Epi-fluorescence imaging of the human brain through a multimode fiber

Over the last decade, innovative and unique fiber based endoscopes have been developed, consisting of multicore fibers,^1,2 fiber bundles,^3,4 and gradient index (GRIN) lenses.⁵ Their main application areas are endoscopic medical imaging and biomedical research in challenging surroundings or difficult to reach places. Multicore fiber endoscopes suffer from reoccurring image replicas due to the periodicity of multiple cores,² while honeycomb structured fiber bundles are limited in resolution and exhibit a characteristic pixelation effect.^3,4

The multimode (MM) fiber offers a minimally invasive solution due to its flexibility and small footprint. Its adaptation would, therefore, be of paramount importance when imaging in sensitive surroundings, e.g., brain tissue, organs, and generally in biomedical deep-tissue imaging.^6–8 MM fiber endoscopes have in common that the image forming information is lost due to scrambling of the light within the fiber. Therefore, prior to experimental use, the transmission matrix of the fiber needs to be characterized^9–12 in order to decode and exploit its waveguiding properties. If the transmission matrix is measured, light can be focused through a MM fiber by spatially shaping the wavefront.¹³ This allows us to match a specific output mode at the distal end of the fiber, e.g., focal spots, leading to the raster-scan imaging possibility.^8,10,14

We propose to use the scrambling of the light to our advantage. Coherent light entering at one point at the proximal side of the MM fiber creates a random speckle pattern at the distal end. By altering the entry position at the input side, multiple and uncorrelated random speckle patterns can be created. The application of compressive imaging through a multimode fiber combined with speckle patterns as illumination¹⁵ enables data acquisition below the Nyquist–Shannon limit, while at the same time it might offer the ability to image below the Abbe limit, enabling super-resolution imaging.^16–18 An image is recovered via a computational compressive sensing algorithm.¹⁹ In the field of biological microscopy, the implementation of compressive imaging was, for the first time, reported in 2012, demonstrating widefield compressive microscopy.²⁰ Furthermore, it has been used in many applications, such as multiphoton and nonlinear microscopy,^21,22 fluorescence-lifetime imaging microscopy (FLIM),²³ and recently in photoacoustic-fluorescence microendoscopy.²⁴ 

Here, we visualize lipofuscin accumulations. Lipofuscin consists of yellow-brown, autofluorescent, and electron-dense granules. These granules, usually of the size of 0.1–5 μm,²⁵ accumulate in the cytoplasm of cells, reaching sizes up to 20 μm, and are extensively distributed within brain tissue. Neuronal somas can be found filled with lipofuscin granulates.²⁶ Lipofuscin, often called the age pigment, is known as “metabolic waste,”²⁷ containing residues of lysosomal digestion composed of oxidized proteins and lipids,^25,28 and is, therefore, a depot of unexcreted waste products.²⁹ Due to its complex mixture of chromophores, it exhibits a broad banded auto-fluorescence.³⁰ 

In this Letter, compressive sensing through a lensless MM fiber for brain imaging is shown. We demonstrate high-resolution imaging of Alzheimer’s disease (AD) human brain tissue with lipofuscin accumulations through a single MM fiber probe of only 125 μm in diameter. Lipofuscin accumulates progressively through the normal aging process in post-mitotic neurons, both in humans and in animals, and can be linked to a number of pathological conditions, such as tumors, malnutrition, and, e.g., neuronal ceroid lipofuscinosis.³¹ 

The experimental setup is shown in Fig. 1(a). A digital micromirror device (DMD, Vialux V-9501, 1920 × 1080 pixels, pixel size 10.8 μm) is homogeneously illuminated with a 491 nm (Cobolt 05-01 Calypso 200) linearly polarized laser source. A binary amplitude grating is projected on the DMD, and the first diffraction order is filtered and guided into a 20× objective [Olympus Plan, 0.4 NA (O1)] to couple the beam into the MM fiber (Thorlabs, FG050LGA, 0.22 NA, core diameter: 50 μm, cladding diameter: 125 μm). At the distal end of the fiber, a 40× objective [Olympus Plan N, 0.65 NA (O2)] is used to image the output (speckle patterns) of the fiber onto the camera (Basler ace acA1440-220 μm) through a tube lens (L3) of f = 200 mm. The camera is used during the calibration process prior to experimental image acquisition (see Sec. II B). The sample is then placed between the output facet of the fiber and objective O2. The auto-fluorescence signal is measured through the same fiber in epi-direction and deflected by the dichroic mirror [Semrock FF509-FDi02 (DM)] into the detection arm. Two filters (F) are embedded (Semrock FF01-496/LP-25 and Semrock FF518-Di01) in the detection arm to suppress subtle reflections of the pump beam from the fiber input facet and to filter out low energy Raman signals from the silica fiber probe. The entire auto-fluorescence signal is collected with a single area photomultiplier tube (PMT) detector (Thorlabs PMT2101/M).

Computational compressive imaging is based on sparsity. However, the biological samples investigated here are not represented sparse as well separated points in space. Therefore, the whole reconstruction procedure has to be optimized for brain tissue imaging. Here, we investigated several reconstruction algorithms to find a solution to the underdetermined compressive sensing problem: bpdn, the conventional Basis Pursuit with denoising,³² l1_ls,³³ tveq_logbarrier,³⁴ and the total variation minimization algorithm tval3.³⁵ They mainly differ in the way how the next point during the optimization process is chosen. For all solvers, we set non-negative constraints and used the ℓ₁ norm.

l1_ls tries to find a sparse solution for x using the minimum l1-norm that satisfies the incomplete least-square equation Ax = b, where A is our illumination matrix (speckle patterns) and b the observations (measured intensity per speckle pattern).

To be more precise, l1_ls optimizes the following problem: $minAx−b22+λx1$⁠, with λ > 0 as the regularization parameter and $x1=∑in|xi|$ as the l₁-norm of x. Since the l1_ls solver is prone to noise and its reconstruction results often pixelated,¹⁷ we investigated additional denoising reconstruction algorithms. Similar to l1_ls, the bpdn solver tries to find a solution to the underdetermined least-square problem, but only approximates the solution as trade-off between best least-square fit and the l₁ norm.

Furthermore, we tested two commonly used total variation algorithms. We chose the isotropic TV + model for tval3: $minx∈Rn∑iDix2$⁠, s.t. Ax = b, x ≥ 0; with D the discrete gradient vector as described in detail in Ref. 35. We set the following parameters to maxit = 1500, TVnorm = 1, nonneg = true, and isreal = true. The remaining parameters use the initial default values. Similar to tval3, the tveq_logbarrier algorithm, used in a previous publication,¹⁶ assumes a sparse gradient to solve the underdetermined problem.

One of the advantages of the tval3 solver is its superior computational reconstruction speed compared to the other solvers.

A snap frozen Alzheimer’s disease (AD) human brain tissue section with a thickness of around 20 μm was obtained from an aged individual [Netherlands Brain Bank (NBB)] and mounted on a CaF₂ microscope slide. This tissue was collected after written informed consent for brain autopsy and the use of brain tissue and clinical information for research purposes. The brain donor program of the NBB is approved by the local medical ethics committee of the Vrije Universiteit Medical Center (Ref. No. 2009/148).

Figure 1(a) depicts the experimental setup. The detailed description is given in Sec. II. We implement two modalities of imaging through a multimode fiber endoscope: raster-scan (RS) imaging via wavefront shaping and speckle-based compressive imaging (SBCI).

Raster-scan imaging through a MM fiber has been realized by active wavefront shaping. A stepwise sequential wavefront shaping algorithm is used¹³ to match the incident wavefront to the scattering properties of the fiber. The Lee amplitude holography method³⁶ is used to create the optimized wavefront by the digital micromirror device (DMD). In close proximity to the distal end of the fiber (around 50 μm), several focal spots (N = 2500) were created, arranged in a square shaped grid with the size of 50 × 50, with a 540 nm pitch.

After this pre-calibration step, the diffraction limited spots (size ∼1.1 μm) are sequentially scanned across the sample to perform imaging. At each spot position, the obtained (auto-)fluorescence intensity is recorded in epi-direction. Ultimately, pixel by pixel, an image can be formed.

The workflow of compressive imaging is illustrated in Fig. 1(b). During SBCI, the wavefront shaping algorithm is not used. Focal spots on the input fiber facet can be scanned by changing the wave vector of a grating projected on the DMD. By sequentially scanning the spots across the proximal facet of the fiber, (pseudo-)random speckle patterns, SPs, are generated at the output facet. Here, we used a grid of 30 × 30 spots at the input to generate M = 900 speckle patterns at the distal end. Thus, the pre-calibration procedure is simply to record the intensity distributions of all the generated speckle patterns and the background signal. The M two-dimensional speckle patterns, captured with a size of N × N, depicted in Fig. 1(b) are saved in an “illumination matrix” A (M × N²), and their corresponding measured intensities are stored as a 1D vector b_bgrnd (M).

Then, the sample is placed at a distance of ∼50 μm from the fiber facet, and the speckle patterns are used to illuminate the sample. The emitted (auto-)fluorescence is collected via the same MM fiber and guided toward the PMT. The total response for each of the 900 patterns is measured and stored as a 1D vector b. All measurements were performed with a laser power in the range of 1–10 μW at the sample. The acquisition rate was set to 15 Hz. Typically, M ≪ N², meaning x is undersampled. The image of the sample can be computed using a reconstruction algorithm to determine x, which satisfies the underdetermined equation b − b_bgrnd = Ax. There are plenty of reconstruction algorithms available, which vary in their computational speed, preconditions, and other parameters.

In the first set of experiments, the experimental configuration was verified using Nile Red beads (ex/em peaks at 535/575 nm, ThermoFisher FluoSpheres) with a diameter of 1 μm placed on a coverslip. The ground truth image and the image recorded in the SBCI setup (reconstructed via ls_l1) are shown in Fig. 1(b). This particular experiment was carried out with a 532 nm laser source. Further analysis regarding the image quality of the raster-scan and compressive imaging technique are shown in the supplementary material. The experimental results for a 1 μm bead phantom and five different image reconstruction approaches including the “gold standard” raster-scan imaging and four reconstruction algorithms are presented in Fig. S1. The resolution of compressive imaging can be twice as good compared to the raster-scan approach. On top of that, the area imaged with SBCI within the same time is around three times larger. The peak-to-background (p2b) ratio for the tval3 algorithm is superior to the raster-scan result (more than an order of magnitude better), as presented in Fig. S2, because tval3 also functions as a denoising algorithm. To summarize, tval3 is the most suitable algorithm for compressive neuroimaging. Therefore, it has been used for brain imaging.

We confirm the presence of lipofuscin in our brain samples, using Sudan Black B,³⁷ also known as auto-fluorescence quencher. Human brain sections were immersed in 0.1 percent Sudan Black B (Sigma-Aldrich) and 70 percent ethanol for 20 minutes at room temperature. The results are presented in Fig. 2. The auto-fluorescence images of the tissue sample were taken with a Leica DM2500 microscope using a DFC450 color camera and a 40× objective (NA 0.8) and presented in Figs. 2(a) and 3(a). The excitation spans 450–490 nm, while the camera recorded fluorescence signal above 515 nm. The dye binds to the lipofuscin granulates, and the lipofuscin accumulations appear now as black (dotted) areas under bright light, matching exactly the bright yellow/orange areas in the auto-fluorescence image of the same area.

In Fig. 3, we present the imaging results for the brain sample. Figure 3(a) shows the auto-fluorescence image taken with a standard high-NA microscope (described above) and used as a ground truth reference. Prior to the endo-microscopy experiments, the fiber has been bleached by scanning the pump beam across the whole core for over 30 minutes with 100 mW. It helps to reduce fluorescence background from the fiber itself and to suppress interference with the actual auto-fluorescence signal from the sample. After this preparation step, the brain imaging through a thin MM fiber with the two methods has been performed.

In the second set of experiments, we demonstrate raster-scan human brain imaging through a MM fiber probe using the 491 nm source. Before placing the sample at the distal fiber facet, the stepwise sequential algorithm was run to generate focal spots at the output facet of the fiber. Next, the sample was placed and scanned point by point, while the auto-fluorescence signal was measured with the PMT. The diffraction limited raster-scan image of the brain tissue recorded through the MM fiber is presented in Fig. 3(b). Excellent agreement between the reference image and the image acquired through the MM fiber is obtained. With a constant acquisition rate of 15 Hz, a total of N = 2500 points were scanned, resulting in a measurement time of 167 s, covering a field of view (FOV) of around 27 × 27 μm.

In the third set of experiments, we implemented the SBCI approach for human brain imaging through a MM fiber. First, the CaF₂ microscope slide was placed below the fiber and the simplified pre-calibration step was run to generate and record M = 900 speckle patterns at the distal end of the fiber and the background signal b_bgrnd. After the pre-calibration step, the sample was placed at the fiber output facet and the same sequence of speckle patterns was projected while recording the overall emission signal collected through the same fiber probe, b. In the reconstruction procedure, b − b_bgrnd has been used. The acquired signal, background, and final subtracted results are presented in Fig. 3(c). The same acquisition rate of 15 Hz was used, resulting in a measurement time of 60 s for the FOV covering the whole fiber core of 50 μm in diameter. The imaging speed could be further improved by replacing a relatively slow DMD with a resonant galvo mirror system.

Subsequently, the recorded patterns and the signal after subtracting background were fed into the reconstruction algorithm. Since the original images of the speckle patterns are the size of a full camera frame of 1440 × 1080 pixels, and therefore too big for computation, the images were cropped to a square of 1000 × 1000 pixels and subsequently down-sampled to a size of 128 × 128 pixels, resulting in a pixel size of ∼0.42 μm. It takes an additional 3.2 s to run the algorithm on a laptop (Dell, 4-Core Intel processor 1.80 GHz, 16 GB RAM). The reconstructed image is presented in Fig. 3(d). The image shows bright fluorescence at the same location as the ground truth auto-fluorescence microscope image. At the same time, it is pixelated and coarse, which is mainly due to the low and noisy recorded signal. For the images presented in Fig. 3, the peak-to-background (p2b) ratio was calculated from the averaged values obtained within the solid green line box (peak) and the dashed yellow line box (background) for all three images. For peak values normalized to 1, background is equal to 0.39, 0.13, and 0 for the ground truth microscope image, raster-scan MMF-based imaging, and MMF compressive imaging, respectively. Further analysis on the quality of the reconstructed images can be found in the supplementary material (Figs. S1 and S2), showing peak-to-background data and obtained resolution (FWHM) when using compressive imaging. Further results are presented in supplementary material Fig. S4 showing images of different N ranging from 32 to 192 while keeping the number of speckles constant (M = 900). This means that the obtained results are of different compression rates.

The results were acquired below the Nyquist–Shannon criterion and offer, therefore, increased acquisition speed compared to the conventional raster-scan methods, when looking at a similar optical resolution. Since only M = 900 speckle patterns were projected, the TVAL3 reconstruction recovers good results with a compression rate of 18 times (128 × 128/900).

Since the image quality partially depends on the number of speckle patterns, we show in Fig. S5 in the supplementary material additional image reconstructions where the overall image size is kept constant but the number of speckle patterns (and hence the compression rate) varies. Even with a very low number of speckle patterns (M = 90, meaning a CR of 182), the object can already be distinguished. Already for M = 225 speckles, the reconstructed image nicely resolves all the main features. The disadvantage of using only a very low number of acquisitions is the decrease in the image quality. Nevertheless, this could potentially be used to “pre-”scan an area very fast and when something was found a more thorough acquisition can be taken.

The low number of illumination patterns helps to increase the imaging speed by about three times (60 s for SBCI and 167 s for raster-scan imaging) while providing an almost four times larger FOV (2800 μm² for SBCI and 729 μm² for raster-scan imaging). In addition, SBCI allows us to reduce photo-damage effects, which is especially important in the case of bio-imaging applications. Further analysis on the quality of the reconstructed images can be found in the supplementary material (Figs. S1 and S2), showing peak-to-background data and obtained resolution (FWHM) when using compressive imaging. In all the experiments presented in this Letter, camera exposure time and photomultiplier tube (PMT) settings were the same for raster-scan and compressive imaging approaches. If one would want to keep the FOV the same, this will lead to a much lower spatial resolution of the raster-scan approach as presented in Fig. S3, where both raster-scan and compressive sensing images are reconstructed from 225 measurements.

Currently, the main challenge is the stability of speckle patterns because even small variations in the imaging patterns compared to the pre-calibration ones introduce artifacts in the reconstructed image. This also holds for the laser pointing stability. A quantitative analysis of the speckle stability and its influence on the imaging quality over 2 h time is presented in the supplementary material, Fig. S6. We can conclude that the relatively good imaging quality (correlation coefficient between the reconstructed image and the ground truth is more than 0.95) could be expected within the first 30 minutes after the calibration procedure. However, due to the long and free “floating” fiber, a long term stability is not in place yet.

In the future, this problem could be overcome by using a combination of a bundle of stable single mode fibers with a fused MM fiber part.

We demonstrated auto-fluorescence brain imaging through an ultimately thin MM fiber probe with two different approaches: wavefront shaping-based raster-scan imaging and speckle-based compressive imaging. We have shown that both methods allow us to visualize lipofuscin—age-related pigment accumulated in brain tissues—with a diffraction limited resolution of 1.1 μm.

To the best of our knowledge, this is the first demonstration of imaging of unlabeled Alzheimer human brain tissues through an ultimately thin MMF-based endoscopic probe. With SBCI, we obtained promising results with compression rates up to 18 times below the Nyquist–Shannon limit, indicating possible future acquisition times at video rate. Sub-diffraction limited compressive imaging initiating possibilities to image even higher details with SBCI.^16,38,39,40 The proposed approach paves the way toward intravital optical neuroimaging with high spatial and temporal resolutions and represents an important step toward the goal of understanding the brain functioning and dis-functioning mechanisms. This method could be used for minimally invasive structural (morphological) as well as functional imaging, including visualization of the dynamic processes of neuronal activity. Moreover, the proposed approach has the potential for medical applications and can, in the future, help to aid medical procedures, for example, entering of thin veins or assisting imaging during epidural anesthesia. Our results lay the groundwork for further experiments on various biological samples and the potential application of robust and flexible endoscopic in vivo (auto-)fluorescence imaging.

See the supplementary material for further imaging quality and speckle stability analysis.

We would like to acknowledge the software support from Marco Seynen (AMOLF). Part of this work was carried out within ARCNL, a public–private partnership between UvA, VU, NWO, and ASML, and was partly financed by “Toeslag voor Topconsortia voor Kennis en Innovatie (TKI)” from the Dutch Ministry of Economic Affairs and Climate Policy. The authors acknowledge support from the Nederlandse Organisatie voor Wetenschappelijk Onderzoek (Grant No. 15872, WISE).

Yes, The authors declare the following competing interests. A patent application, i.e., No. 2021837 (imaging through multimode fibers), based on these results has been filed by VU University, with L.A. and J.F.d.B. as inventors.

The data that support the findings of this study are available from the corresponding author upon reasonable request.