Computational single-objective scanning light sheet (cSOLS)

Single-objective scanning light sheet (SOLS) using oblique plane (OP) illumination techniques has pushed the limits of image-based biological studies such as the quantification of single-molecule dynamics in living cells¹ to calcium signaling in neuronal circuits in living mice.² All existing single-objective scanning light sheet systems require costly and more complex remote imaging units that comprise one or two complementary objective lenses (secondary and tertiary objective lenses)³ to achieve diffraction-limited imaging and optical sectioning. Additional elements, such as tailor-made objective lens, tilted mirror, or diffraction grating, are required to effectively image an oblique 3D sample slice on a 2D imaging sensor.^4–6 These remote focusing units are incompatible with conventional epifluorescence detection paths with a single tube lens and a 2D camera sensor.⁷ As such, single-objective scanning light sheet systems [e.g., eSPIM (epi-illumination Selective-Plane Illumination microscopy) and SCAPE (Swept Confocally-Aligned Planar Excitation microscopy)] are only limited to specialized microscopy setups.^1,2

Lightfield⁸ is a special class of single-shot volumetric fluorescence imaging that is directly compatible with a standard epifluorescence imaging. Lightfield leverages computational imaging that performs 3D depth retrieval using a single 2D lightfield image data. Because of this, lightfield imaging has been successfully integrated into conventional epifluorescence detection.^8,9 Lightfield imaging systems have recently reached unprecedented recording speeds under ultra-low light¹⁰ and highly scattering conditions,^9,11 expanding the imaging field of view into the millimeter range at single-cell imaging resolution.¹² There are a wide variety of computational lightfield tools ranging from ray optics interpolation⁸ and deconvolution iterations¹³ to deep learning reconstruction.¹⁴ All lightfield computational tools are designed to identify 3D information (x, y, z) of an object based on the angular disparity (r, θ) that is encoded within lightfield images generated by the microlens array (MLA).⁸ 

In conventional epifluorescence detection, 3D sample imaging requires sequential axial scanning. In a SOLS system, the functional use of a remote imaging unit is to (1) image an oblique 3D sample slice without sequential axial scanning and (2) fulfill the Sine and Hershel conditions for spherical aberration-free imaging over depth.¹ Fluorescence emitted from an oblique 3D sample slice is of varying angles along the inclined axial plane.³ This means that OP lightfield images alone possess angular disparities of the fluorescence points. This meant that axial fluorescence points can be retrieved so as to identify oblique 3D sample slices without remote imaging unit or sequential axial scanning. In SOLS, the remote imaging unit fulfills the Sine and Hershel conditions. On the other hand, a computational SOLS that uses a microlens element may no longer match Sine and Hershel conditions. As the result, a cSOLS may operate with a reduced axial imaging range. On the contrary, cSOLS will open up new computational capabilities such as computational adaptive optics to achieve isotropic resolution¹⁰ that is not present in any SOLS systems. In principle, an SOLS system could just require a lightfield imaging unit (a single micro-lens array and a 2D imaging camera) that is fully compatible with conventional epifluorescence detection and replaces the bulky remote imaging unit in SOLS.

However, we observed that OP lightfield images present a computational challenge to process using current lightfield depth retrieval tools. Figure 1 aims to illustrate this challenge. Figure 1(a) shows single-objective lens OP illuminating (blue line) over a single 1 µm fluorescence microsphere (red circle) at different angles (α₁ = 0°, α₂ = 30°, and α₃ = 60°). Figure 1(b) shows the corresponding lightfield images. Using lightfield depth retrieval tools,⁸ we retrieved the point spread function (PSF) from each of the LF images. Figure 1(c) shows that the retrieved PSFs from all three distinctively different angles are almost identical. The result shows that the effective imaging PSF (PSF_imaging) is incorrectly represented. This is because PSF_imaging with an oblique beam is expressed by multiplying OP illumination PSF (ObliquePSF_illumination) with detection PSF (PSF_detection) as shown in the following equation:¹⁵ 

Based on the three different illumination angles, the depth retrieved PSF_imaging, in Fig. 1(c), must present a skewed intensity profile² (gray ellipse). Because lightfield detection results in a lower spatial resolution compared to the diffraction-limit resolution of the objective lens,¹³ we anticipate that PSF_detection has a wider spatial extent than a confined ObliquePSF_illumination of thin thickness. This means ObliquePSF_illumination can result in significant spatial modulation over PSF_imaging in the axial direction through Eq. (1). An incorrect PSF_imaging would, therefore, result in inaccuracy in depth retrieval and results in poor Richardson–Lucy deconvolution.¹⁶ It is, therefore, necessary to restore PSF_imaging. Similar to photon reassignment,¹⁷ we proposed that this can be achieved computationally by reassigning fluorescence signal onto an OP illumination plane.

In this Letter, we proposed to build a computational image reassignment model for OP illuminated lightfield systems called Fast Optical Ray (FOR) model. To distinguish the FOR model from lightfield depth retrieval methods,^8,13,14 we term our computational approach as lightfield depth mapping. FOR model lightfield depth mapping is a ray optics algorithm that rapidly identifies a 3D spatial map of any OP illumination plane. The fluorescence signal from each OP lightfield image is then computationally mapped into their respective spatial positions in 3D (x, y, and z) at two planes per second. The OP illumination is generated using a scanning oblique plane illumination (SOPi) setup that offers greater flexibility to adjust different OP illumination planes.¹⁸ For lightfield detection, we constructed a scheme with an unfocused lightfield detection where a microlens array (MLA) is placed at the image plane of a conventional epi-fluorescence microscope. To ensure lightfield imaging without overlapping lenslet images, we underfilled each microlens by having an imaging-side NA smaller than the NA of the MLA (Thorlabs WFS150M-5C). With the 1.3NA objective lens (Olympus UPLFLN100XOI) used in our system, the maximum oblique angle achieved in our system is ∼60°. By limiting the beam diameter using an iris, the OP illumination possessed an experimentally determined thickness of ∼1 µm at the beam waist. Details of the system can be found in the supplementary material, Sec. S1.

Before we introduce the experimental results, we shall first explain the theoretical outline of the FOR model illustrated in Fig. 2(a). The FOR model uses ray transfer matrix analysis, where individual rays of a voxel V are mapped from the object space to the sensor plane S as shown in the following equation:

where the voxel’s spatial coordinates are x, y, z, rays emitted from the voxel are indexed by N, a ray’s coordinates on the aperture of the objective lens are given by x′, y′, a ray’s coordinates on the sensor plane are x, y, the free space transfer term is [P], and the thin lens transfer terms for the objective lens, tube lens, and MLA are defined as [L_obj], [L_tube], and [L_MLA], respectively. The FOR model performs 4 × 4 ray transfer analysis to model XZ and YZ propagations of rays from a voxel V to the sensor plane S. We model XZ and YZ propagations separately using two sets of 2 × 2 ray transfer matrices. The results are then combined to express a ray’s 2D divergence angles from the optical axis. For simplicity, Fig. 2(a) illustrates the XZ propagation paths and ray transfer terms. V is formed by the discrete sampling of the object space using lateral and axial sampling factors δxy and δz. δxy and δz together represent the smallest voxel in the object space that the FOR model can analyze. Using the focal point of the objective lens as the origin, a dataset consisting of voxels V in the object space with spatial coordinates x, y, z (multiples of δxy and δz) relative to the focal point is generated. By evenly distributing N rays emitted from the voxel V on the aperture of the objective lens with spatial coordinates x′, y′ where the rays enter the imaging system from different angles, the FOR model samples the angular information of signal from the voxel V. By indexing each unique ray with N, $SN$ represents the lateral coordinates of the ray $VN$ on the sensor plane.

In Fig. 2(a), we showed an OP (blue line, denoted by K₂) that is detected on the imaging sensor after the MLA. Next, we determine the expected pixels that would create the following lightfield image. The following equation describes how the sensor plane S is mapped to the lightfield image LF,

where the sensor’s pixel size is U, and the lightfield image pixel coordinates are l, m. For illustration purposes, Fig. 2(a) shows four light rays $V1$ to $V4$ that are focused through the MLA onto the camera sensor at $S1$ to $S4$ and recorded as $LF1$ to $LF4$ . We then used Eq. (4) to calculate the expected intensity of the respective voxel,

The detected intensity I is the summation and averaging of all rays $V1$ to $V4$ . Since a camera sensor samples an image discretely, bilinear interpolation is applied at this step to construct each voxel intensities. Figure 2(b) illustrates how the FOR model maps OP illumination in object space to form a volumetric image. Figure 2(b)(i) shows diagonally tilted voxels across a single volume that are excited by three OP illumination slices (K₁, K₂, K₃). The FOR model identifies the illumination slice, which is then used to determine the corresponding fluorescence intensities (for details, see the supplementary material, Sec. S2). Figure 2(b)(ii) shows that the chosen fluorescence intensities are then rearranged in separate columns of pixels. For a given illumination slice, we plot the respective OP images P shown as P₁, P₂, P₃ in Fig. 2(b)(ii). Oblique fluorescence intensities are then mathematically reassigned back into the 3D positions in four different Z-slices as shown in Fig. 2(b)(iii). The reassignment process is given in the following equation:

where $Pkz$ is the set of OP images and Z_z is the slice at depth z. Both OP images and Z-slices have pixel coordinates i, j. $kz$ is described in the supplementary material, Sec. S2, which is used to locate the OP image with index k, where pixel i, j is to be extracted from.

To fulfill the Nyquist sampling criterion in our discrete FOR model, we set δxy and δz as 0.25 and 0.5 µm, which are more than four times smaller than the theoretical lateral and axial resolution 1.35 and 2.16 µm of our lightfield detection (see details in the supplementary material, Sec. S1). The FOR model with fine lateral and axial sampling allows accurate determination of S_x,y and, therefore, I_x,y,z. Details on the impact of sampling density on FOR model depth mapping can be found in the supplementary material, Sec. S3. The current implementation of the FOR model utilizes paraxial ray transfer matrix analysis and, therefore, is subject to propagation inaccuracies associated with ray optics. However, this allows the FOR model to be highly efficient and adaptive to different optics including objective lenses and MLAs as paraxial thin lens modeling does not rely on the detailed parameters of optics (e.g., refractive index, surface curvature, thickness). To optimize for speed, we pre-compute the FOR model to generate a dataset of S_x,y of the lightfield detection. S_x,y applies to all lightfield images captured for different samples. This allows real-time depth mapping and image reassignment to be performed in MATLAB at 2 OP lightfield images per second. For imaging a volume of 40 × 40 × 14 µm³ with 60° OP illumination that is captured with 50 OP illuminated lightfields, current CPU-based (AMD Ryzen 7 5800X, 32 GB RAM, non-parallel processing) achieves FOR model depth mapping in 25 s. This could be shortened by 25 times after implementing multicore parallel processing, achieving full 3D depth mapping at 1 volume per second. In comparison, for the same imaging volume and the number of lightfields, lightfield depth retrieval based on deconvolution iteration¹³ takes ∼40 minutes to compute (see the supplementary material, Sec. S6). We anticipate that deep learning-based lightfield methods can achieve multiple volumes per second, however, requires a sophisticated model training and validation process.¹⁴ Computational considerations of FOR model generation and real-time depth mapping are summarized in the supplementary material, Sec. S4.

Next, we experimentally validate FOR model depth mapping for an OP illumination angle of 60° and compare it against lightfield depth retrieval based on ray optics.⁸ We used 1 µm diameter fluorescence microspheres (Polysciences Fluoresbrite YG Microspheres) that are below the resolution limit of the optical system. The supplementary material, Sec. S5, outlines FOR model parameters used for processing. Figure 3(a) shows the 1 µm microsphere retrieved without FOR model depth mapping showing a significant amount of out-of-focus blur in both XZ and YZ planes. However, the same microsphere processed by FOR model depth mapping demonstrates a greatly reduced out-of-focus signal by twofold in the vertical direction as shown in Figs. 3(b) and 4. Using FOR model depth mapping, we acquired the effective PSF_imaging with vertical FWHMs (1.39 and 1.37 µm) in XZ and YZ that are close to the theoretical limit of the system, which is 1.35 µm with a profile almost identical to the ideal PSF_imaging and in doing so achieves accurate depth sectioning of the microsphere. In Fig. 3(c), while lightfield depth retrieval and FOR model depth mapping demonstrate comparable XY FWHMs of 1.94 and 1.92 µm in the vertical direction, the effective PSF_imaging acquired by FOR model depth mapping shows an asymmetry profile in the lateral direction of 1.59 µm. Noting that the asymmetry is aligned with the OP illumination sample scan direction in the lateral axis, it is possibly an artifact associated with the sample scanning or data interpolation. We anticipate that this artifact can be resolved by the OP illumination sample scan at finer steps and improved interpolation strategy (e.g., cubic interpolation) during FOR model depth mapping and image reassignment.

To further validate our model on densely packed samples, we performed imaging on the customized laser-written rigid fluorescence microstructure, illustrated in Fig. 5(a) (not drawn to scale). Using multiphoton lithography, we constructed a microstructure consisting of letters “A” (red) and “U” (blue) stacked vertically on a glass coverslip and by supporting structures (green). The 3D fabrication process is described in the supplementary material, Sec. S8. For comparison, we performed deconvolution on images acquired by FOR model depth mapping using the captured PSF_imaging and compared with a lightfield depth retrieval method based on wave optics deconvolution iteration.¹³ Parameters for lightfield deconvolution can be found in the supplementary material, Sec. S6. Figure 5(b) shows XY slices containing letters “A” and “U” at z = −2 µm and z = 2 µm, respectively. This gives a small separation of 4 µm along the z axis between “A” and “U” to test axial sectioning. From Fig. 5(b), both lightfield depth retrieval and FOR model depth mapping appear to show the letters at their designated axial position accurately, excluding any out-of-focus intensity arising from the other letter located 4 µm away in z. However, upon closer examination of the YZ slice [Fig. 5(c)] across the center of the microstructure [blue dashed plane in Fig. 5(a)], we observed that lightfield depth retrieval, Fig. 5(c)(i), only eliminated half of the out-of-focus cones, while depth mapping, Fig. 5(c)(ii), shows no out-of-focus cones. Figure 5(d) plots the normalized axial intensity across the letter “A” [white lines in Fig. 5(c)] and quantifies the out-of-focus region. We found that lightfield depth retrieval results in an out-of-focus region extending over 8 µm (FWHM) in depth. Conversely, FOR model depth mapping achieves accurately mapped to z = −2 µm with an axial profile (FWHM) of 1.9 µm resulting in sharp axial fluorescence signal of the letter “A.” The results of FOR model depth mapping on densely packed fluorescence microstructures show significant improvement in depth sectioning over lightfield depth retrieval, as shown in Fig. 6. However, observed in Fig. 5(b) qualitatively that FOR model depth mapping demonstrates inferior lateral resolvability of the microstructure to the lightfield depth retrieval method.

In conclusion, we demonstrated a novel computational single-objective scanning light sheet (cSOLS) imaging technique enabled by our FOR model depth mapping lightfield algorithm. With the remote imaging unit removed, FOR model depth mapping allows SOLS imaging to be performed on conventional epifluorescence detection scheme. cSLOS can immediately be deployed in handheld SLOS ¹⁹ because of simplified detection scheme. cSOLS with FOR can be integrated into a wide range of existing high throughput imaging systems for high contents screening with the addition of a new OP illumination module. We also confirmed that conventional lightfield depth retrieval methods fail to retrieve OP illuminated images. FOR model depth mapping achieves an effective PSF_imaging with axial FWHMs (1.39 and 1.37 µm) that are close to the lateral resolution limit (1.35 µm) of our lightfield detection.⁸ As shown in Table I, cSOLS with FOR achieved spatial resolution that is comparable to existing single-objective light sheet systems of comparable effective NA.^20,21 We anticipate that the enhancement of optical sectioning achieved by FOR model depth mapping can immediately benefit high-contrast imaging of live developing zebrafishes.¹⁶ FOR model depth mapping achieves optical sectioning on densely packed structures that existing lightfield algorithms cannot.²² By adopting MLA-based lightfield detection, SOLS imaging can benefit from technological advances in lightfield computational imaging. cSOLS with FOR can be further tailored to the structured OP beam such as Airy beam light sheet for potential large field of view (FOV), high-contrast imaging with improved resistance to scattering in biological samples.²³ Implementing a two-photon light sheet can also improve depth penetration when imaging thick biological samples. cSOLS with FOR has the potential to adopt computational adaptive optics for aberration correction without “guide star” or additional wavefront sensing and, in turn, promises imaging deep inside samples.¹⁰ The FOR model can also accommodate multidimensional OP illuminated lightfield spectral imaging²⁴ and is able to computationally resolve multi-direction OP illumination for multi-view light sheet imaging for highly scattering samples.²⁵ A key area of improvement in the FOR model is the remaining 7.5% mismatch between ObliquePSF_illumination and PSF_detection due to experimental uncertainty in determining oblique angles, which can be improved using additional calibration steps.¹⁸ Furthermore, current FOR model depth mapping requires prior manual calibration of the illumination profile for effective optical sectioning. This calibration process can be automated by performing wavefront sensing of the illumination at the back focal plane of the objective lens. The current implementation of FOR model depth mapping considers OP illuminations of a uniform thickness for simplicity, which is not a realistic model as the Gaussian beam diverges in regions outside of the beam waist. The effect of beam divergence could be addressed in the FOR model by incorporating variable OP illumination thickness as a next step when adopting a large FOV. Here, we validated cSOLS with FOR to have a spatial resolution of 1.59 × 1.92 × 1.39 µm³ over an imaging depth range of 14 µm, which is lower than traditional SOLS that uses a remote focusing unit to satisfy the Sine condition (supplementary material, Sec. S7). The limited imaging depth is because of the focusing properties of MLA. Recent advance in liquid crystal MLA with tunable focusing and numerical aperture can improve the imaging depth of cSOLS.²⁶ cSOLS with FOR also has anisotropic spatial resolution across depth, which is a factor inherent to lightfield imaging.^9,13 This can be overcome by complete space multiscale lightfield modeling⁹ and computational adaptive optics¹⁰ to achieve FOR cSOLS imaging with aberration correction and isotropic resolution. We anticipate that FOR model depth mapping will adopt these computational methods to extend the imaging depth to 180 µm. Although the spatial resolution of cSOLS with FOR is around half of high-NA SOLS systems,⁴ this can be overcome by adopting scanning lightfield techniques¹⁰ or Fourier lightfield detection²⁷ to reach ∼300 nm laterally and ∼700 nm axially that are comparable to both high-NA SOLS and orthogonal light sheet systems. An important future step for the FOR model is to adopt wave optics-based propagation and deconvolution to minimize propagation inaccuracy, increase the spatial resolution, and increase the signal-to-noise ratio. MLA-based lightfield detection is subject to the trade-off between spatial resolution and depth resolvability, which ultimately limits biological imaging capabilities. This can be alleviated by adopting MLA and a camera with finer pitch and pixel size. We envision that cSOLS using MLA with tunable optical properties²⁸ and computational adaptive optics¹⁰ will reach subcellular imaging resolution with improved depth resolvability. Recent Compute Unified Device Architecture (CUDA) processing will also accelerate FOR model depth mapping volume rate to up to 30 volumes per second.^8,29 Finally, we expect the FOR model to complement existing lightfield deep-learning methods to reach sub-cellular imaging resolution with simplified signal processing pipelines and at hundreds of volumes per second.¹⁴ 

See the supplementary material for detailed methods. Section S1 describes details of the optical system. Section S2 describes the modeling of scanned oblique plane illumination in the FOR model. Section S3 shows the impact of sampling on FOR model depth mapping performance. Section S4 outlines computational consideration of FOR model depth mapping. Section S5 summarizes FOR model depth mapping parameters for experimental data processing. Section S6 describes parameters for lightfield deconvolution. Section S7 shows resolution and imaging depth comparison with SOLS approaches. Section S8 describes the multiphoton lithography fabrication process.

The authors acknowledge the Australian Research Council (Grant Nos. DE160100843, DP190100039, DP200100364, and CE140100011) for financial support.