Wavelength-scanning pixel-super-resolved lens-free on-chip quantitative phase microscopy with a color image sensor

We report a wavelength-scanning-based lens-free on-chip microscope using a color CMOS sensor and a matching modified phase retrieval algorithm for pixel super-resolution. Compared to traditional monochrome industrial cameras, color sensors favored by the consumer electronics industry have smaller pixel sizes, higher performance, and lower costs. However, the color filtering array (CFA) introduces inherent modulation to the holograms acquired under quasi-monochromatic illumination, which complicates the data processing in lens-free on-chip microscopy. Without physically removing the CFA positioned on the sensor chip, we demonstrate quantitative phase imaging (QPI) with a lateral half-width resolution of 615 nm over a wide field-of-view of 51.88 mm 2 by exploiting the green-channel data from Bayer-masked holo-grams. The resulting spatial bandwidth product is 137.2 megapixels, over 10 times that of a conventional optical microscope. The rationale for using only green-channel data is that the information from each sampling point is not lost during propagation but rather distributed to all pixels in the image. Therefore, the missing data in other channels can be recovered by exploiting the sufficient differences among the raw images captured at different wavelengths. Compared to the scheme with monochrome sensors, this method requires the acquisition of several more images to guarantee the convergence of the algorithm. Experimental results show that we can achieve high-quality QPI performance, thus demonstrating the applicability of cost-effective color sensors in the field of lens-free holographic microscopy.


I. INTRODUCTION
For decades, optical microscopy has been a central tool in fields as diverse as engineering, physics, medicine, and biology.Nevertheless, microscopic imaging of biological cells and tissues remains an active research field due to the low contrast of biological specimens when using bright-field microscopy.Quantitative phase imaging (QPI) is a powerful, label-free method for high-resolution and highcontrast imaging, providing non-invasive observation methods for biomedical experiments. 1,2On the other hand, the pursuit of wide field-of-view (FOV) and high-resolution imaging has been a constant endeavor in the development of microscopic technology. 3owever, optical microscopy is constrained by the conflict between numerical aperture and FOV, which restricts the spatial bandwidth product (SBP). 1Recently developed computational microscopy techniques provide new opportunities for high-resolution phase imaging over a large FOV, such as synthetic aperture holography, 4 transport of intensity equation (TIE), 5,6 Fourier ptychography microscopy (FPM), [7][8][9] differential phase contrast (DPC) microscopy, 10 and lens-free on-chip microscopy (LFOCM). 11,12Among them, LFOCM has drawn the attention of researchers thanks to its simple optical design and low-cost device requirements, making it a promising technique for high-throughput QPI imaging.
Based on the principle of in-line holography, an LFOCM system contains only an illumination source, a sample, and a sensor.The incident coherent beam diffracts when crossing the sample and free propagates forward, and then the intensity is recorded on the sensor.3][14][15] To overcome this problem, many pixel super-resolution (PSR) methods have emerged, including multi-angle illumination, 16 active parallel plate scanning, 17 axial scanning of the sample-to-sensor distance, 13,18 and wavelength scanning. 19,20Among them, the wavelength-scanning method with a medium-step interval 20 can achieve both phase retrieval and resolution enhancement while eliminating mechanical displacement and enhancing system stability, making it suitable for long-term live cell observation.
In the meantime, the ongoing digital revolution has brought us cheaper image sensors with smaller pixels, better dynamic ranges, and higher performance. 21Color optoelectronic sensors are the dominant detectors in consumer electronic devices, such as smartphones, webcams, and digital cameras, with nearly 7 × 10 9 annual sales. 22Given the rapid development of the consumer electronics industry, the pixel size of the color sensors has shrunk significantly, reaching below a micron.In contrast, the pixel size of the monochrome sensors is generally larger due to weaker market demand.The use of cost-effective and powerful color sensors has driven the improvement of various applications, such as point-of-care microscopic imaging.
In LFOCM systems, monochromatic sensors are preferable for recording holograms thanks to the quasi-monochromatic illumination.However, most monochrome sensors on the market fall short when it comes to pixel size and cost efficiency.Therefore, a pressing challenge for LFOCM technology is to achieve highresolution phase reconstruction using readily available, low-cost, and high-performance color sensors.Based on previous research, the application of color sensors in lens-free systems can be divided into three categories: (1) using color sensors for color pathological section imaging or color fluorescence imaging, [23][24][25][26] (2) manually removing the color filtering array (CFA) layer to obtain small-pixel monochromatic sensors, 15 and (3) applying numerical methods to perform Bayer demosaicing based on experimental calibration of the light intensity response curves for each color filter. 27Among these, manually removing the CFA layer is very time-consuming and laborious, which is prone to damaging the sensor.In addition, using numerical methods, the Bayer-masked images are difficult to decouple because of the crosstalk among different channels and the nonlinear response to varying illumination intensities, resulting in an ill-posed problem. 28n this paper, we propose an LFOCM platform using a color sensor and a matching wavelength-scanning iterative phase retrieval algorithm.Unlike the color-imaging methods, our method aims to measure the quantitative phase of the sample accurately by exploiting the phase modulation induced by the variation of the illumination wavelengths.By using only the green-channel data without physically removing the CFA layer or performing Bayer demosaicing numerically, we can achieve phase reconstruction with pixel super-resolution.In comparison with our earlier lens-free on-chip platform 13,17,20 using relatively older generation monochrome sensors, the current system uses a color sensor with a larger active imaging area and smaller pixel sizes, significantly improving the space bandwidth product of our lens-free imaging platform.

II. METHOD A. Lens-free microscopy setup
Our experimental setup is based on a typical LFOCM system, as shown in Fig. 1(a).The wavelength-tunable illumination source consists of a supercontinuum spectrum laser (YSL SC-Pro) and an acousto-optic tunable filter (AOTF, YSL AOTF-Pro, tunable wavelength range 430-1450 nm, bandwidth 2-9 nm, tuning interval 1 nm) [Fig.1(b)].After being spatially filtered by the pinhole (∼100 μm), the quasi-monochromatic spherical wave propagates a sufficiently long distance (Z 1 ∼ 150 mm) to approximate a plane wavefront.The sample is illuminated, and the resulting in-line holograms are captured by a color CMOS sensor (7716 × 5360, the imaging source, DFK AFU420-L62, pixel-size 1.12 μm) with Bayer patterns (one red, one blue, and two green pixels) [Fig.1(c)].To circumvent the problem of artifacts generated by the CFA, we extracted only the green-channel data to reconstruct experimental results.To reduce the interference caused by crosstalk from the red and blue channels, we selected a wavelength range of 490-580 nm, matching the peak response range of the green channel.This range also corresponds to the interval between the intersection points of the blue-green and those of the green-red quantum efficiency curves of the sensor, as indicated in the product manual.Experimentally, we processed only two green channels for each unit of the Bayer pattern.

B. Pixel super-resolved phase retrieval algorithm based on color sensors
Different from the traditional wavelength-scanning phase retrieval algorithm based on full spatial information, the approach was modified to achieve pixel-super-resolved phase reconstruction using only the green pixels of the mosaiced holograms.The distance from the illumination source to the sample (Z 1 ) is large enough such that the illumination wavefront conforms to the plane-wave approximation.The illumination wavelengths used in the experiment were recorded in a sequence, denoted as {λi, i = 1, 2, 3, . . ., T} (T is the number of wavelengths), and the image sensor recorded the corresponding low-resolution holograms {I i cap }.Furthermore, the auto-focusing algorithm is used in advance to calculate the distance between the sample and the sensor. 29,30Figure 2 depicts the flowchart of the reconstruction method, which is divided into three main steps: Step 1: Extract the green-channel data.In the wavelength range we employed (490-580 nm), the pixels coated with the green filters have the maximum response.The green-channel data {I i cap_G } are extracted for reconstruction, and the mathematical expression is written I i cap_G = I i cap ⋅ MaskG, where MaskG is determined by the Bayer format.In particular, we constructed a MaskG matrix by assigning 1 to the green pixel positions in the Bayer mask and 0 to the other positions.
Step 2: Generate an initial guess.The raw data {I i cap_G } are first normalized to the average intensity to compensate for the inhomogeneity of illumination intensity and sensor response efficiency at different wavelengths, producing a sequence of normalized images {I i G } with consistent green-channel intensity.All of the images in {I i G } are interpolated, bilinearly upsampled (the upsampling ratio is set to 4 in this paper), then backpropagated to the object plane, and finally averaged to generate a high-resolution estimated object field U 1 o corresponding to λ 1 .
Step 3: Iterative phase retrieval and PSR.This is the key step of the reconstruction flow, including six sub-steps.
where j is the imaginary unit and arg(⋅) is the operator to obtain the argument.
α is the relaxation parameter controlling the feedback from the previous estimate.
(5) Backpropagation and conversion to the next wavelength: The updated complex amplitude U i s_update on the sensor plane is backpropagated to the object plane to obtain the updated object field U i o_new .Here, we assumed that the dispersion of the sample is independent of the illumination wavelength, which is a reasonable assumption for most weakly scattering biological samples. 31Accordingly, when converting to the next wavelength λ i+1 , the amplitude is considered constant (i.e., A i+1 o = A i o_new ), and the phase component should be changed proportionally (ϕ i+1 o = (λi/λ i+1 )ϕ i o_new ).In addition, the next complex amplitude is denoted as Notably, the wavelength conversion is performed based on the unwrapped phase, so ϕ i o_new requires an additional two-dimensional phase unwrapping operation. 19,32takes 5-20 iterations to converge (depending on the sample complexity and data quality), and the converged complex amplitude U i o provides the lateral distribution of the measured sample with high resolution at wavelength λi.

III. SIMULATION AND EXPERIMENTAL RESULTS
The phase retrieval algorithm is based on the principle of constrained substitution in both the spatial and spectral domains.The solution to the phase recovery problem depends on the correlation and variation of the data.Due to diffraction, the information of a point on the object is distributed to different pixels at different wavelengths.Hence, acquiring more diffraction patterns with variations at different wavelengths can overcome the signal loss caused by under-sampling and Bayer coding.To determine the required number of wavelengths, we performed numerical simulations with parameters consistent with those used in our experimental system (1.12 μm pixel size, 500 μm sample-to-sensor distance, 490-580 nm wavelength-scanning range).We used a standard resolution target (1951 USAF) to quantify the resolution improvement and extract Group 9 features for better definition [Fig.3(a)].We set the downsampling factor to four when generating holograms in the forward model.We simulate the reconstruction process using all pixels from the low-resolution holograms (i.e., holograms captured by a monochrome sensor).It shows that only four raw images at different wavelengths are used to achieve a two-fold resolution improvement up to Group 9 Element 6 [548 nm line width, Fig. 3(b)].In comparison, using only the information from the green pixels, four input images can only be reconstructed up to Group 9 Element 4 [691 nm line width, Fig. 3(c)].The distribution of green pixels in staggered intervals implies that their equivalent pixel size is √ 2 times the real pixel size. 33In addition, the half-width of 0.691 μm surpassed the equivalent pixel size (1.584 μm) by approximately a factor of 2. As a result, the simulation outcomes presented in Fig. 3(c) can be considered reasonable and consistent with the aforementioned observations.As shown in Fig. 3(d), increasing the number of input images to eight essentially enables the resolution of Group 9 Element 6 in the reconstruction.Further increasing the number of inputs to 12 frames enhanced the reconstruction quality and reduced the discrepancy with the ground truth [Fig.3(e)].
Figure 3(f) illustrates the variation of mean squared error (MSE) with the number of iterations in the reconstruction process for different cases.The inset therein shows the profile line plots for Group 9 Elements 4-6.It can be seen that increasing the number of wavelengths can significantly improve the imaging resolution and reduce the steady-state error, producing converged QPI results.The simulations confirmed that enough mosaiced holograms at different wavelengths provide sufficient data variation for consistent reconstructions with full spatial information.Due to the inherent challenges posed by pixel aliasing and mosaic artifacts in the algorithm, a set of 12 images was utilized to safeguard against potential limitations and uncertainties that may arise during the recovering process.In later experiments, we used 12 wavelengths from 490 to 580 nm with a 7 nm step.Then we used a self-developed C++ program to control the AOTF and the sensor to work in collaboration with the whole image acquisition process, lasting about 2 s.
To quantify the resolution of our LFOCM setup based on a color sensor and validate our improved PSR phase retrieval method (Fig. 2), we performed experiments on a quantitative phase target (QPT) etched on glass.In Fig. 4  To demonstrate the applicability of our platform for biological imaging, we performed live-cell imaging experiments.In visualization 1, we provide a time-lapse video of HeLa cells cultured in vitro across the full FOV with multiple enlarged regions.The full-FOV reconstruction at 02:40 in the video is demonstrated in Fig. 5(a).Figure 5(b) shows the division process of a cell [corresponding to ROI 1 in Fig. 5(a)], which spanned nearly 80 minutes.Before the division, the mother cell shrank into a clump (00:01), and then chromatin gathered at the cell girdle to form a bulge (00:28).At time point 01:02, the chromosomes divided into two groups and moved to opposite sides of the cell, and the cell was elongated.At 01:04, the cytoplasm split, and the cell wall contracted and fell off (01:08).Finally, two new daughter cells were individualized and spread out (01:20). Figure 5(c) illustrates the phase image of ROI 2, which contains three cells (labeled as Cells A, B, and C).The cells exhibit high-resolution features in the image.Based on the reconstructed QPI results, we can implement a digital multimodal display without any additional hardware.As an illustration, Figs.5(d1)-5(d3) present the phase contrast image, the differential interference contrast (DIC) image, and the pseudo-3D morphology of Cell C shown in Fig. 5(c).The tunneling nanotubes (TNTs) connecting cells are thought to be an important pathway for the transport of genetic and biochemical information between distant cells.We observed the formation and disappearance of single filamentary bridge (SFB) connecting two cells [Figs.5(e1)-5(e2)].Moreover, using the phase information, we can perform a quantitative analysis of the changes in stem mass during cell culture [Fig.5(f)].And Fig. 5(g) shows the trajectories of the cells in ROI 2. These experimental results demonstrated that the LFOCM systems with color sensors can achieve high-resolution, high-throughput, and long-term QPI of live cells in culture.It shows that color sensors can be applied to microscopic instruments to provide greater information throughput, displaying a bright future in higher-content and more economical observation methods for live cell culture.

IV. DISCUSSION AND CONCLUSION
Traditional demosaic methods usually require pre-measuring the light intensity response curves at different wavelengths for crosstalk removal during data processing. 27,34Furthermore, they need extra lateral sub-pixel shifts to achieve PSR and axial modulation of the sample-to-sensor distance for phase recovery.For another, wavelength-scanning techniques in LFOCM systems are often used to replace axial scanning for multi-height based phase retrieval 35 or to achieve PSR with narrow spectral range scanning (e.g., 10-30 nm) instead of lateral sub-pixel displacement. 19In contrast, we use wavelength scanning within a relatively wide range and achieve pixel-super-resolved phase retrieval 20 based on only the green-channel data, avoiding complex processes such as mechanical displacement, illumination response measurement, and crosstalk removal.
In summary, we have proposed a wavelength-scanning PSR phase retrieval technique for the LFOCM platforms employing color sensors without any pre-processing or physical removal of the CFA on the sensor chip.The use of high-performance color sensors allows us to achieve higher spatial resolution because of the smaller pixel size, while significantly reducing the cost of experimental setups.In our method, when the illumination wavelength is tuned in the sensed range of the green channel (490-580 nm), a series of Bayermasked in-line holograms are captured accordingly.Through the modified iterative method, we can obtain the pixel-super-resolved QPI using only the green pixels of raw measurements for intensity constraint.The experimental results of QPT demonstrated a lateral half-width resolution of 615 nm across a full FOV of 51.88 mm 2 .The corresponding SBP is 137.2 megapixels, exceeding 10 times that of a conventional optical microscope.Compared to monochrome sensors, lens-free systems utilizing color sensors compensate for the loss of spatial sampling due to the Bayer mask by extending data acquisition.A possible direction for future work is to combine the data from the red and blue channels with the green-channel data during the reconstruction process.This could either reduce the number of input measurements or enhance the quality of the reconstruction.Furthermore, we will refine the physical model of the sensor by experimentally measuring the accurate pixel point spread function 36,37 to enhance the imaging resolution.In forthcoming developments, the application of color sensors with reduced pixel size holds the potential for achieving high spatial resolution and facilitating the exploration of high-resolution color imaging techniques.

SUPPLEMENTARY MATERIAL
See the supplementary material for visualization 1.

FIG. 1 .
FIG. 1.(a) Diagram of the optical setup.(b) Components of the wavelengthtunable source.(c) The photograph of the color sensor with Bayer patterns and the corresponding efficiency curves at different wavelengths.

( 1 )FIG. 2 .
FIG. 2.Flowchart of the reconstruction algorithm for the modified wavelength-scanning PSR only using the green pixels.

( 3 )
Intensity constraint based on the green-channel data: The green-channel data of the image field intensity is updated based on the intensity constraint imposed by the green pixels of the captured low-resolution hologram: I i update = I i G_up /I i s_bin ⋅ I i s , where I i G_up is the upsampling result of I i G .I i s_bin is calculated from I i s_G by pixel-binning and then upsampling with the nearest neighborhood interpolation.(4) Adaptive update scheme: The updated values of the green channel [in sub-step (3)] are combined with the values of the un-updated channels with an adaptive strategy to obtain the updated complex amplitude on the sensor plane:

( 6 )FIG. 3 .
FIG. 3. Effect of the number of wavelengths simulated on the resolution of this method.(a) The USAF target used for simulation.(b) Reconstruction of all pixels using four wavelengths.(c)-(e) Reconstruction results of the USAF target using the different number of wavelengths using green pixels.(f) The MSE vs. iteration numbers.In the inset of (f), the profiles corresponding to the white lines in the reconstruction results in (b)-(e), respectively.

FIG. 4 .
FIG. 4. The experimental result of a QPT.(a) A QPT area includes the elements of groups 6-7: the raw image (upper) acquired directly by the color sensor and the interpolated version (bottom).For the boxed area, both mosaic and interpolated images are enlarged for display.(b) Reconstruction using backpropagation.(b1) The enlarged view of Groups 8-9 corresponds to the white-boxed area in (b).(c) Super-resolution reconstruction using the proposed method.(c1) The enlarged view corresponding to the white-boxed area in (c).(d) A bright-field microscope image of Groups 8-9 (40×, 0.75 NA) for visual comparison.

FIG. 5 .
Figure3(f) illustrates the variation of mean squared error (MSE) with the number of iterations in the reconstruction process for different cases.The inset therein shows the profile line plots for Group 9 Elements 4-6.It can be seen that increasing the number of wavelengths can significantly improve the imaging resolution and reduce the steady-state error, producing converged QPI results.The simulations confirmed that enough mosaiced holograms at different wavelengths provide sufficient data variation for consistent reconstructions with full spatial information.Due to the inherent challenges posed by pixel aliasing and mosaic artifacts in the algorithm, a set of 12 images was utilized to safeguard against potential limitations and uncertainties that may arise during the recovering process.In later experiments, we used 12 wavelengths from 490 to 580 nm with a 7 nm step.Then we used a self-developed C++ program to control the AOTF and the sensor to work in collaboration with the whole image acquisition process, lasting about 2 s.To quantify the resolution of our LFOCM setup based on a color sensor and validate our improved PSR phase retrieval method (Fig.2), we performed experiments on a quantitative phase target (QPT) etched on glass.In Fig.4(a), the upper panel displays a ARTICLE pubs.aip.org/aip/appraw image of the QPT acquired directly by the color sensor under 520 nm wavelength, and the interpolated version is shown in the bottom panel.The magnified views of the selected area [white box in Fig. 4(a)] before and after interpolation are presented for comparison.Figure 4(b) shows the phase image obtained by directly backpropagating the interpolated hologram to the focusing plane, suggesting that only a half-pitch resolution close to the pixel size of the image sensor [Group 8 Element 5, 1.23 μm, in Fig. 4(b1)] can be achieved.Through our proposed algorithm, the resolution of Group 9 Element 5 (615 nm line width) can be reconstructed experimentally [Figs.4(c) and 4(c1)], reaching 2.58 times the equivalent pixel size (1.584 μm).The reduced resolution compared to the simulation results may be due to sensor noise.Figure 4(d) provides a conventional bright-field microscope image (40×, 0.75 NA) of the same region for comparison.The region of Group 9 Element 5 is enlarged, and the line profiles are given correspondingly [as shown in Fig. 4(c)].The breakages of the QPT in the Group 9 Element 6 region are also shown comparatively.