Deep image restoration for infrared photothermal heterodyne imaging

Infrared (IR) absorption spectroscopy and microscopy are widely used techniques to detect and identify chemical compounds. This is due to characteristic vibrational transitions that occur in the mid-infrared (MIR) “fingerprint” region of the electromagnetic spectrum. For nearly all optical microscopies and spectroscopies, however, spatial resolution is fundamentally limited to approximately half the wavelength of light by the Abbe diffraction limit. Consequently, in the MIR, a practical spatial resolution is of order ∼5 μm.^1,2 The significance of this is that mid-infrared microscopies/spectroscopies are unsuitable for studying compounds and materials under conditions free of averaging effects. Obtained images do not resolve individual absorbers within illuminated regions while obtained spectra represent the average response of a sampled subensemble. This is especially the case when specimens reside in spatially congested environments.

A number of techniques have been developed to overcome this limitation. They include IR scanning near field optical microscopy (IR-SNOM) and IR-assisted atomic force microscopy (AFM-IR).² The former involves using synchrotron radiation to obtain high brightness broadband IR light.³ This limits the technique’s general applicability as access to a synchrotron is required. Extracting IR-SNOM spectral information is also convoluted by sample scattering and absorption. AFM-IR, in contrast, provides high spatial resolution (<20 nm) images with easy-to-extract spectral information. However, the technique relies on the thermal expansion of samples upon absorbing IR light. This restricts its use to materials with relatively large thermal expansion coefficients.⁴ 

A recently developed, all-optical, table top IR microscopy/spectroscopy circumvents many above-mentioned issues. Called IR photothermal heterodyne imaging (IR-PHI),^5–7 it operates in a pump/probe fashion and uses a MIR pump laser to induce local heating in specimens. Resulting photothermal changes are detected using a visible wavelength probe laser. In turn, the use of visible probe wavelengths means that IR-PHI’s spatial resolution is dictated by the visible diffraction limit. Consequently, IR-PHI’s spatial resolution can be an order of magnitude smaller than the MIR diffraction limit.

As described by Pavlovetc et al.,⁷ IR-PHI signal contrast for measurements conducted on specimens surrounded by air predominantly originates from thermally induced changes to sample backscattering cross sections (σ_backscat). For particles with radius $>50$ nm, dominant contributions to Δσ_backscat originate from temperature-induced changes to specimen refractive indices, as captured by its thermo-optical parameter $∂n∂T$⁠. Consequently,

Although heat flow occurs into the surrounding medium, associated IR-PHI contrast contributions are small, given the much smaller thermo-optical coefficient of air [i.e., $∂nair∂T=−8.6×10−7$ vs $∂nparticle∂T=−1.3×10−4$].^8,9 Consequently, IR-PHI contrast predominantly originates from thermally induced changes to the sample, as opposed to the surrounding medium when a significant difference in specimen/surrounding medium thermo-optic coefficients exists.

In practice, IR-PHI has a demonstrated spatial resolution of $∼300$ nm.^10–13 It has been used to image and conduct sensitive MIR spectroscopic measurements on a variety of material systems. This includes individual live cells,¹⁴ single viruses,¹³ semiconductor thin films,^10,15 environmental matrices,^16,17 and many biomolecular systems.^5,18,19 Like AFM-IR, this technique takes advantage of photothermal changes induced in specimens upon absorbing MIR light to generate optical contrast. Unlike AFM-IR, though, IR-PHI is suitable for studying condensed phase samples where large changes in morphology are not expected. Case in point is the recent application of IR-PHI to study the localized MIR plasmon resonances of individual gold nanowires.¹¹ 

Although IR-PHI’s spatial resolution is readily an order of magnitude better than that of conventional IR microscopes, it is still limited by the probe diffraction limit. IR-PHI images are also negatively impacted by diffraction-induced image blurring as well as background artifacts, acquired during raster scanning. These issues are not unique to IR-PHI and exist in images acquired using other optical imaging modalities, such as conventional fluorescence microscopy.^20–22 Improvements to IR-PHI’s spatial resolution are therefore required to expand its capabilities so that it can reliably detect and image individual objects in spatially congested environments.

A readily accessible means of improving IR-PHI’s spatial resolution entails image processing to deconvolute its point spread function (PSF) from images. This is generally accomplished using Fourier transform techniques as evidenced by image deconvolution algorithms present in many popular software packages, such as ImageJ,²³ scikit-image,²⁴ and their subpackages. In the absence of direct PSF deconvolution, alternative approaches to improving the spatial resolution of microscopy images involve multiphoton,²⁵ structured illumination,²⁶ stimulated emission depletion,²⁷ or single molecule localization microscopies.²⁸ While these latter approaches enhance resolution from $2$ to 50 times, they rely on single molecule fluorescence detection along with high excitation intensities. This ultimately limits their applicability to photosensitive and/or nonemissive materials. For the topic at hand, this prevents their immediate use in the MIR region of the spectrum.

Image deconvolution is therefore pursued as a general means of enhancing the spatial resolution of IR-PHI images. The primary challenge with image deconvolution, however, is the exact modeling of various image degradation processes, which occur during a measurement. In addition to diffraction-induced blur, background noise and optical aberrations degrade IR-PHI images. These degradation contributors must be modeled properly lest they introduce new artifacts when images are processed and restored.^29,30 Prior work has therefore focused on modeling/addressing specific types of image degradation mechanisms.^31,32 Unfortunately, no general approach exists to simultaneously address all degradation mechanisms.³³ This has spurred recent efforts in implementing deep learning algorithms to address the broad issue of image degradation in microscopy.^34–37 

Deep learning is a data-driven methodology, where a convolutional neural network (CNN) is trained to learn an image restoration function by leveraging big data.³³ Accurate models as well as knowledge of particular noise types are no longer required. This approach has already delivered promising results. For example, Shajkofci and Liebling³⁴ used a CNN to predict parameters in a spatially variant PSF model. Following Richardson–Lucy (RL)^31,32 deconvolution, deblurred optical microscopy images were recovered.

Of particular note is an advanced network architecture, referred to as cycle-consistent general adversarial network (CycleGAN),³⁸ that has drawn significant attention within the microscopy deconvolution community. CycleGAN not only trains a network to achieve image deconvolution but also trains a second network to recover desired deconvoluted/deblurred images using the output of the first network. Lim et al.³⁶ improved CycleGAN by replacing its second network with a known linear PSF to effectively remove blur artifacts encountered in fluorescence microscopy. Lim et al.’s network thus represents the state of the art in image restoration and will hereafter be referred to as optimal transport driven CycleGAN (OTD-CycleGAN).

Deep learning networks can be divided into two types. The first trains a network to directly restore high resolution images.^39,40 This type of network requires pairs of low and high resolution images produced using different imaging techniques. Restricting the approach, though, are blur artifacts present in said high resolution images or, alternatively, the inability to acquire them. The second network type leverages prior information about an instrument’s PSF and learns suitable deconvolution operations for yielding artifact-free restored images.^36,37 Restricting this approach are output spatial resolutions limited by the image resolution of initial input images.

In this study, a deep learning CNN, employing image restoration (i.e., deconvolution, deblurring, and denoising), is adopted to improve both the spatial resolution and image quality of IR-PHI images. Elements of both deep learning network types are utilized wherein IR-PHI’s PSF is estimated a priori using experimental images of IR-absorbing particles with known physical sizes below the MIR diffraction limit. This PSF is utilized in a loss function that is part of the network. In tandem, synthetic datasets, consisting of ground-truth and degraded ground-truth image pairs, are created. Employed ground-truth images are theoretical high resolution images of studied IR-absorbing particles. Degraded ground-truth images are ground-truth images with IR-PHI’s PSF applied to deliberately degrade said images. These image pairs are then used to train the network.

A multi-scale structure, hereafter called a generator, is now developed to process experimental IR-PHI images. In the generator, initial low resolution images, containing measurement-induced artifacts, are iteratively passed into a network structure called U-Net⁴¹ and an upsampling block. The U-Net removes blur/background artifacts while the upsampling block increases image resolution fourfold. What results are high quality deblurred/denoised IR-PHI images that exhibit twofold improvements in feature resolution. This makes possible application of IR-PHI to probing materials residing in spatially congested environments.

The IR-PHI pump consists of a pulsed, tunable, mid-infrared optical parametric oscillator (OPO; M Squared Lasers) (λ = 5.4–9.6 μm, $ν̃=1040–1840cm−1$⁠, and 20 kHz repetition rate). A continuous wave diode-pumped 532 nm solid-state laser is used as the probe. Pump and probe lasers are arranged in a counterpropagating geometry and are brought to focus on opposing sides of a specimen. The pump is focused using a reflective objective [Ealing, 0.65 numerical aperture (NA)], while the probe is focused using a standard high NA refractive objective (Nikon, 0.95 NA). Specimens sit atop a closed loop piezo stage (Mad City Labs) that raster scans samples under the mutual foci of pump and probe beams during image acquisition. Specimens consist of dilute solutions of materials drop cast onto CaF₂ coverslips (25.0 × 0.5 mm² thickness, Crystran).

IR-PHI image sizes are limited by the scanning range of the piezo stage (300 μm). In a typical measurement, 50 × 50 μm² areas are scanned (pixel dimensions: 100 × 100), where the piezo step size is 500 nm. For a 10 × 10 μm² area (pixel dimensions: 100 × 100), the piezo step size is 100 nm. Corresponding lock-in integration times range from 10 to 100 ms with an associated pixel dwell time three times the length of the lock-in integration time and a software overhead time of ∼25 ms. Associated (total) image acquisition times therefore range from 24 to 36 min. Measurements are performed using pump and probe intensities of I_pump ∼ 6–300 MW m⁻² and I_probe ∼ 22 GW m⁻², respectively.

At each point of a scan, backscattered probe light from the specimen is collected with the refractive objective and is sent to a homebuilt autobalanced photodetector.¹ The autobalanced photodetector is referenced using a fraction of the probe light split off prior to the IR-PHI microscope. The autobalanced photodetector’s output is then sent to a lock-in amplifier (Zurich Instruments, MFLI), referenced to the OPO repetition rate. Lock-in-extracted signals are plotted using a home-written Python data acquisition program. More details about the IR-PHI instrument can be found in Ref. 2.

For network benchmarking purposes, commercial polystyrene (PS, Sigma-Aldrich) and polymethylmethacrylate (PMMA, Magsphere, Inc.) particles were purchased from their respective vendors. Particle mean radii ranged from 60 to $∼1100$ nm with each ensemble possessing a 20% size distribution (r_actual = 60 ± 10, 300 ± 50, and 1100 ± 350). Such ensembles are referred to as 60 nm PS, 300 nm PMMA, and 1100 nm PS hereafter. PS and PMMA particles were both imaged at 1450 cm⁻¹, which corresponds to the asymmetric bending modes of their methylene (CH₂) and methoxy (O–CH₃) groups, respectively.

Samples for IR-PHI imaging of real specimens consisted of micro- and nano-plastics (MNPs) leached from commercial nylon teabags.⁴² Dilute suspensions of MNPs were prepared by steeping store-bought and emptied teabags in ultrapure deionized water at both room temperature and 95 °C. Nylon MNPs were imaged at 1637 cm⁻¹ on resonance with their amide I resonance. Additional details regarding nylon MNP samples can be found in Refs. 16 and 17.

For fluorescence microscopy measurements, samples consisted of commercial, 50 nm radius, dye-doped PS particles (Molecular Probes, F8800) drop cast onto glass coverslips. Specimens were imaged using the above IR-PHI instrument, modified to conduct fluorescence measurements. In particular, 532 nm light was focused onto specimens using the 0.95 NA refractive objective. Induced emission was collected using the same objective with a 550 nm long pass filter (Chroma) introduced to block any scattered 532 nm light. Filtered emission was then focused onto a single photon counting avalanche photodiode (Perkin Elmer, SPCM-AQR-14) connected to both a counter and a pulse to analog voltage converter. Signals were then read using a universal serial bus (USB)-based data acquisition card (National Instruments).

Scanning electron microscopy (SEM) images of samples were acquired using a FEI Magellan 400 SEM with an accelerating voltage of 1 kV.

The developed generator uses a three-scale U-Net to generate restored artifact-free images with a fourfold increase in image resolution. Image resolution refers to the center-to-center distance between adjacent pixels in an image. Each scale beyond the first successively enhances image resolution twofold using upsampling. Upsampling is a computational process wherein original images are enlarged to twice their original size using bilinear interpolation to fill in gaps between the original pixels.⁴³ In whole, each scale of the network removes image blur and noise at a given image resolution to produce sharp deblurred/denoised images. These images can be fed into subsequent scales for further image enhancement. Details of the utilized network structure are provided in the supplementary material along with visual schematics (Figs. S1–S6).

Weights in the CNN are crucial to its performance. An iterative optimization process, called gradient descent, is therefore used to find optimal values for these weights, which collectively minimize a loss function.⁴⁴ Updated weights are equivalent to their original value minus a gradient of the loss function, taken with respect to the weight in question. In general, the loss function measures the distance between the network’s output and a ground-truth image.

For the current study, the employed loss function is a combination of four distance metrics: mean squared error (MSE) loss, gradient loss, recovery loss, and classification loss. MSE and gradient loss are used to measure the similarity between the ground-truth image and the network’s output. Recovery loss measures the similarity between the network’s input and its output where the latter has been convolved with the instrument’s PSF. Classification loss measures the similarity of a binary thresholded image of the network’s output to the ground-truth image.

The developed network requires training using thousands of ground-truth and degraded ground-truth image pairs to establish optimal CNN weights. A data generation framework thus produces synthetic datasets, consisting of theoretical high resolution IR-PHI images (referred to as ground-truth images) and corresponding degraded ground-truth images. Degraded ground-truth images are obtained by degrading ground-truth images through application of (a) experimental PSF-induced blurring, (b) addition of Gaussian noise, and (c) fourfold downsampling. The synthetic data generation framework is summarized in Algorithm 1 of the supplementary material. Training and testing of the three-scale network was performed on a Nvidia 1080-Ti Graphics Processing Unit (GPU). The resulting CNN recovers high resolution IR-PHI images in 40 ms on the same Nvidia 1080-Ti GPU. Software for the three-scale network is available in GitHub (https://github.com/Agchai52/MicroscopyDeblur_IR-PHI).

Figure 1(a) shows a representative, large area, 4 × 8 μm² IR-PHI image of individual 300 nm PMMA particles. Single particle identification is made based on the employed dilution of particle stock solutions. Analogous images of 60 nm PS, 300 nm PMMA and, 1100 nm PS can be found in Fig. S7 of the supplementary material.

Figures 1(b), 1(e), and 1(h) show 3 × 3 μm² zoomed-in IR-PHI images of individual 60 nm PS, 300 nm PMMA, and 1100 nm PS particles. Subsequent linescan analyses of image full width at half maxima (FWHM) yield apparent sizes of r_degraded = 346, 381, and 916 nm, respectively. Extracted single particle sizes follow the Abbe diffraction limit wherein particles with sizes smaller than the PSF adopt sizes dictated by the PSF.⁴⁵ This is most apparent in the 60 nm PS specimen where a fivefold increase in apparent size is observed. To conduct network deconvolution of IR-PHI images, Fig. 1(b) is used to approximate IR-PHI’s PSF. This is because the imaged particle’s actual size is significantly smaller than the probe laser’s diffraction limit.

Figures 1(c), 1(f), and 1(i) show restored images from the developed three-scale network. In all cases, dramatic enhancements to the image resolution as well as clear improvements to image sharpness are observed. This is especially apparent in Fig. 1(i) where additional structure to the particle is revealed.

Figures 1(d), 1(g), and 1(j) summarize achieved resolution improvements via before and after linescans of individual particles. Linescans (blue dashed lines) are taken along the center horizontal of individual particles and reveal FWHM reductions of 56%, 37%, and 3% for the 60 nm PS, 300 nm PMMA, and 1100 nm PS particles, respectively. Recovered back-estimated sizes are r_restored = 152, 240, and 890 nm. These values generally agree with above-mentioned sizes and ensemble size distributions. For the 60 nm PS particle, the recovered size is noticeably larger than what would be expected from an ideal deconvolution of the instrument’s PSF. This is a consequence of approximating IR-PHI’s PSF using the image of a finite sized particle [Fig. 1(b)] rather than an infinitely small point source.

Other sources contribute to the observed deviations. From an imaging standpoint, input image resolution (i.e., the choice of step size during piezo raster scanning) limits network recovery, especially given that it impacts experimental approximations of the instrument’s PSF. Choosing a smaller step size will, in principle, improve experimental rendering of the PSF and, by extension, the fidelity of network-restored images. However, given that data collection times increase quadratically with decreasing step size, other factors, such as instrument drift, likely come into play to adversely impact image quality. Chosen step sizes, therefore, reflect a trade-off between realizable experimental resolution and practical constraints to PSF image quality.

To assess the three-scale network’s performance, comparisons to other image deconvolution algorithms are now made. Figure 2 shows results of deconvoluting IR-PHI’s PSF from Figs. 1(b) and 1(e) [here, Figs. 2(a) and 2(b)] using the classic Richardson–Lucy (RL) algorithm^31,32 as well as Lim et al.’s³⁶ OTD-CycleGAN. The former RL approach is a standard deconvolution strategy whereby (input) degraded images are divided by estimated PSFs in the frequency domain to isolate desired images. PSF-free images are then reverse Fourier transformed to obtain corresponding real space deconvoluted images. In practice, the main drawback of the RL algorithm is its implicit assumption that shot noise dominates all other noise sources in measurements. Consequently, extrinsic noise contributions to image degradation are amplified in RL-restored images.

Results of the RL algorithm are shown in Figs. 2(c) and 2(d), where deconvolution has been performed using the RL deconvolution algorithm in scikit-image,²⁴ an open-source image processing library for the Python programming language. Restored images show improvements in image resolution with all single particle images showing reductions in apparent particle size. For both 60 nm PS and 300 nm PMMA particles, FWHM reductions of 21% are seen in Figs. 2(i) and 2(j). At the same time, it is evident that background artifacts are amplified. Comparing the background of restored images with those of the initial degraded images reveals that sharp artifacts are introduced by RL deconvolution. As suggested earlier, this occurs because RL amplifies extrinsic background noise.

Figures 2(e) and 2(f) show results of OTD-CycleGAN.³⁶ Evident are improvements in restored images where for the 60 nm PS and 300 nm PMMA particles, FWHM reductions of 20% and 29%, respectively, are realized [Figs. 2(i) and 2(j)]. Beyond this, the deep learning network produces cleaner backgrounds, especially when compared to Figs. 2(c) and 2(d).

Comparing results from RL deconvolution and OTD-CycleGAN with those from the three-scale network [Figs. 2(g) and 2(h)] reveals that the latter readily surpasses the former two in terms of resolution enhancement. FWHM reductions of 56% and 37% are realized, yielding restored particle sizes in closer agreement with r_actual for the 60 nm PS and 300 nm PMMA particles [Figs. 2(i) and 2(j), respectively]. This is especially evident for the 60 nm PS particle (r_actual = 60 ± 10 nm) where the three-scale network yields r_{restored, three-scale} = 152 nm vs r_{restored, RL} = 270 and r_{restored, OTD-CycleGAN} = 262 [Fig. 2(i)]. Background restoration is also better as there is no amplification of artifacts in three-scale network-restored images.

Between networks, performance differences primarily stem from differences in scale. OTD-CycleGAN employs a U-Net scale factor of one, whereas the three-scale network employs three. OTD-CycleGAN additionally lacks an upsampling block. Restored images, therefore, preserve the original image resolution of degraded images. The three-scale network, by contrast, uses bilinear interpolation to increase image resolution by an overall factor of four.

To demonstrate the performance of the three-scale network on IR-PHI images of real specimens, IR-PHI measurements have been conducted on MNPs leached from commercial nylon teabags. It is known that anthropogenic MNPs are ubiquitous in the environment and result from polymer breakdown.^46–48 Recently, MNPs have been discovered in solutions produced from steeped commercial nylon teabags with concentrations up to 1.5 × 10¹⁰ particles per teabag.^16,17,42

MNP impact on environmental and human health is now being investigated. However, hindering advances in the field is a lack of suitable analytical techniques that are simultaneously non-destructive, quantitative, sensitive and which possess both high spatial resolution and chemical specificity. Efforts are therefore underway to apply IR-PHI towards studying MNPs via bond-selective imaging.² 

Figure 3(a) shows a SEM image of nylon MNPs leached from a commercial nylon teabag, following room temperature steeping. Evident are irregularly shaped particles, many possessing fiber-like morphologies. This is highlighted by the accompanying SEM image of a single fiber [Fig. 3(b)], which shows that individual fibers possess irregular surface features, likely due to local variations in polymer morphology and density. Estimated room temperature-leached MNP lengths are of the order of 1–10 μm.

Figure 3(c) shows a 6 × 4 μm², 1637 cm⁻¹ IR-PHI image of a single MNP (FWHM = 986 nm). Accompanying larger area IR-PHI images are provided in the supplementary material (Fig. S8). The image reveals variations in IR-PHI intensity, attributed to local “hotspots,” arising from temperature inhomogeneities induced in the fiber. These temperature inhomogeneities stem from local variations in particle morphology, density, and, by extension, heat capacity [Fig. 3(b)]. Using IR-PHI’s previously estimated PSF [Fig. 1(b)], the input degraded image is processed using the three-scale network. For comparison purposes, the image is also processed using the RL algorithm and OTD-CycleGAN.

Figures 3(d)–3(f) show results of the processing. As with earlier polymeric particles, all algorithms improve the feature resolution of images. Comparison of perpendicular linescans (Fig. S9) show FWHM reductions of 1%, 14%, and 18% using the RL-algorithm, OTD-CycleGAN, and the three-scale network, respectively.

In general, OTD-CycleGAN [Fig. 3(e)] increases contrast and suppresses noise in restored images. The three-scale network [Fig. 3(f)], however, performs better in both regards. This is highlighted by parallel linescans shown in Fig. 3(g) (white dashed lines). Furthermore, the three-scale network performs best among the approaches in suppressing background noise.

A caveat that must be considered when imaging large objects with sizes comparable to or larger than the PSF is heat diffusion. In Fig. 3(b), this specifically refers to irregular surface features that arise from intraparticle structural heterogeneity. Because such features are large and are essentially embedded within a medium having identical/near identical thermo-optical parameters—unlike the case of small isolated particles surrounded by air—heat diffusion manifests itself as an intrinsic “blurring” of features not readily accounted for by any of the above restoration algorithms. Thus, while the three-scale network clearly improves the resolution of observed features, its use of a PSF, based on a small isolated particle surrounded by air, does not fully restore images in this case.

Figure 4 shows the result from IR-PHI measurements conducted on MNPs extracted from 95 °C-steeped nylon teabags. Figure 4(a), in particular, shows a SEM image of leached MNPs. Evident are numerous irregularly shaped particles. Figure 4(b) shows a 5 × 5 μm² IR-PHI image of the same sample acquired at 1637 cm⁻¹.

Figures 4(c)–4(e) show restored images of Fig. 4(b) using RL deconvolution [Fig. 4(c)], OTD-CycleGAN [Fig. 4(d)], and the three-scale network [Fig. 4(e)]. In all cases, imaged MNP sizes are reduced. A linescan comparison (Fig. S10) of these approaches for the topmost MNP (highlighted by a green box, apparent size 425 nm) shows size reductions of 28%, 39%, and 46%, respectively. As before, background noise is enhanced by the RL algorithm, whereas it is suppressed using both network approaches.

Finally, a clear performance difference exists between OTD-CycleGAN and the three-scale network. In Fig. 4(b), the input IR-PHI image shows five small MNPs (numbered). Although both OTD-CycleGAN and the three-scale network successfully recover small and barely distinguishable MNPs (e.g., particles 1 and 5), background noise is fully suppressed using the latter network. This is due to its use of a classifier block that effectively distinguishes signal from noise.

The developed three-scale network is broadly applicable beyond IR-PHI and can be trained to enhance the spatial resolution of other optical microscopies where spatial resolution is limited by the Abbe diffraction limit. To demonstrate this, the three-scale network is applied to improve fluorescence microscopy images. Shown in Figs. 5(a) and 5(c) are fluorescence images of 50 nm radius dye-doped PS microspheres. Additional images are shown in Fig. S11. Estimated particle sizes from linescan profiles (Fig. S11) reveal apparent single particle sizes of r = 181 ± 25 nm. Figure S12 shows the single particle fluorescence image used as the measurement’s PSF.

Figures 5(b) and 5(d) show corresponding three-scale network restored images. In all cases, evident are sizable $∼58$% decreases in r to r_{restored, three-scale} = 75 ± 20 nm. Also evident are improved backgrounds free of artifacts. Figure S11 compares before and after restoration single particle linescans. Full network recovery of actual particle sizes is prevented by use of an experimental PSF. Summarizing, Figs. 1–5 demonstrate the broad applicability of deep learning algorithms to enhance the quality of diffraction-limited optical microscopy images.

In this study, a deep learning network has been developed to improve both image and spatial resolution of IR-PHI images. Efficacy of the proposed network has been demonstrated on both control specimens and real samples consisting of MNPs leached from nylon teabags. In all cases, sizable enhancements to IR-PHI images are realized. Twofold improvements in feature resolution are realized from 300 to $∼150$ nm. Network performance has been benchmarked against both the conventional RL algorithm and a more recent deep learning network developed by Lim et al. In either case, the three-scale network with upsampling and classifier blocks performs better, simultaneously improving feature resolution while discriminating against background artifacts in restored images. This and future machine learning efforts therefore portend advances in super-resolution microscopy whether in the visible or, more relevantly, in the MIR region of the spectrum.

See the supplementary material for details of the three-scale network, including U-Net structure, point spread function estimate, classifier structure, synthetic data generation pipeline, and network training details; IR-PHI images of 60 nm PS, 300 nm PMMA, and 1100 nm PS particles; comparisons of RL, OTD-CycleGAN, and the three-scale network; IR-PHI images of MNPs from a 25 °C-steeped nylon teabag solution and linescans of MNPs from a 95 °C-steeped nylon teabag solution; and fluorescence images and linescans of individual dye-doped PS microspheres.

M.K. acknowledges the NSF (Grant Nos. CHE-1563528 and CHE-1954724) for financial support. M.K. also acknowledges the AFOSR under the MURI:MARBLe project (Award No. FA9550-16-1-0362) for partial financial support.

The authors have no conflicts to disclose.

S.Z. and K.K. contributed equally to this work.

The data that support the findings of this study are available within the article and its supplementary material.