Retrieving positions of closely packed subwavelength nanoparticles from their diffraction patterns Open Access

Imaging, localization, and retrieval of the number of subwavelength objects closely packed, although extremely challenging, are problems that are very often encountered in applications, such as environmental monitoring,¹ semiconductor optical inspection,² materials,³ and biomedical analysis.⁴ This problem cannot be tackled by conventional microscopy, which is bound by the Abbe–Rayleigh diffraction limit to a resolution of about half the wavelength of the incident light. Improved resolution can be obtained by using optical techniques such as PALM and STED, which work with photoactivated labels^5–7 or near-field methods,^8,9 which require contact with the sample and are, therefore, unsuitable in many instances because of their complexity and invasiveness.¹⁰ 

It was recently reported that deep learning-enabled analysis of single-shot diffraction patterns of coherent light scattered by subwavelength objects can be used to obtain unlabeled super-resolution optical metrology^11,12 and to correctly predict the number of nano-objects in clusters of subwavelength objects.¹³ Here, we show that AI-empowered analysis of the optical diffraction patterns of closely packed subwavelength nanoholes, using a neural network trained on similar a priori known objects, allows us to retrieve their positions, even when the nanoholes are touching.

We conducted numerical experiments using a coherent plane wave illumination (λ = 633 nm) of clusters of subwavelength nanoholes with a diameter of λ/6.33 perforated in an opaque film, randomly placed within a 2.2 λ × 2.2 λ area. We image the diffraction patterns created by the nanoholes clusters at a distance, H = 1 λ, away from the sample, over a 22 λ × 22 λ field of view, accounting for the numerical aperture, NA = 0.9, of a real imaging system [Fig. 1(a)]. Pairs of the far-field diffraction pattern and the corresponding position map of nanoholes in the cluster were used to train a modified U-Net encoder–decoder convolutional neural network. U-Net is a convolutional neural network (CNN) architecture specifically designed for semantic image segmentation,¹⁴ i.e., categorizing each pixel in an image into a class or object, which continues to generate widespread interest and has found application in medical imaging¹⁵ and optical microscopy.¹⁶ The network has a U-shaped architecture consisting of an encoder, or dimension-reducing path, followed by a decoder or dimension-increasing path, with a symmetric design that reduces the risk of information loss during the encoding and decoding processes.¹⁷ The encoder path captures and condenses information of inputs at multiple levels of abstraction through convolutional and pooling layers, as in traditional CNNs.¹⁸ The decoder path, instead, uses transposed convolutions to recover the dimension, conditioned by skip connections from the correspondingly encoded information at the same level. This approach enables the network to produce precise segmentation masks and effectively addresses the vanishing gradient problem.¹⁹ This phenomenon, where the network loses its capacity to capture long-term dependencies, is mitigated through this design. To promote efficient information propagation and resolve the category imbalance challenge in the particle localization problem (i.e., the small fraction of nanoholes with respect to the background), we modify the U-Net by introducing a residual architecture²⁰ with multiple deep convolutional layers and novel hybrid loss function that combines binary cross-entropy loss and a class-balanced loss (details are in the supplementary material Secs. 1–5). The trained network is then able to retrieve the positions of the particle in the cluster from previously unseen diffraction patterns [Fig. 1(b)].

Each sample contains up to 10 nanoholes that may form clusters with an inter-particle distance smaller than the Rayleigh limit of resolution of a conventional microscope, 0.61 λ/NA. We define the sizes of the sub-Rayleigh clusters by counting the number of nanoholes whose inter-particle distance is smaller than the Rayleigh limit [Fig. 2(a)] and characterize each sample by the largest sub-Rayleigh cluster size within the 2.2 λ × 2.2 λ area. It shall be noted how, often, not only pairs but all particles fall within the Rayleigh distance [e.g., nanoholes A, B, and C in Fig. 2(a)]. The groundtruth maps used to supervise the network are binary images of 512 × 512 pixels size (corresponding to an area of 2.5 λ × 2.5 λ), where white pixels (value = 1) represent the nanoholes and black pixels (value = 0) represent the Cr film [second column in Fig. 2(b)]. The corresponding diffraction patterns were generated plotting the total electric field intensity profiles calculated by finite-difference-time-domain (FDTD) full Maxwell solver (supplementary material Sec. 6), at a distance of 1 λ from the sample surface, over a field of view of 22 λ × 22 λ [first column in Fig. 2(b)]. Figure 2(b) shows that the light propagating through the nanoholes generated very rich interference patterns in the diffraction maps, thus making it very difficult to correlate with a specific number and distribution of nanoholes on the sample. Nonetheless, the trained network can not only retrieve the number and positions of the nanoholes in the clusters but also return 512 × 512 pixels images [third column in Fig. 2(b)], where the sizes of the nanoholes match well those of the groundtruth and, therefore, can be regarded as a form of super-resolution imaging.

Figure 3(a) shows that the accuracy exceeds 0.81 for samples with the sub-Rayleigh cluster index of 3 and remains as high as 0.71 for samples containing ten nanoholes within the same sub-Rayleigh cluster, which is an indication of the trained network's robustness against the size of sub-Rayleigh clusters. The decrement of the image reconstruction accuracy with the increasing size of sub-Rayleigh clusters can be justified by the increase in the complexity of the interference patterns.

To explore further our technique and test its resilience against the problem of closely paced particles, we tested its performance with an example of two closely spaced nanoholes. In this case, we use 800 diffraction patterns of two λ/6.33 nanoholes with center-to-center separation decreasing from 0.677 λ (Rayleigh distance) to 0.158 λ (touching) as a test. The Pearson correlation coefficient calculated in Fig. 3(b) shows that the two nanoholes could be resolved with an accuracy higher than 0.8 across almost the entire range of sub-Rayleigh distances, with a slight drop to 0.75 in the only case of touching nanoholes.

In conclusion, we report on a far-field, single-shot super-resolution optical technique based on the deep learning of the light diffracted on the clusters of subwavelength particles. It allows retrieving maps showing the number, positions, and sizes of the nanoparticles in the cluster and, therefore, constitutes a form of imaging. Our technique is scalable to different wavelengths and film materials as long as the film material is opaque for the wavelength. The image reconstruction accuracy measured as the correlation coefficient between the ground truth and reconstructed maps of the nanoparticles depends on the number of nanoparticles in the largest cluster of sub-Rayleigh spaced particles and varies from 0.82 to 0.71 when the cluster size increases from 2 to 10. In addition, we showed that the technique resolves nanoholes separated significantly smaller than the Rayleigh distance.

See the supplementary material for details on the network (architecture, optimization objectives, hyperparameters, and tailored objective functions) and the dataset distribution used for training, numerical simulations, and performance with noise.

This work was supported by the Singapore National Research Foundation (Grant No. NRF-CRP23-2019-0006), the Singapore Ministry of Education (Grant No. MOE2016-T3-1-006), and the Engineering and Physical Sciences Research Council UK (Grants No. EP/T02643X/1).

The authors have no conflicts to disclose.

Benquan Wang: Conceptualization (lead); Data curation (lead); Formal analysis (equal); Investigation (equal); Methodology (lead); Software (equal); Validation (lead); Visualization (lead); Writing – original draft (equal); Writing – review & editing (equal). Ruyi An: Formal analysis (equal); Software (equal); Validation (supporting); Visualization (supporting); Writing – review & editing (equal). Eng Aik Chan: Formal analysis (equal); Investigation (equal); Project administration (lead); Software (equal); Writing – review & editing (equal). Giorgio Adamo: Formal analysis (equal); Investigation (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal). Jin-Kyu So: Formal analysis (equal); Investigation (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal). Yewen Li: Formal analysis (equal); Investigation (equal); Software (equal); Writing – review & editing (equal). Zexiang Shen: Funding acquisition (equal); Resources (equal); Supervision (equal); Writing – review & editing (equal). Bo An: Funding acquisition (equal); Supervision (equal); Writing – review & editing (equal). Nikolay I. Zheludev: Funding acquisition (lead); Resources (lead); Supervision (lead); Writing – review & editing (equal).

The data that support the findings of this study are available from the corresponding author upon reasonable request.