Enhancing hyperspectral EELS analysis of complex plasmonic nanostructures with pan-sharpening

Electron energy loss spectroscopy (EELS) in a scanning transmission electron microscope (STEM) has provided nanoscale hyperspectral analysis of complex nanostructures across multiple materials systems and disciplines.^1–4 In particular, it has seen extensive use in plasmonic nanostructures, where the focused probe can detect resonances directly at the length scales of the plasmonic field enhancement.^5–9 However, the serial nature of STEM-EELS experiments engenders a natural conflict between pixel density/dwell time and experiment duration. For systems with weaker signals or high degrees of nanoscale complexity, the required scan parameters to efficiently sample the entire system can become prohibitive in terms of the stability of the microscope, forcing a trade-off between spatial resolution and spectral fidelity.^10–12 Additionally, for beam sensitive materials that cannot withstand high doses of electron irradiation, short experiments are required to avoid destroying the sample.^13,14

Consequentially, new experimental methodologies enhanced with post-acquisition computational processes have been developed to improve the spectral resolution and fidelity of hyperspectral datasets without sacrificing spatial resolution.^15–20 An alternative approach that has found widespread use in satellite imaging is pan sharpening (PS): a class of techniques where multiple datasets containing different qualities of spatial and spectral information are combined into a single dataset with the beneficial properties of both.²¹ Only recently have PS algorithms been turned to nanoscience, seeing applications in nano-infrared atomic force microscopy experiments²² and optical microscopy.²³ It has also seen direct use in electron microscopy through correlation of microscopy images correlated with secondary ion mass spectrometry.^24–26 However, as of yet, the technique has not been used directly with hyperspectral EELS analysis.

Here, we utilize a PS-driven STEM-EELS methodology, called coupled non-negative matrix factorization (CNMF), to achieve a robust analysis of complex clusters of doped-semiconductor plasmonic nanoparticles with improved spatial resolution and spectral fidelity and reduced electron irradiation and experimental run time. Our CNMF approach is one of several varieties of PS,²⁷ where a high-spatial-resolution (HSR) dataset is combined with a high-spectral-fidelity (HSF) dataset to create a single dataset with the positive aspects of both. Our technique applies NMF to the individual datasets to break them down into discrete spectral (endmembers) and spatial (abundance maps) components.²⁸ We perform a single NMF decomposition on the HSF dataset to access the spectral components with high fidelity and then use those components for a second decomposition on the HSR dataset, resulting in a single dataset with HSR abundance maps and HSF spectral endmembers. The CNMF result outperforms other denoising techniques in both spatial and spectral signal-to-noise ratios (SNRs), while retaining the experimental linewidths of the plasmonic response, reducing the total dose put into the sample and providing a robust guard against noise-induced artifacts. Additionally, we demonstrate that the dose-rate can be even further reduced by defocusing the probe during acquisition, with negligible changes to the PS result.

The plasmonic particles studied here are fluorine doped indium oxide and fluorine/tin doped indium oxide (F:IO and FT:IO), which possess natively infrared plasmonic resonances with narrow linewidths and nanoscale physical dimensions^29–31 (additional details about the nanoparticles is provided in the supplementary material, Figs. S1 and S2). The clusters have a high degree of nanoscale complexity, meaning that a high pixel density is needed to access the true physical response of larger clusters. Additionally, the energy-resolution of EELS is mediated by the elastic scattering peak or the zero-loss peak (ZLP), which is convolved with every peak in the EEL spectrum. Thus, for IR plasmon modes with linewidths in the range of ∼50 meV to 100 meV, a high degree of monochromation is required in order to prevent the ZLP convolution from significantly altering the measured linewidths from their genuine values. This is achieved by cutting out the tails of the ZLP in a pre-specimen monochromator with a physical slit but has the unfortunate side effect of significantly reducing the total probe current. As a result, IR EELS experiments have naturally low signals that necessitate long dwell times to achieve high SNR.³² Ergo, these nanoparticles are a prototypical system for materials that require advanced denoising techniques and an ideal candidate for PS-CNMF, as we cannot sacrifice pixel density or pixel dwell time in the experimental parameters.

Figure 1 shows representative SI slices and spectra for the HSR [Figs. 1(a)–1(c)], HSF [Figs. 1(d)–1(f)], and PS []. Here, the HSR dataset has a pixel density of 150 × 150 and a dwell time of 20 ms, while the HSF dataset has a pixel density of 35 × 35 and a dwell time of 200 ms. We compare two probe positions, one near to a corner of the trimer, where the plasmonic signal is strong, and one ∼50 nm away, where the signal is much weaker (marked by the white squares in the SI slices). The SI slices are taken at 425 meV, the peak plasmon intensity exhibited in the spectra at the two probe positions.

Comparing the HSR, HSF, and PS datasets in terms of the SI slices [Figs. 1(a), 1(d), and 1(g)] and the position 1 spectra for [Figs. 1(b), 1(e), and 1(h)], we can see the dramatic improvement in the signal-to-noise ratio in both the spectral and spatial dimensions of the SI. There is almost no noise visible in the SI slice, and the SNR of the spectrum is even higher in the PS spectrum than in the HSF spectrum. Furthermore, we see that the comparison has retained both qualitative and quantitative accuracy since all three spectra exhibit a double peak structure with a dominant peak at 425 meV and a secondary peak at ∼550 meV, and the PS and HSR spectra match in intensities. While such a match between the three datasets is encouraging, this also means that no new information is gained from the PS process and that these improvements are cosmetic if the HSR dataset has sufficient signal to make out the plasmonic line shapes.

However, in situations where the plasmonic signatures cannot be directly derived from the HSR dataset, we see that PS still works exactly as well. In the position 2 spectra [Figs. 1(c), 1(f), and 1(i)], only the faintest traces of plasmonic excitation is visible in the HSR dataset and peak intensities and linewidths cannot be inferred, but with the long dwell times of the HSF dataset, we can see that the plasmon is still excited at this probe position (albeit with intensity reduced by a factor of 3). Thus, in the PS dataset, we can still easily detect this peak and see that it matches with the HSF dataset qualitatively and with the HSR data quantitatively, enabling accurate retrieval of the plasmonic structure at high spatial resolution in extremely low-signal environments.

It is important to note that a large portion of the denoising comes from the reconstruction of the PS dataset from the most significant CNMF components. Even if one were to take the NMF decomposition of the HSR dataset by itself (without the HSF spectral endmembers) and reconstruct it from its top components, this would significantly denoise the dataset. However, the disadvantage of this strategy is that NMF is a mathematical decomposition and hence has no way to objectively distinguish between a weak signal and noise. As a result, the reconstruction can lose physical information if part of the plasmon response is below the noise threshold in the dataset and noise fluctuations become more mathematically significant than the genuine signal. This is the key advantage of the CNMF approach: the HSF dataset lowers the noise floor and identifies more physical components in the NMF decomposition that can be fed back into the HSR dataset allowing for a more physical reconstruction.

This is demonstrated in Fig. 2 by comparing the NMF decompositions of the HSR, HSF, and PS datasets shown in Fig. 1. Here, we define two types of NMF components: “sample” and “noise.” The “sample” components are the ones where the spectral endmembers and abundance maps are actually connected to real energy loss events in the system and not extraneous effects (such as noise or instrumental instabilities). In simple systems, such components can correspond directly to individual plasmon modes, but in some cases (especially more complex structures), mathematical artifacts can emerge that are still connected to the real plasmonic response of the system but are not real plasmon modes. Such artifacts would include an edge mode and face mode of a cube (which would be spatially overlapped in a projection view), being merged into one double-peaked component, or a red-shift of a large peak manifesting as an NMF component with a smaller, lower energy peak that, when combined with the component for the main peak, represents the red-shifted frequency. While these complex components are more difficult to physically interpret, they are still connected to the genuine plasmonic response of the system and hence provide valuable information. The “noise” components come up in low-signal scenarios, where noise, scanning distortions, and dark references are dominant enough to appear as a separate component. These components are mathematically significant to the dataset as a whole but not to the plasmonic response of the nanostructure. Here, we define “sample” components as ones possessing single (or even multiple) peaks in the endmembers and a clear localization to geometric features of the sample (e.g., edge, corner, interface, and substrate) in the abundance maps and the “noise” components as the ones that do not.

In the two left-most columns, we see the standard NMF linear unmixing of the HSR dataset (i.e., not informed by the HSF endmembers). Here, we can see that even for the weak signal of the HSR dataset, we return some “sample” NMF components (components 1, 3, 5, and to an extent 6), meaning that much of the plasmonic response is still captured without pan-sharpening. However, most of the components (components 2, 4, 7, 8, and 9) are “noise” components. component 2 is a product of the reduced total EELS cross section, while the beam transmits through the ∼150 nm thick particles (increasing the total influence of the dark reference with respect to the signal). The other “noise” components (components 4, 7, 8, and 9) all have similar spectral and spatial features. The spectral profile is mostly noise with a tail coming in on the low energy side, corresponding to the SiN phonon at 130 meV that is cut out of the dataset. The spatial profile is also mostly noise but with visible diagonal lines, which arise from the post-acquisition alignment of the ZLP (more details are shown about this effect in the supplementary material, Fig. S3). It is also important to note that NMF (as well as CNMF-PS) orders the components in terms of their mathematical significance, meaning that the lower order components are a more dominant part of the signal compared to the higher order components. Ergo, the fact that components 2 and 4 are both “noise” components (as well as the fact that the majority of the 9 components end up as noise components here) indicates that the HSR dataset is dominated by noise and has lost a lot of information due to the lack of signal.

The benefit of using long dwell times to undo that lack of signal and achieve high spectral fidelity is shown by examining the next two columns, which illustrate the NMF decomposition of the HSF dataset. Here, there is only one “noise” component (component 8), which corresponds to the decrease in total inelastic cross section when transmitting through the thick sample. It is important to note that this component is equivalent to component 2 in the HSR decomposition; however, it indicates that the dark reference is the second most mathematically significant component in the HSR dataset but the eighth most significant in the HSF dataset, indicating that the NMF decomposition is now accessing the plasmonic response of the material much more effectively.

As a result, the majority of the spectral endmembers in the PS dataset are also “sample” components, aside from component 8, thanks to the long dwell times of the HSF acquisitions. In the last column, we show the pan-sharpened CNMF abundance maps. As stated previously, the CNMF approach refines the HSR dataset using the spectral endmembers from the HSF dataset. Thus, the abundance maps in the HSR and the PS datasets are obtained from the exact same data, the only difference being that HSR maps are determined by standard NMF, while the PS datasets are determined by PS-CNMF. In the PS results, we see all “sample” abundance maps with the same spatial localizations as were observed in the HSF maps only now with the spatial resolution to examine important regions like corners and gaps effectively. Additionally, some of the “sample” modes in the PS decomposition represent individual plasmon modes both spatially and spectrally, even in this more complex system. For instance, the component 1 abundance map shows the intensity focused in the corners of all three nanoparticles, which would be consistent with the individual particle edge mode and has a single peak in the spectral endmember at 460 meV. We can compare this with near-field enhancement (NFE) simulations of a trimer of nanoparticles with the same geometry and see that at 460 meV, the plasmonic response is dominated by the signal at the edges of the individual nanoparticles, indicating that component 1 corresponds to an “edge” plasmon mode in the individual nanoparticles (direct comparison shown in Fig. S4 of the supplementary material). Additionally, components 4 and 5 have dipole-like spatial-profiles while possessing the lowest peak energies of any components, indicating that they could correspond to the dipole modes of the system in the vertical and horizontal axes of the trimer. However, others contain some of the mathematical artifacts discussed earlier. For instance, components 7 and 9 are double peaked and hence not physical representations of individual plasmon modes, but even these artifactual “sample” modes are capturing mathematically relevant aspects of plasmonic response in the system and when combined with the other CNMF components generate accurate linewidths, shapes, and intensities, as observed in Fig. 1.

The technique also performs favorably compared to other standard denoising techniques. In Fig. 3, we examine a more complex cluster of FT:IO nanoparticles. Here, we directly compare the HSR dataset after pan-sharpening to the HSR dataset after denoising through Gaussian, Fourier, and Savitzky-Golay filtering. The SI slices at an energy of 600 meV are shown for the as-acquired HSR and HSF datasets in Figs. 3(a) and 3(b), and for the denoised/PS datasets in Figs. 3(c)–3(n) [Gaussian filtering—Figs. 3(c)–3(e); Fourier filtering—Figs. 3(f)–3(h); Savitzky-Golay filtering—Figs. 3(i)–3(k); CNMF-PS—Figs. 3(l)–3(n). For each method, we show the SI slice at 600 meV and spectra from a point just off the edge of the cluster, where the plasmonic signal is maximized, and from a point in the center of one of the particles, where the plasmonic signal is reduced due to thickness induced scattering effects. For each method, the HSF and HSR spectra from the equivalent positions are overlaid with the processed spectrum as to compare both the degree of denoising from the original data (by comparing with the HSR spectrum) and the match with the genuine spectral shape and frequency (by comparing with the HSF spectrum).

The Gaussian filtering is performed in the spatial dimension by averaging each pixel partially with its surrounding pixels to reduce the influences of readout noise and dark references. We can see from the spectra in Figs. 3(d) and 3(e) that the spectra compare very favorably with the HSF spectrum for both probe positions, but in the SI slice [Fig. 3(c)], we see that the spatial resolution is not significantly improved from the HSF dataset either, limiting the usefulness of the technique.

Conversely, the Fourier and Savitzky–Golay filtering are performed in the spectral dimension, with each spectrum in the SI either being Fourier transformed and having the high frequency components removed (Fourier) or being processed with a sliding window that considers the surrounding spectral features to remove spurious noise (Savitzky–Golay). We see that these methodologies provide excellent denoising in the image plane, and the SI slices for both methods in Figs. 3(f) and 3(i) show the gaps between particles going from invisible to highly pronounced. Additionally, in the spectra, we can see that the random noise is almost entirely eliminated in both methods; however, in the spectra, we see a different problem. Comparing the line shapes of the peaks in both Figs. 3(g) and 3(j), we see some subtle differences in the peak shape and widths that were not present in the Gaussian filtered spectrum. These artifacts are subtle in the probe position by the edge of the cluster, where the plasmonic signal is strong, but when penetrating through the FT:IO nanoparticle, the effect is exacerbated. If we examine the 600 meV peak here [Figs. 3(h) and 3(k)], we can see that the peak is broadened significantly compared to the HSF spectrum. Moreover, if we compare with the dashed line that is squarely at 600 meV, we can see that both the Fourier and Savitzky–Golay filtering have shifted the maximum frequency of the peak to a higher energy (∼640 meV). For the Fourier filtering, we show additional details about how the parameters for the fit were chosen and their sensitivity to slight changes in the parameters in the supplementary material (Figs. S5 and S6).

However, in the PS SI slice [Fig. 3(l)], we see that we have obtained the same high quality of image plane denoising obtained from the single spectrum filters, but by examining the spectra [Figs. 3(m) and 3(n)], we see that here, we obtain both a high degree of denoising without artificially altering the shape or frequency of the peaks in the high-noise scenario of the second probe position. This is driven by the fact that our processing here is guided by a dataset that samples the true plasmonic response as opposed to an averaging process.

These simpler filters are extremely effective at denoising spectra. In fact, they are significantly more powerful in denoising than PS, which is fundamentally limited by the noise in the HSF acquisition, and can provide meaningful results in many analyses. However, when the noise is of comparable intensity to the signal, they can result in artifacts and errors to many of the critical parameters to measure in plasmonics, such as peak position and linewidth.^33,34 Conversely, PS refines noisy spectra with high fidelity spectra, ensuring that if the plasmonic behavior is captured by an NMF component in the HSF acquisition, it is quantitatively reproduced in the PS dataset in terms of both peak position and linewidth, as shown in Fig. 3.

PS also compares well with simply increasing the dwell time at the same pixel density. Fig. S7 in the supplementary material shows a comparison of the above PS dataset with an SI from the same region with three times the total exposure time as the entire PS acquisition that results in visibly worse spatial and spectral denoising. Thus, PS provides a unique complementary alternative to standard methodologies of noise reduction and signal extraction, with its own unique combination of strengths and weaknesses that can reduce dose and experiment time while providing with quantitatively and qualitatively accurate results.

As a final point, while this technique is most relevant in systems where long experiment durations (i.e., long pixel acquisition times and high pixel densities) are unfeasible, it can also be applied to systems where the dose induced by the electron beam is unsustainable. The converged probe of the STEM results in all the probe current being focused to a single point and generating huge areal dose rates that can induce significant beam damage even for low total doses. Thus, the reduced dwell times of the CNMF result automatically present a strong benefit for beam-sensitive samples, but it can improve even further by exploiting the relationship between the focal-plane of the electron probe and its diameter.

For the HSF dataset, maximum spatial resolution is needed, so the beam is focused to the same plane as the sample, but since the dwell time is minimized here, so is the beam damage. For the HSR dataset, the long dwell times would result in a huge areal dose at the probe position, so even if the total dose is comparable to the HSF acquisition, far more beam damage would result. However, if the probe is defocused such that the focal plane of the probe is not in the same plane as the sample, the total dose is spread out over a larger diameter because of the convergence angle of the beam. This reduces the spatial resolution of the experiment, but since the spatial resolution of the PS dataset is provided by the HSR dataset anyway, the reduced spatial resolution in the HSF dataset comes at no cost, while reducing the areal dose rate by potentially orders of magnitude.

Figure 4 shows the difference between CNMF reconstructions of the plasmonic response of a single F:IO nanocube with the same HSR dataset but with two different HSF datasets: one with a focused probe (defocus = 0 nm) and one with a defocused probe (defocus = 500 nm). High angle annular dark field (HAADF) images of the tilted nanocube in both the focused and defocused condition are shown in Figs. 4(a) and 4(b), respectively, which clearly demonstrate the blurring and reduced spatial resolution caused by the defocused probe. The experiment is performed with a convergence semiangle of 30 mrad at an accelerating voltage of 30 kV in a high current mode, which means that the beam that is nominally around 2 Å in diameter at the focal point expands to illuminate a 30 nm diameter region of the sample plane at 500 nm defocus. As a result, the spatial area over which the same beam current is spread increases (and the areal dose rate decreases) by four orders of magnitude between the focused and defocused probe (beam current = 20 pA, focused areal dose rate = 4 × 10⁹e nm⁻² s⁻¹, and defocused areal dose rate = 2 × 10⁵e nm⁻² s⁻¹). It is important to note that defocus does not change the total dose experienced by the sample during the HSF acquisition, only the areal dose rate. However, for many samples, it is the highly focused electron beam irradiating the same point position for an extended period that generates the damage, and spreading out the beam can improve sample stability even for comparable total electron doses, ergo other techniques (such as sub-pixel scanning) that reduce the areal dose rate could also be applied in the same manner.

Additionally, the particle is tilted by ∼20° such that the face of the cube that contacts the substrate (called the proximal face) is shifted to the left and the face of the cube that is in the vacuum (called the distal face) is shifted to the right in the 2D projection. The tilt enables us to spatially separate the corner and edge that would be spatially degenerate in a 2D projection to examine each mode individually.

The abundance maps from the HSF-NMF decomposition along with the CNMS-PS abundance maps are shown for the corner mode and edge mode in Figs. 4(c) and 4(d), with the HSF/PS spectral endmembers for the two modes shown in Figs. 4(e) and 4(f), respectively. Here, we can see that in the HSF components, the defocus significantly affects the abundance maps, with the modes being blurred and spanning over more pixels in the HSF₅₀₀ components compared to the HSF₀ components. However, in the PS components, the abundance maps are almost identical. The reason is because even though the localization of the plasmon modes changes when the probe is defocused, the frequency of the plasmon modes is unaffected, and as a result, the NMF decomposition of the HSF datasets results in almost identical spectral endmembers for both the corner mode [see Fig. 4(e)] and the edge mode [see Fig. 4(f)]. The CNMF-PS is based off of the spectral endmembers of the HSF decomposition, so the similarity of the endmembers between the focused and defocused dataset results in nearly identical PS abundance maps and a nearly identical PS reconstruction.

The similarity can be seen by examining the PS-SI slices at the corner mode frequency (450 meV) and the edge mode frequency (550 meV). While the slices are reasonably similar, there are subtle differences in the localizations of the peak intensities around the edges and corners that are the same between the focused and defocused PS-SI. More critically, in Fig. 4(h), the spectra from the edge and the corner of the distal face are shown (to get the closest representation of the true corner and edge modes). Here, we can see that the edge mode at 550 meV is strongly excited in both the corner and edge spectrum (due to the projection of the overlapped corner and the edge) but that the shoulder corresponding to the corner mode is absent in the edge spectrum. Additionally, we pick up the higher energy face mode at ∼700 meV in the edge probe position. These CNMF plasmon modes agree well with the energy and localization of near-field simulations of an F:IO cube (shown in the supplementary material, Fig. S8), further demonstrating the ability of the reconstructed PS dataset to accurately represent the plasmonic response of a simple nanostructure.

We note that for a converged probe, defocus fundamentally alters the local density of states (LDOS) that is sampled by EELS. Thus, the LDOS experienced by the probe at −500 nm defocus will not be identical to the LDOS experienced at 0 nm defocus, which is why the HSF NMF components are visibly different between the two. However, the linewidth and energy of plasmon modes do not change via defocus, so as long as the defocus is not so significant that the spatial localization of the individual plasmon modes is lost, the spectral components should be near identical and, thus, the PS results should be nearly identical (as observed here). Potentially, for more complex systems (especially ones with 3D complexity) and/or higher defocus values, the change in the experienced LDOS of the probe may be so significant as to prevent reasonable comparison between the in focus and out of focus datasets. However, for the defocus used here, we can see if we can produce near identical signals in the CNMF components shown in Figs. 4(e) and 4(f), demonstrating that it is possible to use defocused-PS to significantly enhance spectral fidelity and significantly reduce areal dose without compromising on the accuracy of the spectral linewidths and frequencies.

In conclusion, the CNMF-driven pan sharpening of multidimensional EELS datasets is a robust technique that can significantly improve the signal-to-noise ratio without compromising spatial or spectral information. This technique is not computationally intensive or complicated to use and can be straightforwardly incorporated into conventional EELS workflows. In order to help interested researchers, we have included a sample Python notebook in the supplementary material that can be utilized to test CNMF-PS on existing datasets or to aid in future analyses. Additionally, the code could be easily modified to accommodate different modes of correlating the HSR and HSF datasets, such as principle component analysis³⁵ or multivariate curve resolution.³⁶ Furthermore, if the transformation matrices are known, the correlation could be performed between different spectroscopic techniques. Such correlative experiments have already begun to yield significant benefits in nanoscience,³⁷ and the addition of the PS could be used to combine EELS datasets with complementary nanoscale techniques, such as cathodoluminescence or near-field optical spectroscopy. The PS approach should enable efficient access to novel and unique aspects of nanoscale excitations and provide new insights into complex and difficult systems and is expected to be applicable, not just in electron microscopy but in all nanoscale hyperspectral techniques.

See the supplementary material for additional information on the materials and methods, post-synthesis macroscopic characterization, electromagnetic near field simulations, and additional validation of the methodology. Additionally, an iPython notebook exemplifying the CNMF-PS process can be found at https://github.com/hachteja/CNMF-PS.