Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution STEM imaging

The advent of aberration correction in (scanning) transmission electron microscopes (S/TEMs), significantly increased the spatial resolution of images obtained from electron “beam stable” samples.^1,2 However, this resolution improvement was accompanied by higher beam currents, and electron “beam damage” now represents the main limiting factor hindering the wide use of S/TEM techniques. This “beam damage” is an especially limiting factor for imaging of biological samples,^3,4 soft and sensitive materials,^5,6 in-situ imaging under environmental conditions,^7,8 and imaging that requires multiple or long acquisitions, such as 3D tomography,^9,10 ptychography,¹¹ or spectroscopic mapping.¹² Although relatively new to mainstream S/TEM applications in inorganic materials, the issue of electron “beam damage” has been a long-standing problem in the biological sciences, and there have been various strategies developed to keep it at a minimal level. Some of these strategies involve choosing the electron beam energy in order to minimize the primary beam damage mechanism¹³ or utilizing methods that rely on the minimization of pre-exposure or fractionation of the electron dose.¹⁴ While all of these methods are constantly being improved by the development of new detector technologies and methods that operate with higher efficiency,^15–17 it is clear that minimizing the electron dose on the sample can go a long way to improving the reliability and reproducibility of images from all types of samples in the S/TEM.

Independent of these traditional low-dose S/TEM methods, the concepts of compressive sensing and inpainting have recently emerged as a viable means for electron dose reduction.^18–20 Compressive sensing (CS) and inpainting is based on the foundation that images/data can be well represented in a much sparser form using a suitable basis set, and that this suitable basis set can be fully recovered from significantly fewer measurements than conventional pixel-by-pixel acquisition (hence reducing the acquisition time and dose). The theory of compressive sensing was developed by Candes et al.²¹ and Donoho,²² and it is now widely applied in various fields. In an electron microscopy, compressive sensing has been applied recently for tomographic reconstruction^20,23 and has also been proposed as a method for the TEM video rate enhancement.²⁴ 

Because STEM is a serial imaging technique, it can, in theory, be readily adapted for sparse sampling, leading to an immediate dose reduction. The expected reduction of the dose will depend on the type of sparse sampling, the reconstruction algorithms, and the acceptable image quality. In the original theoretical STEM paper by Stevens et al.,¹⁹ it was shown in image simulations that inpainting reconstructions using Bayesian dictionary learning²⁵ could provide suitable reconstructions with better than an order of magnitude sub-sampling. The practical realization of sparse sampling in STEM can be achieved either directly with the scan coils (by steering the beam to the sparse pixel locations) or by applying beam blanking on top of a conventional scan (blanking the beam at all but a few randomly selected pixels). Directly controlling the beam, rather than using the beam blanker, is a very attractive option since it offers an improvement in acquisition time in addition to dose reduction. However, for a typical STEM image with a pixel dwell time of only few μs, electron beam controlled sampling may not be easily adopted due to the dynamic response of the scan coils causing significant delays between desired and current probe position. A viable approach has been previously proposed and demonstrated for SEM, where the scan coil dynamics were characterized, thus allowing a post-acquisition correction of the positional assignments.²⁶ The beam-blanking approach eliminates the positional uncertainties, but only permits the reduction of dose. Possibly, a more important issue, however, is the fact that the beam-blanking approach will also be associated with extra dose and time due to the beam blanking itself. As the beam is moved on and off the sample (blanking), a finite time for stabilization has to be accounted for, leading to additional dose and acquisition time. The experimental demonstration of sparse acquisition based on an electromagnetic blanker was demonstrated by Béché et al.,²⁷ and a faster electrostatic blanker by Muecke-Herzberg et al.²⁸ In this work, we pursue an approach to sparse sampling that directly controls the scan coils and eliminates the uncertainties with beam position assignment. It also offers several additional advantageous as compared to the previously applied scan-coils controlled sparse sampling.²⁶ 

A “fully random” or 2D Bernoulli pixel sampling, as shown in Figure 1(a), represents one of the most advantageous strategies for sparse sampling and inpainting.²⁹ This sampling has a very low degree of coherence, which permits a high degree of under-sampling without the introduction of reconstruction distortions.²⁹ One can easily envision that Bernoulli sampling could be accomplished by randomly changing the pixel step size during image acquisition. However, in order to practically implement sparse sampling by changing the step size, it has to be assured that the beam steps can be realized in a much shorter time than the dwell time, which in the case of conventional STEM imaging can be as short as few μs. To assess the viability of beam controlled sparse sampling on the used JEOL ARM200CF, we evaluated the dynamic response of the beam by measuring the beam rise time, which represents the response time for the beam to reach a desired set location. The rise time was evaluated from a pair of images acquired in interlaced fashion under different dwell times. Details regarding derivation of the beam rise time will be presented elsewhere. As shown in Figure 1(b), we find that the time required for the beam to reach 90% of the desired location is approximately 60 μs, which is significantly higher than the dwell time used for conventional STEM imaging. Given this relatively long time and the complex response, it is clear that even for relatively long dwell times (∼tens of μs), performing sparse sampling by changing the step size would lead to a situation where the beam movement is delayed from a specified setting during detector integration, and the set location will not only be dictated by current settings but also by after-effects of several previous beam settings. It should be noted that in a regular full scan (all pixels), a dwell time of only a few μs will not cause any image distortions because after the initial acceleration, which can take few pixels, the beam inevitably reaches a steady state speed with the desired pixel size/dwell time. This is without any other additional corrections applied to the scan generator.

To overcome the issue of slow beam response, we developed an approach where the sparse sampling is accomplished by a continuous lateral scan that imposes a random vertical perturbation. In this sparse sampling approach, as shown in Figure 1(c), the beam moves at constant speed along the horizontal (x) direction, while sampling a range of pixels along the vertical (y) direction. The step size is kept constant and each step corresponds to only a one pixel jump along x and y directions. Since the movement of the beam is constant along the x direction, just as in the case of regular scan, the average beam position along x is well defined. For the y direction, the beam randomly hops up and down within a pre-defined range of pixels. To assure that the beam positions can be reliably driven and assigned to the underlying pixel grid, the dwell time has to be restricted to a predefined range, as dictated by the rise time. It can be shown that the dwell times corresponding to ∼70%–90% of the rise are adequate for implementation of this sparse line-hopping approach. The simulated beam positions for the movement along the y direction (as well as the x direction) for a dwell time of 80% of the rise is shown in Figures 1(d) and 1(e). As seen in Figure 1(e), the average position of the beam along the y direction is well within the center of the periodic grid, thus enabling a reliable pixel assignment. Dwell time that corresponds to only 70% of rise time translates to just over 30 μs on the JEOL ARM200CF, which is higher than the conventional time of ∼10 μs, but still results in a time reduction when sparse sampling is employed, and it allows the total acquisition time to be relevant for high resolution imaging applications.

To compare the theoretical performance of line-hopping sampling with Bernoulli random sampling, we have performed a test reconstructions on digitally sparsely sampled images, and compared them on the basis of peak signal to noise ratio (PSNR), following the procedures by Stevens et al.¹⁹ As shown in Figure 1(d), the highest quality of reconstruction (highest PSNR) is obtained for Bernoulli sampling. However, the line-hopping sampling gets very close—for example, the line-hoping method is expected to yield the same PNSR at ∼75% under-sampling as 80% under-sampling in the Bernoulli sampled image. At all but the highest levels of under-sampling the line-hopping approach is thus expected to provide an accurate reconstruction while optimizing the time for acquisition.

The line-hopping acquisition described above has been implemented on an aberration corrected JEOL ARM200CF. The scan coils and the imaging detector were controlled with a custom-built acquisition unit, using a National Instrument PCIe-6361 acquisition card. The Analog Output (AO or DAC) is 16 bit, 2.86 MHz to control the X and Y positions. The Analog input (AI or ADC) is 16 bit, 2 MHz to read the High Angle Annular Dark Field (HAADF) detector. A Labview program was developed to control the acquisition of both regular (to test against the microscope acquisition system) and sparse acquisitions. The sparse position for the line-hopping acquisition was pre-loaded as an input file, which was generated by a separate Fortran program. For the images shown here, the microscope was operated at 200 kV and the images were recorded on a High Angle Annular Dark Field (HAADF) detector. The inpainting reconstructions were performed with the beta-process factor analysis (BPFA) algorithm¹⁹ and run in Matlab. Although for the purposes of this demonstration the system for attaining a line-hopping scan was separate from the microscope's standard controls, there is no physical limitation that stops the approach being integrated with the normal microscope control systems.

Figure 2 shows the atomic level microstructure of NiTiO₃, obtained with full sampling and two sparse samplings at 50% and 20%, using the line-hopping acquisition. The images were acquired with a dwell time of 31.35 μs and pixel array of 512 × 512. As shown in Figure 2(a) of a fully sampled image of NiTiO₃ (R $3¯$⁠) from the [100] projection displays dumbbell motifs, with super-lattice intensity variation along the [001] direction. A comparison of the original and reconstructed images of Figures 2(a), 2(c), and 2(e) shows that a very high degree of image fidelity is maintained down to 20% sampling. Only small differences in the image quality for the 20% sampled image are observed as compared to the fully or 50% sampled images. Perhaps most noticeably, the definition of some reconstructed atomic columns is noisier. Nevertheless, the 20% sampled image maintains a high resolution and contrast that is representative of the fully sampled microstructure.

Figure 3 shows a sparsely acquired atomic resolution image of CaCO₃ along the [241] zone axis at 20% sampling, together with the corresponding inpainting reconstruction. The image was acquired with a 6 pA probe, and a total dose of ∼3175 e^–/Ų. CaCO₃ is a high beam sensitive material (unlike NiTiO₃), which is very difficult to image at atomic resolution. The previous work by Hoffman et al.³⁰ indicated that a total dose of the order of ∼1000 e^–/Ų represents a threshold limit for the onset degradation, which results in transformation to CaO and CO₂. The use of the line-hopping approach enabled us to reduce the total dose to 3175 e^–/Ų, which in turn enabled the resolution of the atomic structure of CaCO₃ in HAADF mode; the main lattice spacings of (112), (102), (104), and (210) can be clearly observed in the images. However, even under the reduced dose of the line-hopping acquisition, the onset of amorphization can be seen in the image. It is also worth noting that any subsequently acquired images from identical regions of the sample suffered from serious beam damage and amorphization.

The benefit of sparse sampling has thus far been described in the context of dose reduction. It is also important to point out that this sparse sampling approach can simultaneously lower acquisition time, when comparing it on the basis of constant dwell time per pixel to full sampling, or lower the dose-rate (beam current) when comparing it on the basis of total acquisition time. Imaging with lower beam currents conventionally requires increasing dwell time to obtain images with a comparable signal to noise ratio. The increase in dwell time results in prolonged exposures that are adversely affected by sample drift. Due to the time saving associated with sparse sampling, however, lowering the beam current and prolonging dwell times can be afforded while keeping the image acquisition time at a constant value, thus minimizing the drift. The reduction of beam current is especially important for imaging groups of oxides, carbonates, and other materials where both the total dose and dose rate play an important role in controlling the electron beam damage.³¹ 

As the dwell time for line-hopping has to be above 30 μs, this method may not be suitable for very high frame rate imaging/recording as compared to conventional imaging. For example, the overall acquisition time for the line-hopping approach with a dwell time of 30 μs and 20% sampling will be comparable to 6 μs dwell time with a conventional full scan. As the beam dwell time can be reduced significantly below 6 μs for full sampling, a much higher frame rate for imaging can be obtained. Note that if the rise time of the scan is decreased by using faster electrostatic deflectors, the sparse sampling approach can be used for faster imaging. On the other hand, in the case of spectroscopic imaging where the scan dynamics no longer limits the minimum dwell time, significant reduction in acquisition time is expected. For spectroscopic imaging the minimum dwell time is dictated by the signal to noise ratio in the acquired data, and the dwell time is normally an order/s of magnitude higher. The sparse sampling approach by controlling scan coils can be thus much more readily adopted in this case, and would provide a significant reduction in acquisition time, in addition to the total dose reduction.

The line-hop sampling approach has several attributes that are advantageous when compared to the sparse sampling method introduced by Anderson et al.²⁶ for fast image acquisition in SEM. The sparse line-hopping approach allows taking into account the dynamic scan response in much simpler ways; it does not require a magnification specific, highly accurate beam response characterization. It also provides an extra reduction of acquisition time/dose by eliminating a substantial portion of the fly-back time as the beam scans multiple lines at once. In addition, the reconstruction can be treated in a simpler way, as the sampling does not lead to a variable dwell time and signal to noise ratio at each pixel. In comparison to the beam blanking approach, the line-hopping method offers simultaneous reduction of acquisition time/beam current, and most importantly is not associated with extra dose/time due to beam blanking. Nevertheless, the beam blanking approach can truly achieve Bernoulli random sampling, and thus in theory provides higher quality reconstructions at a constant level of under-sampling.

A topic that was not explored in this work is how to optimize imaging parameters, including the degree of under-sampling, dwell time, and beam current for a constant dose of e-/A² in low dose imaging. Very recent work by Reed et al.³² suggested that Compressive sensing (CS) denoising alone on fully sampled images may be the most optimum strategy for low-dose imaging. It will be experimentally worthwhile to further explore this topic, not only for single image acquisition but also in the context of higher dimensional sparse sampling such as, for example, used for tomography reconstructions.¹⁰ 

In conclusion, we have shown that efficient sparse sampling can be achieved in STEM images by randomly sampling over several rows of pixels as the beam moves along the scanning direction. This line-hop sampling is demonstrated to lower the dose and acquisition time by a factor of at least 5× for atomic-resolution images, without requiring significant modifications to the instrument hardware. In the case of beam sensitive materials, the reconstructions allow high-resolution images to be obtained while at the same time allowing areas of beam damage to be identified. This method can reduce the dose/speed of any operating procedure in STEM, meaning that any improvements in detector efficiency or in the speed and stability of the scan system (for example, by moving to an electrostatic scan system) will also be improved further by this sampling procedure. Such an approach can facilitate the use of STEM methods for a much wider variety of materials, including biological samples.

The development of Compressive sensing for EM applications was supported by the Chemical Imaging Initiative, a Laboratory Directed Research and Development Program at Pacific Northwest National Laboratory (PNNL). PNNL is a multi-program national laboratory operated by Battelle for the U.S. Department of Energy (DOE) under Contract No. DE-AC05-76RL01830. A portion of the research was performed using the Environmental Molecular Sciences Laboratory (EMSL), a national scientific user facility sponsored by the Department of Energy's Office of Biological and Environmental Research and located at PNNL. We would like to acknowledge Dr. Tamas Varga and Dr. Sebastien Kerisit for their kind help in providing samples for this study.