Deep learning-enabled probing of irradiation-induced defects in time-series micrographs

In situ transmission electron microscopy (TEM) is a powerful technique that allows us to observe high resolution phenomena at the nanoscale while simultaneously analyzing the dynamics of an undergoing process.^1–4 Accordingly, with the developments in in situ TEM over the past few years and the dynamic frame rates of modern cameras,^5,6 experimentalists often have hundreds, sometimes thousands, of micrographs to process to understand an individual physical process at the appropriate time scale. Automated analysis tools available now should not only increase efficiency but also decouple subtle interactions that humans miss during data analysis.⁷ However, there is an added layer of complexity when trying to model microstructural changes occurring during radiation damage to structural materials.^8,9

In radiation materials science, in situ TEM is used as a unique tool to analyze complex defect structures produced in response to extreme environments.^10–14 Of practical importance, materials that can survive high doses of radiation are needed as we explore the dimensional stability of Generation IV reactor materials. Fundamental studies encompass the impact high energy particles have on the microstructure of a variety of materials, citing the interaction of point defects leading to dislocation networks, vacancy clusters, interstitial clusters, and inert gas bubbles.¹⁵ By analyzing the number of defects and their size and shape, we gain insight into the irradiation doses that begin to fundamentally alter materials’ properties and cause long timescale performance degradation, often limiting the performance of engineering systems.¹⁶ However, microstructural damage varies widely in different materials, even changing with the number of preexisting defects prior to irradiation.¹⁷ Accordingly, a large framework of data and experimentation is needed to explore as many enumerations as possible to design radiation tolerant materials for advanced nuclear energy technology.

In response, the nuclear materials community has made valiant efforts to automate defect characterization and extract the most information from large datasets. A faster regional-based convolutional neural network (R–CNN) was previously used to identify helium bubbles in irradiated X-750 from a nuclear reactor fleet.¹⁸ However, the automated technique detected a lot of small features that were missed in human inspection of low-resolution images, an indication that the success of the algorithm was very reliant on imaging conditions in the TEM. When looking at time-series data from in situ experiments, a “you only look once” (YOLO) algorithm was used to detect translations of mobilized defects, and a quantitative assessment of the size of the defects was performed.^19–21 Cluster analysis has also proven to be successful based on grouping aberrations in the micrographs into a feature of interest.²² While these advancements are tremendous, the community could benefit from a comprehensive end-to-end image analysis process that can handle images of various imaging conditions, quantify parameters of defects that served as a barrier in the past, and make the integration process more pervasive for experimentalists. In these types of experiments, we hypothesize that deep learning is the clear and most suitable bridge between experimentation and theory.

In this article, in situ ion irradiation is used in line with post mortem TEM to model the evolution of defects in metals. Individual frames are passed through a CNN to segment features from the images, and post-processing is used on the output segmentation maps to understand the physics of the reaction. Uniquely, we develop an end-to-end process flow to extrapolate a large volume of information from individual electron microscopy images that could be scaled to other datasets. By doing so, we observe defect interactions and annihilation, morphological distortions, and size effects as a function of radiation dose. Ultimately, we determine that advanced image processing techniques help us unveil complex reactions, and this methodology can be scaled to resolve several other problems in radiation materials science.

Experiments were performed using in situ ion irradiation at a 200-kV JEOL-JEM 2100 TEM. A TVIPS 1k camera enabled in situ imaging, capturing 5 fps. A 6 MV EN tandem ion accelerator was used to generate a 3 MeV Cu³⁺ broad beam. The stage was tilted 30° toward the beamline to form a 60° angle between the ion beam and the sample normal so that the sample faced the ion beam. The temporally uniform but spatially graded Cu beam spot size current was measured using a Faraday cup located upstream of the TEM on the beamline. For further details on the sample preparation, the method used to form a two-beam image, and calculating the thickness of the TEM foil, refer to our previous work by Li et al.²³ Accordingly, to further diversify the training set, we include an additional in situ dataset from a self-ion irradiation of Au, with a similar snapshot shown in Fig. S2(a). Additionally, the hand-labeled masks for the Au “still” frames are displayed in Fig. S2(b). Details on the Au irradiation can be found in our previous work.²⁴ There were two separate sequences of videos collected in this experiment to collect the data used for the Cu irradiation and the Au irradiation, respectively.

Over 200 000 defects were hand-labeled to establish the ground-truth data. In situ observation yielded insight into the defect generation rate and interaction of defects. All defects in the micrographs were hand-labeled pixel-by-pixel by domain experts to establish the ground-truth data. In Fig. S1(a), we give a snapshot of the training dataset for the Cu foils, which is an extrapolation of “still” frames from the video captured in Movie S1. In Fig. S1(b), the corresponding hand-labeled masks are shown to display a comprehensive labeling process from the still frames. This in situ dataset from the Cu foils had similar microstructures throughout the experiment, which increases the probability of overfitting the model during training.

Extrapolating features from images requires an end-to-end process, from image filtering prior to training to extracting features from the output segmentation maps. The multi-step process here was built using a combination of previously studied steps,^25–29 with some new processes integrated. Figure 1 provides a synopsis of this process, where we break each section down into the following subparts:

1. Data normalization: The input image is analogous to the “still” frame introduced in Fig. 1(a). The size of the image is then reduced by a factor of 2, making it easier for UNet++ layers to handle. A patch-based learning strategy is then implemented to recognize salient features of an image with a higher degree of accuracy. The input data then goes through linear stretching to reduce the intensity ratio between the defects and the background, making it seamless to filter out artifacts or unwanted characteristics from the image, as shown in Fig. S3. Finally, an 80/20 random split between the training/test set was implemented.

2. UNet++ training: 80% of the entire dataset analyzed was implemented into the training set. To increase the number of analyzed micrographs and add complexity to the dataset, the micrographs were augmented with three modulations. They were rotated in iterations of 90°, zoomed in at five different magnifications, and introduced with noise through a low-filter Gaussian blur. Visual representations of the augmentation are found in Fig. S4. Next, training and passing of the images through the UNet++ architecture³⁰ were performed.

3. Probability map post-processing: The probability maps of the patches are a segmentation of the features of the patches generated in Sec. I. A binary classification was adopted to yield absolute probability values, where each pixel in the image was given a value based on the intensity of the original image. Pixels below a particular threshold were given a value of 0 (copper metal), and pixels above a certain threshold were given a value of 1 (defect clusters). The threshold value (set to 0) is used to classify the pixel values. The maximum value (255) is assigned to pixel values exceeding the threshold. We then used Otsu’s binary threshold since this approach determines the optimal global thresholding values for the user based on the intensity distribution in the image histogram. The complete image was then reconstructed by stitching the patches of the threshold back together. The pixel values of 1 were then given a label, thereby giving every defect cluster a unique label in the returned array. Finally, we get the full output segmentation map, which can distinguish between overlapping defects, touching defects, and defects that coalesced.

4. Feature extraction: To detect the contours around the defects, a loop in the range of contours is run and iterated through. An outline of the shape was then drawn to determine the centroid of the boundary. The shape is then detected based on the number of contour points within the outline, the detected shape is characterized, and the center point is given a shape. Finally, we can extract the size and shape metrics from each of the labels (defects) from the output segmentation map.

This sequence of steps was used for all the frames in the dataset for consistency in analysis. Note that some steps (i.e., data augmentation in the section of UNet++ training as opposed to Data Normalization) could be applied to a different section; however, emphasis is placed on the sequence in which each process is performed.

Model performance was based on metrics we can quantify by calculating the intersection over union (IoU) and computing the Dice score. First, we provide a comparative visual of the input (raw) images, the ground-truth dataset, and the predicted images from the output of the process flow, as shown in Fig. 2(a). Accordingly, we take two samples from the training set and two from the test set and calculate the defect count from the ground truth and predicted dataset, displayed as a bar plot in Fig. 2(b). The ground truth dataset for Cu irradiation (Training Set 1) and Au irradiation (Training Set 2) had a defect count of 143 and 23, respectively. The corresponding predicted masks had a defect count of 144 and 24 defects, respectively. It should be noted that both experiments characterized defects produced during self-ion irradiation experiments. However, to determine the robustness of this approach to any defect observed in a TEM, we applied this algorithm to a dataset of implanted He bubbles in Pd metal to see if the segmentation algorithm was transferable to a defect class that has an intensity gradient across its domain. The He was implanted into a thin Pd slab at 1 × 10¹⁷ cm⁻² and thermally annealed at 400, 450, 500, and 550 °C, respectively. For this, the ground truth had a defect count of 317 defects, while the predicted had 241 defects. The large variance is a good indication that a separate training set or alternate segmentation algorithm would be required to get better comparative results among feature characteristics from a different class of defects. Overall, we can conclude that the machine performed well relative to the human-based identification of defects in the systems with in situ ion irradiation.

In Fig. 2(c), the IoU for the training and test sets saturates at 0.73 and 0.80, respectively. The computed Dice score saturated close to 200 epochs above 0.8. The IoU is a good judge of the accuracy of the model, while the Dice score is a good judge of how precise the model is, as the evaluation criteria for the Dice score penalizes more for false-positives. The training set for the IoU displayed an accelerated learning pace within the first 15 epochs before slowing to a linear learning rate up to 200 cycles. This means that the model learns the mean of the target variable in the first 15 epochs, which causes the large decrease in loss, then deciphers subtle differences in the data from epochs 15–200, attributed to the reduced slope in this region.³¹ The test set reached more ideal IoU values than the training set at the end of the training, which we attribute to the training loss being calculated before each epoch, while the test loss is calculated after each epoch. The hyperparameters were optimized to generate the best performance, with values summarized in Table I. Mainly, these hyperparameters allowed the loss function to begin convergence at a smaller number of epochs, not miss any local minima, and lead to better model fitting with a lower value on the weight decay.

As mentioned previously, one of the main benefits of in situ TEM is the ability to perform real-time experiments while analyzing local regions of a material under an external stimulus and simultaneously probing the dynamics of a reaction. In Fig. 3, we highlight the defect interactions commonly observed in our experiments by indicating regions of interest with arrows. First, defect generation is shown in Fig. 3(a), as impinging ions generate vacancies at the site of impact. The energy of the ions is sufficiently high to overcome the threshold energy of Cu atoms in the target material.²³ Next in Fig. 3(b) is defect coalescence, in which we discuss the interplay between dislocations and stacking fault tetrahedra (SFT). These defects interact aggressively with each other at high strain rates,³² which we impart on the Cu slab from prolonged exposure to the ion beam stimulus. Third is defect fragmentation in Fig. 3(c), which occurs when an impinging ion interacts with a preexisting defect in the target material. Fragmentation rates increase with the number of dislocations in a close vicinity,³³ so their occurrence is expected to increase with fluence.

In Fig. 3(d), defect annihilation is highlighted, which depends strongly on the separation between dislocations. In general, there is more free energy for dislocations in a closer vicinity to each other,³⁴ making these dislocations easier to annihilate in the presence of impinging ions. A defect is annihilated within 0.4 s, which is quantitatively visualized from the intensity decay of the cluster [visualized in the line profile in Fig. 3(e)]. It should be noted that defect clusters leave debris behind after annihilation because the annihilation is triggered by a dipole transformation; additionally, dislocations rarely have a fixed orientation. The center of mass remains stationary relative to the surrounding defects, as shown in the line profile, and the intensity variance reduces in contrast relative to the Cu grain. Accordingly, even with the small debris left behind, the model does not pick up the defect after it is annihilated, as the contrast is closer to that of the Cu grain than that of the defect. The fluences in Fig. 3(f) were selected based on the inflection points indicated by the red X annotations in the plot, which are used to describe the stages of defect evolution in Cu.^23,35 Here, when the density of defects is proportional to the irradiation dose (i.e., n = 1, where one impinging ion generates 1 defect), the defects are displaced directly in the displacement cascades. The incubation zone represents the onset of Cu being displaced from equilibrium with a slope greater than 1. As the fluence increases from prolonged exposure to the ion beam, the slope decreases, indicating the interaction of adjoining defects.

Figure 4(a) provides a visual of how defects are tracked using the OpenCV object tracking software. A tiff stack of output segmentation maps was converted to a NumPy array in their respective time series. Each defect was given a unique label based on its (x, y) coordinate list in the image and its occurrence in time. Accordingly, this allows one to visualize how a particular defect changes its displacement through irradiation. Figure 4(b) provides a breakdown of the count of defects present pre-irradiation and arising from irradiation. The defect clusters that were present pre-irradiation decrease with increasing fluence, while the irradiation-induced defects oscillate between increasing and decreasing in density with increasing fluence. We attribute this to a combination of the defect interaction events discussed in Fig. 3 since the ions impinging on the Cu slab have a steady dose, and without interactions, a progressive increase in the number of defects would be observed. Figure 4(c) displays the TEM images and corresponding masks that complement Fig. 4(b) and the annotation in Fig. 3(f). The preexisting defects are shown in black, while the irradiation-induced defects are displayed in gold. Irradiation-induced defects accumulate aggressively both within the Cu grain and along the grain boundary.

In Figs. 5(a)–5(c), a density plot of the defect density, eccentricity, and area are given relative to each metric for the irradiation of Cu. The plots represent the mean value of each variable across the entire experiment. Each plot provides a distribution of a single variable vs the relationship of two variables in a dataset and is used here as an effective method to identify trends between independent variables. Plots 5a-c have the same density spread throughout the micrographs, meaning the kernel smoothing representing the peak of each plot corresponds to how well-correlated each feature is relative to each other. This means the mean target value for each of the three variables being compared has a co-dependence on each other, with the closest correlation occurring between defect density and eccentricity in Fig. 5(b). Likely, this indicates that the relative proximity of defects has a direct effect on the shape of the neighboring defect, as uniform displacement fields emanate from dislocation loop cores to create geometric non-uniformities in the surrounding lattice.³⁶ 

In Fig. 5(d), the median, mean, and max area are shown throughout the experiment. The median value of the defect sizes remains the same, a good indication that irradiation-induced defects generated are the same size as fabrication-induced defects, while the slight fluctuations in size can be attributed to nanoscale interactions and defects interacting. In Fig. 5(e), a plot of the fluence vs the eccentricity is displayed. The eccentricity increases at the initial onset of the irradiation, indicating a distortion in the morphology of the defects becoming more elongated and a larger variance in the major and minor axis lengths. This onset occurs when the Faraday cup is removed to initiate the irradiation. There is a slight thermal shock on the metal that causes a small amount of stage drift. This creates false (ghosting) on the defects, making the size read higher than what is real. In both plots, the max and mean have an unusually higher value for the first data point, which we attribute to a false identifier at the onset of irradiation. The plot in Fig. 5(d) shows a poor correlation of the fluence vs area, as the size of the defects oscillates around a mean through the experiment. The plot in Fig. 5(e) is a plot of the fluence vs eccentricity, showing a steady increase during the first half of the experiment, followed by the values fluctuating around a mean. The outliers in Figs. 5(a)–5(e) are explained by fighting drift as a microscopist at the beginning of an experiment. This is when a small area and low defect density (and, therefore, low fluence) are present simultaneously. Figure S5 is included to show how the standard deviation of area and eccentricity both evolve over time. Notably, the standard deviations of the area assist in highlighting an outlier at the beginning, as the “ghosting” around the defects during drift will make the defects appear larger in a non-uniform manner.

Correlative methods of tying together in situ TEM and data science serve as a platform for rapid, accurate, precise, and non-biased identification of defects. We correlated the properties of defects based on output segmentation maps to understand the physics at multiple time scales by stitching back together micrographs in their temporal sequence. With this approach, it becomes possible to sift through massive amounts of time-series data and decouple the kinetics of a given reaction, only limited by the specifications of the electron microscope used. Uniquely, we emphasize a transferable approach by diversifying our micrographs through various in situ experiments and augmentation. We also show how many features of defects can be compared to each other to globally understand an irradiation response. However, we have also seen that one training regime is not universal to all types of defect characterization, so future work will be geared toward developing a customized process flow for the identification of He and Xe bubbles in similar materials systems.

We have developed an end-to-end process that allows one to plug in time-series TEM images and extrapolate feature characteristics to probe the physics of any nanoscale reaction that can be captured within the capabilities of the camera frame rate and the resolution of the microscope. To date, we have assembled the largest labeled in situ ion irradiation dataset in two-beam (diffraction contrast) conditions, which includes over 1000 images and over 250 000 defects, making for one of the most comprehensive data-driven studies in this field. Notably, we found that

• hyperparameter optimization, including multiple experiments in the training set and increasing variability in the input, was paramount for preventing the overfitting of the model,

• defect classification does not scale 1:1 from dislocation loops/SFTs to helium bubbles. The primary driving force behind this issue is the opposite contrast produced by the electron beam between the intrinsic defects in Cu and Au and the extrinsic defects in Pd. Accordingly, a separate model, or a vastly expanded model with thousands of micrographs with a high density of He bubbles in the training set, would have to be trained for an algorithm to classify broader datasets in radiation materials science,

• in situ datasets can often battle drift when the onset of an external stimulus is introduced to the materials. The noise aberrations can create “false” metrics in the images and generate inaccuracies in the data. It is important to selectively go through a dataset and understand where this happens (as seen in this work) or develop an approach that removes drift-related noise (i.e., a denoising autoencoder) and,

• the end-to-end process developed in this work was suitable for characterizing irradiation-induced defects in face centered cubic metals such as Cu and Pd, with rapid analysis and good accuracy.

Last, this approach can be adopted to quantify the characteristics of defects in similar materials’ systems and understand the dynamics of processes occurring at multiple time scales. Further work is needed to separate dislocation loops and SFTs in the micrographs.

See the supplementary material for additional details: the Cu dataset and the hand labeled masks (S1); the Au dataset and the hand labeled masks (S2); unsupervised image segmentation on regions of interest (S3); data augmentation (S4); and the effect of the LIF neuron’s threshold (S5).

This work was primarily funded by the US Department of Energy in the program “4D Camera Distillery: From Massive Electron Microscopy Scattering Data to Useful Information with AI/ML.” This work was performed, in part, at the Center for Integrated Nanotechnologies, an Office of Science User Facility operated for the U.S. Department of Energy (DOE) Office of Science. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for the U.S. DOE’s National Nuclear Security Administration under contract DE-NA-0003525. The views expressed in the article do not necessarily represent the views of the U.S. DOE or the United States Government. Los Alamos National Laboratory, an affirmative action equal opportunity employer, is managed by Triad National Security, LLC, for the U.S. Department of Energy’s NNSA under Contract No. 89233218CNA000001.

The authors have no conflicts to disclose.

Kory Burns: Conceptualization (equal); Data curation (equal); Formal analysis (equal); Investigation (equal); Methodology (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal). Kayvon Tadj: Conceptualization (supporting); Data curation (supporting); Investigation (supporting). Tarun Allaparti: Data curation (supporting). Liliana Arias: Data curation (supporting). Nan Li: Conceptualization (supporting); Investigation (supporting); Writing – review & editing (supporting). Assel Aitkaliyeva: Writing – review & editing (supporting). Amit Misra: Investigation (supporting); Project administration (supporting); Supervision (supporting); Writing – review & editing (supporting). Mary C. Scott: Funding acquisition (equal); Project administration (equal); Supervision (equal); Writing – review & editing (equal). Khalid Hattar: Formal analysis (equal); Funding acquisition (equal); Project administration (equal); Supervision (equal); Writing – review & editing (equal).

All the code supporting this work is available via GitHub.³⁷ All images are available via a downloadable link on The Box.³⁸