Deep convolutional neural networks for multi-scale time-series classification and application to tokamak disruption prediction using raw, high temporal resolution diagnostic data

Plasma phenomena contain a wide range of temporal and spatial scales, often exhibiting multi-scale characteristics (see Fig. 1). In fusion energy plasmas, many disparate diagnostic instruments are simultaneously used in order to capture these various spatiotemporal scales and to cover the multiple physics present in these plasmas. In addition, fusion experiments are increasingly built to run longer pulses, with the goal of eventually running a reactor continuously. The confluence of these facts leads to large, complex datasets with phenomena manifest over long sequences. A key challenge is enabling scientists/engineers to utilize these long sequence datasets to, for example, automatically catalog events of interest or predict the onset of phenomena.

Machine learning, and specifically the variant deep learning, has been proven to be highly successful in automating a number of tasks, such as identifying objects in images, language translation, and even playing strategic games such as Go.¹ Many deep learning architectures have been created and successfully applied to sequence learning^1–3 problems, in areas of time-series analysis or natural language processing. However, many of the typical architectures used for learning from sequences [e.g., recurrent neural networks (RNNs) and their most popular variant long short-term memory (LSTM) networks] suffer from memory loss; long-range dependencies in sequences are difficult for these architectures to track.

In this paper, we discuss recent advances in neural networks, specifically an architecture that uses dilated convolutions in a deep convolutional neural network (CNN), which was designed to overcome these problems of learning on long sequences. We use this architecture to predict oncoming disruptions in fusion plasma discharges of the DIII-D tokamak utilizing only raw data from a single, high temporal resolution imaging diagnostic (Electron Cyclotron Emission imaging diagnostic or ECEi). Because the ECEi diagnostic is sensitive to a range of multi-scale dynamics in the plasma related to disruptions,⁴ it offers the potential to more accurately predict them. Avoiding disruptions is a great challenge for tokamak fusion devices on the road to fusion energy.⁵ While much research has gone into utilizing machine learning for disruption prediction,^6–8 often global, reduced 0D features are used in shallow machine learning methods. Recently, work utilizing deep LSTM networks also added the use of low temporal resolution 1D plasma profiles,⁸ and another work used a combination CNN/LSTM on resampled, low temporal resolution bolometer data.⁹ The work we present here takes inspiration from these works in utilizing higher dimensional signals and shows how to use newer deep learning architectures to learn on high-temporal resolution data with long-range dependencies due to multi-scale physics.

The outline for the rest of this paper is as follows: Sec. II discusses a paradigm for understanding deep learning and its usefulness in fusion plasma physics, Sec. III discusses typical sequence learning architectures and the need for newer neural network architectures to capture long-range dependencies in sequences, Sec. IV applies deep convolutional neural networks with dilated convolutions to ECEi data for disruption prediction, and Sec. V discusses future applications and directions for these networks in fusion.

Deep learning has been tremendously successful in recent years in achieving state-of-the-art results for many machine learning tasks, including image classification, language translation, and speech recognition. Reference 1 provides a good review of the underlying principles of deep learning. Deep learning refers to neural networks with many hidden layers. Each hidden layer provides a linear transform (with a number of weights, W_j, along with a bias term b_j), followed by a non-linear transform (called the activation). By stacking up several layers, a deep neural network can potentially learn complicated non-linear functions. A task specific loss function is defined and produces a measure of the error between predictions of a network and the user-labeled targeted outcomes. Using this measure of error from the loss function, neural networks use a method called backpropagation¹¹ to update the weights of the neural network, with the goal of making the predictions match the targets. In this sense, a paradigm for deep learning is that deep neural networks are a series of filters whose coefficients (or weights) are “learned” instead of user-prescribed.

One of the reasons for deep learning's great success is the ability to learn multiple filters for high-dimensional data, avoiding the need for humans to do feature extraction.¹ This allows the deep learning algorithms to learn directly from raw data, for example, using directly the pixels from a camera image to predict whether a cat is in the picture, instead of having humans to specify filters, which can find features such as ovals (for eyes) and triangles (for ears).

An example from fusion can aid in understanding how feature extraction is applied everyday by fusion physicists and the potential for deep learning to learn the filters necessary to perform tasks. In fusion plasmas, often a variety of filters, transforms, and models are applied to measured data to extract physically relevant quantities. Figure 2 shows the example of extracting the normalized internal inductance, $ℓi$⁠, from the raw magnetic diagnostic measurements. A series of transforms are applied: low pass filter, Grad–Shafranov equation using the EFIT code,¹² and finally volume averages and normalization to calculate the normalized internal inductance. $ℓi$ is often a parameter passed into shallow machine learning architectures (e.g., Random Forests,⁷ SVM,⁶ etc.) for use in disruption prediction. Deep learning offers the potential to bypass this process and allow the algorithm to learn filters for identifying particular phenomena such as oncoming disruptions, neoclassical tearing modes, and Alfvén eigenmodes.

For learning on sequences, such as time series or sentences in a document, several types of neural network architectures can be used. In this section, we point out difficulties with traditional architectures used for sequence learning and discuss how deep convolutional neural networks with dilated convolutions overcome these difficulties.

Traditionally, special neural network architectures called Recurrent Neural Networks (RNNs) have been employed for sequence learning problems. These architectures have feedback connections among layers, allowing the use of information from previous parts in the sequence.² For years, the dominant flavor of RNNs has been the long short-term memory (LSTM) network,¹³ which overcame difficulties with training RNNs on sequences with long-range dependencies. LSTM networks use memory gates to enable them to effectively remember important pieces of previous sequences, in theory making them able to remember dependencies in infinitely long sequences.

However, in practice, LSTM networks (and even more so RNNs) tend to suffer from memory loss, forgetting sequence events that occurred much earlier.¹⁴ The length of the sequence that LSTM can remember will be highly dependent on the dataset, but a general empirical observation is sequence lengths in the thousands are the limit, i.e., a general rule of thumb for LSTM networks to be useful is $Tlong/Tshort ≲ 1000$ (where T_short is a fast timescale of interest in a sequence and T_long is a longer timescale that needs to be captured for a particular prediction).

CNNs have also been used for sequence learning^15,16 though traditionally they are not as common as the RNN/LSTM network. One difficulty in using CNNs for sequence learning is for scenarios that need to respect causality (i.e., do not use future time points to make predictions); to be sensitive to long sequences, you must increase the convolutional filter size and/or the number of layers in the network, both of which significantly increase the number of parameters for the network to learn.¹⁷ 

Recently, there has been much research into deep learning architectures, which can overcome the deficiencies of RNN/LSTM networks and handle long, multi-scale sequences. A seminal paper presented one such architecture, WaveNET,¹⁷ which is a convolutional neural network (CNN) focused on generating realistic audio. One of the key insights of this paper was to use dilated convolutions to increase the receptive field of the network (i.e., the number of sequence points used by a neural network to make a prediction at a single time point). This overcomes the dilemma faced with using normal convolutions in causal networks, where to be sensitive to long sequences, you must increase the convolutional filter size and/or the number of layers in the network. Dilated convolutions have a dilation factor (d) that represents the number of input points skipped between filter parameters, e.g., the sequence output $y[n]$ from a dilated convolution with dilation d is

where w represents the weights of the 1D dilated convolution filter of length k and $x[n]$ is the input sequence. A normal convolution results by setting d = 1. By stacking layers of dilated convolutions and increasing the dilation factor in each layer, the receptive field of the network can be increased while maintaining a tractable number of model parameters. The difference between normal and dilated convolution architectures is shown in Fig. 3.

Dilated convolutions impose an inductive bias or specific structure to the architecture, which guide the transformations learned by the neural network. Specifically, dilated convolutions have a natural connection with wavelet structures, which have been used for separating out the structure in multi-scale data, including turbulent flows.¹⁸ In a loose sense, these neural networks allow us to learn the wavelet coefficients needed to accomplish our classification task.

A simplified yet powerful architecture, named the temporal convolutional network (TCN),¹⁴ is built upon this WaveNET work, utilizing dilated convolutions and many modern neural network techniques, such as weight normalization (normalizes layer weights to speed up training) and residual connections (connections which skip layers, found to stabilize deep neural networks). Bai et al.¹⁴ showed that the TCN could outperform the LSTM network and a similar architecture, Gated Recurrent Unit (GRU),¹⁹ on many common sequence learning tasks, especially for long sequences with long-range dependencies. It is this TCN architecture that we will now apply to the problem of disruption prediction using ECEi data.

Before continuing, we mention that besides deep CNNs with dilated convolutions, there are other neural network architectures being researched and developed to handle long range dependencies, most focusing on Natural Language Processing (NLP) applications. Many of these use attention mechanisms, which learn which parts of sequences are most important for certain predictions, to better focus the neural network on relevant parts of the sequence. The most popular of these is transformer networks.^20–22 Non-local networks²³ also use a mechanism similar to attention. In addition, there are CNN architectures that have been enhanced with attention mechanisms.¹⁵ While attention can be computationally expensive, research is ongoing into making them more efficient.²⁴ It is left for a future study to explore applying these architectures to plasma time-series predictions.

Disruptions in tokamak plasmas are a sudden loss of control, which cause a termination of the plasma and potentially large destructive forces and/or heating on the containment vessel and protective wall materials. Next-step devices such as ITER and beyond will have low tolerance for disruptions.²⁵ We need to ensure that disruptions can be avoided by accurate prediction of oncoming disruptions and mitigation techniques if necessary.

Here, we apply the TCN architecture to high-temporal resolution, raw ECEi imaging data from the DIII-D tokamak for the purpose of predicting oncoming disruptions (code available at https://github.com/rmchurch/disruptcnn).

The ECEi diagnostic²⁶ records RF emission intensity at 16 tunable frequencies in the range of second harmonic electron gyromotion and along 20 lines of sight. An example time series of the DIII-D ECEi data near a disruption is shown in Fig. 4. Under conditions that are common in the core of many mid-sized tokamaks (optically thick, Maxwellian plasma having modest gradients in pressure, magnetic field, etc.), this signal is well-correlated with a local electron temperature.^27,28 The diagnostician locates the spatial region of interest by tuning the heterodyne receiver to a harmonic of the local cyclotron frequency (⁠$ωc=eB/me$⁠) and adjusting the quasi-optical lenses that couple radiation to the two ECEi antenna arrays. However, for the analysis presented in Sec. IV B, none of the metadata describing those operations is utilized; the neural net knows nothing of the instrument's design, operation, or potential correlation of the signal with physical quantities. The filtering and interpretation of raw ECEi data are spawned entirely from the machine learning procedure.

There are a number of different root causes for disruptions, including edge radiation, too high density, and MHD instabilities.²⁹ ECEi systems acquire data at such a high sampling rate and spatial resolution under ideal conditions that well-behaved signals span spatiotemporal scales to reflect the dynamics of turbulent fluctuations, Alfvén eigenmodes, tearing modes, sawteeth, ELMs, and other potential pre-disruption markers. Non-ideal conditions also impact the signal in ways that can be difficult for a human diagnostician to interpret, but are rich in information. For example, a sudden loss of signal can be the result of density cutoff. Alternatively, a sudden spike in the signal can be the result of a non-thermal electron distribution. A wide range of other conditions can impact the signal, producing fluctuations or other features that machine learning techniques might become sensitive to, even when the human data analyst finds them to be troublesome or ambiguous. By using raw, un-processed ECEi data, we make full use of all these signal features.

A dataset of 2747 DIII-D shots (⁠$∼42%$ disruptive and $∼58%$ non-disruptive, ranging from shot 144199 to 167542) was selected from a shot database in the DISRUPTIONS module³⁰ of the modeling framework OMFIT.³¹ Of the disruptive shots, about 45% were in the flattop phase of the discharge (full plasma current disruption). Shots were selected specifically where good ECEi data were available, defined as shots where each ECEi channel has a signal-to-noise ratio SNR > 3 (as a result, this excludes shots where ECEi was in cutoff). Even though the spatial location of the diagnostics can change between shots depending on the plasma magnetic field and ECEi user settings, we purposely made no effort to down select to only aligned shots or do any kind of spatial interpolation. Instead, we chose to rely on the neural network to learn pre-disruption markers from a variety of locations. Only data from the ECEi antenna measuring the low-field side of the plasma were used (20 vertical channels by 8 radial channels). The time length of each shot varies, typically between 5 and 10 s. The entire dataset (approximately 10 TB) was transferred to the Princeton Research Compute center, to enable the use of the Princeton Tiger GPU cluster, which has 320 Nvidia P100 GPUs, with 4 GPUs per node.

The ECEi data were taken in its raw, digitizer voltage output, i.e., without calibration to the actual electron temperature. The only corrections made were removing digitizer offsets by subtracting the average value before the shot begins (t < 0). As is common for neural networks,³² the input ECEi data were then normalized before input into the neural network, using the z-normalization. This is done by creating a per-ECEi channel mean (⁠$x¯$⁠) and standard deviation (σ_x) over the entire dataset and then normalizing the input as $xnorm=(x−x¯)/σx$⁠. Since $x¯$ and σ_x are constant over the dataset, it should still allow meaningful correlations to be learned by the network, including relative changes in absolute temperature between channels.

For ease of training the neural network, we decided in this initial paper to temporally downsample the ECEi data to 100 kHz (i.e., factor of 10× less data). We also pass the ECEi channels into the TCN without applying any initial 2D convolutions in R, Z space. This means that the TCN will learn the temporal correlations between channels, but that we do not explicitly tell the network about the spatial relationship between channels (the spatial relationship between channels is known implicitly by the network, by passing the channels in the same order every time).

We treat the problem of disruption prediction as a binary classification problem, where we predict whether each time slice corresponds to a “non-disruptive” or “disruptive” class. Rea et al.⁷ showed that data from time slices in a disruptive shot but far from the disruption time, t_disrupt, had similar data distributions as time slices from non-disruptive shots. Therefore, we label all time slices within 300 ms of a disruption as disruptive (⁠$tdisrupt−t<300$ ms), and all other time slices as non-disruptive⁷ (sequences from shots without disruptions are taken during established times of the discharge, i.e., during the plasma current flattop). Note that we could have instead selected by hand a number of events important for predicting disruptions (e.g., tearing modes, temperature drop from radiation, etc.) and had the network instead predict these events (i.e., multi-class classification), which may have some advantages in identifying important characteristics of disruptions. Choosing a binary target (is this time slice close to a disruption or not?) allows the network to learn implicitly a variety of these events through experience. Also note that we will aggregate time slice prediction into shot predictions, and they will relax the requirement that only time slices 300 ms before a disruption are considered disruptive. The intuition here is that we expect disruptive precursors to be present in a majority of disruptive shots at least 300 ms before a disruption. Further work could shorten this time, attempting to guarantee disruptive precursors during the disruptive time slices, or lengthen this time, to better capture pre-disruption markers that may occur much earlier.

Typical binary cross-entropy loss was used for the loss function during the neural network training,

where n is the number of time slices in a batch, y_n the target (binary, i.e., discrete, either 0 or 1), $ŷn$ the network prediction (continuous, ranging from 0 to 1), and w_n a constant class weight applied to help balance between disruptive and non-disruptive time slices.

We define our TCN model to have a receptive field of $Nrecept∼30 000$⁠. This is an order of magnitude larger than receptive fields in the original TCN¹⁴ or WaveNET¹⁷ papers. With the 100 kHz sampling rate, this means that each time slice prediction uses the previous $∼300$ ms in order to make the prediction. With our definition of disruptive time slices as within 300 ms of the disruption, this implicitly assumes that 600 ms before a disruption is sufficient to predict oncoming disruptions. We use a 4 hidden layer TCN with dilations $[1,10,100,961]$ (i.e., increasing by a factor of about 10 each layer), with a filter kernel size of 15. The number of filters per hidden layer was held constant at 80 (varying the number of filters per hidden layer was not attempted).

The TCN architecture allows parallelization of the sequence prediction by inputting sequences of length N_seq, which are longer than N_recept, resulting in $Nseq−Nrecept+1$ predictions per sequence. We are limited in the length possible due to memory constraints of the GPU and the need to process a number of sequences from a variety of shots for best learning (the group of sequences processed for a single update of network weights is called a “batch”). Empirically, it was found that sequence lengths of $Nseq=78 125$ allowed model computations that fit inside the GPU memory constraints, while allowing a batch size of N_batch = 12 (per GPU) to ensure a sufficient variety within each batch for training with stochastic gradient descent (i.e., each GPU processes $Nbatch×(Nseq−Nrecept+1)$ number of sequences for each weight update).

A multi-node, multi-GPU setup was used to parallelize the training. The Pytorch built-in synchronous data parallel training routine DistributedDataParallel was used,³³ training on 16 GPUs over 2 days. This makes the total effective batch size with data parallelism $Nbatch·Ngpu=192$⁠. A larger batch size can be achieved by reducing the sequence length, though at an increased computational cost due to more data reads.

Stochastic Gradient Descent (SGD) with Nesterov momentum 0.9 was used to train the model, with an initial learning rate of 0.5 that was decreased automatically upon plateau of the loss (ReduceLROnPlateau). A warmup period was used for the first 5 epochs (1 epoch is a single pass through the complete training dataset), increasing the learning rate from 0.0625 to 0.5 to enable larger batch training.³⁴ 

The set of sequences with time slices consisting of only the majority class (non-disruptive) was undersampled such that there were balanced disruptive and non-disruptive sequences,³⁵ leading to a total data size of $∼66$ GB for training. 10% of the data was set apart as a holdout validation set to determine during training how well the neural network generalizes. Caution was taken to ensure that the training and validation datasets had no shots in common. Two validation metrics were tracked: accuracy (how many time slices were predicted correctly as disruptive or non-disruptive) and F1-score (a harmonic mean between precision and recall). Because the time slice classes are imbalanced (even though the sequence sets are balanced), the F1-score gives a better indication of how well our classifier does on the minority class (disruptive), as can be seen with its equation,

where T stands for true (correctly predicted), F for false (incorrectly predicted), P for positive (disruptive), and N for negative (non-disruptive). Since the output of the TCN network $ŷn$ is continuous, we need to determine a threshold to map the continuous output to a discrete prediction. We use the common technique of calculating the accuracy and F1-score using a range of thresholds (for our case from 0.05 to 0.95 in 0.05 increments) and then select the threshold that maximizes the F1-score.

The results of training this TCN model on ECEi data for disruption prediction on DIII-D are shown in Fig. 5. The results are plotted over 1000 training epochs. The training binary cross-entropy loss continually decreases over the training, showing that our model has the capacity to learn the task from this dataset. The validation loss also continually decreases, slightly flattening toward the end, indicating that the model is reaching the limit of its generalizability after 1000 epochs. The two validation metrics are also shown in Fig. 5; the metric of accuracy reaches $∼94%$⁠, but more importantly, the metric of F1-score reaches $∼91%$⁠, showing that the neural network has learned to predict individual time slices of both disruptive and non-disruptive time slices very well.

We use a hysteresis threshold method³⁶ to consolidate the time slice predictions to shot predictions. This method triggers a disruptive alarm if the neural network output rises above a high threshold, σ_high, and afterward remains higher than a lower threshold σ_low for a time window τ_alarm. In this manner, it avoids spurious spikes in the predictions. A Bayesian optimization scheme using the optuna package³⁷ was used to find the optimal thresholds and time window parameters to produce the highest performance. Unlike for the time slice prediction, we do not only consider 300 ms before the disruption as disruptive but rather accept alarm triggers anytime during the shots (i.e., we allow an arbitrary class parameter from Ref. 36, $τclass→∞$⁠), up to 30 ms before the disruption (a common quoted minimum time needed for disruption mitigation). This means that alarms anytime during a disruption shot will be considered a success (true-positive). The Bayesian optimization routine was set to minimize the distance to a perfect true-positive rate $TPR=TP/(TP+FN)=1$ and false-positive rate $FPR=FP/(FP+TN)=0$⁠, i.e., the optimization objective was

where $θ*=(σlow,σhigh,τalarm)$ is the optimal set of parameters and y the neural network predictions. This produced parameters $σhigh=0.96, σlow=0.96$⁠, and $τalarm=1$ ms, with an shot F1-score of 0.944. A full grid search of the parameters was done to verify, and the resulting receiver-operator curve showing FPR vs TPR for each parameter combination on the validation dataset is shown in Fig. 6, along with the convex hull showing the best points. Using the convex hull, this gives an AUC of 0.963.

These optimal parameters were then used with the holdout test dataset predictions, in order to test how this predictor and set of alarm parameters would perform in the future. This produced a shot F1-score of 0.868, with the number of shots correctly and incorrectly predicted (i.e., confusion matrix) shown in Table I. As seen the true positive rate is encouraging, at $∼94.9%$⁠, however, the false positive rate is too high, at $∼17.8%$⁠. We hypothesize that the high false positive rate is due to the undersampling of non-disruptive shots and that with further training, and instead oversampling on the disruptive shots, this could be improved. Current machine learning disruption predictors typically achieve a true-positive rate in the low 90% on shots,^6–9 but with more reasonable false-positive rates (⁠$<10%$⁠). The rough goal is a true-positive rate of $>95%$ with a false-positive rate of $<5%$⁠.²⁵ The results presented here show that further work needs to be done using this machine learning technique, but, considering that this utilizes a single diagnostic, shows promise in being able to contribute to a machine learning disruption prediction solution.

The warning times given by these disruption predictions are significantly before the minimum time needed for disruption mitigation (30 ms), as shown in Fig. 7. The majority arrive at least 200 ms before the disruption, giving a sufficient warning time to, in principle, use control algorithms for disruption avoidance.³⁰ As seen, approximately 15% of the detected disruptions were detected more than 1 s before the disruption although more detailed analyses are needed to determine whether these were based on true disruption precursors or should be counted as false-positives.

Examining the output of the TCN disruption predictor can give insight into what the neural network has learned from the ECEi data. We first show in Fig. 8 an example shot with a flattop disruption, which the TCN predictor correctly identified roughly 260 ms before the disruption. Significant n = 1 mode activity is present, and the mode appears to lock starting near t = 3.0 s. The plasma survives another 500 ms, with the TCN disruption predictor alarm raised 220 ms after the mode locks (within the receptive field of the TCN). Also notice, though, that the TCN disruption predictor has high output during sudden drops in electron temperature near 2.0 s, though never sustained over 1 ms to raise the alarm.

There is ongoing work identifying the failure mechanisms of the TCN disruption predictor, particularly the false-positives (i.e., alarm raised during non-disruptive shots). Figure 9 shows such a shot, where the TCN disruption predictor appears very certain that a disruption is imminent. Examining the plasma time traces, this shot appears to have a minor disruption,³⁸ where there is a strong drop in core electron temperature at t = 2.0 s, but the plasma recovers. There is MHD activity leading up to and during the drop, as seen in Fig. 9. The TCN disruption predictor appears to trigger during the locking of the n = 2 mode, but closer analysis of magnetics also shows an $m/n=2/1$ mode present just previous to when the TCN disruption predictor triggers. Further techniques to identify the salient features that cause the neural network to trigger will be discussed in Sec. V. As the sawtooth amplitude and period are often modified near disruptions (as in Fig. 8), due to MHD mode coupling, the fact that the sawteeth do not substantially change could have been an indicator that this shot would not completely disrupt (though there is some evidence of sawteeth modification near minor disruptions also).³⁹ The ECEi diagnostic does capture sawteeth behavior; nevertheless, this does suggest that additional diagnostics, or more reduced physics-based extracted features, could augment the ECEi raw data in this disruption predictor, along with further training looking at more shots. This point will be further discussed in Sec. V.

These results show the usefulness of deep convolutional neural networks with dilated convolutions for fusion problems where the multi-scale, multi-physics nature mandates capturing long-range dependencies in time series. The results also demonstrate that the ECEi diagnostic captures rich pre-disruption dynamics sufficient enough to by itself be useful for disruption predictions. The techniques applied here show that deep learning can apply directly to raw data from diagnostics with high temporal resolution in order to make useful disruption predictions, a topic critical to the success of magnetic confinement fusion. These techniques can also be applied to various time-series sequence analysis problems, many of which are applied to data gathered using scientific instruments. They also show that training TCN networks with large receptive fields on the order of $∼30k$ are possible, allowing learning on long sequences with long-range dependencies. This initial paper presented the proof-of-principle of the technique. Comparisons using other machine learning techniques are needed to definitively show the benefit of these TCN architectures for this particular application of disruption prediction and the sole use of ECEi.

Looking forward to disruption prediction on machines such as ITER and beyond to fusion reactors, this work can help inform how to best utilize the diagnostic sets for data-driven disruption predictions and what diagnostic coverage would be needed for accurate predictions. The results presented here motivate the installation and use of high-resolution spatiotemporal diagnostics such as ECEi in future fusion reactors, which are feasible to install but may be in different forms compared to diagnostics in current devices.⁴⁰ The TCN algorithm is quite general and can be applied to other diagnostics and/or physics-based features, expanding the temporal information used in disruption prediction. As diagnostic coverage is expected to be reduced in fusion reactors due to a number of nuclear environment issues, this work could be expanded to help make the most of available diagnostics and inform a minimal set of diagnostics needed for accurate disruption prediction. Additional lines of research related to utilization of the work presented here in future devices such as ITER are detailed below.

For future work, there are many areas that can extend and enhance the work presented here.

First, pertaining to using deep learning with the ECEi diagnostic for disruption prediction, there is large potential to further increase the disruption prediction performance. Using the full dataset (no temporal downsampling and no undersampling of non-disruptive sequences) could further improve the results shown here. With the current computational setup, this may require a hybrid model/data parallelism in the parallelization of the training routine, where the model will be decomposed into on-node GPUs, and data parallelism will be used across nodes.⁴¹ Varying model parameters, such as the dilation factor, may also be a viable solution. Also, although the TCN architectures allow training on raw data, it may be that certain pre-training transforms (e.g., short-time Fourier transforms) may be sufficient for the task at hand and can simplify the neural network architecture needed (this would be an example of human crafted features, see Sec. II).

Second, combining multiple diagnostics (in machine learning parlance often referred to as multiple modalities) will most likely be mandatory for multi-physics problems such as disruption predictions. Disruptions have a number of different physics root causes, which may be more accurately predicted using diagnostics to be more sensitive to those physics, for example, bolometry diagnostics for impurity radiation-induced disruptions. Separate neural networks could be trained on each diagnostic and combined at the end to give a prediction, but newer techniques, such as feature-wise transformations, offer the potential to integrate multiple diagnostics into a single network,⁴² allowing correlations between diagnostics to be utilized in the disruption prediction. We can also explore inputting into the neural network more physics-based features, extracted from the raw diagnostic signals, like those that have been traditionally used in disruption prediction such as the n = 1 mode amplitude and frequency. Care must always be taken when using processed signals that causality is not violated by filters or transforms used to extract the features.

Third, understanding how to transfer these trained models to newer machines, such as ITER, is critical, where not many disruptions can be tolerated.²⁵ Purely data-driven techniques utilizing the trained neural networks can be used, namely, a technique known as transfer learning that enables learning on a small number of examples by retraining a neural network that has already been trained on a different, larger dataset from similar but not necessarily completely overlapping data distributions.^43,44 For similar diagnostics, it may be possible to apply transfer learning for use on different machines, which can be tested now, for example, by attempting to retrain the disruption predictor in this paper using the ECEi diagnostics on the KSTAR tokamak.⁴⁵ However, there are difficulties with this purely data-driven approach, as mismatches between spatial sizes, temporal sampling rates, and device physical timescales may make scaling or interpolation of data between devices impossible. Techniques, which can be viewed as more hybrid, including physics information with the data-driven approach, may be necessary. One possibility for including physics constraints can come from including simulation data in retraining models, making use of synthetic diagnostics for the new machine.⁴⁶ Another possibility is to create models for plasma behavior, including disruption, and extract model parameters from existing machines and their diagnostics.⁴⁷ 

Fourth, interpretability of the neural network predictions is greatly desired. While these algorithms can be treated as black-boxes for certain engineering uses (one example may be disruption prediction for triggering mitigation), when extrapolating to new machines, understanding why the algorithm is giving a particular prediction is very beneficial. There are various research studies into neural network interpretability,⁴⁸ including saliency methods that have been applied to, for example, neural networks on self-driving cars, to identify which pixels of an image most informed the neural network in the prediction on the direction to go.⁴⁹ Caution must be used to ensure that the inductive bias (i.e., selected structure) of the networks does not dominate the outputs of these methods.⁵⁰ Recent work using context activation vectors,⁵¹ a small set of samples with a domain-expert determined salient feature, has been used to identify when a neural network utilizes such features for a particular prediction. This could allow fusion physicists to isolate pre-disruption markers of interest (e.g., locked modes) and allow the neural network to output how important these markers were in its disruption prediction.

Fifth, recent upgrades to the DIII-D ECEi system give more accurate absolute electron temperature measurements, providing sharper details of modes in the plasma. System-on-chip (SoC) technology^52,53 provides superelectronics noise suppression and outstanding shielding performance against out-of-band interference. Also, the working frequency has been upgraded into the W-band (75–110 GHz), which is able to set the ECEi observation window in physically interesting regions of the pedestal or core. The new W-band SoC ECEi system has been calibrated with a standard electron cyclotron emission radiometer and Thomson scattering to provide real-time electron absolute temperature profiles. These upgrades give the promise of allowing neural networks to even more accurately capture and learn the pre-disruption dynamics.

Finally, this technique can of course be used to detect various fusion plasma phenomena, besides disruptions, which are of interest to operators and physicists. Specifically in high temporal resolution diagnostics, the data are often cumbersome to manually review. Creating an “automated logbook” using a neural network trained to identify phenomena of interest could create tremendous value in helping physicists sift through the data intelligently. For situations where labeling the data may be cumbersome, self-supervised techniques can be used,⁵⁴ where the network trains on a large diagnostic dataset and attempts to predict next time points for the diagnostics are used. This trained model can then be retrained on a small labeled dataset of a particular phenomenon in order to accurately identify other instances of the phenomena.

The main author would like to thank Dave Schissel, C. S. Chang, Bill Tang, Julien Kates-Harbeck, Raffi Nazikian, Cristina Rea, Bob Granetz, Neville Luhmann, Sean Flanagan, Ahmed Diallo, Brian Grierson, Nikolas Logan, and Ken Silber for various contributions to this work. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Fusion Energy Sciences, under Nos. AC02–09CH11466, DE-FC02–04ER54698, and FG02–99ER54531. We also recognize the Princeton Research Computing center for the computational resources used in this paper. Part of the data analysis was performed using the OMFIT integrated modeling framework.³¹ 

This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.

The data that support the findings of this study are openly available in Dataset at Princeton University at http://arks.princeton.edu/ark:/88435/dsp013b591c48v, Ref. 55.