Machine learning for modern power distribution systems: Progress and perspectives

AI

Artificial intelligence

ANN

Artificial neural network

D-OPF

Distribution optimal power flow

DER

Distributed energy resource

DNN

Deep neural network

DSSE

Distribution system state estimation

EV

Electric vehicle

FLISR

Fault location, isolation and service restoration

GAN

Generative adversarial network

GP

Gaussian process

GNN

Graph neural network

k-NN

k-nearest neighbors

ML

Machine learning

OPF

Optimal power flow

PCA

Principal components analysis

PMU

Phasor measurement unit

PV

Photovoltaic

PES

Power and energy systems

SCADA

Supervisory control and data acquisition

SE

State estimation

SSL

Semi-supervised learning

SVM

Support vector machines

V2G

Vehicle-to-grid

WLS

Weighted least square

Countries worldwide have set imperative targets for lowering greenhouse gas emissions to combat the worst impacts of climate change.¹ As part of these climate measures, efforts to decarbonize the electric power sector rely heavily on deploying renewable energy sources—predominantly wind and solar—at scale.² Reaching greenhouse gas emissions reduction targets also necessitates shifting transportation energy demand from fossil fuels to electricity through extensive vehicle electrification and supporting charging infrastructure, leading to a substantial increase in electricity demand.³ As the demand for electricity grows and the requirements for decarbonization persist, both the importance and the complexity of electric distribution systems are rising.

The integration of small-scale distributed energy resources (DERs), such as solar photovoltaics (PVs) and electric vehicles (EVs), into power distribution systems has increased dramatically in recent years. However, due to their intrinsic variability, uncertainty, and limited controllability, these new assets and technologies bring new challenges to the reliable and secure operations of electric distribution networks. Increasing numbers of DERs are exposing the limitations of existing analytical tools for operating and managing electric distribution networks, primarily because these tools were designed for traditional distribution networks wherein uncertainty arises only from the consumption profiles, and power is supplied predominantly from the transmission system to the distribution system. Meanwhile, the variability and uncertainty from weather-dependent renewable generation, along with limited sensing and visibility into distribution network operations, pose new challenges. These challenges will loom larger in future grids without advanced operation and planning methodologies to improve operational flexibility and guarantee network security, thereby maintaining stable voltages and frequencies.²³ Artificial intelligence (AI) and, particularly, machine learning (ML) have attracted considerable research attention as promising solutions to help address these challenges. The implementation of ML solutions is envisioned to complement or potentially even replace traditional, long-established physics-based modeling approaches used in various aspects of power system analysis. In this context, ML applications cover nearly every area of interest, including generation, transmission, distribution, and consumption, and extend into coupled energy infrastructure systems, such as heating, gas, and transportation. Similarly, ML applications in power systems can cover a broad range of timescales, ranging from sub-second intervals for transient stability to decades for planning. Table I summarizes recent review papers on various power and energy system (PES) domains.

A conventional notion of the ML process involves a learning algorithm using input data to find hidden patterns and structures in data, extract new insight from the data, or make predictions automatically. When a priori information (e.g., initial conditions) and knowledge (e.g., physical and mathematical principles) are incorporated into the learning process, what was previously purely data-driven (also commonly referred to as physics-agnostic or model-free) now becomes a hybrid methodology (also commonly referred to as physics-informed or scientific ML), more suitable for engineering applications (Fig. 1). A concrete example of this emerging learning methodology is the category of deep learning algorithms referred to as physics-informed neural networks.²⁴ The latest survey by Huang and Wang²⁵ provides a thorough review of the application of physics-informed neural networks in the domain of power systems, making it a valuable read for those interested in the topic.

In general, ML encompasses a wide variety of methods that can learn from (i) experimental data (in the case of PES, this is often data derived from computational simulations), (ii) observational data describing physical processes within the system (e.g., power system measurements), or (iii) both (observational and experimental/simulation data), to build predictive or explanatory models. As such, these methods may offer many advantages compared to traditional power system planning, operations, and control methodologies. Some examples include, but are not limited to: (i) reduced computational complexity for solving optimization-based problems prevalent in power systems; (ii) higher accuracy in linear modeling of nonlinear power flow equations and other linear function approximations, leading to improved solution accuracy and/or reduced computation time; (iii) the ability to process and analyze large volumes of measurements collected from heterogeneous data sources to derive valuable insight or predictions from such measured data; (iv) the capability to manage ill-posed problems; (v) more informed decision-making under growing uncertainty from DERs; (vi) intelligent support for fault diagnosis and monitoring in complex distribution networks; and (vii) learning hard-to-model functions, such as consumer behavior for demand response mechanisms that do not provide a closed system of equations. See Fig. 2 for a depiction of some potential application areas of ML in power distribution systems.

Consistent advances in AI have been fueled by the advent of massive amounts of data and the utilization of high-performance parallel computing using AI accelerator hardware like graphics processing units.²⁶ More recent advances in ML are characterized by the development of new algorithms, particularly within deep neural networks (DNNs)²⁷ and graph neural networks (GNNs).²⁸ As AI/ML advances at an unprecedented rate, it will likely have a significant impact on how power systems are operated and planned. Despite active research devoted to applying state-of-the-art ML techniques to power system problems, there is often a lack of attention paid to evaluating the suitability of the technique for a specific problem, as discussed in Sec. VI. For instance, data-intensive methods like DNNs may not be well-suited for power system problems where large amounts of data are not readily available. Similarly, while GNNs show promising advantages, many obstacles remain to be overcome for their usage in power distribution systems.²⁹ To address this gap, this article aims to provide insight into important topics concerning (i) ML for electric power systems with a particular focus on electric distribution networks, and (ii) challenges unique to this interdisciplinary research area. Unlike previous works that have either broadly covered these topics or targeted specific application areas or ML methodologies, this paper concentrates on the general ML efforts required for electric distribution systems, specifically the D and C domains from Table I. It does not aim to be a comprehensive survey of all plausible applications nor a thorough literature review, as that would be too ambitious. Instead, this paper discusses applications within the distribution system domain that can benefit from a broad range of ML concepts and techniques by identifying suitable application areas. The goal is to foster and stimulate interdisciplinary research between the two disciplines to guide future research and realize the potential of ML in power distribution systems.

The remainder of this paper is organized as follows: Secs. II and III provide background reading on power systems and ML, respectively, from basic concepts and theoretical fundamentals to state-of-the-art techniques. Section IV provides relevant examples from the distribution system domain in the context of each main ML paradigm. Section V overviews selected distribution system application areas where ML can realize greater potential but still needs to be explored. Finally, Sec. VI discusses open problems and recommendations on research directions, while Sec. VII concludes the work.

This section provides fundamental background reading on electric power systems, specifically focusing on power distribution systems. It is intended for readers with little familiarity with the topic and sets the stage for later discussions by considering various distribution system problems where ML can realize great potential. The section begins with an introduction to key concepts pertinent to power system operations and planning. It then proceeds to describe the basic power system modeling aspects. Additionally, it briefly touches on the advantages offered by ML in power system analysis, reserving more detailed discussions for Secs. IV and V. Finally, the section concludes with an overview of power system data.

An electric power grid is a complex system consisting of a transmission network that transmits electricity over long distances using high-voltage transmission lines and a distribution network that delivers electricity to consumers using low-voltage distribution lines. In traditional power systems, electricity flows unidirectionally from centralized, large-scale synchronous generators to loads. However, the increasing prevalence of DERs in modern power distribution systems has given rise to bi-directional power flows, resulting in a greater incidence of over-voltages, as illustrated in Fig. 2. These and other issues arising from the clean energy transition pose challenges to the operational reliability of power grids.

An essential requirement for reliable and secure power grid operations is a continuous balancing of electricity supply and demand, differences between which are manifested in changes in the grid frequency (preferably a constant 60 or 50 Hz depending on the country).³⁰ Any imbalance leads to a change in frequency, which drops from its nominal value in the case of insufficient generation and increases with oversupply. Traditionally, generation and load balancing have been achieved by scheduling (as part of the unit commitment problem) and dispatching (as part of the economic dispatch problem) a controllable generation fleet, such as coal and natural gas power plants, one day ahead and varying their output in real-time to match the varying load (as part of automatic generation control). Unit commitment and economic dispatch comprise a production cost model formulated as a mixed-integer programming problem that minimizes the bulk power system's total operating costs while adhering to the transmission network's and other physical constraints. In contrast, the capacity expansion model—also an optimization problem but with different objectives—focuses on planning and policy aspects of bulk power systems. The so-called optimal power flow is at the heart of these and other optimization problems and is ubiquitous in power systems (see Sec. II C 2).

Security of supply, paramount to reliable and secure power grid operations, is becoming more challenging with a high share of variable renewable energy.²³ These changes call for new services at all system levels, from ancillary services provided by renewable energy sources to optimal market frameworks for obtaining those services. Distribution utilities can only meet these new requirements by significantly improving current operational and planning practices, creating opportunities for active ML research in various distribution system contexts, as shown in Fig. 2.

The structure of the electric power system (known as the topology)—referring to the physical arrangement and connection of system components—can be conceptualized as a graph composed of nodes (buses) to which generators and loads are connected and edges (lines) connecting these nodes, as illustrated in Fig. 2. Depending on the problem at hand, the topology of the power system can be represented by a directed or undirected graph.³¹ This representation facilitates the mathematical derivation of the power system model used in all power system problems, from operations and control to planning. In general, the power system model comprises a set of equations describing the relationships between variables of interest within the timeframe under study while accounting for various components and their more-or-less simplified models, such as transformers, circuit breakers, loads, lines, and cables. Notably, the distribution network model differs from the transmission network model in that it has (i) a radial or weakly meshed network topology that frequently changes; (ii) high resistance-to-reactance ratios; and (iii) unbalanced phases due to the presence of many asymmetrical loads (single-phase and two-phase loads, primarily in North America). These network model differences imply that the distribution and transmission systems analysis also differs. For example, a direct implication of (ii) is that line resistances cannot be neglected, a simplification commonly used for traditional transmission system analysis.

This section presents a brief overview of three fundamental problems in power system analysis—the power flow problem, optimal power flow (OPF), and state estimation (SE)—with a focus on challenges related to distribution system analysis. The discussion frames the challenges in the context of ML applications, thereby serving as an introduction to the use of ML in power system analysis. As such, this section is a good primer for readers interested in exploring the intersection of power systems and ML.

The power flow problem, also known as the load flow problem, is instrumental for steady-state power system analysis. It entails determining the state of the system under normal operating conditions, including the voltage magnitudes and angles at all nodes within the network.^32,33 Once the system state is known, any other relevant quantity such as the real and reactive power flow and losses on each line can be analytically calculated.^32,33 This information is crucial for power system operators (hereafter referred to as operators for short) to ensure reliable and secure power system operations as it enables them to understand system behavior under different operating conditions, identify potential issues, and take corrective measures to maintain safe operating conditions. As a result, power flow analysis is a fundamental aspect of key power system operational tasks, such as OPF and SE, which will be further discussed below.

To avoid physical inconsistencies or implausible outcomes resulting from imperfect data or noisy measurements that can negatively impact the performance of ML models, it is necessary to move beyond purely data-driven models like (5).³⁴ This is where physics-informed models (Fig. 1) come into play and offer a promising research direction to pursue; see Sec. VI A 3 for a detailed discussion.

Remark 2. The OPF problem is nonlinear, nonconvex, and generally large-scale (with tens of thousands of variables involved), making it nondeterministic polynomial-time (NP-) hard to solve.³⁵ As a result, it is common to use less computationally expensive solution approaches to this problem, including:

• Linear approximations. This set of solution approaches is computationally efficient; however, it yields less accurate, sub-optimal solutions that may violate operational constraints.

• Convex relaxations. Although computationally more intensive than the previous solution methods, the main advantage of convex relaxation approaches is convergence to the global optimum and feasibility guarantees.

• Black-box or hybrid solvers. This set of solution approaches is ML-based, may be data-prohibitive, and typically does not provide optimality guarantees. Within this group, we can distinguish between end-to-end learning, which is the most popular in the PES literature, and learning-to-optimize approaches for OPF, as discussed in Ref. 36. The end-to-end learning approach employs supervised learning to map an input, such as nodal net power injections, to an optimal solution, such as voltages or power generation, resulting in high-fidelity optimization proxies for OPF solutions. Conversely, the latter approach leverages ML techniques to accelerate the computational speed of existing optimization algorithms used to solve OPF; one such example is using ML to facilitate the warm starting of OPF solvers.³⁶ As such, some of the learning-to-optimize approaches for OPF can be sub-categorized into physics-based ML models, which are also known as hybrid solvers.

For a detailed survey on approximations and relaxations of power flow equations in the context of OPF, particularly for the first two solution approaches listed above, the reader is referred to Ref. 37. More details on the third solution approach listed above can be found in Ref. 36.

OPF techniques have been effectively used to optimize bulk power system operations, such as economic dispatch of the generation fleet. However, applying the same OPF methods to distribution systems is challenging due to the differences outlined in Sec. II B. Moreover, the increasing presence of DERs in modern distribution systems exacerbates uncertainty, making distribution OPF (D-OPF) even more complex. As a result, operators need D-OPF solutions that can be executed in near real-time to accurately reflect system operating conditions and make timely decisions under uncertainties. Despite the significant number of published papers on the D-OPF problem, it remains an active area of research.³⁷ 

The recent literature has seen many instances of ML being applied directly to OPF/D-OPF or its variants, including security-constrained OPF and probabilistic OPF that takes uncertainty factors into consideration. For a brief overview of the use of ML in OPF, refer to Ref. 38. One exciting and promising research direction involves physics-informed neural networks for solving OPF while imposing constraints on neural network optimization functions.³⁹ The interested reader is referred to Ref. 40 and references therein for additional details.

SE is a critical inference task for enabling system-wide situational awareness by determining the most-likely operating state of the power system, typically represented by complex voltage phasors comprising voltage magnitudes and phase angles. To achieve this, SE requires two key inputs: (i) telemetry data available in sufficient quantity to make the network observable, and (ii) an up-to-date network model that includes network parameters and topology. The primary objective of SE is to provide an accurate real-time estimate of the system state (denoted by x) that is consistent with the available measurements, primarily current and voltage magnitudes in distribution systems.⁴¹ Specifically, the measurement vector z can be represented as $z = h ( x ) + e$⁠, where e denotes the measurement error. The nonlinear function $h ( · )$ relates system states to these measurements by using the previously defined (inverse) power flow equations.

Remark 3. Measurement redundancy and accurate knowledge of the network model are prerequisites for a classic SE framework, such as weighted least squares (WLSs).^41,42 The WLS state estimator has been used for decades in transmission systems; however, it is unsuitable for distribution systems where the SE problem remains undetermined due to the low observability of the network, as discussed in the Challenges section below. Furthermore, even if the network is observable, the WLS state estimator, or similar formulations, cannot guarantee a solution in all cases and may encounter convergence problems, especially for large-scale systems. Moreover, these state estimators are sensitive to measurement errors and bad data.^41,42

The direct application of the classic SE framework in distribution systems is severely challenged for three reasons:^41,42 (i) low observability due to scarce measurement devices and insufficient communications infrastructure; (ii) poor network models resulting from network model uncertainty, that is, imprecise network parameters and incomplete topology information; and (iii) unbalanced operations. Presently, distribution state estimators rely on so-called pseudo-measurements⁴³ due to the difficulties in addressing (i) and (ii) without substantial upgrades to existing communications and metering infrastructures. Unfortunately, such infrastructure upgrades are economically impractical due to the vast scale of distribution systems. On the other hand, load-derived pseudo-measurements used to compensate for insufficient telemetry data can propagate significant errors through the distribution state estimator, leading to their unreliable performance.^41,42 Moreover, (iii) renders the decoupled versions of state estimators used in transmission networks inadequate for distribution systems, requiring the development of three-phase state estimators. Hence, there is a growing interest in exploring techniques that enable accurate distribution system state estimation (DSSE) despite limited measurements and uncertainties in system models. Among these techniques, researchers have focused on ML^44,45 and sparsity-based approaches to DSSE, which are discussed below.

ML techniques have shown promising performance in various aspects directly or indirectly related to SE, such as bad data detection,⁴⁶ topology identification, including transformer-to-customer mapping and phase connectivity identification,⁴⁷ and generation and modeling of higher fidelity pseudo-measurements.^48–50 Furthermore, numerous ML-based state estimators have been proposed that can be broadly classified into two different categories, depending on whether they require difficult-in-practice knowledge of the distribution network model (model-augmented) or not (model-agnostic or model-free). Notably, many ML-based state estimators proposed in the literature assume a level of observability that exceeds what is usually observed in real-world distribution systems or employ sensor placement optimization strategies.⁵¹ Therefore, there are numerous opportunities for further research in this field.

Last, but not least, it is worth highlighting a specific methodological concept utilized within the realm of recommendation (or recommender) systems that is relevant to research in power system SE. For an introduction to recommender systems, see, for example, Ref. 52. While recommendation systems are commonly linked to platforms such as Netflix, YouTube, and Amazon, where ML is utilized to suggest items of potential interest to individual users based on data from other users, the mathematical concepts behind these systems can also be applied in the power system domain. Specifically, low-rank matrix completion has been used in the area of DSSE, particularly in the context of low-observable distribution networks. For further information, see Refs. 53 and 54, which provide examples of model-augmented and model-free approaches to voltage estimation, respectively. Apart from matrix completion techniques, the domain of sparsity-based DSSE has seen active research on other pertinent techniques, such as tensor completion.^55–57 

In power distribution system analysis, high-fidelity modeling is essential for obtaining trustworthy results. Traditional power distribution system modeling relies on complex physics-based models and rigorous mathematical principles. However, these models can lead to intractable optimization and simulation problems given the large scale of these systems. To address this scalability issue, power distribution system modeling is often simplified through assumptions and approximations that come in many forms, like linearization or convex relaxation. Such simplified models may be satisfactory for most operating points but can lead to sub-optimal solutions for others. This becomes more pronounced with increasing variability and uncertainty in system operating conditions due to the increased integration of DERs.

There are many examples where ML can offer an efficient alternative to conventional simplifications, especially where obtaining timely solutions is a primary concern, at the expense of solution accuracy. One such example is the use of ML to develop surrogate models—high-accuracy, computationally cheap approximations of a detailed analytical model—to accelerate analysis. Examples may include convex and nonconvex optimization problems (e.g., D-OPF, DSSE), Monte Carlo simulations, and approximation of nonlinear power flow equations embedded in optimization-based problems.

Computationally intensive Monte Carlo simulations are traditionally used to generate hundreds to thousands of scenarios pertaining to the different system operating conditions (i.e., power generation, consumption, and voltage level) for various relevant studies, such as security assessment under different DERs penetration levels. Identifying the most critical scenario (i.e., operating point) is highly challenging because the system state space can be vast.⁵⁸ To that end, ML techniques can be leveraged to optimally search the state space, thus circumventing the need to run distribution power flow analysis for each scenario individually.

By reducing the time burden of various analyses, ML can allow for more comprehensive and/or more frequent optimization, therefore improving the efficiency and reliability of the system. Additionally, because time-intensive model solving often limits the above problems to offline applications, ML methods are promising tools for shifting model solving from offline to online settings, which are more suitable for time-varying operational conditions. It is important to note that by replacing the original detailed physics-based analysis with ML surrogates, certain guarantees on performance may be lost, therefore limiting the application of ML to high-regret applications, especially in the context of system stability.

We close this section with an overview of the various types of data sources used in power distribution systems, accompanied by a brief discussion of important considerations to keep in mind when working with these data. As depicted in Fig. 3, diverse data sources, such as supervisory control and data acquisition (SCADA), weather stations, meteorological databases, and metering devices, including μ phasor measurement units (μ-PMUs) and smart meters, can be exploited for various ML applications. However, it is important to note that these data come with unique challenges, such as asynchronous measurements and limited access to real-world measurements.

The inconsistency in the temporal resolution of distribution measurements originates from the spatiotemporal heterogeneity of sensors installed throughout the electric distribution network, such as smart meters, μ-PMUs, and field equipment monitors. These sensors have varying sampling rates and reporting times, as shown in Fig. 4, which results in measurements of different temporal resolutions. To effectively utilize these measurements in ML systems, it is necessary to standardize their resolutions. This requires downsampling the higher-resolution data to match the lower-resolution data or vice versa (the so-called upsampling). This can be done in many ways, from trivial linear interpolation methods to more advanced ML methods; we highlight recent work⁵⁹ where low-resolution load profiles are upsampled into high-resolution load profiles using generative adversarial networks (GANs).

Another important consideration is the limited availability of measured data from real systems due to a lack of sensors in addition to privacy and security concerns. For these reasons, researchers are often limited to using simulated and open-source data to develop ML systems. However, simulations typically assume error-free measurements and no communication delays, which is unrealistic for real-world measurements. To address this issue, researchers may introduce noise that follows certain assumed distributions, which may not accurately reflect the noise patterns found in actual measurements. Additionally, open-source datasets may have undergone pre-processing and may lack certain features that could be relevant to the problem being studied.

This section serves as a brief introduction to the field of ML for audiences outside that community. It delves into the fundamental principles of ML and categorizes the different learning paradigms. By providing examples from a distribution systems perspective, it illustrates the concepts discussed and serves as a starting point for researchers to explore the potential of ML within the power engineering domain.

A learning task $T$ can be described as the process of optimizing a performance measure $L$ driven by a training experience $D$⁠.²⁶ This task typically involves three main components: the input data (also known as the training set or feature set), the model (or algorithm), and the output or prediction (also known as the label or target).²⁶ To simplify the discussion, in the following, we will formally define the learning task assuming the supervised learning setting, which is covered in more detail in Sec. IV A. Not only has this type of learning been widely studied in the field of PES, but it also provides a sound basis for explaining the fundamental concepts of ML.

To begin, let us briefly apply the abstract ML terminology above to a concrete example within the distribution systems context. In power systems, a microgrid is defined as a small electricity network that can operate either connected to the larger grid or in an autonomous mode. Consider the example of microgrid formation to facilitate distribution network recovery during low-probability, high-impact events, such as extreme weather events. The learning task in this scenario can be formulated as segmenting the distribution network into autonomous (microgrid) zones. The performance measure to be improved in such a learning task may be the minimum time required for the distribution network's restoration or alternatively the number of critical customers facing power outages. The training experience may involve defining all possible microgrids within the observed network beforehand.

In (8), $ℓ$ is a loss function used as a learning criterion for the optimization problem (8a), and $R$ is the optional regularization term used to reduce the risk of overfitting.

Remark 4. What does a good model mean in the definition above? In the field of ML for PES, the term good model is not clearly defined and is open to interpretation. Generally, a good model is one that generalizes well to previously unseen data. However, when it comes to PES, other criteria should be considered when defining a good model. For example, features of a model that address barriers to adoption in the real world should be considered. Nevertheless, the lack of a clear consensus on what constitutes a good model in PES research can lead to difficulties in the practical application of ML methods in this field. Therefore, researchers must establish clear standards and definitions for good models in PES in order to make the use of ML techniques successful in the real world. Developing realistic benchmark problems can help the community at large agree upon a consistent definition of a good model for each application of ML to PES.

The accuracy of an ML algorithm is quantified using the loss function—a mathematical function that measures the difference between the predicted output (also known as the prediction or estimate) produced by the ML algorithm and the actual output (also known as the ground truth). The loss function is optimized during the training process using techniques such as gradient descent to enhance the model's predictive efficacy.⁶¹ 

As discussed above, the learning task is driven by the training experience, or in other words, the data. However, raw data are rarely used in the learning process; they often require prior processing and formatting. These steps generally fall under the umbrella of data engineering and are vital to developing quality ML systems.

The first step (data pre-processing) may include some, if not all, of the following sub-steps:

• Data cleaning, which is used to detect and correct or remove corrupt observations—missing data and outliers;

• Feature selection, which involves selecting an “optimal” subset of independent variables (features) from among many less useful ones; this implies ignoring other features as irrelevant. These techniques are instrumental when there are many features and comparatively few data observations;

• Feature scaling, which involves normalizing the range of data features. Examples include mean normalization, min–max normalization, and standardization.

The second step (data formatting) is to ensure that the input data are well formatted. The most common is the tabular format (the so-called vector–matrix data representation) where each row and column of the table represents a particular example (also called instance) and feature (also called covariate), respectively. Let us consider a single distribution feeder and the corresponding past measurements at each node as an illustrative example of how this system can be represented in tabular form. In such a table, each row corresponds to a time instance when the measurement snapshots were made, and columns correspond to different measured variables at each node (e.g., voltages, injected complex power, consumed complex power).

Remark 5. In the context of ML applications within the distribution system domain, the data utilized are primarily structured in a tabular format. There are, however, certain exceptions to this, such as satellite imagery that can be used to estimate behind-the-meter solar generation or thermal images from distribution lines and transformers inspections that can be used for predictive maintenance.

The evaluation of an ML model's performance should take place on an independent test dataset. In general, the full dataset is sampled into two (train/test) or three (train/validation/test) distinct sets:

• Training set, typically the largest sample of the data, is used to learn a predictive ML model;

• Validation set is a smaller data sample used to evaluate the model's performance during training and fine-tune its hyperparameters to prevent overfitting or underfitting to the training data;

• Test set is an independent dataset that is not used in the training or hyperparameter tuning process. Instead, it is used to estimate the performance of the predictive ML model on new, unseen data.

Another technique for evaluating and selecting ML models is cross-validation.⁶³ In this approach, the data are partitioned into subsamples, and the error rate is estimated as the mean of the error rates calculated from all the data subsamples.⁶³ 

The field of ML encompasses various learning paradigms, including supervised, unsupervised, and reinforcement learning. These paradigms have been extensively studied and widely adopted across various application domains.²⁶ A summary of their key characteristics and related algorithms is presented in Fig. 5, color-coded to help differentiate them in Sec. IV, where they will be discussed individually in more detail. In Secs. IV A–IV C, we will focus on these main learning paradigms, including semi-supervised learning (SSL) in Sec. IV D. For a comprehensive overview of recent advancements in the field, including transfer, multitask, and multiview learning, the interested reader is referred to Refs. 64 and 65.

Among the different ML paradigms, a noteworthy distinction is the difference between offline and online learning. In offline learning, a batch of data samples is processed simultaneously, while in online learning, data samples are processed sequentially as they arrive over time. Although supervised and unsupervised learning methods can be implemented using either offline or online learning strategies, these methods are traditionally associated with offline realization. In contrast, reinforcement learning, which relies on sequential interactions between the agent and the environment, primarily operates online. Nonetheless, there are also less represented forms of reinforcement learning, such as batch reinforcement learning that decouples data collection and policy training processes, thereby updating an agent's policy with a batch of pre-collected data.⁶⁶ Another such form is pure batch (or offline) reinforcement learning, which aims to learn policies solely from a suitably diverse and sizable dataset, devoid of any online interactions.⁶⁷ 

Another distinction is between discriminative and generative models. Discriminative models learn the conditional probability distribution $p ( y | x )$ while generative models learn the joint probability distribution p(x, y).⁶⁸ While discriminative models are primarily used for classification tasks, they can also be used for other types of supervised learning tasks, such as regression and structured prediction, wherein the output is a structured object, such as a sequence or a tree.⁶⁹ On the other hand, generative models are primarily used for unsupervised learning tasks of generating new data samples that resemble the training data. Generative models are particularly promising for applications where the training dataset is small or difficult to obtain (see Sec. IV B).

This section provides a succinct overview of the primary learning paradigms outlined in Sec. III B and displayed in Fig. 5. At the end of each subsection are examples in order to demonstrate how the methods can be utilized in the PES domain. Most of the selected examples, including but not limited to predictive maintenance, phase identification, and voltage control, have been extensively studied over the years and are therefore well-suited to illustrate the considered ML paradigms in the context of power distribution systems.

In supervised learning, the goal is to learn a model from labeled data $D = { ( x ( i ) , y ( i ) ) } i = 1 m$ to generate an output y for each input x (or a probability distribution over y given x), which can then be used to make predictions on new, unseen (test) data. There are two primary tasks within supervised learning—classification, where the output y is categorical, and regression, where the output y is numerical:

• Classification can be further divided into two tasks. The first is binary classification, which aims to determine the class (out of two possible classes) to which a given data instance belongs. The second task is multiclass classification, which assigns a data instance to one of K categories. In the context of power systems, this could include determining if a fault is an open-circuit fault, symmetrical short-circuit fault, or asymmetrical short-circuit fault. Some popular classification algorithms include logistic regression, support vector machines (SVM), k-nearest neighbors (k-NN), and decision trees. Ensemble variants of decision trees, such as random forests, are also commonly used.

• Regression is a task that involves making predictions about future outcomes by learning the relationship between a dependent variable and one or more independent variables. In the context of power systems, an example of a dependent variable could be the generated power from a PV system, while independent variables could include factors such as solar irradiance, ambient temperature, and PV system specifications. The common regression algorithms include linear regression, polynomial regression, SVM, neural networks, and random forests. These algorithms are widely used in the PES field to make predictions and projections based on historical data.

It is important to note that many supervised learning techniques, including decision trees, random forests, and artificial neural networks (ANNs), can describe complex, nonlinear relationships between input–output pairs. Among them, ANNs have been widely recognized as universal nonlinear function approximators, capable of approximating arbitrary functions with high accuracy.⁷⁰ Recent advances in DNNs have further improved the capabilities of ANNs. DNNs utilize multiple layers of nodes to capture increasingly complex data relationships, making them the most powerful ML technique studied extensively across various research fields, including PES.

Supervised learning is arguably the most extensively studied learning paradigm in PES research. Two selected examples showcase the application of supervised learning in forecasting tasks related to distribution systems (Example 1) and for predictive maintenance (Example 2).

Unsupervised learning analyzes unlabeled data $D = { ( x ( i ) ) } i = 1 m$ with the aim of identifying underlying structures or uncovering patterns that may be present in the data. Some examples of tasks that fall under unsupervised learning include:

• Clustering is a task within unsupervised learning that groups unlabeled data into clusters based on their similarities. Standard clustering algorithms are Gaussian mixture models, k-means, hierarchical, and spectral clustering. For a comprehensive overview of clustering algorithms, see Ref. 76.

• Dimensionality reduction is a technique that is frequently employed in the pre-processing stage of data analysis to represent high-dimensional data with significantly fewer dimensions while maintaining the integrity of the data. There are various dimensionality reduction techniques that can be used, with the most popular being principal components analysis (PCA),⁷⁷ and autoencoders.⁷⁸ 

• Anomaly detection is a commonly employed technique for identifying observations that deviate from the norm, also known as outliers or anomalies. Among the various algorithms for anomaly detection, two notable examples are one-class SVM and isolation forest.⁷⁹ In the context of PES, anomaly detection can identify abnormal patterns in measured data that may indicate a deviation from expected system behavior, such as diagnosing faults using anomaly detection algorithms on voltage data.

• New sample generation involves generating new samples or scenarios with distributions that are representative of the actual distribution of the given data. It is important to note, however, that the quality of the generated scenarios is contingent upon the quality of the input data. One prominent algorithm employed for this task is the GANs, which consist of a pair of neural networks—a generator and a discriminator.⁸⁰ The generator is trained to produce data samples that closely resemble the training data, while the discriminator is trained to accurately distinguish between the generated samples and the actual training data.⁸⁰ This process is conducted in an adversarial manner where both models are trained simultaneously.⁸⁰ Unlike traditional scenario generation methods like Monte Carlo simulations, GANs do not require a priori assumptions about the probability distributions of the input data, making them a more attractive option for PES applications. As a result, GANs have gained considerable attention in recent years, and various studies have demonstrated their efficacy in generating near-realistic scenarios for a wide range of applications. These include profiles for electricity demand, solar PV, and wind generation.^81,82

Unsupervised learning has received less interest in PES research due to its lower accuracy compared to its supervised counterpart. Nonetheless, unsupervised learning can be highly useful, particularly for analyzing unlabeled data or in situations where there is not enough labeled data to use supervised learning techniques. In the context of power distribution systems, two common applications of unsupervised learning techniques are phase identification and customer segmentation. Two selected examples (Example 1: Phase identification and Example 2: Customer segmentation) are presented below to illustrate these applications.

Reinforcement learning is a popular ML paradigm for learning an optimal policy or set of actions through a trial-and-error search and a reward system.⁹⁰ The process, diagramed in Fig. 8, involves an agent interacting with a dynamic environment through a series of actions, each of which leads to a state transition and affects the agent's subsequent actions, all aimed at maximizing a cumulative reward function. In the context of distribution voltage control, exemplified in Fig. 8, reinforcement learning is used to govern the power output of a PV system, thereby minimizing voltage fluctuations and maintaining acceptable voltage levels across the distribution grid. This topic has been covered in more detail in Sec. V B 1.

A key distinction between reinforcement learning and supervised learning is the type of feedback provided during training. In supervised learning, training data provide quantitative feedback in the form of actual output for a given input (ground-truth). In contrast, reinforcement learning utilizes qualitative feedback, indicating whether the action taken is correct or not. Algorithmic formulations of reinforcement learning include Q-learning, deep variational learning, policy gradient, actor-critic, and state-action–reward–state–action (SARSA), among others.

Reinforcement learning has been extensively studied in recent years for decision-making and control problems in power systems. Within this framework, power system problems are typically modeled as Markov decision processes or variants thereof, such as partially observable Markov decision processes. Broadly, the potential applications of reinforcement learning in power systems can be categorized into two main groups, namely, game-based studies and search-based studies, as indicated in Ref. 58. Game-based studies cover various tasks, such as demand-side management (e.g., Example 1: V2G), strategic bidding for different electricity markets, and power system control tasks that involve coordination over multiple agents (e.g., Example 2: Voltage control). On the other hand, search-based studies include⁵⁸ system fault diagnosis, security assessment, and cascading outage prediction, to name a few. For a detailed analysis of the advantages and limitations of reinforcement learning in decision-making and control applications for power systems, we recommend referring to the latest review paper by Chen et al.⁶ 

We conclude this section with semi-supervised learning (SSL), which is conceptually situated between supervised and unsupervised learning.⁹⁷ This learning paradigm is particularly relevant when labeled data are scarce or costly to obtain, as it allows for the utilization of unlabeled data to improve model performance. In SSL, the model is trained upon a combination of labeled and unlabeled data $( x ( i ) , y ( i ) , ) i = 1 m ∪ ( x ( j ) ) j = 1 m ′$⁠, the ratio of which usually significantly favors the latter (i.e., $m ≪ m ′$⁠). The labeled data are utilized to establish the relationships between input and output variables, while the unlabeled data are employed to enhance the model's generalization capabilities.⁹⁷ The most popular and widely used SSL methods include:

• Self-training⁹⁸ uses an initial set of a small amount of labeled data to train a supervised learning model and then applies that model to classify the unlabeled data. The most confident predicted labels are then added to the labeled dataset and used to retrain the model.

• Co-training⁹⁹ is a variation of self-training where two or more classifiers are trained on different views of data, and their predictions are combined to predict labels for the unlabeled data.

• Transductive SVM¹⁰⁰ is a variation of SVM specifically designed for SSL that leverages the unlabeled data to find the decision boundary that best separates the labeled data.

• Pseudo-labeling¹⁰¹ involves training an initial model to predict “pseudo-labels” for previously unlabeled data. The pseudo-labeled and original labeled data are then combined, and the model is retrained on this larger training set. For example, consider the task of classifying a distribution line as either undergoing an outage or operating normally using historical measurements. If only a small portion of the dataset is annotated with the status of the line, the labeled dataset alone is likely insufficient to robustly train a supervised learning model. In this situation, pseudo-labeling can augment the original labeled dataset with pseudo-labeled data, potentially increasing the accuracy of the resulting outage detection model.

• MixMatch¹⁰² combines the concepts of data augmentation and “mixup” on both labeled and unlabeled data to generate new examples, and then trains a model on the mixed data using a combination of supervised and unsupervised loss functions, thereby improving the model's performance.

SSL presents a significant area of research potential within the field of PES, particularly when labeled data are scarce. One example is the classification of rare events that disrupt normal power system operations due to extreme weather events or cyber-events. In such scenarios, traditional supervised learning solutions may result in sub-optimal decision boundaries. However, SSL can overcome this limitation by incorporating both labeled and unlabeled data into the learning process to aid in such classification tasks. Specifically, the inclusion of unlabeled data allows the SSL algorithm to infer close-to-optimal decision boundaries, thereby improving classification accuracy. Previous studies have applied SSL techniques for fault diagnosis and event detection,^103,104 detection of cyber-attacks such as false data injection attacks,¹⁰⁵ electricity theft detection,¹⁰⁶ and nonintrusive load monitoring.^107,108 However, the full potential of SSL has yet to be explored in various distribution system applications. As an illustration of one such application, the detection of voltage harmonic distortions is chosen, although it should be noted that SSL can also be used for the detection of other power quality disturbances.

Remark 6. The distribution system examples presented in this section assume a particular learning paradigm. However, the formulation of certain tasks can be approached from alternative perspectives, thereby enabling the potential utilization of learning paradigms other than those previously demonstrated. For instance, supervised learning techniques may be utilized for phase identification tasks,¹¹⁰ while unsupervised learning approaches may be employed for predictive maintenance. Additionally, SSL can be utilized for customer segmentation tasks, such as identifying household profiles,¹¹¹ as an alternative to unsupervised learning methods. Furthermore, many problems encountered in distribution systems exhibit a high degree of complexity, which can be mitigated by decomposing them into smaller sub-problems, akin to the divide-and-conquer algorithm approach. Subsequently, various learning paradigms can be utilized to address these sub-problems.

This section aims to identify application areas in power distribution systems that could greatly benefit from ML contributions in the future. The identified areas are organized into subsections according to the shared challenges that make them suitable for ML applications. Figure 9 provides an illustrative representation of these categories, aiding readers in comprehending the potential of ML in each application domain. Although this section does not intend to comprehensively survey all relevant ML applications in distribution systems, the insight presented herein offer valuable perspectives on how ML can improve the operational and planning practices of distribution systems, especially in cases where conventional tools are inadequate or inapplicable.

The following examples showcase the use of ML in tackling the task of constructing models pertaining to distribution system operations that are difficult to develop based on first principles alone. ML-based models can help predict system behavior and enhance system resiliency in response to evolving consumer and environmental factors, where traditional first-principles models are not available. Here, we can distinguish between data-driven models that leverage the vast amount of data to capture complex behaviors that are difficult to model and agent-based models, which allow for the representation of complex interactions between system components and agents.

A highly promising area of ML applications in the power distribution system domain involves modeling consumer behavior for demand-response programs. For example, intelligent building control systems rely on accurate modeling of occupant behavior in buildings. However, the complexity of consumer behavior—driven primarily by human habits and weather—renders it challenging to model explicitly using mathematical equations. Likewise, modeling the impacts of household-owned batteries and EV charging on power distribution systems is also confronted with similar challenges. Mathematical equations alone may not fully capture the complex consumer behavior regarding energy consumption patterns, whereas ML techniques can excel in capturing such unknown complex relationships. More accurate models can, in turn, facilitate the development of coordination strategies for DERs to provide efficient demand-response. At present, research in this area is largely focused on reinforcement learning and deep reinforcement learning. Interested readers are directed to Refs. 112 and 113 for an in-depth review of occupant behavior modeling in buildings and EV charging management using reinforcement learning techniques, respectively.

Given the many successful examples of ML predicting consumer behavior in other application areas (e.g., recommender systems, targeted advertising), there is a powerful set of techniques available that could be extended and applied to understanding, predicting, and shaping consumer impacts on the distribution system. Nevertheless, leveraging ML techniques to model consumer behavior and/or impacts requires careful consideration of equity and fairness. Reference 114 provides a comprehensive overview for readers seeking a general understanding of the increasing concerns surrounding bias, fairness, and equity in ML-driven applications. There are numerous steps that can be taken to mitigate bias and ensure that ML systems are not only accurate but also fair and equitable.¹¹⁴ This matter is particularly critical for equitable market predictions in future electricity markets with a plethora of diverse and small-scale participants (commonly known as prosumers), or when executing demand-response initiatives to guarantee an equitable and unbiased outcome for all parties involved.

While many of the detailed power system analyses rightfully focus on analyzing the power system in isolation, in some cases, the power system is dramatically impacted by external systems. Of particular interest is modeling and predicting the impact of natural disasters on the distribution system as such events become more common and more destructive due to the effects of climate change. ML is a promising approach to capture such complex interactions and take advantage of the rich and varied weather datasets to build predictive models relevant to the power system.¹¹⁵ Wildfires are another example of a natural system that is bidirectionally coupled to the distribution system—wildfires can cause power outages and power lines can ignite wildfires—and becoming more prevalent due to climate change. Recent works have begun to address how wildfire risk should factor into power system operations.^116,117

The subsequent application areas of ML in power distribution systems underscore its capacity to speed up computations or simplify modeling and simulation. To avoid repetition, this section will not reiterate on other pertinent topics that were previously discussed, such as the utilization of ML to accelerate D-OPF by optimizing the candidate space or linearizing power flow equations.

Power distribution systems worldwide are witnessing an increasing adoption of power electronics in various forms, chiefly inverter-interfaced renewable generators (such as PV systems and wind turbines) and power electronics loads (such as power electronics-backed air conditioners). Accordingly, the traditional approach to voltage control in power electronics-dominated distribution networks is becoming increasingly challenged.¹¹⁸ Instead, the active participation of distributed local power electronics will play a key role in various system-wide control mechanisms (e.g., voltage and frequency control), further increasing the complexity of these problems. Furthermore, conventional voltage regulation devices, such as transformers' on-load tap changers and capacitor banks, may not be adequate to respond to rapidly changing operating conditions due to their slow response times, which can range from hours to days. Moreover, traditional optimization-based methods for voltage control, rooted in the D-OPF framework, may experience slow computational speed and convergence difficulties.

Given these challenges, ML and data-driven techniques—in particular reinforcement learning and deep reinforcement learning—have emerged as a promising avenue for research that can help to (i) enable faster convergence of existing voltage control methods for real-time control or (ii) develop novel control methods adapted to the use of local power electronics controls. The basic concepts of the application of reinforcement learning to the voltage control problem were presented previously in Sec. IV C, Example 2, and are therefore omitted here for brevity. Reinforcement learning has proven effective in this area to deal with the complexity of the voltage control problem without requiring detailed system models (which are often not available) and in a distributed manner that can scale to large systems. Recent work in this area has focused on algorithms that not only learn good policies for nominal conditions, but also provide some guarantee that safety constraints will be met⁹⁶ or that the system will remain stable.^119–121 These works can serve as an example for other application areas in providing guarantees that will aid in the adoption of data-driven techniques (see Sec. VI B 4).

Traditionally, protection relay schemes in distribution systems have been programmed for unidirectional power flows.^122,123 The most common is overcurrent protection, which is designed to operate when the short-circuit (fault) current exceeds some predefined threshold (i.e., $I f ≥ I min )$⁠. However, the increasing displacement of synchronous generators with inverter-based resources may lead to the miscoordination of overcurrent protection schemes.¹²³ This is due to these resources' notably lower fault current contributions than those in traditionally operated distribution networks. In addition to potential miscoordination, reduced fault currents in power electronics-dominated distribution networks can make fault detection more difficult. Another concern is related to more frequent reverse power flows that can adversely affect normal network operations by causing relay protection to malfunction (i.e., unwanted tripping). This calls for new protection concepts that should gradually replace the functionality of traditional protection systems to ensure stable and safe distribution system operations during both normal and contingency operations.¹²³ 

The idea of adaptive protection has emerged as a potential solution;¹²² however, it is still in its infancy, as is the application of ML for protection.^124–126 Nevertheless, ML has the potential to play a crucial role in advancing the field of relay protection in distribution systems. Specifically, ML techniques can aid in developing adaptive relay protection schemes, making it an exciting future research direction. By utilizing look-ahead search techniques, such as Monte Carlo Tree Search,¹²⁷ the accuracy and speed of fault detection and isolation can be improved. Moreover, ML-based solutions can replace the conventional lookup tables used for protection relay settings, thereby enabling the establishment of more accurate and adaptive protection settings. However, implementing such solutions requires extensive offline analysis of all possible future outcomes, which can be prohibitively expensive. Despite this, the advantages that ML can bring to relay protection for active distribution systems make it a worthwhile topic for future research and development.

Another potential application is in developing data-driven models that capture the aggregate dynamic behavior of an active distribution system for use in transmission level analysis (modeling both domains in detail is computationally prohibitive). Future research should focus on how ML-based dynamic models can be integrated seamlessly into existing dynamic simulations which are to-date centered around physics-based models.

The following application areas demonstrate how ML can be employed to alleviate issues related to missing, incomplete, or nonexistent data. In addition to the topics covered below, several other ML applications are relevant to this category, including synthetic data generation for creating pseudo load profiles used in DSSE (discussed in Sec. II C 3) and topology and phase identification (see Example 1 in Sec. IV B).

Advanced demand-side management can help mitigate new operational challenges facing system operators by allowing them to shape and modulate flexible loads optimally. However, systemic missing or out-of-date information (e.g., size and location) on installed solar PV panels on residential and commercial rooftops makes demand-side management difficult at scale. Some reasons for this lack of data include older installations made prior to regulations being enacted, lack of owners' awareness of permitting rules, unauthorized installations to avoid permit fees, and discrepancies between reported and installed configurations.¹²⁹ At the same time, registration of unregistered PVs can be highly challenging due to their large numbers in poorly visible secondary distribution networks.

Because accurate and up-to-date repositories of distributed PVs and other flexible loads are a prerequisite to effectively employing demand response, ML-based solutions are gaining research attention. In one line of research, ML has been proposed to classify and segment rooftop PVs using aerial and satellite imagery, thereby enabling the creation of more accurate registers (databases) according to the spatial scale of interest (e.g., street or city level). An overview of opportunities and challenges on this topic can be found in Ref. 130. Other data-driven solutions proposed in the literature rely on smart meter measurements (time-series data, as opposed to images) to detect abnormal energy consumption behaviors, including unauthorized PV installations. An exemplary work in this category is Ref. 131, where the corresponding solution is developed using a change-point detection algorithm.

The traditional operations and “fit-and-forget” approach to distribution network planning (capacity hosting analysis) are becoming less than adequate with the widespread adoption of DERs. The necessary changes in operational and planning practices rely on increasing the grid's last-mile visibility, which has historically been difficult to achieve due to a lack of metering devices and insufficient communications infrastructure.^41,42 However, the massive rollout of smart meters has significantly increased the number of metering devices installed at the grid's edge.¹³² In addition to smart meters, new untapped sensing potential comes from broadband cable television networks, with sensors deployed across the countries and their own high-speed, low-latency communications network.¹³³ Unlocking the potential of these sensors can enable greater insight into the operations of last-mile networks, thus allowing operators to react promptly.

Despite the influx of new data, missing measurements due to metering sensor errors or communication disruptions will become an inevitable challenge. However, leveraging the low-rank property of the streaming data matrix can allow for the utilization of low-rank matrix completion and tensor completion techniques, as well as certain ML methods, to recover these missing measurements.^134–136 Although data recovery is an important topic, the lack of visibility into distribution system operations currently presents more pressing challenges. For instance, outage detection in distribution systems remains a nontrivial task for operators who still rely on customer calls to identify power outages, sending crews to locate and eliminate the cause. To overcome this, fault location isolation and service restoration (FLISR) is a promising application in the advanced management of distribution systems, but it requires improved observability. To improve situational awareness at the distribution system level, the PES community has recognized the numerous opportunities that ML can offer, in addition to those discussed in Sec. II C 3 in the context of DSSE. For example, ML can be applied for outage detection,^137,138 fault location,¹³⁹ and service restoration,^140,141 among others. Readers interested in further information on ML for FLISR are encouraged to refer to Ref. 10, which provides a comprehensive literature review on AI applications for distribution system operations.

ML methods and implementations must be characterized by well-posedness, reliability, and robustness in order to be accepted for practical application in distribution systems beyond the research community. Furthermore, their outcomes must be readily verifiable, validated, and reproducible. To attain this, several key challenges must be effectively addressed, including: (i) data availability, quality, nonstationarity, and privacy; (ii) robustness, scalability, and performance guarantees of selected models; (iii) adherence to physical principles and laws; (iv) explainability and visualization of obtained solutions (i.e., interpretability); and (v) a unified evaluation methodology for ML performance, encompassing well-defined metrics and structured baselines for comparison. Figure 10 outlines the identified research directions aimed at addressing these critical challenges through the lens of applied ML. The discussion in this section is centered on challenges and opportunities, emphasizing the critical research directions necessary for making ML a credible and trusted methodology in distribution systems. Nevertheless, we argue that the presented discussion can be generalized to other systems within the PES domain, including bulk power systems.

The first research theme is centered around three main strategies: (i) leveraging domain knowledge (i.e., expert feedback) on data acquisition and model selection; (ii) incorporating physical principles and governing laws into ML models; and (iii) domain-aware decision-making or policy-making. Each of these strategies comprises multiple challenges that will be discussed in subsequent subsections VI A 1–VI A 4.

The centrality of data in ML highlights the fundamental importance of data adequacy for successful ML implementations. Data adequacy refers to data availability, quality, and nonstationarity that may significantly constrain ML applications in distribution systems. ML can provide novel and valuable perspectives on these data adequacy challenge, guiding data acquisition and ensuring the adequacy and quality of data for distribution system applications.

ML relies heavily on a substantial amount of data—often hundreds or thousands of samples—to train its algorithms. The fastest-growing area of ML, deep learning, requires even more extensive data. In practice, the data used for learning ML models in distribution systems are often obtained through computational simulations, as observational data may be scarce. However, generating these data through simulations can be costly. Therefore, it is of interest to design data generation and collection procedures that minimize the amount of data required and its associated costs. Research is needed to identify the most relevant data to produce or gather to improve model results. Using unsupervised and semi-supervised methods with small samples can also prove to be difficult. Gaussian processes (GPs)¹⁴² and their variants (e.g., sparse approximations¹⁴³) offer a potential solution in these cases. Another approach to overcome the scarcity of data is through data augmentation using GANs, which can synthesize additional training samples from the existing data to supplement the original sparse datasets. Nevertheless, maintaining diversity in the synthesized data and accurately representing out-of-sample or rare events are major ongoing challenges when using generative techniques.

Data quality is vital to the efficiency of the learning process and encompasses various aspects, such as completeness, consistency in representation, informative nature, and trustworthiness, of the data. Real-world distribution system measurements and operational data are often plagued by issues such as missing values, noise, and nonuniform sampling rates, requiring extensive pre-processing for effective utilization. Additionally, the available data may not accurately reflect the overall distribution of measurements, leading to unrepresentative results. Therefore, ML methods must be able to accommodate the presence of noise, communication delays, and missing data in real-world scenarios. Furthermore, while ML models are often trained and tested on data obtained from high-fidelity simulators and accurate distribution system models, these simulators may contain modeling errors and simplifications themselves and may not always have access to the exact parameters and topology of the distribution network. Therefore, it is essential to incorporate domain knowledge to guide data acquisition and ensure its quality and adequacy.

Data drift in the field of ML refers to changes in the distribution of data over time relative to the data distribution upon which the model was originally trained. Within the context of PES, data drift can result from changes in the topology of distribution networks, such as reconfigurations of loads and the addition of DERs, as well as transitions between grid-connected and islanded modes in microgrids. The current literature on this issue generally suggests retraining the ML model in response to each topological change, but this may not be a feasible solution for rapidly changing operational settings with a large number of network topology combinations. Further research is thus necessary to fully understand and address the challenges posed by data drift in power distribution systems.

In the PES community, there are growing concerns about the accessibility of unencrypted consumer electricity usage data collected by smart meters and other data sources.¹⁴⁴ The aforesaid concerns represent a significant obstacle to the safeguarding of privacy and, thus, call for the deployment of privacy-centric solutions that prioritize and preserve privacy. The adoption of cloud services is a viable option for addressing these challenges. For instance, software running on a distribution utility's server—collecting data from various end-users—may only transmit aggregated or anonymous statistics to the cloud, thereby allowing for data analysis without exposing individual consumer data. While this approach may benefit certain problems, its usefulness in demand-response applications where granular data are required to generate personalized incentives may be limited. In addition, a major hurdle lies in getting utilities to embrace cloud services as well as overcoming the inherent cybersecurity risks associated with this approach.

In the PES field, applying new ML methodologies, particularly DNNs and GNNs, requires caution. DNNs and GNNs are rapidly evolving and growing research areas within ML, therefore new models are frequently developed and made available. In turn, PES research is often focused on applying these state-of-the-art neural networks, potentially at the expense of more pressing needs. For example, the reliance on large amounts of data—crucial requirement for deep learning—is frequently overlooked in distribution system research. This shift toward more data-intensive ML methods has also diverted attention away from traditional, less data-intensive ML methods that may be better suited for the problems at hand. To address these concerns, it is imperative to carefully consider the suitability of state-of-the-art ML methods for distribution systems and make informed decisions on their use. This requires a collaborative effort between researchers and practitioners from both disciplines, focusing on understanding the context and needs of the distribution system domain.

In response to the increasing complexity of the distribution system, researchers have proposed a variety of ML-based alternatives to existing tools. As discussed in examples throughout this paper, these ML algorithms can effectively deal with complexity, reduce the computational burden of analysis, or leverage existing datasets, to name a few of the many potential benefits. At the same time, there exist decades of progress in PES developing traditional physics-based models and techniques that are governed by domain knowledge and theoretical foundations. In many cases, the best solution is not simply choosing one technique over the other, but rather combining the two in novel ways to achieve the benefits of each, while also limiting the drawbacks of each (Fig. 11). This has numerous benefits, including reduced data requirements or enhanced performance; for example, models can be trained with fewer data points, or the models can converge faster to optimal solutions.¹⁴⁵ 

Incorporating physics into ML models can be achieved through the enforcement of soft and/or hard constraints during the training process, prediction time, or both. Hard constraints involve transforming an optimization problem into a constrained one, while soft constraints involve modifying the objective function (i.e., the loss function) by incorporating additional physics-based terms.¹⁴⁵ This results in the creation of physics-constrained neural networks and physics-informed neural networks, respectively, which are characterized by their increased capability for generalization and enhanced interpretability.¹⁴⁵ The interested reader is referred to Ref. 34 for a list of software libraries specifically designed for physics-informed ML. Further research is necessary to explore hybrid and physics-informed approaches to achieve better performance and explainability in various distribution system applications.

The core principles of ML assume minimal human intervention, which often contradicts traditional practices in distribution system management. Despite the growing trend toward digitization and automation in power distribution systems, the adoption of ML-based solutions in these systems requires expert input, particularly in the areas of operations, control, and planning. Thus, it is unrealistic to expect that ML alone can revolutionize the operations of power distribution systems. Instead, a more viable approach is to leverage ML-based solutions in conjunction with expert input to improve system operations by offloading certain tasks and allowing operators to focus on optimizing operations. Such co-occurrence of ML and human expertise would reduce the need for human intervention and specialized expertise.

The second research theme is centered around three main strategies: (i) explainable; (ii) understandable; and (iii) repeatable ML results. To gain a full understanding of these concepts in a general sense, it is suggested that the reader refers to Ref. 146.

Traditionally, power engineers' confidence in the suitability of a model for a particular application is closely tied to their understanding of how the model works, its adherence to fundamental physical principles, and the degree to which its outputs facilitate effective decisions. However, this can be challenging with ML models, which are often hard to understand due to their lack of interpretability. To overcome this issue, research into methods that incorporate domain knowledge is important to make these models understandable and trustworthy. This ties into the research theme of physics-informed ML, which has been previously discussed.

Numerous ML frameworks used in distribution systems predominantly employ data-driven methodologies that produce black-box solutions with restricted information, making them difficult to explain. Although these techniques have exhibited good performance, their lack of interpretability renders them unfit for deployment in safety-critical systems. It is therefore essential to incorporate interpretability to establish trust in ML models and to effectively communicate their outcomes to system operators and decision-makers. However, as ML models become more complex—a trend also evident in PES research—they become harder to explain, let alone understand. This leads to a trade-off between complexity and interpretability. One approach is to quantify this trade-off and focus on developing interpretable solutions tailored to specific distribution system applications. This way, operators can understand and trust the results of the models. Despite the extensive body of literature on interpretable ML,^147,148 much progress still needs to be made in creating interpretable ML models tailored specifically to the PES domain.

Visualization is a frequently overlooked, yet crucial aspect of decision-making for operators in control centers. Apart from facilitating their comprehension of system processes, it also enables them to swiftly identify critical states in the system and take prompt actions accordingly. When it comes to ML systems, it is imperative to take into account the unique needs and preferences of decision-makers and system operators concerning both the visualization of the learning processes and the resulting outcomes. To advance research in this area, the focus should be on developing advanced but intuitive explanatory approaches that incorporate domain knowledge. Such practices would aid operators in understanding and interacting with complex ML models. Therefore, the promising avenue for future research is the development of post-hoc visualization and explanatory techniques, which provide a way to visualize and explain the results of ML models after they have been trained rather than during the training process. Although showcasing the accuracy of a learning model during training is useful, it is even more useful to identify cases where the predictions were incorrect and focus on them. Another prospective area for future development is how to properly visualize the evaluation metrics employed for evaluating the performance of ML models, the topic discussed next.

Researchers tend to prioritize finding the ML model that optimizes a chosen performance metric. However, solely relying on evaluation metrics to evaluate model performance can be misleading. For instance, when dealing with an unbalanced dataset wherein classes are not equally represented, using a performance metric like accuracy can provide a skewed perspective of the model's performance. For example, consider a classification task that aims to detect three-phase faults—the rarest variety—in power distribution systems. If the dataset used for this task contains 100 000 fault currents, with only 100 of them being the faults of interest and the rest being single- and two-phase faults, a trivial algorithm that classifies all faults as single- and two-phase faults (i.e., “non-three-phase faults”) would still have an accuracy of 99.9%. This could lead to the incorrect conclusion that the classifier is highly accurate, despite missing the faults of interest. Therefore, it is crucial to rigorously quantify the performance of learning systems, including the estimation of prediction quality and effective confidence bounds, to increase the reliability and credibility of ML in distribution systems. Additionally, efforts must be made to make performance measures understandable to operators by aligning with their expectations and translating them into more user-friendly metrics.

For certain high-regret applications in power systems (e.g., determining system stability), the costs associated with failure are so large that even a very reliable model will not be trusted by operators without performance guarantees. Therefore, an important area of future research will be to augment ML approaches with performance guarantees or adopt hybrid approaches which combine ML with traditional analysis that can provide guarantees. For example, in Refs. 120 and 121, techniques from robust control theory and Lyapunov stability theory, respectively, are combined with reinforcement-learning controllers to achieve stability guarantees. In the supervised learning domain, some efforts have been made to provide worst-case performance guarantees when learning OPF solutions.¹⁴⁹ Despite these promising examples, there remain many proposed applications of ML in power systems that would benefit from work in this area. By incorporating guarantees into ML techniques researchers can help remove a major barrier to adoption. In some cases, when strict guarantees are not possible, probabilistic ML methods can help to accurately convey the uncertainty inherent in the model (see Sec. VI C).

Finally, it is worth noting that although ML robustness could be a subsection of its own, we cover it briefly here, as the notion of ML model robustness is closely related to this topic. Namely, many data-driven models assume that the inputs to the model are unaltered; however, those designed for power system applications may be susceptible to various perturbations, both adversarial and nonadversarial.¹⁵⁰ Therefore, another important area of research will be analyzing and quantifying the robustness of the ML-based solutions against altered data instances. A relevant work on this particularly important topic is Ref. 150.

Many of the most successful applications of ML, such as image classification, speech recognition, and natural language processing, have consistently been improved on by outperforming prior works on well-established benchmarks (available at the UCI repository of ML databases¹⁵¹). In contrast, the lack of a unified and well-established set of baselines or benchmarks in the existing PES literature presents a challenge for accurately comparing and evaluating the performance of ML models and reproducing reported results. This results in a fragmented approach to model evaluation, with individual works often presenting specific quality tradeoffs without a direct comparison to prior works. To improve the characterization of methodological performance and enable a fair comparison, it is essential to establish a systematic and standardized framework for defining datasets and benchmarks for each ML application area. This will provide a more robust foundation for evaluating the performance of models and facilitate meaningful comparisons. Moreover, there is a need for research to develop quantitative measures that are both meaningful and interpretable, or to replace existing measures with qualitative characteristics that are better suited for the intended audience.

The last research theme focuses on probabilistic ML and includes two primary strategies: (i) probabilistic ML modeling and (ii) translating predictions with uncertainty estimates into actionable decisions. In contrast to traditional ML methods that typically provide deterministic predictions, probabilistic ML models generate predictions in the form of prediction intervals or probability distributions, encompassing not only a single-point estimate but also a measure of uncertainty. To gain a better understanding of probabilistic ML, refer to Ref. 152 for a general discussion.

Regarding (i), the use of ML in distribution system applications necessitates incorporating probabilistic modeling to account for three intrinsic types of uncertainty. These uncertainties include those arising from the physics involved (see Sec. VI C 1 for examples), the data used to train ML models, and the model itself. The second type of uncertainty typically involves both aleatoric uncertainty resulting from noise in data and epistemic uncertainty resulting from limited data and knowledge, as discussed in Ref. 34.

Modern power distribution systems must be better equipped to handle uncertainty from various sources (such as measurement errors, load fluctuations, weather-dependent renewable generation, and network parameters uncertainty) to guarantee secure operations. One of the key ways to achieve this is through reliable forecasting. Historically, energy forecasting, including demand, wind, and solar, relied on deterministic or single-valued forecasting methods.¹⁵³ However, probabilistic energy forecasting, which provides a range of probable values in addition to single-point estimates, is now considered more suitable for power distribution systems and is expected to become a fundamental aspect of their effective planning and operations. Despite the growing popularity of probabilistic ML techniques, particularly in the field of probabilistic forecasting (see, e.g., Ref. 154), the majority of forecasting models in practice still rely on deterministic modeling approaches. The transition from deterministic to probabilistic forecasting in power distribution systems has yet to occur, making it crucial to develop best-practice guidelines for their design and standardize evaluation metrics that are suitable for probabilistic ML models at this early stage. In addition to the emphasis on probabilistic forecasting, there is also a significant focus on developing probabilistic estimation methods, such as probabilistic SE using Bayesian networks,^155,156 which offer many advantages over traditional deterministic SE methods.

In the face of growing uncertainty in power distribution systems, relying solely on deterministic forecasting and point estimation is not sufficient for reliable decision-making and operational risk assessment. Additionally, the predictions made by ML models are often subject to a high degree of uncertainty. To enhance the credibility and reliability of ML-based solutions, it is crucial to incorporate uncertainty quantification and provide quantifiable confidence measures for decision-makers to utilize. This will allow them to make more informed decisions based on a better understanding of the uncertainty involved. This important issue has received much attention in recent years, supplementing traditional analytical methods with stochastic formulations to deal with uncertainty. The most common techniques for quantifying uncertainty in PES research are Monte Carlo simulations,¹⁵⁷ Bayesian inference, quantile regression, and polynomial chaos expansion,¹⁵⁸ with methods such as Bayesian deep learning,¹⁵⁹ Monte Carlo dropout,¹⁶⁰ (deep) GPs,^142,161 gaining in popularity. Future research in this area should focus on developing statistical learning methods that allow for the requisite uncertainty quantification in ML predictions that are specifically tailored to distribution system applications. This also entails characterizing and quantifying all sources of uncertainty during model development and determining the factors that contribute most to uncertainty. Providing insight into sources of uncertainty, let alone quantifying their contributions, can significantly increase the chances of making ML systems more accessible to systems operators and decision-makers.

In light of the prior discussion, it is evident that deterministic ML is not best suited for decision-making under uncertainty. Instead, probabilistic ML is more appropriate and can enhance the trustworthiness and robustness of ML model predictions in power distribution systems. The challenge, however, is translating the outputs of ML models (i.e., predictive uncertainty estimates) into effective decisions.¹⁶² Currently, the use of ML methods for decision-making under uncertainty is extensively researched within the PES domain. A prime example of this is cost-oriented ML, which endeavors to improve the practical value of probabilistic forecasting in power system decision-making procedures.¹⁶³ Similarly, future research should prioritize evaluating the value of probabilistic ML models in the context of decision-making processes within distribution systems rather than just examining the accuracy and precision of their outputs. Another important area for future research is exploring ways to seamlessly integrate probabilistic forecasts and uncertainty estimates into existing decision-making practices. One potential approach is to utilize the upper and lower bounds of uncertainty estimates as inputs into planning frameworks, considering them as the most favorable and unfavorable scenarios.¹⁶⁴ Nevertheless, there is still ample opportunity for improvement in incorporating the full uncertainty estimates into decision-making processes rather than just the distribution tails. Effective visualization that is specifically tailored for distribution system control centers is important in this context, and it relates to the topic of visualization discussed earlier.

Emerging from the previous discussion are several key questions:

• How can ML models' interpretability, robustness, and performance guarantees be enhanced so that they can be utilized confidently in complex distribution systems?

• How to effectively present the results of ML models to improve their understanding by system operators and decision-makers?

• What steps need to be taken to make the performance evaluation of complex ML models more accessible to system operators?

• What measures should be taken to validate ML findings and translate them into actionable insight for decision-makers?

• What methods should be employed to combine actual distribution system operational data with simulated data to enhance sparse datasets while ensuring data quality standards are met?

• How can the uncertainty in ML model predictions be converted into quantifiable confidence measures for system operators and decision-makers?

The answers to these questions will vary depending on the application and require careful consideration to guide future research. Nonetheless, concerning future published works, four key pieces of information are indispensable for developing robust and reliable ML models for distribution system applications:

• A comprehensive list of assumptions, parameters, and algorithmic choices used;

• A complete description of the raw input data and data processing steps utilized;

• A complete description of the model implementation to enable independent replication;

• Verification and validation of the model implementation to ensure accuracy and reliability.

In addition to the information above, careful consideration should be given to standardizing model performance evaluation across benchmark datasets and problems and developing appropriate benchmarks in novel application areas if they do not already exist, thereby encouraging model-consistent testing. Additionally, a concerted effort by researchers will be needed to devise meaningful metrics that test the accuracy of newly proposed ML-based solutions, especially if the assessment of model fidelity at rare events (i.e., tails of distributions) is also factored into such evaluations. Last, but not least, by including the code used to generate results in conjunction with papers, researchers can make the methods more accessible (and comparable) to the community to encourage further improvements.

In this paper, we summarize the current state of ML for power distribution systems in a form that is accessible to readers from different engineering backgrounds in order to encourage interdisciplinary research on this topic. While a vast body of literature has focused on reviewing novel ML techniques and their applications in PES, our attention is directed toward power distribution systems. These systems are at the forefront of the ongoing electric power sector's transformation and come with distinctive data-related challenges related to ML applications. In addition to providing background material on power systems—with a specific emphasis on distribution systems—and relevant ML theory, we also explore the current capabilities and limitations of ML for power distribution systems problems. We highlight various ML applications, ranging from distribution system operations and control to planning. We finally discuss potential research directions that can facilitate the adoption of ML beyond the research community and academic circles, and promote its implementation in real-world applications. One of the key takeaways from our discourse is that the current use of new data-intensive methods—widely explored in PES research—is not practical for power distribution systems at this stage due to the limited availability of data in distribution systems planning, and in particular, operations. Consequently, until data availability increases and the data-related issues discussed in this paper are addressed, it is advisable to redirect research efforts toward ML approaches that require less data or enable trustworthy augmentation of existing limited data. In this regard, physics-based ML appears to offer viable and promising research directions to pursue.

This work was authored in part by the National Renewable Energy Laboratory (NREL), operated by the Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36–08GO28308. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes. This work was also partially funded by the Climate Change AI Innovation Grants program, hosted by Climate Change AI with the support of the Quadrature Climate Foundation, Schmidt Futures, and the Canada Hub of Future Earth.

The authors have no conflicts to disclose.

Marija Markovic: Conceptualization (lead); Visualization (lead); Writing – original draft (lead); Writing – review & editing (equal). Matthew Bossart: Conceptualization (supporting); Visualization (supporting); Writing – original draft (supporting); Writing – review & editing (equal). Bri-Mathias Hodge: Supervision (lead); Writing – original draft (supporting); Writing – review & editing (equal).

Data sharing is not applicable to this article as no new data were created or analyzed in this study.