A Review of Safe Reinforcement Learning Methods for Modern Power Systems

MDPs are defined by a tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},R,\rho_{0},\gamma)$ρろー start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_γがんま ) which are, respectively, the state space, action space, probability distribution, reward function, initial state $\rho_{0}\in{\cal S}$ρろー start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∈ caligraphic_S and discount factor. When the decision problem fits in an MDP, the objective is to determine the policy $\pi$πぱい that maximizes the expected discounted reward $\mathcal{J}_{R}^{\pi_{\theta}}$πぱい start_POSTSUBSCRIPT italic_θしーた end_POSTSUBSCRIPT end_POSTSUPERSCRIPT, i.e.[4, 7, 8]:

where $\tau\sim\pi$τたう ∼ italic_πぱい indicates that the distribution over trajectories depends on the policy $\pi$πぱい; similarly $\bm{s}_{0}\sim\rho_{0}$ρろー start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, $\bm{a}_{t}\sim\pi(\cdot|\bm{s}_{t})$πぱい ( ⋅ | bold_italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ), $\bm{s}_{t+1}\sim\mathcal{P}(\cdot|\bm{s}_{t},\bm{a}_{t})$. Even if the transition probabilities and reward function are fully known, this task is often intractable. However, the approach taken normally is to learn the policy, using some parametrization.

The CMDP $\mathcal{M}_{C}=(\mathcal{S},\mathcal{A}_{t},\mathcal{P},R,\rho_{0},\gamma,\mathcal{C})$ρろー start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_γがんま , caligraphic_C ) is an extension of a standard MDP, that addresses a frequent model variation: the case in which the action space ${\cal A}_{t}$ is a function of the state space ${\cal S}$, i.e. $\bm{s}_{t}\mapsto{\cal A}_{t}$, because the change in the environment affects what is a safe or feasible action, or due to the state-dependent cost of the action, which in the formulation needs to be below a threshold. This occurs in physical systems in which the boundary conditions, the state and the laws of physics limit what is feasible, what would lead to operations that are unsafe and how expensive is a certain agent action. In a nutshell, what differentiates the various instances of CMDP from a conventional MDP is the class of constraints that characterize the action space as a function of the system dynamics and the specific engineering problem and context that define the constraints. In this review, we define the CMDP for power system problems:

where $\bm{a}_{t}$ is feasible not only means that $\bm{a}_{t}$ is constrained within its upper and lower limits, but also that the resulting $\bm{s}_{t}$ falls within specified feasible sets. In power systems, constraints on the upper and lower bounds of $\bm{a}_{t}$ relate to the control ranges of various controllable devices, such as the power output of SGs, RESs, and ESSs, as well as the temperature setpoint of HVAC systems, which can typically be enforced by simply restricting the action space of RL. $\bm{s}_{t}$ falls within specified feasible sets means that the state adheres to safe and stable operation constraints, such as boundary constraints of voltages, line flows, and building temperatures, as well as stability constraints of voltages, frequency, and rotor angles. Due to the highly non-linear and non-convex nature of power systems, obtaining feasible $\bm{a}_{t}$ that guarantees feasible $\bm{s}_{t}$ is challenging. This is also the main challenge of training safe RL.

In power system applications, the classification of constraints into instantaneous and cumulative constraints is related to the required degree of constraint satisfaction and the safe RL algorithms used. Typically, bus balance equations, upper and lower power limits of various equipment, ESS capacity constraints, certain voltage amplitude constraints, and some stability constraints are considered hard constraints. Safe RL algorithms capable of ensuring the satisfaction of hard constraints include projection method III-B, Lyapunov method III-C, shielding method III-E, safety layer method III-F and barrier function method III-G. For example, [15] uses the logarithmic barrier function to make the $\mathcal{J}_{h_{i}}^{\pi_{\theta}}$πぱい start_POSTSUBSCRIPT italic_θしーた end_POSTSUBSCRIPT end_POSTSUPERSCRIPT approach infinity when voltage exceeds bounds, thereby satisfying hard voltage constraints. Due to discrepancies between models and real systems, various uncertainties of RESs and loads, and algorithmic shortcomings, even if constraints are theoretically satisfied, they may not be guaranteed in actual deployment. Therefore, GP methods III-D and RRL III-G have been proposed, using the probabilistic/chance constraint (6). However, their application in power systems remains underexplored. A more common approach is to use constrained game-theoretic RL within RRL [14, 16]. Furthermore, by design some safe RL algorithms can only encourage constraint satisfaction while maximizing rewards. Such algorithms include Lagrangian relaxation III-A and penalty functions. For example, [17] uses the voltage constraint metric $\mathcal{J}_{h_{i}}^{\pi_{\theta}}=\sum_{i\in\cal N}\max\left\{|\bm{v}_{i,t}-1|-0.05|,0\right\}$πぱい start_POSTSUBSCRIPT italic_θしーた end_POSTSUBSCRIPT end_POSTSUPERSCRIPT = ∑ start_POSTSUBSCRIPT italic_i ∈ caligraphic_N end_POSTSUBSCRIPT roman_max { | bold_italic_v start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT - 1 | - 0.05 | , 0 } and employs Lagrangian relaxation for voltage control, which cannot guarantee absolute adherence to voltage constraints, thus classifying it as a soft constraint. For some constraints, instead, such as user satisfaction with EV charging and voltage control at certain nodes, the goal is to approach standard values as closely as possible, making them inherently soft constraints. The illustrations of different constraints of safe RL are shown in Fig. 2.

Lagrangian relaxation, also known as primal-dual method, is the most common technique in safe RL. The key idea of this method is to transform the CMDP problem into an unconstrained dual problem. This is achieved by employing adaptive Lagrange multipliers to penalize constraints [19]:

The solution of (7) relies on Danskin’s theorem and convex analysis [20]. Due to its straightforward implementation and compatibility with both on-policy and off-policy methods, Lagrangian relaxation has been integrated with other RL algorithms, fostering the creation of numerous variants, such as DDPG-Lag, PPO-Lag, TRPO-Lag, TD3-Lag, SAC-Lag, MAPPO, RCPO, PDO, TRPO-PID, CPPO-PID, DDPG-PID, TD3-PID, SAC-PID [21, 22, 19, 23].

The Lagrangian relaxation method is the most commonly used approach in power systems, capable of being easily integrated with various algorithms for application across a wide range of domains. Based on instantaneous or hard constraints, [24] utilizes a primal-dual approach to optimize the control of power generation and BESS charging and discharging actions in a multi-stage real-time stochastic dynamic OPF. Additionally, [25] applies constrained SAC to the Volt-VAR control problem by synergistically combining the merits of the maximum-entropy framework, the method of multipliers, a device-decoupled neural network structure, and an ordinal encoding scheme. Furthermore, [26] employs constrained RL for the predictive control of OPF, paired with EV charging control. On the other hand, based on cumulative or soft constraints, [27] approximates the actor gradients by solving the Karush-Kuhn-Tucker conditions of the Lagrangian, instead of constructing reward critic networks and cost critic networks through interactions with the environment. Then, the interior point method is incorporated to derive the parameter updating rule for the DRL agent. Similarly, [28] develops a soft-constraint enforcement method to adaptively encourage the control policy in the safety direction with nonconservative control actions and find decisions with near-zero degrees of constraint violations.

The TRM ensures constraint satisfaction at every step and enhances performance by updating the trust region policy gradient and projecting the policy into a safe feasible set during each iteration [29]. Typical projection methods include CPO [9], PCPO [30], FOCOPS [31], CUP [32], and MACPO[22], among which PCPO is implemented through a two-step process: first, conducting a local reward update, and then projecting the policy back onto the constraint set to address any constraint violations, as depicted in Fig. 3.

In the power system domain, TRMs have also seen widespread application. For instance, [33] introduced a projection-embedded MA-DRL algorithm that smoothly and effectively restricts the DRL agent action space to prevent any violations of physical constraints, thereby achieving decentralized optimal control of distribution grids with a guaranteed 100% safety rate. Additionally, in the area of EV charging problems, [34] utilizes a penalty function to penalize the neural network output if it exceeds the action space and uses a projection operator to avoid incurring a negative reward when no EV is occupying the charging bay. In addition, [35] employs CPO for volt-VAR control to minimize the total operation costs while satisfying the physical operation constraints. However, TRMs, primarily based on TRPO or PPO, are not easily integrated with other RL types and are computationally intensive in high dimensions, limiting their suitability for large-scale safe RL problems [36].

Lyapunov functions, widely used in control engineering for controller design [37], were first applied to safe RL in [38]. The application of the Lyapunov method in power systems is limited because it requires prior knowledge of a Lyapunov function. If the model of environmental dynamics is unknown, identifying a suitable Lyapunov function can be challenging. For example, [39] integrates a Lyapunov function into the structural properties of primary frequency controllers, guaranteeing local asymptotic stability over a large set of states. Additionally, [40] utilizes Lyapunov theory to design the controller that satisfies specific Lipschitz constraints for decentralized inverter-based voltage control. In addition, [41] utilizes a stability-constrained RL method for real-time voltage control in distribution grids, providing a formal voltage stability guarantee using the Lyapunov function.

GP [42] is widely utilized in numerous approaches to estimate uncertainty and identify unsafe areas. Consequently, assessments based on GP can be incorporated into the learning process to enhance agent safety [43]. GP-based safe RL algorithms include SafeOpt [44] and PILCO [45]. The application of GP method-based safe RL in power systems is limited, meriting further research to adequately address the various uncertainties inherent in power systems. The potential disadvantage of GP methods is their computational complexity and scalability issues, especially as the dimensionality of the problem space increases [36].

In [46], the shield is introduced for the first time in RL. This shield is computed in advance, based on the safety component of the system specification provided and an abstraction of the dynamics of the agent’s environment. It guarantees safety with minimal interference, implying that the shield limits the agent’s actions as little as necessary, only prohibiting actions that could jeopardize the safe behavior of the system. The shielded RL is shown in Fig. 4.

Shielding is a method that enforces constraint satisfaction, making it highly suitable for power system problems with hard constraints. For instance, in [47], actions that would lead to dangerous states, such as the SoC of BESSs being fully charged or depleted, are substituted by the shielding mechanism with safe actions to maintain system stability. Additionally, [48] combines a correction model adapted from gradient descent with the prediction model as a post-posed shielding mechanism to enforce safe actions in computer room air conditioning unit control problems. In addition, in unit commitment scheduling, [49] utilizes action space clipping to ensure that uncertainty estimates are reasonable and within appropriate bounds obtained from historical data. A potential drawback of the shielding method is the challenge of identifying feasible, safe actions based on infeasible ones, which requires underlying knowledge of the system. This can be difficult for certain complex systems or specific control scenarios [36].

Both the safety layer and shielding method integrate safety into the RL process, but they differ in their implementation: the safety layer acts as an additional check within the RL framework, whereas shielding employs an external system (the shield) that intervenes only when necessary to prevent unsafe actions. The safety layer method, first proposed in [50] for continuous action spaces in RL, emphasizes maintaining zero-constraint violations throughout the learning process. It expresses safety constraints as linear functions of action through a first-order approximation. Assuming that at most one constraint is violated at any time, an analytical solution to the safety layer optimization problem can be directly obtained. The linearization equation and visualization of the safety layer are shown in (8) and 5, respectively.

where $w_{i}$ are weights of NN; $g(s_{t};w_{i})$ denotes first-order approximation to $h_{i}(s_{t},a_{t})$ with respect to $a_{t}$.

The safety layer method has been widely applied in power systems. For example, in optimal power generation dispatch, [51] proposes a hybrid knowledge-data-driven safety layer to convert unsafe actions into the safety region, which is accelerated by a security-constrained linear projection model. Additionally, in volt-VAR control, [52] adds a safety layer to the policy neural network to enhance operational constraint satisfaction during both the initial exploration phase and the convergence phase. In addition, [53] uses action clipping, reward shaping, and expert demonstrations to ensure safe exploration and accelerate the training process during the online training stage for the assist service restoration problem. However, the linear approximation in the safety layer might not accurately capture the complexities of underlying dynamics in highly non-linear systems, and iterating at every time step could introduce a significant computational burden. Moreover, assuming only one constraint at a time may not be valid in complex environments where multiple safety constraints are concurrently active.

The barrier function method involves adding a barrier function penalty term to the original objective function. When the system state approaches the safety boundary, the value of the constructed barrier function tends to infinity, thereby ensuring that the state remains within the safe boundary [54]. The most typical barrier function method is IPO, which augments the objective with logarithmic barrier functions, drawing inspiration from the interior-point method [55]:

where $t_{i}$ is a hyperparameter for $h_{i}$. The illustration of IPO is shown in Fig. 6.

Barrier function method and IPO have been widely applied in power systems to ensure the safety of constraints. For example, [12] utilizes IPO to ensure the fulfillment of distribution network constraints without the need for designated penalty terms and the associated tuning of penalty factors, or repeatedly solving optimization problems for action rectification. Additionally, [56] uses IPO to facilitate desirable learning behavior towards constraint satisfaction and policy improvement simultaneously during online preventive control for transmission overload relief. In addition, [57] proposes a safe RL method for emergency load shedding in power systems, where the reward function includes a barrier function that approaches negative infinity as the system state approaches safety bounds. However, the accurate formulation and tuning of barrier functions necessitate knowledge of system dynamics, which can be challenging in complex environments.

One of the challenges in RL is generalization under uncertainties not seen during training. To address this, RRL frameworks have been developed, focusing on enhancing the reliability and robustness of RL agents for the worst-case scenarios [58, 59]. Two notable approaches in this context are chance-constrained RRL and constrained game-theoretic RL. It is important to note that RRL is not universally recognized as a safe RL algorithm in other fields. However, due to the significant uncertainties in power systems, RRL is employed to enhance control robustness and is reviewed here.

Benchmarks include both benchmark environments and benchmark algorithms. Safety Gym, developed by OpenAI, is the first widely recognized safe benchmark environment. It includes an environment-builder and a suite of pre-configured benchmark environments [21, 67]. Correspondingly, Safety Starter Agents, a benchmark algorithm library, has been developed based on Safety Gym [68]. The supported algorithms in this library include PPO, PPO-Lag, TRPO, TRPO-Lag, SAC, SAC-Lag, and CPO. This package has been tested on Mac OS Mojave and Ubuntu 16.04 LTS and is likely compatible with most recent Mac and Linux operating systems.

Safety Gymnasium, an update and extension of Safety Gym, has currently become the mainstream platform in use [69, 70]. Correspondingly, a benchmark repository for safe RL algorithms has been proposed, named SafePO [71]. SafePO is tested on the Linux platform and potentially supports Mac or Windows, requiring only modifications to the Linux path and sort functions for compatibility.

SafePO further extends the variety of supported safe RL algorithms, as illustrated in Fig. 7.

OmniSafe emerges as the first unified learning framework in the field of safe RL, featuring a highly modular framework that includes a comprehensive collection of algorithms specifically developed for safe RL across various domains. Its versatility comes from an abstracted algorithm structure and a well-designed API, facilitating seamless integration of different components, thereby simplifying extension and customization for developers. Additionally, OmniSafe enhances algorithm learning speeds through process parallelism, supporting both environment-level and agent asynchronous parallel learning. OmniSafe is supported and tested on Linux and also supports M1 and M2 versions of macOS. However, it does not support Windows [72, 73]. The supported safe RL algorithms of OmniSafe are shown in Table I.

Overall, Safety Gymnasium is the current mainstream benchmark environment, and OmniSafe has also integrated Safety Gymnasium to ensure overall code compatibility. It is important to remark that Safety Gymnasium was primarily developed for control in gaming, robotics, autonomous driving, etc., featuring a series of agents such as point, car, dog, and ant, among others. It offers several specific environments tailored for challenges such as safe navigation, safe velocity, and safe vision, but it is not directly applicable to power systems problems’ formulations. Hence, there is a need to develop corresponding power system control environments based on the environment templates provided by Safety Gymnasium. In terms of benchmark algorithms, OmniSafe offers a more comprehensive set of algorithms but currently does not support Windows due to difficulties with Python library installations. In contrast, SafePO is more easily expanded on Windows. Since most power system professional software is developed for Windows, with less support for Linux and macOS, this may limit the application of OmniSafe in model-based environments. However, if surrogate models are used to substitute for physical models in a model-free environment, OmniSafe can be utilized in Linux or macOS.

Optimal power generation dispatch considering various constraints, ranging from simplified versions to security constraints, including economic dispatch, DC-OPF, AC-OPF, and SCOPF. The operation of a power system must meet both security and economic requirements. Considering credible contingencies, AC-OPF has been widely used [79, 86]. Most existing methods for solving OPF rely on analytical methods; however, given the inherently large scale of these problems, real-time computation is very challenging. A new variation of OPF is the SCOPF. This type of problem requires significantly longer computation times due to the additional security constraints [27]. To accelerate the calculation of SCOPF, methods such as DC-PF approximation [87], convex power flow approximation [88], and convex security constraint approximation [89] have been proposed. However, the accuracy of these methods has been questioned, and they remain time-consuming for large-scale systems. To accelerate computation and achieve better solutions, RL methods have been widely applied. Since traditional RL struggles to handle safety constraints effectively, safe RL has been further applied to address these issues.

The details of the applications of safe RL in optimal power generation dispatch are shown in Table II. Based on Table II, we summarize the foundational framework for implementing safe RL in optimal power generation dispatch with a specific example with SGs, RESs, and BESSs, incorporating strict physics-based constraints such as AC- and DC-PF constraints. If the system encompasses additional power system devices, the presented equations are designed to be readily scalable to accommodate them. Note that the models presented below are examples for illustration, and there are other RL formulations and models for optimal power generation dispatch depending on the specific problem setting. This is also true for other application domains. The state, action, reward, and constraints of optimal power generation dispatch are shown as follows.