Robust Deep Reinforcement Learning for Inverter-based Volt-Var Control in Partially Observable Distribution Networks
0
Sign in to get full access
Overview
- Investigates the use of robust deep reinforcement learning for volt-var control in partially observable distribution networks
- Focuses on controlling inverter-based distributed energy resources (DERs) to maintain voltage within limits
- Presents a deep reinforcement learning (DRL) approach to learn an optimal volt-var control policy
Plain English Explanation
The paper explores using a robust deep reinforcement learning approach to control inverter-based distributed energy resources (DERs) in power distribution networks. The goal is to maintain the voltage within acceptable limits, even when the network conditions are only partially observable.
In a power distribution network, voltage levels can fluctuate due to factors like changing loads and renewable energy generation. Inverter-based DERs, like solar panels or battery storage, can help regulate the voltage by adjusting their reactive power output. However, determining the optimal control actions is challenging, especially when the full state of the network is not known.
The researchers propose using a deep reinforcement learning algorithm to learn an effective volt-var control policy. The algorithm takes limited observations about the network state as input and learns to output the optimal reactive power settings for the DERs to maintain voltage within limits. The approach is designed to be robust to uncertainties and disturbances in the network.
Technical Explanation
The paper presents a deep reinforcement learning (DRL) framework for inverter-based volt-var control in partially observable distribution networks. The key elements include:
- Network Model: The distribution network is modeled as a partially observable Markov decision process (POMDP), where the full state of the network is not directly observable.
- DRL Agent: A DRL agent is trained to learn an optimal volt-var control policy. The agent takes limited observations about the network state as input and outputs the reactive power settings for the DERs.
- Reward Function: The reward function encourages the agent to maintain the voltage within acceptable limits while minimizing the reactive power usage.
- Robust Training: The training process employs techniques like domain randomization to improve the agent's robustness to uncertainties in the network.
The proposed DRL approach is evaluated on a realistic distribution network simulation and demonstrates improved voltage regulation performance compared to traditional control methods.
Critical Analysis
The paper presents a promising approach to addressing the challenges of volt-var control in partially observable distribution networks. However, there are a few potential limitations and areas for further research:
- The paper focuses on a single-agent DRL framework, but real-world distribution networks may involve multiple DERs owned by different stakeholders. Extending the approach to a multi-agent setting could be an interesting direction.
- The simulation-based evaluation does not account for all real-world complexities, such as communication delays, sensor inaccuracies, or unexpected network disturbances. Further testing on physical testbeds or field deployments would be valuable to assess the method's practical feasibility.
- The paper does not discuss the computational and memory requirements of the DRL agent, which could be an important consideration for real-time deployment in distribution networks with limited computational resources.
Overall, the research presents a promising step towards addressing the volt-var control challenge in partially observable distribution networks using robust deep reinforcement learning techniques.
Conclusion
The paper investigates the use of robust deep reinforcement learning for inverter-based volt-var control in partially observable distribution networks. The proposed DRL-based approach learns an optimal control policy that can maintain voltage within acceptable limits, even when the full network state is not directly observable. The simulation results demonstrate the method's effectiveness in improving voltage regulation compared to traditional control techniques.
While the research shows promising results, further work is needed to address potential limitations and expand the approach to real-world deployment scenarios. Exploring multi-agent settings, validating the method's performance in more realistic testbeds, and assessing the computational requirements are all important areas for future research.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Robust Deep Reinforcement Learning for Inverter-based Volt-Var Control in Partially Observable Distribution Networks
Qiong Liu, Ye Guo, Tong Xu
Inverter-based volt-var control is studied in this paper. One key issue in DRL-based approaches is the limited measurement deployment in active distribution networks, which leads to problems of a partially observable state and unknown reward. To address those problems, this paper proposes a robust DRL approach with a conservative critic and a surrogate reward. The conservative critic utilizes the quantile regression technology to estimate conservative state-action value function based on the partially observable state, which helps to train a robust policy; the surrogate rewards of power loss and voltage violation are designed that can be calculated from the limited measurements. The proposed approach optimizes the power loss of the whole network and the voltage profile of buses with measurable voltages while indirectly improving the voltage profile of other buses. Extensive simulations verify the effectiveness of the robust DRL approach in different limited measurement conditions, even when only the active power injection of the root bus and less than 10% of bus voltages are measurable.
Read more8/14/2024
0
New!Fair Reinforcement Learning Algorithm for PV Active Control in LV Distribution Networks
Maurizio Vassallo, Amina Benzerga, Alireza Bahmanyar, Damien Ernst
The increasing adoption of distributed energy resources, particularly photovoltaic (PV) panels, has presented new and complex challenges for power network control. With the significant energy production from PV panels, voltage issues in the network have become a problem. Currently, PV smart inverters (SIs) are used to mitigate the voltage problems by controlling their active power generation and reactive power injection or absorption. However, reducing the active power output of PV panels can be perceived as unfair to some customers, discouraging future installations. To solve this issue, in this paper, a reinforcement learning technique is proposed to address voltage issues in a distribution network, while considering fairness in active power curtailment among customers. The feasibility of the proposed approach is explored through experiments, demonstrating its ability to effectively control voltage in a fair and efficient manner.
Read more9/17/2024
🏅
0
End-to-End Reinforcement Learning of Curative Curtailment with Partial Measurement Availability
Hinrikus Wolf, Luis Bottcher, Sarra Bouchkati, Philipp Lutat, Jens Breitung, Bastian Jung, Tina Mollemann, Viktor Todosijevi'c, Jan Schiefelbein-Lach, Oliver Pohl, Andreas Ulbig, Martin Grohe
In the course of the energy transition, the expansion of generation and consumption will change, and many of these technologies, such as PV systems, electric cars and heat pumps, will influence the power flow, especially in the distribution grids. Scalable methods that can make decisions for each grid connection are needed to enable congestion-free grid operation in the distribution grids. This paper presents a novel end-to-end approach to resolving congestion in distribution grids with deep reinforcement learning. Our architecture learns to curtail power and set appropriate reactive power to determine a non-congested and, thus, feasible grid state. State-of-the-art methods such as the optimal power flow (OPF) demand high computational costs and detailed measurements of every bus in a grid. In contrast, the presented method enables decisions under sparse information with just some buses observable in the grid. Distribution grids are generally not yet fully digitized and observable, so this method can be used for decision-making on the majority of low-voltage grids. On a real low-voltage grid the approach resolves 100% of violations in the voltage band and 98.8% of asset overloads. The results show that decisions can also be made on real grids that guarantee sufficient quality for congestion-free grid operation.
Read more6/21/2024
🏅
0
Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control
Yang Qu, Jinming Ma, Feng Wu
Active voltage control presents a promising avenue for relieving power congestion and enhancing voltage quality, taking advantage of the distributed controllable generators in the power network, such as roof-top photovoltaics. While Multi-Agent Reinforcement Learning (MARL) has emerged as a compelling approach to address this challenge, existing MARL approaches tend to overlook the constrained optimization nature of this problem, failing in guaranteeing safety constraints. In this paper, we formalize the active voltage control problem as a constrained Markov game and propose a safety-constrained MARL algorithm. We expand the primal-dual optimization RL method to multi-agent settings, and augment it with a novel approach of double safety estimation to learn the policy and to update the Lagrange-multiplier. In addition, we proposed different cost functions and investigated their influences on the behavior of our constrained MARL method. We evaluate our approach in the power distribution network simulation environment with real-world scale scenarios. Experimental results demonstrate the effectiveness of the proposed method compared with the state-of-the-art MARL methods. This paper is published at url{https://www.ijcai.org/Proceedings/2024/}.
Read more9/4/2024