End-to-End Reinforcement Learning of Curative Curtailment with Partial Measurement Availability
2405.03262
0
0
🏅
Abstract
In the course of the energy transition, the expansion of generation and consumption will change, and many of these technologies, such as PV systems, electric cars and heat pumps, will influence the power flow, especially in the distribution grids. Scalable methods that can make decisions for each grid connection are needed to enable congestion-free grid operation in the distribution grids. This paper presents a novel end-to-end approach to resolving congestion in distribution grids with deep reinforcement learning. Our architecture learns to curtail power and set appropriate reactive power to determine a non-congested and, thus, feasible grid state. State-of-the-art methods such as the optimal power flow (OPF) demand high computational costs and detailed measurements of every bus in a grid. In contrast, the presented method enables decisions under sparse information with just some buses observable in the grid. Distribution grids are generally not yet fully digitized and observable, so this method can be used for decision-making on the majority of low-voltage grids. On a real low-voltage grid the approach resolves 100% of violations in the voltage band and 98.8% of asset overloads. The results show that decisions can also be made on real grids that guarantee sufficient quality for congestion-free grid operation.
Create account to get full access
Overview
- The paper presents a novel approach to resolving congestion in distribution grids using deep reinforcement learning.
- This method can make decisions to curtail power and set appropriate reactive power to maintain a non-congested, feasible grid state.
- It enables decisions under sparse information, only requiring observations from some buses in the grid, making it suitable for distribution grids that are not fully digitized.
- The approach resolved 100% of voltage violations and 98.8% of asset overloads on a real low-voltage grid, demonstrating its effectiveness for congestion-free grid operation.
Plain English Explanation
As the energy transition progresses, the expansion of energy generation and consumption will change, with technologies like solar panels, electric vehicles, and heat pumps influencing power flow, especially in distribution grids. Scalable methods are needed to enable congestion-free operation of these distribution grids.
This paper introduces a new approach that uses deep reinforcement learning to tackle this challenge. The method learns to make decisions, such as reducing power generation or adjusting reactive power, to maintain a grid state that avoids congestion and overloads. Importantly, it can do this without needing detailed measurements from every part of the grid, which is often the case in real-world distribution grids that are not fully digitized.
The researchers tested the approach on a real low-voltage grid and found it was able to resolve 100% of voltage issues and 98.8% of asset overloads. This demonstrates the method's potential to help distribution grid operators manage the changing energy landscape and ensure reliable, congestion-free grid operation.
Technical Explanation
The paper presents a deep reinforcement learning approach to resolving congestion in distribution grids. The key innovation is the ability to make decisions under sparse information, leveraging observations from only some buses in the grid, rather than requiring detailed measurements from every bus, as is the case for traditional optimal power flow methods.
The architecture learns to determine the appropriate actions, such as curtailing power generation or setting reactive power, to maintain a non-congested, feasible grid state. This is particularly important as distribution grids become more dynamic due to the integration of distributed energy resources like solar panels and electric vehicles.
The researchers tested the approach on a real low-voltage grid and found it was able to resolve 100% of voltage violations and 98.8% of asset overloads. This performance is notable, as many distribution grids are not yet fully digitized and observable, making traditional optimization-based methods challenging to apply. The presented approach demonstrates the potential for deep reinforcement learning to enable congestion-free grid operation in the face of these real-world constraints.
Critical Analysis
The paper presents a promising approach to a critical challenge in the energy transition, but it also acknowledges several limitations and areas for further research. One key limitation is the reliance on a single, real-world grid for testing, which may limit the generalizability of the results. Additional testing on a wider range of grid topologies and scenarios would help establish the robustness of the method.
The authors also note that the approach currently assumes perfect knowledge of grid parameters, such as line impedances, which may not always be the case in practice. Addressing this uncertainty, perhaps through the integration of robust control techniques, could further enhance the method's applicability.
While the overall performance is impressive, the paper does not provide a detailed comparison to other state-of-the-art approaches, such as decentralized coordination or model-predictive control methods. Such a comparison could help contextualize the strengths and weaknesses of the proposed approach.
Conclusion
This paper presents a novel deep reinforcement learning-based method for resolving congestion in distribution grids, a critical challenge as the energy transition progresses. The key innovation is the ability to make decisions under sparse information, making it suitable for real-world distribution grids that are not fully digitized and observable.
The approach demonstrated impressive results on a real low-voltage grid, resolving 100% of voltage violations and 98.8% of asset overloads. This suggests the method has the potential to enable congestion-free grid operation, supporting the integration of new energy technologies like solar panels and electric vehicles. While further research is needed to address limitations and provide more comprehensive comparisons, this work represents an important step forward in the quest to modernize and optimize our electricity distribution systems.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🤿
Closed-form congestion control via deep symbolic regression
Jean Martins, Igor Almeida, Ricardo Souza, Silvia Lins
0
0
As mobile networks embrace the 5G era, the interest in adopting Reinforcement Learning (RL) algorithms to handle challenges in ultra-low-latency and high throughput scenarios increases. Simultaneously, the advent of packetized fronthaul networks imposes demanding requirements that traditional congestion control mechanisms cannot accomplish, highlighting the potential of RL-based congestion control algorithms. Although learning RL policies optimized for satisfying the stringent fronthaul requirements is feasible, the adoption of neural network models in real deployments still poses some challenges regarding real-time inference and interpretability. This paper proposes a methodology to deal with such challenges while maintaining the performance and generalization capabilities provided by a baseline RL policy. The method consists of (1) training a congestion control policy specialized in fronthaul-like networks via reinforcement learning, (2) collecting state-action experiences from the baseline, and (3) performing deep symbolic regression on the collected dataset. The proposed process overcomes the challenges related to inference-time limitations through closed-form expressions that approximate the baseline performance (link utilization, delay, and fairness) and which can be directly implemented in any programming language. Finally, we analyze the inner workings of the closed-form expressions.
5/3/2024
🏅
Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies
Seyed Soroush Karimi Madahi, Gargya Gokhale, Marie-Sophie Verwee, Bert Claessens, Chris Develder
0
0
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.
5/1/2024
Decentralized Coordination of Distributed Energy Resources through Local Energy Markets and Deep Reinforcement Learning
Daniel May, Matthew Taylor, Petr Musilek
0
0
As the energy landscape evolves toward sustainability, the accelerating integration of distributed energy resources poses challenges to the operability and reliability of the electricity grid. One significant aspect of this issue is the notable increase in net load variability at the grid edge. Transactive energy, implemented through local energy markets, has recently garnered attention as a promising solution to address the grid challenges in the form of decentralized, indirect demand response on a community level. Given the nature of these challenges, model-free control approaches, such as deep reinforcement learning, show promise for the decentralized automation of participation within this context. Existing studies at the intersection of transactive energy and model-free control primarily focus on socioeconomic and self-consumption metrics, overlooking the crucial goal of reducing community-level net load variability. This study addresses this gap by training a set of deep reinforcement learning agents to automate end-user participation in ALEX, an economy-driven local energy market. In this setting, agents do not share information and only prioritize individual bill optimization. The study unveils a clear correlation between bill reduction and reduced net load variability in this setup. The impact on net load variability is assessed over various time horizons using metrics such as ramping rate, daily and monthly load factor, as well as daily average and total peak export and import on an open-source dataset. Agents are then benchmarked against several baselines, with their performance levels showing promising results, approaching those of a near-optimal dynamic programming benchmark.
4/23/2024
🏅
Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control
Yang Qu, Jinming Ma, Feng Wu
0
0
Active voltage control presents a promising avenue for relieving power congestion and enhancing voltage quality, taking advantage of the distributed controllable generators in the power network, such as roof-top photovoltaics. While Multi-Agent Reinforcement Learning (MARL) has emerged as a compelling approach to address this challenge, existing MARL approaches tend to overlook the constrained optimization nature of this problem, failing in guaranteeing safety constraints. In this paper, we formalize the active voltage control problem as a constrained Markov game and propose a safety-constrained MARL algorithm. We expand the primal-dual optimization RL method to multi-agent settings, and augment it with a novel approach of double safety estimation to learn the policy and to update the Lagrange-multiplier. In addition, we proposed different cost functions and investigated their influences on the behavior of our constrained MARL method. We evaluate our approach in the power distribution network simulation environment with real-world scale scenarios. Experimental results demonstrate the effectiveness of the proposed method compared with the state-of-the-art MARL methods.
5/15/2024