Multistep Criticality Search and Power Shaping in Microreactors with Reinforcement Learning

Overview

• Reducing operation and maintenance costs is a key objective for advanced nuclear reactors, especially microreactors.
• Developing robust autonomous control algorithms is essential to ensure safe and autonomous reactor operation.
• Reinforcement learning (RL) algorithms have shown promise in control problems, such as in fusion tokamaks and building energy management.
• This work explores the use of RL for intelligent control in nuclear microreactors.

Plain English Explanation

Nuclear power plants, including small-scale microreactors, need to be operated efficiently and cost-effectively. One way to achieve this is by developing advanced control systems that can automatically manage the reactor without constant human supervision.

Reinforcement learning (RL) is a type of artificial intelligence that has shown promise in controlling complex systems. RL algorithms learn by trial-and-error, exploring different actions and learning which ones work best to achieve their goals.

In this research, the authors used two cutting-edge RL techniques, Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C), to train an RL agent to control a simulated nuclear microreactor. The agent learned how to adjust the position of the reactor's control drums to maintain the reactor in a safe, stable, and efficient state as the fuel is used up over time.

The results demonstrate that the PPO-based RL agent was able to very effectively control the reactor, keeping the power distribution balanced and the reactor critical (i.e., maintaining the nuclear chain reaction). This suggests that RL could be a powerful tool for enabling real-time autonomous control of nuclear reactors through "digital twins" - virtual simulations of the actual reactor.

Technical Explanation

The researchers used a high-fidelity simulation of a nuclear microreactor design inspired by the Westinghouse eVinci[^1] concept to train their RL control policies. They first utilized a Serpent[^2] model to generate data on the reactor's control drum positions, core criticality, and power distribution. This data was then used to train a feedforward neural network surrogate model, which could quickly estimate the reactor's state given the control drum positions.

The researchers then trained two state-of-the-art RL algorithms, Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C), to learn control policies that could identify the optimal drum positions to maintain critical core conditions and a symmetrical power distribution across the reactor's six core sections.

The results showed that the PPO-based RL agent was able to achieve excellent performance, keeping the "hextant power tilt ratio" (a measure of power distribution symmetry) within the strict limit of less than 1.02 and maintaining criticality within a 10 pcm (percent millirho) range. In contrast, the A2C agent did not perform as well on these metrics.

These findings suggest that well-trained RL control policies can quickly identify the necessary control actions, opening the door for real-time autonomous control of nuclear reactors through digital twins.

[^1]: The Westinghouse eVinci is a microreactor design that aims to provide a compact, passively safe, and easily deployable nuclear power solution. [^2]: Serpent is a Monte Carlo neutron transport code used for reactor physics calculations and simulations.

Critical Analysis

The paper demonstrates the potential of reinforcement learning to enable autonomous control of nuclear microreactors, which could lead to significant reductions in operation and maintenance costs. However, it is important to note that this research was conducted in a simulated environment, and further validation in real-world scenarios would be necessary before deploying such RL-based control systems in actual nuclear reactors.

Additionally, the paper does not address potential safety concerns or regulatory hurdles that would need to be overcome before RL-based control systems could be implemented in nuclear reactors. Integrating RL with traditional model-predictive control approaches may be one way to address these concerns and provide a more comprehensive control solution.

Overall, this research represents an important step towards the development of advanced autonomous control systems for nuclear microreactors, but more work is still needed to fully validate and deploy these techniques in real-world settings.

Conclusion

This paper explores the use of reinforcement learning (RL) algorithms, specifically Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C), for intelligent control of nuclear microreactors. The results demonstrate that a well-trained PPO-based RL agent can effectively maintain critical core conditions and a symmetrical power distribution in a simulated microreactor, suggesting that RL could be a promising approach for enabling real-time autonomous control of nuclear reactors through digital twins. However, further research is needed to address safety and regulatory concerns before such RL-based control systems could be implemented in real-world nuclear power plants.