Adaptive Energy Regularization for Autonomous Gait Transition and Energy-Efficient Quadruped Locomotion

Introduction

The text discusses how humans and animals exhibit different locomotion behaviors, such as walking at low speeds and running at higher speeds, to optimize their energy efficiency. Prior research has shown that quadrupeds exhibit different optimal gaits, such as four-beat walking at low speeds, trotting at intermediate speeds, and trotting/galloping at high speeds. Using gait information as guidance for locomotion policies is a popular approach in reinforcement learning-based methods. However, creating a versatile and robust locomotion policy that can adapt to different speeds and generalize across platforms is challenging, particularly in terms of reward design. Previous works have used various reward terms, such as gait reference, feet-air time, and contact force penalties, to induce specific behaviors and stabilize the training process. While these additional reward components aim to promote certain behavioral traits, they are also aligned with the broader objective of reducing energy costs. This observation leads to the question of whether a more straightforward, energy-centric reward term could be used to replace the specifically designed terms used in prior policy training.

This paper investigates a streamlined reward formulation for energy-efficient locomotion in quadruped robots. Prior work has shown that energy-efficient gaits correlate with speed and that energy minimization at pre-selected speeds leads to the emergence of specific gaits. This study aims to verify if a simplified approach focusing solely on energy minimization can yield stable and effective velocity-tracking across a range of speeds, without the need for multiple velocity-specific energy-optimal policies.

The paper examines the influence of energy regularization weights on policy performance, finding that both excessively low and high weights can lead to unnatural movements or immobility. To address this, the researchers design a non-negative energy reward function and an adaptive reward form by interpolating the maximum energy weights at selected speeds. This adaptive reward structure is employed within the IsaacGym framework to train robust policies for the ANYmal-C and Unitree Go1 quadruped robots.

The trained policies autonomously adopt different energy-efficient gaits, such as four-beat walking at lower speeds and trotting at higher speeds, without preset gait knowledge. These policies outperform baseline approaches in terms of energy efficiency and significantly improve velocity tracking and energy consumption performance compared to a policy trained with fixed-weight energy rewards.

The paper concludes by deploying the trained single policy on a real Unitree Go1 robot, verifying its stable moving and transition locomotion skills in the real world. The main contributions of this paper include the introduction of a streamlined reward formula integrating velocity-tracking and adaptive energy minimization, the demonstration of autonomous gait adoption at varying speeds, and the evaluation and real-world application of the derived policies.

Related Works

This text summarizes recent research on using reinforcement learning (RL) to generate locomotion skills for legged robots. Early RL approaches achieved walking policies on flat ground, but tended to converge to a single trotting gait that may not be optimal for all terrains and speeds.

To overcome this limitation, researchers have explored methods to promote behavioral diversity, such as using hierarchical frameworks with predefined gait primitives or directly incorporating gait-related inputs into the RL model. However, these approaches still require manual specification of gaits or rely on training multiple velocity-specific policies.

The text also discusses the importance of energy efficiency in legged locomotion, with the cost of transport being a key metric. Prior work has used conceptual legged models and optimal control to study energy-efficient gaits at different speeds. The current work aims to generate a single RL policy that can transition between energy-optimal gaits across a range of velocities, rather than relying on complex reward engineering.

The provided figures show example gait patterns and robot simulations demonstrating different gaits at varying speeds.

Locomotion Reward Design

This paper proposes a general form of an energy-regularized locomotion reward function for training legged robot control policies using reinforcement learning. The reward function consists of three components:

1. Motion rewards: These encourage the robot to accurately track linear and angular velocity reference commands.

2. Energy rewards: These discourage energy consumption by the robot's motors. The energy reward is expressed as an exponential function of the absolute product of joint torques and velocities.

3. Adaptive energy reward weight: The weight of the energy reward is made adaptive to the reference velocity, as the authors found that fixed energy reward weights only work well within a narrow range of velocities. They collect velocity-weight sample pairs and linearly interpolate between them to get the adaptive energy reward weight.

4. Normalization index: Due to the adaptive energy reward weight, the overall reward scale varies across different velocities. To stabilize training, the authors also record the final achieved reward for each velocity-weight sample and linearly interpolate to get a normalization index curve.

The paper claims this reward function design allows training energy-efficient locomotion policies across a wide range of reference velocities.

V Locomotion Skill Training Details

The paper describes the reward design and training settings used for reinforcement learning (RL) in quadruped robot locomotion. The energy reward alone is not sufficient for RL training, so it is used together with basic locomotion rewards.

For the ANYmal-C robot, the training uses the legged-gym framework and Proximal Policy Optimization (PPO) algorithm. The system takes in various sensor inputs and outputs joint position commands. The training is done on flat ground with randomized friction and mass. Modifications are made to the velocity tracking reward function to improve convergence. The energy reward scaling and its relationship with reference velocity are determined through fixed-velocity policy training.

For the Unitree Go1 robot, the training follows the walk-these-ways framework. In addition to the energy reward, an auxiliary reward based on safety considerations is added. Curriculum learning on the velocity sampling range is found to be essential for successful training. The energy reward scaling is determined similarly to the ANYmal-C case.

The paper provides detailed descriptions of the training settings and reward design for the two quadruped robot platforms, which constitute the key technical contributions.

Experiments

The experiments were designed to show the following after the legged robot was trained using the adaptive energy-regularized reward shown in Equation (2):

1. With the adaptive weighting functions α(v̂x) and Z(v̂x), the legged robot automatically selects suitable behaviors to move with the reference velocity. The trained policy is also deployable to a real quadruped robot.

2. If the weighting functions α(v̂x) and Z(v̂x) are constant, the legged robot may not be able to find an energy-efficient walking policy for all target velocities.

3. The proposed method can generate a more energy-efficient walking policy compared to established baselines.

The paper demonstrates how a trained walking policy for a legged robot can exhibit different locomotion gaits at different reference velocities. At low speeds (0.5 m/s), the policy exhibits a four-beat walking gait, moving one leg at a time. At medium speeds (1.0 m/s), it shows an intermediate gait between walking and trotting. At high speeds (2.0 m/s), the policy exhibits a standard trotting gait.

The paper also shows that the swing ratio of the robot's legs is regularized - the legs only lift high enough to reach the reference velocity, avoiding unnecessary energy expenditure. This contrasts with other methods that tend to lift the legs redundantly high.

Ablation studies are performed to examine the effects of using a fixed energy reward scale versus an adaptive one that varies with velocity. The results show the adaptive approach achieves the lowest energy consumption while maintaining good velocity tracking accuracy.

Finally, the paper compares the trained policy to other methods on a real-world Go1 robot, demonstrating that the policy swings the legs only to the necessary height, making it the most energy-efficient.

Discussions

This paper presents a reinforcement learning approach to develop energy-efficient locomotion strategies for quadruped robots. However, the method has some limitations.

The key limitation is the requirement to pre-run experiments to determine appropriate energy regularization weights. This reliance on empirical observations limits the flexibility of the reinforcement learning system. While the developed policy can be applied across different speeds after training, it cannot be obtained with only one training session for a new quadruped platform. The paper also did not verify the adaptive capability of energy rewards in different environments, which would be crucial for real-world deployment.

Future research could address these limitations by developing methodologies to automatically tune energy regularization weights within a single reinforcement learning training. This would enable the system to dynamically adjust its strategy in response to multi-task reinforcement learning or cross-embodiment settings.

Beyond locomotion, the underlying principle of leveraging energy efficiency could be applied to other robotic tasks, such as manipulation and interaction. Implementing an energy-centric approach across different tasks could help align robotic systems more closely with sustainability principles and environmental consciousness.

Conclusions

The paper presented a new approach to energy-efficient locomotion in quadruped robots. It used a simplified, energy-focused reward strategy within a reinforcement learning framework. The method allowed quadruped robots, such as ANYmal-C and Unitree Go1, to autonomously develop and transition between various gaits at different velocities without relying on predefined gait patterns or complex reward designs. An adaptive energy reward function, adjusted based on velocity, enabled the robots to naturally select the most energy-efficient locomotion strategies. The policy demonstrated energy-efficient behaviors and gait transitions in both simulation (ANYmal-C) and hardware (Go1) experiments. The paper also showed the usefulness of adaptive energy regularization through ablation studies.

Acknowledgements

This research was funded by several organizations, including the Agency of Science, Technology and Research of Singapore through the National Science Scholarship, the FANUC Advanced Research Laboratory, and the InnoHK program of the Hong Kong government's Hong Kong Centre for Logistics Robotics.