HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model

Overview

• Large language model (LLM) agents can operate in various domains, processing environmental observations to generate executable actions for target tasks.
• The effectiveness of these agents is significantly influenced by their memory mechanism, which records historical experiences as sequences of action-observation pairs.
• Memory can be categorized into two types: cross-trial memory (accumulated across multiple attempts) and in-trial memory (working memory, accumulated within a single attempt).
• While cross-trial memory has been extensively optimized, enhancing agent performance through improved working memory utilization remains underexplored.
• Existing approaches often involve directly inputting entire historical action-observation pairs into LLMs, leading to redundancy in long-horizon tasks.

Plain English Explanation

Large language model (LLM) agents are computer systems that can process information from their environment and take actions to achieve specific goals. These agents are very powerful, but their effectiveness is heavily influenced by how they store and use their past experiences.

Agents can store two types of memories: cross-trial memory and in-trial memory (working memory). Cross-trial memory is what the agent has learned over multiple attempts at a task, while working memory is what the agent remembers during a single attempt.

Researchers have done a lot of work to optimize cross-trial memory, but improving how agents use their working memory has not been explored as much. Existing approaches often try to directly feed the agent's entire history of actions and observations, which can be inefficient for long-term tasks.

The paper introduces a new framework called HiAgent that aims to help LLM agents use their working memory more effectively. HiAgent prompts the LLM to break down a task into smaller sub-goals and only keep track of the information relevant to the current sub-goal, rather than the full history. This can help the agent perform better on long-term tasks.

Technical Explanation

The paper introduces HiAgent, a framework that leverages subgoals as memory chunks to manage the working memory of LLM-based agents hierarchically. Specifically, HiAgent prompts LLMs to formulate subgoals before generating executable actions and enables LLMs to decide proactively to replace previous subgoals with summarized observations, retaining only the action-observation pairs relevant to the current subgoal.

Experimental results across five long-horizon tasks demonstrate that HiAgent achieves a twofold increase in success rate and reduces the average number of steps required by 3.8. The analysis also shows that HiAgent consistently improves performance across various steps, highlighting its robustness and generalizability.

Critical Analysis

The paper acknowledges that while considerable research has optimized performance through cross-trial memory, the enhancement of agent performance through improved working memory utilization remains underexplored. This presents an important gap in the existing literature that the HiAgent framework aims to address.

One potential limitation mentioned is the reliance on LLMs to formulate subgoals, which could introduce biases or errors if the LLM is not sufficiently capable. Additionally, the paper does not explore the scalability of HiAgent as the complexity of tasks or the size of the LLM increases.

Further research could investigate alternative approaches to working memory management, such as incorporating external memory modules or exploring the use of memory-augmented neural networks. Comparisons to other working memory-focused techniques would also help contextualize the contributions of HiAgent.

Conclusion

The HiAgent framework represents a promising step towards enhancing the performance of LLM-based agents by improving the utilization of their working memory. By prompting LLMs to formulate subgoals and selectively retain relevant action-observation pairs, HiAgent achieves significant improvements in success rates and task completion efficiency across various long-horizon tasks.

This work highlights the importance of working memory management in the development of more capable and efficient LLM-based agents, which have the potential to impact a wide range of applications, from robotics and planning to interactive software systems and game AI.