LLM+Reasoning+Planning for supporting incomplete user queries in presence of APIs

Overview

• Recent advancements in Large Language Models (LLMs) have enabled the development of natural language interfaces for various end-user tasks.
• These end-user tasks often require orchestrating multiple APIs, but user queries can be incomplete, lacking all the necessary information.
• While LLMs excel at natural language processing, they can struggle with handling missing information or orchestrating APIs.
• The proposed approach leverages logical reasoning and classical AI planning along with an LLM to accurately answer user queries, including identifying and gathering any missing information.

Plain English Explanation

The recent availability of Large Language Models (LLMs) has led to the creation of numerous natural language interfaces for helping users with various tasks. These tasks often involve using a set of pre-existing software tools, or APIs, in a specific order.

However, users don't always provide all the necessary information in their initial requests. While LLMs are great at understanding natural language, they can sometimes make up information or have trouble coordinating the right sequence of API calls to fully address the user's needs.

The researchers in this paper have come up with a new approach that combines the power of LLMs with classical AI planning techniques. Their method first translates the user's request into a formal representation that can be understood by an AI planning system. This system then figures out the correct set of API calls, including any additional information that needs to be gathered, to properly answer the user's query.

The key idea is to use both the natural language processing capabilities of LLMs and the logical reasoning abilities of classical AI planning to robustly handle incomplete user requests. This hybrid approach has been shown to significantly outperform using an LLM alone, achieving over 95% success in answering both simple and more complex multi-step queries.

Technical Explanation

The paper proposes a novel approach that leverages logical reasoning and classical AI planning along with an LLM to accurately answer user queries, even when they are missing information.

The system first uses an LLM to translate the user's natural language request into an intermediate representation in Answer Set Programming (ASP). It then further translates this ASP representation into a Planning Domain Definition Language (PDDL) format, which can be understood by a classical AI planner.

The key innovation is the introduction of a special get_info_api that the planner can use to identify and gather any missing information required to fully address the user's query. The researchers also model all the available APIs as PDDL actions, capturing the dataflow between them.

The planner then generates an orchestration of API calls, including calls to get_info_api, to answer the user's request. The evaluation shows that this hybrid approach significantly outperforms a pure LLM-based solution, achieving over 95% success on a dataset of both complete and incomplete single-goal and multi-goal queries.

Critical Analysis

The paper presents a promising approach to leveraging both LLM and classical AI planning techniques to robustly handle incomplete user queries. However, there are a few potential caveats and areas for further research:

• The evaluation was limited to a specific dataset, and it's unclear how well the approach would generalize to a wider range of real-world user requests.
• The paper doesn't discuss the performance or computational overhead of the planning system, which could be a practical concern for real-time applications.
• The researchers mention that the get_info_api mechanism relies on the availability of relevant information sources, but they don't explore how this could be scaled or automated in a practical system.
• While the hybrid approach outperforms a pure LLM-based solution, it's still not clear how it would compare to other potential methods, such as training the LLM to reason about missing information or using the LLM to generate planning domains.

Overall, the proposed NL2Plan approach is a valuable contribution to the field of natural language interfaces and AI planning, but further research and real-world testing would be needed to fully evaluate its practical benefits and limitations.

Conclusion

This paper presents a novel approach that combines the strengths of Large Language Models (LLMs) and classical AI planning to enable robust natural language interfaces for end-user tasks. By leveraging logical reasoning and a specialized get_info_api, the system can accurately answer user queries, even when they are missing information.

The key innovation is the hybrid architecture that translates natural language requests into a formal planning representation, allowing a classical planner to orchestrate the necessary API calls, including those to gather any missing information. The evaluation shows this approach significantly outperforms a pure LLM-based solution, suggesting it could be a valuable tool for building more capable and reliable natural language interfaces.

While the paper presents promising results, further research is needed to explore the scalability, performance, and generalization of this approach in real-world applications. Nonetheless, the proposed NL2Plan system represents an important step forward in the integration of modern language models and classical AI planning techniques to enhance the capabilities of natural language interfaces.