You Only Look at Screens: Multimodal Chain-of-Action Agents

Overview

• This paper introduces Auto-GUI, a multimodal solution that allows large language models (LLMs) to directly interact with graphical user interfaces (GUIs) without the need for external tools or application-specific APIs.
• The paper proposes a chain-of-action technique that leverages a series of intermediate previous action histories and future action plans to help the agent decide what action to execute.
• The approach is evaluated on a new device-control benchmark called AITW, which includes 30,000 unique instructions spanning multi-step tasks such as application operation, web searching, and web shopping.

Plain English Explanation

The paper presents a new way for AI models to interact with graphical user interfaces (GUIs) and automate tasks without manual intervention.

Traditionally, approaches that use large language models (LLMs) to interact with GUIs have relied on external tools and application-specific APIs to parse the environment into textual elements and interpret the predicted actions. This can lead to inefficient inference and the risk of errors propagating through the system.

To address these challenges, the authors introduce Auto-GUI, a multimodal solution that allows the AI model to directly interact with the GUI, bypassing the need for environment parsing or reliance on application-dependent APIs. The key innovation is a "chain-of-action" technique that helps the model decide what action to take by considering a series of previous actions and future action plans.

The approach is evaluated on a new benchmark called AITW, which includes 30,000 unique instructions for tasks like operating applications, web searching, and web shopping. The results show that Auto-GUI achieves state-of-the-art performance, with an action type prediction accuracy of 90% and an overall action success rate of 74%.

Technical Explanation

The Auto-GUI framework directly interacts with the graphical user interface, bypassing the need for environment parsing or reliance on application-dependent APIs. This allows the system to avoid the inefficient inference and error propagation risks associated with traditional approaches that rely on external tools and APIs.

The core innovation is the chain-of-action technique, which helps the agent decide what action to take by considering a series of intermediate previous action histories and future action plans. This allows the agent to reason about the consequences of its actions and develop a more coherent strategy for accomplishing complex, multi-step tasks.

The system is evaluated on the AITW benchmark, which includes 30,000 unique instructions across a diverse range of GUI-based tasks, such as application operation, web searching, and web shopping. The results show that Auto-GUI achieves state-of-the-art performance, with an action type prediction accuracy of 90% and an overall action success rate of 74%.

Critical Analysis

The paper presents a promising approach for enabling AI models to directly interact with graphical user interfaces, but there are a few potential limitations and areas for further research:

• The evaluation is limited to a single benchmark, AITW, which may not capture the full range of challenges and use cases in real-world GUI interactions. Further evaluation on a more diverse set of benchmarks would help validate the generalizability of the approach.
• The paper does not provide a detailed analysis of the types of errors or failure modes encountered by the system. Understanding these failure cases could inform future improvements to the chain-of-action technique or the overall system design.
• The integration of multimodal inputs and outputs (e.g., vision, touch, speech) could potentially enhance the system's capabilities and robustness, but this is not explored in the current work.

Overall, the Auto-GUI approach represents an important step forward in enabling AI agents to effectively interact with graphical user interfaces, with promising results that warrant further investigation and development.

Conclusion

The Auto-GUI framework introduces a novel solution for enabling large language models to directly interact with graphical user interfaces, bypassing the need for external tools and application-specific APIs. By leveraging a chain-of-action technique to reason about the consequences of its actions, the system can effectively automate complex, multi-step tasks in a diverse range of GUI-based environments.

The promising results on the AITW benchmark suggest that this approach could have significant implications for the development of autonomous agents capable of assisting humans with a wide variety of computer-based tasks. As the field of AI-powered user interfaces continues to evolve, the insights and techniques presented in this paper may pave the way for more intelligent and seamless interactions between humans and machines.