MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

Overview

• This paper introduces a novel approach to automatically mine user interface (UI) data from Android apps using Large Language Models (LLMs).
• Existing UI datasets are often outdated and contain noisy data, presenting challenges in modeling UI understanding.
• The proposed method leverages LLMs to mimic human-like exploration and employs best practices in UI noise filtering and human annotation to ensure dataset quality.
• The result is a large dataset, MUD, containing 18,000 human-annotated UIs from 3,300 apps, which is useful for common UI modeling tasks like element detection and UI retrieval.

Plain English Explanation

Designing good mobile user interfaces (UIs) is crucial, but it requires high-quality data to train the AI models that help with this task. Unfortunately, existing UI datasets are often outdated and messy, making it hard to accurately model how people interact with UIs in the real world.

To address this, the researchers in this paper developed a new way to automatically collect UI data from Android apps. They used large language models (LLMs), which are AI systems trained on vast amounts of text data, to mimic how a human would explore and interact with different apps. This allowed them to gather a large dataset of UI screenshots, while also using techniques to filter out low-quality or noisy data.

The end result is a new dataset called MUD, which contains over 18,000 UI screenshots from 3,300 different apps. This dataset has been carefully annotated by humans to ensure it is high-quality and useful for training AI models. The researchers demonstrate how MUD can be used for two common UI-related tasks: detecting individual UI elements and retrieving similar UIs.

Overall, this work provides a new, modern, and well-curated dataset that can help advance research into understanding and modeling mobile user interfaces, which is crucial for building better and more usable mobile apps.

Technical Explanation

The key innovation in this paper is the use of LLMs to automate the process of gathering UI data from Android apps. Traditionally, creating high-quality UI datasets has been a labor-intensive task, requiring manual exploration and annotation of apps.

To address this, the researchers developed a system that uses LLMs to mimic human-like app exploration. The LLMs are trained on a large corpus of text data, which allows them to understand natural language and make decisions about where to navigate within an app, similar to how a human user would. This automated exploration is combined with techniques to filter out low-quality or irrelevant UI screenshots, ensuring the resulting dataset (MUD) is clean and useful for downstream tasks.

The dataset curation process also includes a final validation step where human annotators review the collected UI screenshots and provide additional metadata, such as labels for individual UI elements. This human-in-the-loop approach helps ensure the dataset meets high standards of quality and accuracy.

The researchers demonstrate the usefulness of the MUD dataset through experiments on two common UI modeling tasks: element detection and UI retrieval. They show that models trained on MUD outperform those trained on existing datasets, highlighting the value of having a large, modern, and well-curated dataset for advancing research in this area.

Critical Analysis

One potential limitation of this work is the reliance on LLMs, which can be opaque and difficult to interpret. While the researchers demonstrate the effectiveness of their LLM-based approach, there may be concerns about the reproducibility and transparency of the data collection process. It would be valuable to see further analysis or ablation studies to better understand the contribution of the LLM component compared to other aspects of the system.

Additionally, the paper does not provide a detailed analysis of the types of UI elements or app categories represented in the MUD dataset. It would be helpful to understand the diversity and distribution of the data, as this could impact the generalizability of the models trained on this dataset.

Finally, while the researchers mention the potential for MUD to support multimodal UI understanding and large-scale data manipulation, these applications are not explored in depth. Further research could investigate how MUD can be leveraged for these and other emerging areas of UI-related research.

Conclusion

This paper presents a novel approach to automatically mining high-quality UI data from Android apps using Large Language Models. The resulting MUD dataset, containing over 18,000 human-annotated UI screenshots, has the potential to serve as a valuable resource for advancing research in mobile UI understanding and modeling. By addressing the challenges of outdated and noisy UI datasets, this work lays the foundation for developing more robust and effective AI-powered tools for designing and optimizing mobile user interfaces.