CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

Overview

• As large multimodal models (LMMs) become more advanced, evaluating their performance is increasingly important.
• There is a significant gap in evaluating the knowledge and reasoning abilities of LMMs in non-English contexts, such as Chinese.
• The paper introduces CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to assess LMMs on tasks requiring college-level subject knowledge and deliberate reasoning in a Chinese context.

Plain English Explanation

The paper discusses the growing need to evaluate the capabilities of large multimodal models (LMMs) - models that can process and understand different types of data, such as text, images, and audio. As these models become more advanced, it's important to assess how well they perform on various tasks.

One area that has been overlooked is evaluating LMMs in non-English contexts, such as Chinese. The researchers created a new benchmark called CMMMU to address this gap. CMMMU is designed to test an LMM's ability to demonstrate college-level subject knowledge and engage in complex reasoning using a Chinese-language dataset.

The benchmark includes over 12,000 multimodal questions (questions that combine text and images) from college exams, quizzes, and textbooks, covering a wide range of subjects like art, business, science, and technology. The questions test the LMM's ability to understand different types of visual information, such as charts, diagrams, and chemical structures, and apply that knowledge to answer the questions.

By evaluating LMMs on this benchmark, the researchers aim to identify areas for improvement and drive the development of more capable artificial intelligence systems that can excel in diverse language contexts.

Technical Explanation

The researchers created CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark, to evaluate the performance of large multimodal models (LMMs) in a Chinese-language context. CMMMU is inspired by and follows the same annotation and analysis pattern as the previously developed MMMU benchmark.

CMMMU includes 12,000 manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and comprise 39 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures.

The researchers evaluated 11 open-source large language models (LLMs) and one proprietary GPT-4V(ision) model on the CMMMU benchmark. Even the high-performing GPT-4V model only achieved an accuracy of 42%, indicating a significant room for improvement in the capabilities of LMMs when it comes to complex perception and reasoning with domain-specific knowledge in the Chinese context.

Critical Analysis

The CMMMU benchmark provides a valuable tool for evaluating the advanced knowledge and reasoning abilities of large multimodal models in a non-English context. By focusing on the Chinese language, the researchers have addressed an important gap in the existing literature, which has primarily focused on English-language benchmarks.

One potential limitation of the CMMMU benchmark is the reliance on manually collected questions from college-level materials. While this approach ensures the questions are challenging and representative of real-world knowledge requirements, it may also introduce biases or inconsistencies in the dataset. The researchers acknowledge this and suggest that future work could explore automating the question generation process to create a more scalable and diverse benchmark.

Additionally, the paper does not provide a detailed analysis of the specific areas where the evaluated LMMs struggled the most, such as particular types of visual information or subject domains. A more granular understanding of the models' strengths and weaknesses could help guide future research and development efforts.

Overall, the CMMMU benchmark is a valuable contribution to the field of large multimodal model evaluation, and the researchers' findings highlight the need for continued advancements in artificial intelligence to achieve expert-level performance in diverse language and knowledge domains.

Conclusion

The introduction of the CMMMU benchmark represents a significant step forward in evaluating the capabilities of large multimodal models in a Chinese-language context. By focusing on tasks that require college-level subject knowledge and complex reasoning, the benchmark aims to push the boundaries of what these models can achieve.

The researchers' evaluation of several leading LLMs, including the high-performing GPT-4V, reveals that there is still a considerable gap in the models' ability to excel in the Chinese-language domain. This underscores the importance of developing more robust and adaptable artificial intelligence systems that can thrive in diverse language and cultural contexts.

The CMMMU benchmark's impact on the field could be significant, as it provides a standardized way to measure progress and inform the development of the next generation of large multimodal models. By promoting the democratization of LMMs across languages, the benchmark has the potential to bring the benefits of advanced AI to a wider global audience.