GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation

Overview

• Large vision-language models (LVLMs) have shown impressive abilities in image perception and language understanding
• Existing multimodal benchmarks focus on primary perception and commonsense knowledge, but don't fully reflect the comprehensive capabilities of LVLMs
• Researchers propose GAOKAO-MM, a new multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO)
• GAOKAO-MM tests LVLMs on a wide range of subjects and image types, including diagrams, function graphs, maps, and photos
• The benchmark is designed to assess LVLMs' perception, understanding, knowledge, and reasoning abilities at a human level

Plain English Explanation

The researchers wanted to create a more comprehensive test for large vision-language models, which are AI systems that can understand both images and language. Existing tests for these models focused mainly on basic visual perception and common sense knowledge, but the researchers felt this didn't fully capture the models' true capabilities.

To address this, they developed a new benchmark called GAOKAO-MM, which is based on the Chinese college entrance exam (known as the GAOKAO). This test covers 8 different subject areas and 12 types of images, like diagrams, graphs, maps, and photos. The goal is to assess the models' abilities in perception, understanding, knowledge, and reasoning at a human level.

When the researchers evaluated 10 different large vision-language models on this new benchmark, they found that the highest-performing models only scored around 50% accuracy. This suggests these models still have significant room for improvement before reaching human-level performance across a diverse set of multimodal tasks.

Technical Explanation

The researchers developed GAOKAO-MM, a new multimodal benchmark derived from the Chinese College Entrance Examination (GAOKAO). GAOKAO-MM encompasses 8 subject areas and 12 types of images, including diagrams, function graphs, maps, and photos. This benchmark is designed to assess the comprehensive capabilities of large vision-language models in perception, understanding, knowledge, and reasoning, going beyond the primary perception and commonsense knowledge tested in existing multimodal benchmarks.

The researchers evaluated 10 different LVLMs on GAOKAO-MM and found that the highest-performing model, GPT-4-Vision, achieved an accuracy of only 48.1%. The next best-performing models were Qwen-VL-Plus at 41.2% and Gemini-Pro-Vision at 35.1%. These results indicate that current LVLMs still have significant room for improvement to reach human-level performance on this comprehensive multimodal assessment.

The multi-dimensional analysis provided insights into the strengths and weaknesses of the evaluated LVLMs. This suggests that while these models have made substantial progress, they still have a moderate distance to travel towards the goal of artificial general intelligence (AGI). The findings of this study can help guide the development of more capable, multilingual large vision-language models.

Critical Analysis

The GAOKAO-MM benchmark provides a valuable new tool for evaluating the capabilities of large vision-language models. By testing a broader range of subjects and image types beyond just basic perception and commonsense, the benchmark offers a more comprehensive assessment of these models' true abilities.

However, one potential limitation is that the benchmark is based on the Chinese GAOKAO exam, which may introduce cultural and linguistic biases that could advantage or disadvantage certain models. It would be interesting to see if similar benchmarks could be developed for other educational contexts and languages to provide a more globally representative evaluation.

Additionally, while the results suggest current LVLMs still have significant room for improvement, the paper does not delve deeply into the specific strengths, weaknesses, and failure modes of the evaluated models. Further analysis in this area could yield additional insights to guide future model development.

Overall, the GAOKAO-MM benchmark represents an important step forward in multimodal evaluation and highlights the continued challenges in developing AI systems that can match human-level performance across a diverse range of tasks and modalities.

Conclusion

The researchers have proposed GAOKAO-MM, a new multimodal benchmark derived from the Chinese College Entrance Examination, to more comprehensively evaluate the capabilities of large vision-language models. By testing perception, understanding, knowledge, and reasoning abilities across a wide range of subjects and image types, GAOKAO-MM provides a more rigorous assessment than existing benchmarks.

The evaluation of 10 LVLMs on this benchmark revealed that even the best-performing models only achieved around 50% accuracy, suggesting these systems still have significant room for improvement before reaching human-level multimodal proficiency. The insights from this study can help guide the development of more capable, multilingual large vision-language models as the field progresses towards the long-term goal of artificial general intelligence.