How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

Overview

• This paper introduces InternVL 1.5, an open-source multimodal large language model (MLLM) that aims to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding.
• The model includes three key improvements: a stronger vision encoder, dynamic high-resolution support, and a high-quality bilingual dataset.
• InternVL 1.5 is evaluated on a series of benchmarks, outperforming both open-source and proprietary models in 8 out of 18 tasks.

Plain English Explanation

The researchers have developed a new open-source multimodal language model called InternVL 1.5 that can understand and process both text and images. This model aims to provide capabilities similar to expensive commercial models, but in an open-source format that is freely available.

To improve the model's performance, the researchers made three key changes:

1. Stronger Vision Encoder: They enhanced the model's ability to understand visual information by improving the underlying vision-based machine learning component.
2. Dynamic High-Resolution: The model can now handle high-quality, high-resolution images, up to 4K resolution, by intelligently dividing the images into smaller tiles.
3. High-Quality Bilingual Dataset: The researchers created a new multilingual dataset with English and Chinese question-answer pairs, covering common scenes and document images. This helps the model perform better on tasks involving text in multiple languages, such as optical character recognition (OCR) and Chinese-related tasks.

When evaluated on a range of benchmark tests, InternVL 1.5 showed competitive performance compared to both open-source and proprietary models, achieving the best results in 8 out of 18 tasks.

Technical Explanation

The researchers introduced three key improvements to the InternVL model:

1. Strong Vision Encoder: They explored a continuous learning strategy for the large-scale vision foundation model, InternViT-6B, to boost its visual understanding capabilities. This makes the vision encoder more transferable and reusable in different large language models (LLMs).
2. Dynamic High-Resolution: The model can divide images into tiles ranging from 1 to 40 of 448x448 pixels, depending on the aspect ratio and resolution of the input images. This supports up to 4K resolution input, enabling the model to handle high-quality visual data.
3. High-Quality Bilingual Dataset: The researchers carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs. This significantly enhances the model's performance on OCR- and Chinese-related tasks.

The researchers evaluated InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 showed competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks.

Critical Analysis

While the improvements made to InternVL 1.5 are significant, the paper does not address some potential limitations or areas for further research:

• The performance comparison with proprietary models is limited, as the paper only provides high-level results without detailed analysis. More in-depth comparisons could help better understand the model's strengths and weaknesses.
• The scope of the bilingual dataset is not clearly defined, and it is unclear how representative it is of real-world multimodal data. Expanding the dataset's diversity and complexity could further enhance the model's capabilities.
• The specific use cases where InternVL 1.5 excels or falls short are not discussed in detail. Contextual analysis of the model's performance on different types of tasks or domains would provide more insights.

Overall, the research presents promising advancements in open-source multimodal language models, but further investigation and transparency could strengthen the conclusions and implications of this work.

Conclusion

The researchers have introduced InternVL 1.5, an open-source multimodal language model that aims to bridge the capability gap between open-source and proprietary commercial models. By incorporating a stronger vision encoder, dynamic high-resolution support, and a high-quality bilingual dataset, the model has achieved competitive performance on a range of benchmarks, outperforming both open-source and proprietary models in several tasks.

This research represents a significant step forward in the development of accessible and capable multimodal language models, which can have far-reaching implications for various applications, such as natural language processing, video understanding, and multimodal fusion. By expanding the capabilities of open-source models, the researchers are contributing to the democratization of advanced AI technologies and paving the way for more inclusive and accessible multimodal AI applications.