Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

Overview

• Multimodal large language models (MLLMs) can process visual, textual, and auditory data to gain insights that complement human analysis.
• Existing video question-answering (VidQA) datasets often exhibit a bias toward a single modality, despite the goal of requiring advanced reasoning skills that integrate diverse modalities.
• This research introduces the modality importance score (MIS) to identify such bias and proposes a method using state-of-the-art MLLMs to estimate the modality importance.

Plain English Explanation

Large language models that can understand multiple types of data, like text, images, and audio, have the potential to provide insights that humans might miss. However, existing datasets used to test these models' abilities often focus more on a single type of information, even though the goal is to test how well the models can combine different types of information to answer questions.

The researchers in this paper developed a way to measure how important each type of information is for answering a particular question. They call this the "modality importance score" (MIS). By using the latest multimodal language models, they can estimate the MIS automatically, which can serve as a proxy for how humans perceive the importance of each modality.

Using the MIS, the researchers showed that many existing datasets have a bias towards a single type of information, rather than truly testing the models' ability to integrate diverse types of information. They also found that current models don't effectively combine information from different sources due to this modality imbalance in the datasets.

The researchers suggest that the MIS can be used to create more balanced datasets that will better assess multimodal learning and improve the capabilities of these powerful language models to understand the relationships between different types of information.

Technical Explanation

The researchers introduce the modality importance score (MIS) to quantify the bias in existing video question-answering (VidQA) datasets towards a single modality, despite the goal of requiring advanced reasoning skills that integrate diverse modalities.

They propose an innovative method using state-of-the-art multimodal large language models (MLLMs) to estimate the modality importance, which can serve as a proxy for human judgments of modality perception. By applying this MIS analysis, the researchers demonstrate the presence of unimodal bias and the scarcity of genuinely multimodal questions in existing datasets.

The researchers further validate the modality importance score with multiple ablation studies to evaluate the performance of MLLMs on permuted feature sets. Their results indicate that current models do not effectively integrate information due to modality imbalance in existing datasets.

The proposed MLLM-derived MIS can guide the curation of modality-balanced datasets that advance multimodal learning and enhance MLLMs' capabilities to understand and utilize synergistic relations across modalities.

Critical Analysis

The researchers acknowledge several caveats and limitations in their work. They note that the MIS estimation may not fully capture human perception of modality importance, as their method relies on the performance of current MLLMs, which may not perfectly align with human judgments.

Additionally, the researchers do not provide a detailed analysis of the specific types of unimodal bias present in the examined datasets, which could limit the insights gained from their findings.

Further research could explore alternative methods for assessing modality importance, potentially involving more direct human evaluation or the development of novel MLLM architectures specifically designed for multimodal integration.

Despite these limitations, the modality importance score represents a valuable tool for identifying and mitigating biases in multimodal datasets, which is a crucial step in advancing the field of multimodal learning and enhancing the capabilities of large language models.

Conclusion

This research introduces the modality importance score (MIS) to quantify the bias in existing multimodal datasets towards a single modality, despite the goal of requiring advanced reasoning skills that integrate diverse modalities.

By using state-of-the-art multimodal large language models (MLLMs) to estimate the MIS, the researchers demonstrate the presence of unimodal bias and the scarcity of genuinely multimodal questions in current datasets.

The proposed MLLM-derived MIS can guide the curation of modality-balanced datasets that advance multimodal learning and enhance MLLMs' capabilities to understand and utilize synergistic relations across modalities, ultimately improving the performance and robustness of these powerful language models.