MELA: Multilingual Evaluation of Linguistic Acceptability

Overview

• Researchers present the largest benchmark to date on linguistic acceptability, called Multilingual Evaluation of Linguistic Acceptability (MELA), covering 10 languages from diverse language families.
• They establish baseline performance of large language models (LLMs) on this benchmark and investigate cross-lingual transfer in acceptability judgments using XLM-R.
• The researchers also conduct probing experiments to explore how syntax capability is acquired by fine-tuning XLM-R on the MELA dataset.

Plain English Explanation

The researchers have created a new benchmark, called MELA, to test how well AI language models can judge the grammatical correctness of sentences in 10 different languages. This is the largest such benchmark to date, covering a diverse set of languages.

The researchers tested several language models, including the powerful GPT-4o and the open-source XLM-R, to see how well they can perform on this benchmark. They found that GPT-4o exhibited strong multilingual abilities, outperforming the fine-tuned XLM-R model. The open-source XLM-R model lagged behind the more capable GPT-4o.

The researchers also looked at how well the language models can transfer their knowledge from one language to another. For example, they found that training the XLM-R model on just 500 examples in Icelandic led to a 23% improvement in its performance on a completely different language, Chinese. This suggests that the models are learning some general principles about language structure that can be applied across languages.

To better understand how the models acquire these language skills, the researchers probed the inner workings of the fine-tuned XLM-R model. They found that training on the MELA dataset improved the model's performance on tasks related to syntax, indicating that the model is learning important grammatical concepts through this type of training.

The researchers have made the MELA dataset publicly available for other researchers to use and build upon. This could lead to further advancements in the development of multilingual language models that can better understand and generate grammatically correct text across a wide range of languages.

Technical Explanation

The researchers present the Multilingual Evaluation of Linguistic Acceptability (MELA) benchmark, which is the largest to date for assessing the linguistic acceptability of sentences across 10 diverse languages. They establish baseline performance of large language models (LLMs), including the powerful GPT-4o and the open-source XLM-R, on this benchmark.

To investigate cross-lingual transfer in acceptability judgments, the researchers conduct experiments using the XLM-R model. They find that transfer in acceptability judgment is non-trivial, with 500 Icelandic fine-tuning examples leading to a 23% improvement in performance on a completely unrelated language, Chinese.

In pursuit of multilingual interpretability, the researchers perform probing experiments by fine-tuning XLM-R on the MELA dataset. Their results indicate that this training improves the model's performance on syntax-related tasks, suggesting that the model is learning important grammatical concepts through this type of training.

The researchers' findings show that GPT-4o exhibits strong multilingual abilities, outperforming the fine-tuned XLM-R model, while open-source multilingual models like XLM-R lag behind by a noticeable gap. This highlights the importance of continued research and development in scaling multilingual capabilities of LLMs.

The MELA dataset is publicly available at https://github.com/sjtu-compling/MELA, and the researchers' work contributes to the growing body of research on evaluating and mitigating linguistic discrimination in LLMs.

Critical Analysis

The researchers acknowledge that their experiments only cover a small set of languages, and more work is needed to fully understand the cross-lingual transfer capabilities of language models. Additionally, the Megaverse benchmark could provide a more comprehensive evaluation of multilingual performance across a wider range of tasks and languages.

While the researchers demonstrate the utility of the MELA dataset for probing syntax capabilities, it would be valuable to explore the model's performance on other linguistic phenomena, such as semantics and pragmatics, to gain a more holistic understanding of the model's linguistic competence.

Furthermore, the researchers' findings highlight the need for continued research and development in improving the multilingual capabilities of LLMs, as the open-source models still lag behind the more capable GPT-4o. Addressing this gap could have significant implications for the accessibility and inclusivity of language technology.

Conclusion

The researchers have presented the largest benchmark to date on linguistic acceptability, MELA, covering 10 diverse languages. Their findings demonstrate the strong multilingual capabilities of the GPT-4o model and the potential for cross-lingual transfer in acceptability judgments. The researchers' probing experiments also provide insights into how language models acquire syntax capabilities.

This work contributes to the growing body of research on evaluating and mitigating linguistic discrimination in LLMs and highlights the importance of continued development in scaling multilingual capabilities of LLMs. The publicly available MELA dataset and the researchers' insights can serve as a valuable resource for the broader research community.