KMMLU: Measuring Massive Multitask Language Understanding in Korean

Overview

• Researchers propose a new Korean language benchmark called KMMLU
• KMMLU contains 35,030 expert-level multiple-choice questions across 45 subjects
• Unlike prior Korean benchmarks, KMMLU is based on original Korean exams, capturing linguistic and cultural aspects
• They test 27 public and proprietary large language models (LLMs) on KMMLU
• The best public model scores 50.5%, suggesting significant room for improvement
• Current Korean-focused LLMs like Polyglot-Ko perform poorly, and even top proprietary models like GPT-4 do not exceed 60%
• This highlights the need for further work to improve LLMs for the Korean language

Plain English Explanation

Researchers have developed a new benchmark called KMMLU to evaluate how well large language models (LLMs) can understand the Korean language. Unlike previous Korean benchmarks that were translated from English, KMMLU is based on original Korean exam questions, which helps capture the unique linguistic and cultural aspects of the Korean language.

The KMMLU benchmark contains 35,030 expert-level multiple-choice questions across 45 different subjects, ranging from humanities to science and technology. The researchers tested 27 different public and proprietary LLMs on this benchmark. They found that the best-performing public model scored around 50.5%, leaving significant room for improvement.

Interestingly, even the most capable proprietary LLMs, such as GPT-4 and HyperCLOVA X, did not exceed 60% on the KMMLU benchmark. This suggests that current LLMs, including those specifically tailored for Korean like Polyglot-Ko, still struggle to fully understand the complexities of the Korean language.

The researchers believe that the KMMLU benchmark provides a valuable tool to track progress in improving LLM performance on Korean language understanding. By making the dataset publicly available on the Hugging Face Hub and integrating it into the EleutherAI Language Model Evaluation Harness, they hope to encourage further research and development in this area.

Technical Explanation

The researchers developed the KMMLU (Korean Multi-Modal Language Understanding) benchmark, which contains 35,030 expert-level multiple-choice questions across 45 subjects. Unlike previous Korean benchmarks that were translated from existing English benchmarks, such as MMLU and CFLUE, KMMLU is collected from original Korean exams, capturing the linguistic and cultural aspects of the Korean language.

The researchers tested 27 public and proprietary large language models (LLMs) on the KMMLU benchmark. They found that the best-performing public model scored 50.5%, suggesting significant room for improvement. This model was primarily trained for English and Chinese, not Korean.

Current LLMs tailored for Korean, such as Polyglot-Ko, performed far worse on the KMMLU benchmark. Surprisingly, even the most capable proprietary LLMs, like GPT-4 and HyperCLOVA X, did not exceed 60% on the benchmark.

These results highlight the need for further work to improve LLM performance on the Korean language. The researchers believe that the KMMLU benchmark provides a valuable tool to track progress in this area and have made the dataset publicly available on the Hugging Face Hub and integrated it into the EleutherAI Language Model Evaluation Harness.

Critical Analysis

The researchers have acknowledged several limitations and areas for further research in their paper. One key limitation is the reliance on multiple-choice questions, which may not fully capture the depth of language understanding required in real-world applications. Additionally, the benchmark is focused solely on the Korean language, and it would be interesting to see how LLMs perform on cross-lingual or multilingual tasks involving Korean.

Another potential concern is the subjectivity inherent in the expert-level questions used in the KMMLU benchmark. While the researchers have taken steps to ensure the quality and consistency of the questions, there may be room for further refinement or validation of the benchmark.

Furthermore, the researchers have not delved deeply into the specific reasons why current LLMs struggle with the KMMLU benchmark. Investigating the linguistic and cultural factors that contribute to this performance gap could provide valuable insights for future model development.

Overall, the KMMLU benchmark represents an important step forward in the evaluation of language understanding for the Korean language. By making the dataset publicly available and encouraging further research, the researchers have laid the groundwork for improving LLM performance in this domain.

Conclusion

The KMMLU benchmark developed by the researchers offers a valuable tool for assessing the language understanding capabilities of large language models in the context of the Korean language. The benchmark's focus on original Korean exam questions, rather than translated content, helps capture the unique linguistic and cultural aspects of the Korean language.

The researchers' findings suggest that current LLMs, even the most capable proprietary models, still struggle to achieve high performance on the KMMLU benchmark. This highlights the need for further research and development to improve LLM performance on the Korean language.

By making the KMMLU dataset publicly available and integrating it into the EleutherAI Language Model Evaluation Harness, the researchers have created opportunities for the broader research community to contribute to advancing the state of the art in Korean language understanding. This work has the potential to lead to more accurate and culturally-aware language models, which could have significant implications for a wide range of applications involving the Korean language.