NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages

Overview

• The researchers create a new multi-choice Reading Comprehension dataset called NaijaRC for three native Nigerian languages based on high-school reading comprehension exams.
• They provide baseline results by performing cross-lingual transfer using existing English RACE and Belebele datasets with a pre-trained encoder-only model.
• Additionally, they provide results by prompting large language models (LLMs) like GPT-4.

Plain English Explanation

The researchers have developed a new dataset called NaijaRC that contains reading comprehension questions and answers in three native Nigerian languages. This dataset is based on high-school level reading exams in these languages.

To establish a baseline for how well machine learning models can perform on this new dataset, the researchers used two existing English reading comprehension datasets, RACE and Belebele, and fine-tuned a pre-trained encoder-only model on them. This allowed the model to transfer its learning from English to the Nigerian languages.

The researchers also tested how well large language models (LLMs) like GPT-4 could perform on the NaijaRC dataset when prompted to answer the reading comprehension questions.

Technical Explanation

The researchers created the NaijaRC dataset, which contains multi-choice reading comprehension questions and answers in three native Nigerian languages: Hausa, Igbo, and Yoruba. They sourced the questions and passages from high-school level reading comprehension exams in these languages.

To establish baseline performance, the researchers used cross-lingual transfer learning. They fine-tuned a pre-trained encoder-only model, like BERT, on the English RACE and Belebele datasets. This allowed the model to leverage its English language understanding to perform well on the Nigerian language datasets.

Additionally, the researchers prompted large language models (LLMs) like GPT-4 to answer the NaijaRC questions directly, without any fine-tuning. This provided a sense of how capable these powerful language models are at reading comprehension in non-English languages.

Critical Analysis

The researchers acknowledge that their dataset, NaijaRC, is relatively small compared to widely used English reading comprehension datasets. This may limit the ability of machine learning models to fully generalize and perform well on this new dataset.

Furthermore, the cross-lingual transfer learning approach relies on the assumption that the English datasets used for fine-tuning are similar enough in structure and content to the Nigerian language datasets. This may not always be the case, and could impact the performance of the transferred model.

The researchers did not provide extensive analysis on the types of errors made by the models or the specific challenges presented by the Nigerian language datasets. Further research into these areas could provide valuable insights for improving reading comprehension in low-resource languages.

Conclusion

The researchers have made a valuable contribution by creating the NaijaRC dataset, which can help drive progress in reading comprehension for three native Nigerian languages. Their use of cross-lingual transfer learning and prompting of large language models provides a solid baseline for future work in this area.

As the field of multilingual natural language processing continues to advance, datasets like NaijaRC will become increasingly important for ensuring that the benefits of these technologies are accessible to speakers of diverse languages, particularly those that have been historically underrepresented in AI research.