Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Dataset

Overview

• Open-ended questions that require students to provide detailed responses are a valuable tool for formative assessment, as they give teachers a better understanding of what students know and don't know.
• However, grading these open-ended questions can be time-consuming, leading teachers to use simpler question formats or conduct fewer formative assessments.
• Researchers have long been interested in automating the grading of short-answer responses (Can Large Language Models Make the Grade?), but previous approaches have been technically complex, limiting their use in formative assessment contexts.
• The newest generation of Large Language Models (LLMs) may make grading short-answer questions more feasible.

Plain English Explanation

This research paper explores the potential of using the latest LLMs, such as GPT-4, to automatically grade short-answer questions for formative assessments. Formative assessments are tests or quizzes that teachers use throughout the learning process to gauge what students have understood and where they may be struggling.

Open-ended questions, where students have to provide multi-word answers, are a common type of formative assessment. These questions give teachers more detailed insights into students' knowledge compared to simple multiple-choice or true/false questions. However, grading these open-ended responses can be very time-consuming for teachers.

Researchers have tried to solve this problem by developing systems to automatically grade short-answer questions, but these systems have often been technically complex, making them difficult for teachers to use.

The researchers in this paper believed that the newest generation of powerful language models, like GPT-4, might be able to grade short-answer questions more effectively and with less complexity. To test this, they:

1. Created a new dataset of short-answer reading comprehension questions from students in Ghana, providing a new context to evaluate LLMs beyond the typical data from high-income North American countries.
2. Tested how well various LLM configurations could grade the short-answer responses compared to expert human raters.

The results showed that GPT-4, with minimal prompting, was able to grade the short-answer responses extremely well, nearly matching the performance of expert human raters. This suggests that advanced LLMs could potentially be used by teachers to efficiently grade formative assessments, saving time and allowing them to gather more detailed insights into student learning.

Technical Explanation

The researchers first created a novel dataset of short-answer reading comprehension questions. This dataset was drawn from a set of reading assessments conducted with over 150 students in Ghana, providing a new context to evaluate LLMs beyond the typical North American data they are usually trained on.

Next, the researchers empirically evaluated how well various configurations of generative LLMs, such as GPT-4, could grade the student short-answer responses compared to expert human raters. They used metrics like Quadratic Weighted Kappa (QWK) and F1 score to measure the performance of the LLMs.

The findings showed that GPT-4, with minimal prompt engineering, performed extremely well on grading the novel dataset, reaching a QWK of 0.92 and an F1 score of 0.89. This performance was near-parity with expert human raters.

Critical Analysis

The researchers acknowledge that their dataset, while novel, is relatively small, and further evaluation on larger and more diverse datasets would be beneficial. Additionally, they note that the performance of the LLMs may depend on the specific prompting and fine-tuning approaches used, and more research is needed to explore the optimal configurations for ASAG tasks.

One potential concern not addressed in the paper is the risk of LLMs perpetuating biases present in the training data, which could lead to unfair or inaccurate grading for certain student populations. Further research is needed to explore the effectiveness of LLMs as annotators and ensure their fairness and reliability in educational assessment contexts.

Conclusion

This research provides promising evidence that the newest generation of LLMs, such as GPT-4, could be effectively used to grade short-answer questions for formative assessments, potentially saving teachers time and allowing them to gather more detailed insights into student learning. While further research is needed to address potential limitations and concerns, this work represents an important step forward in the automated grading of proficiency-based assessments using advanced language models.