Can GPT-4 do L2 analytic assessment?

Overview

• Automated essay scoring (AES) has been used for decades to evaluate second language (L2) proficiency in educational contexts.
• While holistic scoring in AES has advanced to match or exceed human performance, analytic scoring still faces issues inherited from the human scoring process.
• The recent introduction of large language models presents new opportunities for automating the evaluation of specific aspects of L2 writing proficiency.

Plain English Explanation

Automated essay scoring (AES) is a technology that has been used for a long time to assess how well people can write in a second language (L2) for educational purposes. AES can provide holistic scores that are as good as or better than scores given by human experts. However, the more detailed "analytic" scores that break down different aspects of writing proficiency still have some problems, as they inherit limitations from the human scoring process.

The development of large language models, such as GPT-4, has opened up new possibilities for automatically evaluating specific components of L2 writing ability. In this study, the researchers wanted to see if they could use GPT-4 in a "zero-shot" way (without any additional training) to extract detailed information about the underlying analytic components of L2 writing proficiency, based on a publicly available dataset that already had holistic scores.

Technical Explanation

The researchers performed a series of experiments using the GPT-4 large language model in a zero-shot fashion on a publicly available dataset. This dataset contained essays annotated with holistic scores based on the Common European Framework of Reference (CEFR) for language proficiency.

The goal was to see if GPT-4 could automatically predict analytic scores for different aspects of L2 writing proficiency, without any additional training or fine-tuning. The researchers looked at how the GPT-4-generated analytic scores correlated with various features associated with the individual proficiency components.

The results showed significant correlations between the automatically predicted analytic scores and multiple features linked to the individual writing proficiency components. This suggests that large language models like GPT-4 have the potential to be used for a more detailed, automated evaluation of L2 writing skills, going beyond just holistic scoring.

Critical Analysis

The paper presents an interesting exploration of using large language models, such as GPT-4, to automate the assessment of specific aspects of second language writing proficiency. This could be a valuable tool for educational applications, as it could provide more detailed and nuanced feedback to students and teachers.

However, the research is still in a preliminary stage, and the authors acknowledge several limitations. For example, the study was conducted on a single dataset, and the performance of the model may vary depending on the quality and characteristics of the training data. Further research is needed to evaluate the model's performance across a wider range of datasets and assessment frameworks.

Additionally, the zero-shot approach used in this study may not capture the full potential of large language models for this task. Exploring fine-tuning or other adaptation techniques could potentially improve the model's ability to accurately assess specific proficiency components.

Conclusion

This paper demonstrates the potential of large language models, such as GPT-4, to automate the evaluation of second language writing proficiency in a more detailed, analytic way, going beyond just holistic scoring. The significant correlations between the model's predicted analytic scores and various features of writing proficiency are a promising result.

However, further research is needed to fully understand the capabilities and limitations of this approach, as well as to explore ways to optimize the model's performance. If successful, this technology could lead to more personalized and effective feedback for language learners, ultimately supporting their development of writing skills in a second language.