Graded Relevance Scoring of Written Essays with Dense Retrieval

Overview

• This paper proposes a novel approach for automatically scoring the relevance of student essays, a key aspect of essay quality.
• The method uses dense retrieval encoders to create embeddings that cluster essays by relevance level, allowing a simple 1-nearest neighbor classifier to determine the relevance score.
• The researchers leverage the Contriever unsupervised dense encoder and test their approach on the ASAP++ dataset.
• Their method achieves state-of-the-art performance on task-specific scenarios and performs on par with the best cross-task models.
• The approach also shows promise in reducing labeling costs for practical few-shot scenarios.

Plain English Explanation

Grading student essays can be a time-consuming process for teachers. Automated Essay Scoring aims to automate this task, helping improve student writing skills. While previous research has focused on holistic essay scoring, this paper zeroes in on a specific aspect of essay quality: relevance.

Relevance refers to how well a student stays on topic throughout their essay. The researchers propose a new way to automatically score this relevance. Their approach uses dense retrieval encoders to create embeddings, or numerical representations, of essays at different relevance levels. These embeddings form distinct clusters, with the centroids (average positions) of the clusters representing different relevance levels.

To score a new essay, the researchers simply find the 1-nearest neighbor - the cluster centroid that is closest to the essay's embedding. This gives them the relevance score. The key is that the Contriever encoder they use is trained in an unsupervised way, without needing manually labeled essay data.

When tested on the ASAP++ dataset, this method achieved new state-of-the-art performance for scoring relevance in task-specific scenarios. It also performed on par with the best cross-task models, meaning it can be applied to essays on new topics. Additionally, the approach showed promise in reducing the amount of labeled data needed, which is valuable for practical applications.

Technical Explanation

The paper focuses on automatically scoring the relevance of student essays, a key aspect of essay quality that has received less attention than holistic essay scoring. The researchers propose a novel unsupervised approach that leverages dense retrieval encoders to create essay embeddings.

The core idea is that essays with different relevance levels will form distinct clusters in the embedding space, with the centroids of those clusters representing the different relevance levels. The researchers then use a simple 1-Nearest Neighbor classifier over those centroids to determine the relevance score of a new, unseen essay.

As the unsupervised dense encoder, the researchers use Contriever, which has demonstrated comparable performance to supervised dense retrieval models. They test their approach on both task-specific (training and testing on the same task) and cross-task (testing on unseen tasks) scenarios using the ASAP++ dataset.

In the task-specific scenario, the proposed method establishes a new state-of-the-art performance for relevance scoring. For the cross-task scenario, the extended version of the approach exhibits performance on par with the best existing model.

The researchers also analyze the performance of their method in a few-shot learning setting, where only a small amount of labeled data is available. They find that their approach can significantly reduce the labeling cost while sacrificing only about 10% of its effectiveness.

Critical Analysis

The paper presents a compelling approach to automated essay scoring, focusing on the important but understudied aspect of essay relevance. The use of unsupervised dense encoders like Contriever is an interesting innovation, as it avoids the need for manually labeled essay data.

However, the paper does not delve into potential limitations or caveats of the proposed method. For example, it would be valuable to understand how the approach performs on essays with varying levels of quality or complexity, or how sensitive it is to differences in writing styles or essay prompts.

Additionally, while the researchers test their method on the widely used ASAP++ dataset, it would be helpful to see evaluations on other essay datasets to better understand the generalizability of the approach. Transformer-based joint modelling and generalized contrastive learning techniques could also be interesting avenues for future research in this area.

Overall, the paper presents a novel and promising approach to automated essay scoring, but further research and analysis would be valuable to fully understand its capabilities and limitations.

Conclusion

This paper introduces a novel unsupervised approach for automatically scoring the relevance of student essays, a key aspect of essay quality. By leveraging dense retrieval encoders to create essay embeddings that cluster by relevance level, the researchers demonstrate state-of-the-art performance on task-specific scenarios and on-par results for cross-task settings.

The method's ability to perform well in few-shot learning scenarios, where labeled data is scarce, is particularly promising for practical applications of automated essay scoring. While the paper could benefit from further analysis of limitations and generalizability, it represents an interesting contribution to the field of automated essay scoring.