Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models

Overview

• Generative large language models (LLMs) like ChatGPT have shown impressive capabilities across various natural language processing (NLP) tasks, including machine translation (MT) and text summarization.
• Recent research has found that while LLMs perform well at the system level for MT quality assessment, they perform poorly at the segment level.
• To address this, the researchers investigated different prompting techniques and proposed a new method called "Error Analysis Prompting" (EAPrompt) that combines Chain-of-Thoughts and Error Analysis.
• EAPrompt aims to produce explainable and reliable MT evaluations at both the system and segment level, emulating the Multidimensional Quality Metrics (MQM) framework for human evaluation.

Plain English Explanation

Large language models (LLMs) like ChatGPT have become incredibly capable at various language tasks, including translating between languages. Researchers have found that these models can accurately evaluate the overall quality of machine translations, but they struggle to provide detailed feedback on specific problems within the translations.

To address this, the researchers in this study tried out different ways of asking the LLMs to assess translations. They proposed a new technique called "Error Analysis Prompting" (EAPrompt) that combines two existing methods: Chain-of-Thoughts and Error Analysis. This new approach aims to make the LLMs' evaluations more human-like, with clear explanations of the types of errors they find, similar to how human experts would assess translations.

The researchers tested EAPrompt on a variety of LLMs and found that it performed well at both the overall system level and the individual sentence level. EAPrompt was able to reliably identify major errors versus minor ones, and the distribution of errors it found matched up with what human evaluators using the standard Multidimensional Quality Metrics (MQM) framework would find.

This suggests that EAPrompt could be a powerful tool for getting high-quality, explainable feedback on machine translations from large language models, which could be useful for improving translation systems and for educational purposes.

Technical Explanation

The researchers investigated different prompting techniques to improve the performance of LLMs on machine translation (MT) quality assessment. They proposed a new method called "Error Analysis Prompting" (EAPrompt) that combines two existing techniques: Chain-of-Thoughts (Wei et al., 2022) and Error Analysis (Lu et al., 2023).

EAPrompt is designed to emulate the Multidimensional Quality Metrics (MQM) framework used in human evaluation of MT. The technique involves asking the LLM to first provide an overall quality score for the translation, then to analyze the specific errors it identifies and classify them into different categories (e.g., major vs. minor errors).

The researchers tested EAPrompt on various LLM architectures, including GPT-3, PaLM, and FLAN, using data from the WMT22 metrics shared task. They found that EAPrompt outperformed other prompting techniques and achieved state-of-the-art performance at both the system and segment level for MT quality assessment.

Further analysis showed that EAPrompt was able to effectively distinguish major errors from minor ones, and the distribution of errors it identified closely matched that of human evaluations using the MQM framework. These findings suggest that EAPrompt can serve as a reliable, human-like evaluator for machine translations.

Critical Analysis

The researchers provide a thorough evaluation of their EAPrompt technique and demonstrate its effectiveness in improving LLM performance on MT quality assessment. However, there are a few potential limitations and areas for further research:

1. The study focuses on evaluating EAPrompt on a specific dataset from the WMT22 metrics shared task. It would be valuable to test the technique on a broader range of MT datasets and language pairs to further validate its generalizability.

2. The researchers acknowledge that the current implementation of EAPrompt relies on predefined error categories, which may not capture the full nuance and complexity of human evaluation. Exploring more flexible, open-ended error analysis prompting could be an interesting area for future research.

3. While EAPrompt aims to emulate human-like evaluation, it is still based on an LLM and may not fully capture the contextual understanding and reasoning that human experts bring to the task. Further research is needed to understand the limitations and potential biases of LLM-based evaluation techniques.

Overall, the EAPrompt approach represents a promising step forward in leveraging large language models for more reliable and explainable MT evaluation. The researchers have demonstrated the potential of this technique, but continued exploration and refinement will be important to fully realize its benefits.

Conclusion

This research proposes a novel prompting technique called "Error Analysis Prompting" (EAPrompt) that aims to improve the performance of large language models (LLMs) on machine translation (MT) quality assessment. By combining Chain-of-Thoughts and Error Analysis, EAPrompt is able to produce explainable and reliable evaluations at both the system and segment level, emulating the Multidimensional Quality Metrics (MQM) framework used in human evaluation.

The experimental results validate the effectiveness of EAPrompt across different LLM architectures, highlighting its potential as a powerful tool for assessing and improving MT systems. This research contributes to the ongoing efforts to leverage the capabilities of large language models for various NLP applications, while also underscoring the importance of developing techniques that can provide meaningful, human-like feedback and explanations.