DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking

Overview

• This paper presents a simple, generic framework for generating effective distractors (incorrect answer choices) for multiple-choice questions (MCQs) using pre-trained language models.
• Previous methods have struggled with distractor generation, but this approach relies solely on pre-trained models without additional training.
• The framework consists of two stages: candidate generation and candidate selection, outperforming prior techniques.
• Human evaluations confirm the distractors generated by this method are more effective and engaging.

Plain English Explanation

Multiple-choice questions (MCQs) are a common assessment tool, but creating good distractors (incorrect answer choices) for these questions is challenging. Distractor Generation is the process of generating plausible but incorrect options to accompany the correct answer.

This research presents a simple, generic framework that uses Pre-trained Language Models (PLMs) to generate effective distractors without needing additional training. Unlike previous methods, this approach relies only on the language models, not specialized datasets or fine-tuning.

The framework has two key steps:

1. Candidate Generation: The system generates a set of potential distractor options using the language model.
2. Candidate Selection: It then selects the best distractors from the candidate set based on certain criteria.

This two-stage process produces distractors that outperform previous techniques, as confirmed by human evaluations. The distractors are more effective at assessing student understanding and more engaging for test-takers.

Technical Explanation

The researchers introduce a two-stage framework for distractor generation that relies solely on pre-trained language models, rather than requiring additional training or fine-tuning.

Candidate Generation: The system uses the language model to generate a set of potential distractors based on the question stem and correct answer. It does this by prompting the model with the question and answer, then extracting the top-ranked generated text as candidate distractors.

Candidate Selection: From the candidate set, the framework selects the best distractors using a combination of criteria, such as semantic similarity to the correct answer, language model probability, and conciseness. This ensures the distractors are plausible but distinct from the correct answer.

The researchers evaluated their approach on several benchmark datasets for distractor generation, and found that it outperformed previous state-of-the-art methods without any additional training. Human evaluations also confirmed that the distractors generated by this framework were more effective and engaging compared to other techniques.

Critical Analysis

The key strength of this approach is its simplicity and generalizability. By relying solely on pre-trained language models, the framework can be easily applied to different domains and question types without the need for specialized datasets or fine-tuning.

However, the paper acknowledges that the framework has some limitations. The distractor quality is still dependent on the underlying language model, and may not perform as well on highly specialized or technical content. Additionally, the selection criteria used in the framework could be further improved to better capture the nuances of effective distractor generation.

Future research could explore ways to fine-tune the language models or incorporate additional signals (e.g., knowledge graphs, domain-specific information) to enhance the distractor generation process. Evaluating the framework's performance on a wider range of question types and real-world educational assessments would also be valuable.

Conclusion

This research presents a simple, generic framework for generating effective distractors for multiple-choice questions using pre-trained language models. By avoiding the need for specialized datasets or fine-tuning, the approach is highly accessible and can be readily applied to a variety of educational and assessment contexts.

The two-stage framework of candidate generation and selection produces distractors that outperform previous methods, as confirmed by human evaluations. This work demonstrates the potential of leveraging large language models to automate the challenging task of distractor generation, with potential applications in areas like educational technology, test development, and more.