Research on the Application of Large Language Models in Automatic Question Generation: A Case Study of ChatGLM in the Context of High School Information Technology Curriculum

Overview

• This study explores the use of the large language model (LLM) ChatGLM for automatically generating high school information technology exam questions.
• Researchers designed prompts to guide ChatGLM in producing diverse questions, which were then evaluated by domain experts.
• The evaluation focused on factors like alignment with teaching content, embodiment of core competencies, clarity of questions, and teachers' willingness to use them.

Plain English Explanation

The researchers wanted to see how well the ChatGLM language model could generate exam questions for high school information technology classes. They used special prompts to get ChatGLM to create a variety of questions, and then had experts review the questions.

The experts looked at things like how well the questions matched the course material, how well they tested key skills, how clear the questions were, and whether teachers would be willing to use the questions in their classes. The results showed that ChatGLM-generated questions were clearer and teachers were more willing to use them, compared to human-written questions. But there wasn't a big difference in how well the questions matched the course content or tested the right skills.

This suggests that ChatGLM could help make the process of generating exam questions more efficient and reduce the workload for teachers. But the researchers think the model could be improved to maintain the strengths while also improving the fit and relevance of the questions.

Technical Explanation

The researchers used prompt engineering strategies to guide the ChatGLM language model in generating diverse exam questions for high school information technology courses. They then had domain experts evaluate the questions across four dimensions:

1. Hitting: The degree of alignment between the questions and the teaching content.
2. Fitting: How well the questions embodied the core competencies being tested.
3. Clarity: How explicitly and unambiguously the questions were described.
4. Willing to use: The extent to which teachers would be willing to use the questions in their teaching.

The results showed that the ChatGLM-generated questions outperformed human-written questions in terms of clarity and teachers' willingness to use them. However, there was no significant difference in hit rate and fit.

These findings suggest that ChatGLM has the potential to enhance the efficiency of exam question generation and reduce the burden on teachers. However, future research may need to explore further optimizations to the model to maintain high fit and hit rates while improving the clarity and teachers' willingness to use the questions.

Critical Analysis

The researchers acknowledged that while ChatGLM showed promise in generating clear and teacher-friendly exam questions, there is still room for improvement in terms of the fit and relevance of the questions to the course content. They suggested that further optimizations to the prompting strategies or the model itself may be necessary to address these limitations.

Additionally, the study focused on a specific domain (high school information technology) and may not generalize to other subject areas or educational levels. Replicating the study in different contexts would help validate the findings and identify any unique considerations for other subject areas.

It's also worth noting that the evaluation of the questions was done by domain experts, which could introduce some subjectivity. Incorporating more objective measures, such as student performance on the generated questions, could provide additional insights into the quality and effectiveness of the questions.

Conclusion

This study demonstrates the potential of large language models like ChatGLM to enhance the efficiency of exam question generation and reduce the workload for teachers. While the ChatGLM-generated questions showed strengths in clarity and teacher acceptance, there is still room for improvement in maintaining high alignment with course content and core competencies.

Further research and optimization of the prompting strategies or the language model itself could help address these limitations, paving the way for more effective and AI-powered educational assessment systems. By leveraging the capabilities of large language models, educators may be able to streamline the question generation process and focus more on other aspects of teaching and learning.