Scalable Two-Minute Feedback: Digital, Lecture-Accompanying Survey as a Continuous Feedback Instrument
2310.19334
0
0
🧠
Abstract
Detailed feedback on courses and lecture content is essential for their improvement and also serves as a tool for reflection. However, feedback methods are often only used sporadically, especially in mass courses, because collecting and analyzing feedback in a timely manner is often a challenge for teachers. Moreover, the current situation of the students or the changing workload during the semester are usually not taken into account either. For a holistic investigation, the article used a digital survey format as formative feedback which attempts to measure student stress in a quantitative part and to address the participants' reflection in a qualitative part, as well as to collect general suggestions for improvement (based on the so-called One-Minute Paper) at two educational institutions. The feedback during the semester is evaluated qualitatively and discussed on a meta-level and special features (e.g. reflections on student work ethic or other courses) are addressed. The results show a low, but constant rate of feedback. Responses mostly cover topics of the lecture content or organizational aspects and were intensively used to report issues within the lecture. In addition, artificial intelligence (AI) support in the form of a large language model was tested and showed promising results in summarizing the open-ended responses for the teacher. Finally, the experiences from the lecturers are reflected upon and the results as well as possibilities for improvement are discussed.
Get summaries of the top AI research delivered straight to your inbox:
Overview
- The paper discusses the importance of feedback for improving courses and lectures, but notes that collecting and analyzing feedback can be challenging, especially in large classes.
- To address this, the paper used a digital survey format to gather both quantitative and qualitative feedback from students at two educational institutions.
- The feedback was evaluated qualitatively and discussed on a meta-level, and the researchers also tested the use of an artificial intelligence (AI) large language model to summarize the open-ended responses.
- The results show a low but consistent rate of feedback, with responses mostly covering lecture content or organizational aspects, and the AI support showed promising results in summarizing the feedback.
Plain English Explanation
Feedback from students is essential for improving courses and lectures, as it helps instructors understand what's working well and what needs to be changed. However, collecting and analyzing this feedback can be difficult, especially in large classes where there are many students.
The researchers in this study tried a different approach. They used a digital survey that asked students both quantitative questions about their stress levels, and open-ended questions to get their reflections and suggestions for improvement. This two-part format allowed them to gather both numerical data and more detailed, qualitative feedback.
The researchers found that while the overall rate of feedback was low, the responses they did receive were useful, covering topics like the lecture content and course organization. Interestingly, they also tested using an AI system to help summarize the open-ended feedback, and the results were promising.
Overall, this study shows that getting regular, detailed feedback from students can be challenging, but using a mixed-method approach and incorporating AI tools can help make the process more manageable for instructors. By better understanding how students are experiencing a course, instructors can make changes to improve the learning experience.
Technical Explanation
The study used a digital survey format to collect both quantitative and qualitative feedback from students at two educational institutions. The quantitative portion aimed to measure student stress levels, while the qualitative part asked for open-ended reflections and suggestions for improvement, similar to a One-Minute Paper.
The feedback was evaluated qualitatively, with the researchers looking for common themes and issues that were raised. They also discussed the feedback on a meta-level, considering factors like student work ethic and connections to other courses.
In addition, the researchers tested the use of an AI large language model to automatically summarize the open-ended responses. This AI-based approach showed promising results in helping the instructors quickly understand the key points raised by students.
The overall findings indicate a low but consistent rate of student feedback, with responses typically covering lecture content and organizational aspects of the courses. The researchers reflected on their experiences and discussed potential avenues for further improving the feedback process, such as better integrating the AI summarization into the instructors' workflow.
Critical Analysis
The paper highlights an important challenge faced by many instructors - how to effectively collect and use student feedback to improve their courses. The researchers' approach of using a mixed-method digital survey is a reasonable attempt to address this, as it allows for both quantitative and qualitative data to be gathered.
However, the relatively low response rate suggests that more work may be needed to encourage students to consistently provide feedback. The researchers could have explored potential reasons for the low engagement, such as students' perceptions of the usefulness of the feedback process or concerns about anonymity.
Additionally, while the AI-based summarization showed promise, the paper does not provide a thorough evaluation of its accuracy or usefulness from the instructors' perspective. Further research could delve deeper into the practical implementation and impact of this AI support.
Overall, the paper provides a valuable starting point for thinking about how to improve feedback practices in educational settings. By continuing to experiment and refine these methods, instructors can better understand and respond to the needs and experiences of their students.
Conclusion
This study demonstrates the challenges of collecting and using student feedback to improve courses and lectures, but also highlights some promising approaches. The use of a mixed-method digital survey and the integration of AI tools show potential for making the feedback process more manageable and effective for instructors.
While the overall response rate was low, the qualitative feedback that was gathered proved useful in identifying areas for improvement and reflecting on the student experience. As educational institutions continue to grapple with how to best support student learning, strategies like those explored in this paper can help instructors stay attuned to their students' needs and make data-driven changes to enhance the quality of their courses.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
How Well Can You Articulate that Idea? Insights from Automated Formative Assessment
Mahsa Sheikhi Karizaki, Dana Gnesdilow, Sadhana Puntambekar, Rebecca J. Passonneau
0
0
Automated methods are becoming increasingly integrated into studies of formative feedback on students' science explanation writing. Most of this work, however, addresses students' responses to short answer questions. We investigate automated feedback on students' science explanation essays, where students must articulate multiple ideas. Feedback is based on a rubric that identifies the main ideas students are prompted to include in explanatory essays about the physics of energy and mass, given their experiments with a simulated roller coaster. We have found that students generally improve on revised versions of their essays. Here, however, we focus on two factors that affect the accuracy of the automated feedback. First, we find that the main ideas in the rubric differ with respect to how much freedom they afford in explanations of the idea, thus explanation of a natural law is relatively constrained. Students have more freedom in how they explain complex relations they observe in their roller coasters, such as transfer of different forms of energy. Second, by tracing the automated decision process, we can diagnose when a student's statement lacks sufficient clarity for the automated tool to associate it more strongly with one of the main ideas above all others. This in turn provides an opportunity for teachers and peers to help students reflect on how to state their ideas more clearly.
4/19/2024
New!Beyond Repetition: The Role of Varied Questioning and Feedback in Knowledge Generalization
Gautam Yadav, Paulo F. Carvalho, Elizabeth A. McLaughlin, Kenneth R. Koedinger
0
0
This study examines the effects of question type and feedback on learning outcomes in a hybrid graduate-level course. By analyzing data from 32 students over 30,198 interactions, we assess the efficacy of unique versus repeated questions and the impact of feedback on student learning. The findings reveal students demonstrate significantly better knowledge generalization when encountering unique questions compared to repeated ones, even though they perform better with repeated opportunities. Moreover, we find that the timing of explanatory feedback is a more robust predictor of learning outcomes than the practice opportunities themselves. These insights suggest that educational practices and technological platforms should prioritize a variety of questions to enhance the learning process. The study also highlights the critical role of feedback; opportunities preceding feedback are less effective in enhancing learning.
5/17/2024
Investigating Automatic Scoring and Feedback using Large Language Models
Gloria Ashiya Katuka, Alexander Gain, Yen-Yun Yu
0
0
Automatic grading and feedback have been long studied using traditional machine learning and deep learning techniques using language models. With the recent accessibility to high performing large language models (LLMs) like LLaMA-2, there is an opportunity to investigate the use of these LLMs for automatic grading and feedback generation. Despite the increase in performance, LLMs require significant computational resources for fine-tuning and additional specific adjustments to enhance their performance for such tasks. To address these issues, Parameter Efficient Fine-tuning (PEFT) methods, such as LoRA and QLoRA, have been adopted to decrease memory and computational requirements in model fine-tuning. This paper explores the efficacy of PEFT-based quantized models, employing classification or regression head, to fine-tune LLMs for automatically assigning continuous numerical grades to short answers and essays, as well as generating corresponding feedback. We conducted experiments on both proprietary and open-source datasets for our tasks. The results show that prediction of grade scores via finetuned LLMs are highly accurate, achieving less than 3% error in grade percentage on average. For providing graded feedback fine-tuned 4-bit quantized LLaMA-2 13B models outperform competitive base models and achieve high similarity with subject matter expert feedback in terms of high BLEU and ROUGE scores and qualitatively in terms of feedback. The findings from this study provide important insights into the impacts of the emerging capabilities of using quantization approaches to fine-tune LLMs for various downstream tasks, such as automatic short answer scoring and feedback generation at comparatively lower costs and latency.
5/2/2024
Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences
John Stamper, Ruiwei Xiao, Xinying Hou
0
0
The field of Artificial Intelligence in Education (AIED) focuses on the intersection of technology, education, and psychology, placing a strong emphasis on supporting learners' needs with compassion and understanding. The growing prominence of Large Language Models (LLMs) has led to the development of scalable solutions within educational settings, including generating different types of feedback in Intelligent Tutoring Systems. However, the approach to utilizing these models often involves directly formulating prompts to solicit specific information, lacking a solid theoretical foundation for prompt construction and empirical assessments of their impact on learning. This work advocates careful and caring AIED research by going through previous research on feedback generation in ITS, with emphasis on the theoretical frameworks they utilized and the efficacy of the corresponding design in empirical evaluations, and then suggesting opportunities to apply these evidence-based principles to the design, experiment, and evaluation phases of LLM-based feedback generation. The main contributions of this paper include: an avocation of applying more cautious, theoretically grounded methods in feedback generation in the era of generative AI; and practical suggestions on theory and evidence-based feedback design for LLM-powered ITS.
5/14/2024