Enhancing Student Feedback Using Predictive Models in Visual Literacy Courses
0
Sign in to get full access
Overview
- This paper presents a study on enhancing student feedback using predictive models in visual literacy courses.
- The researchers used a Naive Bayes model to automatically assess student peer reviews of visual communication assignments.
- The goal was to provide students with more timely and detailed feedback to improve their visual communication skills.
Plain English Explanation
The researchers wanted to find a way to give students more helpful feedback on their visual communication assignments, such as graphic designs or multimedia projects. Typically, instructors would have students peer review each other's work, but this can be time-consuming for the instructor to provide detailed feedback on all the peer reviews.
To address this, the researchers developed a machine learning model that could automatically analyze the student peer reviews and provide a prediction of the quality of the visual communication project. This allowed the instructors to quickly identify projects that needed more feedback or support, and give students more detailed guidance to improve their skills.
The model used a Naive Bayes approach, which is a common technique for text classification. The researchers trained the model on a dataset of previous student peer reviews and project quality assessments. Then, they could use the model to analyze new peer reviews and provide a predicted score for each project.
This allowed the instructors to focus their time on providing more personalized feedback to students who needed extra support, rather than having to review every single peer review. The goal was to help students improve their visual communication abilities and prepare them for careers that require strong visual skills.
Technical Explanation
The researchers developed a Naive Bayes model to automatically assess student peer reviews of visual communication assignments. They trained the model on a dataset of previous peer reviews and instructor assessments of student work.
The Naive Bayes approach works by calculating the probability of a given peer review text belonging to different quality categories (e.g. high, medium, low). The model learns the relationships between the text features and the quality labels from the training data.
When presented with a new peer review, the model can then predict the likely quality of the corresponding visual communication project. This allows instructors to quickly identify which projects need more detailed feedback and support, rather than having to manually review every peer review.
The researchers evaluated the performance of their predictive model on a held-out test set of peer reviews. They found the model was able to accurately predict the quality categories, with an average F1-score of 0.85. This suggests the approach is effective at automating the analysis of student peer reviews in visual literacy courses.
Critical Analysis
One limitation of the study is that it was conducted in a single course, so the generalizability to other visual literacy courses is uncertain. The researchers acknowledged this and suggested further testing in different educational contexts.
Additionally, the study did not explore the impacts of providing the automated feedback predictions to students. It's possible that simply showing students the model's quality assessment could influence their perceptions and willingness to engage in the peer review process. Further research is needed to understand the behavioral effects.
Another potential concern is the interpretability of the Naive Bayes model's predictions. While the model achieved high accuracy, the specific reasons behind its quality assessments may not be transparent to students or instructors. Exploring more explainable AI techniques could help address this issue.
Overall, this study presents a promising approach to automating the analysis of student peer reviews in visual literacy courses. However, additional research is needed to fully understand the impacts and limitations of this predictive modeling technique.
Conclusion
This paper demonstrates how machine learning models can be leveraged to enhance the feedback and support provided to students in visual literacy courses. By automatically assessing student peer reviews, instructors can focus their time on giving more personalized guidance to those who need it most.
The use of a Naive Bayes model allows for relatively accurate predictions of project quality based on the peer review text. This could lead to improvements in students' visual communication skills and better prepare them for careers that require strong visual abilities.
Further research is needed to explore the behavioral impacts of providing automated feedback predictions, as well as ways to improve the interpretability of the model's assessments. But overall, this study offers a promising approach to enhancing student learning and support in visual literacy courses.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Enhancing Student Feedback Using Predictive Models in Visual Literacy Courses
Alon Friedman, Kevin Hawley, Paul Rosen, Md Dilshadur Rahman
Peer review is a popular feedback mechanism in higher education that actively engages students and provides researchers with a means to assess student engagement. However, there is little empirical support for the durability of peer review, particularly when using data predictive modeling to analyze student comments. This study uses Naive Bayes modeling to analyze peer review data obtained from an undergraduate visual literacy course over five years. We expand on the research of Friedman and Rosen and Beasley et al. by focusing on the Naive Bayes model of students' remarks. Our findings highlight the utility of Naive Bayes modeling, particularly in the analysis of student comments based on parts of speech, where nouns emerged as the prominent category. Additionally, when examining students' comments using the visual peer review rubric, the lie factor emerged as the predominant factor. Comparing Naive Bayes model to Beasley's approach, we found both help instructors map directions taken in the class, but the Naive Bayes model provides a more specific outline for forecasting with a more detailed framework for identifying core topics within the course, enhancing the forecasting of educational directions. Through the application of the Holdout Method and $mathrm{k}$-fold cross-validation with continuity correction, we have validated the model's predictive accuracy, underscoring its effectiveness in offering deep insights into peer review mechanisms. Our study findings suggest that using predictive modeling to assess student comments can provide a new way to better serve the students' classroom comments on their visual peer work. This can benefit courses by inspiring changes to course content, reinforcement of course content, modification of projects, or modifications to the rubric itself.
Read more5/27/2024
0
Eliciting Informative Text Evaluations with Large Language Models
Yuxuan Lu, Shengwei Xu, Yichi Zhang, Yuqing Kong, Grant Schoenebeck
Peer prediction mechanisms motivate high-quality feedback with provable guarantees. However, current methods only apply to rather simple reports, like multiple-choice or scalar numbers. We aim to broaden these techniques to the larger domain of text-based reports, drawing on the recent developments in large language models. This vastly increases the applicability of peer prediction mechanisms as textual feedback is the norm in a large variety of feedback channels: peer reviews, e-commerce customer reviews, and comments on social media. We introduce two mechanisms, the Generative Peer Prediction Mechanism (GPPM) and the Generative Synopsis Peer Prediction Mechanism (GSPPM). These mechanisms utilize LLMs as predictors, mapping from one agent's report to a prediction of her peer's report. Theoretically, we show that when the LLM prediction is sufficiently accurate, our mechanisms can incentivize high effort and truth-telling as an (approximate) Bayesian Nash equilibrium. Empirically, we confirm the efficacy of our mechanisms through experiments conducted on two real datasets: the Yelp review dataset and the ICLR OpenReview dataset. We highlight the results that on the ICLR dataset, our mechanisms can differentiate three quality levels -- human-written reviews, GPT-4-generated reviews, and GPT-3.5-generated reviews in terms of expected scores. Additionally, GSPPM penalizes LLM-generated reviews more effectively than GPPM.
Read more9/4/2024
💬
0
Leveraging Large Language Models for Actionable Course Evaluation Student Feedback to Lecturers
Mike Zhang, Euan D Lindsay, Frederik Bode Thorbensen, Danny B{o}gsted Poulsen, Johannes Bjerva
End of semester student evaluations of teaching are the dominant mechanism for providing feedback to academics on their teaching practice. For large classes, however, the volume of feedback makes these tools impractical for this purpose. This paper explores the use of open-source generative AI to synthesise factual, actionable and appropriate summaries of student feedback from these survey responses. In our setup, we have 742 student responses ranging over 75 courses in a Computer Science department. For each course, we synthesise a summary of the course evaluations and actionable items for the instructor. Our results reveal a promising avenue for enhancing teaching practices in the classroom setting. Our contribution lies in demonstrating the feasibility of using generative AI to produce insightful feedback for teachers, thus providing a cost-effective means to support educators' development. Overall, our work highlights the possibility of using generative AI to produce factual, actionable, and appropriate feedback for teachers in the classroom setting.
Read more7/2/2024
💬
0
Grading Massive Open Online Courses Using Large Language Models
Shahriar Golchin, Nikhil Garuda, Christopher Impey, Matthew Wenger
Massive open online courses (MOOCs) offer free education globally to anyone with a computer and internet access. Despite this democratization of learning, the massive enrollment in these courses makes it impractical for one instructor to assess every student's writing assignment. As a result, peer grading, often guided by a straightforward rubric, is the method of choice. While convenient, peer grading often falls short in terms of reliability and validity. In this study, we explore the feasibility of using large language models (LLMs) to replace peer grading in MOOCs. Specifically, we use two LLMs, GPT-4 and GPT-3.5, across three MOOCs: Introductory Astronomy, Astrobiology, and the History and Philosophy of Astronomy. To instruct LLMs, we use three different prompts based on the zero-shot chain-of-thought (ZCoT) prompting technique: (1) ZCoT with instructor-provided correct answers, (2) ZCoT with both instructor-provided correct answers and rubrics, and (3) ZCoT with instructor-provided correct answers and LLM-generated rubrics. Tested on 18 settings, our results show that ZCoT, when augmented with instructor-provided correct answers and rubrics, produces grades that are more aligned with those assigned by instructors compared to peer grading. Finally, our findings indicate a promising potential for automated grading systems in MOOCs, especially in subjects with well-defined rubrics, to improve the learning experience for millions of online learners worldwide.
Read more6/18/2024