TAMIGO: Empowering Teaching Assistants using LLM-assisted viva and code assessment in an Advanced Computing Class
0
Sign in to get full access
Overview
- The paper "TAMIGO: Empowering Teaching Assistants using LLM-assisted viva and code assessment in an Advanced Computing Class" explores the use of large language models (LLMs) to assist teaching assistants in an advanced computing class.
- The researchers developed TAMIGO, a system that leverages LLMs for viva (oral examination) questions and code assessment.
- TAMIGO aims to empower teaching assistants by automating some of their tasks, allowing them to focus on more high-level activities.
Plain English Explanation
The paper looks at using advanced AI models, called large language models (LLMs), to help teaching assistants in an advanced computer science class. The researchers created a system called TAMIGO that uses these LLMs for two key tasks:
-
Generating viva questions: Vivas are oral exams where students are asked questions about the material. TAMIGO can automatically generate relevant and insightful questions for the teaching assistants to use during these vivas.
-
Assessing student code: Students in the class also have to write code as part of their assignments. TAMIGO can analyze the students' code and provide feedback to the teaching assistants, helping them grade the code more efficiently.
The goal of TAMIGO is to empower the teaching assistants by taking care of some of the more repetitive and time-consuming parts of their job. This allows the teaching assistants to focus more on higher-level tasks, such as providing personalized guidance and support to the students.
Technical Explanation
The researchers developed the TAMIGO system, which integrates large language models to assist teaching assistants in an advanced computing class. TAMIGO has two main components:
-
LLM-Powered Viva Question Generation: TAMIGO uses an LLM to automatically generate relevant and thoughtful questions for the viva (oral exam) sessions. The LLM is trained on course materials and past viva questions to generate high-quality questions that assess the students' understanding.
-
LLM-Powered Code Assessment: TAMIGO also leverages an LLM to analyze and provide feedback on students' code submissions. The LLM is trained to identify common coding issues, provide suggestions for improvement, and evaluate the overall quality of the code.
The researchers conducted a study to evaluate the effectiveness of TAMIGO in the advanced computing class. They found that the teaching assistants were able to save time and focus more on higher-level tasks, while the students received timely and valuable feedback on their performance.
Critical Analysis
The paper provides a promising approach to leveraging LLMs to support teaching assistants in an advanced computing class. However, the research has a few potential limitations:
-
The study was conducted in a single course, and the findings may not generalize to other educational contexts or subject areas. Further research is needed to explore the broader applicability of the TAMIGO system.
-
The paper does not delve deeply into the specific LLM architectures or training approaches used. More technical details about the models and their training would help readers better understand the system's capabilities and limitations.
-
The study primarily focuses on the benefits to teaching assistants, but it would be valuable to also assess the impact on student learning outcomes and perceptions. Understanding how TAMIGO affects the overall educational experience would provide a more comprehensive evaluation.
Despite these potential limitations, the TAMIGO system represents an interesting and innovative approach to leveraging LLMs in higher education. As AI technology continues to advance, exploring ways to integrate it into the classroom to support both instructors and students is an important area of research.
Conclusion
The "TAMIGO: Empowering Teaching Assistants using LLM-assisted viva and code assessment in an Advanced Computing Class" paper explores the use of large language models to assist teaching assistants in an advanced computing course. The TAMIGO system automates the generation of viva questions and the assessment of student code, allowing teaching assistants to focus on more high-level tasks.
The research findings suggest that TAMIGO can be an effective tool for supporting teaching assistants and potentially improving the overall educational experience. As AI continues to advance, integrating these technologies into the classroom in thoughtful and responsible ways represents an exciting avenue for future work in the field of education.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
TAMIGO: Empowering Teaching Assistants using LLM-assisted viva and code assessment in an Advanced Computing Class
Anishka IIITD, Diksha Sethi, Nipun Gupta, Shikhar Sharma, Srishti Jain, Ujjwal Singhal, Dhruv Kumar
Large Language Models (LLMs) have significantly transformed the educational landscape, offering new tools for students, instructors, and teaching assistants. This paper investigates the application of LLMs in assisting teaching assistants (TAs) with viva and code assessments in an advanced computing class on distributed systems in an Indian University. We develop TAMIGO, an LLM-based system for TAs to evaluate programming assignments. For viva assessment, the TAs generated questions using TAMIGO and circulated these questions to the students for answering. The TAs then used TAMIGO to generate feedback on student answers. For code assessment, the TAs selected specific code blocks from student code submissions and fed it to TAMIGO to generate feedback for these code blocks. The TAMIGO-generated feedback for student answers and code blocks was used by the TAs for further evaluation. We evaluate the quality of LLM-generated viva questions, model answers, feedback on viva answers, and feedback on student code submissions. Our results indicate that LLMs are highly effective at generating viva questions when provided with sufficient context and background information. However, the results for LLM-generated feedback on viva answers were mixed; instances of hallucination occasionally reduced the accuracy of feedback. Despite this, the feedback was consistent, constructive, comprehensive, balanced, and did not overwhelm the TAs. Similarly, for code submissions, the LLM-generated feedback was constructive, comprehensive and balanced, though there was room for improvement in aligning the feedback with the instructor-provided rubric for code evaluation. Our findings contribute to understanding the benefits and limitations of integrating LLMs into educational settings.
Read more7/25/2024
0
Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
Wenhan Lyu (Rachel), Yimeng Wang (Rachel), Tingting (Rachel), Chung, Yifan Sun, Yixuan Zhang
The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.
Read more5/6/2024
🛸
0
Analyzing LLM Usage in an Advanced Computing Class in India
Anupam Garg, Aryaman Raina, Aryan Gupta, Jaskaran Singh, Manav Saini, Prachi Iiitd, Ronit Mehta, Rupin Oberoi, Sachin Sharma, Samyak Jain, Sarthak Tyagi, Utkarsh Arora, Dhruv Kumar
This study examines the use of large language models (LLMs) by undergraduate and graduate students for programming assignments in advanced computing classes. Unlike existing research, which primarily focuses on introductory classes and lacks in-depth analysis of actual student-LLM interactions, our work fills this gap. We conducted a comprehensive analysis involving 411 students from a Distributed Systems class at an Indian university, where they completed three programming assignments and shared their experiences through Google Form surveys. Our findings reveal that students leveraged LLMs for a variety of tasks, including code generation, debugging, conceptual inquiries, and test case creation. They employed a spectrum of prompting strategies, ranging from basic contextual prompts to advanced techniques like chain-of-thought prompting and iterative refinement. While students generally viewed LLMs as beneficial for enhancing productivity and learning, we noted a concerning trend of over-reliance, with many students submitting entire assignment descriptions to obtain complete solutions. Given the increasing use of LLMs in the software industry, our study highlights the need to update undergraduate curricula to include training on effective prompting strategies and to raise awareness about the benefits and potential drawbacks of LLM usage in academic settings.
Read more7/29/2024
0
How Do Students Interact with an LLM-powered Virtual Teaching Assistant in Different Educational Settings?
Pratyusha Maiti, Ashok K. Goel
Jill Watson, a virtual teaching assistant powered by LLMs, answers student questions and engages them in extended conversations on courseware provided by the instructors. In this paper, we analyze student interactions with Jill across multiple courses and colleges, focusing on the types and complexity of student questions based on Bloom's Revised Taxonomy and tool usage patterns. We find that, by supporting a wide range of cognitive demands, Jill encourages students to engage in sophisticated, higher-order cognitive questions. However, the frequency of usage varies significantly across deployments, and the types of questions asked depend on course-specific contexts. These findings pave the way for future work on AI-driven educational tools tailored to individual learning styles and course structure, potentially enhancing both the teaching and learning experience in classrooms.
Read more7/29/2024