Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs
0
📈
Sign in to get full access
Overview
- Online programming videos contain a wealth of expert knowledge, but effectively utilizing them for targeted learning can be challenging.
- Unlike direct tutoring, video content lacks tailored guidance, personalized feedback, and interactive engagement necessary for support and monitoring.
- The research paper introduces Tutorly, a JupyterLab Plugin that transforms programming videos into one-on-one tutoring experiences using the cognitive apprenticeship framework.
Plain English Explanation
The paper discusses the challenges of using online programming videos for targeted learning. While these videos are widely popular and full of expert knowledge, they lack the personalized guidance, feedback, and interaction that direct tutoring provides. This can make it difficult for learners to achieve their specific learning goals.
To address this, the researchers developed a tool called Tutorly, a JupyterLab Plugin that aims to transform programming videos into a more interactive, one-on-one tutoring experience. Tutorly uses the cognitive apprenticeship framework to guide learners through the content, allowing them to set personalized goals, engage in "learning-by-doing" with a conversational AI mentor, and receive guidance and feedback tailored to their individual needs.
In a study with 16 participants learning exploratory data analysis from a streamcast, Tutorly was found to significantly improve their performance, increasing their post-test scores from 61.9% to 76.6%. This demonstrates the potential for enhancing programming video learning experiences with LLMs and learner modeling.
Technical Explanation
The paper presents Tutorly, a JupyterLab Plugin that aims to transform programming video learning experiences using the cognitive apprenticeship framework. The key components of Tutorly include:
-
Personalized Learning Goals: Learners can set their own targeted learning goals to guide their engagement with the video content.
-
Conversational LLM-based Mentor Agent: Tutorly provides a virtual mentor agent that engages learners in a conversational, "learning-by-doing" experience, guiding them through the video content.
-
Student Model and Adaptive Guidance: Tutorly maintains a student model that tracks the learner's progress and understanding, and uses this to provide personalized guidance and feedback from the mentor agent.
To evaluate the effectiveness of Tutorly, the researchers conducted a within-subject study with 16 participants learning exploratory data analysis from a streamcast. The results showed that Tutorly significantly improved the participants' performance, increasing their post-test scores from 61.9% to 76.6%.
Critical Analysis
The paper presents a compelling approach to enhancing programming video learning experiences using LLMs and learner modeling. However, the small sample size and single-topic focus of the study may limit the generalizability of the findings.
Additionally, the paper does not provide detailed information on the specific algorithms or techniques used to build the student model and generate the mentor agent's responses. Further research would be needed to understand the underlying mechanisms and potential limitations of these components.
It would also be valuable to explore the long-term impact of Tutorly on learner engagement, retention, and transfer of knowledge to real-world programming tasks. Longitudinal studies or comparisons to other learning interventions could provide deeper insights into the effectiveness of this approach.
Conclusion
The research paper introduces Tutorly, a JupyterLab Plugin that aims to transform programming video learning experiences by leveraging the cognitive apprenticeship framework, LLMs, and learner modeling. The results of a within-subject study demonstrate the potential for Tutorly to significantly improve learners' performance, suggesting that this approach could be a promising way to enhance the effectiveness of online programming video resources.
While the paper provides a solid foundation, further research is needed to explore the generalizability, underlying mechanisms, and long-term impact of this technology. Nonetheless, the work highlights the exciting possibilities for integrating advanced AI techniques, such as LLMs and learner modeling, to create more personalized and effective learning experiences for programming education.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
📈
0
Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs
Wengxi Li, Roy Pea, Nick Haber, Hari Subramonyam
Online programming videos, including tutorials and streamcasts, are widely popular and contain a wealth of expert knowledge. However, effectively utilizing these resources to achieve targeted learning goals can be challenging. Unlike direct tutoring, video content lacks tailored guidance based on individual learning paces, personalized feedback, and interactive engagement necessary for support and monitoring. Our work transforms programming videos into one-on-one tutoring experiences using the cognitive apprenticeship framework. Tutorly, developed as a JupyterLab Plugin, allows learners to (1) set personalized learning goals, (2) engage in learning-by-doing through a conversational LLM-based mentor agent, (3) receive guidance and feedback based on a student model that steers the mentor moves. In a within-subject study with 16 participants learning exploratory data analysis from a streamcast, Tutorly significantly improved their performance from 61.9% to 76.6% based on a post-test questionnaire. Tutorly demonstrates the potential for enhancing programming video learning experiences with LLM and learner modeling.
Read more5/22/2024
0
Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
Wenhan Lyu (Rachel), Yimeng Wang (Rachel), Tingting (Rachel), Chung, Yifan Sun, Yixuan Zhang
The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.
Read more5/6/2024
👨🏫
0
CS1-LLM: Integrating LLMs into CS1 Instruction
Annapurna Vadaparty, Daniel Zingaro, David H. Smith IV, Mounika Padala, Christine Alvarado, Jamie Gorson Benario, Leo Porter
The recent, widespread availability of Large Language Models (LLMs) like ChatGPT and GitHub Copilot may impact introductory programming courses (CS1) both in terms of what should be taught and how to teach it. Indeed, recent research has shown that LLMs are capable of solving the majority of the assignments and exams we previously used in CS1. In addition, professional software engineers are often using these tools, raising the question of whether we should be training our students in their use as well. This experience report describes a CS1 course at a large research-intensive university that fully embraces the use of LLMs from the beginning of the course. To incorporate the LLMs, the course was intentionally altered to reduce emphasis on syntax and writing code from scratch. Instead, the course now emphasizes skills needed to successfully produce software with an LLM. This includes explaining code, testing code, and decomposing large problems into small functions that are solvable by an LLM. In addition to frequent, formative assessments of these skills, students were given three large, open-ended projects in three separate domains (data science, image processing, and game design) that allowed them to showcase their creativity in topics of their choosing. In an end-of-term survey, students reported that they appreciated learning with the assistance of the LLM and that they interacted with the LLM in a variety of ways when writing code. We provide lessons learned for instructors who may wish to incorporate LLMs into their course.
Read more6/26/2024
0
AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails
Sankalan Pal Chowdhury, Vil'em Zouhar, Mrinmaya Sachan
Large Language Models (LLMs) have found several use cases in education, ranging from automatic question generation to essay evaluation. In this paper, we explore the potential of using Large Language Models (LLMs) to author Intelligent Tutoring Systems. A common pitfall of LLMs is their straying from desired pedagogical strategies such as leaking the answer to the student, and in general, providing no guarantees. We posit that while LLMs with certain guardrails can take the place of subject experts, the overall pedagogical design still needs to be handcrafted for the best learning results. Based on this principle, we create a sample end-to-end tutoring system named MWPTutor, which uses LLMs to fill in the state space of a pre-defined finite state transducer. This approach retains the structure and the pedagogy of traditional tutoring systems that has been developed over the years by learning scientists but brings in additional flexibility of LLM-based approaches. Through a human evaluation study on two datasets based on math word problems, we show that our hybrid approach achieves a better overall tutoring score than an instructed, but otherwise free-form, GPT-4. MWPTutor is completely modular and opens up the scope for the community to improve its performance by improving individual modules or using different teaching strategies that it can follow.
Read more4/26/2024