Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT
2405.19132
0
0
š
Abstract
Large Language Models (LLMs) have taken the world by storm, and students are assumed to use related tools at a great scale. In this research paper we aim to gain an understanding of how introductory programming students chat with LLMs and related tools, e.g., ChatGPT-3.5. To address this goal, computing students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT as part of their weekly introductory course exercises. Then students (n=213) submitted their chat protocols (with 2335 prompts in sum) as data basis for this analysis. The data was analyzed w.r.t. the prompts, frequencies, the chats' progress, contents, and other use pattern, which revealed a great variety of interactions, both potentially supportive and concerning. Learning about students' interactions with ChatGPT will help inform and align teaching practices and instructions for future introductory programming courses in higher education.
Create account to get full access
Overview
ā¢ This paper analyzes the chat protocols of novice programmers solving introductory programming tasks with the help of ChatGPT, a large language model.
ā¢ The researchers aim to understand how novice programmers interact with ChatGPT and the types of support the model provides during the problem-solving process.
Plain English Explanation
The researchers in this study wanted to understand how beginner programmers use ChatGPT, a powerful AI assistant, to solve basic programming problems. They analyzed the conversations, or "chat protocols," between the novice programmers and ChatGPT to see how the AI model helps the students during the problem-solving process.
The key idea is to gain insights into how novice programmers, who are just starting to learn how to code, interact with an advanced AI system like ChatGPT. By studying these interactions, the researchers hope to learn more about the types of support and guidance that ChatGPT can provide to help novice programmers develop their programming skills.
Technical Explanation
The researchers conducted an observational study where they had novice programmers solve introductory programming tasks using ChatGPT. They recorded the conversations between the students and the AI model, and then analyzed these "chat protocols" to understand the interactions.
The study focused on several key aspects, including:
- The types of questions and requests the novice programmers made to ChatGPT
- The ways in which ChatGPT responded and provided support
- The overall problem-solving strategies employed by the students with the help of the AI model
By analyzing these interactions, the researchers aimed to shed light on how novice programmers can leverage the capabilities of large language models like ChatGPT to enhance their learning and problem-solving abilities in the context of introductory programming tasks.
Critical Analysis
The study provides valuable insights into the potential benefits and limitations of using ChatGPT to support novice programmers. However, the researchers acknowledge that the study is limited to a small sample size and specific programming tasks.
It would be interesting to see if the findings hold true for a larger and more diverse set of novice programmers and programming problems. Additionally, the researchers suggest that future work could explore the long-term impacts of using ChatGPT on the development of programming skills and problem-solving abilities.
Another potential area for further research is to investigate how the interactions between novice programmers and ChatGPT may differ from those between experienced programmers and the AI model. This could provide a more comprehensive understanding of the role of large language models in programming education and support.
Conclusion
This study offers a glimpse into the interactions between novice programmers and the ChatGPT AI model during the process of solving introductory programming tasks. The findings suggest that ChatGPT can provide valuable support and guidance to beginner coders, but also highlight the need for further research to fully understand the implications and potential limitations of using such AI assistants in programming education.
The insights from this paper contribute to the growing body of research on the applications of large language models in various domains, including coding and programming. As AI systems like ChatGPT become more prevalent, it is important to continue exploring their impact on learning, skill development, and problem-solving in the field of computer science and beyond.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
š
Beyond the Hype: A Cautionary Tale of ChatGPT in the Programming Classroom
Grant Oosterwyk, Pitso Tsibolane, Popyeni Kautondokwa, Ammar Canani
0
0
Due to the proliferation of Large Language Models research and the use of various Artificial Intelligence (AI) tools, the field of information systems (IS) and computer science (CS) has evolved. The use of tools such as ChatGPT to complete various student programming exercises (e.g., in Python) and assignments has gained prominence amongst various academic institutions. However, recent literature has suggested that the use of ChatGPT in academia is problematic and the impact on teaching and learning should be further scrutinized. More specifically, little is known about how ChatGPT can be practically used with code (programming) writing to complete programming exercises amongst IS and CS undergraduate university students. Furthermore, the paper provides insights for academics who teach programming to create more challenging exercises and how to engage responsibly in the use of ChatGPT to promote classroom integrity. In this paper, we used Complex Adaptive Systems (CAS) theory as a theoretical guide to understand the various dynamics through classroom code demonstrations. Using ChatGPT 3.5, we analyzed the various practical programming examples from past IS exercises and compared those with memos created by tutors and lecturers in a university setting. This paper highlights common ways of assessment, programming errors created by ChatGPT and the potential consideration for IS academics to ensure the development of critical programming skills among students.
6/18/2024
š
ChatGPT Is Here to Help, Not to Replace Anybody -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Courses
Bruno Pereira Cipriano, Pedro Alves
0
0
Large Language Models (LLMs) like GPT and Bard are capable of producing code based on textual descriptions, with remarkable efficacy. Such technology will have profound implications for computing education, raising concerns about cheating, excessive dependence, and a decline in computational thinking skills, among others. There has been extensive research on how teachers should handle this challenge but it is also important to understand how students feel about this paradigm shift. In this research, 52 first-year CS students were surveyed in order to assess their views on technologies with code-generation capabilities, both from academic and professional perspectives. Our findings indicate that while students generally favor the academic use of GPT, they don't over rely on it, only mildly asking for its help. Although most students benefit from GPT, some struggle to use it effectively, urging the need for specific GPT training. Opinions on GPT's impact on their professional lives vary, but there is a consensus on its importance in academic practice.
4/29/2024
āØ
Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Ranim Khojah, Mazen Mohamad, Philipp Leitner, Francisco Gomes de Oliveira Neto
0
0
Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user's personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.
5/22/2024
š¬
ChatGPT as an inventor: Eliciting the strengths and weaknesses of current large language models against humans in engineering design
Daniel Nyg{aa}rd Ege, Henrik H. {O}vreb{o}, Vegar Stubberud, Martin Francis Berg, Christer Elverum, Martin Steinert, H{aa}vard Vestad
0
0
This study compares the design practices and performance of ChatGPT 4.0, a large language model (LLM), against graduate engineering students in a 48-hour prototyping hackathon, based on a dataset comprising more than 100 prototypes. The LLM participated by instructing two participants who executed its instructions and provided objective feedback, generated ideas autonomously and made all design decisions without human intervention. The LLM exhibited similar prototyping practices to human participants and finished second among six teams, successfully designing and providing building instructions for functional prototypes. The LLM's concept generation capabilities were particularly strong. However, the LLM prematurely abandoned promising concepts when facing minor difficulties, added unnecessary complexity to designs, and experienced design fixation. Communication between the LLM and participants was challenging due to vague or unclear descriptions, and the LLM had difficulty maintaining continuity and relevance in answers. Based on these findings, six recommendations for implementing an LLM like ChatGPT in the design process are proposed, including leveraging it for ideation, ensuring human oversight for key decisions, implementing iterative feedback loops, prompting it to consider alternatives, and assigning specific and manageable tasks at a subsystem level.
4/30/2024