Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study
0
🤖
Sign in to get full access
Overview
- Prompt engineering is critical for effectively interacting with large language models (LLMs) like ChatGPT.
- However, efforts to teach this skill to students have been limited.
- This study designed and implemented a prompt engineering intervention to examine its influence on undergraduate students' AI self-efficacy, AI knowledge, and prompt engineering proficiency.
Plain English Explanation
The study explored how teaching prompt engineering can benefit students. Prompt engineering is the art of crafting effective prompts to get the desired responses from large language models like ChatGPT.
The researchers designed a workshop to teach prompt engineering strategies to 27 university students in Hong Kong. During the workshop, students learned techniques for creating effective prompts and applied them to plan an essay assignment for their history course.
The study found that after the workshop, the students showed:
- Higher confidence in their ability to use AI
- Better understanding of AI concepts
- Improved skills in crafting effective prompts
These findings highlight the importance of teaching prompt engineering as part of AI literacy education. By learning prompt engineering, students can more effectively leverage LLMs to support their academic work.
Technical Explanation
The study used a quasi-experimental design, with 27 undergraduate students participating in a 100-minute prompt engineering workshop during their history course at a university in Hong Kong.
During the workshop, students were:
- Introduced to prompt engineering strategies and best practices
- Guided in applying these strategies to plan the course's final essay task
Multiple data sources were collected, including:
- Pre- and post-workshop questionnaires to measure students' AI self-efficacy and knowledge
- Pre- and post-workshop prompt libraries to assess their prompt engineering skills
- Written reflections from the students
The study's findings revealed that the prompt engineering intervention led to:
- Increased AI self-efficacy among the students
- Enhanced understanding of AI concepts
- Improved ability to create effective prompts
These results highlight the value of incorporating prompt engineering training into AI literacy education to help students leverage large language models more effectively in their academic work.
Critical Analysis
The study provides valuable insights into the importance of teaching prompt engineering skills to students. However, there are a few potential limitations and areas for further research:
-
The small sample size of 27 students limits the generalizability of the findings. Larger-scale studies would be needed to validate the results.
-
The study did not include a control group, so it's difficult to attribute the observed changes solely to the prompt engineering intervention. Future research could incorporate a control group for more robust comparisons.
-
The study focused on a specific university setting and course (history). Investigating the impact of prompt engineering education across diverse academic disciplines would provide a more comprehensive understanding.
-
The study did not explore long-term effects of the prompt engineering intervention. Longitudinal research is needed to assess the sustainability of the observed improvements in AI self-efficacy, knowledge, and prompt engineering skills.
Despite these limitations, the study's findings highlight the value of prompt engineering education and its potential to enhance students' ability to effectively leverage large language models in their academic pursuits.
Conclusion
This study demonstrates the benefits of incorporating prompt engineering education into AI literacy curricula. By teaching students how to craft effective prompts, educators can empower them to better utilize large language models like ChatGPT to support their academic work and deepen their understanding of AI concepts.
The findings highlight the importance of prompt engineering as a critical skill for navigating the rapidly evolving landscape of AI-powered technologies. As large language models continue to shape the future of education and beyond, equipping students with prompt engineering expertise will be essential for their academic and professional success.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🤖
0
Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study
David James Woo, Deliang Wang, Tim Yung, Kai Guo
Prompt engineering is critical for effective interaction with large language models (LLMs) such as ChatGPT. However, efforts to teach this skill to students have been limited. This study designed and implemented a prompt engineering intervention, examining its influence on undergraduate students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts. The intervention involved 27 students who participated in a 100-minute workshop conducted during their history course at a university in Hong Kong. During the workshop, students were introduced to prompt engineering strategies, which they applied to plan the course's final essay task. Multiple data sources were collected, including students' responses to pre- and post-workshop questionnaires, pre- and post-workshop prompt libraries, and written reflections. The study's findings revealed that students demonstrated a higher level of AI self-efficacy, an enhanced understanding of AI concepts, and improved prompt engineering skills because of the intervention. These findings have implications for AI literacy education, as they highlight the importance of prompt engineering training for specific higher education use cases. This is a significant shift from students haphazardly and intuitively learning to engineer prompts. Through prompt engineering education, educators can faciitate students' effective navigation and leverage of LLMs to support their coursework.
Read more8/15/2024
🤖
0
Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering
Jonas Oppenlaender, Rhema Linder, Johanna Silvennoinen
We are witnessing a novel era of creativity where anyone can create digital content via prompt-based learning (known as prompt engineering). This paper investigates prompt engineering as a novel creative skill for creating AI art with text-to-image generation. In three consecutive studies, we explore whether crowdsourced participants can 1) discern prompt quality, 2) write prompts, and 3) refine prompts. We find that participants could evaluate prompt quality and crafted descriptive prompts, but they lacked style-specific vocabulary necessary for effective prompting. This is in line with our hypothesis that prompt engineering is a new type of skill that is non-intuitive and must first be acquired (e.g., through means of practice and learning) before it can be used. Our studies deepen our understanding of prompt engineering and chart future research directions. We conclude by envisioning four potential futures for prompt engineering.
Read more7/8/2024
👀
0
Unleashing the potential of prompt engineering: a comprehensive review
Banghao Chen, Zhaofeng Zhang, Nicolas Langren'e, Shengxin Zhu
This comprehensive review delves into the pivotal role of prompt engineering in unleashing the capabilities of Large Language Models (LLMs). The development of Artificial Intelligence (AI), from its inception in the 1950s to the emergence of advanced neural networks and deep learning architectures, has made a breakthrough in LLMs, with models such as GPT-4o and Claude-3, and in Vision-Language Models (VLMs), with models such as CLIP and ALIGN. Prompt engineering is the process of structuring inputs, which has emerged as a crucial technique to maximize the utility and accuracy of these models. This paper explores both foundational and advanced methodologies of prompt engineering, including techniques such as self-consistency, chain-of-thought, and generated knowledge, which significantly enhance model performance. Additionally, it examines the prompt method of VLMs through innovative approaches such as Context Optimization (CoOp), Conditional Context Optimization (CoCoOp), and Multimodal Prompt Learning (MaPLe). Critical to this discussion is the aspect of AI security, particularly adversarial attacks that exploit vulnerabilities in prompt engineering. Strategies to mitigate these risks and enhance model robustness are thoroughly reviewed. The evaluation of prompt methods is also addressed, through both subjective and objective metrics, ensuring a robust analysis of their efficacy. This review also reflects the essential role of prompt engineering in advancing AI capabilities, providing a structured framework for future research and application.
Read more9/6/2024
💬
0
A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks
Shubham Vatsal, Harsh Dubey
Large language models (LLMs) have shown remarkable performance on many different Natural Language Processing (NLP) tasks. Prompt engineering plays a key role in adding more to the already existing abilities of LLMs to achieve significant performance gains on various NLP tasks. Prompt engineering requires composing natural language instructions called prompts to elicit knowledge from LLMs in a structured way. Unlike previous state-of-the-art (SoTA) models, prompt engineering does not require extensive parameter re-training or fine-tuning based on the given NLP task and thus solely operates on the embedded knowledge of LLMs. Additionally, LLM enthusiasts can intelligently extract LLMs' knowledge through a basic natural language conversational exchange or prompt engineering, allowing more and more people even without deep mathematical machine learning background to experiment with LLMs. With prompt engineering gaining popularity in the last two years, researchers have come up with numerous engineering techniques around designing prompts to improve accuracy of information extraction from the LLMs. In this paper, we summarize different prompting techniques and club them together based on different NLP tasks that they have been used for. We further granularly highlight the performance of these prompting strategies on various datasets belonging to that NLP task, talk about the corresponding LLMs used, present a taxonomy diagram and discuss the possible SoTA for specific datasets. In total, we read and present a survey of 44 research papers which talk about 39 different prompting methods on 29 different NLP tasks of which most of them have been published in the last two years.
Read more7/25/2024