Can Developers Prompt? A Controlled Experiment for Code Documentation Generation

Overview

• Large language models (LLMs) have great potential to automate tedious development tasks like creating and maintaining code documentation.
• However, it's unclear how effectively developers can prompt LLMs to generate concise and useful documentation.
• The study reports on an experiment with 20 professionals and 30 computer science students tasked with generating code documentation for two Python functions.
• The experimental group used ad-hoc prompts in a ChatGPT-like Visual Studio Code extension, while the control group used a predefined few-shot prompt.

Plain English Explanation

The paper explores whether developers can effectively use large language models (LLMs) to automate the creation of code documentation. This is an important task, as maintaining detailed and accurate documentation is crucial for software development but can be time-consuming.

The researchers conducted an experiment to see how well developers could generate useful documentation by prompting an LLM-powered tool. They had two groups - one that could freely enter their own prompts, and one that used a predefined prompt. The goal was to see if the flexibility of ad-hoc prompting led to better results than a standardized approach.

The findings suggest that both professionals and students struggled to craft effective prompts. Students especially found the documentation generated from their own prompts to be less readable, concise, and helpful compared to the predefined prompt. Some professionals had more success by including the keyword "Docstring" in their prompts.

Overall, the participants recognized the LLM tool as a support system to iteratively refine the documentation, rather than expecting perfect outputs. The research highlights the need to better understand the prompting skills and preferences of developers for this type of task, and how to best support them.

Technical Explanation

The researchers conducted a controlled experiment to assess how effectively developers can prompt large language models (LLMs) to generate useful code documentation. They recruited 20 professional developers and 30 computer science students, and tasked them with creating documentation for two Python functions.

The experimental group used a ChatGPT-like extension of Visual Studio Code, where they could freely enter ad-hoc prompts to generate the documentation. The control group used a predefined few-shot prompt to complete the same task.

The results revealed that both professionals and students struggled to apply effective prompt engineering techniques. Students in particular perceived the documentation generated from their own prompts as significantly less readable, concise, and helpful compared to the documentation from the prepared prompt.

Some professionals were able to produce higher quality documentation by including the keyword "Docstring" in their ad-hoc prompts. However, participants in both groups rarely considered the output to be perfect, and instead viewed the LLM tool as a support system to iteratively refine the documentation.

The study highlights the need for further research to understand the prompting skills and preferences of developers for tasks like code documentation generation. It also suggests that developers may require additional support and guidance in formulating effective prompts, especially for less experienced users.

Critical Analysis

The paper provides valuable insights into the challenges developers face when trying to leverage large language models (LLMs) to automate code documentation tasks. The controlled experiment design and inclusion of both professional developers and computer science students offer a nuanced perspective on the issue.

One limitation of the study is the relatively small sample sizes, especially for the professional developer group. Expanding the participant pool could help confirm and further explore the observed trends.

Additionally, the paper does not delve deeply into the specific prompt engineering techniques that were successful or unsuccessful. More detailed analysis of the prompts used and their impact on the quality of the generated documentation could provide more actionable guidance for developers.

The researchers also acknowledge that their study focused on a narrow set of tasks and programming languages. Expanding the scope to explore a wider range of development activities and technologies could uncover additional insights and challenges.

Despite these limitations, the paper raises important questions about the role of LLMs in automating software engineering tasks and the support developers may need to effectively leverage these powerful tools. Encouraging critical thinking and further research in this area is crucial as the use of LLMs in software development continues to evolve.

Conclusion

This study sheds light on the potential and limitations of using large language models (LLMs) to automate the creation of code documentation. While LLMs hold great promise for streamlining tedious development tasks, the findings suggest that developers, especially less experienced ones, may struggle to craft effective prompts to generate high-quality documentation.

The research highlights the need for further exploration of the prompting skills and preferences of developers, as well as the development of tools and guidance to support them in leveraging LLMs for specific software engineering tasks. As the use of LLMs in software development continues to grow, understanding how to best integrate these technologies into the workflow of developers will be crucial for unlocking their full potential.