Perceived Usability of Collaborative Modeling Tools
0
✅
Sign in to get full access
Overview
- Online collaborative creation of models is becoming more common.
- Collaborative modeling using chatbots and natural language may lower barriers for users from different domains.
- The study compares the perceived usability of two online collaborative modeling tools: the SOCIO chatbot and the Creately web-based tool.
Plain English Explanation
The paper examines how collaborative modeling tools that use chatbots and natural language can make it easier for people from diverse backgrounds to work together on creating models. The researchers conducted an experiment to compare the user experience of two online collaboration tools - a chatbot called SOCIO and a web-based tool called Creately. They wanted to see if the natural language communication enabled by the chatbot could enhance the efficiency and enjoyment of the collaborative modeling process.
Technical Explanation
The researchers performed a crossover experiment with 66 participants to evaluate the system usability of the two collaborative modeling tools. They used the System Usability Scale (SUS) as the evaluation instrument and employed both quantitative and qualitative analysis techniques, including inferential statistics and thematic analysis.
The results indicate that the chatbot-based tool enhanced communication and collaboration efficiency and improved the overall user experience compared to the web-based tool. However, the chatbot-based tool still needs to improve its guidance and help features for novice users.
Critical Analysis
While the study suggests that chatbot-based collaborative modeling tools can enhance the user experience, the researchers acknowledge that more work is needed to address the guidance and support needs of inexperienced users. Additionally, the study was conducted in a controlled experimental setting, and the findings may not fully transfer to real-world scenarios with diverse user populations and complex modeling tasks.
Further research could explore the long-term impacts of chatbot-based collaboration, the specific design features that contribute to the improved user experience, and the scalability of the approach to larger, more complex modeling projects.
Conclusion
This study provides evidence that chatbot-based collaborative modeling tools can enhance communication, collaboration, and the overall user experience compared to traditional web-based tools. While more work is needed to address the needs of novice users, the findings suggest that this approach holds promise for lowering the barriers to collaborative modeling, particularly for users from diverse backgrounds.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
✅
0
Perceived Usability of Collaborative Modeling Tools
Ranci Ren, John W. Castro, Santiago R. Acu~na, Oscar Dieste, Silvia T. Acu~na
Context: Online collaborative creation of models is becoming commonplace. Collaborative modeling using chatbots and natural language may lower the barriers to modeling for users from different domains. Objective: We compare the perceived usability of two similarly online collaborative modeling tools, the SOCIO chatbot and the Creately web-based tool. Method: We conducted a crossover experiment with 66 participants. The evaluation instrument was based on the System Usability Scale (SUS). We performed a quantitative and qualitative exploration, employing inferential statistics and thematic analysis. Results: The results indicate that chatbots enabling natural language communication enhance communication and collaboration efficiency and improve the user experience. Conclusion: Chatbots need to improve guidance and help for novices, but they appear beneficial for enhancing user experience.
Read more8/27/2024
🤯
0
Designing and Evaluating Multi-Chatbot Interface for Human-AI Communication: Preliminary Findings from a Persuasion Task
Sion Yoon, Tae Eun Kim, Yoo Jung Oh
The dynamics of human-AI communication have been reshaped by language models such as ChatGPT. However, extant research has primarily focused on dyadic communication, leaving much to be explored regarding the dynamics of human-AI communication in group settings. The availability of multiple language model chatbots presents a unique opportunity for scholars to better understand the interaction between humans and multiple chatbots. This study examines the impact of multi-chatbot communication in a specific persuasion setting: promoting charitable donations. We developed an online environment that enables multi-chatbot communication and conducted a pilot experiment utilizing two GPT-based chatbots, Save the Children and UNICEF chatbots, to promote charitable donations. In this study, we present our development process of the multi-chatbot interface and present preliminary findings from a pilot experiment. Analysis of qualitative and quantitative feedback are presented, and limitations are addressed.
Read more7/1/2024
⚙️
0
From Human-to-Human to Human-to-Bot Conversations in Software Engineering
Ranim Khojah, Francisco Gomes de Oliveira Neto, Philipp Leitner
Software developers use natural language to interact not only with other humans, but increasingly also with chatbots. These interactions have different properties and flow differently based on what goal the developer wants to achieve and who they interact with. In this paper, we aim to understand the dynamics of conversations that occur during modern software development after the integration of AI and chatbots, enabling a deeper recognition of the advantages and disadvantages of including chatbot interactions in addition to human conversations in collaborative work. We compile existing conversation attributes with humans and NLU-based chatbots and adapt them to the context of software development. Then, we extend the comparison to include LLM-powered chatbots based on an observational study. We present similarities and differences between human-to-human and human-to-bot conversations, also distinguishing between NLU- and LLM-based chatbots. Furthermore, we discuss how understanding the differences among the conversation styles guides the developer on how to shape their expectations from a conversation and consequently support the communication within a software team. We conclude that the recent conversation styles that we observe with LLM-chatbots can not replace conversations with humans due to certain attributes regarding social aspects despite their ability to support productivity and decrease the developers' mental load.
Read more5/22/2024
0
BotEval: Facilitating Interactive Human Evaluation
Hyundong Cho, Thamme Gowda, Yuyang Huang, Zixun Lu, Tianli Tong, Jonathan May
Following the rapid progress in natural language processing (NLP) models, language models are applied to increasingly more complex interactive tasks such as negotiations and conversation moderations. Having human evaluators directly interact with these NLP models is essential for adequately evaluating the performance on such interactive tasks. We develop BotEval, an easily customizable, open-source, evaluation toolkit that focuses on enabling human-bot interactions as part of the evaluation process, as opposed to human evaluators making judgements for a static input. BotEval balances flexibility for customization and user-friendliness by providing templates for common use cases that span various degrees of complexity and built-in compatibility with popular crowdsourcing platforms. We showcase the numerous useful features of BotEval through a study that evaluates the performance of various chatbots on their effectiveness for conversational moderation and discuss how BotEval differs from other annotation tools.
Read more7/26/2024