Talk to the Wall: The Role of Speech Interaction in Collaborative Visual Analytics
0
Sign in to get full access
Overview
- Explores the role of speech interaction in collaborative visual analytics using large vertical displays
- Examines how speech can complement other input modalities like touch and mouse for collaborative tasks
- Conducted user study to understand benefits and challenges of speech-based interaction
Plain English Explanation
The paper explores how using speech can help people work together more effectively when analyzing data visualizations on large wall-mounted screens. The researchers wanted to understand the advantages and difficulties of using speech, in addition to other ways of interacting like touch and mouse, for collaborative tasks.
They conducted a user study where groups of people used a visual analytics system with different input options - speech, touch, and mouse. The goal was to see how the various ways of interacting impacted the collaborative process and outcomes.
Technical Explanation
The paper focuses on the role of speech interaction in the context of collaborative visual analytics on large vertical displays. The researchers investigated how speech can complement other input modalities like touch and mouse for collaborative tasks.
They conducted a user study where groups of participants used a visual analytics system that supported different input methods - speech, touch, and mouse. The study explored the benefits and challenges of using speech for collaborative data analysis and interpretation tasks. The researchers analyzed the interactions, task performance, and user feedback to understand the impact of speech-based interaction in this setting.
Critical Analysis
The paper provides valuable insights into the potential benefits and challenges of incorporating speech interaction into collaborative visual analytics tools. However, the study was relatively small in scale, and more research may be needed to fully understand the broader implications and generalizability of the findings.
Additionally, the paper does not delve deeply into the technical implementation details or the specific speech recognition and natural language processing techniques used. Further exploration of the underlying technology and its limitations could help identify areas for improvement or inform the design of more robust speech-based interaction systems.
Conclusion
This paper offers a thought-provoking exploration of the role of speech interaction in collaborative visual analytics on large vertical displays. The findings suggest that speech can be a valuable complement to other input modalities, enabling more natural and efficient collaboration. However, the researchers also highlight the need to carefully consider the trade-offs and challenges associated with integrating speech-based interaction into such systems. This research serves as a valuable starting point for further investigations into multimodal interaction techniques for data analysis and visualization.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Talk to the Wall: The Role of Speech Interaction in Collaborative Visual Analytics
Gabriela Molina Le'on, Anastasia Bezerianos, Olivier Gladin, Petra Isenberg
We present the results of an exploratory study on how pairs interact with speech commands and touch gestures on a wall-sized display during a collaborative sensemaking task. Previous work has shown that speech commands, alone or in combination with other input modalities, can support visual data exploration by individuals. However, it is still unknown whether and how speech commands can be used in collaboration, and for what tasks. To answer these questions, we developed a functioning prototype that we used as a technology probe. We conducted an in-depth exploratory study with 10 participant pairs to analyze their interaction choices, the interplay between the input modalities, and their collaboration. While touch was the most used modality, we found that participants preferred speech commands for global operations, used them for distant interaction, and that speech interaction contributed to the awareness of the partner's actions. Furthermore, the likelihood of using speech commands during collaboration was related to the personality trait of agreeableness. Regarding collaboration styles, participants interacted with speech equally often whether they were in loosely or closely coupled collaboration. While the partners stood closer to each other during close collaboration, they did not distance themselves to use speech commands. From our findings, we derive and contribute a set of design considerations for collaborative and multimodal interactive data analysis systems. All supplemental materials are available at https://osf.io/8gpv2
Read more8/13/2024
0
Chart What I Say: Exploring Cross-Modality Prompt Alignment in AI-Assisted Chart Authoring
Nazar Ponochevnyi, Anastasia Kuzminykh
Recent chart-authoring systems, such as Amazon Q in QuickSight and Copilot for Power BI, demonstrate an emergent focus on supporting natural language input to share meaningful insights from data through chart creation. Currently, chart-authoring systems tend to integrate voice input capabilities by relying on speech-to-text transcription, processing spoken and typed input similarly. However, cross-modality input comparisons in other interaction domains suggest that the structure of spoken and typed-in interactions could notably differ, reflecting variations in user expectations based on interface affordances. Thus, in this work, we compare spoken and typed instructions for chart creation. Findings suggest that while both text and voice instructions cover chart elements and element organization, voice descriptions have a variety of command formats, element characteristics, and complex linguistic features. Based on these findings, we developed guidelines for designing voice-based authoring-oriented systems and additional features that can be incorporated into existing text-based systems to support speech modality.
Read more4/9/2024
0
Leveraging Speech for Gesture Detection in Multimodal Communication
Esam Ghaleb, Ilya Burenko, Marlou Rasenberg, Wim Pouw, Ivan Toni, Peter Uhrig, Anna Wilson, Judith Holler, Asl{i} Ozyurek, Raquel Fern'andez
Gestures are inherent to human interaction and often complement speech in face-to-face communication, forming a multimodal communication system. An important task in gesture analysis is detecting a gesture's beginning and end. Research on automatic gesture detection has primarily focused on visual and kinematic information to detect a limited set of isolated or silent gestures with low variability, neglecting the integration of speech and vision signals to detect gestures that co-occur with speech. This work addresses this gap by focusing on co-speech gesture detection, emphasising the synchrony between speech and co-speech hand gestures. We address three main challenges: the variability of gesture forms, the temporal misalignment between gesture and speech onsets, and differences in sampling rate between modalities. We investigate extended speech time windows and employ separate backbone models for each modality to address the temporal misalignment and sampling rate differences. We utilize Transformer encoders in cross-modal and early fusion techniques to effectively align and integrate speech and skeletal sequences. The study results show that combining visual and speech information significantly enhances gesture detection performance. Our findings indicate that expanding the speech buffer beyond visual time segments improves performance and that multimodal integration using cross-modal and early fusion techniques outperforms baseline methods using unimodal and late fusion methods. Additionally, we find a correlation between the models' gesture prediction confidence and low-level speech frequency features potentially associated with gestures. Overall, the study provides a better understanding and detection methods for co-speech gestures, facilitating the analysis of multimodal communication.
Read more4/24/2024
0
Mixing Modes: Active and Passive Integration of Speech, Text, and Visualization for Communicating Data Uncertainty
Chase Stokes, Chelsea Sanker, Bridget Cogley, Vidya Setlur
Interpreting uncertain data can be difficult, particularly if the data presentation is complex. We investigate the efficacy of different modalities for representing data and how to combine the strengths of each modality to facilitate the communication of data uncertainty. We implemented two multimodal prototypes to explore the design space of integrating speech, text, and visualization elements. A preliminary evaluation with 20 participants from academic and industry communities demonstrates that there exists no one-size-fits-all approach for uncertainty communication strategies; rather, the effectiveness of conveying uncertain data is intertwined with user preferences and situational context, necessitating a more refined, multimodal strategy for future interface design.
Read more4/15/2024