Multisensory extended reality applications offer benefits for volumetric biomedical image analysis in research and medicine
2311.03986
0
0
🖼️
Abstract
3D data from high-resolution volumetric imaging is a central resource for diagnosis and treatment in modern medicine. While the fast development of AI enhances imaging and analysis, commonly used visualization methods lag far behind. Recent research used extended reality (XR) for perceiving 3D images with visual depth perception and touch but used restrictive haptic devices. While unrestricted touch benefits volumetric data examination, implementing natural haptic interaction with XR is challenging. The research question is whether a multisensory XR application with intuitive haptic interaction adds value and should be pursued. In a study, 24 experts for biomedical images in research and medicine explored 3D medical shapes with 3 applications: a multisensory virtual reality (VR) prototype using haptic gloves, a simple VR prototype using controllers, and a standard PC application. Results of standardized questionnaires showed no significant differences between all application types regarding usability and no significant difference between both VR applications regarding presence. Participants agreed to statements that VR visualizations provide better depth information, using the hands instead of controllers simplifies data exploration, the multisensory VR prototype allows intuitive data exploration, and it is beneficial over traditional data examination methods. While most participants mentioned manual interaction as best aspect, they also found it the most improvable. We conclude that a multisensory XR application with improved manual interaction adds value for volumetric biomedical data examination. We will proceed with our open-source research project ISH3DE (Intuitive Stereoptic Haptic 3D Data Exploration) to serve medical education, therapeutic decisions, surgery preparations, or research data analysis.
Create account to get full access
Overview
- 3D data from high-resolution volumetric imaging is crucial for diagnosis and treatment in modern medicine.
- While AI advancements enhance imaging and analysis, common visualization methods lag behind.
- Recent research has explored using extended reality (XR) for perceiving 3D images with visual depth perception and touch, but this has been limited by restrictive haptic devices.
- Unrestricted touch benefits volumetric data examination, but implementing natural haptic interaction with XR is challenging.
- The research question is whether a multisensory XR application with intuitive haptic interaction adds value and should be pursued.
Plain English Explanation
3D medical imaging data is crucial for doctors and researchers to diagnose and treat patients. However, the common ways of visualizing and interacting with this 3D data are not keeping up with the latest advancements in technology.
Recent studies have looked at using extended reality (XR) - which includes virtual reality (VR) and augmented reality - to allow people to see and even touch 3D medical images. This provides a more immersive and natural way to examine the data. But the haptic (touch) devices used in these studies have been restrictive, limiting the full potential of this approach.
Being able to freely touch and manipulate the 3D data could be very beneficial for medical professionals. But figuring out how to implement this kind of natural haptic interaction with XR technology is challenging. The researchers wanted to explore whether developing a multisensory XR application with intuitive haptic controls would be worthwhile.
Technical Explanation
The study had 24 experts in biomedical imaging, from both research and clinical settings, explore 3D medical shapes using three different applications:
- A multisensory VR prototype that used haptic gloves for touch interaction.
- A simple VR prototype that used standard controllers.
- A standard PC-based visualization tool.
The researchers used standardized questionnaires to assess the usability and sense of presence (feeling of "being there") for each application. They found no significant differences in usability or sense of presence between the two VR prototypes.
However, the participants agreed that the VR visualizations provided better depth information, using their hands instead of controllers simplified data exploration, the multisensory VR prototype allowed more intuitive data exploration, and it was generally more beneficial than traditional data examination methods. While most participants cited the manual interaction as the best aspect, they also saw it as the most improvable feature.
Critical Analysis
The study provides promising initial results suggesting that a multisensory XR application with intuitive haptic interaction could add value for examining volumetric biomedical data. However, the research is still in an early stage, and the authors acknowledge that improving the manual interaction capabilities is an important area for further development.
Some additional limitations and considerations that could be explored in future work include:
- Evaluating the applications with a larger and more diverse sample of users, including clinicians and researchers with varying levels of experience.
- Assessing the impact of the multisensory XR approach on specific medical tasks, such as diagnosis, treatment planning, or surgical preparation.
- Exploring the potential of the multisensory XR approach for medical education and training or research data analysis.
Conclusion
This research suggests that a multisensory XR application with intuitive haptic interaction could provide significant benefits for examining 3D medical imaging data, compared to traditional visualization methods. While the manual interaction capabilities need further improvement, the findings indicate that this approach is worth pursuing and developing further to serve medical education, therapeutic decisions, surgery preparations, and research data analysis.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
💬
Pseudo-Haptics Survey: Human-Computer Interaction in Extended Reality & Teleoperation
Rui Xavier, Jos'e Lu'is Silva, Rodrigo Ventura, Joaquim Jorge
0
0
Pseudo-haptic techniques are becoming increasingly popular in human-computer interaction. They replicate haptic sensations by leveraging primarily visual feedback rather than mechanical actuators. These techniques bridge the gap between the real and virtual worlds by exploring the brain's ability to integrate visual and haptic information. One of the many advantages of pseudo-haptic techniques is that they are cost-effective, portable, and flexible. They eliminate the need for direct attachment of haptic devices to the body, which can be heavy and large and require a lot of power and maintenance. Recent research has focused on applying these techniques to extended reality and mid-air interactions. To better understand the potential of pseudo-haptic techniques, the authors developed a novel taxonomy encompassing tactile feedback, kinesthetic feedback, and combined categories in multimodal approaches, ground not covered by previous surveys. This survey highlights multimodal strategies and potential avenues for future studies, particularly regarding integrating these techniques into extended reality and collaborative virtual environments.
6/4/2024
🤿
From Virtual Gains to Real Pains: Potential Harms of Immersive Exergames
Sebastian Cmentowski, Sukran Karaosmanoglu, Frank Steinicke
0
0
Digitalization and virtualization are parts of our everyday lives in almost all aspects ranging from work, education, and communication to entertainment. A novel step in this direction is the widespread interest in extended reality (XR) [2]. The newest consumer-ready head-mounted displays (HMD) such as Meta Quest 3 or Apple Vision Pro, have reached unprecedented levels of visual fidelity, interaction capabilities, and computational power. The built-in pass-through features of these headsets enable both virtual reality (VR) and augmented reality (AR) with the same devices. However, the immersive nature of these experiences is not the only groundbreaking difference from established forms of media.
5/10/2024
Effects of Multisensory Feedback on the Perception and Performance of Virtual Reality Hand-Retargeted Interaction
Hyunyoung Jang, Jinwook Kim, Jeongmi Lee
0
0
Retargeting methods that modify the visual representation of real movements have been widely used to expand the interaction space and create engaging virtual reality experiences. For optimal user experience and performance, it is essential to specify the perception of retargeting and utilize the appropriate range of modification parameters. However, previous studies mostly concentrated on whether users perceived the target sense or not and rarely examined the perceptual accuracy and sensitivity to retargeting. Moreover, it is unknown how the perception and performance in hand-retargeted interactions are influenced by multisensory feedback. In this study, we used rigorous psychophysical methods to specify users' perceptual accuracy and sensitivity to hand-retargeting and provide acceptable ranges of retargeting parameters. We also presented different multisensory feedback simultaneously with the retargeting to probe its effect on users' perception and task performance. The experimental results showed that providing continuous multisensory feedback, proportionate to the distance between the virtual hand and the targeted destination, heightened the accuracy of users' perception of hand retargeting without altering their perceptual sensitivity. Furthermore, the utilization of multisensory feedback considerably improved the precision of task performance, particularly at lower gain factors. Based on these findings, we propose design guidelines and potential applications of VR hand-retargeted interactions and multisensory feedback for optimal user experience and performance.
4/8/2024
🏋️
Evaluating the efficacy of haptic feedback, 360{deg} treadmill-integrated Virtual Reality framework and longitudinal training on decision-making performance in a complex search-and-shoot simulation
Akash K Rao, Arnav Bhavsar, Shubhajit Roy Chowdhury, Sushil Chandra, Ramsingh Negi, Prakash Duraisamy, Varun Dutt
0
0
Virtual Reality (VR) has made significant strides, offering users a multitude of ways to interact with virtual environments. Each sensory modality in VR provides distinct inputs and interactions, enhancing the user's immersion and presence. However, the potential of additional sensory modalities, such as haptic feedback and 360{deg} locomotion, to improve decision-making performance has not been thoroughly investigated. This study addresses this gap by evaluating the impact of a haptic feedback, 360{deg} locomotion-integrated VR framework and longitudinal, heterogeneous training on decision-making performance in a complex search-and-shoot simulation. The study involved 32 participants from a defence simulation base in India, who were randomly divided into two groups: experimental (haptic feedback, 360{deg} locomotion-integrated VR framework with longitudinal, heterogeneous training) and placebo control (longitudinal, heterogeneous VR training without extrasensory modalities). The experiment lasted 10 days. On Day 1, all subjects executed a search-and-shoot simulation closely replicating the elements/situations in the real world. From Day 2 to Day 9, the subjects underwent heterogeneous training, imparted by the design of various complexity levels in the simulation using changes in behavioral attributes/artificial intelligence of the enemies. On Day 10, they repeated the search-and-shoot simulation executed on Day 1. The results showed that the experimental group experienced a gradual increase in presence, immersion, and engagement compared to the placebo control group. However, there was no significant difference in decision-making performance between the two groups on day 10. We intend to use these findings to design multisensory VR training frameworks that enhance engagement levels and decision-making performance.
4/16/2024