MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on Eyeglasses
2405.21004
0
0
Abstract
We introduce MunchSonic, an AI-powered active acoustic sensing system integrated into eyeglasses, designed to track fine-grained dietary actions like hand-to-mouth movements for food intake, chewing, and drinking. MunchSonic emits inaudible ultrasonic waves from a commodity eyeglass frame. The reflected signals contain rich information about the position and movements of various body parts, including the mouth, jaw, arms, and hands, all of which are involved in eating activities. These signals are then processed by a custom deep-learning pipeline to classify six actions: food intake, chewing, drinking, talking, face-hand touching, and other activities (null). In an unconstrained user study with 12 participants, MunchSonic achieves a 93.5% macro F1-score in a user-independent evaluation with a 2-second time resolution, demonstrating its effectiveness. Additionally, MunchSonic accurately tracks eating episodes and the frequency of food intake within those episodes.
Create account to get full access
Overview
ā¢ This paper, titled "MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on Eyeglasses", explores a novel approach to monitoring dietary behavior using acoustic sensors embedded in eyeglasses.
ā¢ The researchers developed a system called MunchSonic that can detect and classify a wide range of fine-grained eating and drinking actions, such as chewing, swallowing, and sipping, using active acoustic sensing technology.
ā¢ The system leverages the unique acoustic signatures generated by these actions and uses machine learning algorithms to recognize them with high accuracy, providing a more comprehensive and objective way to track dietary intake compared to traditional self-reporting methods.
Plain English Explanation
The paper describes a new way to monitor what people eat and drink throughout the day using special sensors in eyeglasses. The researchers created a system called MunchSonic that can detect different actions related to eating and drinking, such as chewing, swallowing, and sipping, by listening to the unique sounds they make.
Instead of relying on people to remember and report what they've eaten, which can be inaccurate, MunchSonic can automatically track these dietary actions in real-time using tiny microphones and speakers built into the frames of the eyeglasses. The system uses machine learning algorithms to analyze the acoustic patterns and reliably identify the specific actions, providing a more comprehensive and objective record of a person's dietary behavior.
This technology could be useful for healthcare applications, such as monitoring nutritional intake for patients with dietary requirements or disorders, as well as for research on eating habits and the relationship between diet and health.
Technical Explanation
The MunchSonic system uses active acoustic sensing, where the eyeglasses emit high-frequency sounds that are inaudible to the human ear and then analyze the reflections to detect specific actions related to eating and drinking. This approach is similar to AcTsonic, which used active acoustic sensing to recognize everyday activities.
The researchers built a prototype of the MunchSonic system, integrating small speakers and microphones into the frames of the eyeglasses. They then collected a dataset of acoustic signals corresponding to various eating and drinking actions, such as chewing, swallowing, sipping, and biting. Using this data, they trained machine learning models to recognize the unique acoustic signatures of these actions.
In their evaluation, the MunchSonic system achieved high accuracy in detecting and classifying the different dietary actions, outperforming other approaches like MECIFACE, which used a combination of mechanomyography and inertial sensors in glasses, and AutoRecFI, which recognized food intake from audio and inertial sensors in the environment.
The paper also discusses how MunchSonic could be integrated with other sensing modalities, such as those used in HowMuchYouAte and AugGlasses, to provide a more comprehensive dietary monitoring solution.
Critical Analysis
The researchers have presented a novel and promising approach to tracking fine-grained dietary actions using active acoustic sensing in eyeglasses. The high accuracy demonstrated by the MunchSonic system suggests that it could be a valuable tool for healthcare and research applications.
However, the paper does not address several potential limitations and areas for further research. For example, it's unclear how the system would perform in noisy or diverse real-world environments, or how it would handle variations in individual chewing and swallowing patterns. Additionally, the long-term comfort and acceptance of wearing the specialized eyeglasses is not discussed.
Further research is needed to understand the broader implications and practical challenges of deploying such a system in real-world settings. Addressing privacy concerns and ensuring user consent and control over the data collected will also be crucial for the widespread adoption of this technology.
Conclusion
The MunchSonic system presented in this paper represents an innovative approach to tracking fine-grained dietary actions using active acoustic sensing in eyeglasses. By leveraging the unique acoustic signatures of eating and drinking behaviors, the system can provide a more comprehensive and objective record of dietary intake compared to traditional self-reporting methods.
The high accuracy demonstrated in the researchers' evaluation indicates the potential of this technology for healthcare applications, such as monitoring nutritional intake for patients with dietary requirements or disorders, as well as for research on the relationship between diet and health. However, further work is needed to address the practical challenges and ethical considerations of deploying such a system in real-world settings.
Overall, the MunchSonic system showcases the power of leveraging advanced sensing and machine learning techniques to gain deeper insights into human dietary behavior, which could have significant implications for improving individual and public health outcomes.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
ActSonic: Everyday Activity Recognition on Smart Glasses using Active Acoustic Sensing
Saif Mahmud, Vineet Parikh, Qikang Liang, Ke Li, Ruidong Zhang, Ashwin Ajit, Vipin Gunda, Devansh Agarwal, Franc{c}ois Guimbreti`ere, Cheng Zhang
0
0
We present ActSonic, an intelligent, low-power active acoustic sensing system integrated into eyeglasses that can recognize 27 different everyday activities (e.g., eating, drinking, toothbrushing) from inaudible acoustic waves around the body with a time resolution of one second. It only needs a pair of miniature speakers and microphones mounted on each hinge of eyeglasses to emit ultrasonic waves to create an acoustic aura around the body. Based on the position and motion of various body parts, the acoustic signals are reflected with unique patterns captured by the microphone and analyzed by a customized self-supervised deep learning framework to infer the performed activities. ActSonic was deployed in a user study with 19 participants across 19 households to evaluate its efficacy. Without requiring any training data from a new user (leave-one-participant-out evaluation), ActSonic was able to detect 27 activities, achieving an average F1-score of 86.6% in fully unconstrained scenarios and 93.4% in prompted settings at participants' homes.
5/9/2024
EchoGuide: Active Acoustic Guidance for LLM-Based Eating Event Analysis from Egocentric Videos
Vineet Parikh, Saif Mahmud, Devansh Agarwal, Ke Li, Franc{c}ois Guimbreti`ere, Cheng Zhang
0
0
Self-recording eating behaviors is a step towards a healthy lifestyle recommended by many health professionals. However, the current practice of manually recording eating activities using paper records or smartphone apps is often unsustainable and inaccurate. Smart glasses have emerged as a promising wearable form factor for tracking eating behaviors, but existing systems primarily identify when eating occurs without capturing details of the eating activities (E.g., what is being eaten). In this paper, we present EchoGuide, an application and system pipeline that leverages low-power active acoustic sensing to guide head-mounted cameras to capture egocentric videos, enabling efficient and detailed analysis of eating activities. By combining active acoustic sensing for eating detection with video captioning models and large-scale language models for retrieval augmentation, EchoGuide intelligently clips and analyzes videos to create concise, relevant activity records on eating. We evaluated EchoGuide with 9 participants in naturalistic settings involving eating activities, demonstrating high-quality summarization and significant reductions in video data needed, paving the way for practical, scalable eating activity tracking.
6/18/2024
MeciFace: Mechanomyography and Inertial Fusion-based Glasses for Edge Real-Time Recognition of Facial and Eating Activities
Hymalai Bello, Sungho Suh, Bo Zhou, Paul Lukowicz
0
0
The increasing prevalence of stress-related eating behaviors and their impact on overall health highlights the importance of effective and ubiquitous monitoring systems. In this paper, we present MeciFace, an innovative wearable technology designed to monitor facial expressions and eating activities in real-time on-the-edge (RTE). MeciFace aims to provide a low-power, privacy-conscious, and highly accurate tool for promoting healthy eating behaviors and stress management. We employ lightweight convolutional neural networks as backbone models for facial expression and eating monitoring scenarios. The MeciFace system ensures efficient data processing with a tiny memory footprint, ranging from 11KB to 19 KB. During RTE evaluation, the system achieves an F1-score of < 86% for facial expression recognition and 94% for eating/drinking monitoring, for the RTE of unseen users (user-independent case).
4/4/2024
SonicID: User Identification on Smart Glasses with Acoustic Sensing
Ke Li, Devansh Agarwal, Ruidong Zhang, Vipin Gunda, Tianjun Mo, Saif Mahmud, Boao Chen, Franc{c}ois Guimbreti`ere, Cheng Zhang
0
0
Smart glasses have become more prevalent as they provide an increasing number of applications for users. They store various types of private information or can access it via connections established with other devices. Therefore, there is a growing need for user identification on smart glasses. In this paper, we introduce a low-power and minimally-obtrusive system called SonicID, designed to authenticate users on glasses. SonicID extracts unique biometric information from users by scanning their faces with ultrasonic waves and utilizes this information to distinguish between different users, powered by a customized binary classifier with the ResNet-18 architecture. SonicID can authenticate users within 0.12 seconds, with an energy consumption of 19.8 mAs per trial. A user study involving 24 participants confirms that SonicID achieves a true positive rate of 96.5%, a false positive rate of 4.1%, and a balanced accuracy of 96.2% using just 4 minutes of training data collected for each new user. This performance is relatively consistent across different remounting sessions and days. Given this promising performance, we further discuss the potential applications of SonicID and methods to improve its performance in the future.
6/13/2024