SiNC+: Adaptive Camera-Based Vitals with Unsupervised Learning of Periodic Signals
2404.13449
0
0
Abstract
Subtle periodic signals, such as blood volume pulse and respiration, can be extracted from RGB video, enabling noncontact health monitoring at low cost. Advancements in remote pulse estimation -- or remote photoplethysmography (rPPG) -- are currently driven by deep learning solutions. However, modern approaches are trained and evaluated on benchmark datasets with ground truth from contact-PPG sensors. We present the first non-contrastive unsupervised learning framework for signal regression to mitigate the need for labelled video data. With minimal assumptions of periodicity and finite bandwidth, our approach discovers the blood volume pulse directly from unlabelled videos. We find that encouraging sparse power spectra within normal physiological bandlimits and variance over batches of power spectra is sufficient for learning visual features of periodic signals. We perform the first experiments utilizing unlabelled video data not specifically created for rPPG to train robust pulse rate estimators. Given the limited inductive biases, we successfully applied the same approach to camera-based respiration by changing the bandlimits of the target signal. This shows that the approach is general enough for unsupervised learning of bandlimited quasi-periodic signals from different domains. Furthermore, we show that the framework is effective for finetuning models on unlabelled video from a single subject, allowing for personalized and adaptive signal regressors.
Create account to get full access
Overview
- This paper introduces SiNC+, a camera-based system that can remotely measure vital signs like heart rate and respiration rate without specialized equipment.
- SiNC+ uses unsupervised learning to automatically identify and track periodic signals in the video, allowing it to work on a wide range of subjects without calibration.
- The system demonstrates accurate vitals measurement across hundreds of participants, outperforming previous camera-based approaches.
Plain English Explanation
SiNC+ is a new technology that can measure a person's vital signs, like their heart rate and breathing rate, just by using a regular camera. It doesn't require any special equipment or sensors attached to the person. Instead, SiNC+ uses a technique called "unsupervised learning" to automatically identify the periodic signals in the video that correspond to the person's heartbeat and breathing.
This is an important advance because previous camera-based approaches for measuring vital signs often required a lot of setup and calibration to work properly. SiNC+ is more flexible and can work on a wide range of people without needing to be specially configured for each individual. The researchers tested SiNC+ on hundreds of participants and found it could measure vital signs very accurately, outperforming earlier camera-based systems.
The key innovation in SiNC+ is how it uses unsupervised learning to figure out the relevant signals in the video on its own, rather than needing to be explicitly trained. This makes it more adaptable and practical for real-world use cases, like monitoring people's health remotely or measuring engagement during online meetings.
Technical Explanation
The core of SiNC+ is an unsupervised learning approach that can automatically identify and track periodic signals in video data. Unlike previous camera-based vital sign measurement techniques that relied on supervised learning and required extensive calibration, SiNC+ uses a self-supervised learning framework to discover the relevant signals from the video without any labeled training data.
The SiNC+ pipeline first extracts spatial features from the video frames using a convolutional neural network. It then applies a temporal convolutional network to identify periodic motions, which are likely to correspond to the subject's heartbeat and breathing. An optimization process refines these periodic signals, aligning them to find the most prominent heart rate and respiration rate.
Experiments on hundreds of participants showed that SiNC+ could measure heart rate and respiration rate with high accuracy, outperforming prior camera-based approaches. The system was also robust to factors like subject head orientation and lighting conditions.
Critical Analysis
The researchers acknowledge several limitations of SiNC+. First, the system currently only works on a single subject at a time, so it may not be practical for monitoring multiple people simultaneously. Additionally, the unsupervised learning approach means SiNC+ requires a longer initialization period compared to supervised methods.
The paper also does not address the potential privacy concerns around using cameras to measure vital signs without explicit consent. There are open questions about the ethical implications and data security considerations of such remote physiological sensing technologies.
Overall, while SiNC+ represents an impressive technical advancement, further research is needed to explore the real-world applicability and address the ethical challenges of camera-based vital sign monitoring.
Conclusion
The SiNC+ system demonstrates a novel approach to camera-based vital sign measurement that uses unsupervised learning to automatically identify and track periodic signals. By eliminating the need for calibration and specialized equipment, SiNC+ has the potential to enable more accessible and widespread remote health monitoring applications. However, the technology also raises important privacy and ethical concerns that require careful consideration as the field continues to evolve.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
RhythmMamba: Fast Remote Physiological Measurement with Arbitrary Length Videos
Bochao Zou, Zizheng Guo, Xiaocheng Hu, Huimin Ma
0
0
Remote photoplethysmography (rPPG) is a non-contact method for detecting physiological signals from facial videos, holding great potential in various applications such as healthcare, affective computing, and anti-spoofing. Existing deep learning methods struggle to address two core issues of rPPG simultaneously: extracting weak rPPG signals from video segments with large spatiotemporal redundancy and understanding the periodic patterns of rPPG among long contexts. This represents a trade-off between computational complexity and the ability to capture long-range dependencies, posing a challenge for rPPG that is suitable for deployment on mobile devices. Based on the in-depth exploration of Mamba's comprehension of spatial and temporal information, this paper introduces RhythmMamba, an end-to-end Mamba-based method that employs multi-temporal Mamba to constrain both periodic patterns and short-term trends, coupled with frequency domain feed-forward to enable Mamba to robustly understand the quasi-periodic patterns of rPPG. Extensive experiments show that RhythmMamba achieves state-of-the-art performance with reduced parameters and lower computational complexity. The proposed RhythmMamba can be applied to video segments of any length without performance degradation. The codes are available at https://github.com/zizheng-guo/RhythmMamba.
4/10/2024
Joint Spatial-Temporal Modeling and Contrastive Learning for Self-supervised Heart Rate Measurement
Wei Qian, Qi Li, Kun Li, Xinke Wang, Xiao Sun, Meng Wang, Dan Guo
0
0
This paper briefly introduces the solutions developed by our team, HFUT-VUT, for Track 1 of self-supervised heart rate measurement in the 3rd Vision-based Remote Physiological Signal Sensing (RePSS) Challenge hosted at IJCAI 2024. The goal is to develop a self-supervised learning algorithm for heart rate (HR) estimation using unlabeled facial videos. To tackle this task, we present two self-supervised HR estimation solutions that integrate spatial-temporal modeling and contrastive learning, respectively. Specifically, we first propose a non-end-to-end self-supervised HR measurement framework based on spatial-temporal modeling, which can effectively capture subtle rPPG clues and leverage the inherent bandwidth and periodicity characteristics of rPPG to constrain the model. Meanwhile, we employ an excellent end-to-end solution based on contrastive learning, aiming to generalize across different scenarios from complementary perspectives. Finally, we combine the strengths of the above solutions through an ensemble strategy to generate the final predictions, leading to a more accurate HR estimation. As a result, our solutions achieved a remarkable RMSE score of 8.85277 on the test dataset, securing textbf{2nd place} in Track 1 of the challenge.
6/10/2024
Camera-Based Remote Physiology Sensing for Hundreds of Subjects Across Skin Tones
Jiankai Tang, Xinyi Li, Jiacheng Liu, Xiyuxing Zhang, Zeyu Wang, Yuntao Wang
0
0
Remote photoplethysmography (rPPG) emerges as a promising method for non-invasive, convenient measurement of vital signs, utilizing the widespread presence of cameras. Despite advancements, existing datasets fall short in terms of size and diversity, limiting comprehensive evaluation under diverse conditions. This paper presents an in-depth analysis of the VitalVideo dataset, the largest real-world rPPG dataset to date, encompassing 893 subjects and 6 Fitzpatrick skin tones. Our experimentation with six unsupervised methods and three supervised models demonstrates that datasets comprising a few hundred subjects(i.e., 300 for UBFC-rPPG, 500 for PURE, and 700 for MMPD-Simple) are sufficient for effective rPPG model training. Our findings highlight the importance of diversity and consistency in skin tones for precise performance evaluation across different datasets.
4/9/2024
🌐
Evaluation of Video-Based rPPG in Challenging Environments: Artifact Mitigation and Network Resilience
Nhi Nguyen, Le Nguyen, Honghan Li, Miguel Bordallo L'opez, Constantino 'Alvarez Casado
0
0
Video-based remote photoplethysmography (rPPG) has emerged as a promising technology for non-contact vital sign monitoring, especially under controlled conditions. However, the accurate measurement of vital signs in real-world scenarios faces several challenges, including artifacts induced by videocodecs, low-light noise, degradation, low dynamic range, occlusions, and hardware and network constraints. In this article, we systematically investigate comprehensive investigate these issues, measuring their detrimental effects on the quality of rPPG measurements. Additionally, we propose practical strategies for mitigating these challenges to improve the dependability and resilience of video-based rPPG systems. We detail methods for effective biosignal recovery in the presence of network limitations and present denoising and inpainting techniques aimed at preserving video frame integrity. Through extensive evaluations and direct comparisons, we demonstrate the effectiveness of the approaches in enhancing rPPG measurements under challenging environments, contributing to the development of more reliable and effective remote vital sign monitoring technologies.
5/3/2024