DD-rPPGNet: De-interfering and Descriptive Feature Learning for Unsupervised rPPG Estimation
0
Sign in to get full access
Overview
- Presents a new unsupervised method called DD-rPPGNet for estimating remote photoplethysmography (rPPG) signals from video.
- Addresses the challenge of interference from various sources that can degrade rPPG estimation accuracy.
- Combines a de-interfering module with a descriptive feature learning module to improve unsupervised rPPG estimation.
Plain English Explanation
The paper introduces a new computer vision technique called DD-rPPGNet that can accurately measure a person's heart rate just by looking at a video of their face. This is done through a process called remote photoplethysmography (rPPG), which uses subtle changes in the color of the skin to detect the pulse.
One of the challenges with rPPG is that various interfering factors, like head movements or changes in lighting, can disrupt the signal and make it hard to get an accurate heart rate measurement. DD-rPPGNet addresses this by first "de-interfering" the video to remove these unwanted disturbances. It then uses a "descriptive feature learning" module to extract relevant information about the pulse from the cleaned-up video.
By combining these two techniques, DD-rPPGNet is able to estimate heart rate in an unsupervised way - without needing any labeled training data. This makes it a flexible and practical tool for applications like health monitoring, emotion recognition, and biometric authentication, where being able to unobtrusively measure heart rate from a video feed is valuable.
Technical Explanation
The core of DD-rPPGNet is a two-module architecture:
-
De-interfering Module: This module aims to remove various interfering factors, like head movement and illumination changes, that can corrupt the rPPG signal. It does this by learning a mapping between the raw video frames and a "de-interfered" version that preserves the pulsatile information while suppressing unwanted disturbances. This is achieved through an adversarial training process.
-
Descriptive Feature Learning Module: The de-interfered video frames are then fed into this module, which learns a set of "descriptive" features that are strongly correlated with the underlying rPPG signal. This is done in an unsupervised way by exploiting the local spatial-temporal similarity of the pulse-related information.
By cascading these two modules, DD-rPPGNet is able to produce robust and accurate rPPG estimates without requiring any labeled training data. The authors demonstrate the effectiveness of their approach through experiments on publicly available datasets, showing improvements over prior unsupervised rPPG methods.
Critical Analysis
The authors thoroughly evaluate DD-rPPGNet and compare it to existing unsupervised rPPG approaches, providing a comprehensive technical analysis. However, a few caveats are worth noting:
- The experiments are conducted on controlled datasets, and the model's performance on more unconstrained, real-world scenarios is not addressed.
- The paper does not discuss potential privacy or ethical concerns around the unobtrusive measurement of physiological signals from video.
- While the unsupervised nature of DD-rPPGNet is a strength, it would be interesting to see how it compares to supervised methods when labeled data is available.
Overall, DD-rPPGNet represents a promising advance in unsupervised rPPG estimation, but further research is needed to understand its real-world applicability and implications.
Conclusion
This paper presents DD-rPPGNet, a novel unsupervised approach for estimating rPPG signals from video. By combining de-interfering and descriptive feature learning modules, the method is able to produce robust and accurate heart rate measurements without requiring any labeled training data.
The technical contributions and experimental results demonstrate the potential of DD-rPPGNet to enable unobtrusive physiological monitoring in a wide range of applications, from health tracking to biometric authentication. As the field of remote vital sign estimation continues to evolve, techniques like this will play an increasingly important role in making these capabilities more accessible and practical.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
DD-rPPGNet: De-interfering and Descriptive Feature Learning for Unsupervised rPPG Estimation
Pei-Kai Huang, Tzu-Hsien Chen, Ya-Ting Chan, Kuan-Wen Chen, Chiou-Ting Hsu
Remote Photoplethysmography (rPPG) aims to measure physiological signals and Heart Rate (HR) from facial videos. Recent unsupervised rPPG estimation methods have shown promising potential in estimating rPPG signals from facial regions without relying on ground truth rPPG signals. However, these methods seem oblivious to interference existing in rPPG signals and still result in unsatisfactory performance. In this paper, we propose a novel De-interfered and Descriptive rPPG Estimation Network (DD-rPPGNet) to eliminate the interference within rPPG features for learning genuine rPPG signals. First, we investigate the characteristics of local spatial-temporal similarities of interference and design a novel unsupervised model to estimate the interference. Next, we propose an unsupervised de-interfered method to learn genuine rPPG signals with two stages. In the first stage, we estimate the initial rPPG signals by contrastive learning from both the training data and their augmented counterparts. In the second stage, we use the estimated interference features to derive de-interfered rPPG features and encourage the rPPG signals to be distinct from the interference. In addition, we propose an effective descriptive rPPG feature learning by developing a strong 3D Learnable Descriptive Convolution (3DLDC) to capture the subtle chrominance changes for enhancing rPPG estimation. Extensive experiments conducted on five rPPG benchmark datasets demonstrate that the proposed DD-rPPGNet outperforms previous unsupervised rPPG estimation methods and achieves competitive performances with state-of-the-art supervised rPPG methods.
Read more8/1/2024
🤷
0
Self-similarity Prior Distillation for Unsupervised Remote Physiological Measurement
Xinyu Zhang, Weiyu Sun, Hao Lu, Ying Chen, Yun Ge, Xiaolin Huang, Jie Yuan, Yingcong Chen
Remote photoplethysmography (rPPG) is a noninvasive technique that aims to capture subtle variations in facial pixels caused by changes in blood volume resulting from cardiac activities. Most existing unsupervised methods for rPPG tasks focus on the contrastive learning between samples while neglecting the inherent self-similar prior in physiological signals. In this paper, we propose a Self-Similarity Prior Distillation (SSPD) framework for unsupervised rPPG estimation, which capitalizes on the intrinsic self-similarity of cardiac activities. Specifically, we first introduce a physical-prior embedded augmentation technique to mitigate the effect of various types of noise. Then, we tailor a self-similarity-aware network to extract more reliable self-similar physiological features. Finally, we develop a hierarchical self-distillation paradigm to assist the network in disentangling self-similar physiological patterns from facial videos. Comprehensive experiments demonstrate that the unsupervised SSPD framework achieves comparable or even superior performance compared to the state-of-the-art supervised methods. Meanwhile, SSPD maintains the lowest inference time and computation cost among end-to-end models.
Read more9/24/2024
0
Fully Test-Time rPPG Estimation via Synthetic Signal-Guided Feature Learning
Pei-Kai Huang, Tzu-Hsien Chen, Ya-Ting Chan, Kuan-Wen Chen, Chiou-Ting Hsu
Many remote photoplethysmography (rPPG) estimation models have achieved promising performance in the training domain but often fail to accurately estimate physiological signals or heart rates (HR) in the target domains. Domain generalization (DG) or domain adaptation (DA) techniques are therefore adopted during the offline training stage to adapt the model to either unobserved or observed target domains by utilizing all available source domain data. However, in rPPG estimation problems, the adapted model usually encounters challenges in estimating target data with significant domain variation. In contrast, Test-Time Adaptation (TTA) enables the model to adaptively estimate rPPG signals in various unseen domains by online adapting to unlabeled target data without referring to any source data. In this paper, we first establish a new TTA-rPPG benchmark that encompasses various domain information and HR distributions to simulate the challenges encountered in real-world rPPG estimation. Next, we propose a novel synthetic signal-guided rPPG estimation framework to address the forgetting issue during the TTA stage and to enhance the adaptation capability of the pre-trained rPPG model. To this end, we develop a synthetic signal-guided feature learning method by synthesizing pseudo rPPG signals as pseudo ground truths to guide a conditional generator in generating latent rPPG features. In addition, we design an effective spectral-based entropy minimization technique to encourage the rPPG model to learn new target domain information. Both the generated rPPG features and synthesized rPPG signals prevent the rPPG model from overfitting to target data and forgetting previously acquired knowledge, while also broadly covering various heart rate (HR) distributions. Our extensive experiments on the TTA-rPPG benchmark show that the proposed method achieves superior performance.
Read more8/16/2024
0
Continual Learning for Remote Physiological Measurement: Minimize Forgetting and Simplify Inference
Qian Liang, Yan Chen, Yang Hu
Remote photoplethysmography (rPPG) has gained significant attention in recent years for its ability to extract physiological signals from facial videos. While existing rPPG measurement methods have shown satisfactory performance in intra-dataset and cross-dataset scenarios, they often overlook the incremental learning scenario, where training data is presented sequentially, resulting in the issue of catastrophic forgetting. Meanwhile, most existing class incremental learning approaches are unsuitable for rPPG measurement. In this paper, we present a novel method named ADDP to tackle continual learning for rPPG measurement. We first employ adapter to efficiently finetune the model on new tasks. Then we design domain prototypes that are more applicable to rPPG signal regression than commonly used class prototypes. Based on these prototypes, we propose a feature augmentation strategy to consolidate the past knowledge and an inference simplification strategy to convert potentially forgotten tasks into familiar ones for the model. To evaluate ADDP and enable fair comparisons, we create the first continual learning protocol for rPPG measurement. Comprehensive experiments demonstrate the effectiveness of our method for rPPG continual learning. Source code is available at url{https://github.com/MayYoY/rPPGDIL}
Read more7/22/2024