Domain-Adaptive Full-Face Gaze Estimation via Novel-View-Synthesis and Feature Disentanglement

Overview

• The paper discusses the challenge of cross-domain gaze estimation, where gaze estimation models perform well within the same domain but struggle when applied to different domains.
• The key factors believed to impact performance are the range of head poses and gaze directions in the training data.
• The proposed approach includes a data synthesis pipeline to expand the range of head poses, and an unsupervised domain adaptation method to bridge the gap between synthetic and real data.

Plain English Explanation

Gaze estimation, the process of determining where someone is looking, has seen significant progress thanks to deep learning. However, these models tend to work well only when the training and testing data come from the same domain, such as the same camera setup or environment.

When applied to different domains, the performance drops dramatically. This is because factors like the range of head poses and gaze directions in the training data can greatly impact the model's accuracy.

Collecting a large, diverse dataset with extensive head pose and gaze variations is expensive and time-consuming. To address this, the researchers developed a two-part solution:

1. Data synthesis: They use single-image 3D reconstruction to generate synthetic images with a wider range of head poses, without needing a 3D facial shape dataset.

2. Unsupervised domain adaptation: They propose a method to bridge the gap between the synthetic and real images, allowing the model trained on the synthetic data to perform well on real-world data.

The goal is to create a gaze estimation system that can be deployed in various real-world applications, even when the training and test data come from different sources.

Technical Explanation

The paper presents a comprehensive pipeline for training gaze estimation models that can perform well across different domains.

The data synthesis component leverages single-image 3D reconstruction techniques, as described in Domain-Transferred Synthetic Data Generation for Improving Monocular 3D Hand Pose Estimation, to generate synthetic full-face images with a wider range of head poses. This eliminates the need for a 3D facial shape dataset, which can be expensive to acquire.

To bridge the gap between the synthetic and real data, the researchers propose an unsupervised domain adaptation method inspired by Domain Adaptive Pose Estimation via Multi-Level Consistency Regularization. Their approach uses a disentangling autoencoder network to separate gaze-related features from background information, and introduces a background augmentation consistency loss to better utilize the characteristics of the synthetic source domain.

Comprehensive experiments show that the model trained solely on the synthetic data can perform comparably to one trained on real data with a large label range. The proposed domain adaptation method further improves the performance on multiple target domains.

Critical Analysis

The researchers acknowledge that their synthetic data generation approach may not capture all the nuances of real-world data, and that the domain adaptation method still has room for improvement. Additionally, the paper does not address the potential for bias in the synthetic data or the implications of deploying such a system in real-world applications.

While the results are promising, further research is needed to ensure the robustness and fairness of the gaze estimation system, especially when applied to diverse populations and environments. Strategies like learning unsupervised gaze representation via eye mask and power of data augmentation for head pose estimation could be explored to address these concerns.

Conclusion

This paper presents an innovative approach to address the challenge of cross-domain gaze estimation, a crucial step towards deploying gaze-aware systems in real-world applications. By combining synthetic data generation and unsupervised domain adaptation, the researchers have developed a method that can perform well even when the training and test data come from different domains.

While the results are promising, further research is needed to ensure the robustness and fairness of the gaze estimation system. Nonetheless, this work represents an important advancement in the field and opens up new possibilities for gaze-based applications, such as learning gaze-aware compositional GAN for image synthesis.