Latent Embedding Clustering for Occlusion Robust Head Pose Estimation

INTRODUCTION

Head pose estimation is the process of predicting the orientation and position of a person's head relative to a camera. This is an important topic in computer vision, as head pose estimation is crucial for many applications, such as human-computer/robot interaction, surveillance systems, driver attention monitoring, virtual/augmented reality, healthcare, and marketing.

One major challenge that existing head pose estimation methods often struggle with is the presence of occlusions. Occlusions can be caused by external objects, facial accessories, or even body parts, and they can make it difficult to capture reliable facial features, leading to inaccurate head pose estimates. This is a significant issue in real-world, unconstrained environments, and it has not been investigated in detail.

This paper proposes a new deep learning methodology to improve the robustness of human pose estimation (HPE) to occlusions. The key innovations are:

1. Combining fine-grained Euler angle regression with unsupervised latent embedding clustering to refine the feature representation for pose estimation. This approach requires significantly fewer ground truth latent embedding points compared to prior latent space regression methods.

2. The ability to augment the occluded training dataset with non-occluded images, overcoming the limitation of prior methods that require occluded/non-occluded image pairs.

3. Lower computational cost compared to fully supervised latent space regression approaches.

The proposed method achieves state-of-the-art results on occluded images in benchmark datasets, while also performing competitively on non-occluded images. An ablation study demonstrates the impact of the unsupervised embedding clustering component.

RELATED WORK

This section reviews the existing literature on head pose estimation (HPE) and unsupervised deep clustering. Regarding HPE, there are two main classes of approaches: those based on facial landmarks and model fitting, and those using deep learning on image features without the need for landmarks. The landmark-based approaches fit a head mesh to detected facial landmarks or keypoints, with some recent works using 3D morphable models to improve generalization. The deep learning approaches avoid the need for landmarks and can directly predict head pose, with recent works proposing solutions to address common HPE problems like ambiguity in rotation labels and perspective distortion.

The literature on occlusions in HPE is limited, but a few works have tackled this problem by using optical flow tracking, estimating landmark visibility probabilities, or combining landmark, pose, and deformation estimation. Some works have also explored the use of synthetically occluded datasets and multi-loss functions to improve pose prediction in the presence of occlusions.

The section also discusses unsupervised deep clustering, which aims to group data without ground truth labels based on feature similarity. Simple methods like k-means can struggle with high-dimensional data, so deep learning approaches that map data to a lower-dimensional feature space have become popular. Recent deep clustering works incorporate both clustering and reconstruction losses to preserve local data structure, and use augmented data and contrastive losses to maximize the similarity of positive pairs and penalize negative ones.

Methodology

The proposed LEC-HPE methodology aims to address the difficulties mentioned in Section I. It uses a small number of ground truth latent embeddings (K) to obtain the representation of a larger number of images (N), where K << N. To accomplish this, the method uses unsupervised latent embedding clustering motivated by previous work.

The overall LEC-HPE architecture includes a backbone encoder and four separate branches. Three branches are used for predicting each Euler angle (yaw, pitch, roll) using a multi-loss framework for classification and fine-grained estimation. The remaining branch is responsible for clustering and fine-tuning the latent space.

The training strategy has two stages. In the first stage, the model is optimized for bin prediction using classification loss and fine-grained Euler angle estimation using regression loss. In the second stage, a clustering term is added while maintaining the multi-loss functions from the first stage.

The fine-grained losses for feature learning and preservation include a cross-entropy loss for classification and a mean squared error loss for regression. These losses are combined to yield the final loss for each Euler angle.

The unsupervised latent clustering is performed by measuring the pairwise similarity between latent embedded points and cluster centers, and optimizing the cluster centers and encoder parameters by minimizing a Kullback-Leibler divergence objective.

The overall loss is the sum of the Euler angle losses and the clustering loss, with a regularization coefficient to adjust the impact of the clustering term.

V Experimental Evaluation

This section describes the extensive experimental evaluation of the proposed head pose estimation (HPE) method using several benchmark datasets. The datasets used include:

• 300W-LP: A synthetic dataset with 61,225 face samples used for training. It covers a wide range of individuals, illumination conditions, and poses.
• BIWI: A dataset with over 15,000 images of 20 individuals used for testing. It provides depth, RGB images, and ground truth pose annotations.
• AFLW2000: A dataset of 2,000 in-the-wild images used for testing, with diverse head poses and variations in lighting and background.
• Pandora: A dataset with over 250,000 RGB and depth images, simulating driving poses, used for testing with real-life occlusions.

The network structure uses a ResNet-50 backbone encoder. The training is done in two stages: (1) using the original 300W-LP dataset, and (2) using the synthetically occluded version of the same dataset.

The paper compares the proposed method's performance to state-of-the-art approaches on the BIWI, AFLW2000, and Pandora datasets, both in original and occluded scenarios. The results show the proposed method delivers competitive performance, surpassing other methods in occluded scenarios by a significant margin.

An ablation study is also conducted to analyze the impact of the clustering term in the training loss. The results indicate that including the clustering term (with an optimal β value of 100) reduces the estimation error for both occluded and non-occluded images compared to not using the clustering term.

Conclusion

The paper presents an efficient method to address the challenge of occlusion in head pose estimation, a significant issue in computer vision. The proposed framework, called LEC-HPE, combines unsupervised latent embedding clustering in the latent feature space with a fine-grained Euler angle multi-loss scheme to improve occlusion robustness. The key idea is to enhance feature representation for pose estimation while refining the latent embedding space through clustering, eliminating the need for labeled embedding data for each training image. This approach offers a more efficient alternative compared to recent occlusion-focused state-of-the-art methods, without requiring a constrained expansion of the training dataset. Experiments show the method can achieve similar results without the need for ground truth for each latent embedding label and surpass the state of the art for standard head pose estimation in occluded images. An ablation study quantifies the impact of the clustering term and the need to include the fine-grained Euler angles scheme. Future work will include automatic selection of the optimal number of cluster centroids and evaluation with smaller, more efficient backbone encoders for low-power applications. The use of clustering losses for the classification component in the multi-loss scheme will also be explored.

Acknowledgements

The work described in this paper was funded by several sources. These include LARSyS, which provided funding through the specified DOI numbers. Funding also came from the Fundação para a Ciência e a Tecnologia. Additionally, the SmartRetail project, managed by IAPMEI - Agência para a Competitividade e Inovação, provided funding through the specified project number.