Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AI

Overview

• Facial expression recognition is crucial for understanding human behavior.
• Deep learning has enabled models that can outperform humans in this task.
• However, it's unclear how closely these deep neural networks mimic human perception.
• This study aims to explore the similarity between deep neural networks and human perception of facial expressions.

Plain English Explanation

Facial expressions are an important way that humans communicate and understand each other's emotions. Deep learning models have become very good at recognizing facial expressions, even better than humans in some cases. But it's not clear how these deep learning models actually "see" and process facial expressions compared to how humans do it.

This study looked at 12 different deep learning models, both general object classifiers and models specifically designed for recognizing facial expressions. The researchers used a special technique to generate heatmaps that showed which parts of the face were most important for each model when recognizing different facial expressions.

They then compared these heatmaps to the facial regions that are known to be important for human perception of facial expressions, based on previous research. The goal was to see how similar the deep learning models' processing was to human perception.

Technical Explanation

The researchers employed an explainable AI method to generate heatmaps revealing the crucial facial regions used by 12 different deep neural networks when classifying 6 basic facial expressions. This allowed them to assess how closely the models' processing aligned with human perception of facial expressions.

They quantitatively compared the heatmaps to ground truth masks based on Friesen and Ekman's well-established description of facial expression cues. Metrics used included Intersection over Union (IoU) and normalized correlation coefficients. Qualitatively, they observed that models with pre-trained weights showed more similarity in heatmaps compared to those without pre-training.

Across all expressions and architectures, the average IoU was low at 0.2702, indicating limited alignment between the models and human perception. The best-performing architecture averaged 0.3269, while the worst-averaged 0.2066. Dendrograms based on normalized correlation revealed two main clusters - models with and without pre-training.

Critical Analysis

The findings suggest that current deep learning models for facial expression recognition, while highly capable, do not closely mimic human perceptual processing. The researchers note that network architecture seems to influence the level of similarity, as models with similar architectures prioritized similar facial regions.

This work highlights the importance of not solely relying on model performance metrics, but also examining the internal workings and decision-making processes of these systems. Without this deeper understanding, it's difficult to have confidence that they are truly replicating human-level intelligence and decision-making.

Further research is needed to better understand the factors that drive the divergence between machine and human facial expression processing. Exploring model architectures, training data, and other design choices could yield insights to bring these systems closer to human-level perception and reasoning.

Conclusion

This study provides valuable insights into the limitations of current deep learning models for facial expression recognition. While these models can outperform humans on the task, they do not closely mirror human perceptual processing of facial cues.

Understanding these differences is crucial as we develop AI systems intended to interact with and understand humans. Bridging the gap between machine and human facial expression processing could lead to more natural, intuitive, and trustworthy human-AI interactions in the future.