How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps

Overview

• This paper systematically reviews 68 user studies to investigate the effectiveness of saliency maps in helping humans understand how deep neural networks classify images.
• The review found that while saliency maps can enhance human performance in some cases, null effects or even costs are quite common.
• The authors analyzed various factors that may modulate these effects, including the nature of the human tasks, AI performance, explanation methods, images, human participants, and comparison conditions.

Plain English Explanation

Saliency maps are visual tools that highlight the parts of an image that a deep learning model "pays attention to" when making a classification. The idea is that these saliency maps can help humans understand how the model is making its decisions, similar to how explaining AI errors can help humans.

However, this review suggests that the benefits of saliency maps for humans may not be as straightforward as one might expect. In some cases, saliency maps did improve human performance on tasks related to understanding the model's behavior. But in many other cases, there was no improvement or even a negative impact.

The researchers looked at a number of factors that might influence whether saliency maps are useful. For example, they found that saliency maps tended to be more beneficial when the task was focused on understanding the model, rather than just classifying the images. They also found that the effects depended on whether the model's prediction was correct or incorrect.

Interestingly, the specific explanation method used to generate the saliency maps didn't seem to have much impact. And the effects were quite variable depending on the comparison conditions used in the studies.

Overall, this review suggests that the value of saliency maps for human understanding is not as clear-cut as it might seem. The benefits appear to depend on the details of the task and the model's performance, rather than just the saliency maps themselves.

Technical Explanation

This systematic review examines 68 user studies that investigated the effectiveness of saliency maps in helping humans understand how deep neural networks classify images. Saliency maps are visual representations that highlight the regions of an image that a model pays the most attention to when making a classification.

The authors organized the empirical outcomes of these studies along several factors, including:

• The nature of the human tasks (e.g. AI-focused vs. image-focused)
• The performance of the AI model (correct vs. incorrect predictions)
• The specific XAI methods used to generate the saliency maps
• The characteristics of the images being classified
• The demographics of the human participants
• The comparison conditions used in the studies

They found that while saliency maps can enhance human performance in some cases, null effects or even costs (i.e. worse performance) are quite common.

In tasks focused on understanding the AI model's behavior, benefits of saliency maps were more common. But in image-focused tasks (e.g. classifying the images themselves), the effects were more mixed and often dependent on the specific cognitive requirements of the task.

Interestingly, the specific XAI method used to generate the saliency maps had surprisingly little impact on the outcomes. The effects were also highly dependent on the comparison conditions used in the studies.

The evidence was more limited for factors related to the images and human participants. But the impacts of these factors tended to be quite variable and context-dependent.

Critical Analysis

This systematic review provides a nuanced and insightful look at the potential benefits and limitations of using saliency maps to help humans understand deep learning models. The authors did a commendable job of carefully analyzing a large body of prior research to identify the key factors that can influence the effectiveness of this explanatory technique.

One strength of the paper is its balanced, objective tone. The authors acknowledge the complexity of the issue and don't overstate the conclusions. They rightly point out that the effects of saliency maps appear to be highly dependent on the specific context and research design.

That said, the review also highlights some gaps in the existing literature that warrant further investigation. For example, the limited evidence on how image and human characteristics impact the usefulness of saliency maps suggests a need for more targeted studies in these areas.

Additionally, while the authors touch on the importance of the comparison conditions used in the studies, they don't delve too deeply into the potential issues with the experimental designs. It would be helpful to see a more critical exploration of methodological factors that could be influencing the inconsistent findings.

Overall, this paper makes a valuable contribution by shedding light on the complex relationship between saliency maps and human understanding of AI systems. The results caution against overly simplistic assumptions about the explanatory power of these visualizations and point to the need for more nuanced, context-dependent approaches to XAI.

Conclusion

This systematic review casts doubt on the assumption that saliency maps are inherently useful for helping humans understand how deep neural networks classify images. While saliency maps can enhance human performance in some cases, the benefits appear to be highly dependent on factors like the specific task, the model's performance, and the comparison conditions used.

The findings suggest that the value of saliency maps may be more limited than previously thought, at least in their current form. Researchers and practitioners should be cautious about relying on saliency maps as a universal solution for explainable AI, and instead focus on developing more contextualized, user-centric approaches to model interpretability.

Overall, this paper highlights the importance of carefully evaluating the real-world effectiveness of XAI techniques through rigorous user studies. It sets the stage for further research to better understand the conditions under which saliency maps and other explanatory tools can meaningfully support human understanding of complex AI systems.