Causal Analysis of Shapley Values: Conditional vs. Marginal

Overview

• Shapley values are a popular tool for explaining machine learning models
• Two common approaches, conditional and marginal, can lead to different results when features are correlated
• This has led to conflicting recommendations in the literature about which approach to use

Plain English Explanation

Shapley values are a way to understand how much each "feature" or input to a machine learning model contributes to the final prediction. This is useful for explaining why a model made a certain decision.

The two main ways to calculate Shapley values are the conditional approach and the marginal approach. However, when the features in the model are correlated, these two methods can give different results. This has caused confusion, as different researchers have recommended using different approaches.

The paper aims to resolve this controversy by looking at the problem from a causal perspective. It shows that the differences arise from the assumptions each method makes about missing causal information. The paper concludes that the marginal approach is the better choice.

Technical Explanation

The paper examines the two main methods for calculating Shapley values - the conditional approach and the marginal approach. It shows that when features in the model are correlated, these methods can produce different results.

The authors use causal arguments to analyze the underlying assumptions of each method. They demonstrate that the conditional approach is fundamentally flawed from a causal perspective, as it makes unsound assumptions to deal with missing causal information.

In contrast, the marginal approach is shown to be more aligned with causal principles. Combined with previous work, this leads the authors to conclude that the marginal approach should be preferred over the conditional one for explaining machine learning models.

Critical Analysis

The paper provides a rigorous causal analysis of the differences between the conditional and marginal approaches to calculating Shapley values. This is a valuable contribution, as it helps resolve the conflicting recommendations in the literature.

However, the paper does not discuss the practical implications of their findings. It would be helpful to see examples of how the choice of approach can affect the explanations provided to users of machine learning models in real-world scenarios.

Additionally, the paper focuses solely on the causal issues with the conditional approach. It would be interesting to see if there are other theoretical or empirical considerations that might inform the choice between the two methods.

Conclusion

This paper offers a principled, causal perspective on the ongoing debate around calculating Shapley values for explaining machine learning models. By demonstrating the fundamental issues with the conditional approach, the authors make a compelling case for preferring the marginal method.

While there may be additional factors to consider, this research represents an important step forward in providing guidance on how to best interpret the contributions of different features to a model's predictions. As machine learning systems become more widespread, tools like Shapley values will be crucial for building trust and accountability.