Metrizing Fairness

Overview

• The paper explores supervised learning problems for predicting properties of individuals who belong to one of two demographic groups.
• The goal is to find predictors that are fair according to the principle of statistical parity, meaning the distributions of the predictions within the two groups should be similar.
• The paper showcases the benefits of using integral probability metrics (IPMs) other than the Kolmogorov distance to measure unfairness.

Plain English Explanation

The paper looks at a common problem in machine learning where you want to predict some property of individuals, and those individuals belong to one of two different demographic groups. For example, you might want to predict someone's income level, and the individuals could be divided into two groups based on their gender or race.

The key challenge is to make these predictions in a fair way, where the predictions for the two groups are similar. This is known as achieving statistical parity - the distributions of the predictions should be close between the two groups.

Traditionally, researchers have used the Kolmogorov distance to measure how different the prediction distributions are between the groups, and then tried to minimize this distance in the machine learning model. This paper introduces the use of other types of distance metrics, called integral probability metrics (IPMs), to measure unfairness.

The paper shows that these alternative IPM metrics have some conceptual and computational benefits over the Kolmogorov distance. Conceptually, they can be interpreted as measuring the divergence in "expected utility" between the two groups. Computationally, certain IPMs allow for unbiased gradient estimates, which means the machine learning model can be trained more efficiently using stochastic gradient descent.

The paper also identifies conditions where achieving statistical parity can actually improve the overall prediction accuracy, beyond just making the model fair. This is an important finding, as it suggests there can be benefits to fairness beyond just meeting ethical or regulatory requirements.

Technical Explanation

The core of the paper is exploring the use of integral probability metrics (IPMs) to measure and optimize for fairness in supervised learning problems with two demographic groups.

Traditionally, the Kolmogorov distance has been used to quantify the unfairness between the prediction distributions of the two groups, and this unfairness metric is then incorporated into the machine learning model's objective function. The paper shows that other IPM metrics, such as the squared L2 distance and squared maximum mean discrepancy, have conceptual and computational advantages over the Kolmogorov distance.

Conceptually, the authors prove that any IPM can be interpreted as measuring the divergence in expected utility between the two demographic groups. This provides an intuitive interpretation of unfairness. Computationally, they show that for certain IPMs, the unfairness-regularized prediction loss admits unbiased gradient estimators, making the optimization problem amenable to efficient stochastic gradient descent (SGD) algorithms.

The paper presents extensive numerical experiments on real-world data, demonstrating that these SGD-based fair learning algorithms can achieve superior accuracy-unfairness trade-offs compared to state-of-the-art fair learning methods - sometimes orders of magnitude faster.

Finally, the authors identify conditions under which statistical parity (i.e., similar prediction distributions between groups) can actually improve prediction accuracy. This is an important finding, as it suggests fairness can have benefits beyond just ethical or regulatory requirements.

Critical Analysis

The paper makes a compelling case for the use of IPMs as a more flexible and powerful way to measure and optimize for fairness in supervised learning problems. The conceptual and computational advantages they identify are well-justified and supported by the theoretical analysis and empirical results.

However, it's worth noting that the paper focuses solely on the binary classification setting. While the authors mention the potential extension to multiclass problems, further research would be needed to validate the effectiveness of IPMs in those more complex scenarios.

Additionally, the paper does not address the issue of individual fairness, which is another important fairness criterion beyond just group-level statistical parity. Techniques like reweighting or hyperparameter tuning may be necessary to achieve individual fairness alongside group-level fairness.

Finally, the paper's findings on the potential accuracy benefits of statistical parity are intriguing, but the authors acknowledge that the conditions for this to hold may be fairly narrow. [Further research is needed to fully understand the intrinsic fairness-accuracy tradeoffs and the circumstances under which fairness can lead to improved predictive performance.

Conclusion

This paper makes a significant contribution to the field of fair machine learning by introducing the use of IPMs as a flexible and powerful way to measure and optimize for fairness in supervised learning problems. The conceptual and computational advantages of IPMs over traditional Kolmogorov-based approaches are well-demonstrated, and the authors' findings on the potential accuracy benefits of statistical parity are an important step towards understanding the interplay between fairness and predictive performance.

While the paper is focused on the binary classification setting, the insights and techniques it presents have the potential to be extended and applied to a wide range of fair machine learning applications, further advancing the field's ability to develop models that are both accurate and equitable.