Robust Design and Evaluation of Predictive Algorithms under Unobserved Confounding

Overview

• This paper proposes a unified framework for designing and evaluating predictive algorithms in situations where the outcome is selectively observed based on choices made by human decision makers.
• The authors introduce general assumptions about how much the outcome may vary on average between unselected and selected units, given observed covariates and identified nuisance parameters.
• They develop debiased machine learning estimators to calculate bounds on various predictive performance measures, such as likelihood, mean squared error, and true/false positive rates, under these assumptions.
• The paper illustrates how varying assumptions about unobserved confounding can lead to meaningful changes in default risk predictions and evaluations of credit scores across sensitive groups, using administrative data from a large Australian financial institution.

Plain English Explanation

Predictive algorithms are used to make important decisions in many real-world settings, such as lending or insurance. However, the data used to train and evaluate these algorithms may be biased because the outcomes are only observed for the people who were selected for the relevant action (e.g., who received a loan).

This paper proposes a way to account for this "selective observation" when designing and evaluating predictive algorithms. The authors introduce general assumptions about how much the outcomes might differ on average between the selected and unselected groups, based on the observed information. Using these assumptions, they develop new statistical methods to calculate bounds on how well the predictive algorithms are performing, in terms of metrics like accuracy, error rates, and fairness across different groups.

The paper demonstrates how these methods can lead to different conclusions about the performance and fairness of credit risk prediction models, compared to standard approaches that don't account for selective observation. By being more careful about potential biases in the data, the researchers can get a better understanding of how well these algorithms are really working in practice.

Technical Explanation

The key innovation in this paper is a unified framework for designing and evaluating predictive algorithms in situations with selectively observed data. The authors introduce general assumptions about how much the outcome may vary on average between unselected and selected units, conditional on observed covariates and identified nuisance parameters.

These assumptions formalize popular strategies for handling missing data, such as using proxy outcomes or instrumental variables. The authors then develop debiased machine learning estimators to calculate bounds on a wide range of predictive performance measures, including likelihood, mean squared error, and true/false positive rates.

To illustrate their framework, the researchers analyze an administrative dataset from a large Australian financial institution. They show how varying the assumptions about unobserved confounding can lead to meaningful changes in default risk predictions and evaluations of credit scores across sensitive groups.

Critical Analysis

The authors acknowledge several limitations and areas for further research. First, the general assumptions they introduce about unobserved confounding may not hold in all real-world situations, and the resulting bounds may still be quite wide. More investigation is needed into how sensitive the results are to the specific assumptions made.

Additionally, the paper focuses on predictive performance metrics, but does not directly address other important considerations like model interpretability or deployment feasibility. Further work is needed to understand how this framework integrates with the broader challenges of responsible AI development and deployment.

Another potential issue is the reliance on debiased machine learning estimators, which can be computationally intensive and may require careful tuning. The practicality of applying these methods at scale in real-world settings remains an open question.

Overall, this paper represents an important step forward in addressing selective observation bias in predictive algorithms. By making the underlying assumptions explicit and developing principled ways to bound performance, the authors provide a useful toolbox for researchers and practitioners working to build more robust and fair predictive systems.

Conclusion

This paper presents a unified framework for designing and evaluating predictive algorithms in the presence of selectively observed data. By formalizing assumptions about unobserved confounding, the authors develop new statistical methods to calculate bounds on a variety of predictive performance metrics.

The demonstrated ability to uncover meaningful differences in default risk predictions and credit score evaluations, compared to standard approaches, highlights the importance of accounting for selective observation bias. As predictive algorithms continue to inform high-stakes decisions in areas like lending and insurance, this work represents a valuable contribution toward building more transparent, robust, and fair AI systems.