Sample Observed Effects: Enumeration, Randomization and Generalization

Overview

• The paper proposes a new "Combinatorial" definition for the external validity (EV) of intervention effects, which aims to address the limitations of the widely used "Counterfactual" definition.
• The Combinatorial definition focuses on the concept of an effect observation "background" and formulates conditions for effect generalization based on observed and unobserved backgrounds.
• The approach reveals two key limits for effect generalization: (1) when effects are observed under all their enumerable backgrounds, or (2) when backgrounds have become sufficiently randomized.
• The authors use this Combinatorial framework to re-examine several issues in the original Counterfactual formulation, such as out-of-sample validity, concurrent estimation of multiple effects, bias-variance tradeoffs, statistical power, and connections to current predictive and explanatory techniques.

Plain English Explanation

The paper introduces a new way to think about the "external validity" of research findings - that is, how well the results from a study can be applied to different situations or populations beyond the original study.

The traditional "Counterfactual" approach to this issue has focused on ensuring the results are unbiased and accurate within the original study. However, the authors argue that this definition doesn't fully capture the ability to generalize the findings to new contexts.

Their "Combinatorial" approach instead looks at the "background" conditions under which the effects were observed. This could include things like the characteristics of the study participants, the setting, the timing, and other factors that might influence the results.

The key insight is that for effects to be truly generalizable, they need to be observed under all the possible "backgrounds" that might be relevant, or the backgrounds need to be sufficiently randomized. This provides two routes to establishing external validity.

Using this new framework, the authors re-examine several important issues in causal inference, such as how to handle out-of-sample data, estimate multiple effects concurrently, manage the tradeoffs between bias and variance, and leverage modern machine learning techniques.

Overall, this paper offers a fresh perspective on a fundamental challenge in empirical research - how to ensure that the findings from one study can be meaningfully applied in the real world.

Technical Explanation

The paper introduces a "Combinatorial" definition of external validity (EV) for intervention effects, as an alternative to the widely used "Counterfactual" definition. The Counterfactual approach is focused on ensuring unbiasedness and accuracy within the original study sample, but does not fully capture the ability to generalize the findings to new contexts.

The Combinatorial definition centers on the concept of an effect observation "background" - the set of observed and unobserved factors that may influence the intervention effects. The authors formulate conditions for effect generalization based on the enumeration and randomization of these backgrounds.

Specifically, the paper shows that effect generalization is possible in two cases: (1) when effects are observed under all their enumerable backgrounds, or (2) when backgrounds have become sufficiently randomized. This provides a new perspective on issues like out-of-sample validity, concurrent estimation of multiple effects, bias-variance tradeoffs, statistical power, and connections to modern predictive and explanatory techniques.

Methodologically, the Combinatorial framework allows the authors to reframe the parametric estimation problems of the Counterfactual approach as combinatorial enumeration and randomization problems in non-experimental samples. They use this non-parametric approach to demonstrate tradeoffs in the performance of popular supervised, explaining, and causal-effect estimators.

The paper also illustrates how the Combinatorial framework can enable the use of supervised and explanatory methods in non-i.i.d. (independent and identically distributed) samples - an increasingly important capability given the COVID-19 pandemic's impact on data availability and distribution.

Critical Analysis

The paper makes a valuable contribution by introducing a new perspective on the external validity of intervention effects. The Combinatorial definition provides a more nuanced and contextual understanding of generalizability, moving beyond the limitations of the traditional Counterfactual approach.

One potential limitation is the reliance on the ability to enumerate or sufficiently randomize the "backgrounds" of effect observations. In practice, this may be challenging, especially for complex real-world settings with many unobserved variables. The authors acknowledge this and discuss potential strategies, but further research may be needed to address these challenges.

Additionally, while the paper demonstrates the application of the Combinatorial framework to various issues in causal inference, it would be helpful to see more detailed case studies or empirical evaluations to fully assess the practical implications and tradeoffs of this approach.

Overall, this paper offers a thought-provoking and methodologically rigorous approach to a critical issue in empirical research. By shifting the focus to the contextual factors that shape intervention effects, it encourages researchers to think more deeply about the conditions under which their findings can be reliably applied beyond the original study.

Conclusion

This paper proposes a novel "Combinatorial" definition of external validity (EV) for intervention effects, which aims to address the limitations of the widely used Counterfactual approach. The key innovation is the focus on the "background" conditions that influence the observed effects, and the formulation of two routes to establishing generalizability: enumerating all relevant backgrounds or achieving sufficient randomization.

This Combinatorial framework allows the authors to re-examine several important issues in causal inference, such as out-of-sample validity, concurrent estimation of multiple effects, bias-variance tradeoffs, and the use of modern predictive and explanatory techniques. By shifting the perspective to the contextual factors that shape intervention effects, this paper offers a valuable contribution to the ongoing efforts to improve the real-world relevance and applicability of empirical research findings.