Meta-experiments: Improving experimentation through experimentation

Overview

• A/B testing is widely used by companies to optimize their customer-facing websites
• Many companies employ experimentation specialists to manage the A/B testing process
• This paper explores the idea of running "meta-experiments" - experiments on the experimentation process itself
• The goal is to improve the A/B testing process through empirical study

Plain English Explanation

A/B testing is a common technique used by many companies to improve their websites and online services. It involves showing different versions of a web page or feature to users and measuring which one performs better. Companies often have teams of experimentation specialists who oversee this process.

This paper explores the idea of taking the A/B testing process itself and making it the subject of experimentation. The researchers call these "meta-experiments" - experiments on the experiment process. The goal is to find ways to make the A/B testing process more effective and efficient.

As an example, the paper discusses a meta-experiment that helped experimenters design A/B tests with sufficient statistical power - meaning the tests were more likely to detect meaningful differences between the variants. This shows how running experiments on the experimentation process can lead to tangible improvements.

The paper also highlights the benefits of the experimentation specialists "dogfooding" - using their own tools and processes to run experiments on themselves. This can provide valuable insights to make the overall experimentation approach stronger.

Technical Explanation

The paper explores the concept of "meta-experiments" - running experiments on the A/B testing process itself in order to improve it. The researchers provide an example of a meta-experiment they conducted that helped experimenters design A/B tests with sufficient statistical power.

The meta-experiment tackled the challenge of ensuring A/B tests were sufficiently powered to detect meaningful differences between variants. The researchers developed a tool to help experimenters calculate the appropriate sample size and duration for their A/B tests, leading to more reliable results.

Additionally, the paper discusses the benefits of "dogfooding" - the experimentation specialists using their own tools and processes to run experiments on themselves. This "eating your own dog food" approach can provide valuable insights to strengthen the overall experimentation framework.

Critical Analysis

The paper presents a compelling case for running meta-experiments to improve the A/B testing process. However, it does not delve deeply into potential pitfalls or limitations of this approach.

For example, the researchers do not address how to handle potential biases that could arise when the experiment designers are also the subjects of the meta-experiments. There may be unconscious motivations to produce favorable results.

Additionally, the paper does not discuss the challenges of scaling meta-experiments across an entire organization, or how to ensure consistency and reliability when multiple teams are running their own meta-experiments.

Further research could explore these areas in more depth, as well as investigate the long-term impacts of meta-experiments on the quality and effectiveness of A/B testing programs over time.

Conclusion

This paper introduces the idea of "meta-experiments" - running experiments on the A/B testing process itself to drive continuous improvement. The researchers provide a concrete example of a meta-experiment that helped experimenters design more reliable A/B tests.

The paper also highlights the value of "dogfooding" - experimentation specialists using their own tools and methods to run experiments on themselves. This can lead to important insights to strengthen the overall experimentation framework.

While the paper makes a compelling case for meta-experiments, it does not fully address potential limitations or scaling challenges. Further research could explore these areas to provide a more holistic understanding of the meta-experiment approach and its long-term impacts on A/B testing programs.