Back

A/B Testing, Design and Analysis

95-819

Units: 6

Description

This class teaches methods for measuring the true impact of business changes in the digital economy. Because of the prevalence of A/B testing in practices, we especially emphasize how A/B tests work and how to apply them. We will have hands-on discussions of how managers can measure impact in business situations and how to evaluate claims of impact made by others. While the focus will lie on A/B testing, we will also discuss other methods that can be used when experiments are not feasible. Methods are illustrated with examples from a variety of digital businesses including Microsoft, Amazon, eBay and Uber. We will cover examples in a variety of topical areas including e-mail marketing, technology adoption, advertising, pricing, among others.

Guest speakers from the New York Times and Spotify will discuss the practical importance and challenges of implementing A/B tests.

Learning Outcomes

At the end of the course, students should be able to:

1. Explain why measuring impact by using the correct causal methods is important
2. Recognize whether claims are causal when they encounter them in the real world and to be appropriately skeptical of claims.
3. Identify the conditions that allow one to learn a causal effect from data
4. Describe why experimentation solves the causal inference problem and identify when natural experiments can approximate actual experiments
5. Implement field experiments in a variety of empirical contexts, such as advertising or pricing.
6. Understand the challenges of why firms do not experiment more often and how to potentially overcome those.

Prerequisites Description

The course assumes that you have a working understanding of basic statistical concepts such as hypothesis testing, statistical significance, t-tests and regression. I build on this knowledge and will not be able to teach it from scratch. If you do not have knowledge or are uncertain how much you recall, I strongly encourage you to review these concepts prior to the start of the class.

The course likewise assumes that you are able to manipulate data and use software to plot data, conduct t-tests and run regressions. Again, if you are unsure how to do so or how much you remember, you should familiarize yourself with the techniques prior to class. I do not require that you use a specific statistical software (you can use Stata, R, Excel, or other tools) but you will be asked to document how you came to your results.