Randomized Experiments

Why Does This Exist?

In observational data, you can only control for the confounders you measured. If you believe income and age confound the relationship between a marketing email and a purchase, you can add them to your model. But what if "boredom" is also a confounder? What if "propensity to check email at 3 AM" is a confounder? If you didn't measure them, your estimate of the email's effect will be biased, and no amount of math can save it.

Randomized Controlled Trials (RCTs)—or A/B tests—exist to solve the problem of unmeasured confounders. By taking the decision of who gets the treatment completely out of the hands of humans, nature, and algorithms, and handing it to a random number generator, we force the treatment assignment to be completely independent of every other variable in the universe.

Think of It Like This

Imagine trying to figure out if wearing a wetsuit makes you swim faster.

How It Actually Works

In the language of Potential Outcomes, we want to estimate the Average Treatment Effect (ATE): $\text{ATE} = \mathbb{E}[Y(1)] - \mathbb{E}[Y(0)]$

We can only estimate this using the simple difference in observed means ($\mathbb{E}[Y | T=1] - \mathbb{E}[Y | T=0]$) if the treatment $T$ is independent of the potential outcomes: $(Y(0), Y(1)) \perp \!\!\! \perp T$. This property is called Exchangeability. It means the treated group and the control group are completely interchangeable; if we swapped their treatments, their average outcomes would swap perfectly.

In the language of Causal Graphs, a confounder $Z$ creates a backdoor path $T \leftarrow Z \rightarrow Y$. When we randomize $T$, we are physically intervening in the system. The equation that normally determines $T$ is replaced by a coin flip: $T = \text{Random}()$. Because $T$ now listens only to the coin, the arrow $Z \rightarrow T$ is deleted from the graph. The backdoor path is cut.

The Power of Randomization

Randomization achieves identification—the ability to calculate a causal effect from data—without requiring us to assume we know all the confounders. It handles:

1. Measured confounders: Income, age, location.
2. Unmeasured confounders: Mood, hidden preferences, systemic bias.

Because the coin flip distributes all these traits equally (on average, given a large enough sample size) between the treatment and control groups, the only remaining difference between the groups is the treatment itself.

Show Me the Code

Let's simulate a scenario with a massive unobserved confounder, and see how observational regression fails while randomization succeeds perfectly.

1import numpy as np2import statsmodels.api as sm3
4np.random.seed(42)5N = 50006
7# 'Hidden Motivation' is a massive confounder we cannot measure8hidden_motivation = np.random.normal(0, 1, N)9
10# Treatment: Using an AI study tool (T=1) or not (T=0)11# Outcome: Final exam score (Y)12
13# True causal effect: The tool adds exactly +2 points to the score14true_effect = 2.015
16# Outcome is driven by baseline (70), the true effect, motivation, and noise17Y_0 = 70 + 5 * hidden_motivation + np.random.normal(0, 2, N)18Y_1 = Y_0 + true_effect19
20# --- Observational Data ---21# Highly motivated students are much more likely to use the tool22prob_T_obs = 1 / (1 + np.exp(-2 * hidden_motivation))23T_obs = np.random.binomial(1, prob_T_obs)24Y_obs = np.where(T_obs == 1, Y_1, Y_0)25
26# Naive regression on observational data27X_obs = sm.add_constant(T_obs)28model_obs = sm.OLS(Y_obs, X_obs).fit()29print(f"Observational Estimate: +{model_obs.params[1]:.2f} points")30# -> +10.15 points. WRONG! It credits the tool for the students' motivation.31
32# --- Randomized Experiment (A/B Test) ---33# We force 50% of students to use the tool, 50% to not, completely randomly34T_rand = np.random.binomial(1, 0.5, N)35Y_rand = np.where(T_rand == 1, Y_1, Y_0)36
37# Naive regression on randomized data38X_rand = sm.add_constant(T_rand)39model_rand = sm.OLS(Y_rand, X_rand).fit()40print(f"Randomized Estimate: +{model_rand.params[1]:.2f} points")41# -> +2.07 points. CORRECT! Randomization severed the link to motivation.

Watch Out For

The Quick Version

• Observational causal inference requires you to measure and control for every single confounder. This is usually impossible.
• Randomization solves this by breaking the causal arrow from the confounder to the treatment assignment.
• By assigning treatment via a coin flip, you ensure the treatment and control groups are exchangeable, meaning any difference in their outcomes must be caused exclusively by the treatment.
• In causal graph terms, randomization is a structural intervention that deletes all incoming arrows to the treatment node.

What to Read Next

• ab-testing-for-ml — How to design, run, and measure randomized experiments in a production ML context.
• sequential-testing — Why you can't just stop an A/B test the moment it looks statistically significant, and what to do instead.
• instrumental-variables — The technique used when you run a randomized experiment but people refuse to comply with their assigned group.