Every “Variant B increased conversions by 12%!” headline you’ve seen rests on a statistical test, and most of the ways those tests get misused are quiet and easy to miss — nobody’s lying, the math just gets applied a little wrong. This post builds the actual test up from its assumptions, then — instead of asking you to trust the formulas — runs thousands of simulated A/B tests where the true answer is known in advance, so you can watch the theory’s claims actually hold (or fail to) in the numbers.

1. The setup

A typical A/B test: visitors are randomly split between a control (A) and a variant (B), and you measure how many in each group convert (buy, sign up, click — whatever the goal is). The question isn’t just “which rate is higher” — with any finite sample, B’s rate will almost never be exactly equal to A’s, purely from random noise, even when the underlying, true conversion probability is identical for both. The question is: is the gap we observed bigger than random noise alone would typically produce?

That’s what a hypothesis test answers. We set up two competing claims:

H0:pA=pBHa:pA≠pBH_0: p_A = p_B \qquad \qquad H_a: p_A \ne p_B

H0H_0 (the null hypothesis) says there’s no real difference — any gap we see is noise. HaH_a (the alternative) says there’s a genuine difference. The test doesn’t “prove” either one; it asks how surprising the observed data would be if the null were true, and that’s the only question a p-value ever answers.

2. The two-proportion z-test

For comparing two conversion rates, the standard tool is a two-proportion z-test. With xAx_A conversions out of nAn_A visitors in the control, and xBx_B out of nBn_B in the variant:

p^A=xAnA,p^B=xBnB\hat{p}_A = \frac{x_A}{n_A}, \qquad \hat{p}_B = \frac{x_B}{n_B}

Under H0H_0, both groups share one true rate, so we pool them to get the best estimate of it:

p^=xA+xBnA+nB\hat{p} = \frac{x_A + x_B}{n_A + n_B}

That pooled rate gives us the standard error — how much random spread to expect in the difference between two sample proportions of these sizes, if the null is true:

SE=p^(1−p^)(1nA+1nB)SE = \sqrt{\hat{p}(1 - \hat{p}) \left( \frac{1}{n_A} + \frac{1}{n_B} \right)}

The z-statistic is just the observed gap measured in units of that expected noise:

z=p^B−p^ASEz = \frac{\hat{p}_B - \hat{p}_A}{SE}

And the p-value is the probability, under H0H_0, of seeing a gap at least this extreme purely by chance:

p-value=2(1−Φ(∣z∣))p\text{-value} = 2 \left(1 - \Phi(|z|)\right)

where Φ\Phi is the standard normal CDF. If that probability is small — conventionally, below a threshold α\alpha (almost always 0.050.05) — the result is called “statistically significant.”

3. What a p-value does not mean

This is the part that gets misquoted constantly, so it’s worth being precise before touching any code:

  • A p-value is not the probability the null hypothesis is true. p=0.03p = 0.03 does not mean “there’s a 3% chance there’s really no difference.” It means: if there really were no difference, a gap this large (or larger) would show up about 3% of the time by chance alone.
  • p<0.05p < 0.05 does not mean the effect is real, or big, or important. It means the result would be unusual under the null. With a large enough sample, even a trivially small, practically meaningless difference becomes statistically significant.
  • p≥0.05p \ge 0.05 does not mean there’s no effect. It might just mean your sample was too small to detect a real effect that’s actually there — which is exactly what the simulation in section 5 will show happening.

4. Statistical power and sample size

Power is the probability your test correctly detects a real effect, given it exists — formally, 1−β1 - \beta, where β\beta is the probability of a false negative (missing a real effect). Power depends on three things pulling against each other: the size of the true effect, the sample size, and your significance threshold α\alpha. A rough sample-size formula for a two-proportion test, targeting power 1−β1 - \beta at significance α\alpha:

n≈(zα/2+zβ)2⋅2pˉ(1−pˉ)(pB−pA)2n \approx \frac{\left(z_{\alpha/2} + z_{\beta}\right)^2 \cdot 2\bar{p}(1 - \bar{p})}{(p_B - p_A)^2}

The intuition, even without the formula: smaller effects need more data to detect reliably, and most underpowered A/B tests fail not because the effect isn’t real, but because nobody ran the numbers on how many visitors it would actually take to see it.

5. Watching all of this happen, in a simulation

Formulas describing long-run frequencies are easy to state and easy to misjudge intuitively. So instead of taking “a 5% significance threshold gives a 5% false-positive rate” on faith, the simulation below actually runs thousands of A/B tests where the true answer is fixed in advance by the code, and checks whether the test’s real-world behavior matches what the theory promises — for the false-positive rate, for statistical power, and for a specific way A/B tests commonly go wrong in practice: peeking.

That last one deserves a preview. Checking your test’s p-value every day and stopping as soon as it dips below 0.05 feels harmless — you’re just “keeping an eye on it.” It isn’t harmless: each additional look is another chance for random noise to cross the threshold, so the true false-positive rate of “check repeatedly, stop at the first significant result” is well above the 5% you think you signed up for. The simulation measures by how much.

A few things worth stopping on once you’ve run it:

The p-value histogram is (close to) flat. When the null hypothesis is exactly true, a p-value is, by construction, uniformly distributed between 0 and 1 — every band is equally likely, including the band below 0.05. That’s not a coincidence or an approximation; it’s the literal definition of what “false positive rate = α\alpha” means, made visible as a shape instead of a sentence.

The single test from section 5.1 came back non-significant — but the effect it was testing for was real (the simulation fixed it at a true 10%-to-12% lift, we just don’t normally get to know that in practice). Section 5.3’s power curve shows why: at n=1,000n=1{,}000 per variant, power for a 2-point lift on a 10% baseline is low, well under a coin flip’s worth of reliability. “Not significant” quietly meant “underpowered,” not “no effect” — precisely the misreading section 3 warned about, now with a number attached.

Peeking roughly quintuples the false-positive rate in this simulation — checking every 100 visitors and stopping at the first p < 0.05 finds “significance” far more often than checking once at the end does, on data engineered to have no real difference at all. This is exactly why serious experimentation platforms either fix the sample size in advance and don’t look until it’s reached, or use sequential-testing methods specifically designed to allow early stopping without inflating the false-positive rate (see Evan Miller’s writeup for a good treatment of the fix, not just the problem).

6. Practical takeaways

  • Decide your sample size before you start, from a target effect size and power (section 4’s formula, or any online A/B test sample-size calculator), and don’t stop early just because the p-value dipped below 0.05 — section 5.4 shows exactly what that costs you.
  • A non-significant result is not evidence of “no effect.” It’s frequently evidence of “not enough data to tell” — check your test’s power before concluding a variant made no difference.
  • Statistical significance and practical significance are different questions. A large enough sample makes almost any real (even minuscule) difference statistically significant. Whether a 0.1 percentage-point lift is worth shipping is a business question the p-value doesn’t answer.
  • If you must peek, use a method built for it (sequential testing, always-valid p-values), not repeated ordinary z-tests with an informal “stop when it looks good” rule.

If hypothesis testing from first principles is interesting, I did a similar paired t-test analysis — on Stroop-effect reaction times rather than conversion rates, but the same “is this gap bigger than noise” logic — as a grad-school project, on the Projects page.