Skip to content
home

Experiments that work · Part 3

Don't peek: why early test results lie

Early results swing. Locking the readout protects the decision.

Terry Manu Hennah

23 August 2026 · 2 min read

Week one of a geo test. Someone checks the dashboard: the test region is already down 9%. The campaign clearly works. Can we call it?

Week four: down 2%. Week eight: down 5%, where it settles. If you'd called it at week one, you'd have overstated the effect almost twofold. If you'd called it at week four, you might have cut a campaign that works.

Why early results swing

Early in a test there's very little data. A few unusual weeks (a local event, a delivery hiccup, the weather) move the estimate a lot. As weeks accumulate, those bumps average out and the estimate settles.

The range around the estimate tells the same story: very wide at first, narrowing as the test runs.

Illustrative example: a test's running estimate of the effect week by week, with its range. Early weeks swing; the estimate settles by the planned end.
Example: a test's running estimate of the effect week by week, with its range. Early weeks swing; the estimate settles by the planned end.
A test that's checked every week until it looks good will always look good eventually. That's the problem.

Why peeking makes it worse

Looking isn't the problem. Acting on what you see is. If you check every week and stop as soon as the result looks good, you'll stop at a lucky high point far more often than chance suggests. A test that's checked every week until it looks good will always look good eventually.

The same happens in reverse: stop when it looks bad and you'll kill campaigns that work.

How to protect the decision

  • Agree the minimum duration before launch, from the power analysis, and write it down.
  • Monitor the test's health, not its result. Is spend delivering as planned? Is the data arriving? Has anything contaminated the comparison regions? Those are the things to watch weekly.
  • Lock the readout. Some teams literally hide the result until the minimum duration has passed. It feels strict; it saves bad decisions.
  • Read out once, at the agreed end, using the analysis you agreed before launch.

What if something goes wrong mid-test?

Stopping early for a real problem (spend leaking into the test region, a major outage) is fine. Stop, record why, and treat the test as incomplete rather than reading a result from it.

The discipline is the point

The value of a test comes from deciding the rules before you see the data. Peeking quietly removes that, and turns a careful experiment back into an opinion with a chart attached.

Newsletter

Get the next essay by email

No spam. Unsubscribe in one click. I’ll only use your email for this newsletter.