What a placebo test tells you that a p-value can't

A geo test ends. The analysis says sales in the test region were 6% lower than its synthetic comparison during the holdout. Is that the campaign's effect, or just how much that comparison wobbles anyway?

A p-value tries to answer that with a formula and a set of assumptions. A placebo test answers it by looking.

The idea

Take every region you didn't touch. For each one, run exactly the same analysis, over exactly the same period, as if that region had been the test region. Build its synthetic comparison the same way, measure the gap the same way.

Nothing happened in those regions, so any "effect" you find there is noise: the natural wobble of the method on your data.

How to read it

Line up all the results: your real test region and every placebo.

• If your test region stands clearly apart, the largest effect by a distance, your result is very unlikely to be noise.
• If it sits among the placebos, the analysis finds effects that size where nothing happened. Your result might be real, but this test can't show it.
• The rank is the evidence. If your test region ranks first out of twelve, roughly one in twelve is the chance of that happening by luck, with no formula required.

Why it's more convincing than a p-value

A p-value depends on assumptions about the data that are hard to check and harder to explain. A placebo test uses your own regions and your own data, and the result is a picture anyone in the room can understand: here's our region, here are eleven where nothing happened, ours is the outlier.

Use it twice

Placebo tests are most useful before the test, not just after. Run them during design on the period you plan to use. If the placebos already show "effects" as large as the one you hope to find, the design won't work, and you've found out before spending the budget.

The bottom line

The most convincing evidence a test worked is showing what the same analysis finds where nothing happened. If your result can't stand apart from the placebos, it isn't ready to move a budget.