Holdout tests when the stakes are high

Suggest switching off a channel in part of the country and you'll often hear the same worry: what if sales drop? In some businesses that worry comes with extra weight: regulated products, brand reputation, campaigns senior leaders care about.

Those concerns are reasonable. They're also manageable, and the alternative (never testing) has a cost too.

Put the risk in proportion

A well-designed holdout typically pauses or reduces one channel, in a small share of the market, for a fixed number of weeks. If the channel works, you lose some of its contribution in that region for that period. That's the cost, and it's bounded and knowable in advance.

The cost of never testing is less visible and larger: every future budget decision for that channel rests on numbers you can't verify.

Limit the exposure

• Test part of a channel, not all of it. Reduce spend by half, or pause one campaign type, rather than switching everything off.
• Use a small share of the market. Choose test regions that together represent a modest slice of sales.
• Keep it time-boxed. Agree the duration from the power analysis and stick to it.
• Prefer reductions to blackouts where brand presence matters. A lower level still tells you a lot.

Choose what to test carefully

Start with the channel where the decision is biggest and the evidence weakest: often one with large spend, uncertain models and confident platform reporting. Avoid testing during periods that matter disproportionately, such as a product launch or peak trading week.

Get the sign-off right

In high-stakes settings, the design document matters as much as the analysis:

• What we're testing and why, and the decision the result will inform.
• The expected cost, stated plainly, with an upper bound.
• Stop conditions: what would make us end the test early, and who decides.
• Compliance and brand review of anything that changes what customers see.
• How results will be shared, including if the answer is "we can't tell".

Make the first one boring

The best first holdout is uneventful: small, careful, well documented, and read out exactly as planned. It builds the trust that makes the second test easier to approve. The cost of a holdout is a few weeks of uncertainty in one region. The cost of never testing is every budget decision after it.