How a geo test should change your mix model

Mix models and experiments are often treated as rival methods. One team trusts the model, another trusts the tests, and the two numbers sit on different slides.

They're more useful together. A mix model and an experiment answer the same question from different directions. Calibration makes them talk to each other.

What each is good at

• Mix models see all your channels at once, over years, including everything else that moves sales. But they can be fooled by channels that move together or barely vary.
• Experiments answer one question about one channel, in one place, for one period. But they're hard to fool, because they create the comparison rather than inferring it.

Each covers the other's weakness.

How calibration works

Once a geo test is complete, its result (the incremental effect, with its range) becomes evidence the model uses for that channel. In a Bayesian model this is natural: the test result informs what the model believes about the channel before it looks at the history, and the history updates it from there.

The test's range matters. A precise test pulls the model strongly; a vague one only nudges it. And because tests happen in one place and time, it's sensible to widen the range a little before using it, to allow for differences elsewhere.

What changes

• The tested channel's range narrows, usually a lot, because the model no longer has to infer its effect from tangled history alone.
• Other channels often sharpen too. If the tested channel was tangled with another, pinning one down helps separate the other.
• Disagreements resolve. If two models disagreed about the channel, calibration often brings them together.

When the test and model disagree

If the test says the channel works and the uncalibrated model said it didn't, believe the test first, then ask why the model disagreed. Common reasons: the channel barely varied, it moved with another channel, or the model's carryover was wrong. Each is worth fixing.

Make it a cycle

1. The model shows where evidence is weakest.
2. Test there.
3. Calibrate with the result.
4. Refresh the model, and see where evidence is weakest now.

Over a year, that cycle turns a model with wide, contested ranges into one the finance team can rely on.