Five checks before you trust a mix model

You don't need to build a mix model to judge one. You need five questions, and the willingness to ask them before the budget moves.

Mix models are the best tool most businesses have for seeing what their marketing caused. They are also easy to get wrong, and the output always looks confident: a neat table of channels, a return for each, a recommended budget. The table looks the same whether the model is sound or not. These checks are how you tell the difference.

1. Could the data answer this at all?

A model can only learn what a channel does if the channel's spend actually changed over time. If TV ran at the same weekly level all year, the data contains no information about what happens when TV goes up or down. The model will still produce a number, but it comes from its assumptions, not your data.

Ask: which channels barely varied, and how were they handled?

2. Is there a range on every number?

A return of £2.40 per pound means little without its range. If the range runs from 40p to £4, the honest answer is "we don't know yet", and the budget shouldn't move much on it.

Ask for every headline number with its likely range. If the model can't give one, that's your answer about the model.

3. How well does it predict weeks it hasn't seen?

Any model can fit the data it learned from. The useful test is how well it predicts weeks it was never shown. Ask for its accuracy on a held-out period, usually the most recent few months, and look at whether the misses are random or systematic.

4. Which channels move together?

If TV and social always rise and fall together, because they're planned together, the model can't separate them. It will split the credit somehow, and the split will be fragile: small changes to the model move a lot of credit between the two.

Ask which channels are strongly correlated, and what the model did about it. Good answers include combining them, using prior evidence, or running a test that moves one without the other.

5. Does it agree with your tests?

Where a well-designed experiment exists, a sound model should broadly agree with it. If a geo test says a channel works and the model says it doesn't, believe the test and ask why the model disagrees. The best models use test results as evidence (calibration) so the two check each other.

One more thing: disagreement is useful

Some teams now run two different modelling approaches on the same data. When they agree, confidence goes up. When they disagree, you've learned where the evidence is thin, which is exactly where to test next. A single model hides that.

The short version

Before a mix model moves your budget, ask: could the data answer it, where are the ranges, how does it do on unseen weeks, which channels are tangled, and does it agree with our tests? A model that answers all five well is worth listening to. One that can't answer them is worth questioning.