The most trustworthy number in marketing usually comes from switching something off.

A holdout is the whole idea in one move: take the thing you want to measure; a channel, a campaign, a nurture program, and deliberately withhold it from a comparable slice of your audience. Then watch what that group does without it. The gap between the group that got the thing and the group that didn’t is the number you’re after.

That gap is the only figure in marketing that answers the question everyone pretends their dashboard answers: did this actually work, or would it have happened anyway? Every other number tells you what occurred and then assigns credit for it. The holdout tells you what you caused. Those are not the same thing, and the difference is usually bigger than anyone wants to admit.

Everything else is correlation in a suit

Here’s why the holdout is different in kind, not just in quality, from the rest of the measurement stack.

Attribution, last-touch, first-touch, marketing mix models, the whole dashboard, every one of them works the same way underneath. It observes what happened and then divides the credit. The customer who converted saw an ad, attended a webinar, opened three emails, and the model decides how much each one gets. But observing a sequence can never tell you what would have happened without it. Maybe that customer was going to buy regardless and the ads just got to stand next to the sale and take a bow. Attribution structurally cannot see that, because it only has the world where the marketing ran.

A holdout has both worlds. It builds the missing one on purpose, a group that lived without the thing, and measures the distance between them. That’s the move from correlation to causation, and there’s no amount of modelling sophistication that gets you there from observation alone. You either withheld the thing from someone and looked, or you’re guessing with extra steps.

This is the constructive flipside of the attribution problem. Attribution is a story you tell about who deserves credit. A holdout is a measurement of what you actually changed. One is editorial; the other is evidence.

How to actually run one

The mechanics are simpler than the mystique around them suggests. Four steps.

Pick one thing to test and one metric that matters, not a dashboard of twelve, the single number the thing is supposed to move, usually pipeline or revenue rather than clicks. Split your population into two genuinely comparable groups: one that gets the thing (treatment) and one that doesn’t (the holdout). Run it long enough for the result to clear the noise, a few weeks of data is usually a coincidence, not a finding. Then the answer is one subtraction: lift equals what the treatment group did minus what the holdout group did. That gap, and only that gap, is what the thing caused.

The forms differ by what you can cleanly withhold. A geo holdout turns a channel off in one region and compares it to a similar one, the standard way to test brand, paid social, anything you can’t target at the individual level. An audience holdout withholds a program from a random slice of contacts, which is how you test nurture emails, retargeting or a lifecycle campaign. Ghost ads and PSA tests show a neutral placeholder to the control group so even the ad server is comparing like with like.

The one thing that makes or breaks it is that the two groups are actually comparable. Randomly assigned, similar in size and makeup, treated identically except for the one thing. Get that right and the subtraction is trustworthy. Get it wrong, cherry-pick the regions, let the groups drift — and you’ve just built a more expensive way to fool yourself.

Why almost nobody does it

If the holdout is so much better, the obvious question is why it’s rare. The answer is that it costs something real, and the costs land in exactly the places people flinch.

The first cost is reach you deliberately give up. Running a holdout means choosing not to market to a group you could have marketed to, on purpose, to learn something. That feels like leaving money on the table, and it is, a small, deliberate amount, spent to buy a true answer. Most teams can’t stomach the trade, so they keep the reach and keep the guessing.

The second cost is that it’s not tidy. A dashboard gives you a confident number today. A holdout gives you an uncomfortable wait and then a messy, caveated result. In a status meeting the dashboard wins every time, which is precisely why the dashboard is usually wrong.

The third cost is the one nobody says out loud: a holdout can tell you the program you built, championed and staffed does nothing. That’s a career-shaped risk, and it keeps a lot of measurement conveniently vague. But flip all three costs around and they’re the reason the method is an edge. The teams willing to pay them know which of their programs actually work. Everyone else has a dashboard full of credit and no idea what they caused.

When not to bother

Here’s where I’d push back on my own enthusiasm, because a holdout zealot is as annoying as a dashboard apologist, and wrong more expensively.

Holdouts are costly to run and read, so they’re for the decisions big enough to earn the cost, a major channel’s worth of spend, a flagship program you’d actually kill or double down on. Running one to decide a button colour is theatre with better intentions. For the small, reversible stuff, a fast proxy metric is the right tool; save the holdout for the questions where being wrong is expensive.

It’s also worth being clear about what a holdout does and doesn’t answer. It measures incrementality, how much this thing added, not attribution. It tells you whether to keep doing something, not who in the funnel deserves the credit. Don’t ask it the question it can’t answer.

And it needs scale, patience and discipline. Too few people in the groups and the result is noise wearing a lab coat. Too short a window and you’ve measured a fluke. Let the groups drift and you’ve measured nothing. The method is only as honest as the setup, which is the real reason to respect it: it punishes sloppiness instead of hiding it in a weighted average. Used on the few questions that matter, run properly, it’s the closest thing marketing has to the truth.

Marketing is drowning in confident numbers. Attribution dashboards, sourced-revenue reports, engagement scores, all precise, all authoritative, almost none of them answering the only question that matters, which is what would have happened anyway.

The holdout answers it, and it charges you for the privilege: a little reach, some patience, and the risk of learning something you didn’t want to. That price is exactly why the number is worth trusting. Attribution tells you a story. A holdout tells you what’s true. In a field this full of stories, the one figure that cost you something to get is the one I’d believe.

Frequently asked

What is a holdout test in marketing?

It’s a controlled experiment where you deliberately withhold a marketing activity, a channel, campaign or program, from a comparable group, then measure the gap between the group that got it and the group that didn’t. That gap is the incremental lift: the result the activity actually caused, as opposed to the result it merely stood next to.

How is a holdout different from attribution?

Attribution observes one world, the one where your marketing ran, and divides credit after the fact, so it can never tell you what would have happened anyway. A holdout builds the missing world on purpose (a group without the thing) and measures the difference. Attribution is correlation and editorial judgment; a holdout is causation. One is a story about credit, the other is evidence of effect.

When should you run a holdout, and when not?

Run one for decisions big enough to justify the cost, major channel spend or a flagship program you’d actually kill or scale. Don’t run one for small, reversible tweaks; a fast proxy metric is better there. Holdouts need scale, a long enough window, and genuinely comparable groups, or the result is just noise. Used on the few questions that matter, run with discipline, they’re the most trustworthy measurement marketing has.