The Panel Your Incrementality Vendor Uses to Build the Holdout Group Determines What 'Incremental' Means for Your Campaign

Incrementality testing has become a standard recommendation for media buyers who want to move beyond last-touch attribution. The core idea is simple: hold back a portion of your audience from seeing ads, compare their conversion rate to those who saw ads, and attribute the difference to your campaign. What gets less attention is that the holdout group itself is a constructed artifact, not a neutral sample, and the vendor who builds it is making decisions that shape your results before a single impression runs.

Understanding those decisions does not require a statistics background. It requires knowing which questions to ask before you sign a measurement contract.

How holdout groups get built

A holdout group is typically drawn from the same audience pool your campaign would otherwise target. The intent is to create two groups that are statistically similar in every way except exposure to your ads. In practice, vendors use different methods to achieve that similarity.

Some vendors assign holdout status at the device or identifier level, randomly selecting a percentage of matched records before the campaign launches. Others assign holdout status at the household level, which changes the size and composition of the group. Still others use geographic splitting, designating certain markets as test markets and others as control markets. Each method answers a slightly different question, and none of them is universally correct.

Device-level holdouts, for example, can be contaminated if a person who is held out on one device sees your ad on another. Household-level holdouts reduce that problem but may introduce geographic clustering effects if households in certain areas are systematically more likely to convert regardless of ad exposure. Geographic holdouts avoid identifier-level contamination but measure market-level lift rather than person-level lift, which may not match your campaign's actual targeting logic.

The vendor you work with has likely defaulted to one of these approaches without necessarily explaining the tradeoff in plain terms.

What 'incremental' is actually being measured

Once the holdout construction method is set, the definition of an incremental conversion follows from it. If your holdout is a device-level random sample, an incremental conversion is a conversion that happened in the exposed group at a rate that exceeds the holdout group's rate, as measured against device-linked conversion signals. If your vendor cannot observe a conversion event on a held-out device, that conversion disappears from the denominator, which inflates your measured lift.

This is not a flaw in any one vendor's product. It is a structural feature of how holdout measurement works. Every measurement system has a boundary around what it can observe, and the lift it reports is lift within that observation boundary, not lift in the real world.

A hypothetical example illustrates this cleanly. Suppose your campaign targets a B2B audience and conversions happen through a sales team after a prospect fills out a form. If your measurement vendor tracks only pixel-fired form submissions and cannot observe downstream pipeline activity, your incrementality test will tell you how many additional form submissions your campaign drove. It will not tell you whether those additional form submissions converted to revenue at a different rate than the holdout group's organic form submissions. Those are two different questions.

Why the holdout percentage matters more than most buyers realize

Vendors typically default to holdout sizes between five and twenty percent of the target audience. Larger holdouts produce more statistically reliable results because there are more conversion events to observe in the control group. Smaller holdouts reduce the audience withheld from your campaign, which matters when your addressable universe is already small.

The tradeoff is real: if your campaign is designed to reach five thousand matched accounts in a B2B context, a twenty percent holdout means one thousand of those accounts never see your ads during the test period. For a long sales cycle category, those one thousand accounts may represent pipeline you genuinely sacrificed to generate a measurement number. Whether that tradeoff is worth it depends on how much you trust the methodology and how much the learning changes your future spending decisions.

Some vendors offer adaptive holdout designs that release held-out audience members after a defined observation window, which can reduce the cost of testing. Understanding whether your vendor offers this, and what triggers the release, is a practical procurement question worth raising before a test launches.

Questions to ask your measurement vendor before the test begins

The goal is not to challenge the vendor but to understand exactly what your results will mean when they arrive. A few specific questions help frame the conversation.

First, ask how holdout assignment is made and at what unit: device, household, or geography. Ask whether a person can appear in both the exposed group and the holdout group if they hold multiple identifiers, and how the vendor handles that case.

Second, ask what conversion events are observable within the holdout group. If the vendor can only observe conversions through a pixel that fires on a page you control, ask whether offline or CRM-sourced conversions are included and how they are attributed to holdout status.

Third, ask what the minimum detectable effect is for your proposed holdout size and expected conversion volume. This is a statistical concept, but vendors can usually provide a plain-language answer: something like, the test can reliably detect a lift of X percent or more given your expected traffic. If your actual lift is below that threshold, the test will not distinguish it from noise, and you should know that before you run it.

Fourth, ask whether the holdout group will be refreshed if the campaign extends beyond its original flight. Audience composition can shift as your campaign optimizes, and a holdout built at launch may no longer be comparable to the exposed group several weeks later.

What to do with the results once you have them

Incrementality results are an input into spending decisions, not a final answer. A test that shows strong lift for one audience, in one channel, during one time period, tells you something useful about that specific combination. It suggests, but does not confirm, that a similar combination would perform similarly in a future campaign.

The most durable use of an incrementality result is to compare it against your attribution model's claim for the same campaign. If your last-touch model credited your campaign with five hundred conversions and your incrementality test suggests the true incremental number is closer to two hundred, that gap tells you something about how much your attribution model is over-counting. That recalibration is often more valuable than the lift number itself.

Building a measurement practice around incrementality means running tests repeatedly across different audiences and channels, not treating a single result as a permanent truth. The holdout methodology your vendor uses will be consistent across those tests, which means its assumptions will be consistently baked into every result. Knowing those assumptions makes your results easier to interpret and your vendor conversations more productive.