Incrementality

Attribution looks at the journeys that ended in a purchase and decides how to divide the credit among the advertising that appeared along the way. Every one of those journeys, by construction, included advertising. The people who would have bought anyway are in that data, indistinguishable from the people who were persuaded.

Incrementality asks the question attribution structurally cannot: how many of those sales would have happened with no advertising at all. Answering it requires observing a world where the ads did not run, which means deliberately withholding them from somebody.

That is the entire difference, and it is why no amount of better modelling closes the gap. A more sophisticated attribution model still only sees converting paths. It is a better way of dividing a number that may be too large to begin with.

Bidding on the client own brand name produces spectacular reported performance. The clicks are cheap because nobody is more relevant to that query, the conversion rate is high because the person was already looking for the client, and the ROAS on the report is the best figure in the account.

Most of those people would have arrived anyway through the organic result immediately below the ad. The advertising did not create the demand; it charged the client for intercepting it.

Whether it is nonetheless worth buying is a real empirical question rather than a rhetorical one. If competitors bid on the brand, the ad defends a position that would otherwise be taken. If the organic result is weak or sits below other content, the ad covers a gap. If neither is true, it is a subsidy.

A two-week holdout on brand terms answers it directly, and it is the single easiest incrementality test to run because the traffic is high and the effect, if there is one, is large.

A geo holdout is the most practical. Pick regions that behave similarly, switch the advertising off in some of them, leave everything else alone, and compare total conversions across the two groups rather than attributed ones. The output is a lift figure no attribution debate can touch.

Both major platforms also run their own conversion lift studies, which randomise at the user level and are more statistically efficient than geography. The trade is that the platform is marking its own homework, and the study measures only that platform.

The third design, and the least satisfying, is an on-off test in time: run, pause, compare. It is confounded by everything that changed between the two periods, which in marketing is usually a lot. Use it only when geography is unavailable and treat the result as suggestive.

Whichever design, the discipline is the hard part. A test that gets adjusted halfway through because the paused regions looked bad has measured nothing, and the fortnight is spent.

It usually will be, and an agency that has not prepared the client for that is about to have a difficult meeting about its own reporting.

The framing that works is to say beforehand that the two numbers measure different things and the test number will be lower, so the lower number is the test succeeding rather than the advertising failing. Said afterwards, the same sentence sounds like an excuse.

The commercially useful output is not a verdict on whether a channel works. It is a correction factor: if measured lift is 60% of attributed conversions, the attributed figures remain usable for day-to-day steering with a known discount applied to anything commercial.

And a channel that tests as genuinely low-incremental is not necessarily one to cut. Branded search that defends against competitor bidding is doing a defensive job that shows up as poor incrementality and would cost more to stop doing.