A/B testing

The step most agency tests skip is the one that determines whether the test can work. Before launching, decide how large a difference would be worth acting on, then work out roughly how much traffic each variant would need for a difference that size to be distinguishable from noise.

Run without that calculation, a test has two ways to end and both are bad. It runs out of time with no clear winner, which reads as a waste of a month. Or somebody calls it early on a gap that was noise, which is worse, because it produces a confident wrong answer that then gets rolled out across the account.

The arithmetic is unforgiving on small numbers. Detecting a ten per cent relative improvement on a baseline conversion rate of three per cent needs on the order of thousands of conversions per variant, which is more than many client accounts produce in a quarter. Halve the effect you want to detect and the traffic needed roughly quadruples.

This is why the honest conclusion is sometimes that the test should not run. An account generating forty conversions a month cannot resolve a ten per cent difference before the season changes underneath it, and saying so is more useful than starting anyway.

A significance test is built on the assumption that you look once, at a point decided in advance. Checking every morning and stopping the day it crosses the threshold breaks that assumption, because each look is an independent opportunity for random variation to cross the line.

The effect is large rather than marginal. Repeatedly peeking at a test with no real difference will eventually show a significant result a substantial share of the time, and the earlier you are willing to stop, the more often that happens.

It is also the most natural thing in the world to do, which is why it is worth guarding against structurally rather than by intention. Decide the end date and the sample size at the start, write them down, and treat the test as unread until then.

Where continuous monitoring genuinely matters, the fix is to use a method designed for it, such as a sequential test with corrected thresholds, rather than to apply a fixed-horizon test more often. Guessing at an adjustment is not a method.

Comparing one period against the next is the most common substitute for a real test in agency work, usually because splitting traffic felt complicated. It measures the change plus everything else that moved between the two periods, and everything else is normally larger.

Consider what shifts across four weeks: the season, competitor budgets, a public holiday, the mix of returning and new visitors, a change in the ad platform nobody announced. Any of those can produce a swing bigger than the creative difference being tested.

The version that looks more rigorous and is not is testing across matched periods, such as the same fortnight last year. It relies on the assumption that the market was comparable, which is the assumption the test was supposed to check.

If traffic genuinely cannot be split, then say the result is directional and treat it that way. The failure is not running a before-and-after comparison, it is presenting one as though it were an experiment.

The size of the detectable effect determines what is worth testing, and that reverses the usual priority order. Button colours and headline tweaks produce small differences that a small account can never resolve; offers, pricing presentation and page structure produce large ones that it sometimes can.

So on a modest budget, test things that could plausibly move the result by a third rather than by a twentieth. A different offer, a materially shorter form, a landing page built around a different objection. If the honest expectation is a few per cent, the test will not settle it.

Splitting by user rather than by campaign matters as much as what is tested. Two campaigns will not deliver to the same audience mix, so a campaign-level split compares variants and audiences together and cannot separate them.

Finally, a result is a result for the context that produced it. A variant that won during a promotion, on mobile, for returning visitors, has been shown to win there. Rolling it everywhere is an extrapolation, and it is worth labelling it as one in the report.