Write the decision rule before your next marketing test

In brief: Before launching a marketing experiment, write down the decision it is meant to change, the single variable, the primary outcome, the guardrail, the observation window and the rule for what happens next. If the result cannot change a decision, the test is probably not worth running.

“Let’s test it” can sound admirably practical. Yet many marketing tests begin with a new headline, image or audience and end with a dashboard full of numbers that nobody knows how to act on.

The missing piece is usually not another metric. It is a decision rule written before the result is visible.

That matters because experiments are not content slots. They are a way to reduce uncertainty between two plausible actions. A useful test should make Monday’s decision easier: adopt the change, keep the control, extend the test or stop because the question was poorly framed.

A measured result can still be useless

A platform can split traffic and report a winner, but the business question may still be unclear. Click-through rate can rise while qualified enquiries fall. A new landing page can appear stronger after three days simply because it ran during a different demand pattern. A team can also change the headline, visual and offer together, then learn only that “the new version” behaved differently.

The UK government’s A/B testing guidance, updated on 3 September 2020, starts with a hypothesis or goal, a control and a variation, a sample size and a duration. It also notes a practical limitation that small brands should take seriously: statistically meaningful results often require many users.

Google’s current custom experiment guidance makes the operating choices explicit. It asks advertisers to select success goals, recommends a 50% traffic and budget split for comparison, and warns that changing the original or experiment while it is running can make the result harder to interpret.

The shared lesson is simple: the result becomes credible only when the question and conditions are stable enough to read.

The 20-minute test card

Before opening the platform, complete these six lines.

1. The decision

Write the choice the result can change.

Not “improve the landing page”. Try: “Decide whether the service page should lead with reassurance or speed for paid-search visitors.”

If neither outcome changes what the team will do, save the production time.

2. The hypothesis

Use a cause-and-effect sentence: “For this audience, leading with proof of experience will increase completed enquiries because perceived risk is the main barrier.”

This exposes the assumption. It also makes an inconclusive result useful: perhaps the barrier was not trust, or the variation did not express trust clearly enough.

3. The single variable

Name what changes and what stays fixed. For example: headline changes; offer, image, form, audience, budget allocation and page structure remain the same.

One-variable discipline is not always possible in a full redesign. When several elements must change together, call it a package test and limit the claim: you can compare packages, but you cannot say which element caused the difference.

4. The primary outcome and guardrail

Choose one outcome closest to the business decision: completed enquiry rate, qualified booking rate or completed purchase rate. Then add a guardrail that protects against a hollow win, such as refund rate, lead quality or cost per qualified enquiry.

Secondary metrics can explain the journey. They should not be allowed to replace the primary outcome after the preferred version underperforms.

5. The window and exposure

Set the minimum duration and required exposure before launch. Include a full trading cycle where relevant, and avoid stopping the first time one line moves ahead.

For lower-traffic businesses, this may reveal that a formal A/B test is not feasible. That is not failure. Run a directional pilot, use moderated usability sessions, or test a larger change that is more likely to create a detectable signal. Label the evidence honestly.

6. The action rule

Pre-write the next move:

• adopt the variation if the primary outcome clears the agreed threshold and the guardrail holds;
• keep the control if it does not;
• extend only if the planned sample or window is incomplete;
• redesign the test if tracking failed or the two versions did not isolate the intended difference.

The Competition and Markets Authority’s experiment principles, published on 19 April 2023, explain why pre-specification matters: planning the analysis before seeing the result reduces the temptation to adjust the method until a significant finding appears.

Rigour should match the decision

A six-line card is not a substitute for statistical expertise. High-spend media, pricing, health claims and decisions affecting vulnerable people need stronger design, larger samples and specialist review. A tiny organic post comparison cannot support the same causal claim as a randomised experiment.

The opposite mistake is to make every low-risk creative choice wait for laboratory-grade evidence. The right level of rigour depends on the cost of being wrong.

For the coming week, set one rule: no marketing experiment enters production without a completed decision card. The document can be short. The thinking cannot be skipped.