Testing without fooling yourself
Lesson 11 of 16 in the course “Marketing 2027”. Free, with animations and a quiz.
The idea that waited six months
At Microsoft Bing, one employee suggested something simple: make the search ad headline longer by moving part of the text from the line below into it. The idea was judged minor and sat in the queue for more than six months. Then an engineer spent a couple of days and launched a test: half the people saw the old version, half the new one. Within hours the system raised a “too good to be true” alert. They checked, and there was no bug: the change brought +12% revenue, which the authors estimate at over 100 million dollars a year in the US alone. The case was described in Harvard Business Review in 2017. The lesson is simple: you can’t eyeball how important an idea is. You can only test it.
Most ideas don’t work
The same article has a sobering number. At Google and Bing, only 10–20% of experiments produce a positive result. Across Microsoft, roughly a third of ideas help, a third change nothing, and a third actually hurt. And these are companies with huge analytics teams. Why? People are good at inventing explanations but bad at predicting how thousands of strangers will behave. For ads this means two things. First: don’t be upset if your “genius” creative lost, that’s normal. Second: without a test you would have launched an idea that hurts and never found out. So a test isn’t bureaucracy, it’s insurance against your own confidence.
One test, one difference
An honest test works like this: the audience is split in two, and each person sees only one variant. The variants differ in exactly one thing, for example the hook in the first seconds. Everything else is the same: budget, offer, dates, platform. Why so strict? If Nika changes the hook, the price and the music at once and sales go up, she won’t know what actually worked and won’t be able to repeat the success. And if A runs on Monday and B on Friday, the winner isn’t the creative, it’s the day of the week. So use the built-in tools: Meta has A/B tests, Google Ads has experiments, TikTok has split tests. They split people so the groups don’t overlap.
How much data you need
Flip a coin 10 times. You could easily get 7 heads and 3 tails, even though the coin is fair. Over 1000 flips it will be roughly 500 to 500. Ads work the same way. If variant A has three sales and B has five, that isn’t a win for B yet, just ordinary random variation. The fewer events, the louder the noise. What to do in practice? First: pick a metric with lots of events. If sales are few, compare clicks to the store or add-to-carts. Second: don’t stop the test until each variant has at least a few dozen of those events. Third: check what the platform’s built-in tool says. Meta, for example, shows how likely the winner is to win again if you repeat the test.
The day-one trap
The most common mistake is peeking and stopping the test as soon as one variant pulls ahead. On day one there’s little data and the leader is often random. A week later the gap can disappear or even flip. That’s why test length is set in advance, usually at least a week, so both weekdays and weekends are included. The second trap is a result that’s too pretty. Experimentation researchers from Microsoft call it Twyman’s law: if a number looks incredibly good, there’s probably a mistake somewhere. Remember Bing? They looked for a bug first and celebrated only after. Do the same: if a new creative suddenly brings 5x more sales, check the pixel, the links and whether a discount reached only one group.
The test log: your team’s memory
One test is one line of knowledge. Ten tests are a real understanding of your audience. But only if you write them down. Set up a simple table: idea, what changed, how long the test ran, result and decision. The example below is made up: Nika tested a question hook, there was no difference, and the idea went to the archive. The price in the first frame did better, so she added it to every video. The test “customer review vs studio shoot” is still running. After a couple of months this log answers questions people used to argue about for hours. And remember: “no difference” is also a result. It saves money because you stop spending time on that idea.
Do it yourself: your first honest test
Take your best creative and write one hypothesis using the formula “if X, then Y, because Z”. For example: “if we show the price in the first frame, clicks will go up, because people instantly see whether they can afford it.” Make variant B that differs only in that. Launch it through the platform’s built-in A/B test, not as two separate campaigns. Write down the length in advance, at least a week, and the main metric. Don’t peek before the end. Then log the result, even if there’s no winner. Reread the log once a month: patterns will show up that become your own rules.
Check yourself
- What should Nika do?
Correct answer: Wait until the end of the week, as planned. One day is too little data, and the leader is often random. The length is set in advance and you don’t peek, so the week covers weekdays and weekends. - Which conclusion is honest?
Correct answer: Too little data: compare clicks or collect more sales. Three sales versus five is ordinary random variation, like 7 heads out of 10 flips. You need more events or a metric with lots of them. - What’s wrong with this test?
Correct answer: Too many differences: you can’t tell what worked. Hook, price and music changed at once. There’s growth, but Nika doesn’t know what caused it and can’t repeat it. One difference per test. - What should she do first?
Correct answer: Check the pixel, links and discounts in both groups. That’s Twyman’s law: a result that’s too good usually means a mistake. Bing also looked for a bug first and celebrated only after.
← AI labels and platform rules Automated campaigns: Advantage+, PMax, AI Max, Smart+ →