A founder pulled me his test dashboard last month, he’s proud of it. Forty-one trials are in effect. I asked him to name three that made a real difference last quarter. He walked in silence, scrolled for a while, and sat down in another. It is possible.
He doesn’t care. He quickly faced a problem that comes with all growing groups. The hard part of making an experiment was usually building it. Notified the designer, waited for ad variants, added tracking, built the page. A week of work to get one test live, with maybe an hour of real thought behind it. The construction week is now over. He can present 40 exercises in the time it once took him to present one, so he does, and almost none of them teach him anything.
Capacity has never been a factor holding teams back. What stopped them was telling the real result from the random noise, and finding the muscle to kill the losers before they spent the budget. AI solved the cheap problem and left the expensive one sitting where it was. It then gave everyone a quick way to be wrong.
So, here is the important rule now. The framework you want is the one that becomes harder to master as the tests become easier to use.
What’s Cheap
The asymmetry I wrote about in the team building goes right through the test pipeline. Different spinning costs nothing today. Writing a testable hypothesis costs what you used to do. The model will size your test in seconds and record a weekly reading in a minute, and it can’t tell you to believe that reading. That takes someone who has been burned with good enough curves to not trust the next one.
Point AI at production work and keep a clear head on the hypothesis, design, and execution call, as well as everything else involved. Figure it all out, and you’ve built a sound machine faster than you can handle.
Start with Fewer Bets
My first move with the new team is to reduce the test backlog, not feed it. Ask the model for ideas, and she will gladly give you 200. A list of 200 unrated ideas is not a strategy. It’s a way to keep busy while important bets wait their turn. The task is to choose five who are counted in this quarter and say no to the other 195 out loud, where the group can’t hear.
We frame all ideas with three questions:
- How big is the win when it comes?
- How sure are we going in?
- How much will it cost to run?
Cheap, high-confidence, high-end ideas. The one the founder saw on LinkedIn at breakfast waits in line like everything else, unless it clears the same bar. A score sheet is not a discipline. Discipline kills common sense before it eats for three weeks.
One client wanted to outsource all of their onboarding workflows with instinct. It scored poorly for confidence and even worse for cost, so we did a three-screen test against the flow it already had. His instinct was wrong. The cheap test returned a quarter of the engineering time he was about to set on fire.
The model can write comments and output points. It won’t tell you what bet your company can afford. That phone is yours.
Create a Test so that the Answer Counts
Many “failing” tests never had a chance to succeed, because they weren’t designed to answer anything. A clean test moves one variable against the original control, runs to the sample size you fixed before you started, and keeps the guardrail at the number you refuse to hurt. Change the subject line and the structure and the audience at the same time, and the elevator will fly for you. You never know which move did the work. Read the result on the second day because the line goes up, and encourage the sound to have a strategy.
This is where AI comes in, in a narrow and practical way. I rely on it to figure out how long a test has to run before it says anything, to simulate the result before I spend a dollar, and to catch the obvious confusion I remember at six in the morning. One thing I never let him do is pick a metric. Give a target to the model, and it will get you a nice win on the number that nobody pays, while the number that keeps the lights on slides quietly otherwise. The human-in-the-loop rule everyone repeats about AI content holds strong in experimental design.
Run the Machine, Not the Judgement
This is where AI finds its seat the most. Layout, unique permissions, QA, resizing, cross-platform formatting, hard-to-learn first draft: it’s all in the tools. Meta Advantage+ and Google Performance Max are available with creatives and bids. GrowthBook and Statsig use statistics and keep your test teams honest. Google Analytics 4 with Mixpanel or Heap handles event data. A model can convert raw results into plain English, so your analyst spends an hour reading instead of formatting slides. I posted the full stack elsewhere and won’t repeat it here.
What is left to man: the hypothesis, the definition of the metric, the judgment that the result is real, and the call to measure it or bury it. Stop work. Reserve judgment. Most of this framework resides in that one line.
Cadence You Can Trust
Speeding without a rhythm just puts you in danger quickly. We read one a week. Every live test leaves that room with one decision: scale, kill, or repeat. There is no “give it a few more days” unless the test has not reached the sample size we set. And each decision goes into the log, next to the hypothesis we tested and what we concluded.
That log does the quiet, unglamorous job that keeps the entire system honest. A year later, that’s why the new recruiting pitch comes across as “we did that in March, here’s what happened,” and why last quarter’s actual win doesn’t disappear a week after the ship. Exams are cheaper now. The log is what turns a bunch of them into something you really know.
One Series B client came to us running north of 20 “tests” a month and had absolutely no confidence in them. We cut it down to six powerful tests, took production to tools, and put a one-week scale-or-kill decision in front of one decision maker. Within the quarter, the hit rate on the tests they drove increased from a penny to nearly two-thirds, and the cost per acquisition dropped by 24%. They run for the third time as the majority of the test and end up trusting those who run.
How Budgeting Really Pays Off
The same few mistakes appear in almost every account, and the AI accelerates them all. Teams call the winner on the second day because the dashboard updates live and the curve looks friendly. They do a test that is too small to reach significance, and then read the fortunes from the static. They chase the number the model can’t move while the important number goes in the wrong direction. And it’s the most expensive practice of all: they never kill anything, so the backlog swells, spending is spread thin, and not a single test gets the right shot.
There is nothing new about this. The AI just put it on a fast clock, which is the whole reason the frame keeps its shape under speed.
The Takeaway
The teams that win in performance marketing in the AI era aren’t the ones with the most testing going on. They are the only ones who still believe in their results when the volume increases. Cheap execution is a real gift. It only pays if your standards rise as quickly as your output does. Make the system as difficult to pass as it gets easier to operate, keep the human from judging, and let the machine do the rest. That’s what holds up when the price of other tests drops to almost nothing.
Additional resources:
Featured Image: Prostock-studio/Shutterstock
