AI Visibility Levels Are Unstable – New Research Shows Too Much Statistical Noise
AI visibility tracking data is not completely reliable. Because generating models often generate different responses, the quoted stocks and rates on your dashboard are simply snapshots of an ever-changing target, not static facts.
The difference between you and your competitor can be real or just a variation between measurements. A new IQRush paper due out next week (we had access to the pre-release) offers a way to break this down, showing that no fixed amount of data can answer the question definitively.
The paper is by Ron Sielinski, founder of IQRush, which sells software that measures the visibility of AI as the paper argues it should. The reason it’s worth your time is that a different team published a similar repeated measurement finding in April, so IQRush isn’t the only one making this case.
How Much Do These Numbers Go?
Repeatedly querying SearchGPT, Gemini, or Perplexity for the same query can produce different sources each time. They are designed to add some randomness to each response, so each citation is one of the many URLs it might pull. A previous paper by the same author examined this variation, showing that, for example, when testing SearchGPT on running gear, Tom’s Guide accounted for about 9.5% of the citations, while Runner’s World averaged 6.0%. On the dashboard, Tom’s Guide appeared more often, but a large margin of error meant that the figures overlapped. With only one sample, it was not true to say that Tom’s Guide outperformed Runner’s World, as the difference of 3.5 points was within the margin of error. A new paper aims to prevent this error by addressing a simple but often overlooked question: How much data is needed before a standard is truly meaningful?
When a Position is Worthy of Trust
The answer has two parts, and both need to be true for the standard to be reliable. First, the order must stop changing.
Initially, the rankings may change frequently as new answers are added because no site has a clear limit yet. It is only after enough responses have been collected that the top sites begin to stand out clearly, allowing the ranking to stabilize. Also, it is important that the top sites are well spaced; if they are very close, the ranking may not matter, as the strong competition does not really show who is really ahead. The paper looks at whether the difference between the top sites is greater than the margin of error for each. If so, the rate reflects the real difference. If not, it’s probably just statistical noise. Both conditions need to be true at the same time, and one is not enough. In the 30 field topic tests, the number of answers required in both conditions to be met ranges from 33 to 94, counting only answers with a quote.
Three of the 30 didn’t get to this point even after 125 queries, all on SearchGPT, where the top sites were too similar to be separated. No single cut works everywhere; what works for one platform and title may not work for another.
We’ve Been Around This
In January, I discussed SparkToro’s discovery that AI tools provide a different list of recommended products more than 99% of the time you ask the same question. That article left one question unanswered: how many times do you have to ask yourself before the results are stable? This paper provides the clearest answer I have come across.
Rand Fishkin, who led that study, offers some helpful advice. Before spending any money on AI visibility tracking, he suggests making sure your provider is “showing their stats.” The IQRush paper is a great way to do this because it provides a simple stop rule, so you don’t have to rely solely on knowing how many runs are enough.
It’s also consistent with a series of studies released by SEJ over the past year, each reporting AI numbers cited as if they were fixed. This one turns around, checks the ratio itself, and asks if those numbers are stable enough to be compared in the first place.
What This Changes in Your Reporting
The number on your dashboard is just one sample. Before trusting it, check if your tracker does the same test over and over and reports the range, or if it pulls the data once and shows a clean value. A clean figure can be a warning, not a guarantee.
Profit after content change is easy to misinterpret. For example, a three-point increase in your SearchGPT citation share may be seen as evidence that your effort has paid off, but such a change may fall within the natural variability of sequential runs, according to the original paper’s data.
For a win, measure before and after more than once each. A single before and after reading cannot distinguish your change from normal sound.
The platform you scale changes how much data you need, not in the way you might guess. It comes down to how much independent information each answer carries, not how many quotes it gives you. Gemini aggregates quotes from several similar sites into one answer, many of which tell you the same thing. SearchGPT provides several citations per answer but spreads them out, so each answer carries more independent information than the raw count suggests. The same number of answers in two engines does not buy the same confidence, and the budget that solves Gemini can leave you guessing at SearchGPT.
Sometimes the honest answer is that you can’t say yet. Three of the 30 tests did not cleanly separate their top sites within the budget. For those, it’s the right call to hold, not publish a rate that the data won’t support. A tracker that can tell you “not enough data” is worth more than one that prints a reliable order every time you ask.
The top level area is the part you can protect the most. With enough responses, the leaders move from the middle of the tail, albeit unintentionally. Margins of error widened rapidly below the front, until the neighborhoods turned on a coin, and even the top 10 were spotless, with the standard margin of error in the top-10 running about five places and one in five wider than 10. Trust the leaders, treat the middle and bottom as a bad thing, and don’t directly report past places before the list.
What Paper Doesn’t Prove
None of this comes from a completed, peer-reviewed study. It’s a preprint built on 30 forum topic tests across three engines, using ChatGPT generated queries rather than actual user searches, over a single cluster. Exact numbers won’t transfer cleanly to your topics, so treat them like problem shapes, not lookup tables.
Those statistics include answers that only contain citations, which is very important to SearchGPT, because half of its queries return no citations at all. In one article, 125 questions produced 104 usable answers, a 17% miss rate, so you’ll need to submit more questions than those numbers suggest.
Method testing is in, too. The paper compares the early rate against the final rate of the same cohort, not against any external fact. That checks if the stopping rule is consistent with it, which is why the same result from the unrelated group does the real work here. The authors of that April paper, Julius Schulte, Malte Bleeker, and Philipp Kaufmann, are researchers at the University of St. Gallen. They used a different dataset and came to the same conclusion, that a single reading is unreliable and you have to sample the engine repeatedly to trust what it tells you.
Where This Goes
Paper stops short of something most people will want, which is a way to know your operating budget before you start collecting. Sielinski leaves that to later work and notes that the number depends on the citation pattern of each platform, so a single international budget is unlikely to help.
The big change is that AI impression reporting is moving in the same way that ad reporting and analytics has gone, to numbers that carry a margin of error instead of a false decimal point. That happens when plumbing isn’t available yet, as Search Console can’t tell you which clicks came from AI. Until then, the onus is on you to run the check more than once and report the range, not the single number your dashboard gives you.
More resources
Featured Image: Stokkete/Shutterstock
