Looking at an AI Visibility Report? It May Be Measuring Nothing

AI answers rotate the brands they name every couple of days. A single check measures nothing, and a vendor selling one is selling noise. What a real measurement looks like, and what to ask before you pay for one.

A founder forwarded me a report last month with a proposal stapled to it. The report showed one screenshot: a competitor named in the AI answer to the buyer’s question, her company absent. The proposal quoted a five-figure fee to fix that.

I ran the same question through the same assistant six times. Her company appeared in three of the answers and the competitor in four. Nobody was invisible, and nobody owned the answer. The vendor had measured one die roll and sold her the result as a verdict.

At ESBO Ltd, the link-building and brand-mentions agency I run, we have checked what assistants say about client brands every month since early 2025, and the routine’s first lesson was to distrust our own screenshots. The early ones were just as misleading as the report that the founder received.

The answer changes every two days

The instability has now been measured at scale. Ahrefs tracked more than 43,000 queries with AI Overviews for about a month and found that the answer had changed in 70% of cases between one check and the next, with the average version lasting about two days. Consecutive answers shared only 54.5% of their cited sources, and the entities named in them, which is where brands live, overlapped 54%.

What held still was the meaning. The semantic similarity between versions averaged 0.95 out of 1, so the assistant keeps its opinion and rotates its cast. For any one brand, that turns presence into a rate. You are named in some share of the answers to a question, and that share is the only number worth knowing.

One question has at least four different answers

A single measurement can move for reasons that have nothing to do with you. Ask the same question twice and the sample changes. Rephrase it slightly, and the shortlist will change with it. Ask a different assistant and the sources change, since each reads a different library. Ask in a different language and, as a controlled study showed this summer, the whole market can change.

Academics now treat this as a sampling problem. A framework posted to arXiv in September by marketing researcher Edward Malthouse and colleagues starts from the observation that repeated responses to one query produce different brands and rankings, and measures how often a brand appears, and how prominently, by sampling the same question many times. Applied to six models, it found prominence tracked online conversation about a brand more closely than conventional popularity, which is the mention thesis arriving from another direction. The model to hold in your head is a draw from a distribution, and you can only describe a distribution by drawing from it repeatedly.

A real number is a rate with a method attached

A trustworthy measurement reads like a lab note: named in seven of 20 runs this month, against five of 20 last month, on the same ten questions, on the same two assistants, in the same language, from the same location. Each of those conditions has to be stated, because changing any one of them changes the result. A report that leaves them out cannot be compared with anything, including itself a month later.

That means the useful questions for any vendor or for your own team are about methods rather than findings. How many runs per prompt? Fewer than ten is a screenshot with extra steps. Which assistants, logged in or logged out, and from which country? In which languages? Can I see the full prompt list and the raw answers, not a dashboard summary? And will use the identical method next month, so the trend is real? A vendor who answers all five plainly is worth talking to. One who changes the subject to the dashboard is selling you a slideshow of dice.

Panic and celebration are both measurement errors

The practical damage from bad measurement runs in both directions. A single zero sends a founder into a crisis budget for a problem that may not exist, which is what the report I was sent was designed to do. A single win does the opposite: someone screenshots the one run that named the company, posts it on LinkedIn and stops doing the work. (I have done the second one myself. The next run did not name us.)

The work is the same whichever way the screenshot points, and it has not changed since I made the case for brand mentions here: get the name into the sources the assistants read, in the places and languages your buyers use. Good measurement does not replace that work. It tells you whether the work is moving the rate, which is a question a screenshot can never answer.

Run the check yourself before anyone sells you one. Ten questions a buyer would ask, two assistants, ten runs each, once a month, logged in a spreadsheet. It takes an afternoon and costs nothing, and in three months you will own the only kind of AI visibility number that means anything: a rate, measured the same way twice, heading in a direction.

The founder who forwarded that proposal did not hire the vendor. She hired an intern for a day a month to roll the dice properly, which is a better deal than most in this market. A die that shows a six once is not a six. It is a die, and the only honest thing anyone can tell you about it is how often it lands there.

AI answers rotate the brands they name every couple of days. A single check measures nothing, and a vendor selling one is selling noise. What a real measurement looks like, and what to ask before you pay for one.

A founder forwarded me a report last month with a proposal stapled to it. The report showed one screenshot: a competitor named in the AI answer to the buyer’s question, her company absent. The proposal quoted a five-figure fee to fix that.

I ran the same question through the same assistant six times. Her company appeared in three of the answers and the competitor in four. Nobody was invisible, and nobody owned the answer. The vendor had measured one die roll and sold her the result as a verdict.