How to Measure AI Visibility: Sample Sizes and Honest Baselines
A 9.5% vs 6.0% citation share looks like a win until the confidence intervals overlap. How many repeats you actually need to measure AI visibility.
Measuring AI visibility means sampling a distribution, not reading an instrument. Ask ChatGPT, Gemini or Perplexity the same question twice and you get different answers citing different sources. So any single-run number — your share of voice, your citation rate — is an estimate with error bars, and the error bars are wide. The practical work is deciding how many repetitions you need before a difference means anything.
Why does the same question give different answers every time?
Because non-determinism is built into how these systems run, not into a setting you can switch off. Thinking Machines Lab sampled 1,000 completions at temperature 0 from Qwen3-235B-A22B-Instruct-2507 with the prompt “Tell me about Richard Feynman”, 1,000 tokens each.
“Surprisingly, we generate 80 unique completions, with the most common of these occuring 78 times.” — Thinking Machines Lab, Horace He, September 2025
Eighty distinct answers, all diverging from token 103 onward. The root cause they identify (September 2025) is not temperature but a lack of batch-size invariance: server load varies unpredictably, which changes the order of floating-point accumulation, which changes the output. You cannot control that from outside. You can only sample repeatedly.
The vendors agree, quietly. OpenAI’s own documentation (checked 16 July 2026) promises only “(mostly) deterministic outputs” when you fix the seed, and warns that configuration changes on their side alter results. An academic study — arXiv:2408.04667 (August 2024), five models including GPT-4o and Llama-3-70B-Instruct, eight MMLU and BBH tasks, ten repeats each with temperature=0, top-p=1 and a fixed seed — found no model maintained 100% consistency. Accuracy gaps between identical runs reached 5-15 points, and up to 70 points in the extreme case (Mixtral on college maths: 75% on its best run, 3% on its worst).
Google documents the same thing for search. Per Google Search Central: “AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary.” Variability is documented behaviour, not a bug you should wait out.
Why is a single measurement almost always misleading?
Because your metrics are sample estimates, and the industry reports them as facts. The clearest published demonstration comes from Ronald Sielinski’s Quantifying Uncertainty in AI Visibility (arXiv:2603.08924v2, June 2026), a study of 200 queries per topic across three topics and three platforms, sampled over nine consecutive days plus 25 samples at ten-minute intervals — 374,052 citations in the daily dataset.
“These findings show that single-run visibility metrics provide a misleadingly precise picture of domain performance in generative search.” — Sielinski, Quantifying Uncertainty in AI Visibility, June 2026
His running-gear example on SearchGPT is the one to remember. Over 200 queries, tomsguide.com showed 9.5% citation share and runnersworld.com 6.0% — an apparent 3.5-point advantage. But the 95% bootstrap confidence intervals — approximately 5.5%-12.5% and 4.0%-8.0% — overlap substantially. It is a statistical tie dressed up as a lead.
Two more numbers quantify how loose the ground is. Repeating an identical query returns mostly different sources: median Jaccard overlap at domain level is 0.29-0.31 on Gemini, 0.33-0.40 on SearchGPT and 0.50 on Perplexity, with identical-citation rates of 0.01-0.10% on Gemini and 3-8% on the other two. And volatility itself is large: a log-std of 0.5 (platform means: 0.504 Gemini, 0.421 Perplexity, 0.417 SearchGPT) means a domain’s citation share typically swings between roughly 60% and 165% of its central value.
Sielinski, citing Hu et al., also reports a 33% inconsistency rate in whether an AI Overview appears at all for a given query. We could not verify that figure in the original source, so treat it as reported rather than measured — but the implication stands: the first source of variance is not what the answer says, it is whether an AI answer exists.
The rule worth memorizing: any citation-share difference below 5-7 percentage points between two brands should be treated as a tie. Overlapping intervals in that range are, in Sielinski’s words, the norm rather than the exception.
How many repetitions do you actually need?
Enough to make the interval narrower than the difference you care about — and the paper publishes the numbers, which is exactly what almost no vendor blog does.
| Metric | Target interval (95% CI) | Gemini | Perplexity | SearchGPT |
|---|---|---|---|---|
| Citation share | 5 points wide (≈ ±2.5 pp) | ~40-50 queries | ~100 queries | ≥150 queries |
| Citation prevalence | 15 points wide | ~140-150 queries | ~140-150 queries | ~60-80 queries |
Source: Sielinski, §5.7, June 2026.
One critical caveat comes with that table. Do not sample until the interval looks narrow and then stop. On SearchGPT, convergence is non-monotonic: interval width narrows, then widens again. A practitioner who stops at a fortuitously narrow point reports uncertainty lower than the truth. Commit to a fixed sample size in advance, based on the platform and topic — never use the running interval as a stopping criterion.
Can you measure ChatGPT with the OpenAI API?
You can, but you will not be measuring what your customers see. Surfer SEO (February 2026) reports that only 24% of brands overlap between API results and scraped ChatGPT results, and that for cited sources the overlap collapses to 4%. On Perplexity, API-versus-UI source overlap is 8%.
Methodology note, because it matters: Surfer describes “1,000 prompt executions” without specifying whether those are unique prompts or repetitions, which model versions were used, or whether web search was enabled on the API calls. It is not peer-reviewed, and the vendor has a commercial interest in the conclusion. We publish it as directional evidence, not as a measured figure.
Surfer’s own structural data points the same way, though it comes from the same vendor and the same unrefereed study: ChatGPT’s API averages 406 words versus 743 for the scraped interface, around 23% of API calls never trigger a web search (the interface always does), around 25% of API responses cite no sources at all (7 on average when they do, versus 16 in the UI), and the API fails to detect brands around 8% of the time.
Which metrics actually mean something?
Start by separating a mention from a citation: a mention is the model writing your name; a citation is the model linking you as a source. Different mechanics, different fixes. Beyond that, Sielinski formalizes three:
- Citation count — absolute citations to your domain. Not comparable across platforms.
- Citation share — your citations divided by total citations. Normalized, comparable.
- Citation prevalence — the fraction of responses containing at least one citation to you. Measures breadth of coverage, not depth.
All three are sample estimators, not fixed properties of the system. For prominence inside the answer, the GEO paper (Aggarwal et al., KDD 2024, 10,000-query GEO-bench) offers Position-Adjusted Word Count — words attributed to your source, exponentially discounted by position — and Subjective Impression, which scores seven dimensions including relevance, influence, uniqueness and click probability.
One hard rule: never compare raw counts across platforms. Median citation volume runs 5-7 per response on SearchGPT versus 36-40 on Gemini. A count that looks impressive on Gemini may be below average there and excellent on SearchGPT. Our comparison of AI visibility tools goes deeper on which platforms report which metric.
How do you build a free baseline, step by step?
You do not need enterprise software to start, and Google Search Console will not help you: per Google Search Central, traffic from AI features is not broken out separately — it is folded into the “Web” search type. That gap is why you sample yourself.
- Fix 30-50 real buying-intent questions. Comparisons, “best X for Y”, “alternatives to Z” — the ones your customers actually ask, not the ones you wish they asked.
- Run logged out. Personalization contaminates the sample.
- Decide your repetition count in advance. Use the table above; write the number down before you start.
- Log four fields per run. Do you appear? In what position? Alongside which competitors? Are you cited or merely mentioned?
- Compare by interval, not by point. If the gap is under 5-7 points, you are tied. Say so.
Also worth knowing: Google states there are no additional requirements or special markup — no AI text files, no bespoke schema — to appear in AI Overviews or AI Mode. Anyone selling you a magic file is selling nothing. If you are new to the discipline, our primer on what GEO is covers the fundamentals first.
What does API sampling really cost?
Tens of dollars, not thousands — and the web search tool, not the tokens, dominates. Using OpenAI’s official pricing (checked 16 July 2026): gpt-5.4-mini is $0.75 per million input tokens and $4.50 per million output; gpt-5.4-nano is $0.20 and $1.25; gpt-5.6-luna is $1.00 and $6.00. The web search tool bills separately at $10.00 per 1,000 calls plus the retrieved content at model rates.
CALCULATED — a baseline of 50 prompts × 30 repetitions = 1,500 search-enabled calls. Search tool: 1.5 × $10.00 = $15.00. Assuming (ASSUMED, not measured) ~5,000 input tokens per call on gpt-5.4-mini — a figure meant to cover the prompt and the retrieved search content, which bills at model rates — and ~800 output tokens: 7.5M input × $0.75 = $5.63, and 1.2M output × $4.50 = $5.40. Total ≈ $26. Re-verify prices on the day you run it; model names and rates in this family change often. And remember what you bought: this measures the API, not ChatGPT.
What moves the needle once you have a baseline?
Correlational evidence points at brand notoriety rather than at how much you publish. Ahrefs (December 2025) studied 75,000 domains with DR above 40 across ChatGPT, AI Mode and AI Overviews:
| Factor | Correlation with AI mentions |
|---|---|
| YouTube mentions | ~0.737 |
| Branded web mentions | 0.656-0.709 |
| Branded anchors | 0.511-0.628 |
| Branded search volume | 0.352-0.466 |
| Domain Rating | 0.266-0.326 |
| Content volume (number of site pages) | ~0.194 (“almost no relationship”) |
This is correlation, not causation. Launching a YouTube channel will not make ChatGPT cite you; the likelier reading is that both YouTube mentions and AI mentions measure the same underlying brand notoriety. We will not promise otherwise, and neither should anyone else. For context on how fast these surfaces are growing, see our note on AI search adoption.
When should you pay for a tool, and what should you demand?
The threshold is arithmetic: prompts × repetitions × platforms ÷ one person’s available time. Below it, a spreadsheet wins. Above it, buy — but interrogate the vendor first:
- Does it measure the real product or the API? (Remember the 24%.)
- Does it report confidence intervals, or just a number?
- Is the sample size declared?
- What is the re-sampling frequency, and is it fixed in advance?
If a provider hands you a share of voice with no error bar, they are selling you noise with a decimal point.
What should an SME do this week?
Pick your 30-50 questions, decide the repetition count before you look at any result, run them logged out on ChatGPT, Perplexity and Google, and log mention versus citation separately. Then re-run the same fixed sample monthly and compare intervals, not points. Anything under 5-7 points of movement is weather, not climate.
If you would rather have that baseline built and defended for you, that is what our AI visibility service does — and our method labels every number as measured, calculated or assumed, so you always know which is which.
Frequently asked questions
How many times do I need to repeat the same query to measure AI visibility properly?
It depends on the platform and the metric, and there are published figures. According to Ronald Sielinski's paper (arXiv:2603.08924, June 2026), a 95% confidence interval spanning about five percentage points on citation share, meaning a margin of roughly plus or minus 2.5 points, requires roughly 40-50 queries on Google Gemini, about 100 on Perplexity, and 150 or more on OpenAI SearchGPT. For citation prevalence with intervals spanning fifteen points, SearchGPT needs 60-80 queries while Gemini and Perplexity need nearly twice as many, around 140-150. The reason is that repeating the same query returns mostly different sources: the median domain-level overlap between two identical runs is only 0.29-0.31 on Gemini and 0.50 on Perplexity. A single measurement is not a measurement. It is one sample from a distribution.
Why does the same question give ChatGPT different answers if I set temperature to zero?
Because temperature only controls the token sampling step, not the rest of the system. Thinking Machines Lab showed in September 2025 that sampling 1,000 completions at temperature 0 with the same prompt produced 80 unique completions. The root cause they identify is not temperature but a lack of batch-size invariance: server load varies unpredictably and changes the order of floating-point accumulation. OpenAI's own documentation only promises outputs that are mostly deterministic even when you set the seed parameter. And an academic study (arXiv:2408.04667) confirmed that with temperature=0, top-p=1 and a fixed seed, none of the five models tested maintained 100% consistency. Variability is not eliminated. It is measured.
Can I measure my ChatGPT visibility using the OpenAI API?
You can, but you will not be measuring what your customers see. Surfer SEO published in February 2026 that only 24% of brands overlap between API responses and scraped ChatGPT responses, and that for cited sources the overlap falls to 4%. The structural differences explain it: the API averages 406 words versus 743 in the interface, around 23% of API calls do not trigger web search at all while the interface always does, and around 25% return no sources. That study does not publish model versions or web search configuration, so treat it as directional rather than precise. The direction is still coherent: the real product carries layers such as search, system prompts and personalization that the API does not reproduce.
Which metrics should I track: mentions, citations, position or sentiment?
First distinguish a mention from a citation. A mention is the AI writing your name. A citation is the AI linking you as a source. They are different things and they are optimized differently. The literature formalizes three citation metrics (Sielinski, 2026): citation count, which is not comparable across platforms; citation share, your fraction of total citations, which is comparable; and citation prevalence, the fraction of responses where you appear at least once. The academic GEO paper presented at KDD 2024 adds prominence inside the answer with Position-Adjusted Word Count and Subjective Impression, which covers seven dimensions including relevance, influence and click probability. Sentiment is a separate layer: it does not change your share, it changes what each appearance is worth. For a small business, starting with prevalence and share is enough.
Can I build a baseline for free or do I need to pay for a tool?
You can start for free, and for many small businesses that is the sensible choice. The method: fix a set of 30-50 real buying-intent questions your customers ask, decide in advance how many repetitions you will run, execute them on ChatGPT, Perplexity and Google, and log in a spreadsheet whether you appear, in what position, and alongside which competitors. Two evidence-based warnings. First, do not use the rule of repeating until the number stabilizes and then stopping. Sielinski shows that on SearchGPT the interval width narrows and widens again, so anyone stopping at a favorable point is fooling themselves. Second, run logged out so personalization does not contaminate the sample. You pay for a tool when prompts times repetitions times platforms stops fitting into one person's time, not before.
How much does API sampling cost and how often should I re-measure?
OpenAI's official prices (checked 16 July 2026) let you calculate it: gpt-5.4-mini costs 0.75 dollars per million input tokens and 4.50 per million output, and gpt-5.4-nano costs 0.20 and 1.25. The dominant cost is not tokens but the web search tool, billed separately at 10 dollars per 1,000 calls plus the tokens of the retrieved content. So a baseline of 50 prompts with 30 repetitions, meaning 1,500 search-enabled calls, lands in the tens of dollars rather than thousands, but remember it measures the API and not the real product. On frequency: the reference study shows volatility day to day and even at ten-minute intervals, so re-sampling daily without confidence intervals only produces noise that looks like a trend. For a small business, a monthly cycle with a fixed sample size, compared by intervals rather than by point, is defensible. Weekly only if your sector moves fast.