Measuring AI Visibility Across a 50,000 Page Health Content Library
Every AI visibility conversation inside a large publisher starts in the same place. Somebody asks which tool we should buy, somebody else asks whether we can see it in the analytics, and a third person asks how often we should check. All three are tracking questions, and tracking is the wrong discipline for this problem. A rank tracker works because there is a finite, enumerable list of keywords and a stable, deterministic result for each one. Neither condition holds here. The list of things a reader might ask an AI engine is effectively unbounded, and the same question asked twice does not reliably return the same answer.
That makes this a survey design problem, and survey design is a mature discipline with settled mathematics. The questions it asks are different and more useful: what population are we estimating, how precisely, with what sample, at what cost, and how large a change could we detect if one happened. A measurement programme that cannot answer those five questions is producing numbers, not evidence.
This page supplies the design. The strategic framework and the metric definitions sit in the generative engine optimisation pillar and are used here rather than restated. The commercial case for publishers is in GEO for medical publishers. The evidence on which sources engines actually cite, and the three study comparison behind it, is in how AI engines decide which health publisher to cite, which explicitly deferred the method to this page.
How can a large publisher measure its visibility in AI answers?
By drawing a stratified random sample of the questions readers ask, running it on a fixed cadence across more than one engine, and reporting every result with a margin of error attached. The sample is drawn from the question space rather than from the page library, which is why its size is set by the precision required and not by how many pages the publisher owns. A workable design at any library size above a few thousand pages is in the region of 600 prompts, allocated across intent and topic strata by commercial value, run three times each on at least one OpenAI surface and one Google surface. That is 5,400 queries per cycle and yields a margin of error of roughly plus or minus 2.2 percentage points on a portfolio citation rate near 10%. Analytics and server logs are supporting instruments rather than the measurement itself: referral data captures only the small fraction of AI exposure that produces a click, and crawler logs capture ingestion rather than visibility. The discipline that makes this credible is not the tooling, it is declaring the sample frame, the allocation and the detection threshold before the first query is run.
Why the census instinct is the first thing to drop
The instinct inside a large publisher is to want coverage. If there are 50,000 pages, the programme should look at 50,000 pages. That instinct is expensive, slow and, more importantly, measures the wrong thing.
| Census design | Queries per cycle | At a weekly cadence | Verdict |
|---|---|---|---|
| 3 prompts per page, 3 engines, 5 runs | 2,250,000 | 117,000,000 queries a year | Not fundable at any realistic unit cost, and that is before anybody classifies the results |
| 2 prompts per page, 2 engines, 3 runs | 600,000 | 31,200,000 a year | Still not fundable, and now also not informative, because two prompts cannot represent a page's demand |
| 1 prompt per page, 2 engines, 3 runs | 300,000 | 15,600,000 a year | The cheapest census is the least valid. One prompt per page assumes you already know the single question each page should answer |
| 600 stratified prompts, 3 engines, 3 runs | 5,400 | 280,800 a year | Fundable, defensible, and produces an interval you can put in a board paper |
The cost argument is the weaker of the two arguments against a census, and it is the one people usually make. The stronger argument is conceptual. A census of pages answers the question which of our pages get cited. That is not the question the business needs answered. The business needs to know what share of the demand in a given topic and intent segment we are present in, how that compares with the competitors in that segment, and whether it is moving. Demand does not arrive page shaped. A single well constructed condition hub might be the answer to four hundred distinct questions, and three hundred thin archive pages might be the answer to none.
So the first design decision is to stop enumerating pages and start enumerating questions. The page library then becomes what it actually is: the inventory you deploy against the demand, not the thing you measure.
The pricing tell
If a vendor prices an AI visibility programme per URL, per page or per domain crawled, they are selling you a rank tracker with a new label. The statistics of this problem do not have a per page unit in them anywhere.
The legitimate units are prompts, runs and engines, because those are the things that consume budget and move precision. Ask any prospective supplier to state the margin of error their standard package produces at a 10% citation rate. A supplier who cannot answer that question has not designed a measurement, they have built a dashboard.
The sample size mathematics, shown rather than asserted
This is the section that is missing from every vendor page in this category, and it is three lines of arithmetic. A citation rate is a proportion. The sample size needed to estimate a proportion to a stated precision at a stated confidence level is a solved problem, and the formula contains no term for the size of the population. That single absence is why a 50,000 page library needs no larger a sample than a 5,000 page one.
The formula, for the reader who wants to check our arithmetic n = z squared, times p, times one minus p, divided by e squared. z is 1.96 for 95% confidence, p is the citation rate you expect, and e is the margin of error you are willing to live with, expressed as a decimal. There is no population term. A finite population correction exists and it matters only when your sample is a large fraction of the population. The population here is the set of questions readers could ask, which is effectively unbounded, so the correction is irrelevant and the sample size is set entirely by precision and by the rate itself. |
Run that formula across the range of citation rates a health publisher will realistically see and the planning table looks like this. Every cell is the number of distinct prompts required, at 95% confidence.
| Expected citation rate | Margin of error ±5pp | ±3pp | ±2pp | ±1pp |
|---|---|---|---|---|
| 3% | 45 | 125 | 280 | 1,118 |
| 5% | 73 | 203 | 457 | 1,825 |
| 10% | 139 | 385 | 865 | 3,458 |
| 20% | 246 | 683 | 1,537 | 6,147 |
| 50% (the worst case, used when you have no prior) | 385 | 1,068 | 2,401 | 9,604 |
Two things in that table are worth sitting with. The first is how small the numbers are at the precision most businesses actually need. If your citation rate is around 5% and you can act on a reading of plus or minus 3 percentage points, 203 prompts is a defensible sample. A great many programmes are running tens of thousands of queries to produce a number they could have produced with two hundred, and spending the difference on repeats that add almost nothing.
The second is how brutally the cost of precision escalates. Going from plus or minus 3 points to plus or minus 1 point at a 10% rate takes you from 385 prompts to 3,458, which is nine times the budget for a threefold improvement. That is the square relationship in the denominator and it is not negotiable. Precision is the most expensive thing you can buy in this programme and it should be bought deliberately, at the level the decision actually requires.
Read it the other way: what your current budget actually buys
Most teams arrive with a budget rather than a precision target. Inverted, the same formula tells you what you are entitled to claim. The numbers below treat the prompt as the unit of analysis, which is the conservative and correct stance explained in the next section.
| Distinct prompts in the sample | Margin of error at a 5% rate | At 10% | At 20% | At 50% | What you can honestly say |
|---|---|---|---|---|---|
| 100 | ±4.27pp | ±5.88pp | ±7.84pp | ±9.80pp | Directional only. Adequate for is this zero or not. Never show a trend line |
| 200 | ±3.02pp | ±4.16pp | ±5.54pp | ±6.93pp | A usable single reading for one segment. Not enough to split further |
| 300 | ±2.47pp | ±3.39pp | ±4.53pp | ±5.66pp | The practical floor for a portfolio number you would put in front of a board |
| 600 | ±1.74pp | ±2.40pp | ±3.20pp | ±4.00pp | The recommended design point. Good portfolio precision, supports four to six reporting segments |
| 1,000 | ±1.35pp | ±1.86pp | ±2.48pp | ±3.10pp | Strong. Supports eight to ten segments at usable precision |
| 2,000 | ±0.96pp | ±1.31pp | ±1.75pp | ±2.19pp | Research grade. Appropriate for an annual benchmark, wasteful as a monthly tracker |
The 600 row is the recommendation, and the reason is the segment column rather than the portfolio column. A portfolio citation rate on its own is close to useless to an editorial team, because it cannot be acted upon. The number that changes behaviour is the segment reading: our clinical reference content holds this share, our commercial comparison content holds that one. 600 prompts is roughly where you can carry four to six segments at a precision that survives scrutiny, and that is the real constraint.
Detecting a change is a different and much harder problem
Estimating a rate and detecting a change in a rate are not the same calculation, and conflating them is the most common statistical error in this field. A programme sized to estimate a rate to plus or minus 2 points will routinely report movements of 2 to 3 points that are pure sampling noise. Detection requires power, and power is expensive.
| Change you want to detect | In relative terms | Prompts needed per period | Comment |
|---|---|---|---|
| 5% to 6% | a 20% improvement | 8,156 | Effectively undetectable for any programme at a normal budget. Do not promise it |
| 5% to 7% | a 40% improvement | 2,210 | Detectable with a serious quarterly benchmark, not with a monthly tracker |
| 5% to 7.5% | a 50% improvement | 1,468 | The realistic lower bound for a well funded programme |
| 5% to 10% | a doubling | 432 | Comfortably detectable at the recommended design point. This is the size of change a real content programme produces over two to three quarters |
| 10% to 12% | a 20% improvement | 3,839 | The number that ends most monthly reporting ambitions |
| 10% to 15% | a 50% improvement | 683 | Detectable at the recommended design point |
| 20% to 25% | a 25% improvement | 1,092 | Larger base rates are easier to move and harder to detect proportionally |
All of those figures are at 95% confidence with 80% power, which are the conventional settings and the ones a sceptical finance director will expect. The practical consequence is uncomfortable and worth saying plainly: a monthly AI visibility report at a normal budget cannot reliably detect the size of change a good content programme actually produces in a month. It can detect a catastrophe, and it can detect a doubling. Everything in between is noise being read as signal.
The right response is not to abandon the monthly report, it is to change what the monthly report claims. Report the monthly number as an estimate with its interval and no trend assertion. Make the quarterly reading the one that carries a verdict, and size it for detection rather than for estimation. That is the two cadence design set out later on this page, and it is cheaper than running a single over specified monthly programme.
Decide the smallest change worth acting on, first Before any sampling decision, answer this: how large would a movement have to be before we would do something differently? If the answer is a 50% relative improvement, you need 683 prompts at a 10% base and you are done. If the answer is any movement at all, you cannot have it at any price and the honest thing is to say so. This single question removes more waste from a measurement budget than any tooling decision. It converts an open ended monitoring commitment into a sized, finite, defensible programme. |
How many times should you run each prompt?
This is where most measurement budgets are quietly wasted. The same prompt asked twice does not reliably return the same sources, so the instinct is to run each prompt several times and average. The instinct is right and the implementation is usually wrong, because repeats of the same prompt are not independent observations and the statistics treat them accordingly.
Two answers to the same prompt are correlated: whatever drives the engine toward a particular source on run one is still operating on run two. The standard way to account for that is the design effect, which inflates your required sample in proportion to how correlated the repeats are. The effective sample size is the raw number of queries divided by one plus the number of extra runs multiplied by the intra prompt correlation.
Here is what that does to a fixed budget of 1,000 queries. Read across each row: the budget is identical in every cell, only the allocation changes.
| Intra prompt correlation | 1 run × 1,000 prompts | 3 runs × 333 prompts | 5 runs × 200 prompts | 10 runs × 100 prompts |
|---|---|---|---|---|
| 0.30, weakly correlated | n effective 1,000 | 625 | 455 | 270 |
| 0.50 | n effective 1,000 | 500 | 333 | 182 |
| 0.70, a realistic figure for citation presence | n effective 1,000 | 417 | 263 | 137 |
| 0.85, highly stable prompts | n effective 1,000 | 370 | 227 | 116 |
At a realistic correlation of 0.7, moving from one run to five costs you three quarters of your statistical power for the same money. That is not a marginal inefficiency. A programme running 10,000 queries as 1,000 prompts at ten runs each has roughly the effective sample of 1,370 well chosen single run prompts, and has paid seven times over for the privilege.
The independent evidence agrees, from a different direction A 2026 variance components study decomposed non determinism across 12,933 LLM responses, fully crossed over 20 brands, 8 query languages and 3 models, with about five repeats per cell. It was measuring sentiment rather than citation, which is a real limitation, and its conclusion about measurement design is the relevant part. It found that reliability for a single answer was approximately 0.01, and approximately 0.36 for the full crossed design. It also found that a repeat beyond the fifth reduced relative error variance by 0.0003, while adding languages and models reduced it substantially more per unit of query budget. Different outcome variable, same design conclusion: spend the budget on breadth, not on repetition. |
The exception that justifies repeats, and it is a real one
None of the above means run everything once. It means repeats are the wrong purchase for a portfolio estimate. They are the right purchase, and the only possible one, when you need a verdict on an individual prompt.
Suppose a prompt matters commercially and your true per run citation rate on it is 30%. A single run tells you nothing useful, because you will see a citation only 30% of the time and will conclude you are absent from a prompt you win roughly once in three.
| True per run citation rate on this prompt | 1 run | 3 runs | 5 runs | 10 runs |
|---|---|---|---|---|
| 20% | 0.200 | 0.488 | 0.672 | 0.893 |
| 30% | 0.300 | 0.657 | 0.832 | 0.972 |
| 50% | 0.500 | 0.875 | 0.969 | 0.999 |
Each cell is the probability that you observe at least one citation. At a 30% true rate, five runs catch you 83% of the time and ten runs catch you 97% of the time. If the question is does this prompt ever cite us, the answer requires repeats and there is no clever sampling around it.
So the design resolves into two tiers rather than one compromise. This is the single most useful structural idea on the page.
| Tier 1 — portfolio tracker | Tier 2 — diagnostic set | |
|---|---|---|
| Question it answers | What share of demand are we present in, and is it moving | Do we win this specific prompt, and what does the engine put there instead |
| Unit | The prompt, treated as a single observation | The individual prompt, examined in depth |
| Design | Many prompts, few runs. Breadth is everything | Few prompts, many runs. Depth is everything |
| Typical shape | 600 prompts × 3 runs × 3 engines per cycle | 30 to 60 high value prompts × 10 runs × 3 engines, refreshed quarterly |
| Queries per cycle | 5,400 | 900 to 1,800 |
| Reported as | Rates with confidence intervals, by segment | Named prompts with win rates and the competing sources, as a list |
| Audience | Board, finance, audience analytics | Editorial and commissioning. This is the tier that changes what gets written |
| Cadence | Monthly estimate, quarterly verdict | Quarterly, or on demand before a commissioning decision |
Most programmes run a single design and then try to answer both questions from it, which is why they produce portfolio numbers nobody trusts and prompt level findings nobody can act on. Separating the two costs very little, because the diagnostic tier is small by construction, and it is the change that most improves how the output is received internally.
Stratification: where the design earns its keep
A simple random sample of 600 prompts from the question space would be defensible and wasteful. Stratification, which means dividing the question space into groups and sampling within each, buys precision for free and, more importantly, makes the output usable by segment. It is also where the division of labour with the prompt map page sits.
Division of labour with the prompt map This page owns the statistical frame: how many prompts, how they are allocated across strata, how many runs, how many engines, and what precision results. The prompt map page owns the content of the question space: the four stage, four persona model of how readers move from an open question to a decision, and the commissioning calendar that follows from it. Build the map first and sample from it second. A sampling design applied to a question space nobody has modelled will be perfectly precise about the wrong population. |
Choose the strata before you choose the sample size
The number of reporting segments is the binding constraint on the whole design, because precision inside a stratum depends on the prompts allocated to that stratum, not on the total. A 600 prompt sample split twenty ways is twenty samples of thirty.
| Reporting segments | Prompts per segment at n = 600 | Margin of error per segment at a 10% rate | Usable? |
|---|---|---|---|
| 4 | 150 | ±4.8pp | Yes. Tight enough to compare segments against each other |
| 6 | 100 | ±5.9pp | Borderline. Fine for a single reading, too coarse for trend |
| 10 | 60 | ±7.6pp | No. Any two segments will overlap almost always |
| 20 | 30 | ±10.7pp | No, and actively misleading. A segment reading 8% and one reading 18% are not distinguishable at this precision |
Four to six segments at 600 prompts. That is the honest limit. A publisher who wants ten usable segments needs roughly 1,500 prompts, and should decide whether the tenth segment is worth two and a half times the budget. In almost every case the better answer is to report six segments properly and to rotate a seventh and eighth through the diagnostic tier.
Allocate by value, not by inventory
Once the strata are fixed, the allocation question is how many of the 600 prompts each stratum gets. There are three defensible answers and one that is almost always wrong.
- Proportional to demand, which is the usual default. Each stratum gets a share of the sample equal to its share of the estimated question volume. Simple, unbiased for the portfolio total, and it under samples small high value segments.
- Proportional to variance, the classical optimal allocation. Segments where your citation rate is near 50% carry the most uncertainty per observation and benefit most from extra sample. Mathematically efficient and commercially naive, because it spends the budget where you are most uncertain rather than where the money is.
- Proportional to commercial value, which is what we recommend. Weight each stratum by the revenue or strategic importance of the decisions it influences, with a floor of roughly 60 prompts per reporting segment so that nothing is reported at a precision that cannot support a sentence. A specialist oncology segment worth a quarter of licensing revenue deserves sample weight regardless of its share of question volume.
- Proportional to your page inventory, which is the wrong one and the most common one. Allocating sample according to how many pages you happen to have in a topic measures your publishing history rather than your market. If you have 4,000 legacy pages on a topic nobody asks about any more, that is a finding, not a reason to spend 40% of the sample there.
Whichever allocation you choose, write it down and keep it fixed between cycles. Changing allocation between periods makes the periods incomparable, and an allocation that drifts toward the segments that are performing well is a slow, invisible form of cheating that will eventually be noticed by somebody who matters.
What metrics replace rankings when traffic moves to AI?
Five, and they are defined in our GEO pillar rather than here: presence, visibility, share of voice, recommendation rate and average position. What this page adds is the measurement layer those definitions need in order to be reportable. Every one of them is a sample statistic, not a count, which means each must be published with a margin of error, a sample size, a stratum and a method note, or it is not a metric at all. The most important distinction among them is between presence, meaning you were cited somewhere in the answer, and recommendation, meaning the answer actively directs the reader toward you. Those two move independently, and a programme reporting only presence will show improvement while the commercially meaningful number stays flat. A ranking was a single integer that one person could check. None of these five is checkable that way, and the organisational adjustment that matters more than the metric choice is accepting that your headline number is an estimate with an interval around it.
What each metric needs before it can be reported
The definitions are settled. The reporting contract is not, and it is where programmes fall over. Each row below is the minimum a number needs carried alongside it before it goes into a deck.
| Metric, as defined in the pillar | Reporting unit | What must travel with it | The failure mode when it does not |
|---|---|---|---|
| Presence | Percentage of sampled prompts where the brand or domain appears at all | Sample size, margin of error, stratum, engine, cycle dates | Presence is the easiest metric to inflate, because adding low competition prompts to the sample raises it without anything improving |
| Visibility | Presence weighted by prominence within the answer | The position convention used, and whether a second mention counts | Two teams computing visibility differently will produce different numbers from identical data. The convention must be written down |
| Share of voice | Your presence as a proportion of the total citations in the sampled answer set | The competitor set, stated explicitly and kept fixed | A share of voice that improves because a competitor was dropped from the set is the most common fake win in this category |
| Recommendation rate | Percentage of sampled prompts where the answer actively directs the reader to you | The classification rule, and the inter rater agreement if humans are classifying | Recommendation requires a judgement call that presence does not. Without a written rule it drifts cycle to cycle |
| Average position | Mean ordinal position of your citation within answers where you appear | Whether absent prompts are excluded or scored, which changes the number enormously | Excluding absences flatters you as coverage falls. State the convention and never change it |
The column that causes the most arguments is the last one on average position. If you exclude the prompts where you were not cited, your average position can improve while your presence collapses, because the only answers left are the ones where you were cited early. Both conventions are defensible. Only one can be in use, and it has to be the same one every quarter.
Presence and recommendation move independently, which is the finding that matters
The reason this distinction carries so much weight at library scale is that the two metrics respond to different work. Presence responds to coverage: publishing on a topic at all, being crawlable, being structurally extractable. Recommendation responds to the comparative judgement the engine makes between you and the alternatives, which is a function of what the rest of the web says about you rather than of what you published.
A large publisher expanding its library will reliably see presence rise. It will not see recommendation rise from the same work, and when the two are reported as a single AI visibility score the dilution hides the problem for several quarters. The practical instruction is to report them as two lines on the same chart and to brief the board that they are expected to diverge.
A worked illustration of the dilution Suppose a publisher adds 8,000 pages across twelve new condition areas in a year. Presence in the sampled question set rises from 9% to 14%, which is a real and detectable improvement at the recommended design point. Recommendation rate over the same period moves from 3.1% to 3.3%, which at a 600 prompt sample is well inside the margin of error and is therefore no movement at all. Reported as a single blended score, this looks like a programme working. Reported as two lines, it is a clear instruction: the coverage strategy succeeded and the authority strategy has not started. Those are different budgets and different teams. |
GET ACTIONABLE AI AND HEALTHCARE INSIGHTS THAT HELP YOU STAY AHEAD IN A FAST-CHANGING PHARMA WORLD
The reporting trap nobody adjusts for
If you report six segments, three engines and five metrics, you are running ninety comparisons every cycle. Test each of them at the conventional 5% level and the arithmetic is unforgiving.
| Segments tested each cycle at the 5% level | Expected false movers per cycle | Probability of at least one false mover |
|---|---|---|
| 5 | 0.25 | 22.6% |
| 10 | 0.50 | 40.1% |
| 20 | 1.00 | 64.2% |
| 40 | 2.00 | 87.1% |
At twenty segments you will find something that moved roughly two cycles in three, whether or not anything happened. This is not a subtle statistical nicety. It is the mechanism by which a monthly AI visibility report always has a headline, and it is the reason those headlines so rarely survive the following month.
There are two acceptable fixes and one unacceptable one. The first acceptable fix is to adjust for multiple comparisons, controlling the false discovery rate rather than using a blunt Bonferroni correction, which at ninety tests would be so conservative that nothing would ever be reported. The second, and the one we prefer for a commercial audience, is to stop making movement claims at segment level altogether: publish each segment as an estimate with its interval, let the reader see that this quarter's interval overlaps last quarter's, and reserve movement language for the portfolio number and for changes that were pre registered as the thing you were trying to achieve.
The unacceptable fix is the status quo, which is to run the comparisons unadjusted and report whichever segment moved most. If your agency's monthly report always contains a win, that is not evidence of a good agency.
Why your analytics cannot answer this question
Every publisher tries the cheap route first, which is to look in the analytics for AI referrals and count them. It is a reasonable instinct and it fails for four separate reasons, each of which is sufficient on its own.
Reason one: the panels do not agree, and the disagreement is the methodology
Four credible 2026 measurements of AI referral traffic as a share of visits, each from a large panel:
| Panel | Figure | Sample | Window |
|---|---|---|---|
| Contentsquare, published January 2026 | 0.20% of traffic mix | 6,500 websites | Q4 2025 |
| Ahrefs, published May 2026 | 0.28% of total web traffic | 74,752 websites | March 2026 |
| SE Ranking, published June 2026 | 0.32% of all website traffic | 101,574 websites | January to April 2026 |
| Conductor, page updated July 2026 | 1.08% of all website traffic | 1,215 US enterprise domains across 10 industries | May to September 2025, the oldest window of the four |
That is a 5.4 times spread between the highest and lowest readings of the same quantity. The highest figure also comes from the oldest window and from enterprise domains rather than typical sites, so the apparent growth story runs backwards. Sample, window and denominator explain the whole range, which is precisely the lesson of the citation studies compared in our page on how engines choose health sources. If you cannot say which of those four numbers describes a site like yours, you cannot use any of them as a benchmark.
Health specifically sits at the low end. One 2026 vertical analysis put AI penetration of the health vertical at 0.17%, down from 0.23%, and a publisher specific reading at 0.08% of organic sessions. Those use different denominators again and are not comparable with each other, which is the point.
Reason two: the referral lands on the wrong page
Even where the referral is captured, it frequently cannot be attributed to the content that earned it. One 2026 panel found that roughly 60% of AI referred visits land on a homepage, against 17% of visits from organic search. A citation of a specific clinical page can therefore arrive in your analytics as an unattributed homepage session.
The attribution has also been moving. The share of ChatGPT referrals attributed to its own homepage rather than to a specific conversation rose from around 26 to 29% to around 62 to 63% during 2026. A publisher who built a dashboard on the earlier pattern saw their AI referral data change shape for reasons that had nothing to do with their content. Referrer strings are a product surface decision made by somebody else, and they can change without notice.
Reason three: most of the exposure never produces a click at all
The click is no longer the event. Pew Research Center, in a study of around 900 users and nearly 70,000 Google searches, found:
| Behaviour | Without an AI Overview | With an AI Overview | Implication for measurement |
|---|---|---|---|
| Search led to an organic click | about 15% | about 8% | Roughly half the click rate. Any referral based metric is measuring a shrinking fraction of the same exposure |
| Clicked a link inside the AI summary | Not applicable | about 1% | The citation itself almost never produces a visit. A programme measuring citations by their clicks is measuring about one in a hundred of them |
| Ended the browsing session | 16% | 26% | The answer completed the task. This is the outcome the publisher model was built to prevent and it is now the modal case |
Google has disputed the study's methodology, which should be noted and does not change the direction. If roughly one percent of AI Overview appearances produce a click on a cited link, then referral data is a one percent sample of your citation exposure, non randomly selected, with no way to weight it back up. That is not a measurement instrument. It is a conversion signal, and it is valuable as one.
Reason four: crawler logs measure a different thing entirely
Server logs showing AI crawler activity tell you about ingestion, not about visibility. The gap between the two is enormous and has been quantified. Crawl to refer ratios derived from Cloudflare Radar data, for July 2026, with July 2025 shown alongside:
| Operator | July 2025 | June 2026 | July 2026 | Direction |
|---|---|---|---|---|
| Anthropic | about 38,744 to 1 | about 3,386 to 1 | about 1,917 to 1 | Narrowing sharply, from an extreme base |
| OpenAI | about 1,104 to 1 | about 647 to 1 | about 251 to 1 | Narrowing |
| Perplexity | about 195 to 1 | about 207 to 1 | about 289 to 1 | Widening, the only one of the four moving the wrong way for publishers |
| about 5.5 to 1 | about 5.0 to 1 | about 4.7 to 1 | Stable, and in a different order of magnitude from the rest |
Read those as orders of magnitude rather than as precise figures. They are Cloudflare Radar derived and relayed through secondary reporting, the periods are not perfectly aligned, and Cloudflare itself has cautioned that the ratios may overstate the gap because referrals from native apps carry no referrer header at all. The direction is clear and the measurement lesson is clearer: a thousand crawler hits is not visibility, and counting them as a proxy will produce a number that moves for reasons entirely outside your control.
Crawler logs remain worth keeping. They answer the question of whether your content is being ingested at all, which is a prerequisite for citation and a live question for any publisher considering blocking. That decision, and the licence, block or optimise framework behind it, belongs to a separate page in this cluster and is not argued here.
What the analytics layer is actually for
Keep all of it, and move it to the right job. Referral data is your conversion and value instrument: it tells you what an AI referred visitor is worth relative to an organic one, and the evidence on that is encouraging rather than otherwise, with multiple 2026 panels reporting materially higher conversion rates from AI referrals despite the low volumes.
Crawler logs are your ingestion instrument, and they feed the crawler policy decision.
Neither is your visibility instrument. The sampled prompt panel is. Three instruments, three questions, and the failure is in asking one of them to answer all three.
The complete design, assembled
Pulling the pieces together for a health publisher with a library in the tens of thousands of pages. Every figure below follows from the sections above and can be re derived from them.
| Design parameter | Recommendation | Why this value |
|---|---|---|
| Sample frame | The question space, modelled before sampling begins | Pages are inventory, not demand. The prompt map page supplies the model |
| Reporting segments | Four to six | Precision per segment collapses beyond six at this sample size |
| Prompts per cycle, tier 1 | 600 | ±2.4pp at a 10% portfolio rate, and 100 to 150 per segment |
| Allocation | Weighted by commercial value, floor of 60 prompts per reporting segment | Spends the budget where the decisions are, without leaving any segment unreportable |
| Runs per prompt, tier 1 | 3 | Enough to damp the worst single run noise, few enough that the design effect stays near 2.4 |
| Engines, tier 1 | 3, including at least one OpenAI surface and one Google surface | Engine is a large source of variance and the two families behave differently |
| Tier 1 queries per cycle | 5,400 | 600 × 3 × 3 |
| Prompts, tier 2 diagnostic | 30 to 60 high commercial value prompts | Small by construction. This tier is read as a list, not as a rate |
| Runs per prompt, tier 2 | 10 | Catches a 30% true rate 97% of the time. Repeats are the whole point of this tier |
| Tier 2 queries per cycle | 900 to 1,800 | 30 to 60 × 10 × 3 |
| Total per cycle | 6,300 to 7,200 queries | Modest. The expensive part of this programme is classification, not querying |
| Cadence | Tier 1 monthly as an estimate, quarterly as a verdict. Tier 2 quarterly | Monthly cannot detect realistic change. Quarterly can detect a 50% relative move |
| Detection threshold, declared in advance | A 50% relative change in the portfolio rate | 683 prompts detects 10% to 15% at 95% confidence and 80% power. Anything smaller is not funded and is not claimed |
| Fixed between cycles | Strata, allocation, competitor set, position convention, classification rules | Changing any one of them makes the cycles incomparable. Change them at a stated annual reset, never mid year |
On cost, the querying is the cheap part and this surprises people. At a few cents of effective cost per query, a 6,000 query cycle is a few hundred dollars. The real expense is classification: deciding whether a mention counts as a recommendation, whether a competitor belongs in the set, and whether an answer changed because your content changed or because the engine did. Budget for the analyst, not for the API.
The reporting template
A number from this programme is not reportable until it carries five fields. Adopt this or write your own that carries the same five, and apply it to every figure without exception, including the ones that flatter you.
| Field | What it contains | Example |
|---|---|---|
| Point estimate | The rate itself, to one decimal place and no more | 11.3% |
| Interval | The 95% confidence interval, always shown, never collapsed into the headline | 9.1% to 13.9% |
| Sample | Distinct prompts, runs per prompt, engines, and the stratum | n = 150 prompts, 3 runs, 3 engines, clinical reference segment |
| Comparison | The prior period's estimate and interval, with an explicit overlap verdict | Q2: 10.1%, 8.0% to 12.6%. Intervals overlap: no detected change |
| Method note | Anything changed since last cycle, and the convention used for anything contestable | Position convention: absent prompts excluded. Competitor set unchanged since January. Engine mix unchanged |
The comparison row is the one that will be resisted internally and it is the one that matters most. Writing intervals overlap: no detected change next to a number that rose from 10.1% to 11.3% feels like underselling the work. It is instead the thing that makes the programme credible the first time a number falls, because a team that reports honest non findings in good quarters is believed in bad ones.
One further discipline: pre register the quarter's hypothesis At the start of each quarter, write down in one line what you are trying to move and in which segment. Then report that number first, whatever it did, before any other segment is discussed. This single habit removes the multiple comparisons problem from your headline, because the headline is no longer whichever segment happened to move. It also converts the measurement programme from a monitoring function into an experimental one, which is the only form in which it will ever justify its cost. |
Limitations of this method, and of this page
Several of the numbers above rest on assumptions that a careful reader should see clearly.
| Limitation | Why it matters | What to do about it |
|---|---|---|
| The intra prompt correlation of 0.7 is an illustrative value, not a measurement | It drives the entire design effect argument. If the true correlation for health citation presence is 0.3, repeats are much less wasteful than this page implies | Estimate your own from tier 2 data. Ten runs on 40 prompts gives you a direct empirical read on how stable your prompts are. Do this in the first quarter and re size tier 1 accordingly |
| The variance components study measured sentiment, not citation | Its specific coefficients do not transfer to a citation outcome | Used here only for its design conclusion, which is that breadth beats repetition per unit of budget. The design effect arithmetic stands on its own and does not depend on that study |
| The normal approximation behind the sample size tables breaks down at low rates and small samples | At a 3% rate with 60 prompts, the approximation can produce an interval that includes impossible values | Use a Wilson or Agresti Coull interval whenever the expected count falls below about ten. At the recommended 600 prompt design point the approximation is sound |
| The referral panel figures use different denominators and are not strictly comparable | That is the point being made with them, and it means none of the four should be quoted as the benchmark | Quote the range, never a single figure. If you need a benchmark for a site like yours, you will have to measure your own |
| The crawl to refer ratios are Cloudflare Radar derived and relayed through secondary reporting | Periods are not perfectly aligned and Cloudflare has cautioned that native app referrals carry no referrer header, so the ratios may overstate the gap | Treated as orders of magnitude throughout this page and used for direction only |
| The design assumes a stable question space within a cycle | A news event or a guideline change can reshape health demand inside a month | Hold the core sample fixed for comparability and add a clearly labelled supplementary set for the event. Never quietly fold event prompts into the tracking sample |
| Nothing here measures whether being cited is worth anything | Citation is exposure. The value of that exposure is a separate and largely unsolved question | Pair the visibility programme with the referral conversion data, which is the only part of the stack that touches value at all |
Conclusion: design the measurement before you buy the tool
The reason AI visibility reporting has such a poor reputation inside large publishers is not that the tools are bad. It is that the measurements were never designed. A dashboard was bought, it produced numbers, the numbers moved, and nobody could say whether the movement meant anything. Every one of the problems in this page follows from skipping the design step, and every one of them is solved by a few hours of arithmetic before any money is spent.
The design itself is not complicated. Model the question space, choose four to six segments, sample six hundred prompts weighted by commercial value, run each three times across three engines, keep a small deep diagnostic tier alongside it, declare the smallest change you would act on, and publish every number with its interval. That is a programme that costs less than most teams are currently spending and can survive a conversation with a sceptical finance director, which is a test very little in this category currently passes.
For what the question space should contain, see the health publisher's prompt map. For the evidence on which sources engines actually cite and why three published studies disagree, see how AI engines decide which health publisher to cite. For the metric definitions used throughout, see the generative engine optimisation pillar, and for the commercial case, GEO for medical publishers. The crawler ingestion question raised in the logs section is decided in licence, block or optimise, and the structural work that follows a measurement finding sits in condition hub architecture for AI answers and the medical review workflow.
Where Multiplier AI fits This is the method our platform runs, and we have published it in enough detail that a competent analytics team could execute it without us. That is deliberate. A measurement supplier who will not show you the sampling design is asking you to trust a number you cannot audit, and in a regulated industry that is not a position any buyer should accept. The pharma AI visibility and GEO platform implements the two tier design described here, with the intervals and method notes attached to every figure. Our wider healthcare professional data work sits on the GenAI doctor data platform and the HCP data platform for healthcare recruitment, and everything we publish is indexed on the Multiplier AI blog. |
Frequently Asked Questions For AI Visibility Measurement for Publishers
By sampling the question space rather than auditing the page library. You model the questions your audience asks, divide them into four to six segments, draw a stratified sample of roughly 600 prompts weighted by commercial value, run each three times across at least three engines, and report every result with a 95% confidence interval attached. That is 5,400 queries per cycle and gives a margin of error of about plus or minus 2.4 percentage points on a portfolio citation rate near 10%. The key insight is that sample size is set by the precision you need, not by how many pages you publish.
No, and this is the most expensive misconception in the field. The formula for the sample size needed to estimate a proportion contains no term for the size of the population. 600 prompts buys the same precision for a 500 page site as for a 500,000 page one. What a larger library may justify is more reporting segments, and segments are what drive sample size upward, not pages. If a vendor prices an AI visibility programme per URL or per page, the pricing model does not match the statistics.
Three for portfolio measurement and around ten for a verdict on an individual prompt. Repeats of the same prompt are correlated, so they add much less information than independent prompts do. At a realistic intra prompt correlation of 0.7, a budget of 1,000 queries spent as 200 prompts at five runs each has the effective sample of about 263 independent prompts, against 1,000 if spent as single runs. Repeats are still essential when you need to know whether a specific prompt ever cites you, because at a 30% true rate a single run finds you only 30% of the time.
Between 139 and 3,458 depending on the precision you want, at a citation rate near 10% and 95% confidence. Plus or minus 5 percentage points needs 139 prompts, plus or minus 3 points needs 385, plus or minus 2 points needs 865 and plus or minus 1 point needs 3,458. 600 is the practical design point because it gives roughly plus or minus 2.4 points on the portfolio and still leaves 100 to 150 prompts per reporting segment.
No. Referral analytics measures a small, non random and shrinking fraction of your AI exposure. Four credible 2026 panels put AI referral traffic at between 0.20% and 1.08% of visits, a 5.4 times spread caused by differences in sample, window and denominator. Roughly 60% of AI referred visits land on a homepage rather than the cited page, and Pew Research found that about 1% of searches with an AI Overview produced a click on a link inside it. Keep analytics as your conversion and value instrument and measure visibility with a sampled prompt panel.
No. They tell you about ingestion, which is a prerequisite for citation and not a measure of it. The gap is large: crawl to refer ratios derived from Cloudflare Radar data for July 2026 run at roughly 1,917 to 1 for Anthropic, 251 to 1 for OpenAI, 289 to 1 for Perplexity and 4.7 to 1 for Google. Crawler logs belong in the decision about whether to allow, block or licence access, not in the visibility report.
Presence, visibility, share of voice, recommendation rate and average position. The most important distinction is between presence, meaning you appear in the answer at all, and recommendation, meaning the answer actively directs the reader to you. Those two move independently, and a publisher expanding its library will usually see presence rise while recommendation stays flat. Every one of the five is a sample statistic rather than a count, so each must be published with a sample size, a confidence interval and a method note.
Monthly as an estimate and quarterly as a verdict. A monthly reading at a normal budget cannot reliably detect the size of change a good content programme produces in a month. Detecting a move from 10% to 12% requires roughly 3,839 prompts per period, while detecting 10% to 15% requires 683. Report the monthly number as an estimate with its interval and no trend claim, and reserve movement language for the quarterly reading, which can be sized for detection.
Almost certainly because of unadjusted multiple testing. If you test twenty segments each cycle at the conventional 5% level, there is a 64.2% probability that at least one of them shows a false movement, and you expect one false mover every cycle on average. Either control the false discovery rate, or stop making movement claims at segment level and publish segments as estimates with intervals. A monthly report that always contains a win is a statistical artefact rather than a performance record.
Four to six at a 600 prompt sample. Precision inside a segment depends on the prompts allocated to it, so 600 prompts split twenty ways gives 30 prompts per segment and a margin of error of about plus or minus 10.7 percentage points, at which point two segments reading 8% and 18% are not reliably distinguishable. Ten usable segments needs roughly 1,500 prompts. The better answer for most publishers is six segments reported properly, with additional topics rotated through a small deep diagnostic set.
The design, and it takes a few hours rather than a procurement cycle. Decide the sample frame, the four to six reporting segments, the allocation rule, the runs per prompt, the engine mix, the position and recommendation conventions, and above all the smallest change you would actually act on. That last decision sizes everything else. A tool bought before those decisions are made will produce numbers that nobody can interpret, which is the normal state of AI visibility reporting today.
GET ACTIONABLE AI AND HEALTHCARE INSIGHTS THAT HELP YOU STAY AHEAD IN A FAST-CHANGING PHARMA WORLD
Let's Discuss Your Requirements
