You type “best coffee roaster in Austin” into ChatGPT. Your brand shows up. Good news? Maybe. Ask again tomorrow and it might not. Ask from a different account, a different country, or a slightly different wording, and the list can change again.
This post explains why a single AI answer is not a measurement, how many answers you need before a number means something, and why we would rather show “Not enough data” than a number we cannot stand behind.
AI answers change on their own
A search results page is fairly stable. If you rank third today, you will probably rank near third tomorrow. AI answers do not work like that.
Each answer is generated fresh. Several things can change it:
- Sampling. Language models pick words with some randomness. The same prompt can produce a different list of brands.
- Retrieval. Engines that search the web before answering may pull different pages each time.
- Model updates. The engine behind the answer can change without notice.
- Context. Country, language and wording all shift what comes back.
So when you check once and see your brand, you have seen one draw from a range of possible answers. It is a bit like flipping a coin once and announcing it always lands heads.
What we actually want to know
The useful question is not “did my brand appear in that one answer?” It is “out of all the times a buyer asks this, how often is my brand named?”
That is a rate. We call it the mention rate. If your true mention rate is 40%, you appear in about 4 out of every 10 answers over time. Any single answer is either a yes or a no. Only many answers together can tell you the rate.
How many answers do you need?
Here is a real example. Say the true mention rate is about 40%. We ask the same question a number of times and compute a 95% range around the result using the Wilson score interval, a standard method for rates.
| Answers collected | 95% range for the mention rate |
|---|---|
| 1 | 0% to 79% |
| 5 | 12% to 77% |
| 10 | 17% to 69% |
| 20 | 22% to 61% |
| 50 | 28% to 54% |
| 100 | 31% to 50% |
Read the first row again. With one answer, the honest statement is “your mention rate is somewhere between 0% and 79%.” That tells you almost nothing. It cannot tell a brand that is nearly invisible from one that shows up most of the time.
With 20 answers, the range is 22% to 61%. Still wide, but now you know you are not invisible and not dominant. With 100 answers, 31% to 50%, which is narrow enough to compare against a competitor or against last month.
What a 95% range means in plain words
A 95% range is a way of saying “given the answers we have, the true rate is very likely inside this span.”
More precisely: if you repeated the whole process many times and built a range each time, about 95 out of 100 of those ranges would contain the true rate. The other 5 would miss. So the range is not a promise, but it is a fair, standard way to show how sure we are.
Two practical rules follow from this:
- A narrow range means you can act on the number. If you are at 31% to 50% and a competitor is at 60% to 75%, they are clearly ahead.
- Overlapping ranges mean you cannot call a winner yet. If you are at 22% to 61% and a competitor is at 35% to 70%, you do not know who is ahead. You need more answers.
Why the range shrinks slowly
Look at the table one more time. Going from 1 answer to 20 cuts the range a lot. Going from 50 to 100 helps much less. That is how sampling works: to halve the width of a range you need roughly four times as many answers.
This is why Massyl does not ask each question a few hundred times. It would cost a lot and add little. Instead, each question is asked 20 times per engine per month, spread across the weeks, so one busy day or one odd model update does not dominate the result.
Combining questions gives a sharper picture
Twenty answers on one question gives a wide range. But you rarely care about one question alone. You care about how visible your brand is across the questions your buyers ask.
When you combine questions, the answers add up. Take 24 questions, each asked 20 times:
24 x 20 = 480 answers.
With a true rate of about 40%, those 480 answers narrow the overall range to about 36% to 44%. That is precise enough to track month to month.
So the per-question view tells you where to look. The combined view tells you how you are doing overall. Massyl shows both, each with its own range.
Why “Not enough data” is an honest answer
A single percentage with no range looks clean. “Your AI visibility: 42%.” But if that 42% came from 5 answers, the real figure could be anywhere from about 12% to 77%. The clean number hides that.
When a range is too wide to act on, Massyl says “Not enough data” instead of showing a number. We think that is more useful than a confident number that could mislead you into rewriting pages that were fine, or ignoring a problem that was real.
“Not enough data” is not a failure. It tells you one of two things:
- The question needs more answers before it can be judged. Wait for the rest of the month’s checks.
- The question is rarely triggering any brand at all, which is a finding in itself.
The opposite case also matters. Sometimes the answer is clear early. If a brand is named in nearly every answer, or in none, a handful of checks may already give a narrow enough range. In that case Massyl can stop early for that question and spend the effort where the picture is still blurry.
Every answer is kept
A number is only as good as the answers behind it. In Massyl, every answer is stored with its date, engine, country and language, plus a link you can share. If a rate looks surprising, you can open the answers and read them yourself. If a colleague doubts the result, you can send them the exact answer.
What to do with this today
You do not need any tool to apply the idea:
- Never judge from one answer. Ask the same question several times, on different days, before drawing a conclusion.
- Write down every result. A tally of “named / not named” is enough to start.
- Compare ranges, not single numbers. If you want to see how wide your range is, try our free margin-of-error calculator. It runs in your browser and sends nothing anywhere.
- Start from real buyer questions. The buyer question generator can help you draft them.
If you want Massyl to do the asking, counting and range math for you, you can join early access and see the founding prices. To learn more about the method, read how Massyl measures.