You rewrote a page. A month later your brand shows up more often in ChatGPT answers. Did your change cause that?
Maybe. But AI answers move on their own, even when you change nothing. This post shows how to separate a real gain from background drift, with a worked example you can check by hand.
Before and after is not enough
The obvious test is to measure your mention rate, make a change, and measure again. If the number went up, the change worked.
The problem is that many other things changed in that month too:
- The engine may have been updated.
- Other sites may have published new content the engine now cites.
- A competitor may have dropped out of the answers for reasons of its own.
- Seasonal interest may have shifted what the engine retrieves.
Any of these can move your mention rate up or down. If everything drifted up by 6 points and your fixed pages went up by 15, only part of that 15 was your doing. A before and after comparison cannot tell you which part.
Hold some questions back
The fix is an old idea from experiments: a control group.
Before you make changes, split the questions you track into two groups:
- Changed questions. The questions whose answers your edits are meant to affect.
- Control questions. Similar questions you leave untouched. You do not edit the pages that serve them.
Both groups live through the same month. The same engine updates, the same news, the same seasonal shifts. So whatever happens to the control group is a good estimate of the drift. Whatever extra happens to the changed group is a good estimate of your effect.
The number you care about is:
gain over control = (change in the changed group) minus (change in the control group)
Every number needs a range
Mention rates come from a limited number of answers, so each one has uncertainty. The gain over control is built from four rates, so it carries the uncertainty of all four. That means the gain needs its own 95% range.
The rule is simple:
- If the whole range is above zero, the gain is proven at that level of confidence.
- If the range includes zero, the change is not proven yet. It might have worked. It might not. You do not know.
A worked example: Halden Coffee
Halden Coffee is a made-up coffee roaster in Austin. We use it as an example across this site. The numbers below are invented for illustration, but the arithmetic is real.
Halden tracks 24 buyer questions on one engine. They split them into 12 changed questions and 12 control questions. Each question is asked 20 times in a month, so each group has:
12 x 20 = 240 answers per month.
They improve their wholesale and subscription pages, which serve the 12 changed questions. They leave the pages behind the control questions alone.
| Group | Month before | Month after | Change |
|---|---|---|---|
| Changed | 72 of 240 (30.0%) | 108 of 240 (45.0%) | +15.0 points |
| Control | 70 of 240 (29.2%) | 84 of 240 (35.0%) | +5.8 points |
A before and after view would say “+15 points, great.” But the control group also rose by almost 6 points with no edits at all. So:
Gain over control = 15.0 minus 5.8 = about +9 points.
Now the range. To keep the math visible here we use a simple normal approximation. Each rate p from n answers has a variance of p x (1 minus p) / n. Add the four variances, take the square root, and multiply by 1.96:
- Changed, before: 0.30 x 0.70 / 240 = 0.000875
- Changed, after: 0.45 x 0.55 / 240 = 0.001031
- Control, before: 0.292 x 0.708 / 240 = 0.000861
- Control, after: 0.35 x 0.65 / 240 = 0.000948
Sum: about 0.003715. Square root: about 0.061, or 6.1 points. Times 1.96: about 11.9 points.
So the 95% range for the gain is roughly 9 minus 12 to 9 plus 12, which is about -3 to +21 points.
The range includes zero. Massyl’s verdict would be: not proven yet. The change might be worth 20 points. It might be worth nothing. One month of 240 answers per group cannot tell.
What to do when it is not proven yet
“Not proven yet” is not the same as “it failed.” It means you need more evidence. Some options:
Wait for more answers. Suppose the next month shows the same pattern, so Halden now has 480 answers per group with the same rates. The gain is still about +9 points, but the variances are halved:
| Answers per group | Gain over control | 95% range |
|---|---|---|
| 240 (one month) | +9 points | about -3 to +21 |
| 480 (two months) | +9 points | about +1 to +18 |
With twice the data, the whole range sits above zero. Now the gain is proven, though the lower end is close to zero, so the honest summary is “a real gain, somewhere between small and large.”
That second row assumes the next month looks like the first. It might not. That is exactly why you wait and measure instead of assuming.
Do not keep changing things. If you edit the control pages, or make new edits to the changed pages every week, you break the comparison. Let the test run.
Do not peek and stop on a good day. Checking every day and declaring victory the first time the range clears zero inflates your error rate. Decide in advance how long the test runs, or use a method built for early stopping.
Look at the answers themselves. A range tells you how sure you are about the rate. It does not tell you why. Open the stored answers. Is the engine now citing the page you edited? Is it repeating a fact you added? That is useful evidence alongside the number.
Picking good control questions
The control group only works if it is a fair stand-in. A few guidelines:
- Similar questions. If your changed questions are about wholesale coffee, your controls should be similar buyer questions, not something unrelated.
- Similar starting rates. In the example, both groups started near 30%. If one group starts at 5% and the other at 60%, they may drift differently.
- Split before you look. Choose the groups before you start editing, not after you see which ones moved.
- Enough questions on each side. More questions means more answers, which means narrower ranges.
How Massyl handles this
In Massyl, proof is part of the fix workflow, at launch. When you approve fixes, some questions are held back untouched as a control group. After the changed pages are indexed, Massyl re-checks both groups and reports the gain of the changed pages over the control, with its range. If the ranges overlap, it says the change is not proven yet rather than claiming a win.
Every answer behind those numbers is stored with its date, engine, country and language, and a link you can share, so you can show a client or a colleague exactly what changed.
Summary
- AI answers drift even when you do nothing, so before and after can mislead.
- Hold back similar control questions and leave them untouched.
- Measure gain over control, not raw gain.
- Put a 95% range on the gain. If it includes zero, the change is not proven yet.
- More answers narrow the range. Be patient and keep the test clean.
To size your ranges before you start, try the free margin-of-error calculator. To read more about fixes and proof, see fixes and publishing. Founding prices are on the pricing page.