Why AI answers change when nothing else did

The experiment we did not mean to run

On 2 September 2026 we scanned a customer’s question bank twice by accident. The first scan finished at 06:07:17 UTC. The second started at 06:07:20 — three seconds later, same questions, same engines, same brand, same market. Nothing about the world had changed in between.

Of the 49 question-and-engine verdicts both scans scored, 4 disagreed. Three of them were Gemini, one Perplexity. That is 8.2% of the measurement flipping with no cause upstream of it.

The part worth sitting with: both scans found the brand cited exactly 18 times. The headline was identical. Underneath it, four verdicts had swapped places. A report that showed only the total would have looked perfectly stable while 8% of its evidence churned.

So we measured it properly

That pair was one accident, so we went back through every scan we have stored. For each pair of consecutive scans of the same brand, we compared the recorded verdict for every question and engine that both scans scored. Verdicts are taken after the majority vote, the same way a report collapses repeated runs, and an engine that returned no grounded answer is left out rather than counted as a miss.

That gives 2,024 comparable verdicts across 62 scan pairs and 11 brands, between late July and late October 2026. 197 of them flipped: 9.7% (95% confidence interval 8.4% to 11.0%).

Then we split those pairs by how far apart the two scans ran, because a gap of three weeks can contain a genuine change and a gap of two days effectively cannot:

An earlier version of this page made something of that difference — short-gap pairs moving more than long-gap ones, which would be a strange result if time were the cause. On the data we had in October it did not hold up: 14.3% against 11.4% was well inside what chance produces at those counts (p ≈ 0.19), and we retracted it. On today’s data the gap is wider, 13.0% against 9.0%, and a simple test calls it significant (p ≈ 0.02). We still do not lean on it. Only 8 of the 62 scan pairs ran under 2.5 days apart, and verdicts from one pair of scans rise and fall together, so that test overstates how sure we can be. We are leaving the retraction visible rather than deleting it, because a page about measurement error should show its own.

The conclusion the claim was reaching for holds anyway, by a cleaner route.

The same minute is at least as unstable as the next week

On Growth and above we ask Gemini and Claude the same question three times inside a single scan, and take the majority. That gives a second measurement of churn that has nothing to do with time passing: how often those runs disagree with each other. Set it against how often the same question flips between one scan and the next:

Three calls in the same minute disagree at least as often as the same question asked days or weeks later. For Claude the two are indistinguishable (7.1% against 7.6%, p ≈ 0.78). For Gemini the same-minute figure is the higherone (15.0% against 11.2%, p ≈ 0.03). The two columns are not quite like for like — the first counts any disagreement among a question’s runs, the second compares the majority verdicts, and a majority smooths some disagreement away — but no reading of them makes time the cause. Whatever is driving the churn, it is not the passage of time, and it is not your marketing. It is the engines.

The obvious objection is that the instability might be ours rather than theirs — that we are reading the same answer differently on different days. We are not, and the reason is structural: the verdict is decided by matching the brand’s domain against the sources the engine returned, which is ordinary string comparison. It uses no model, no randomness and no clock. Hand it the same answer twice and it returns the same verdict twice. A language model does read each answer alongside it, for sentiment and for which other sites were cited, but it has no vote on whether you were cited. So when two runs disagree, the thing that changed is the engine’s answer.

Which is the finding, put plainly: on these questions the engines do not hold a settled view of whom to cite. Asked three times, an engine will cite you twice and someone else once. That is not a measurement failing to pin down a fact — it is an accurate measurement of something genuinely unfixed. A brand cited in all three runs occupies a different position from one cited in two, even though both are “cited” once the majority is taken.

It is not the same across engines

Restricting to the short-gap pairs, where a real change is implausible:

On this stratum Claude is nearly four times steadier than Gemini, and the difference is unlikely to be chance (p ≈ 0.04). It is not explained by how often we ask: Gemini and Claude are both asked three times per question on Growth and up, and Perplexity and ChatGPT once. Repetition is not what separates them — the engines simply behave differently.

How large the gap is, though, depends on which pairs you look at, and we would rather say so than pick the flattering cut. Across allpairs rather than short-gap ones alone, where Claude has 301 comparisons instead of 49, it flips 7.6% against Gemini’s 11.2% — still steadier, but about 1.5 times rather than four, and at that sample the difference could still be chance (p ≈ 0.09). The direction is consistent across both cuts. The multiple is not settled. Use the direction: a move on Gemini needs more evidence behind it than the same move on Claude.

Why it happens

Four mechanisms, none of them a fault in anyone’s product:

• The answer is generated, not retrieved. A model samples its output. The same input can produce different text, and a site that was worth citing in one pass gets left out of the next for no reason you could point at.
• The search underneath it is live. Grounded answers run a real web search first. Indexes shift, results reorder, and a source that made the cut at 06:07:17 can miss it at 06:07:20.
• Whether the engine searches at all varies. Sometimes it answers from training knowledge and never looks anything up. We record that as no grounded answer rather than a miss, but it means the set of questions actually measured differs run to run.
• Prominence is unstable even when presence is not. You can be cited in both runs and move from second source to eleventh, which changes the score without changing whether you were cited.

Why this does not contradict our methodology page

How we measure reports that repeating a call within one scan left Perplexity’s verdicts unchanged. This page reports that 13% of Perplexity verdicts flipped between scans under two and a half days apart. Both are true, and the gap between them is the finding: an engine can be stable across three calls in the same minute and unstable across the same call on two different days. Repeating a question inside one scan buys you less than it looks like it should.

What to do with this

• Do not act on one week’s movement. On the numbers above, a week-over-week change of roughly one verdict in eleven is inside the range these engines produce on their own. It is not a result.
• Read direction over several weeks, not levels between two. Four readings trending one way is evidence. One reading against the one before it is a coin landing.
• Weight the engines differently. A Claude move is worth more than a Gemini move of the same size.
• Look at which questions moved, not just the score. The back-to-back pair above had an identical total and four swapped verdicts. If the same three questions flip every week, that is noise with a pattern. If a question that never used to cite your site starts citing it and stays, that is the thing you were looking for.
• Be suspicious of a supplier whose chart only ever goes up. Ours does not, and any honest measurement of this will sometimes report a drop it cannot explain.

How to check this yourself

Everything above comes from two tables, geo_scans and geo_results, and one comparison: group the stored rows by question and engine within each scan, take the majority verdict, then diff consecutive scans of the same brand. We are describing the method rather than only the result so that a customer on Growth or above can run it against their own history and get their own number. If yours disagrees with ours, we would like to see it: hello@mediadesk.asia.

Background: what generative engine optimisation is, and how we measure it.