The Product Page Your Company Didn't Write

2026-10-01

Ask an AI shopping assistant the same question about the same product twice, five minutes apart, and the odds it gives you the same answer are worse than you'd think. A benchmark published on September 22 by Product.ai ran 220 shopping questions five times each across four systems — Gemini, ChatGPT, Claude, and Perplexity — and found that 86 percent of questions produced at least one confirmed factual conflict between runs. Gemini's paid tier invented a product detail outright — a nonexistent ingredient, a wrong spec, a model name that doesn't exist — on 21 percent of questions, and delivered an error costly enough to mislead a buyer's purchase on 54 percent.

Two days later, a separate analysis from Suff Digital asked 300 "what's the best…" questions ten times each. Gemini named a different top pick in 82.7 percent of identical, back-to-back queries. ChatGPT changed its answer 56.3 percent of the time. Neither system has settled on an answer a company could treat as its answer.

The deeper problem sits one layer down from the inconsistency. An academic preprint posted September 16 compared the source domains behind ChatGPT's and Gemini's product answers across 1,536 responses and found the two systems shared, on average, just 5.4 percent of the sites they drew from — and shared none at all in 76.7 percent of comparisons. Two systems, asked about the same product, are not just reaching different conclusions. They are reading almost entirely different material to get there.

Put the three findings together and a real company's real product — not a hypothetical, an actual named item with a real price and a real spec sheet — is being described to an unknown number of shoppers with a detail invented on the spot, a price that may be wrong, and a source trail that two systems answering the same question wouldn't even agree exists. The buyer who gets that answer doesn't know it was improvised. Why would they? The assistant delivers it the way it delivers everything else — confidently, specifically, with no hedge attached. The same quote, tested across eight AI search engines, came back attributed correctly less than half the time, and almost never with a flagged doubt.

What makes the shopping case different from the citation case is that nobody downstream is even asking where the claim came from. A journalist checking a quote might ask for a source. A shopper comparing three products for a Saturday purchase is not going to audit the chatbot's reasoning before clicking buy — or before clicking away because the assistant just said a competitor's version is cheaper, lighter, or better reviewed. The error doesn't surface as a correction request. It surfaces as a return, a one-star complaint about a feature the product never had, or a sale that quietly went somewhere else, with nothing connecting any of it back to the five seconds an AI system spent generating an answer.

This has already happened at scale once, outside the shopping context specifically. A Minnesota company sued Google after its AI Overview invented a regulatory lawsuit that no real source had described, built from four real pages that said nothing of the kind — and the company only found out because someone noticed and said something. Multiply that pattern across every product question an AI shopping assistant answers about a company's catalogue, each one independently fabricated, each one gone by the next time someone asks, and the scale of what goes unrecorded stops being a curiosity about chatbot reliability and starts being a blind spot in how a company understands its own market.

The honest reading of these three studies, taken together, is not that one system is worse than another — though Gemini's numbers are notably worse across all three. It's that none of the major systems has reached a point where a company can assume its products are being described accurately, consistently, or from the same sources twice in a row, and none of them keeps a copy of what they said for a company to check later. The studies exist because researchers ran the same question enough times to catch the pattern. Most companies only get that view by accident, if a customer happens to screenshot the wrong answer before the next version quietly replaces it. The question worth asking isn't whether these systems will get more consistent. It's who is keeping a timestamped copy of what they said today, before the next run says something else entirely.