When the AI sounds sure, who checked the number?

Disclosure: John Farrall, who publishes this site, leads go-to-market for Gemsen. Charlie Ko is Gemsen's co-founder and chief executive.

Six cases that did not exist

In the spring of 2023 a New York lawyer filed a brief in a personal injury suit against the airline Avianca. It cited six earlier court decisions in support of his client. None of them existed. He had found them with ChatGPT, and when he asked it whether the cases were real, it told him they were and that they could be found in the standard legal databases. The judge described the reasoning in one of the invented opinions as "gibberish" and sanctioned the lawyers and their firm.

The case is usually told as a story about careless lawyering. It is also an accurate description of how a language model fails. The model did not hesitate, flag uncertainty or change its tone. Its invented answer read exactly like a correct one would have.

The same failure, without the courtroom

Few businesses will ever file an invented legal citation. Many are about to do something closer to home: put an AI assistant in front of their own data and let people ask it questions. Why did margins fall in the Northeast last month? Which devices are starting to fail? Is the pricing change working?

These are the right questions, and a language model is a good way to ask them. People can use plain words instead of a query language, and an answer comes back in seconds instead of waiting in an analyst's queue. The risk is the one the court found. When an answer contains a number, the reader cannot tell from the prose whether the number came from the data or was produced because it looked right. In a courtroom a judge eventually checks. In a weekly operating review nobody may check at all, and a decision is made on the figure.

Why it happens

Language models are trained to produce the text most likely to follow from what came before. That makes them very good with language and with reasoning about a situation described in words. A figure in an answer is produced the same way as the words around it, which is how a model can state a plausible number with complete confidence. Counting across thousands of records, or applying a rule consistently across them, is a different kind of work, and statistical software has done it reliably for decades.

We measured the difference with a transit ticketing partner. Sixteen language models reviewed the same 64 payment cases, and each had to list every figure it relied on. Code then looked up each figure in the exact file that model had been given.

54% of answers cited a figure the model had never been given when it saw only the raw payment records, against 6% when it was also given figures that had already been computed from those records in code.

How it happened matters more than the rate. When we gave models a comparison with the typical device but withheld the device's own figure, many supplied the missing figure themselves and reported the comparison as if they had measured it. Where a figure required applying a rule across many records, the models did no better than guessing the most common answer, and none of them said so.

A model does not fall silent when information is missing. It fills the gap, in the same voice it uses when it is right.

Dividing the work

The response is to give each kind of system the work it does well, in the way a newsroom separates reporting from fact checking.

• You ask. A question in plain words, the way you would ask a colleague.
• The model plans. It decides what needs to be looked up. It does no arithmetic.
• Code computes. Every count, rate and comparison is worked out from your data.
• Every figure is checked. Each number in the answer is matched to the computed results. If one fails, the answer is blocked.

The checker is deterministic code, not another model, so it does not share the blind spots of the system it checks. An answer that fails is blocked rather than shown with a warning, because a warning leaves the wrong number in front of the reader.

This changes what an assistant can safely be used for. In a trading system built this way, a question about what would happen if a lagging order traded faster came back in under a minute, with every figure matched to the data. Asked about commission costs the data did not contain, the system said so instead of estimating. The second answer is the more important one. An assistant that can say "the data does not contain that" is the one you can let people rely on.

Keeping the checker honest

A check of this kind confirms that each figure exists, with that value, in the evidence. It does not confirm that the figure is attached to the right quantity, or that the reasoning from correct figures to a conclusion is sound. That is why the people who act on an answer still make the decision, and why the next test of any such system should compare its conclusions with human verdicts on the same cases.

The checker is also an instrument that needs calibrating. In our evaluation it surfaced three faults in our own setup, including an instruction that gave every model the wrong name for a field, which the checker then counted against the models. Each correction is recorded with its effect, so drift in the checker stays visible.

A question worth asking

Conversational access to business data is arriving quickly, and it will help the people who use it. Before relying on any such system, including the ones I build, it is worth asking one question: when the AI states a number, who computed it, and who checked it?

Sources. Mata v. Avianca, Inc., No. 1:22-cv-01461 (S.D.N.Y.), Opinion and Order on Sanctions, 22 June 2023. Partner evaluation: 61,440 calls by 16 language models on 64 payment cases from a transit ticketing partner, 10 August 2026; rates are an upper bound on model error. Trading demonstration: real Nasdaq market data, 12 December 2025; orders simulated.