The machine that could not say no

Ask a language model a factual question it has no basis for answering, and the older ones would answer anyway, fluently and with every appearance of confidence. This was not a bug in the ordinary sense. It followed from what the system was trained to do.

A model of this kind is trained to produce text that resembles the text it has learned from. Human writing is overwhelmingly confident: encyclopaedias do not hedge, textbooks state, articles assert. A system trained to produce plausible continuations of such text will produce confident sentences, because confident sentences are what the pattern contains. Nothing in the training distinguished a confident sentence that happened to be true from one that happened not to be.

The difficulty is that confidence is the main signal readers use. In conversation with humans, hesitation usually tracks uncertainty, so we have learned to treat fluency as weak evidence of knowledge. That shortcut works reasonably well with people and fails completely with a system whose fluency is unconnected to whether it knows anything.

Later systems were trained with this in mind, and refuse or qualify far more often. The improvement is real and it is partial, because the underlying asymmetry has not gone away: a useful answer and a confident invention are produced by the same machinery, and the second does not announce itself.

The practical rule for a student is therefore not about trust levels. It is about verifiability. Use such a system freely for anything you can check — a draft to improve, an explanation you can test against a textbook, code you can run. Be far more careful with anything you cannot, particularly a specific figure, date or quotation, which is exactly the kind of detail it will supply most readily and most convincingly.