The model that memorised

A system trained to spot a disease in chest images reached 97 per cent accuracy in testing and failed in the clinic. It had learned to recognise the hospital.

The training data came from two sources: a specialist centre, where most of the positive cases had been collected, and a general hospital that supplied most of the negatives. The two used different scanners, and the images carried faint differences in contrast and in the position of a corner marker. A system asked to separate positives from negatives found a shortcut that separated them perfectly without looking at any anatomy at all.

This failure mode is called shortcut learning, and it is not a bug in the algorithm. The algorithm did exactly what it was asked: minimise error on the data given. The fault is in the dataset, and specifically in a correlation present in the training set that does not hold in the world. Every such correlation is a shortcut waiting to be taken, and the ones that matter are the ones nobody thought to look for, because looking for them requires knowing they exist.

The standard defence is to test on data from a site that contributed nothing to training. The accuracy figure drops, often sharply, and the drop is the only honest estimate. A secondary defence is to show which parts of an image the system is using; when the highlighted region is a corner marker rather than a lung, the problem announces itself.

The general lesson reaches beyond medicine. Any high accuracy obtained on data that shares a source with the training data is a statement about the dataset. Only performance on a genuinely new source is a statement about the task.