The exam that measured the wrong thing

A national reading test was introduced with a sensible aim: to find out whether eleven-year-olds could read well enough for secondary school. Within four years average scores had risen substantially. Within six, teachers and employers were reporting that reading had not improved at all.

Both observations were correct, and the explanation is now a standard example in the study of measurement.

When a measure becomes the thing being judged, behaviour reorganises around the measure. Schools did not cheat. They did something more rational and more damaging: they taught the test. The paper used a limited set of question types, so lessons began to concentrate on those types. The passages were always short, so practice with long texts declined. Inference questions had recognisable shapes, so students learned to answer the shape rather than read the passage. Each decision improved scores. Together they produced students who were better at the test and no better at reading.

This is sometimes stated as a law: a measure that becomes a target stops being a good measure. The formulation is neat and slightly misleading, because it suggests the problem is unavoidable. What the research actually shows is that the damage depends on how narrow the measure is. A test that samples widely and unpredictably from everything a reader should be able to do is much harder to teach to, because teaching to it and teaching reading become the same activity.

That is the practical lesson, and it is more demanding than abandoning tests. It requires building measures broad enough that gaming them is indistinguishable from doing the real work.