The experiment that worked too well

A classroom study of a new teaching method reported an improvement so large that the authors spent most of the paper explaining why they did not believe it.

The effect was about four times the size typically found in education research. Effects that large are occasionally real and usually a clue, and the usual explanations are mundane. The comparison group may have had a worse teacher. The test may have been written by the person who designed the method, and so may measure the method rather than the learning. The students may have been selected, consciously or not, for willingness to try something new. Any of these produces a large number honestly.

The authors' response was to make the finding harder to get. They repeated the study with the test written by an independent team, with classes assigned at random rather than by volunteering teachers, and with a comparison group taught by the same staff. The effect survived at about a quarter of its original size, which is still worth having and no longer remarkable.

This pattern is so common that it has a name: the decline effect, the tendency of striking findings to shrink as the design tightens. It is often reported as a scandal. It is better understood as the system working slowly. The first result was not a lie; it was a measurement taken in the easiest available conditions, and easy conditions flatter every intervention.

The practical reading habit follows. When a paper reports an unusually large effect, the question is not whether the researchers were careless but which of the mundane explanations the design has ruled out. If the paper does not say, it has not ruled out any.