Measuring what matters

A number on a screen feels like a fact. It is not: it is the end of a long chain of decisions about what to measure, how to measure it and what to leave out. Learning to see that chain is probably the most transferable skill in the whole project.

Two words from measurement theory are worth knowing. A measure is reliable if it gives the same answer for the same thing — the same squat, counted twice, should produce the same number. It is valid if it measures what you actually care about. The two are independent, and reliability is the easier one to obtain. A counter that rewards only very deep squats is extremely reliable and not very valid, because what the school cares about is whether students move regularly, not whether they can reach a particular angle. Most arguments about scores are arguments about validity in disguise.

FitForm measures repetitions because repetitions can be counted by a camera. What the project is actually for is participation: how many students in a class train on a normal Tuesday, and how many of them are still training three months later. Those numbers are harder to make exciting on a poster, so there is a constant temptation to report the big, easy figure — total repetitions across the school — and let the audience assume it means something. By itself it does not: one determined student can produce thousands of repetitions while participation falls.

This is where Goodhart's law arrives. When a measure becomes a target, it stops being a good measure, because people optimise the number rather than the thing it stands for. The symptoms are familiar from every school: teach to the test, train for the metric. In FitForm the honest responses are structural rather than moral. Points are capped each day, so grinding is pointless. Classes are compared by the share of students who take part, not by totals, so persuading a quiet classmate is worth more than one more set. Several different numbers are shown together, because a single headline figure is always easier to game than a dashboard.

Then there is the question the team argued about for a week. The public board shows ten virtual training partners — small characters with their own daily routines — so that a student who opens the site at seven in the morning does not see an empty hall. Should they look like real students? It would be easy, and the board would look busier. The answer was no, and the reason is worth stating precisely: the same page claims that the statistics are real and invites teachers to check them. A number that is partly invented cannot be checked, and a measure that cannot be checked is worthless the moment anyone tries. The virtual partners therefore carry a robot icon, belong to a class called "Community", and are excluded from every statistic the project reports. The board is slightly emptier at seven in the morning and completely defensible at a competition.

Three habits follow from this, and none of them is specific to sport. Say what the number counts, in one sentence, next to the number. Separate what was measured from what it suggests — "sessions rose by forty per cent" is a measurement, while "students are fitter" is a claim that needs different evidence. And report the number that could embarrass you: the share of students who tried the platform once and never came back is the single most useful figure the team collects, precisely because nobody would volunteer it.