The homework that was marked twice

A department had every piece of work marked by two teachers for one term. The second marks were not reported to students; they were used to measure the first.

The results were uncomfortable and unsurprising. On questions with a right answer, the two markers agreed almost perfectly. On extended writing, agreement was moderate: about three quarters of scripts received marks within one band, which also means a quarter did not. The disagreements were not random. One marker consistently rewarded structure, the other rewarded range of vocabulary, and each believed they were applying the same published criteria.

This is the ordinary condition of marking rather than a failure of these teachers. Where a judgement is required, two competent people differ, and the published criteria are not detailed enough to remove the difference because criteria detailed enough to remove it would be unusable. The practical response is called moderation: markers score the same few scripts, compare, and discuss the gaps until their interpretations converge. Done before the main marking, it reduces disagreement substantially. Done afterwards, it only measures it.

The department made two changes. Moderation moved to the start of each term, using three anonymised scripts from the previous year. And the criteria were annotated with examples rather than rewritten: beside 'develops ideas' sat two short paragraphs, one that did and one that did not.

Students noticed the second change most. Several reported that they had understood the phrase 'develop your ideas' for the first time, not because the words had changed but because they could finally see the difference the words were pointing at.