The backup that was never restored

An organisation ran nightly backups for four years without a failure and discovered, on the day it needed them, that none of the files could be opened.

The backup job had been succeeding. It copied the database files while the database was running, which produces a file that exists, has the right size, and contains a half-written internal state. Every indicator the team monitored was green, because every indicator measured whether the copy had been made and none measured whether the copy was usable.

This is the standard failure in resilience work and it has a standard name: an untested backup is not a backup. The property that matters is not the existence of a copy but the ability to reach a working system from it, and that property can only be established by doing it. Organisations that take this seriously run a restore on a schedule, into a separate environment, and measure two numbers: whether it worked and how long it took. The second number is usually the surprise. A restore that technically succeeds after eleven hours is a different plan from the one people had in mind when they approved it.

The same logic applies well beyond data. A fire drill tests an evacuation plan; a plan that has never been walked is a document. A contingency for a missing teacher is a theory until someone is absent. In each case what is being tested is not the existence of the arrangement but the parts of it that nobody wrote down, which are the parts that fail.

The organisation's revised procedure was one line long: on the first Monday of each month, restore last night's backup into the test environment and record the time it took.