Scourge · No culprit · January 15 1990

The Great Silence of AT&T

Chronicle

On 15 January 1990 a 4ESS switch in Manhattan reset itself and, by telling its neighbours so, knocked them over one after another. The culprit was a single misplaced C 'break' in an update to the recovery software rolled out in December 1989. For about nine hours nearly half of AT&T's long-distance calls failed to connect: over 50 million calls blocked, with some estimates as high as 70. No attacker: just one line of code.

Lesson learned

It showed that a system built to heal itself can also make itself sick, and that shipping the same update to every node turns one bug into an epidemic.

Prayer of reparation

O blessed stray Break, who leapt out of the wrong block, guide our hands through nested statements and let our tests cover the neighbour who reboots. Amen.

How it is remembered

Remembered by rolling out updates in stages, reading every switch statement with suspicion and keeping the previous version within reach.