Scourge · No culprit · July 19 2024

The Friday of the Blue Screens

Chronicle

On 19 July 2024 at 04:09 UTC CrowdStrike pushed a defective content update for its Falcon sensor, Channel File 291. About 8.5 million Windows computers crashed and were stuck in a blue-screen boot loop. Airlines, banks, broadcasters, hospitals and some 911 call centres were disrupted worldwide, and many machines had to be fixed by hand, one at a time. There was no attack: the cause was a logic error that the content validator had missed.

Lesson learned

It is often called the largest IT outage in history. It taught that security updates too must be rolled out gradually, and reopened the debate about code that runs at the heart of the operating system.

Prayer of reparation

For those who were stranded at airports that day or waited for care or help, and for the technicians who rebooted millions of machines by hand, we ask patience, relief and rest. Grant us the humility to test every update as if it were the first. Amen.

How it is remembered

Remembered by rolling out even content updates in successive rings, keeping recovery keys within reach and rehearsing mass recovery of workstations.