Scourge · No culprit · June 8 2021

The Great Eclipse of Fastly

Chronicle

On 8 June 2021 at 09:47 UTC one customer's perfectly valid configuration change woke a latent bug introduced in a May deployment, and 85% of Fastly's network began returning errors. The New York Times, the Guardian, Reddit, Twitch, gov.uk and many other sites vanished for up to an hour. Fastly found the cause within minutes and restored service in under an hour. No attacker: just a bug waiting for the right customer.

Lesson learned

It was a reminder of how much of the web runs through a few CDN providers, and of the value of a fast diagnosis and an honest public report.

Prayer of reparation

O sleeping bug, who waited three weeks for the right configuration to wake you, stay asleep in our systems and let our tests find you before our customers do. Amen.

How it is remembered

Remembered by rolling out changes gradually, having a fallback plan if the CDN goes down, and writing a post-mortem as frank as Fastly's.