Scourge · No culprit · October 4 2021

The Day Facebook Vanished

Chronicle

On 4 October 2021, at about 15:40 UTC, a maintenance command meant to assess backbone capacity accidentally disconnected all of Facebook's data centres, and an audit tool, because of a bug, failed to stop it. Facebook's DNS servers, no longer able to see the data centres, withdrew their BGP announcements: Facebook, Instagram and WhatsApp disappeared from the Internet for about six hours. Engineers had to reach the routers physically, slowed by the very security measures designed to protect them. No attacker.

Lesson learned

It showed the danger of making the tools that fix a network depend on that same network, and how many billions of people rely on a single group of services to communicate.

Prayer of reparation

O BGP, protocol of mutual trust, do not withdraw our routes without good reason, and let the keys to the server room never be kept inside the server room. Amen.

How it is remembered

Remembered by keeping out-of-band access channels, testing the tools that audit commands and rehearsing what happens when your own network disappears.