Scourge · No culprit · August 1 2012

The Downfall of Knight Capital

Chronicle

On 1 August 2012, at the New York Stock Exchange open, Knight Capital switched on new trading software. On one of its eight servers the new version had not been installed, and a repurposed old flag woke dormant code called 'Power Peg'. In about 45 minutes millions of erroneous orders went out and the firm lost about $440 million (the SEC later put it at more than $460 million), bringing it to the brink of collapse. No attacker: just a botched deployment.

Lesson learned

The SEC fined Knight for lacking adequate controls, and the episode became the favourite story of everyone who argues for automated, verified, reversible deployments and for deleting dead code.

Prayer of reparation

O Power Peg, code buried but never deleted, rest in peace in the git history and never again awaken to a recycled flag. Amen.

How it is remembered

Remembered by automating deployments, checking that every server runs the same version, deleting dead code and keeping a kill switch within reach.