STATUS IS UPDid you know?
  1. Status Is UP
  2. Did you know?
  3. No. 20

Did you know? · No. 20 · Tip ·

One missed server sent millions of orders in 45 minutes

A Status Is UP card reads “45 min” with the line “Partial rollout, millions of orders.” Siu, the green Status Is UP mascot, stands in the corner.

On August 1, 2012, Knight Capital’s automated order router processed 212 customer orders. Seven of its eight servers had received code for the NYSE Retail Liquidity Program. The eighth had not: its older Power Peg function was still present, and a repurposed flag activated it.

In about 45 minutes, the router sent millions of child orders, produced more than 4 million executions, and left Knight with unwanted positions. The SEC order says Knight ultimately realized a $460 million loss. Your website may not trade stocks, but a partial release can still create inconsistent behavior that spreads quickly. Treat deployment completeness and a fast, practiced way to stop a bad release as production requirements.

Try this on your site

This week, make your deploy pipeline prove that every target received the same release, and block completion if one is missing. Rehearse who can disable or roll back a failing release, then document the steps.

Share this fact

Each fact is checked against the primary sources listed. Spotted an error? Email the Status Is UP team. Status Is UP helps owners improve their sites; it is not an uptime monitor. For live outage status, see Status Is Down.