Backbone maintenance command disconnects Facebook's data centers and locks out recovery tooling (Meta, 2021)
A command issued to assess global backbone capacity unintentionally took down all the connections in Facebook's backbone network.
What happened
- A bug in the audit tool designed to prevent such commands stopped it from properly blocking this one.More details about the October 4 outage
- Recovery was slow because engineers could not use normal access paths and DNS also failed, which required onsite intervention at heavily secured data centers.More details about the October 4 outage
The failure pattern
Watch the pattern: Global change without a staged rollout
Safeguards that address it
CollectSEA's engineering analysis of what would have prevented or contained this failure.
- Test the guard that is meant to stop dangerous commandsAudit tools need their own tests and a second independent check for changes that can disconnect the network.
- Keep recovery paths independent of the thing being recoveredEnsure out-of-band access and DNS-independent tooling exist so a network failure does not lock operators out.
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
- ASI03 Identity and Privilege Abuse
- ASI08 Cascading Failures
- LLM03 Excessive Agency
- AC-3 Access Enforcement
- AC-5 Separation of Duties
- AC-6 Least Privilege
- CM-3 Configuration Change Control
- CM-4 Impact Analyses
- RV.3.4 Update the development process
- MAP 5.1 Estimate impacts from past uses and public incident reports
- CWE-862 Missing Authorization
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
Database permissions change doubles a bot-management feature file and crashes edge proxies (Cloudflare, 2025)
A ClickHouse access-control change made table permissions explicit, so a query without a database filter returned duplicate rows and doubled the size of the Bot Management feature file.
Dormant configuration error in legacy routing systems withdraws Cloudflare's 1.1.1.1 DNS resolver worldwide (2025)
A configuration change on June 6, 2025 for a not-yet-used service accidentally included the 1.1.1.1 resolver's IP prefixes in that service's topology; the error sat dormant in production.
Policy change with blank fields crashes Google Cloud's Service Control in a global loop (2025)
A policy change containing blank fields was inserted into the quota policy system; when Service Control read it, the code path hit a null pointer and the binaries went into a crash loop.
Unplanned automatic OS update cuts networking for Heroku apps and its own status tools (2025)
An automated operating-system update ran on production hosts when it should have been disabled, and restarted their networking service.
Faulty Channel File 291 content update crashes Windows hosts running CrowdStrike Falcon (2024)
Rapid deployment channel 291 bypassed staged canary rings; out-of-bounds memory read triggered BSOD on 8.5M Windows hosts.
New Consul streaming feature under heavy load and a BoltDB issue cause a 73-hour outage (Roblox, 2021)
Enabling a relatively new streaming feature in Consul under unusually high read and write load caused excessive contention, compounded by a pathological performance issue in BoltDB when handling large writes.