Mistyped input to a capacity-removal command takes down S3 in us-east-1 (AWS, 2017)
An engineer ran a command meant to remove a small number of servers from an S3 subsystem, but one input was entered incorrectly and more capacity was removed than intended.
What happened
- The disruption affected S3 requests (GET, LIST, PUT, DELETE) and dependent services including EC2 launches, EBS snapshots, Lambda and the S3 console in US-EAST-1.Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region
- AWS changed the tool to remove capacity more slowly and added safeguards that block removals which would take a subsystem below its minimum required capacity.Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region
The failure pattern
Watch the pattern: Privileged action without an enforceable approval gate
Safeguards that address it
CollectSEA's engineering analysis of what would have prevented or contained this failure.
- Enforce minimum-capacity limits inside the toolRefuse removals that would drop a subsystem below its minimum capacity and remove capacity gradually.
- Validate operator input and require a dry runCheck parameters against expected ranges and show what a destructive capacity change will affect before executing it.
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
- ASI03 Identity and Privilege Abuse
- ASI08 Cascading Failures
- LLM03 Excessive Agency
- LLM06 Unbounded Consumption
- AC-3 Access Enforcement
- AC-5 Separation of Duties
- AC-6 Least Privilege
- SC-5 Denial-of-Service Protection
- RV.3.4 Update the development process
- MAP 5.1 Estimate impacts from past uses and public incident reports
- CWE-862 Missing Authorization
- CWE-770 Allocation of Resources Without Limits or Throttling
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
GitHub Actions injection leaks Nx's npm token; malicious packages use local AI coding tools to hunt for secrets (2025)
A pull-request title validation workflow used pullrequesttarget, which runs with repository secrets, and echoed the unsanitized PR title, creating an injection point; the workflow's token also had read/write permissions.
GitHub's RSA SSH host private key briefly exposed in a public repository (2023)
GitHub.com's RSA SSH private key was briefly exposed in a public GitHub repository.
Zillow winds down its algorithmic home-buying business after price forecasts miss (2021)
Machine learning pricing models drifted during market volatility; automated purchasing bought homes above value resulting in $540M inventory write-down.
Retail broker failed best-execution reviews while taking high payment for order flow (Robinhood, SEC 2020)
From July 2016 to June 2019 the broker received high payment for order flow but did not conduct adequate, regular and rigorous reviews of execution quality on customer orders.
SUNBURST: a backdoor inserted into SolarWinds Orion builds reaches customers (2020)
Compromised build server injected SUNBURST malware into compiler memory during build process; distributed to 18,000 customers.
Confusing loan-system interface leads Citibank to wire nearly $900 million in principal to Revlon lenders (2020)
Unintuitive Flexcube interface required checking obscure override box; bank accidentally wired $894M of proprietary funds to Revlon lenders.