Latent race condition in DNS automation leaves DynamoDB's regional endpoint with an empty record (AWS, 2025)
A latent race condition in DynamoDB's DNS management system produced an empty DNS record for the regional endpoint that the automation could not repair, so clients could not resolve the endpoint.
What happened
- The disruption on October 19-20, 2025 spread to dependent services including EC2, Network Load Balancer, Lambda, container services, Security Token Service and IAM authentication.Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region
- AWS disabled the DNS automation worldwide until the race condition is fixed, added velocity controls to NLB health checks and improved EC2 recovery testing.Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region
The failure pattern
Watch the pattern: Automation acts on stale state or authorization
Safeguards that address it
CollectSEA's engineering analysis of what would have prevented or contained this failure.
- Make control-plane automation safe against stale plansVersion and verify plans so an older plan cannot overwrite a newer one, and give operators a safe manual path to repair an empty record.
- Limit how fast health failures remove capacityAdd velocity controls to dependent health checks so one dependency fault cannot cascade through the region.
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
Database permissions change doubles a bot-management feature file and crashes edge proxies (Cloudflare, 2025)
A ClickHouse access-control change made table permissions explicit, so a query without a database filter returned duplicate rows and doubled the size of the Bot Management feature file.
Dormant configuration error in legacy routing systems withdraws Cloudflare's 1.1.1.1 DNS resolver worldwide (2025)
A configuration change on June 6, 2025 for a not-yet-used service accidentally included the 1.1.1.1 resolver's IP prefixes in that service's topology; the error sat dormant in production.
Third-party storage outage behind Workers KV takes down many Cloudflare services (2025)
Workers KV relied on a central data store, partly backed by a third-party cloud provider; when that provider had an outage, cold reads and writes to KV failed.
Unplanned automatic OS update cuts networking for Heroku apps and its own status tools (2025)
An automated operating-system update ran on production hosts when it should have been disabled, and restarted their networking service.
Blank provisioning parameter leads to automatic deletion of a customer's private cloud (UniSuper / Google Cloud, 2024)
An input parameter was left blank in an internal provisioning tool, so the system assigned an unknown default fixed one-year term to the customer's VMware Private Cloud.
Deletion script given site IDs instead of app IDs removes 775 customers' sites (Atlassian, 2022)
A script meant to retire a deprecated app was given IDs for entire customer sites and did not check whether an ID referred to the app or the whole site.