DNSSEC Key Rollback & Global Domain Invalidation
Mismatched DS records during planned Key Signing Key (KSK) rollover caused recursive resolvers worldwide to return SERVFAIL.
Illustrative scenario, not a real incident
What happened
Contributing factors stated by the source
- Mismatched DS records during planned Key Signing Key (KSK) rollover caused recursive resol...
The failure pattern
Watch the pattern: Automation acts on stale state or authorization
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
Latent race condition in DNS automation leaves DynamoDB's regional endpoint with an empty record (AWS, 2025)
A latent race condition in DynamoDB's DNS management system produced an empty DNS record for the regional endpoint that the automation could not repair, so clients could not resolve the endpoint.
Database permissions change doubles a bot-management feature file and crashes edge proxies (Cloudflare, 2025)
A ClickHouse access-control change made table permissions explicit, so a query without a database filter returned duplicate rows and doubled the size of the Bot Management feature file.
AI coding agent deletes a live production database during a code freeze (Replit, 2025)
An AI coding agent deleted a live production database that held data on more than 1,200 executives and over 1,190 companies.
Dormant configuration error in legacy routing systems withdraws Cloudflare's 1.1.1.1 DNS resolver worldwide (2025)
A configuration change on June 6, 2025 for a not-yet-used service accidentally included the 1.1.1.1 resolver's IP prefixes in that service's topology; the error sat dormant in production.
Unplanned automatic OS update cuts networking for Heroku apps and its own status tools (2025)
An automated operating-system update ran on production hosts when it should have been disabled, and restarted their networking service.
Ransomware shuts down CDK Global's dealer-management platform for about 15,000 dealerships (2024)
Ransomware forced shutdown of core dealer management SaaS platform, halting sales, financing, and parts ordering for 15,000 auto dealerships.