43-second network partition triggers database failover and unreplicated writes (GitHub, 2018)
Network maintenance caused a 43-second loss of connectivity between GitHub's US East Coast hub and its primary data center, which led to 24 hours and 11 minutes of degraded service.
What happened
- Orchestrator failed MySQL primaries over to the West Coast while the East Coast still held writes that had not replicated, leaving data inconsistent and preventing a safe failback.October 21 post-incident analysis
- Recovery involved restoring from backups, resynchronising replicas and manually reconciling about 954 unreplicated writes per affected cluster.October 21 post-incident analysis
The failure pattern
Watch the pattern: Partial success leaves systems out of sync
Safeguards that address it
CollectSEA's engineering analysis of what would have prevented or contained this failure.
- Make automatic failover aware of replication stateDo not promote a replica in another region unless it has the writes the old primary acknowledged, or fence the old primary first.
- Plan for split-brain reconciliationKeep tooling and runbooks for detecting and reconciling divergent writes after a partition.
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
Database permissions change doubles a bot-management feature file and crashes edge proxies (Cloudflare, 2025)
A ClickHouse access-control change made table permissions explicit, so a query without a database filter returned duplicate rows and doubled the size of the Bot Management feature file.
Latent race condition in DNS automation leaves DynamoDB's regional endpoint with an empty record (AWS, 2025)
A latent race condition in DynamoDB's DNS management system produced an empty DNS record for the regional endpoint that the automation could not repair, so clients could not resolve the endpoint.
Dormant configuration error in legacy routing systems withdraws Cloudflare's 1.1.1.1 DNS resolver worldwide (2025)
A configuration change on June 6, 2025 for a not-yet-used service accidentally included the 1.1.1.1 resolver's IP prefixes in that service's topology; the error sat dormant in production.
Unplanned automatic OS update cuts networking for Heroku apps and its own status tools (2025)
An automated operating-system update ran on production hosts when it should have been disabled, and restarted their networking service.
Blank provisioning parameter leads to automatic deletion of a customer's private cloud (UniSuper / Google Cloud, 2024)
An input parameter was left blank in an internal provisioning tool, so the system assigned an unknown default fixed one-year term to the customer's VMware Private Cloud.
Deletion script given site IDs instead of app IDs removes 775 customers' sites (Atlassian, 2022)
A script meant to retire a deprecated app was given IDs for entire customer sites and did not check whether an ID referred to the app or the whole site.