Documented incidentIncident 2021-10-28RobloxEngineering postmortem

New Consul streaming feature under heavy load and a BoltDB issue cause a 73-hour outage (Roblox, 2021)

Enabling a relatively new streaming feature in Consul under unusually high read and write load caused excessive contention, compounded by a pathological performance issue in BoltDB when handling large writes.

What happened

  • A single Consul cluster supported multiple workloads and the monitoring systems relied on the same infrastructure, which hampered diagnosis.Roblox Return to Service
  • The outage lasted 73 hours (October 28-31, 2021); recovery included disabling streaming, restoring caches and reintroducing traffic gradually through DNS steering in 10% steps.Roblox Return to Service

The failure pattern

Global change without a staged rolloutA configuration, content or code change reaches every server at once, so one bad value takes everything down together.How to prevent it Retry amplification and runaway resource consumptionRetries, loops or an expensive input multiply load until a partial failure becomes a full outage or a runaway bill.How to prevent it
Watch the pattern: Global change without a staged rollout

Safeguards that address it

CollectSEA's engineering analysis of what would have prevented or contained this failure.

  • Roll out new infrastructure features gradually under realistic loadEnable new capabilities on a small portion of traffic and watch contention metrics before full rollout.
  • Separate critical workloads and their monitoringAvoid one shared cluster for many workloads, and keep observability independent of the system being diagnosed.

Related standards

CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.

Building something similar?

Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.

Analyze your design

Incidents with the same pattern