Failure pattern · 10 documented incidents

Global change without a staged rollout

A configuration, content or code change reaches every server at once, so one bad value takes everything down together.

See it happen

Watch how this failure unfolds, then switch to the same situation with the safeguard in place. Click any step to replay it.

Scroll the diagram sideways to see every system.

12 / 12 s

One mistake, one global outage.

What goes wrong:

  1. A configuration file with one bad value is published.
  2. Within seconds it reaches every server in the world.
  3. Every server fails on the same input at the same time.
  4. One mistake, one global outage.

With the safeguard:

  1. A configuration file with one bad value is published.
  2. It goes to a small canary group first, not everywhere.
  3. Health checks catch the errors and roll back automatically.
  4. The rest of the world never sees the bad value.

What it is

A configuration, content or code change goes out to every server at the same moment. If it carries one bad value, everything fails together, and the tools you’d use to roll back may be down too.

How it happens

  • A change bypasses the staged rollout that code goes through
  • Every region or server consumes the same file or setting
  • There’s no automatic check on health before the change spreads

What it causes

  • Global outages from a single change
  • Recovery slowed because management tools depend on the broken system
  • Every customer affected at once

Warning signs

  • Errors starting in every region within seconds of a change
  • Config or content updates that skip canary stages
  • No one can roll back faster than the change spread

How to prevent it

Stage every change and keep a fast rollback. Validate configuration and content like code, release through canary rings while watching health signals, and keep a tested last-known-good rollback that works even when the control plane is degraded.

  • Every change, including config and content, rolls out in stages starting with a small canary
  • Health checks stop the rollout and roll back automatically
  • New configuration is validated, size and shape included, before anything loads it

Key ideas:Canary releaseKill switchBlast radius

Check your system against the full launch checklist

Real incidents with this pattern

Documented incidents first. Each links to its public source.

Open in the database

In the standards

CollectSEA's mapping; not an official determination by the standards bodies.