Glossary

Canary release

Rolling a change out to a small slice of servers or users first, watching for problems, and only then expanding it.

Also called Staged rollout, Progressive delivery.

In plain terms

The name comes from the canary in a coal mine. A small group gets the change first, and if errors climb you roll back before most people ever see it.

It only protects you if everything goes through it: code, configuration, feature flags and content updates alike. Some of the largest outages came from changes that skipped the stages because they weren’t considered “code”.

See what happens without it

The failure pattern this protects against, global change without a staged rollout, played out step by step.

Scroll the diagram sideways to see every system.

12 / 12 s

One mistake, one global outage.

What goes wrong:

  1. A configuration file with one bad value is published.
  2. Within seconds it reaches every server in the world.
  3. Every server fails on the same input at the same time.
  4. One mistake, one global outage.

With the safeguard:

  1. A configuration file with one bad value is published.
  2. It goes to a small canary group first, not everywhere.
  3. Health checks catch the errors and roll back automatically.
  4. The rest of the world never sees the bad value.

When it mattered

Documented incidentIncident 2024-07-19CrowdStrike

In July 2024 a faulty content update for CrowdStrike Falcon reached Windows machines worldwide at once and crashed them, grounding flights and disrupting hospitals.

Read the full record Source: crowdstrike.com