Glossary
Circuit breaker
A guard around calls to a dependency that stops sending requests when the dependency is failing, then tries again carefully once it recovers.
In plain terms
Named after the electrical kind. When calls to a service keep failing or timing out, the breaker “opens” and fails new calls straight away instead of letting them queue up and time out too.
After a cool-down it lets a few test requests through. If they succeed it closes again, and if not it stays open. The struggling service gets room to recover, and your own system stays responsive.
See what happens without it
The failure pattern this protects against, retry amplification and runaway resource consumption, played out step by step.
Scroll the diagram sideways to see every system.
A brief slowdown has become a full outage.
What goes wrong:
- The database slows down for a moment.
- Every client times out and retries at the same moment.
- The retries time out too, and retry again, multiplying the load.
- A brief slowdown has become a full outage.
With the safeguard:
- The database slows down for a moment.
- Clients back off with random jitter and stop after a small retry budget.
- A circuit breaker sheds the excess before it reaches the database.
- The database recovers and traffic flows normally again.
When it mattered
A surge of traffic after the holidays overloaded part of Slack’s network and stalled the systems meant to add capacity, the kind of feedback loop that shedding load early is meant to break.