Design reference: AWS guidance on timeouts, retries and backoff with jitter
AWS describes retries as powerful but dangerous: when a dependency is struggling, too many retries can amplify load and turn a partial failure into a retry storm.
What happened
- The article recommends timeouts, capped exponential backoff and jitter to keep retries from synchronising.Timeouts, retries, and backoff with jitter
The failure pattern
Watch the pattern: Retry amplification and runaway resource consumption
Safeguards that address it
CollectSEA's engineering analysis of what would have prevented or contained this failure.
- Use capped exponential backoff with jitterSpread retries over time and cap attempts so clients do not synchronise into a storm.
- Retry at one layer with a budgetAvoid multiplying retries across layers; bound retries with a budget relative to normal traffic.
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
Third-party storage outage behind Workers KV takes down many Cloudflare services (2025)
Workers KV relied on a central data store, partly backed by a third-party cloud provider; when that provider had an outage, cold reads and writes to KV failed.
Holiday traffic surge overloads a Transit Gateway and stalls autoscaling provisioning (Slack, 2021)
After the holidays, an unusual surge of traffic from clients with cold caches overloaded one of Slack's AWS Transit Gateways.
Software component issue could make the service unavailable
A public security advisory describes this issue. Review the linked source for the technical details and remediation guidance.
Software component issue could make the service unavailable
A public security advisory describes this issue. Review the linked source for the technical details and remediation guidance.
Software component issue could make the service unavailable
pupnp is an SDK for development of UPnP device and control point applications. Prior to 2.0.2, rawtoint in upnp/src/genlib/net/http/httpparser.c parses Content-Length with strtol and casts the resulting long to a 32-bit…
Software component issue could make the service unavailable
A flaw was found in libtirpc. An authenticated remote attacker could exploit this vulnerability by sending oversized wrapped data requests to an RPC (Remote Procedure Call) service configured with RPCSECGSS (Remote…