Failure pattern · 8 documented incidents

Retry amplification and runaway resource consumption

Retries, loops or an expensive input multiply load until a partial failure becomes a full outage or a runaway bill.

See it happen

Watch how this failure unfolds, then switch to the same situation with the safeguard in place. Click any step to replay it.

Scroll the diagram sideways to see every system.

12 / 12 s

A brief slowdown has become a full outage.

What goes wrong:

  1. The database slows down for a moment.
  2. Every client times out and retries at the same moment.
  3. The retries time out too, and retry again, multiplying the load.
  4. A brief slowdown has become a full outage.

With the safeguard:

  1. The database slows down for a moment.
  2. Clients back off with random jitter and stop after a small retry budget.
  3. A circuit breaker sheds the excess before it reaches the database.
  4. The database recovers and traffic flows normally again.

What it is

Something slows down for a moment, and every client retries at once. The retries add load, which causes more timeouts, which cause more retries. A small hiccup turns into a full outage, or a bill nobody expected.

How it happens

  • Many clients share the same timeout and retry settings
  • Retries fire immediately and without a limit
  • Nothing upstream sheds load when a dependency struggles

What it causes

  • A brief slowdown becomes an outage across services
  • Autoscaling and cloud costs spike
  • Recovery takes far longer because traffic never lets up

Warning signs

  • Request volume climbs while successful responses fall
  • Retry counts rising together across many clients
  • Queues growing faster than they can drain

How to prevent it

Bound retries, loops and expensive work. Use capped exponential backoff with jitter, retry budgets and circuit breakers, and put hard limits on iterations, cost and CPU per request.

  • Clients retry with exponential backoff and random jitter, and give up after a small budget
  • Circuit breakers stop calls to a struggling dependency and fail fast
  • Expensive work has limits on cost, time and concurrency

Key ideas:Exponential backoff with jitterCircuit breaker

Check your system against the full launch checklist

Real incidents with this pattern

Documented incidents first. Each links to its public source.

Open in the database
Documented incidentIncident 2017-02-28

Mistyped input to a capacity-removal command takes down S3 in us-east-1 (AWS, 2017)

An engineer ran a command meant to remove a small number of servers from an S3 subsystem, but one input was entered incorrectly and more capacity was removed than intended.

Safeguard: Enforce minimum-capacity limits inside the tool

Privileged action without an enforceable approval gateaws.amazon.com
Documented incidentDate not recorded

Design reference: AWS guidance on timeouts, retries and backoff with jitter

AWS describes retries as powerful but dangerous: when a dependency is struggling, too many retries can amplify load and turn a partial failure into a retry storm.

Safeguard: Use capped exponential backoff with jitter

Retry amplification and runaway resource consumptionaws.amazon.com
222 security advisories and illustrative scenarios with this pattern
Security advisory2026-10-16

Software component issue could make the service unavailable

pupnp is an SDK for development of UPnP device and control point applications. Prior to 2.0.2, rawtoint in upnp/src/genlib/net/http/httpparser.c parses Content-Length with strtol and casts the resulting long to a 32-bit…

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-16

Software component issue could make the service unavailable

A flaw was found in libtirpc. An authenticated remote attacker could exploit this vulnerability by sending oversized wrapped data requests to an RPC (Remote Procedure Call) service configured with RPCSECGSS (Remote…

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-16

Security issue reported in lopdf

lopdf::Document::loadmem (and the other load entry points) parses nested PDF arrays and dictionaries with unbounded recursion.

Retry amplification and runaway resource consumptiongithub.com
Security advisory2026-10-15

Software component issue could make the service unavailable

A NULL pointer dereference vulnerability has been reported to affect Qsync Central. If a remote attacker gains a user account, they can then exploit the vulnerability to launch a denial-of-service (DoS) attack.

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-15

Software component issue could make the service unavailable

A NULL pointer dereference vulnerability has been reported to affect Qsync Central. If a remote attacker gains a user account, they can then exploit the vulnerability to launch a denial-of-service (DoS) attack.

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-15

Software component issue could make the service unavailable

In Eclipse Vert.x versions [4.0.0, 4.5.34] or [5.0.0,5.0.12] or [5.1.0,5.1.9] or [5.2.0, 5.2.0], the Vert.x MySQL client (vertx-mysql-client) pre-allocates an array for column definitions based on a server-declared…

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-15

Security issue reported in Software component

In Eclipse Vert.x versions [4.0.0, 4.5.34] or [5.0.0,5.0.12] or [5.1.0,5.1.9] or [5.2.0, 5.2.0], the PostgreSQL message decoder reads the 4-byte message length field from the PostgreSQL wire protocol without any maximum…

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-15

Software component issue could make the service unavailable

undici is an HTTP client library for Node.js. In versions 8.11.0 and 8.11.1, its experimental WebSocket client, which is enabled by default when HTTP/2 is allowed, mishandles a failed WebSocket handshake over HTTP/2: the…

Retry amplification and runaway resource consumptionnvd.nist.gov

In the standards

CollectSEA's mapping; not an official determination by the standards bodies.