Documented incidentIncident 2025-06-12CloudflareEngineering postmortem

Third-party storage outage behind Workers KV takes down many Cloudflare services (2025)

Workers KV relied on a central data store, partly backed by a third-party cloud provider; when that provider had an outage, cold reads and writes to KV failed.

What happened

  • Because many Cloudflare products used KV for configuration, authentication and asset delivery, the outage spread to Access, Gateway, WARP, Workers AI, Turnstile and others for 2 hours and 28 minutes.Cloudflare service outage June 12, 2025
  • Cloudflare said it was responsible for its choice of dependencies and is removing the dependency on any single storage provider and limiting each product's blast radius.Cloudflare service outage June 12, 2025

The failure pattern

Retry amplification and runaway resource consumptionRetries, loops or an expensive input multiply load until a partial failure becomes a full outage or a runaway bill.How to prevent it
Watch the pattern: Retry amplification and runaway resource consumption

Safeguards that address it

CollectSEA's engineering analysis of what would have prevented or contained this failure.

  • Remove single dependencies on infrastructure you do not ownReplicate critical configuration and authentication data across independent providers, and test operating with one of them unavailable.
  • Let dependent products degrade instead of failCache last-known-good configuration locally so services keep working when the shared store is unreachable.

Related standards

CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.

Building something similar?

Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.

Analyze your design

Incidents with the same pattern

Security advisory2026-10-16

Software component issue could make the service unavailable

pupnp is an SDK for development of UPnP device and control point applications. Prior to 2.0.2, rawtoint in upnp/src/genlib/net/http/httpparser.c parses Content-Length with strtol and casts the resulting long to a 32-bit…

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-16

Software component issue could make the service unavailable

A flaw was found in libtirpc. An authenticated remote attacker could exploit this vulnerability by sending oversized wrapped data requests to an RPC (Remote Procedure Call) service configured with RPCSECGSS (Remote…

Retry amplification and runaway resource consumptionnvd.nist.gov