Documented incidentIncident 2021-01-04SlackEngineering postmortem

Holiday traffic surge overloads a Transit Gateway and stalls autoscaling provisioning (Slack, 2021)

After the holidays, an unusual surge of traffic from clients with cold caches overloaded one of Slack's AWS Transit Gateways.

What happened

  • The provisioning service hit resource limits (an open-files limit and an AWS quota) while trying to add about 1,200 servers under degraded network conditions.Slack's Outage on January 4th 2021
  • Follow-ups included removing the monitoring dashboards' dependency on the Transit Gateway, regularly load-testing the provisioning service and reviewing autoscaling and health-check settings.Slack's Outage on January 4th 2021

The failure pattern

Retry amplification and runaway resource consumptionRetries, loops or an expensive input multiply load until a partial failure becomes a full outage or a runaway bill.How to prevent it
Watch the pattern: Retry amplification and runaway resource consumption

Safeguards that address it

CollectSEA's engineering analysis of what would have prevented or contained this failure.

  • Load-test scale-up paths, not just steady stateRegularly exercise the provisioning service at burst scale and check OS and cloud quota limits.
  • Keep monitoring independent of the failing pathPlace dashboards and alerting where a network-gateway failure cannot blind operators.

Related standards

CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.

Building something similar?

Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.

Analyze your design

Incidents with the same pattern

Security advisory2026-10-16

Software component issue could make the service unavailable

pupnp is an SDK for development of UPnP device and control point applications. Prior to 2.0.2, rawtoint in upnp/src/genlib/net/http/httpparser.c parses Content-Length with strtol and casts the resulting long to a 32-bit…

Retry amplification and runaway resource consumptionnvd.nist.gov
Security advisory2026-10-16

Software component issue could make the service unavailable

A flaw was found in libtirpc. An authenticated remote attacker could exploit this vulnerability by sending oversized wrapped data requests to an RPC (Remote Procedure Call) service configured with RPCSECGSS (Remote…

Retry amplification and runaway resource consumptionnvd.nist.gov