Failure pattern · 9 documented incidents

Destructive operation without verified recovery

A delete, cleanup or teardown runs against the wrong target, and the backups that should save the day turn out not to work.

See it happen

Watch how this failure unfolds, then switch to the same situation with the safeguard in place. Click any step to replay it.

Scroll the diagram sideways to see every system.

12 / 12 s

The backups were never tested, and they are empty.

What goes wrong:

  1. A cleanup job is supposed to delete the old staging database.
  2. A wrong variable points it at production instead.
  3. The team reaches for the backups.
  4. The backups were never tested, and they are empty.

With the safeguard:

  1. A cleanup job is supposed to delete the old staging database.
  2. Production has deletion protection, and the target must be confirmed by name.
  3. Backups are restored automatically every week to prove they work.
  4. Even a real mistake is recoverable in minutes.

What it is

A delete, cleanup or teardown runs against the wrong thing: production instead of staging, every site instead of one. Then the backups that were supposed to make it survivable turn out to be missing, empty or impossible to restore.

How it happens

  • A script or command takes its target from a variable, a list or user input
  • Nothing double-checks a destructive action against a protected set
  • Backups exist on paper but haven’t been restored in a long time

What it causes

  • Customer data or whole environments deleted
  • Days of downtime while systems are rebuilt by hand
  • Permanent data loss when restores fail

Warning signs

  • Destructive commands that run without a dry run or confirmation
  • Backup jobs that “succeed” but produce tiny files
  • No one can say when a restore was last tested

How to prevent it

Make deletion reversible and prove recovery works. Use soft-delete with a grace period, confirm the exact target before destructive commands, and regularly restore from backups to prove they work within your recovery-time target.

  • Production resources have deletion protection that a script can’t switch off
  • Destructive commands show what they will touch and need the target confirmed by name
  • Backups are restored on a schedule to prove they work, and kept somewhere a single admin can’t delete

Key ideas:Restore testing

Check your system against the full launch checklist

Real incidents with this pattern

Documented incidents first. Each links to its public source.

Open in the database
7 security advisories and illustrative scenarios with this pattern
Security advisory2026-06-18

Security issue reported in Enterprise

Splunk Enterprise contains a missing authentication for critical function vulnerability which could allow an unauthenticated user to create or truncate arbitrary files through a PostgreSQL sidecar service endpoint.

Destructive operation without verified recoveryadvisory.splunk.com
Illustrative scenarioDate not recorded

DNSSEC Key Rollback & Global Domain Invalidation

Mismatched DS records during planned Key Signing Key (KSK) rollover caused recursive resolvers worldwide to return SERVFAIL.

Automation acts on stale state or authorization

In the standards

CollectSEA's mapping; not an official determination by the standards bodies.