Engineer deletes the production database directory and every backup method fails (GitLab, 2017)
While working on replication, an engineer ran a data-directory deletion on the production database server instead of the secondary, removing about 300 GB before stopping it.
What happened
- The backup mechanisms failed: pg_dump backups had failed silently (version mismatch), Azure disk snapshots were not enabled, the LVM snapshot was six hours old, and the secondary's data had been wiped.Postmortem of database outage of January 31
- About 5,000 projects, 5,000 comments and 700 user accounts created in a roughly 6.5-hour window were lost.Postmortem of database outage of January 31
The failure pattern
Watch the pattern: Destructive operation without verified recovery
Safeguards that address it
CollectSEA's engineering analysis of what would have prevented or contained this failure.
- Verify backups by restoring themRun scheduled restore tests and alert on empty or failing backup jobs; do not rely on job success alone.
- Make dangerous commands hard to misdirectDistinguish production from replicas in prompts and require confirmation for destructive operations on primaries.
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
Blank provisioning parameter leads to automatic deletion of a customer's private cloud (UniSuper / Google Cloud, 2024)
An input parameter was left blank in an internal provisioning tool, so the system assigned an unknown default fixed one-year term to the customer's VMware Private Cloud.
Deletion script given site IDs instead of app IDs removes 775 customers' sites (Atlassian, 2022)
A script meant to retire a deprecated app was given IDs for entire customer sites and did not check whether an ID referred to the app or the whole site.
iLearningEngines, Inc. discloses a cybersecurity incident (SEC 8-K, November 2024)
iLearningEngines, Inc. (the “ Company ”) recently became aware of a cybersecurity incident.
HTTP Server and Oracle Weblogic Server Proxy Plug-in issue could allow unauthorized access
Oracle HTTP Server and Oracle Weblogic Server Proxy Plug-in contain an improper access control vulnerability that can result in unauthorized creation, deletion or modification access to critical data as well as…
Security issue reported in Enterprise
Splunk Enterprise contains a missing authentication for critical function vulnerability which could allow an unauthenticated user to create or truncate arbitrary files through a PostgreSQL sidecar service endpoint.
AI coding agent deletes a live production database during a code freeze (Replit, 2025)
An AI coding agent deleted a live production database that held data on more than 1,200 executives and over 1,190 companies.