Zillow winds down its algorithmic home-buying business after price forecasts miss (2021)
Machine learning pricing models drifted during market volatility; automated purchasing bought homes above value resulting in $540M inventory write-down.
The failure pattern
Watch the pattern: Privileged action without an enforceable approval gate
Safeguards that address it
CollectSEA's engineering analysis of what would have prevented or contained this failure.
- Cap automated exposure in deterministic limitsSet hard caps on automated purchasing volume and concentration, with human approval above thresholds.
- Monitor model drift and pause automaticallyCompare model valuations with realised outcomes and halt automated action when error exceeds a tolerance.
Related standards
CollectSEA's mapping through this incident's failure pattern; not an official determination by the standards bodies.
- ASI02 Tool Misuse and Exploitation
- ASI03 Identity and Privilege Abuse
- ASI10 Rogue Agents
- LLM03 Excessive Agency
- LLM10 Improper Output Handling
- AC-3 Access Enforcement
- AC-5 Separation of Duties
- AC-6 Least Privilege
- SI-10 Information Input Validation
- RV.3.4 Update the development process
- MAP 5.1 Estimate impacts from past uses and public incident reports
- CWE-862 Missing Authorization
- CWE-250 Execution with Unnecessary Privileges
Building something similar?
Describe your design and see every precedent that applies, with safeguards you can hand to your coding agent.
Incidents with the same pattern
GitHub Actions injection leaks Nx's npm token; malicious packages use local AI coding tools to hunt for secrets (2025)
A pull-request title validation workflow used pullrequesttarget, which runs with repository secrets, and echoed the unsanitized PR title, creating an injection point; the workflow's token also had read/write permissions.
GitHub's RSA SSH host private key briefly exposed in a public repository (2023)
GitHub.com's RSA SSH private key was briefly exposed in a public GitHub repository.
Retail broker failed best-execution reviews while taking high payment for order flow (Robinhood, SEC 2020)
From July 2016 to June 2019 the broker received high payment for order flow but did not conduct adequate, regular and rigorous reviews of execution quality on customer orders.
SUNBURST: a backdoor inserted into SolarWinds Orion builds reaches customers (2020)
Compromised build server injected SUNBURST malware into compiler memory during build process; distributed to 18,000 customers.
Confusing loan-system interface leads Citibank to wire nearly $900 million in principal to Revlon lenders (2020)
Unintuitive Flexcube interface required checking obscure override box; bank accidentally wired $894M of proprietary funds to Revlon lenders.
Phone spear-phishing gives attackers Twitter's internal support tools; 130 accounts targeted (2020)
A phone spear-phishing attack on a small number of employees gave attackers credentials to internal support tools on July 15, 2020.