The CollectSEA incident dataset

All records were kept in a single stored list that ingestion rewrote every two hours; it was approaching the size and processing limits at which ingestion would have stopped. Every record ever ingested is now kept permanently in a database, together with the original feed item it came from, so nothing is discarded and records can be re-processed when extraction improves. Matching uses a working set of all documented incidents and scenarios plus the newest 1,500 advisories; older advisories remain available on their record pages and through the API.

Visitors and page views were counted from every page request, so automated clients that do not identify themselves were counted as people. From October 10, 2026 a visit is counted when the page’s own script runs, which most bots never do, so visitor numbers from this date are lower and not directly comparable with earlier days. The usage page also shows how many visitors start describing a system, run an example and analyze it.

The incident list embedded all 1,500+ records in every page load (about 1 MB); it now embeds documented incidents and loads advisories and scenarios when you choose those tabs. The methodology benchmark and the usage report were recomputed by page requests and could fail with HTTP 503; they are now computed by the scheduled job and shared. Page headings and the analyzer are readable from the first frame, the header no longer overflows at tablet widths, and failure-pattern examples wrap on phones.

Since October 4 most uncached requests, including every analysis, intermittently failed with HTTP 503 (Cloudflare error 1102) because each request exceeded the Worker CPU allowance. Analysis now prepares each record’s text once instead of on every query (about eight times less CPU per analysis with identical results), records are no longer normalized repeatedly while the corpus loads, and the corpus is rebuilt only when ingestion or a review decision changes it rather than every 30 seconds. In repeated staging tests, successful requests rose from 15 of 39 to 64 of 64.

Some requests also hung for 30–40 seconds: they waited for work that another request had started and that Cloudflare cancelled when that request ended. Shared work is now reused only once it has finished, and a request waits at most eight seconds for unfinished work before doing it itself.

NVD API rate limits caused incomplete refreshes. Ingestion now reads NVD’s official compressed recent feed, checks its publication freshness, and selects recently updated records. CISA entries are selected newest first. OSV adds targeted coverage for MCP, LangChain, LlamaIndex and n8n. Previously collected advisories survive moving feed windows, failures retain each source’s own last-success timestamp, and incomplete refreshes or storage failures are reported explicitly.

Extra browser testing found that unfinished app reviews disappeared on refresh, behavior choices became duplicated notes after returning to the analyzer, and share links lost file names and repository provenance. Drafts now retain editable choices in the browser tab, existing-app share links preserve complete context in a URL fragment, and uploaded files remain separate from pinned repository files. The analyzer workflow strip now wraps on 320-pixel screens.

The analyzer now supports planned designs and existing apps, with public GitHub link import, quick behavior choices and a drag-and-drop file input. Repository imports select readable docs, config and source at one commit and show the selection before analysis. Existing-app reviews accept descriptions and up to 10 text files, record submitted sources and evidence limitations, and offer safeguards to verify. The REST API supports existing_app mode and MCP exposes analyze_existing_app. The classifier now reads the full accepted 25,000-character context instead of silently stopping at 15,000 characters.

Live testing found that designs describing deletion jobs returned no precedents and multi-tenant cache designs ranked unrelated breaches first. The behavior classifier now recognises more deletion, infrastructure and scheduling wording, and a bounded ranking signal favours incidents whose failure pattern the design is exposed to. Details are in the test report on the Verification page.

Analysis results attached a fixed lesson to six of the failure patterns and a generic "validate authority, rate limits and idempotency identifiers" lesson to the other eight. Some pairings were wrong — prompt-injection matches recommended hardware MFA, and privileged-action matches recommended rule engines for customer chat. Lessons are now taken from each pattern's own definition and from the matched record's incident-specific safeguard.

Illustrative scenarios could rank above documented incidents. They are now weighted lower and at most one appears in a result.

Older curated records showed anonymized organization names ("Global Cloud Identity & Enterprise SaaS Provider") although their sources name the company. They now show the organization named in the source and a plain-language headline; the underlying record text is unchanged.

Analysis results contained hand-written explanations for about ten specific incidents. After the sources were re-checked, several described events differently from their records. They were removed; every explanation is now generated from the record's own facts and failure pattern, and no longer calls every case a "verified incident".

Most curated records had been imported with a generic index page, an unrelated article or another company's filing, and all carried one placeholder date (2024-03-15). Each record now links a specific primary source (SEC filing, regulator release, court opinion, or the company's own report) and shows a real date labelled as an incident, disclosure or publication date.

Substantive fixes included: SEA-SEC-013 pointed at a different company's filing index; SEA-SEC-018 is now the US-CERT alert on the 2014 entertainment-company wiper attack; SEA-REG-004 now cites the correct CFTC release; SEA-SEC-016 reports Twilio's final figure of 209 customers (125 was an early estimate); SEA-POST-005, -006 and -007 described events that did not match their links and were rewritten; SEA-POST-001 and -002 are design articles and are labelled "Design reference".

Five records (SEA-REG-005, SEA-POST-012 to -015) had no findable public source and were moved to "Illustrative scenario" rather than deleted. The internal benchmark was rebuilt on real incidents only; its top-5 recall is about 0.46, not the 0.69 previously shown, which counted synthetic scenarios as hits.

69 records titled "Verified Case #N" were composite teaching scenarios linked to two unrelated sources, each duplicated 3–4 times. They are now labelled "Illustrative scenario", carry no primary-source claim, and were reduced to 20 unique scenarios. Retired IDs redirect to their canonical scenario.

The advisory ingester tagged about 85% of GitHub/CISA records as AI-agent platforms because its keyword pattern matched short strings such as "ai" inside ordinary words, and attached the same boilerplate cause to every advisory. Classification is now content-based, boilerplate is labelled as a CollectSEA inference, only reviewed GitHub advisories are ingested, and analysis applies a relevance floor.

A later audit found that URL reachability had been treated too broadly as claim verification; bundled records are now labelled machine-extracted pending source-specific human review.