Methodology

How CollectSEA finds precedents, and what it can’t do

Deterministic where it can be, open about its scores and honest about how strong the evidence is. Everything here is detailed enough to reproduce or pick apart.

Match on behavior, not on names

Failures repeat because the conditions behind them repeat: an ambiguous timeout, an agent reading untrusted text, a config change that reaches every server at once. CollectSEA doesn’t ask what your system is called. It asks what it can do, and what went wrong when other systems could do the same.

That is what NIST SP 800-53 SA-15(8), the SSDF (RV.3) and the AI RMF (MAP 5.1) ask for: learn from similar systems before you build. The framework crosswalk shows each mapping.

How a design is matched

Scroll the diagram sideways to see every system.

15 / 15 s

You get ranked real precedents, each with the safeguard that would have stopped it.

  1. You describe what you’re building in plain words. Nothing is stored.
  2. The description is reduced to behaviors: what it can change, what it handles, how it’s built.
  3. Every record was profiled with the same vocabulary, so behaviors and meaning can be compared directly.
  4. A transparent score keeps only matches backed by more than one signal, so unrelated designs get no padding.
  5. You get ranked real precedents, each with the safeguard that would have stopped it.

Four steps, every time

  1. Step 1Profile the designA rule-based classifier picks out which of 40 behaviors your description shows: what it can change, what data it handles, how it is built, and how reversible its actions are.
  2. Step 2Find candidatesEvery record carries the same profile. Behaviors are compared directly, and a sentence-embedding model compares meaning so different wording still matches.
  3. Step 3Score and filterA published, weighted score ranks candidates. Matches resting on a single weak signal are dropped, so an unrelated design gets nothing rather than filler.
  4. Step 4Explain and adviseEach match says what the two systems share and what went wrong. Safeguards come from the incident itself and its failure pattern, never from a template.

How a match is scored

Each signal adds to a record’s score with a fixed weight. The total is then multiplied by how strong the record’s evidence is.

  • Shared behaviors+0.30
  • Semantic similarity+0.30

    Rescaled from cosine 0.58–0.78 (BAAI bge-base-en-v1.5 on Workers AI)

  • Same kind of consequential action+0.25
  • Similar consequences+0.20
  • Specific technology terms in common+0.20

    0.10 each, up to 0.20

  • Shared trust boundaries+0.15
  • Likely failure pattern (re-ranking only)+0.12

    Can reorder records that already qualify, never qualify one on its own

  • Wording overlap+0.10

When a record qualifies

  • It scores at least 0.30 with two independent signals behind it
  • or it shares a specific technology term with your design
  • or its meaning is a close match (similarity 0.65 or more)

Thresholds were set on fixed test queries: unrelated designs never scored above about 0.57, genuine analogues landed between 0.64 and 0.78. Below 0.58 similarity, shared behaviors alone can’t qualify a record.

The exact formula +
score = 0.30 × shared behaviors (overlap coefficient)
      + 0.25 × same kind of consequential action
      + 0.20 × similar consequences
      + 0.15 × shared trust boundaries
      + 0.10 × wording overlap (stemmed)
      + 0.30 × semantic similarity (cosine 0.58–0.78)
      + 0.10 per shared technology term (max 0.20)
      − small penalty for AI / non-AI mismatch
      + 0.08 per likely failure pattern (max 0.12)
then × evidence weight

Every analysis returns its own score breakdown and the versions it used.

Not all evidence is equal

Every record carries a label, and the label changes its weight in the ranking.

  • Documented incident× 1.0

    A real incident summarised from a public source (postmortem, regulatory filing, court record or advisory). The source link was checked; the summary has not yet been adjudicated by a human reviewer.

  • Incident disclosure× 0.9

    A company’s own SEC filing about a cybersecurity incident (Form 8-K, Item 1.05), ingested automatically. Facts are quoted from the filing; the classification is machine-extracted.

  • Illustrative scenario× 0.9

    A composite failure pattern written by CollectSEA for learning and demos. It is NOT a documented real-world incident and has no primary source.

  • Security advisory× 0.85

    Automatically ingested from GitHub, CISA KEV, NVD or OSV. Machine-extracted: a vulnerability report, not a full incident write-up.

Facts in curated records stay labelled as CollectSEA summaries until a reviewer checks them against the source. Corrections are listed on the dataset page.

Does it keep finding the right incidents?

A regression benchmark of 10 design scenarios checks that retrieval still finds the incidents a maintainer judged relevant, and still leaves out the distractors. Live results from the corpus being served right now (raw data):

Recall@5
Keywords and behaviors only: 50%
With semantic matching: 50%
Precision@3
Keywords and behaviors only: 47%
With semantic matching: 43%
Distractor rate · lower is better
Keywords and behaviors only: 0.02
With semantic matching: 0.02

These numbers guard against regressions. They don’t measure whether CollectSEA makes software safer; that’s what the study below is for.

Can it recognise the system behind each documented incident?

For each of the 65 documented incidents, a short description of the affected system as its builders might have described it before the failure is searched against the documented incidents and teaching scenarios. It never names the organization, a CVE, a date or what went wrong, though it may name technologies such as PostgreSQL. With semantic matching, the incident itself ranks in the top five for 86% of descriptions (95% interval 76%–93%).

Ranked first
Keywords and behaviors only: 23%
With semantic matching: 71%
In the top five
Keywords and behaviors only: 45%
With semantic matching: 86%
Same failure pattern in the top five
Keywords and behaviors only: 69%
With semantic matching: 92%

The descriptions were written by the maintainer from the records, so this measures whether behaviour is recognised, not independent usefulness. Queries and per-incident ranks are in the raw data.

The question that matters

Do precedents change what people build? A planned independent study will find out.

  • Real engineers

    Engineers and AI-agent users who did not write the ground truth, working on the same design tasks with and without CollectSEA.

  • What gets measured

    Hazards spotted, unsafe implementations shipped, review time, false alarms and which safeguards were actually adopted.

  • Published either way

    A frozen corpus and protocol, blinded raters and confidence intervals, with negative results published too.

What it can’t do

  • No precedent ≠ safe

    The corpus leans toward incidents that large organizations disclosed publicly. An empty result is not a clean bill of health.

  • Review is ongoing

    Most curated summaries haven’t yet been checked line by line by a human reviewer, and advisories are machine-extracted.

  • The classifier has blind spots

    It is rule-based and can miss unfamiliar phrasing. Semantic matching narrows the gap but doesn’t close it.

  • Analogy, not prediction

    A matched incident points at a risk worth checking. It doesn’t mean your system will fail the same way.

Spotted a wrong source, date or classification? Every record has a “Report a problem” link that goes straight to the review queue.