Verification
Does everything actually work?
CollectSEA checks itself in production every two hours, and every release is tested end to end against the live site in a real browser. Results are published here as they come, failures included.
Five layers of testing
From every commit to every two hours in production. Test traffic identifies itself and is left out of the usage figures.
- Step 1Unit and integration testsClassifier, matcher, benchmark floor, server rendering, MCP protocol and telemetry privacy. They gate every deploy.
- Step 2Staging load testConcurrent page, API and analysis requests against a pre-release copy must all succeed.
- Step 3Live release suitesEvery route, API endpoint and MCP tool over HTTP, plus real visitor journeys in Chrome.
- Step 4Self-checks every 2 hoursThe platform tests itself in production and publishes the result on this page.
- Step 5Uptime probesPages, analysis and MCP are probed twice an hour from outside; failures alert the maintainer.
Continuous production self-checks
Last run on release 2026.10.17-seo-visuals: 16 of 17 checks passed in 3.1 s. Machine-readable: /api/verification.
125 of 140 recorded runs passed every check.
- 2026-10-10T14:00:24.717Z: 17 of 17 passed
- 2026-10-10T16:01:27.085Z: 17 of 17 passed
- 2026-10-10T18:01:21.698Z: 17 of 17 passed
- 2026-10-10T20:01:16.399Z: 17 of 17 passed
- 2026-10-11T00:27:55.978Z: 17 of 17 passed
- 2026-10-11T02:20:21.570Z: 17 of 17 passed
- 2026-10-11T04:41:11.641Z: 17 of 17 passed
- 2026-10-11T06:40:59.770Z: 17 of 17 passed
- 2026-10-11T18:40:32.466Z: 17 of 17 passed
- 2026-10-11T20:40:31.944Z: 17 of 17 passed
- 2026-10-11T22:40:55.525Z: 17 of 17 passed
- 2026-10-12T00:40:45.358Z: 17 of 17 passed
- 2026-10-12T02:40:46.473Z: 17 of 17 passed
- 2026-10-12T04:40:44.563Z: 17 of 17 passed
- 2026-10-12T06:40:52.616Z: 17 of 17 passed
- 2026-10-12T08:41:18.061Z: 16 of 17 passed
- 2026-10-12T10:41:08.849Z: 17 of 17 passed
- 2026-10-12T12:41:23.054Z: 17 of 17 passed
- 2026-10-12T14:41:24.484Z: 17 of 17 passed
- 2026-10-12T16:41:28.665Z: 16 of 17 passed
- 2026-10-12T18:41:19.602Z: 16 of 17 passed
- 2026-10-12T20:41:04.307Z: 17 of 17 passed
- 2026-10-12T22:40:53.440Z: 17 of 17 passed
- 2026-10-13T00:40:39.417Z: 17 of 17 passed
- 2026-10-13T02:40:47.118Z: 17 of 17 passed
- 2026-10-13T04:40:43.925Z: 17 of 17 passed
- 2026-10-13T06:42:44.259Z: 17 of 17 passed
- 2026-10-13T08:41:15.452Z: 16 of 17 passed
- 2026-10-13T10:41:28.927Z: 17 of 17 passed
- 2026-10-13T12:44:15.748Z: 17 of 17 passed
- 2026-10-13T16:43:58.772Z: 16 of 17 passed
- 2026-10-13T20:41:52.330Z: 17 of 17 passed
- 2026-10-13T22:41:12.903Z: 17 of 17 passed
- 2026-10-14T00:40:14.184Z: 17 of 17 passed
- 2026-10-14T02:41:01.876Z: 17 of 17 passed
- 2026-10-14T04:41:02.610Z: 16 of 17 passed
- 2026-10-14T06:41:02.682Z: 17 of 17 passed
- 2026-10-14T08:41:03.562Z: 17 of 17 passed
- 2026-10-14T10:41:02.702Z: 17 of 17 passed
- 2026-10-14T12:41:02.643Z: 16 of 17 passed
- 2026-10-14T14:41:06.562Z: 16 of 17 passed
- 2026-10-14T16:41:03.726Z: 16 of 17 passed
- 2026-10-14T18:41:05.700Z: 17 of 17 passed
- 2026-10-14T20:41:03.834Z: 17 of 17 passed
- 2026-10-14T22:41:01.062Z: 17 of 17 passed
- 2026-10-15T00:41:01.185Z: 16 of 17 passed
- 2026-10-15T01:59:22.173Z: 17 of 17 passed
- 2026-10-15T02:40:40.601Z: 17 of 17 passed
- 2026-10-15T04:40:40.303Z: 17 of 17 passed
- 2026-10-15T06:40:23.517Z: 17 of 17 passed
- 2026-10-15T08:40:22.524Z: 17 of 17 passed
- 2026-10-15T10:40:22.172Z: 17 of 17 passed
- 2026-10-15T12:40:22.655Z: 17 of 17 passed
- 2026-10-15T14:40:36.766Z: 17 of 17 passed
- 2026-10-15T16:40:34.257Z: 17 of 17 passed
- 2026-10-15T18:40:34.857Z: 17 of 17 passed
- 2026-10-15T20:40:31.261Z: 17 of 17 passed
- 2026-10-15T22:40:28.268Z: 17 of 17 passed
- 2026-10-16T00:40:29.658Z: 17 of 17 passed
- 2026-10-16T02:40:31.534Z: 17 of 17 passed
- 2026-10-16T04:40:29.672Z: 17 of 17 passed
- 2026-10-16T06:40:29.710Z: 17 of 17 passed
- 2026-10-16T08:40:30.739Z: 17 of 17 passed
- 2026-10-16T10:40:31.803Z: 16 of 17 passed
- 2026-10-16T12:40:36.872Z: 17 of 17 passed
- 2026-10-16T14:40:36.066Z: 17 of 17 passed
- 2026-10-16T16:40:34.114Z: 17 of 17 passed
- 2026-10-16T18:40:35.215Z: 17 of 17 passed
- 2026-10-16T20:40:30.474Z: 17 of 17 passed
- 2026-10-16T22:40:28.914Z: 17 of 17 passed
- 2026-10-17T00:40:27.955Z: 17 of 17 passed
- 2026-10-17T02:40:35.500Z: 17 of 17 passed
- 2026-10-17T04:40:31.138Z: 17 of 17 passed
- 2026-10-17T06:40:30.855Z: 16 of 17 passed
- 2026-10-17T08:40:30.092Z: 17 of 17 passed
- 2026-10-17T10:40:30.629Z: 17 of 17 passed
- 2026-10-17T12:40:34.616Z: 17 of 17 passed
- 2026-10-17T14:40:33.748Z: 17 of 17 passed
- 2026-10-17T16:40:35.188Z: 17 of 17 passed
- 2026-10-17T18:40:34.334Z: 17 of 17 passed
- 2026-10-17T20:40:28.972Z: 17 of 17 passed
- 2026-10-17T22:40:29.255Z: 17 of 17 passed
- 2026-10-18T00:40:29.310Z: 16 of 17 passed
- 2026-10-18T02:40:29.896Z: 16 of 17 passed
Slowest checks in the latest run
Measured duration in milliseconds, including failed checks. This measures the self-check run, not visitor page-load speed.
- Refund agent → duplicate-charge and approval precedents486 ms
- Agent reading GitHub issues → prompt-injection safeguard458 ms
- MCP server negotiates the protocol445 ms
- Cloud cleanup job → deletion-without-recovery precedents405 ms
- Unrelated design returns no padded results390 ms
- Home page is server-rendered with the analyzer353 ms
All 17 checks in the latest run
| Check | Area | Observed | Time | Result |
|---|---|---|---|---|
| Home page is server-rendered with the analyzer | Pages | Headline, analyzer and hydration data present | 353 ms | Pass |
| Incident page shows the record and its source | Pages | CrowdStrike record with source link | 5 ms | Pass |
| Failure-pattern page lists real incidents | Pages | Prompt-injection pattern links its incidents | 15 ms | Pass |
| Framework crosswalk page renders every item | Pages | OWASP Agentic ASI01–ASI10 present | 8 ms | Pass |
| Unknown pages return a real 404 | Search engines | HTTP 404 with noindex | 6 ms | Pass |
| Old URLs redirect permanently | Search engines | /stats → 301 /impact | 0 ms | Pass |
| Pages carry canonical URLs and structured data | Search engines | Canonical URL and Article JSON-LD | 9 ms | Pass |
| Sitemap lists documented incidents | Search engines | 179 URLs | 4 ms | Pass |
| Refund agent → duplicate-charge and approval precedents | Analysis | 6 precedents · Hybrid | 486 ms | Pass |
| Agent reading GitHub issues → prompt-injection safeguard | Analysis | Safeguard for indirect prompt injection among 5 | 458 ms | Pass |
| Cloud cleanup job → deletion-without-recovery precedents | Analysis | 6 precedents | 405 ms | Pass |
| Unrelated design returns no padded results | Analysis | Coverage "none", 0 matches | 390 ms | Pass |
| MCP server negotiates the protocol | MCP | protocolVersion 2025-06-18 | 445 ms | Pass |
| MCP analyze_design returns structured precedents | MCP | 3 precedents | 341 ms | Pass |
| Public API serves the corpus | API & data | 1649 records | 4 ms | Pass |
| Dataset downloads in CSV | API & data | 1649 rows | 3 ms | Pass |
| Advisory feeds refreshed recently | Operations | feeds stale | 186 ms | Fail |
Release test report
Run against https://collectsea.com on 2026-10-17 (release 2026.10.17-seo-visuals, commit b6e5100; Google Chrome (headless) via Playwright, macOS; HTTP suite on Node v26.7.0). 137 of 137 live cases passed; 263 of 263 automated unit and integration tests passed.
HTTP suite: pages, search engines, analysis, MCP, API and security
Every public route, redirect, API endpoint and MCP tool, called over the internet like a client would.
Pages 16/16
Search engines 19/19
Analysis 11/11
MCP 11/11
API & data 12/12
Security 4/4
Performance 1/1
Existing apps 3/3
Every case, with what was expected and observed
| ID | Test | Expected | Observed | Result |
|---|---|---|---|---|
| P-01 | / renders on the serverPages | HTTP 200, server-rendered content contains "Learn from real software failures" | HTTP 200 in 562 ms, 79 KB of HTML | Pass |
| P-02 | /incidents renders on the serverPages | HTTP 200, server-rendered content contains "How real software and AI systems failed" | HTTP 200 in 109 ms, 179 KB of HTML | Pass |
| P-03 | /incidents/SEA-SEC-003 renders on the serverPages | HTTP 200, server-rendered content contains "CrowdStrike" | HTTP 200 in 98 ms, 46 KB of HTML | Pass |
| P-04 | /incidents/SEA-AI-001 renders on the serverPages | HTTP 200, server-rendered content contains "Replit" | HTTP 200 in 59 ms, 59 KB of HTML | Pass |
| P-05 | /patterns renders on the serverPages | HTTP 200, server-rendered content contains "The 14 ways software keeps failing" | HTTP 200 in 74 ms, 184 KB of HTML | Pass |
| P-06 | /patterns/duplicate-action-after-timeout renders on the serverPages | HTTP 200, server-rendered content contains "Duplicate action after an ambiguous result" | HTTP 200 in 71 ms, 72 KB of HTML | Pass |
| P-07 | /frameworks renders on the serverPages | HTTP 200, server-rendered content contains "Standards, mapped to the incidents behind them" | HTTP 200 in 55 ms, 26 KB of HTML | Pass |
| P-08 | /frameworks/nist-sp-800-53 renders on the serverPages | HTTP 200, server-rendered content contains "SA-15(8)" | HTTP 200 in 60 ms, 144 KB of HTML | Pass |
| P-09 | /impact renders on the serverPages | HTTP 200, server-rendered content contains "Who CollectSEA reaches" | HTTP 200 in 1459 ms, 93 KB of HTML | Pass |
| P-10 | /verification renders on the serverPages | HTTP 200, server-rendered content contains "Does everything actually work?" | HTTP 200 in 239 ms, 222 KB of HTML | Pass |
| P-11 | /integrations renders on the serverPages | HTTP 200, server-rendered content contains "claude mcp add" | HTTP 200 in 41 ms, 38 KB of HTML | Pass |
| P-12 | /docs renders on the serverPages | HTTP 200, server-rendered content contains "API and MCP reference" | HTTP 200 in 69 ms, 54 KB of HTML | Pass |
| P-13 | /methodology renders on the serverPages | HTTP 200, server-rendered content contains "How CollectSEA finds precedents" | HTTP 200 in 155 ms, 66 KB of HTML | Pass |
| P-14 | /dataset renders on the serverPages | HTTP 200, server-rendered content contains "The CollectSEA incident dataset" | HTTP 200 in 64 ms, 83 KB of HTML | Pass |
| P-15 | /changelog renders on the serverPages | HTTP 200, server-rendered content contains "What has changed" | HTTP 200 in 44 ms, 30 KB of HTML | Pass |
| P-16 | /about renders on the serverPages | HTTP 200, server-rendered content contains "Software keeps failing the same ways" | HTTP 200 in 42 ms, 39 KB of HTML | Pass |
| S-01 | Canonical URL, description and Open Graph tagsSearch engines | Each page has canonical, description, og:image (PNG) | canonical, description and og:image present | Pass |
| S-02 | Structured data on home and incident pagesSearch engines | FAQPage, SoftwareApplication, Article and BreadcrumbList JSON-LD | All expected schema.org types present | Pass |
| S-03 | Dataset page is discoverable by Google Dataset SearchSearch engines | schema.org Dataset with license and downloads | Dataset with CC BY 4.0 license and 3 downloads | Pass |
| S-04 | Unknown page returns 404Search engines | HTTP 404 with noindex | HTTP 404, noindex | Pass |
| S-05 | Unknown incident ID returns 404Search engines | HTTP 404 | HTTP 404 | Pass |
| S-06 | https://www.collectsea.com/incidents redirects permanentlySearch engines | 301 → https://collectsea.com/incidents | 301 → https://collectsea.com/incidents | Pass |
| S-07 | https://docs.collectsea.com/ redirects permanentlySearch engines | 301 → https://collectsea.com/docs | 301 → https://collectsea.com/docs | Pass |
| S-08 | /stats redirects permanentlySearch engines | 301 → /impact | 301 → /impact | Pass |
| S-09 | /research redirects permanentlySearch engines | 301 → /methodology#evaluation | 301 → /methodology#evaluation | Pass |
| S-10 | /incidents?id=SEA-SEC-003 redirects permanentlySearch engines | 301 → /incidents/SEA-SEC-003 | 301 → /incidents/SEA-SEC-003 | Pass |
| S-11 | /incidents/sea-sec-003 redirects permanentlySearch engines | 301 → /incidents/SEA-SEC-003 | 301 → /incidents/SEA-SEC-003 | Pass |
| S-12 | /patterns/SEA-FM-004 redirects permanentlySearch engines | 301 → /patterns/indirect-prompt-injection | 301 → /patterns/indirect-prompt-injection | Pass |
| S-13 | /incidents/ redirects permanentlySearch engines | 301 → /incidents | 301 → /incidents | Pass |
| S-14 | Sitemap lists every documented incident, pattern and frameworkSearch engines | Valid XML with ≥ 80 URLs; no advisories or private pages | 179 URLs | Pass |
| S-15 | robots.txt points to the sitemapSearch engines | Sitemap directive; /review disallowed | Sitemap and Disallow present | Pass |
| S-16 | llms.txt summarises the site for AI assistantsSearch engines | Lists patterns, frameworks and incidents | 160 lines | Pass |
| S-17 | Atom feed of new incidentsSearch engines | Valid Atom with entries | 50 entries | Pass |
| S-18 | Social preview imageSearch engines | PNG served at 1200×630 | PNG 1200×630 | Pass |
| S-19 | Product tour videoSearch engines | Both MP4 renditions and the poster are served; byte ranges return 206 (required by Safari); VideoObject on home | 1080p 16.9 MB, 720p 7.7 MB; poster and VideoObject present | Pass |
| A-01 | Refund agent designAnalysis | Precedents with idempotency or approval-boundary patterns; coverage strong | 6 precedents, coverage strong, 630 ms | Pass |
| A-02 | Agent reading public GitHub issuesAnalysis | GitHub/Supabase MCP prompt-injection incidents; prompt-injection safeguard | Matched SEA-AI-003, SEA-AI-004, SEA-AI-007, SEA-AI-008 | Pass |
| A-03 | Global configuration pushAnalysis | CDN/edge outage precedents (Cloudflare, Fastly, CrowdStrike) | SEA-SEC-003, SEA-POST-006, SEA-POST-005, SEA-SEC-010 | Pass |
| A-04 | Cloud cleanup jobAnalysis | Deletion-without-recovery precedents (GitLab, Atlassian, UniSuper) | SEA-POST-009, SEA-POST-004, SEA-POST-010, SEA-SEC-012 | Pass |
| A-05 | Multi-tenant shared cacheAnalysis | A cross-tenant precedent in the top 3 | SEA-PUB-059, SEA-CVE-2026-108606, SEA-SEC-015 | Pass |
| A-06 | Unrelated design is not paddedAnalysis | No precedents; coverage "none" | coverage none, 0 matches | Pass |
| A-07 | Semantic retrieval activeAnalysis | retrieval_algorithm reports Workers AI embeddings | Hybrid: Workers AI embeddings (bge-base-en-v1.5) + Weighted Capability Overlap | Pass |
| A-08 | At most one illustrative scenario per resultAnalysis | ≤ 1 SEA-PUB record among matches | 1 illustrative of 10 | Pass |
| A-09 | Exclusion list for "show more precedents"Analysis | Excluded IDs never returned | SEA-POST-001 excluded | Pass |
| A-10 | Input validationAnalysis | Empty → 400, malformed JSON → 400, oversize → 413 | 400 / 400 / 413 | Pass |
| A-11 | Legacy "idea" field still acceptedAnalysis | HTTP 200 with matches | HTTP 200 | Pass |
| M-01 | initialize negotiates the protocol versionMCP | Echoes 2025-06-18; serverInfo and instructions present | server 2026.10.17-seo-visuals | Pass |
| M-02 | Notifications acknowledgedMCP | HTTP 202 without body | HTTP 202 | Pass |
| M-03 | GET for SSE stream refusedMCP | HTTP 405 (stateless server) | HTTP 405 | Pass |
| M-04 | tools/listMCP | 6 read-only tools with input schemas | analyze_design, analyze_existing_app, search_incidents, get_incident, list_failure_patterns, get_safeguards | Pass |
| M-05 | analyze_designMCP | Markdown summary plus structured precedents | 4 precedents | Pass |
| M-06 | search_incidentsMCP | Keyword search returns CrowdStrike for "channel file" | 70 results | Pass |
| M-07 | get_incidentMCP | Full record with source for SEA-POST-009 | GitLab record returned | Pass |
| M-08 | list_failure_patternsMCP | 14 patterns | 14 patterns | Pass |
| M-09 | get_safeguardsMCP | AGENTS.md content citing precedents | 6283 characters | Pass |
| M-10 | Errors are reported correctlyMCP | Unknown tool → -32602; unknown method → -32601; bad JSON → 400 | -32602 / -32601 / 400 | Pass |
| M-11 | Legacy endpoint and tool namesMCP | /api/mcp with analyze_architecture_proposal works | works | Pass |
| D-01 | Health endpoint (deployment contract)API & data | status operational with corpus counts | 1649 records, feeds healthy | Pass |
| D-02 | Incident list (full and summary views)API & data | incidents array; summary view is smaller | 1649 records; summary 927 KB vs full 7631 KB | Pass |
| D-03 | Single record and unknown recordAPI & data | 200 for SEA-SEC-003, 404 for unknown | 200 / 404 | Pass |
| D-04 | Patterns and frameworks endpointsAPI & data | 14 patterns; 7 frameworks | 14 patterns, 7 frameworks | Pass |
| D-05 | OpenAPI schemaAPI & data | OpenAPI 3.1 with analyzeDesign operation | 11 paths | Pass |
| D-06 | Dataset download (JSON)API & data | Attachment with all served records | 7631 KB | Pass |
| D-07 | Dataset download (CSV)API & data | Attachment with all served records | 1844 KB | Pass |
| D-08 | Dataset download (JSONL)API & data | Attachment with all served records | 2281 KB | Pass |
| D-09 | Impact report and CSVAPI & data | JSON report with 30 daily points; CSV header | tracking since 2026-10-01 | Pass |
| D-10 | Self-check results publishedAPI & data | Latest run with all checks passing | 17/17 at 2026-10-17T18:40 | Pass |
| D-11 | Feedback validationAPI & data | Missing message → 400; record report needs a valid ID → 400 | 400 / 400 | Pass |
| D-12 | Unknown API routeAPI & data | JSON 404 listing endpoints | JSON 404 | Pass |
| O-01 | Security headers on pagesSecurity | CSP with script hash, HSTS, nosniff, frame protection | CSP (no unsafe-inline scripts), HSTS, nosniff, X-Frame-Options | Pass |
| O-02 | Private review API rejects unauthorized requestsSecurity | HTTP 401 without token or 503 when review is disabled | HTTP 401: unauthorized or disabled | Pass |
| O-03 | Manual feed sync requires a tokenSecurity | HTTP 401 without token | HTTP 401 | Pass |
| O-04 | Record text cannot inject scriptSecurity | Record page escapes HTML in data | Embedded data escapes "<" | Pass |
| O-05 | Pages served quickly under concurrencyPerformance | 40 parallel page and API requests succeed; median < 1.5 s | 0 failures; median 286 ms, max 693 ms | Pass |
| A-EX-01 | Files-only existing-app analysisExisting apps | Returns source context, app mode and refund behavior | 4 precedents; submitted source and evidence limits returned | Pass |
| M-EX-01 | MCP existing-app toolExisting apps | Returns structured app context and safeguards to verify | MCP app review returns precedents, safeguards and evidence limits | Pass |
| A-EX-02 | Public repository context importExisting apps | Returns inspectable files and a pinned public repository revision | 3 selected files at dd32ce8; arbitrary host rejected | Pass |
Browser suite: real user journeys in Google Chrome
Desktop and mobile flows driven by Playwright, with screenshots as evidence.
Home 3/3
Analysis 7/7
Incidents 5/5
Patterns 2/2
Frameworks 1/1
Integrations 1/1
Navigation 2/2
Accessibility 3/3
Mobile 2/2
Feedback 1/1
Data 1/1
Transparency 1/1
Errors 1/1
Motion 1/1
Existing apps 11/11
Quality 1/1
Every case, with what was expected and observed
| ID | Test | Expected | Observed | Result |
|---|---|---|---|---|
| U-01 | Home page loads and hydrates without errorsHome | Hero, analyzer and corpus figures visible; no console errors | Hydrated cleanly; figures: Documented incidents0Postmortems, filings, court recordsLive security advisories…screenshot | Pass |
| U-02 | Live behavior detection while typingHome | Detected capability chips appear under the input | Runs code or shell commands; Changes infrastructure; Changes data; Handles credentials or secrets; AI agent that calls toolsscreenshot | Pass |
| U-03 | Example prompts run an editable analysisHome | Clicking "Cloud cleanup job" opens results with an editable proposal | A nightly background job that finds unused cloud databases and virtual machines and deletes them to save cost.screenshot | Pass |
| U-04 | Run an analysis end to endAnalysis | Safeguards, ranked precedents and profile panel render | 5 safeguards, 8 precedents; proposal kept out of the URLscreenshot | Pass |
| U-05 | Score breakdown is inspectableAnalysis | "How scored" reveals score components | Composite score 0.80. How scoring worksscreenshot | Pass |
| U-06 | Export AGENTS.md for a coding agentAnalysis | Download produces an AGENTS.md citing precedents | AGENTS.md (7429 characters)screenshot | Pass |
| U-07 | Mark a safeguard as already in placeAnalysis | Ticked safeguard moves to "Already in place" in exports | Section jumps, refresh and reading evidence preserve choices; export includes "Already in place"screenshot | Pass |
| U-08 | Share link reproduces the analysisAnalysis | Copied ?q= link opens the same proposal and results | Shared link restored the proposal and resultsscreenshot | Pass |
| U-09 | Edit and re-runAnalysis | Edited proposal produces different precedents | Top precedent changed to "Engineer deletes the production database directory and every backup me"screenshot | Pass |
| U-10 | Unrelated design shows an honest empty stateAnalysis | "No close precedent" message, no padded results | Empty state shownscreenshot | Pass |
| U-11 | Search the incident databaseIncidents | Typing "redis" filters results and updates the URL | 60 → 3 records; URL ?q=outagescreenshot | Pass |
| U-12 | Evidence-type tabs, pattern filter and AI filterIncidents | Each filter narrows the list | advisories 60, prompt-injection 5, AI systems 15screenshot | Pass |
| U-13 | Open an incident by client-side navigationIncidents | Card click opens the record without a full reload; title updates | /incidents/SEA-POST-005 — "Cloudflare bot-management outage (Nov 2025) | CollectSEA…"screenshot | Pass |
| U-14 | Incident page shows evidence, source, safeguards and citationIncidents | Source link, trust panel, safeguards, related incidents, citation copy | Source, safeguards, related incidents and BibTeX citation presentscreenshot | Pass |
| U-15 | Report-a-problem form validates inputIncidents | Empty report is rejected with a message | Validation message shown (no report submitted)screenshot | Pass |
| U-16 | Failure pattern pagePatterns | Definition, prevention guidance and real incidents | 17 incident cardsscreenshot | Pass |
| U-17 | Framework crosswalk deep linkFrameworks | Anchor to ASI01 lands on the item with linked incidents | ASI01 shows 5 incidentsscreenshot | Pass |
| U-18 | MCP setup switches per client and copies configIntegrations | Cursor tab shows .cursor/mcp.json; copy succeeds | Cursor config copiedscreenshot | Pass |
| U-19 | Command palette search (⌘K)Navigation | Searching "crowdstrike" and pressing Enter opens the record | Opened /incidents/SEA-SEC-003 | Pass |
| U-20 | Browser back and forwardNavigation | History navigation restores pages | Back and forward restored the right pages | Pass |
| U-21 | Theme toggle persists without a flashAccessibility | Dark mode set before first paint after reload | Dark class applied by inline script on reloadscreenshot | Pass |
| U-22 | Keyboard: skip link and focusAccessibility | First Tab reveals "Skip to content"; Enter moves focus to main | Skip link works | Pass |
| U-23 | Images and controls are labelledAccessibility | Every button and link has an accessible name; one h1 per page | 6 pages: no unnamed controls, exactly one h1, all images have alt text | Pass |
| U-24 | Mobile layout and menuMobile | No horizontal scrolling; menu opens and navigates | 9 pages fit 390 px; menu navigation worksscreenshot | Pass |
| U-25 | Mobile analysisMobile | Analysis results usable on a phone | Results fit (390 px)screenshot | Pass |
| U-26 | Feedback dialog submitsFeedback | Message accepted and confirmation shown | Confirmation shownscreenshot | Pass |
| U-27 | Dataset download from the pageData | CSV download starts from /dataset | collectsea-incidents-2026.10.17.csv (1649 rows)screenshot | Pass |
| U-28 | Impact and verification pages render live dataTransparency | Charts with table fallback; self-check table | 30-day table view and self-check results visiblescreenshot | Pass |
| U-29 | Unknown pageErrors | Friendly 404 with navigation | HTTP 404 page with linksscreenshot | Pass |
| U-31 | Scroll animations reveal content without hiding itMotion | On-screen content animates in on arrival; below-the-fold cards reveal on scroll and settle fully visible; with reduced motion nothing moves (fades only) | 17 arrival animations; 12 elements revealed on scroll; 0 hidden above the fold; reduced motion: fades onlyscreenshot | Pass |
| U-32 | Animated pattern explainer plays and is interactivePatterns | One timeline advances; safeguard steps update the diagram and caption; index previews stay still | One timeline advances, safeguard step 3 shows "The retry reuses the same key, so the API recognises it and returns the first result.", 14 static previews on /patternsscreenshot | Pass |
| U-33 | Files-only analysis, reload, editing and exportExisting apps | Uploaded context stays private, existing-app mode survives reload and edits, exports include source context | Files-only review, reload, editing, export and removal passed; context absent from URLscreenshot | Pass |
| U-34 | Input errors and mode-specific cachingExisting apps | Invalid files are rejected and identical text retains the selected analysis mode | Invalid archive rejected; same description analyzed in both modes with correct labelsscreenshot | Pass |
| U-35 | Quick review without uploadsExisting apps | Behavior choices start an app review with no file or description required | Review created from behavior choices, with safeguards and no uploaded filesscreenshot | Pass |
| U-36 | Public GitHub import, source preview and reviewExisting apps | A repo link imports pinned context and preserves provenance through reload and export | Public repository imported, sources previewed, commit preserved after reload and in exportscreenshot | Pass |
| U-37 | Unfinished review survives refresh and return navigationExisting apps | Notes and behavior choices remain editable after reload and after reviewing | Mode, notes and removable behavior choices survive refresh and returnscreenshot | Pass |
| U-38 | Shared file review preserves sources in a fresh tabExisting apps | Sharing retains file names, contents and app mode without context in the HTTP query | Fresh recipient retains app mode, file name, contents and export provenancescreenshot | Pass |
| U-39 | Mixed source provenance and file removalExisting apps | Uploaded notes are separate from GitHub files; removing imported files clears the revision | GitHub and uploaded files remain distinct; revision clears with last imported filescreenshot | Pass |
| U-40 | Narrow phone layoutExisting apps | Every analyzer element fits a 320 px screen | All three context paths fit a 320 px dark screenscreenshot | Pass |
| U-41 | Import error recoveryExisting apps | A rejected repository link leaves quick review usable | Actionable import error followed by a successful upload-free reviewscreenshot | Pass |
| U-42 | Offline review fallbackExisting apps | Browser matching retains existing-app context when the API is unavailable | Browser fallback displays its limitations and retains existing-app modescreenshot | Pass |
| U-43 | Malformed shared contextExisting apps | Bad share links show an error and allow returning to the analyzer | Invalid share fragment rejected with recovery navigation and no crashscreenshot | Pass |
| U-30 | No console errors across the siteQuality | Zero JavaScript errors recorded during the whole suite | No console errors | Pass |
Production self-checks (latest scheduled run)
Run by the platform itself every two hours; this run 2026-10-17 18:40 UTC.
Pages 4/4
Search engines 4/4
Analysis 4/4
MCP 2/2
API & data 2/2
Operations 1/1
Every case, with what was expected and observed
| ID | Test | Expected | Observed | Result |
|---|---|---|---|---|
| SC-01 | Home page is server-rendered with the analyzerPages | Pass | Headline, analyzer and hydration data present | Pass |
| SC-02 | Incident page shows the record and its sourcePages | Pass | CrowdStrike record with source link | Pass |
| SC-03 | Failure-pattern page lists real incidentsPages | Pass | Prompt-injection pattern links its incidents | Pass |
| SC-04 | Framework crosswalk page renders every itemPages | Pass | OWASP Agentic ASI01–ASI10 present | Pass |
| SC-05 | Unknown pages return a real 404Search engines | Pass | HTTP 404 with noindex | Pass |
| SC-06 | Old URLs redirect permanentlySearch engines | Pass | /stats → 301 /impact | Pass |
| SC-07 | Pages carry canonical URLs and structured dataSearch engines | Pass | Canonical URL and Article JSON-LD | Pass |
| SC-08 | Sitemap lists documented incidentsSearch engines | Pass | 179 URLs | Pass |
| SC-09 | Refund agent → duplicate-charge and approval precedentsAnalysis | Pass | 6 precedents · Hybrid | Pass |
| SC-10 | Agent reading GitHub issues → prompt-injection safeguardAnalysis | Pass | Safeguard for indirect prompt injection among 5 | Pass |
| SC-11 | Cloud cleanup job → deletion-without-recovery precedentsAnalysis | Pass | 6 precedents | Pass |
| SC-12 | Unrelated design returns no padded resultsAnalysis | Pass | Coverage "none", 0 matches | Pass |
| SC-13 | MCP server negotiates the protocolMCP | Pass | protocolVersion 2025-06-18 | Pass |
| SC-14 | MCP analyze_design returns structured precedentsMCP | Pass | 3 precedents | Pass |
| SC-15 | Public API serves the corpusAPI & data | Pass | 1649 records | Pass |
| SC-16 | Dataset downloads in CSVAPI & data | Pass | 1649 rows | Pass |
| SC-17 | Advisory feeds refreshed recentlyOperations | Pass | healthy; last full success 2026-10-17 17:05 UTC | Pass |
Defects found and fixed
Testing is only useful if it finds problems. Each of these was found on the live site or in its data, fixed, redeployed and re-tested. Newest last.
| ID | Issue | Fix | Verified by |
|---|---|---|---|
| DEF-01 | The home page's own "Cloud cleanup job" example returned no precedents, although GitLab, Atlassian, UniSuper and AWS S3 deletion incidents are in the corpus.Found by HTTP suite probe A-04 (live site). A core example produced an empty result. | Broadened the behavior classifier for deletion wording ("deletes", "tears down", "drops tables"), infrastructure ("virtual machines", "cloud databases") and scheduled jobs ("nightly", "background job"). Internal benchmark recall@5 rose from 0.43 to 0.50. | A-04, U-09, self-check analysis-deletion, unit test |
| DEF-02 | For a shared multi-tenant cache, unrelated breaches (e.g. an unpatched web framework) ranked above cross-tenant precedents.Found by HTTP suite probe A-05 (live site). Less relevant precedents were shown first. | Added a bounded likely-pattern signal: behaviors in the design imply failure patterns (a multi-tenant cache implies cross-tenant access), which re-ranks records that already qualify but can never qualify a record alone. Recall@5 0.50 → 0.525, precision@3 0.47 → 0.50; unrelated designs still return nothing. | A-05, A-06, unit tests |
| DEF-03 | Typing or clicking immediately after a page loaded could be lost, because the page's code was fetched only after the HTML arrived.Found by Browser suite U-11, U-18, U-20 (first run). Fast first interactions could do nothing on slow connections. | The Worker now reads the build manifest and adds modulepreload links for the current page's code, so pages become interactive sooner; the app marks when it is ready. | U-11, U-18, U-20 (re-run) |
| DEF-04 | The edge cache key used only the release name, so for up to five minutes after a deploy, cached pages could reference the previous build's script files.Found by Manual check of preload tags after deploy. Possible broken pages immediately after a deployment. | Cache keys now include a fingerprint of the deployed build. | Post-deploy HTML inspection; P-01 to P-16 |
| DEF-05 | Automated test and self-check traffic would have been counted as real visitors and analyses.Found by Test design review. Usage figures would be inflated. | Requests identifying as CollectSEA test or self-check clients, and maintainer browsers carrying an internal cookie, are excluded from all usage counters. | Unit test (telemetry exclusion) |
| DEF-06 | The GitHub advisory feed failed with HTTP 403 and the NVD feed timed out.Found by Service status panel during pre-release QA. New advisories were not being added. | NVD query narrowed to a 7-day window (1.5 s instead of timing out); GitHub requests send the recommended API headers and accept an optional token. All three feeds succeeded on the next sync. | D-01, self-check feeds |
| DEF-07 | Code samples and long labels made the home and analysis pages wider than a phone screen.Found by Mobile overflow scan during pre-release QA. Sideways scrolling on mobile. | Grid columns may now shrink below their content width; long pattern names wrap. | U-24, U-25 |
| DEF-08 | An internet-wide security scanner (CensysInspect) was listed as an AI client connected over MCP, and its handshakes counted as MCP calls.Found by review of live usage data (October 17, 2026). The impact page overstated MCP use; all three lifetime MCP calls came from the scanner. | Security scanners are now detected by user agent and counted only as automated traffic, never as visitors, analyses, MCP calls or AI clients; past scanner entries are left out of the client list. | tests/telemetry_scanner.test.ts |
Evidence screenshots
Captured automatically by the browser test suite on the live site.
U-01
Home page loads and hydrates without errors
U-02
Live behavior detection while typing
U-03
Example prompts run an editable analysis
U-04
Run an analysis end to end
U-05
Score breakdown is inspectable
U-06
Export AGENTS.md for a coding agent
U-07
Mark a safeguard as already in place
U-08
Share link reproduces the analysis
U-09
Edit and re-run