{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","count":2,"catalogCount":27,"refused":["Replit agent database story (July 2025): The Register URL returned HTTP 403 and no first-party engineering postmortem was fetched. Not indexed.","CanLII HTML for 2024 BCCRT 149 returned a client-side block. The Civil Resolution Tribunal document was fetched instead and is the indexed source.","Cursor agent best-practices post loaded and is a product guide, not a postmortem. Not indexed as an incident. The University of Chicago study it mentions was not fetched.","Cognition 'Introducing Devin' (published 2024-03-12) duplicates the SWE-bench technical report. The report is indexed; the launch post is not a second study.","Anthropic many-shot jailbreaking research page loaded. It is an attack writeup, so it is not indexed and no procedure is stored.","OWASP LLM01:2025 was fetched and is cited on the prompt-injection failure mode only. Attack examples from that page are not copied into a case study.","Calder Lab and similar unsourced production-metric blogs were not fetched and are not indexed.","Google Research abstract for Data Cascades in High-Stakes AI loaded on 2026-10-03 (the abstract says 92% prevalence). Not indexed for data-etl: it is an interview study of ML data practice, not a pipeline job failure. The indexed data-etl study is the dbt Cloud status page."],"studies":[{"id":"swe-bench-verified-2024","title":"Introducing SWE-bench Verified","year":2024,"studyKind":"vendor_writeup","classIds":["coding","eval-judge"],"failureModeIds":["eval-contamination","benchmark-protocol-mismatch","weak-verification"],"lesson":"OpenAI's page says the original SWE-bench can grade a valid patch as wrong when unit tests are overly specific, the issue text is underspecified, or the environment fails to set up. They released a 500-sample human-validated subset, with difficulty slices of 196 tasks they mark as under 15 minutes and 45 they mark as over an hour. Cite the subset name with any score.","sourceUrl":"https://openai.com/index/introducing-swe-bench-verified/","asOf":"2026-10-03","notes":"Page date on the fetch: August 13, 2024. Model score tables on that page were not copied here."},{"id":"osworld-verified-site-2025","title":"OSWorld project site: verified upgrade","year":2025,"studyKind":"capability_ceiling","classIds":["computer-use","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch","eval-contamination"],"lesson":"The project site says that on 2025-07-28 OSWorld was enhanced as OSWorld-Verified, including fixed community-reported examples and updated benchmark results, and that new runs should be compared with those new results. A 2026-06-26 line says OSWorld 2.0 is available. A score from the 2024 paper is not the same number as a score on the later set.","sourceUrl":"https://os-world.github.io/","asOf":"2026-10-03","notes":"Fetched the project homepage. What the 2.0 task set contains was not copied because the homepage line only says it is available."}]}