{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","count":5,"catalogCount":27,"refused":["Replit agent database story (July 2025): The Register URL returned HTTP 403 and no first-party engineering postmortem was fetched. Not indexed.","CanLII HTML for 2024 BCCRT 149 returned a client-side block. The Civil Resolution Tribunal document was fetched instead and is the indexed source.","Cursor agent best-practices post loaded and is a product guide, not a postmortem. Not indexed as an incident. The University of Chicago study it mentions was not fetched.","Cognition 'Introducing Devin' (published 2024-03-12) duplicates the SWE-bench technical report. The report is indexed; the launch post is not a second study.","Anthropic many-shot jailbreaking research page loaded. It is an attack writeup, so it is not indexed and no procedure is stored.","OWASP LLM01:2025 was fetched and is cited on the prompt-injection failure mode only. Attack examples from that page are not copied into a case study.","Calder Lab and similar unsourced production-metric blogs were not fetched and are not indexed.","Google Research abstract for Data Cascades in High-Stakes AI loaded on 2026-10-03 (the abstract says 92% prevalence). Not indexed for data-etl: it is an interview study of ML data practice, not a pipeline job failure. The indexed data-etl study is the dbt Cloud status page."],"studies":[{"id":"visualwebarena-2024","title":"VisualWebArena","year":2024,"studyKind":"capability_ceiling","classIds":["browser","computer-use"],"failureModeIds":["gui-grounding-miss"],"lesson":"The abstract says many web tasks need visual information that text-only agents do not use well, and that the benchmark is visually grounded. A browser agent that reads only the DOM or the HTML is being measured on a different task than this one. No success rate from the abstract is repeated here because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2401.13649","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: accepted to ACL 2024."},{"id":"osworld-2024","title":"OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments","year":2024,"studyKind":"capability_ceiling","classIds":["computer-use"],"failureModeIds":["gui-grounding-miss","weak-verification"],"lesson":"The abstract describes 369 real computer tasks across operating systems and apps, with execution-based checks. It says humans accomplish over 72.36% and the best model 12.24%, and that models struggle with GUI grounding and operational knowledge. This is a computer-use ceiling from the paper, not a production outage.","sourceUrl":"https://arxiv.org/abs/2404.07972","asOf":"2026-10-03","notes":"Fetched arXiv v2, last revised 30 May 2024. Later site notes about OSWorld-Verified are a separate study."},{"id":"osworld-verified-site-2025","title":"OSWorld project site: verified upgrade","year":2025,"studyKind":"capability_ceiling","classIds":["computer-use","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch","eval-contamination"],"lesson":"The project site says that on 2025-07-28 OSWorld was enhanced as OSWorld-Verified, including fixed community-reported examples and updated benchmark results, and that new runs should be compared with those new results. A 2026-06-26 line says OSWorld 2.0 is available. A score from the 2024 paper is not the same number as a score on the later set.","sourceUrl":"https://os-world.github.io/","asOf":"2026-10-03","notes":"Fetched the project homepage. What the 2.0 task set contains was not copied because the homepage line only says it is available."},{"id":"openai-cua-2025","title":"Computer-Using Agent","year":2025,"studyKind":"vendor_writeup","classIds":["computer-use","browser"],"failureModeIds":["benchmark-protocol-mismatch","gui-grounding-miss"],"lesson":"OpenAI's page says Computer-Using Agent drives a GUI from screenshots and reports 38.1% on OSWorld, 58.1% on WebArena, and 87% on WebVoyager. Those are the vendor's reported scores for that model. They are not a re-run of the original papers' baselines, and the page says the model is early and has limitations.","sourceUrl":"https://openai.com/index/computer-using-agent/","asOf":"2026-10-03","notes":"Fetched the OpenAI page dated January 23, 2025 in the article header. Depot did not re-run OSWorld, WebArena, or WebVoyager."},{"id":"anthropic-computer-use-2024","title":"Developing a computer use model","year":2024,"studyKind":"vendor_writeup","classIds":["computer-use","security-review"],"failureModeIds":["prompt-injection"],"lesson":"Anthropic's post says computer use was in public beta and names prompt injection as a case where instructions in content can override prior directions or cause actions that were not the user's intent. It notes that a model reading screenshots from an internet-connected computer can be exposed to that content. The lesson stored here is to treat screen text as untrusted. The post's attack detail is not copied.","sourceUrl":"https://www.anthropic.com/news/developing-computer-use","asOf":"2026-10-03","notes":"datePublished on the fetched page is 2024-10-22. No procedure is stored."}]}