{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","count":3,"catalogCount":27,"refused":["Replit agent database story (July 2025): The Register URL returned HTTP 403 and no first-party engineering postmortem was fetched. Not indexed.","CanLII HTML for 2024 BCCRT 149 returned a client-side block. The Civil Resolution Tribunal document was fetched instead and is the indexed source.","Cursor agent best-practices post loaded and is a product guide, not a postmortem. Not indexed as an incident. The University of Chicago study it mentions was not fetched.","Cognition 'Introducing Devin' (published 2024-03-12) duplicates the SWE-bench technical report. The report is indexed; the launch post is not a second study.","Anthropic many-shot jailbreaking research page loaded. It is an attack writeup, so it is not indexed and no procedure is stored.","OWASP LLM01:2025 was fetched and is cited on the prompt-injection failure mode only. Attack examples from that page are not copied into a case study.","Calder Lab and similar unsourced production-metric blogs were not fetched and are not indexed.","Google Research abstract for Data Cascades in High-Stakes AI loaded on 2026-10-03 (the abstract says 92% prevalence). Not indexed for data-etl: it is an interview study of ML data practice, not a pipeline job failure. The indexed data-etl study is the dbt Cloud status page."],"studies":[{"id":"visualwebarena-2024","title":"VisualWebArena","year":2024,"studyKind":"capability_ceiling","classIds":["browser","computer-use"],"failureModeIds":["gui-grounding-miss"],"lesson":"The abstract says many web tasks need visual information that text-only agents do not use well, and that the benchmark is visually grounded. A browser agent that reads only the DOM or the HTML is being measured on a different task than this one. No success rate from the abstract is repeated here because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2401.13649","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: accepted to ACL 2024."},{"id":"osworld-2024","title":"OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments","year":2024,"studyKind":"capability_ceiling","classIds":["computer-use"],"failureModeIds":["gui-grounding-miss","weak-verification"],"lesson":"The abstract describes 369 real computer tasks across operating systems and apps, with execution-based checks. It says humans accomplish over 72.36% and the best model 12.24%, and that models struggle with GUI grounding and operational knowledge. This is a computer-use ceiling from the paper, not a production outage.","sourceUrl":"https://arxiv.org/abs/2404.07972","asOf":"2026-10-03","notes":"Fetched arXiv v2, last revised 30 May 2024. Later site notes about OSWorld-Verified are a separate study."},{"id":"openai-cua-2025","title":"Computer-Using Agent","year":2025,"studyKind":"vendor_writeup","classIds":["computer-use","browser"],"failureModeIds":["benchmark-protocol-mismatch","gui-grounding-miss"],"lesson":"OpenAI's page says Computer-Using Agent drives a GUI from screenshots and reports 38.1% on OSWorld, 58.1% on WebArena, and 87% on WebVoyager. Those are the vendor's reported scores for that model. They are not a re-run of the original papers' baselines, and the page says the model is early and has limitations.","sourceUrl":"https://openai.com/index/computer-using-agent/","asOf":"2026-10-03","notes":"Fetched the OpenAI page dated January 23, 2025 in the article header. Depot did not re-run OSWorld, WebArena, or WebVoyager."}]}