{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","count":7,"catalogCount":27,"refused":["Replit agent database story (July 2025): The Register URL returned HTTP 403 and no first-party engineering postmortem was fetched. Not indexed.","CanLII HTML for 2024 BCCRT 149 returned a client-side block. The Civil Resolution Tribunal document was fetched instead and is the indexed source.","Cursor agent best-practices post loaded and is a product guide, not a postmortem. Not indexed as an incident. The University of Chicago study it mentions was not fetched.","Cognition 'Introducing Devin' (published 2024-03-12) duplicates the SWE-bench technical report. The report is indexed; the launch post is not a second study.","Anthropic many-shot jailbreaking research page loaded. It is an attack writeup, so it is not indexed and no procedure is stored.","OWASP LLM01:2025 was fetched and is cited on the prompt-injection failure mode only. Attack examples from that page are not copied into a case study.","Calder Lab and similar unsourced production-metric blogs were not fetched and are not indexed.","Google Research abstract for Data Cascades in High-Stakes AI loaded on 2026-10-03 (the abstract says 92% prevalence). Not indexed for data-etl: it is an interview study of ML data practice, not a pipeline job failure. The indexed data-etl study is the dbt Cloud status page."],"studies":[{"id":"webarena-2024","title":"WebArena: A Realistic Web Environment for Building Autonomous Agents","year":2024,"studyKind":"capability_ceiling","classIds":["browser"],"failureModeIds":["weak-verification","instruction-drift"],"lesson":"The abstract describes functional sites for e-commerce, forums, collaborative software development, and content management, plus long-horizon tasks. It reports a best GPT-4-based agent at 14.41% end-to-end success and human performance at 78.24%. Use it as a web-agent ceiling, not as evidence about a named vendor's production site.","sourceUrl":"https://arxiv.org/abs/2307.13854","asOf":"2026-10-03","notes":"Fetched arXiv v4, last revised 16 Apr 2024. Submitted 25 Jul 2023."},{"id":"visualwebarena-2024","title":"VisualWebArena","year":2024,"studyKind":"capability_ceiling","classIds":["browser","computer-use"],"failureModeIds":["gui-grounding-miss"],"lesson":"The abstract says many web tasks need visual information that text-only agents do not use well, and that the benchmark is visually grounded. A browser agent that reads only the DOM or the HTML is being measured on a different task than this one. No success rate from the abstract is repeated here because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2401.13649","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: accepted to ACL 2024."},{"id":"openai-cua-2025","title":"Computer-Using Agent","year":2025,"studyKind":"vendor_writeup","classIds":["computer-use","browser"],"failureModeIds":["benchmark-protocol-mismatch","gui-grounding-miss"],"lesson":"OpenAI's page says Computer-Using Agent drives a GUI from screenshots and reports 38.1% on OSWorld, 58.1% on WebArena, and 87% on WebVoyager. Those are the vendor's reported scores for that model. They are not a re-run of the original papers' baselines, and the page says the model is early and has limitations.","sourceUrl":"https://openai.com/index/computer-using-agent/","asOf":"2026-10-03","notes":"Fetched the OpenAI page dated January 23, 2025 in the article header. Depot did not re-run OSWorld, WebArena, or WebVoyager."},{"id":"openai-operator-2025","title":"Introducing Operator","year":2025,"studyKind":"vendor_writeup","classIds":["browser","personal-assistant"],"failureModeIds":["confirmation-skip","over-refusal"],"lesson":"The page describes a research-preview browser agent and says that before a significant action, such as submitting an order or sending an email, Operator should ask for approval. It also says the model is trained to decline some sensitive tasks, including banking transactions. A browser agent that submits or sends with no approval id is outside the behavior this writeup describes.","sourceUrl":"https://openai.com/index/introducing-operator/","asOf":"2026-10-03","notes":"Fetched page. A July 17, 2025 update on the same page says Operator was integrated into ChatGPT agent and that the standalone site would sunset. That update is a product change, not an incident report."},{"id":"the-agent-company-2025","title":"TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks","year":2025,"studyKind":"capability_ceiling","classIds":["coding","browser","research","orchestrator"],"failureModeIds":["weak-verification"],"lesson":"The abstract describes a simulated software company where agents browse, write code, run programs, and communicate with coworkers. It says the most competitive agent they tested completed 30% of tasks autonomously. Read that as a ceiling inside their simulation, not as a measurement of any employer's production team.","sourceUrl":"https://arxiv.org/abs/2412.14161","asOf":"2026-10-03","notes":"Fetched arXiv v3, last revised 10 Sep 2025. Submitted 18 Dec 2024."},{"id":"workarena-2024","title":"WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?","year":2024,"studyKind":"capability_ceiling","classIds":["browser","support"],"failureModeIds":["weak-verification"],"lesson":"The abstract introduces WorkArena as 33 tasks on ServiceNow and BrowserGym as the environment. It says agents showed promise and that a considerable gap remained before full task automation, with a disparity between open and closed models. No success percentage is stored because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2403.07718","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 12 Mar 2024). Enterprise knowledge-work ceiling, not a ServiceNow incident."},{"id":"willison-prompt-injection-2023","title":"Prompt injection: What's the worst that can happen?","year":2023,"studyKind":"mechanism_study","classIds":["personal-assistant","browser","security-review"],"failureModeIds":["prompt-injection","confirmation-skip"],"lesson":"The post explains that a tool-using assistant which reads untrusted text, such as email or a web page, can be driven to take an action the user did not ask for, including sending mail or reaching data through a connected tool. The lesson for an assistant is to treat fetched text as untrusted and to require confirmation before a send or an export. Attack sketches from the post are not stored.","sourceUrl":"https://simonwillison.net/2023/Apr/14/worst-that-can-happen/","asOf":"2026-10-03","notes":"Fetched the 14 April 2023 post. Hypotheticals in the post are not reproduced and are not marked as incidents that Depot verified."}]}