{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","count":27,"catalogCount":27,"refused":["Replit agent database story (July 2025): The Register URL returned HTTP 403 and no first-party engineering postmortem was fetched. Not indexed.","CanLII HTML for 2024 BCCRT 149 returned a client-side block. The Civil Resolution Tribunal document was fetched instead and is the indexed source.","Cursor agent best-practices post loaded and is a product guide, not a postmortem. Not indexed as an incident. The University of Chicago study it mentions was not fetched.","Cognition 'Introducing Devin' (published 2024-03-12) duplicates the SWE-bench technical report. The report is indexed; the launch post is not a second study.","Anthropic many-shot jailbreaking research page loaded. It is an attack writeup, so it is not indexed and no procedure is stored.","OWASP LLM01:2025 was fetched and is cited on the prompt-injection failure mode only. Attack examples from that page are not copied into a case study.","Calder Lab and similar unsourced production-metric blogs were not fetched and are not indexed.","Google Research abstract for Data Cascades in High-Stakes AI loaded on 2026-10-03 (the abstract says 92% prevalence). Not indexed for data-etl: it is an interview study of ML data practice, not a pipeline job failure. The indexed data-etl study is the dbt Cloud status page."],"studies":[{"id":"swe-bench-2023","title":"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?","year":2023,"studyKind":"capability_ceiling","classIds":["coding","eval-judge"],"failureModeIds":["weak-verification","context-overflow"],"lesson":"The abstract describes 2,294 GitHub issues from 12 Python repositories. Resolving them often means coordinating edits across functions, classes, and files, and interacting with an execution environment. The abstract says Claude 2 solved 1.96% of the issues. Treat that figure as the paper's reported ceiling at publication, not a live Depot score.","sourceUrl":"https://arxiv.org/abs/2310.06770","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (v3, last revised 11 Nov 2024). Not a company incident. Depot did not re-run the harness."},{"id":"swe-bench-verified-2024","title":"Introducing SWE-bench Verified","year":2024,"studyKind":"vendor_writeup","classIds":["coding","eval-judge"],"failureModeIds":["eval-contamination","benchmark-protocol-mismatch","weak-verification"],"lesson":"OpenAI's page says the original SWE-bench can grade a valid patch as wrong when unit tests are overly specific, the issue text is underspecified, or the environment fails to set up. They released a 500-sample human-validated subset, with difficulty slices of 196 tasks they mark as under 15 minutes and 45 they mark as over an hour. Cite the subset name with any score.","sourceUrl":"https://openai.com/index/introducing-swe-bench-verified/","asOf":"2026-10-03","notes":"Page date on the fetch: August 13, 2024. Model score tables on that page were not copied here."},{"id":"cognition-swe-bench-report-2024","title":"Cognition SWE-bench technical report","year":2024,"studyKind":"vendor_writeup","classIds":["coding","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch"],"lesson":"Cognition's report says Devin resolved 13.86% of issues on a random 25% subset, unassisted, against a previous unassisted baseline of 1.96%, and that the best previous model resolved 4.80% when given the files to edit. The page itself distinguishes assisted from unassisted. Do not cite those percentages as the same protocol.","sourceUrl":"https://www.cognition.ai/blog/swe-bench-technical-report","asOf":"2026-10-03","notes":"Vendor self-report. article:published_time on the fetched page is 2024-03-15. Depot did not re-run the harness. The launch post Introducing Devin was not indexed separately."},{"id":"gaia-2023","title":"GAIA: a benchmark for General AI Assistants","year":2023,"studyKind":"capability_ceiling","classIds":["research","personal-assistant"],"failureModeIds":["weak-verification"],"lesson":"The abstract says GAIA questions need reasoning, multi-modal handling, web browsing, and tool use, and that they are conceptually simple for people. It reports human respondents at 92% and GPT-4 with plugins at 15%, on a set of 466 questions. That gap is a ceiling for a general assistant, not an incident at a company.","sourceUrl":"https://arxiv.org/abs/2311.12983","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 21 Nov 2023). Answers for 300 questions were held for a leaderboard, so a public question file is not the full scored set."},{"id":"webarena-2024","title":"WebArena: A Realistic Web Environment for Building Autonomous Agents","year":2024,"studyKind":"capability_ceiling","classIds":["browser"],"failureModeIds":["weak-verification","instruction-drift"],"lesson":"The abstract describes functional sites for e-commerce, forums, collaborative software development, and content management, plus long-horizon tasks. It reports a best GPT-4-based agent at 14.41% end-to-end success and human performance at 78.24%. Use it as a web-agent ceiling, not as evidence about a named vendor's production site.","sourceUrl":"https://arxiv.org/abs/2307.13854","asOf":"2026-10-03","notes":"Fetched arXiv v4, last revised 16 Apr 2024. Submitted 25 Jul 2023."},{"id":"visualwebarena-2024","title":"VisualWebArena","year":2024,"studyKind":"capability_ceiling","classIds":["browser","computer-use"],"failureModeIds":["gui-grounding-miss"],"lesson":"The abstract says many web tasks need visual information that text-only agents do not use well, and that the benchmark is visually grounded. A browser agent that reads only the DOM or the HTML is being measured on a different task than this one. No success rate from the abstract is repeated here because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2401.13649","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: accepted to ACL 2024."},{"id":"osworld-2024","title":"OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments","year":2024,"studyKind":"capability_ceiling","classIds":["computer-use"],"failureModeIds":["gui-grounding-miss","weak-verification"],"lesson":"The abstract describes 369 real computer tasks across operating systems and apps, with execution-based checks. It says humans accomplish over 72.36% and the best model 12.24%, and that models struggle with GUI grounding and operational knowledge. This is a computer-use ceiling from the paper, not a production outage.","sourceUrl":"https://arxiv.org/abs/2404.07972","asOf":"2026-10-03","notes":"Fetched arXiv v2, last revised 30 May 2024. Later site notes about OSWorld-Verified are a separate study."},{"id":"osworld-verified-site-2025","title":"OSWorld project site: verified upgrade","year":2025,"studyKind":"capability_ceiling","classIds":["computer-use","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch","eval-contamination"],"lesson":"The project site says that on 2025-07-28 OSWorld was enhanced as OSWorld-Verified, including fixed community-reported examples and updated benchmark results, and that new runs should be compared with those new results. A 2026-06-26 line says OSWorld 2.0 is available. A score from the 2024 paper is not the same number as a score on the later set.","sourceUrl":"https://os-world.github.io/","asOf":"2026-10-03","notes":"Fetched the project homepage. What the 2.0 task set contains was not copied because the homepage line only says it is available."},{"id":"openai-cua-2025","title":"Computer-Using Agent","year":2025,"studyKind":"vendor_writeup","classIds":["computer-use","browser"],"failureModeIds":["benchmark-protocol-mismatch","gui-grounding-miss"],"lesson":"OpenAI's page says Computer-Using Agent drives a GUI from screenshots and reports 38.1% on OSWorld, 58.1% on WebArena, and 87% on WebVoyager. Those are the vendor's reported scores for that model. They are not a re-run of the original papers' baselines, and the page says the model is early and has limitations.","sourceUrl":"https://openai.com/index/computer-using-agent/","asOf":"2026-10-03","notes":"Fetched the OpenAI page dated January 23, 2025 in the article header. Depot did not re-run OSWorld, WebArena, or WebVoyager."},{"id":"openai-operator-2025","title":"Introducing Operator","year":2025,"studyKind":"vendor_writeup","classIds":["browser","personal-assistant"],"failureModeIds":["confirmation-skip","over-refusal"],"lesson":"The page describes a research-preview browser agent and says that before a significant action, such as submitting an order or sending an email, Operator should ask for approval. It also says the model is trained to decline some sensitive tasks, including banking transactions. A browser agent that submits or sends with no approval id is outside the behavior this writeup describes.","sourceUrl":"https://openai.com/index/introducing-operator/","asOf":"2026-10-03","notes":"Fetched page. A July 17, 2025 update on the same page says Operator was integrated into ChatGPT agent and that the standalone site would sunset. That update is a product change, not an incident report."},{"id":"anthropic-computer-use-2024","title":"Developing a computer use model","year":2024,"studyKind":"vendor_writeup","classIds":["computer-use","security-review"],"failureModeIds":["prompt-injection"],"lesson":"Anthropic's post says computer use was in public beta and names prompt injection as a case where instructions in content can override prior directions or cause actions that were not the user's intent. It notes that a model reading screenshots from an internet-connected computer can be exposed to that content. The lesson stored here is to treat screen text as untrusted. The post's attack detail is not copied.","sourceUrl":"https://www.anthropic.com/news/developing-computer-use","asOf":"2026-10-03","notes":"datePublished on the fetched page is 2024-10-22. No procedure is stored."},{"id":"anthropic-building-effective-agents-2024","title":"Building effective agents","year":2024,"studyKind":"vendor_writeup","classIds":["orchestrator","coding","research"],"failureModeIds":["runaway-spend","instruction-drift"],"lesson":"The post separates workflows, where the path is orchestrated in code, from agents, where the model decides the path. It says agentic systems often trade latency and cost for task performance, and that the successful implementations they saw used simple composable patterns. A note on the fetched page says much of the tooling landscape described has changed since December 2024.","sourceUrl":"https://www.anthropic.com/engineering/building-effective-agents","asOf":"2026-10-03","notes":"Fetched page says Published Dec 19, 2024. This is a design note, not an incident with a measured outage."},{"id":"anthropic-multi-agent-research-2025","title":"How we built our multi-agent research system","year":2025,"studyKind":"vendor_writeup","classIds":["research","orchestrator"],"failureModeIds":["instruction-drift","tool-loop","inter-agent-misalignment"],"lesson":"The post describes an orchestrator-worker research system. Watching traces showed agents continuing after they already had enough results, using overly verbose search queries, or selecting the wrong tool. It also says parallel tool use cut research time by up to 90% for complex queries in their system. That 90% figure is Anthropic's report about their Research feature, not a Depot measurement.","sourceUrl":"https://www.anthropic.com/engineering/built-multi-agent-research-system","asOf":"2026-10-03","notes":"Fetched page says Published Jun 13, 2025."},{"id":"tau-bench-2024","title":"τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains","year":2024,"studyKind":"capability_ceiling","classIds":["support","voice"],"failureModeIds":["non-determinism","policy-conflict","weak-verification"],"lesson":"The abstract says τ-bench checks whether an agent follows domain rules while talking to a simulated user, and grades the database at the end. It reports that even a function-calling agent such as gpt-4o succeeded on under 50% of tasks and was inconsistent, with pass^8 under 25% in the retail domain. One green trial is not reliability.","sourceUrl":"https://arxiv.org/abs/2406.12045","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 17 Jun 2024). Capability ceiling for policy-following tool use, not a named company's outage."},{"id":"the-agent-company-2025","title":"TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks","year":2025,"studyKind":"capability_ceiling","classIds":["coding","browser","research","orchestrator"],"failureModeIds":["weak-verification"],"lesson":"The abstract describes a simulated software company where agents browse, write code, run programs, and communicate with coworkers. It says the most competitive agent they tested completed 30% of tasks autonomously. Read that as a ceiling inside their simulation, not as a measurement of any employer's production team.","sourceUrl":"https://arxiv.org/abs/2412.14161","asOf":"2026-10-03","notes":"Fetched arXiv v3, last revised 10 Sep 2025. Submitted 18 Dec 2024."},{"id":"agentbench-2024","title":"AgentBench: Evaluating LLMs as Agents","year":2024,"studyKind":"capability_ceiling","classIds":["coding","eval-judge"],"failureModeIds":["instruction-drift","hallucinated-tool-args","weak-verification"],"lesson":"The abstract says AgentBench uses 8 environments and that the main obstacles they identified were poor long-term reasoning, decision-making, and instruction following. A coding or tool agent that cannot stay on the instruction over many steps fails this kind of eval even if a single call looks fluent. The abstract's model count did not render as a number on the fetched page, so none is stored.","sourceUrl":"https://arxiv.org/abs/2308.03688","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: published in ICLR 2024."},{"id":"lost-in-the-middle-2023","title":"Lost in the Middle: How Language Models Use Long Contexts","year":2023,"studyKind":"mechanism_study","classIds":["rag","research"],"failureModeIds":["lost-in-the-middle"],"lesson":"The abstract says that on multi-document question answering and key-value retrieval, performance is often highest when the relevant information is at the beginning or the end of the context, and degrades when it is in the middle, including for models billed as long-context. Putting the best passage in the middle of a prompt is not the same as the model using it.","sourceUrl":"https://arxiv.org/abs/2307.03172","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment: accepted to TACL 2023. Not a vendor incident."},{"id":"mast-multi-agent-failures-2025","title":"Why Do Multi-Agent LLM Systems Fail?","year":2025,"studyKind":"mechanism_study","classIds":["orchestrator","eval-judge"],"failureModeIds":["inter-agent-misalignment","weak-verification"],"lesson":"The abstract says performance gains of multi-agent systems on popular benchmarks are often minimal. Their taxonomy, built from 150 traces with annotator kappa of 0.88, groups failures into system design, inter-agent misalignment, and task verification, and identifies 14 modes. The larger MAST-Data collection is described as 1600+ annotated traces. This is a failure taxonomy, not a single production outage.","sourceUrl":"https://arxiv.org/abs/2503.13657","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 2025). The 14 mode names are not copied because the abstract did not list them."},{"id":"appworld-2024","title":"AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents","year":2024,"studyKind":"capability_ceiling","classIds":["coding","personal-assistant"],"failureModeIds":["collateral-side-effects","weak-verification"],"lesson":"The abstract describes 9 apps and 457 APIs, and 750 tasks whose checks allow more than one solution while also flagging unexpected changes. It says GPT-4o solved about 49% of normal tasks and about 30% of challenge tasks. A task that reaches the goal and also mutates something else is still a failed check in this design.","sourceUrl":"https://arxiv.org/abs/2407.18901","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Capability ceiling, not a company incident."},{"id":"workarena-2024","title":"WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?","year":2024,"studyKind":"capability_ceiling","classIds":["browser","support"],"failureModeIds":["weak-verification"],"lesson":"The abstract introduces WorkArena as 33 tasks on ServiceNow and BrowserGym as the environment. It says agents showed promise and that a considerable gap remained before full task automation, with a disparity between open and closed models. No success percentage is stored because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2403.07718","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 12 Mar 2024). Enterprise knowledge-work ceiling, not a ServiceNow incident."},{"id":"memgpt-2023","title":"MemGPT: Towards LLMs as Operating Systems","year":2023,"studyKind":"mechanism_study","classIds":["personal-assistant","rag","local-offline"],"failureModeIds":["context-overflow","silent-truncation"],"lesson":"The abstract says limited context windows handicap document analysis and multi-session chat, and describes moving text between tiers so the model sees a working set rather than the whole history. An assistant that answers a long document or a long memory from a single truncated window is the problem this paper is about. No accuracy number is stored because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2310.08560","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. A system paper, not an incident."},{"id":"moffatt-v-air-canada-2024","title":"Moffatt v. Air Canada, 2024 BCCRT 149","year":2024,"studyKind":"tribunal_decision","classIds":["support"],"failureModeIds":["policy-conflict","identity-confusion","ungrounded-answer","stale-retrieval"],"lesson":"The decision says a customer used Air Canada's website chatbot while looking up a bereavement fare. The chatbot said a reduced rate could be requested within 90 days, including after travel. The linked bereavement page said the policy did not apply after travel was complete. An Air Canada representative later called the chatbot's words misleading. The tribunal found negligent misrepresentation, rejected the argument that the chatbot was a separate legal entity, and ordered Air Canada to pay $812.02. A support agent must not choose between two of the operator's pages, and the operator owns the answer.","sourceUrl":"https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do","asOf":"2026-10-03","notes":"Fetched the tribunal document (PDF) dated February 14, 2024, Tribunal Member Christopher C. Rivers. The order is $650.88 damages, $36.14 pre-judgment interest, and $125 in CRT fees. CanLII's HTML page returned a block and was not used."},{"id":"langchain-chat-rebuild-2025","title":"Why we rebuilt LangChain's chatbot","year":2025,"studyKind":"vendor_writeup","classIds":["support","rag"],"failureModeIds":["ungrounded-answer","stale-retrieval"],"lesson":"LangChain's post says support engineers were not using their own docs chatbot because a docs-only answer was not enough for the questions they actually had, and that citations and context were fragmented while the index had to be rebuilt as docs changed. The rebuild described separate paths for ordinary questions and harder ones, plus middleware for guardrails, retries, model fallback, and caching. No latency or accuracy figure is stored here.","sourceUrl":"https://www.langchain.com/blog/rebuilding-chat-langchain","asOf":"2026-10-03","notes":"Fetched page byline: The LangChain Team, November 5, 2025. Descriptive of their rebuild, not an audited outage."},{"id":"langsmith-cert-incident-2025","title":"LangSmith incident on May 1, 2025","year":2025,"studyKind":"production_incident","classIds":["eval-judge","scheduled-ops"],"failureModeIds":["observability-gap"],"lesson":"LangChain's incident note says the US LangSmith API had an elevated error rate for 28 minutes on May 1, 2025, from 14:35 to 15:03 UTC, and that about 55% of API requests failed, including run ingestion. The cause they describe is a leftover DNS record that stopped certificate renewal until the certificate expired. Agent traces can disappear because the trace host's certificate failed, not because the agent reasoned badly.","sourceUrl":"https://www.langchain.com/blog/langsmith-incident-on-may-1-2025","asOf":"2026-10-03","notes":"Fetched LangChain's own incident writeup. This is an observability outage, not a model-behavior incident."},{"id":"willison-prompt-injection-2023","title":"Prompt injection: What's the worst that can happen?","year":2023,"studyKind":"mechanism_study","classIds":["personal-assistant","browser","security-review"],"failureModeIds":["prompt-injection","confirmation-skip"],"lesson":"The post explains that a tool-using assistant which reads untrusted text, such as email or a web page, can be driven to take an action the user did not ask for, including sending mail or reaching data through a connected tool. The lesson for an assistant is to treat fetched text as untrusted and to require confirmation before a send or an export. Attack sketches from the post are not stored.","sourceUrl":"https://simonwillison.net/2023/Apr/14/worst-that-can-happen/","asOf":"2026-10-03","notes":"Fetched the 14 April 2023 post. Hypotheticals in the post are not reproduced and are not marked as incidents that Depot verified."},{"id":"dbt-cloud-databricks-adapter-arg-2025","title":"dbt Cloud scheduled jobs failed on an unexpected Databricks adapter argument","year":2025,"studyKind":"production_incident","classIds":["data-etl"],"failureModeIds":["schema-drift"],"lesson":"dbt Cloud's status page for EMEA Cell 1 Azure says scheduled jobs for Azure Databricks customers failed with an AdapterGetRelationParams error that named an unexpected argument, needs_information. The page marks the incident identified on Tue, Sep 16, 2025, 05:03 PM, a temporary fix deployed by 05:19 PM the same day, and resolved on Fri, Sep 19, 2025, 04:28 PM. It calls the impact a partial outage of Scheduled Jobs and does not say which component added the argument or name a timezone. A pipeline run that errors on an unexpected argument has not finished the load. Do not mark the job successful because a later update says resolved.","sourceUrl":"https://status.getdbt.com/emea-cell-1-azure/incidents/01K59QNKF7W44ER40KRH1RQJFZ","asOf":"2026-10-03","notes":"Fetched the vendor status page on 2026-10-03. The page does not describe an agent. It does not state a root cause beyond the error string."},{"id":"ftc-verkada-can-spam-2024","title":"FTC press release: Verkada commercial email and the CAN-SPAM Act","year":2024,"studyKind":"regulatory_writeup","classIds":["sales-outreach"],"failureModeIds":["confirmation-skip"],"lesson":"The FTC's August 30, 2024 press release says a Department of Justice complaint, filed on referral from the FTC, alleged that Verkada violated the CAN-SPAM Act by flooding prospective customers with commercial emails and failing to include an unsubscribe or opt-out, honor opt-out requests, and provide a physical postal address. The release says the complaint alleges more than 30 million commercial emails over three years, and that those emails violated the Act in four ways, including not honoring unsubscribe requests. It does not name the other three ways in that sentence. It says a proposed order, which a federal judge still had to approve, would require a $2.95 million penalty, called the largest penalty the FTC had obtained for a CAN-SPAM violation, and would prohibit further violations of the Act. The release states allegations and a proposed order. It does not say an outreach agent sent the mail. A sender that omits an opt-out or ignores one has not completed a lawful commercial send under the allegations on this page.","sourceUrl":"https://www.ftc.gov/news-events/news/press-releases/2024/08/ftc-takes-action-against-security-camera-firm-verkada-over-charges-it-failed-secure-videos-other","asOf":"2026-10-03","notes":"Fetched the FTC press release dated August 30, 2024. The same page also alleges security failures and undisclosed reviews. Those allegations are not the outreach lesson. The signed court order was not the source used here, so this row does not say the order was entered."}]}