{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","count":5,"catalogCount":27,"refused":["Replit agent database story (July 2025): The Register URL returned HTTP 403 and no first-party engineering postmortem was fetched. Not indexed.","CanLII HTML for 2024 BCCRT 149 returned a client-side block. The Civil Resolution Tribunal document was fetched instead and is the indexed source.","Cursor agent best-practices post loaded and is a product guide, not a postmortem. Not indexed as an incident. The University of Chicago study it mentions was not fetched.","Cognition 'Introducing Devin' (published 2024-03-12) duplicates the SWE-bench technical report. The report is indexed; the launch post is not a second study.","Anthropic many-shot jailbreaking research page loaded. It is an attack writeup, so it is not indexed and no procedure is stored.","OWASP LLM01:2025 was fetched and is cited on the prompt-injection failure mode only. Attack examples from that page are not copied into a case study.","Calder Lab and similar unsourced production-metric blogs were not fetched and are not indexed.","Google Research abstract for Data Cascades in High-Stakes AI loaded on 2026-10-03 (the abstract says 92% prevalence). Not indexed for data-etl: it is an interview study of ML data practice, not a pipeline job failure. The indexed data-etl study is the dbt Cloud status page."],"studies":[{"id":"gaia-2023","title":"GAIA: a benchmark for General AI Assistants","year":2023,"studyKind":"capability_ceiling","classIds":["research","personal-assistant"],"failureModeIds":["weak-verification"],"lesson":"The abstract says GAIA questions need reasoning, multi-modal handling, web browsing, and tool use, and that they are conceptually simple for people. It reports human respondents at 92% and GPT-4 with plugins at 15%, on a set of 466 questions. That gap is a ceiling for a general assistant, not an incident at a company.","sourceUrl":"https://arxiv.org/abs/2311.12983","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 21 Nov 2023). Answers for 300 questions were held for a leaderboard, so a public question file is not the full scored set."},{"id":"anthropic-building-effective-agents-2024","title":"Building effective agents","year":2024,"studyKind":"vendor_writeup","classIds":["orchestrator","coding","research"],"failureModeIds":["runaway-spend","instruction-drift"],"lesson":"The post separates workflows, where the path is orchestrated in code, from agents, where the model decides the path. It says agentic systems often trade latency and cost for task performance, and that the successful implementations they saw used simple composable patterns. A note on the fetched page says much of the tooling landscape described has changed since December 2024.","sourceUrl":"https://www.anthropic.com/engineering/building-effective-agents","asOf":"2026-10-03","notes":"Fetched page says Published Dec 19, 2024. This is a design note, not an incident with a measured outage."},{"id":"anthropic-multi-agent-research-2025","title":"How we built our multi-agent research system","year":2025,"studyKind":"vendor_writeup","classIds":["research","orchestrator"],"failureModeIds":["instruction-drift","tool-loop","inter-agent-misalignment"],"lesson":"The post describes an orchestrator-worker research system. Watching traces showed agents continuing after they already had enough results, using overly verbose search queries, or selecting the wrong tool. It also says parallel tool use cut research time by up to 90% for complex queries in their system. That 90% figure is Anthropic's report about their Research feature, not a Depot measurement.","sourceUrl":"https://www.anthropic.com/engineering/built-multi-agent-research-system","asOf":"2026-10-03","notes":"Fetched page says Published Jun 13, 2025."},{"id":"the-agent-company-2025","title":"TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks","year":2025,"studyKind":"capability_ceiling","classIds":["coding","browser","research","orchestrator"],"failureModeIds":["weak-verification"],"lesson":"The abstract describes a simulated software company where agents browse, write code, run programs, and communicate with coworkers. It says the most competitive agent they tested completed 30% of tasks autonomously. Read that as a ceiling inside their simulation, not as a measurement of any employer's production team.","sourceUrl":"https://arxiv.org/abs/2412.14161","asOf":"2026-10-03","notes":"Fetched arXiv v3, last revised 10 Sep 2025. Submitted 18 Dec 2024."},{"id":"lost-in-the-middle-2023","title":"Lost in the Middle: How Language Models Use Long Contexts","year":2023,"studyKind":"mechanism_study","classIds":["rag","research"],"failureModeIds":["lost-in-the-middle"],"lesson":"The abstract says that on multi-document question answering and key-value retrieval, performance is often highest when the relevant information is at the beginning or the end of the context, and degrades when it is in the middle, including for models billed as long-context. Putting the best passage in the middle of a prompt is not the same as the model using it.","sourceUrl":"https://arxiv.org/abs/2307.03172","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment: accepted to TACL 2023. Not a vendor incident."}]}