{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","count":1,"catalogCount":27,"refused":["Replit agent database story (July 2025): The Register URL returned HTTP 403 and no first-party engineering postmortem was fetched. Not indexed.","CanLII HTML for 2024 BCCRT 149 returned a client-side block. The Civil Resolution Tribunal document was fetched instead and is the indexed source.","Cursor agent best-practices post loaded and is a product guide, not a postmortem. Not indexed as an incident. The University of Chicago study it mentions was not fetched.","Cognition 'Introducing Devin' (published 2024-03-12) duplicates the SWE-bench technical report. The report is indexed; the launch post is not a second study.","Anthropic many-shot jailbreaking research page loaded. It is an attack writeup, so it is not indexed and no procedure is stored.","OWASP LLM01:2025 was fetched and is cited on the prompt-injection failure mode only. Attack examples from that page are not copied into a case study.","Calder Lab and similar unsourced production-metric blogs were not fetched and are not indexed.","Google Research abstract for Data Cascades in High-Stakes AI loaded on 2026-10-03 (the abstract says 92% prevalence). Not indexed for data-etl: it is an interview study of ML data practice, not a pipeline job failure. The indexed data-etl study is the dbt Cloud status page."],"studies":[{"id":"agentbench-2024","title":"AgentBench: Evaluating LLMs as Agents","year":2024,"studyKind":"capability_ceiling","classIds":["coding","eval-judge"],"failureModeIds":["instruction-drift","hallucinated-tool-args","weak-verification"],"lesson":"The abstract says AgentBench uses 8 environments and that the main obstacles they identified were poor long-term reasoning, decision-making, and instruction following. A coding or tool agent that cannot stay on the instruction over many steps fails this kind of eval even if a single call looks fluent. The abstract's model count did not render as a number on the fetched page, so none is stored.","sourceUrl":"https://arxiv.org/abs/2308.03688","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: published in ICLR 2024."}]}