{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","id":"coding","name":"Coding agent","definition":"Edits a repository, runs commands in a workspace, and checks the result against tests or a diff.","needs":{"jtbdIds":["coding-agent","diff-aware-pr-review","agent-sandbox","code-interpreter","agent-ephemeral-ci","persistent-dev-env"],"freeKinds":["free_sandbox","dev_tool","free_docs"],"capabilityTags":["coding","agents"],"plain":["A workspace that can be snapshotted","Tests or a diff check that the agent does not grade itself"]},"failureModeIds":["tool-loop","hallucinated-tool-args","partial-tool-success","collateral-side-effects","weak-verification","eval-contamination","benchmark-protocol-mismatch","runaway-spend","silent-truncation","non-determinism","schema-drift","context-overflow","instruction-drift"],"exampleTasks":["Patch a failing test and run that test in a sandbox","Review a diff without rewriting unrelated files","Stop when the same command repeats with the same output"],"failures":[{"id":"tool-loop","name":"Tool loop","whatGoesWrong":"The agent calls the same tool, or a cycle of tools, without new information, so the task never reaches a stop condition.","howToDetect":"Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.","classIds":["coding","research","browser","computer-use","orchestrator","scheduled-ops"],"sourceUrls":["https://www.anthropic.com/engineering/built-multi-agent-research-system"]},{"id":"hallucinated-tool-args","name":"Hallucinated tool arguments","whatGoesWrong":"The agent calls a tool with an id, path, or field that does not exist.","howToDetect":"Reject a call that fails the current schema or names an unknown id, and do that before the tool runs a side effect.","classIds":["coding","data-etl","browser","orchestrator"],"sourceUrls":["https://arxiv.org/abs/2308.03688"]},{"id":"partial-tool-success","name":"Partial tool success","whatGoesWrong":"A tool completes only part of a batch and the agent treats the whole batch as done.","howToDetect":"Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.","classIds":["coding","data-etl","browser","computer-use","voice","orchestrator","scheduled-ops"],"sourceUrls":[]},{"id":"collateral-side-effects","name":"Collateral side effects","whatGoesWrong":"The stated task ends in the desired state, and something outside the allowlist also changes: an unrelated file, row, or app.","howToDetect":"Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.","classIds":["coding","computer-use","data-etl","browser"],"sourceUrls":["https://arxiv.org/abs/2407.18901"]},{"id":"weak-verification","name":"Weak verification","whatGoesWrong":"The run is marked successful because the agent said it was done, without an execution check, a test, or a second pass.","howToDetect":"Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.","classIds":["coding","eval-judge","orchestrator","support","security-review"],"sourceUrls":["https://arxiv.org/abs/2503.13657","https://arxiv.org/abs/2310.06770"]},{"id":"eval-contamination","name":"Eval contamination","whatGoesWrong":"A score is not comparable to the one it is cited against: the gold label or the issue text leaked into the prompt, the harness rejects a valid fix, or a revised benchmark is treated as the old one.","howToDetect":"Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.","classIds":["eval-judge","coding","rag","security-review"],"sourceUrls":["https://openai.com/index/introducing-swe-bench-verified/","https://os-world.github.io/"]},{"id":"benchmark-protocol-mismatch","name":"Benchmark protocol mismatch","whatGoesWrong":"A score is cited next to another score that used a different setup: assisted versus unassisted editing, a subset versus the full set, or a revised harness versus the original.","howToDetect":"Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.","classIds":["eval-judge","coding","computer-use","browser"],"sourceUrls":["https://www.cognition.ai/blog/swe-bench-technical-report","https://openai.com/index/introducing-swe-bench-verified/","https://os-world.github.io/"]},{"id":"runaway-spend","name":"Runaway spend","whatGoesWrong":"Retries, fan-out, or a loop keep calling paid models or tools after the task should have stopped.","howToDetect":"Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.","classIds":["coding","research","orchestrator","scheduled-ops","sales-outreach","voice"],"sourceUrls":["https://www.anthropic.com/engineering/building-effective-agents"]},{"id":"silent-truncation","name":"Silent truncation","whatGoesWrong":"Tool output or a document is cut off without a marker, and the agent answers as if it saw the whole thing.","howToDetect":"Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.","classIds":["rag","coding","data-etl","voice","local-offline","research"],"sourceUrls":["https://arxiv.org/abs/2310.08560"]},{"id":"non-determinism","name":"Non-determinism","whatGoesWrong":"The same task produces a different tool sequence or a different end state on another trial, so one success does not mean the agent is reliable.","howToDetect":"Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.","classIds":["support","coding","browser","scheduled-ops","eval-judge","voice"],"sourceUrls":["https://arxiv.org/abs/2406.12045"]},{"id":"schema-drift","name":"Schema drift","whatGoesWrong":"A tool, table, or file changed shape, and the agent still sends the old arguments or reads the old columns.","howToDetect":"Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.","classIds":["coding","data-etl","local-offline","support"],"sourceUrls":[]},{"id":"context-overflow","name":"Context overflow","whatGoesWrong":"The document or the session does not fit the context window, and the agent answers from whatever fragment was left in view.","howToDetect":"If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.","classIds":["rag","personal-assistant","research","local-offline","coding"],"sourceUrls":["https://arxiv.org/abs/2310.08560"]},{"id":"instruction-drift","name":"Instruction drift","whatGoesWrong":"Over a long trace the agent stops following the user's rule or the stop condition, including continuing after it already has enough information.","howToDetect":"Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.","classIds":["research","orchestrator","coding","browser","personal-assistant"],"sourceUrls":["https://www.anthropic.com/engineering/built-multi-agent-research-system","https://arxiv.org/abs/2308.03688"]}],"caseStudies":[{"id":"swe-bench-2023","title":"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?","year":2023,"studyKind":"capability_ceiling","classIds":["coding","eval-judge"],"failureModeIds":["weak-verification","context-overflow"],"lesson":"The abstract describes 2,294 GitHub issues from 12 Python repositories. Resolving them often means coordinating edits across functions, classes, and files, and interacting with an execution environment. The abstract says Claude 2 solved 1.96% of the issues. Treat that figure as the paper's reported ceiling at publication, not a live Depot score.","sourceUrl":"https://arxiv.org/abs/2310.06770","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (v3, last revised 11 Nov 2024). Not a company incident. Depot did not re-run the harness."},{"id":"swe-bench-verified-2024","title":"Introducing SWE-bench Verified","year":2024,"studyKind":"vendor_writeup","classIds":["coding","eval-judge"],"failureModeIds":["eval-contamination","benchmark-protocol-mismatch","weak-verification"],"lesson":"OpenAI's page says the original SWE-bench can grade a valid patch as wrong when unit tests are overly specific, the issue text is underspecified, or the environment fails to set up. They released a 500-sample human-validated subset, with difficulty slices of 196 tasks they mark as under 15 minutes and 45 they mark as over an hour. Cite the subset name with any score.","sourceUrl":"https://openai.com/index/introducing-swe-bench-verified/","asOf":"2026-10-03","notes":"Page date on the fetch: August 13, 2024. Model score tables on that page were not copied here."},{"id":"cognition-swe-bench-report-2024","title":"Cognition SWE-bench technical report","year":2024,"studyKind":"vendor_writeup","classIds":["coding","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch"],"lesson":"Cognition's report says Devin resolved 13.86% of issues on a random 25% subset, unassisted, against a previous unassisted baseline of 1.96%, and that the best previous model resolved 4.80% when given the files to edit. The page itself distinguishes assisted from unassisted. Do not cite those percentages as the same protocol.","sourceUrl":"https://www.cognition.ai/blog/swe-bench-technical-report","asOf":"2026-10-03","notes":"Vendor self-report. article:published_time on the fetched page is 2024-03-15. Depot did not re-run the harness. The launch post Introducing Devin was not indexed separately."},{"id":"anthropic-building-effective-agents-2024","title":"Building effective agents","year":2024,"studyKind":"vendor_writeup","classIds":["orchestrator","coding","research"],"failureModeIds":["runaway-spend","instruction-drift"],"lesson":"The post separates workflows, where the path is orchestrated in code, from agents, where the model decides the path. It says agentic systems often trade latency and cost for task performance, and that the successful implementations they saw used simple composable patterns. A note on the fetched page says much of the tooling landscape described has changed since December 2024.","sourceUrl":"https://www.anthropic.com/engineering/building-effective-agents","asOf":"2026-10-03","notes":"Fetched page says Published Dec 19, 2024. This is a design note, not an incident with a measured outage."},{"id":"the-agent-company-2025","title":"TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks","year":2025,"studyKind":"capability_ceiling","classIds":["coding","browser","research","orchestrator"],"failureModeIds":["weak-verification"],"lesson":"The abstract describes a simulated software company where agents browse, write code, run programs, and communicate with coworkers. It says the most competitive agent they tested completed 30% of tasks autonomously. Read that as a ceiling inside their simulation, not as a measurement of any employer's production team.","sourceUrl":"https://arxiv.org/abs/2412.14161","asOf":"2026-10-03","notes":"Fetched arXiv v3, last revised 10 Sep 2025. Submitted 18 Dec 2024."},{"id":"agentbench-2024","title":"AgentBench: Evaluating LLMs as Agents","year":2024,"studyKind":"capability_ceiling","classIds":["coding","eval-judge"],"failureModeIds":["instruction-drift","hallucinated-tool-args","weak-verification"],"lesson":"The abstract says AgentBench uses 8 environments and that the main obstacles they identified were poor long-term reasoning, decision-making, and instruction following. A coding or tool agent that cannot stay on the instruction over many steps fails this kind of eval even if a single call looks fluent. The abstract's model count did not render as a number on the fetched page, so none is stored.","sourceUrl":"https://arxiv.org/abs/2308.03688","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: published in ICLR 2024."},{"id":"appworld-2024","title":"AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents","year":2024,"studyKind":"capability_ceiling","classIds":["coding","personal-assistant"],"failureModeIds":["collateral-side-effects","weak-verification"],"lesson":"The abstract describes 9 apps and 457 APIs, and 750 tasks whose checks allow more than one solution while also flagging unexpected changes. It says GPT-4o solved about 49% of normal tasks and about 30% of challenge tasks. A task that reaches the goal and also mutates something else is still a failed check in this design.","sourceUrl":"https://arxiv.org/abs/2407.18901","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Capability ceiling, not a company incident."}]}