{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","id":"eval-judge","name":"Eval and judge agent","definition":"Scores a trace or a patch against a rubric, without the gold label in its prompt, and records the harness version.","needs":{"jtbdIds":["evals","oss-evals","rag-evals","production-guardrail-evals","continuous-eval-ci","adversarial-red-team-evals","synthetic-eval-data","llm-tracing"],"freeKinds":["free_data","dev_tool"],"capabilityTags":["evals","tracing"],"plain":["Gold labels kept out of the judge prompt","Harness version stored next to the score"]},"failureModeIds":["eval-contamination","non-determinism","weak-verification","observability-gap","benchmark-protocol-mismatch","inter-agent-misalignment"],"exampleTasks":["Score a trace against a rubric that does not include the gold answer","Refuse to rank two runs that used different harness versions"],"failures":[{"id":"eval-contamination","name":"Eval contamination","whatGoesWrong":"A score is not comparable to the one it is cited against: the gold label or the issue text leaked into the prompt, the harness rejects a valid fix, or a revised benchmark is treated as the old one.","howToDetect":"Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.","classIds":["eval-judge","coding","rag","security-review"],"sourceUrls":["https://openai.com/index/introducing-swe-bench-verified/","https://os-world.github.io/"]},{"id":"non-determinism","name":"Non-determinism","whatGoesWrong":"The same task produces a different tool sequence or a different end state on another trial, so one success does not mean the agent is reliable.","howToDetect":"Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.","classIds":["support","coding","browser","scheduled-ops","eval-judge","voice"],"sourceUrls":["https://arxiv.org/abs/2406.12045"]},{"id":"weak-verification","name":"Weak verification","whatGoesWrong":"The run is marked successful because the agent said it was done, without an execution check, a test, or a second pass.","howToDetect":"Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.","classIds":["coding","eval-judge","orchestrator","support","security-review"],"sourceUrls":["https://arxiv.org/abs/2503.13657","https://arxiv.org/abs/2310.06770"]},{"id":"observability-gap","name":"Observability gap","whatGoesWrong":"Traces or run ingestion stop, and the failure is outside the agent's reasoning. A certificate or DNS fault can make the trace API fail while agents keep acting.","howToDetect":"Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.","classIds":["eval-judge","scheduled-ops","orchestrator"],"sourceUrls":["https://www.langchain.com/blog/langsmith-incident-on-may-1-2025"]},{"id":"benchmark-protocol-mismatch","name":"Benchmark protocol mismatch","whatGoesWrong":"A score is cited next to another score that used a different setup: assisted versus unassisted editing, a subset versus the full set, or a revised harness versus the original.","howToDetect":"Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.","classIds":["eval-judge","coding","computer-use","browser"],"sourceUrls":["https://www.cognition.ai/blog/swe-bench-technical-report","https://openai.com/index/introducing-swe-bench-verified/","https://os-world.github.io/"]},{"id":"inter-agent-misalignment","name":"Inter-agent misalignment","whatGoesWrong":"A worker follows a different task than the one the lead assigned, or the lead merges results that do not answer the assignment.","howToDetect":"Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.","classIds":["orchestrator","research","eval-judge"],"sourceUrls":["https://arxiv.org/abs/2503.13657"]}],"caseStudies":[{"id":"swe-bench-2023","title":"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?","year":2023,"studyKind":"capability_ceiling","classIds":["coding","eval-judge"],"failureModeIds":["weak-verification","context-overflow"],"lesson":"The abstract describes 2,294 GitHub issues from 12 Python repositories. Resolving them often means coordinating edits across functions, classes, and files, and interacting with an execution environment. The abstract says Claude 2 solved 1.96% of the issues. Treat that figure as the paper's reported ceiling at publication, not a live Depot score.","sourceUrl":"https://arxiv.org/abs/2310.06770","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (v3, last revised 11 Nov 2024). Not a company incident. Depot did not re-run the harness."},{"id":"swe-bench-verified-2024","title":"Introducing SWE-bench Verified","year":2024,"studyKind":"vendor_writeup","classIds":["coding","eval-judge"],"failureModeIds":["eval-contamination","benchmark-protocol-mismatch","weak-verification"],"lesson":"OpenAI's page says the original SWE-bench can grade a valid patch as wrong when unit tests are overly specific, the issue text is underspecified, or the environment fails to set up. They released a 500-sample human-validated subset, with difficulty slices of 196 tasks they mark as under 15 minutes and 45 they mark as over an hour. Cite the subset name with any score.","sourceUrl":"https://openai.com/index/introducing-swe-bench-verified/","asOf":"2026-10-03","notes":"Page date on the fetch: August 13, 2024. Model score tables on that page were not copied here."},{"id":"cognition-swe-bench-report-2024","title":"Cognition SWE-bench technical report","year":2024,"studyKind":"vendor_writeup","classIds":["coding","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch"],"lesson":"Cognition's report says Devin resolved 13.86% of issues on a random 25% subset, unassisted, against a previous unassisted baseline of 1.96%, and that the best previous model resolved 4.80% when given the files to edit. The page itself distinguishes assisted from unassisted. Do not cite those percentages as the same protocol.","sourceUrl":"https://www.cognition.ai/blog/swe-bench-technical-report","asOf":"2026-10-03","notes":"Vendor self-report. article:published_time on the fetched page is 2024-03-15. Depot did not re-run the harness. The launch post Introducing Devin was not indexed separately."},{"id":"osworld-verified-site-2025","title":"OSWorld project site: verified upgrade","year":2025,"studyKind":"capability_ceiling","classIds":["computer-use","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch","eval-contamination"],"lesson":"The project site says that on 2025-07-28 OSWorld was enhanced as OSWorld-Verified, including fixed community-reported examples and updated benchmark results, and that new runs should be compared with those new results. A 2026-06-26 line says OSWorld 2.0 is available. A score from the 2024 paper is not the same number as a score on the later set.","sourceUrl":"https://os-world.github.io/","asOf":"2026-10-03","notes":"Fetched the project homepage. What the 2.0 task set contains was not copied because the homepage line only says it is available."},{"id":"agentbench-2024","title":"AgentBench: Evaluating LLMs as Agents","year":2024,"studyKind":"capability_ceiling","classIds":["coding","eval-judge"],"failureModeIds":["instruction-drift","hallucinated-tool-args","weak-verification"],"lesson":"The abstract says AgentBench uses 8 environments and that the main obstacles they identified were poor long-term reasoning, decision-making, and instruction following. A coding or tool agent that cannot stay on the instruction over many steps fails this kind of eval even if a single call looks fluent. The abstract's model count did not render as a number on the fetched page, so none is stored.","sourceUrl":"https://arxiv.org/abs/2308.03688","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: published in ICLR 2024."},{"id":"mast-multi-agent-failures-2025","title":"Why Do Multi-Agent LLM Systems Fail?","year":2025,"studyKind":"mechanism_study","classIds":["orchestrator","eval-judge"],"failureModeIds":["inter-agent-misalignment","weak-verification"],"lesson":"The abstract says performance gains of multi-agent systems on popular benchmarks are often minimal. Their taxonomy, built from 150 traces with annotator kappa of 0.88, groups failures into system design, inter-agent misalignment, and task verification, and identifies 14 modes. The larger MAST-Data collection is described as 1600+ annotated traces. This is a failure taxonomy, not a single production outage.","sourceUrl":"https://arxiv.org/abs/2503.13657","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 2025). The 14 mode names are not copied because the abstract did not list them."},{"id":"langsmith-cert-incident-2025","title":"LangSmith incident on May 1, 2025","year":2025,"studyKind":"production_incident","classIds":["eval-judge","scheduled-ops"],"failureModeIds":["observability-gap"],"lesson":"LangChain's incident note says the US LangSmith API had an elevated error rate for 28 minutes on May 1, 2025, from 14:35 to 15:03 UTC, and that about 55% of API requests failed, including run ingestion. The cause they describe is a leftover DNS record that stopped certificate renewal until the certificate expired. Agent traces can disappear because the trace host's certificate failed, not because the agent reasoned badly.","sourceUrl":"https://www.langchain.com/blog/langsmith-incident-on-may-1-2025","asOf":"2026-10-03","notes":"Fetched LangChain's own incident writeup. This is an observability outage, not a model-behavior incident."}]}