AP
Agent Provider Depot
Opinionated picks for agents
OpenAPI
AGENT EVALS & BENCHMARKING PLAYBOOK · v0.13.8 · As of 2026-10-28
← /evals Hub/growth/playbooks/agent-evals.mdGET /api/playbooks/agent-evalsGET /api/evals/plan?task=coding

How an AI Agent Should Benchmark & Improve Itself

Scoring single-turn LLM prose tells you almost nothing about whether an autonomous agent can resolve a GitHub issue, complete a multi-step checkout, reconcile a ledger, or survive a flaky tool call. This playbook defines how an agent benchmarks its own task capability, avoids data contamination and judge bias, tracks costPerSolvedTaskUsd and pass^k reliability, and runs a closed-loop self-improvement cycle.

1. The 4-Stage Agent Benchmarking Architecture (16 JTBDs)

Agent evaluation is an engineering pipeline with four distinct jobs: picking the right task rubric, executing trajectories inside clean sandboxes, gating regressions in CI, and calibrating judges with human gold slices.

STAGE 1 · Pick the Benchmark & Trajectory Rubric for Your Task TypeGET /api/evals/plan?task=coding

Match your agent's job to an execution-grounded public benchmark and a multi-step tool-trajectory rubric before touching prompts.

public-agent-benchmarksagent-trajectory-tool-use-evalsrag-evalssynthetic-eval-data
STAGE 2 · Run in Isolated Sandboxes, Browsers & Multi-Turn SimulatorsGET /api/decide?jtbd=coding-agent-evals

Execute untrusted code, browser flows, and adversarial user personas inside deterministic microVMs and headless browsers.

evalsoss-evalscoding-agent-evalscomputer-use-browser-agent-evalssynthetic-persona-pre-flightadversarial-red-team-evals
STAGE 3 · Track Regressions, Cost-per-Solved-Task & Release Gates in CIGET /api/decide?jtbd=cost-latency-per-task-benchmarking

Block merges on quality regressions or cost spikes, and track commit-over-commit scorecards with verifiable receipts.

continuous-eval-ciagent-regression-leaderboardscost-latency-per-task-benchmarking
STAGE 4 · Calibrate Judges, Curate Human Review Queues & Guard ProductionGET /api/decide?jtbd=llm-as-judge-calibration

Close the loop from production failures back into your golden dataset while keeping LLM judges honest and aligned with humans.

llm-as-judge-calibrationhuman-labeled-eval-review-queuesproduction-guardrail-evals

2. Task Taxonomy Mapped to Official Public Benchmarks

Never evaluate a coding agent with a trivia multiple-choice test, or a customer-support tool agent with single-turn BLEU/ROUGE. Map your agent's primary job to a benchmark with a deterministic outcome grader:

SWE-bench Verified coding
https://www.swebench.com/ ↗JTBD: coding-agent-evals

Measures: Human-validated subset of real GitHub issues across Python repositories where an agent must generate a patch that passes the issue's unit test suite (`FAIL_TO_PASS` and `PASS_TO_PASS`).

Grader: Container test suite (git apply + pytest)
Contamination trap: Agents that grep git log history inside the container or overfit to issue phrasing inflate scores; strip post-issue git commits and enforce strict container timeouts.
τ-bench (Tau-bench) multi-turn-policy
https://github.com/sierra-research/tau-bench ↗JTBD: synthetic-persona-pre-flight

Measures: Multi-turn customer-support and operations dialogues (airline, retail) where an agent must converse with a simulated user, follow complex domain policy rules, and mutate database state via tools.

Grader: Final DB state + policy pass^k check
Contamination trap: Single-run pass@1 hides catastrophic unreliability: an agent with 70% pass@1 often drops below 25% on pass^5 when users phrase requests differently.
GAIA (General AI Assistants) research-rag
https://huggingface.co/spaces/gaia-benchmark/leaderboard ↗JTBD: public-agent-benchmarks

Measures: Real-world multi-step assistant questions (Levels 1–3) requiring web search, document/PDF parsing, code execution, and multimodal reasoning to reach a single unambiguous factual answer.

Grader: Exact ground-truth answer + trajectory cost
Contamination trap: Live web pages rot or leak benchmark answers directly into search results; block benchmark-leaderboard domains in your search tool during evaluation.
WebArena & BrowserGym browser
https://webarena.dev/ ↗JTBD: computer-use-browser-agent-evals

Measures: End-to-end browser tasks across self-hosted e-commerce, GitLab, forum, and CMS web applications evaluated through BrowserGym (https://github.com/ServiceNow/BrowserGym) or Playwright harnesses.

Grader: DOM state, URL & DB post-condition check
Contamination trap: Shared stateful web app instances cause cross-episode pollution; reset container state or browser context before every single task episode.
OSWorld computer-use
https://os-world.github.io/ ↗JTBD: computer-use-browser-agent-evals

Measures: Open-ended desktop computer-use tasks across Ubuntu, Windows, and macOS apps (LibreOffice, VS Code, GIMP, Chrome, file managers, multi-app workflows) using screen pixels, mouse, and keyboard.

Grader: OS file & app state evaluation scripts
Contamination trap: Coordinate/resolution drift and slow GUI rendering cause false timeouts; pin screen resolution and capture full step screenshots for failure triage.
Terminal-Bench terminal
https://tbench.ai/ ↗JTBD: coding-agent-evals

Measures: CLI, systems engineering, compilation, debugging, networking, and DevOps tasks where an agent operates inside a real Linux terminal environment.

Grader: In-container shell verification script
Contamination trap: Agents that run destructive commands (`rm -rf`, background fork bombs) wedge the worker; enforce per-episode microVM isolation and CPU/memory caps.
Berkeley Function-Calling Leaderboard (BFCL) tool-use
https://gorilla.cs.berkeley.edu/leaderboard.html ↗JTBD: agent-trajectory-tool-use-evals

Measures: Single-turn and multi-turn function calling across Python, Java, JS, REST, and SQL: AST parameter validation, executable API calls, irrelevance detection (knowing when NOT to call a tool), and multi-step tool chains.

Grader: AST structural match + live execution check
Contamination trap: Models tuned only on positive tool calls fail 'irrelevance' turns by hallucinating parameters when no tool matches; always include no-op / missing-parameter test cases.
MLE-bench ml-engineering
https://github.com/openai/mle-bench ↗JTBD: public-agent-benchmarks

Measures: Autonomous machine-learning engineering on 75 curated Kaggle competitions: data exploration, feature engineering, model training, debugging, and submission generation.

Grader: Held-out test split vs medal thresholds
Contamination trap: Data leakage from validation split into training features or memorizing public notebooks; run with network egress disabled or restricted and audit training scripts.

3. Avoiding Benchmark Contamination & LLM-as-Judge Bias

A. Anti-Contamination Protocol (Sandboxes & Git Hygiene)
  • Git history scrubbing: On SWE-bench Verified and repo-repair benchmarks, strip all git commits, tags, and reflogs after the issue creation timestamp inside the container (e2b / modal).
  • Ephemeral state reset: For WebArena, OSWorld, and τ-bench runs, reset the database, DOM cookies, and filesystem snapshot before every single trial so failed attempts never pollute subsequent trials.
  • Search domain allow/blocklisting: Block public benchmark repos and Hugging Face dataset viewers inside your agent's web-search tool during evaluation runs.
B. 4 Rules for LLM-as-Judge Calibration (/jtbds/llm-as-judge-calibration)
  • Prefer deterministic outcome graders (unit tests, AST match, SQL/DOM state diff, Countersign receipt) over LLM prose scoring whenever possible.
  • When LLM-as-judge is required, use a cross-family judge model (e.g. grade an OpenAI agent with Anthropic or vice versa) to avoid self-preference bias.
  • Randomize candidate order (A/B vs B/A) and normalize length to neutralize position and verbosity bias.
  • Require ≥85% agreement against a 50–100 example human-labeled gold slice before enforcing a judge in CI.

4. Why costPerSolvedTaskUsd + pass^k Beat Token Price & pass@1

A cheap model ($0.20/1M tokens) that loops for 25 turns and solves 30% of tasks costs more per solved task than a frontier model ($3–$15/1M tokens) that solves 75% of tasks in 4 clean tool calls. Likewise, an agent with pass@1 = 80% that drops to pass^5 = 32% (0.8^5) is unusable for autonomous production workflows.

# The two equations every agent benchmark harness must compute:
costPerSolvedTaskUsd = (totalInputOutputCachedTokenUsd + sandboxComputeUsd + paidToolApiUsd) / verifiedSolvedTaskCount
# Note: Failed or policy-violating episodes contribute to the numerator (spend) and 0 to the denominator!

passCaretK = ( episodesWhereAllKTrialsSucceeded ) / ( totalDistinctTasks )
# Use k = 3 for coding/browser regression gates; k = 5 for financial or customer-facing policy agents.
/jtbds/cost-latency-per-task-benchmarking (Pick: Langfuse)/jtbds/agent-regression-leaderboards (Pick: Braintrust)

5. Copy-Pasteable Agent Self-Eval Loop (CLI / CI / Autonomous Reflex)

Drop this loop into your agent's pre-release check or GitHub Action to plan the right benchmark, run Depot Benchmark Suite v2 tasks, and gate deployments on pass^3 and costPerSolvedTaskUsd.

# Step 1 — Ask Agent Depot for the recommended benchmark & tool stack for your task type (< 2 KB JSON):
curl -sS "https://getagentdepot.com/api/evals/plan?task=coding" | jq .

# Step 2 — Pull the canonical decision card + near-tie escape hatches for your primary eval harness:
curl -sS "https://getagentdepot.com/api/decide?jtbd=coding-agent-evals" \
  | jq '{pick: .pick.id, nearTie: .nearTie.coPickProviderId, whenNot: .whenNotTopPick}'

# Step 3 — Fetch runnable Depot Benchmark Suite v2 tasks for your category:
curl -sS "https://getagentdepot.com/api/benchmark/tasks?type=coding" | jq '.tasks[] | {id, title, difficulty}'

# Step 4 — Submit your agent's verified benchmark run (tagged untrusted_community until Countersign-attested):
curl -sS -X POST "https://getagentdepot.com/api/benchmark/runs" \
  -H "Content-Type: application/json" \
  -d '{
    "agentName": "my-agent-v1",
    "modelId": "claude-sonnet-4-6",
    "taskId": "coding-01-json-patch",
    "passed": true,
    "score": 1.0,
    "latencyMs": 3400,
    "costUsd": 0.018,
    "notes": "Verified pass^3=1.0 in E2B sandbox"
  }'

6. All 16 Agent Evals & Benchmarking JTBDs (8 New Gap JTBDs + 8 Core JTBDs)

Agent Trajectory, Tool-Call & Multi-Step Plan Evaluationagent-trajectory-tool-use-evals
Depot Pick: Braintrust (near-tie w/ langsmith) · Runners-up: langsmith, arize-phoenix, langfuse, promptfoo
/api/decide?jtbd=agent-trajectory-tool-use-evals
Public Agent Capability Benchmarks (SWE-bench, τ-bench, GAIA, OSWorld, WebArena, Terminal-Bench, BFCL, MLE-bench)public-agent-benchmarks
Depot Pick: Promptfoo (near-tie w/ braintrust) · Runners-up: braintrust, e2b, modal, weights-biases
/api/decide?jtbd=public-agent-benchmarks
Browser, WebArena & Desktop Computer-Use Agent Evaluationcomputer-use-browser-agent-evals
Depot Pick: Browserbase (near-tie w/ e2b) · Runners-up: e2b, steel-dev, browser-use, browserstack
/api/decide?jtbd=computer-use-browser-agent-evals
Coding Agent, Repo Patch & Terminal Benchmark Harnesses (SWE-bench / Terminal-Bench)coding-agent-evals
Depot Pick: E2B (near-tie w/ daytona) · Runners-up: daytona, modal, braintrust, northflank
/api/decide?jtbd=coding-agent-evals
Private Agent Regression Tracking, Version Scorecards & Internal Leaderboardsagent-regression-leaderboards
Depot Pick: Braintrust · Runners-up: langfuse, weights-biases, langsmith, comet-opik, countersign
/api/decide?jtbd=agent-regression-leaderboards
LLM-as-Judge Calibration, Rubric Alignment & Bias Mitigationllm-as-judge-calibration
Depot Pick: Braintrust · Runners-up: patronus, ragas, galileo, confident-ai, promptfoo
/api/decide?jtbd=llm-as-judge-calibration
Cost-per-Solved-Task, Token Efficiency & Latency Pareto Benchmarkingcost-latency-per-task-benchmarking
Depot Pick: Langfuse (near-tie w/ portkey) · Runners-up: portkey, openrouter, braintrust, logfire
/api/decide?jtbd=cost-latency-per-task-benchmarking
Human-in-the-Loop Annotation Queues, Trace Labeling & Gold-Set Curationhuman-labeled-eval-review-queues
Depot Pick: LangSmith · Runners-up: langfuse, braintrust, arize-phoenix, honeyhive
/api/decide?jtbd=human-labeled-eval-review-queues
Prompt/agent eval harnessevals
Depot Pick: Braintrust · Runners-up: langfuse, langsmith, arize-phoenix, confident-ai, langwatch, comet-opik, honeyhive, logfire
/api/decide?jtbd=evals
OSS / CI-native LLM evalsoss-evals
Depot Pick: Promptfoo (near-tie w/ ragas) · Runners-up: ragas, braintrust, langsmith, confident-ai, comet-opik, langwatch
/api/decide?jtbd=oss-evals
RAG-specific evaluation metricsrag-evals
Depot Pick: Ragas (near-tie w/ promptfoo) · Runners-up: promptfoo, braintrust, galileo
/api/decide?jtbd=rag-evals
Production guardrails & hallucination monitoringproduction-guardrail-evals
Depot Pick: Galileo · Runners-up: braintrust, maxim, langsmith, arthur-ai, patronus, confident-ai, honeyhive
/api/decide?jtbd=production-guardrail-evals
Continuous eval gates in CI/CDcontinuous-eval-ci
Depot Pick: Braintrust (near-tie w/ promptfoo) · Runners-up: promptfoo, confident-ai, langsmith, patronus
/api/decide?jtbd=continuous-eval-ci
Synthetic evaluation & scenario datasynthetic-eval-data
Depot Pick: OpenAI (near-tie w/ anthropic) · Runners-up: anthropic, google-ai, fireworks, promptfoo
/api/decide?jtbd=synthetic-eval-data
Adversarial red-team evaluation suitesadversarial-red-team-evals
Depot Pick: Promptfoo · Runners-up: patronus, confident-ai, galileo, lakera
/api/decide?jtbd=adversarial-red-team-evals
Multi-Turn Synthetic User & Adversarial Persona Simulationsynthetic-persona-pre-flight
Depot Pick: Braintrust (near-tie w/ promptfoo) · Runners-up: promptfoo, patronus, galileo
/api/decide?jtbd=synthetic-persona-pre-flight

7. Sourced Pricing Pins (As of 2026-10-28)