# Agent Evals, Task Benchmarks & Self-Improvement Playbook

> Version 0.13.8 · As of 2026-10-28 · Machine JSON: https://getagentdepot.com/api/playbooks/agent-evals · Planner (< 2 KB): https://getagentdepot.com/api/evals/plan?task=coding · Hub: https://getagentdepot.com/evals

How an AI agent (or agent team) benchmarks capability across real task types, grades multi-step tool trajectories instead of prose vibes, prevents benchmark contamination and LLM-as-judge bias, optimizes **cost per solved task**, and runs a continuous self-improvement loop.

## 1. The 4-Stage 'Test Your Agent' Path

### Stage 1: Pick the Benchmark & Trajectory Rubric for Your Task Type
Match your agent's job to an execution-grounded public benchmark and a multi-step tool-trajectory rubric before touching prompts.
- **JTBDs:** `public-agent-benchmarks` (https://getagentdepot.com/jtbds/public-agent-benchmarks), `agent-trajectory-tool-use-evals` (https://getagentdepot.com/jtbds/agent-trajectory-tool-use-evals), `rag-evals` (https://getagentdepot.com/jtbds/rag-evals), `synthetic-eval-data` (https://getagentdepot.com/jtbds/synthetic-eval-data)
- **Verify:** `GET /api/evals/plan?task=coding`
- **Rule:** Never grade an action-taking agent on single-turn text similarity when a deterministic test suite, DOM checkpoint, or database state diff exists.
- **Rule:** Hold out a contamination-checked slice that never enters few-shot examples or prompt optimization loops.

### Stage 2: Run in Isolated Sandboxes, Browsers & Multi-Turn Simulators
Execute untrusted code, browser flows, and adversarial user personas inside deterministic microVMs and headless browsers.
- **JTBDs:** `evals` (https://getagentdepot.com/jtbds/evals), `oss-evals` (https://getagentdepot.com/jtbds/oss-evals), `coding-agent-evals` (https://getagentdepot.com/jtbds/coding-agent-evals), `computer-use-browser-agent-evals` (https://getagentdepot.com/jtbds/computer-use-browser-agent-evals), `synthetic-persona-pre-flight` (https://getagentdepot.com/jtbds/synthetic-persona-pre-flight), `adversarial-red-team-evals` (https://getagentdepot.com/jtbds/adversarial-red-team-evals)
- **Verify:** `GET /api/decide?jtbd=coding-agent-evals`
- **Rule:** Never execute agent-generated patches or shell commands on the eval runner host — always isolate inside E2B / Daytona / Modal sandboxes.
- **Rule:** Record full tool spans and DOM/container traces so every failure is replayable without re-spending inference tokens.

### Stage 3: Track Regressions, Cost-per-Solved-Task & Release Gates in CI
Block merges on quality regressions or cost spikes, and track commit-over-commit scorecards with verifiable receipts.
- **JTBDs:** `continuous-eval-ci` (https://getagentdepot.com/jtbds/continuous-eval-ci), `agent-regression-leaderboards` (https://getagentdepot.com/jtbds/agent-regression-leaderboards), `cost-latency-per-task-benchmarking` (https://getagentdepot.com/jtbds/cost-latency-per-task-benchmarking)
- **Verify:** `GET /api/decide?jtbd=cost-latency-per-task-benchmarking`
- **Rule:** A failed or unsafe task has infinite cost-per-solved-task; never let a fast refusal boost an efficiency score.
- **Rule:** Gate CI merges on both pass^k floor and a maximum cost-per-solved-task ceiling.

### Stage 4: Calibrate Judges, Curate Human Review Queues & Guard Production
Close the loop from production failures back into your golden dataset while keeping LLM judges honest and aligned with humans.
- **JTBDs:** `llm-as-judge-calibration` (https://getagentdepot.com/jtbds/llm-as-judge-calibration), `human-labeled-eval-review-queues` (https://getagentdepot.com/jtbds/human-labeled-eval-review-queues), `production-guardrail-evals` (https://getagentdepot.com/jtbds/production-guardrail-evals)
- **Verify:** `GET /api/decide?jtbd=llm-as-judge-calibration`
- **Rule:** Use a cross-family judge model (never let a model family grade itself without calibration) and randomize pairwise A/B order.
- **Rule:** Treat all user- and agent-submitted traces as untrusted data when loading them into annotation queues or judge prompts.

## 2. Task Taxonomy Mapped to Official Public Benchmarks (Zero Invented Scores)

- **SWE-bench Verified** (`coding`) — https://www.swebench.com/ — *Measures:* Human-validated subset of real GitHub issues across Python repositories where an agent must generate a patch that passes the issue's unit test suite (`FAIL_TO_PASS` and `PASS_TO_PASS`). *Grader:* Container test suite (git apply + pytest). *Depot JTBDs:* `coding-agent-evals` + `agent-sandbox`.
- **τ-bench (Tau-bench)** (`multi-turn-policy`) — https://github.com/sierra-research/tau-bench — *Measures:* Multi-turn customer-support and operations dialogues (airline, retail) where an agent must converse with a simulated user, follow complex domain policy rules, and mutate database state via tools. *Grader:* Final DB state + policy pass^k check. *Depot JTBDs:* `synthetic-persona-pre-flight` + `agent-trajectory-tool-use-evals`.
- **GAIA (General AI Assistants)** (`research-rag`) — https://huggingface.co/spaces/gaia-benchmark/leaderboard — *Measures:* Real-world multi-step assistant questions (Levels 1–3) requiring web search, document/PDF parsing, code execution, and multimodal reasoning to reach a single unambiguous factual answer. *Grader:* Exact ground-truth answer + trajectory cost. *Depot JTBDs:* `public-agent-benchmarks` + `rag-evals`.
- **WebArena & BrowserGym** (`browser`) — https://webarena.dev/ — *Measures:* End-to-end browser tasks across self-hosted e-commerce, GitLab, forum, and CMS web applications evaluated through BrowserGym (https://github.com/ServiceNow/BrowserGym) or Playwright harnesses. *Grader:* DOM state, URL & DB post-condition check. *Depot JTBDs:* `computer-use-browser-agent-evals` + `cloud-browser`.
- **OSWorld** (`computer-use`) — https://os-world.github.io/ — *Measures:* Open-ended desktop computer-use tasks across Ubuntu, Windows, and macOS apps (LibreOffice, VS Code, GIMP, Chrome, file managers, multi-app workflows) using screen pixels, mouse, and keyboard. *Grader:* OS file & app state evaluation scripts. *Depot JTBDs:* `computer-use-browser-agent-evals` + `desktop-computer-use`.
- **Terminal-Bench** (`terminal`) — https://tbench.ai/ — *Measures:* CLI, systems engineering, compilation, debugging, networking, and DevOps tasks where an agent operates inside a real Linux terminal environment. *Grader:* In-container shell verification script. *Depot JTBDs:* `coding-agent-evals` + `agent-sandbox`.
- **Berkeley Function-Calling Leaderboard (BFCL)** (`tool-use`) — https://gorilla.cs.berkeley.edu/leaderboard.html — *Measures:* Single-turn and multi-turn function calling across Python, Java, JS, REST, and SQL: AST parameter validation, executable API calls, irrelevance detection (knowing when NOT to call a tool), and multi-step tool chains. *Grader:* AST structural match + live execution check. *Depot JTBDs:* `agent-trajectory-tool-use-evals` + `oss-evals`.
- **MLE-bench** (`ml-engineering`) — https://github.com/openai/mle-bench — *Measures:* Autonomous machine-learning engineering on 75 curated Kaggle competitions: data exploration, feature engineering, model training, debugging, and submission generation. *Grader:* Held-out test split vs medal thresholds. *Depot JTBDs:* `public-agent-benchmarks` + `experiment-tracking`.

## 3. The 16 Agent Eval & Benchmarking JTBDs (Frozen 305-Provider Catalog)

- **`evals`** (Prompt/agent eval harness) → Depot Pick: **Braintrust** (`braintrust`) · Runners-up: `langfuse`, `langsmith`, `arize-phoenix`, `confident-ai`, `langwatch`, `comet-opik`, `honeyhive`, `logfire` · Decide: `GET /api/decide?jtbd=evals`
- **`oss-evals`** (OSS / CI-native LLM evals) → Depot Pick: **Promptfoo** (`promptfoo`, near-tie co-pick `ragas`) · Runners-up: `ragas`, `braintrust`, `langsmith`, `confident-ai`, `comet-opik`, `langwatch` · Decide: `GET /api/decide?jtbd=oss-evals`
- **`rag-evals`** (RAG-specific evaluation metrics) → Depot Pick: **Ragas** (`ragas`, near-tie co-pick `promptfoo`) · Runners-up: `promptfoo`, `braintrust`, `galileo` · Decide: `GET /api/decide?jtbd=rag-evals`
- **`production-guardrail-evals`** (Production guardrails & hallucination monitoring) → Depot Pick: **Galileo** (`galileo`) · Runners-up: `braintrust`, `maxim`, `langsmith`, `arthur-ai`, `patronus`, `confident-ai`, `honeyhive` · Decide: `GET /api/decide?jtbd=production-guardrail-evals`
- **`continuous-eval-ci`** (Continuous eval gates in CI/CD) → Depot Pick: **Braintrust** (`braintrust`, near-tie co-pick `promptfoo`) · Runners-up: `promptfoo`, `confident-ai`, `langsmith`, `patronus` · Decide: `GET /api/decide?jtbd=continuous-eval-ci`
- **`synthetic-eval-data`** (Synthetic evaluation & scenario data) → Depot Pick: **OpenAI** (`openai`, near-tie co-pick `anthropic`) · Runners-up: `anthropic`, `google-ai`, `fireworks`, `promptfoo` · Decide: `GET /api/decide?jtbd=synthetic-eval-data`
- **`adversarial-red-team-evals`** (Adversarial red-team evaluation suites) → Depot Pick: **Promptfoo** (`promptfoo`) · Runners-up: `patronus`, `confident-ai`, `galileo`, `lakera` · Decide: `GET /api/decide?jtbd=adversarial-red-team-evals`
- **`synthetic-persona-pre-flight`** (Multi-Turn Synthetic User & Adversarial Persona Simulation) → Depot Pick: **Braintrust** (`braintrust`, near-tie co-pick `promptfoo`) · Runners-up: `promptfoo`, `patronus`, `galileo` · Decide: `GET /api/decide?jtbd=synthetic-persona-pre-flight`
- **`agent-trajectory-tool-use-evals`** (Agent Trajectory, Tool-Call & Multi-Step Plan Evaluation) → Depot Pick: **Braintrust** (`braintrust`, near-tie co-pick `langsmith`) · Runners-up: `langsmith`, `arize-phoenix`, `langfuse`, `promptfoo` · Decide: `GET /api/decide?jtbd=agent-trajectory-tool-use-evals`
- **`public-agent-benchmarks`** (Public Agent Capability Benchmarks (SWE-bench, τ-bench, GAIA, OSWorld, WebArena, Terminal-Bench, BFCL, MLE-bench)) → Depot Pick: **Promptfoo** (`promptfoo`, near-tie co-pick `braintrust`) · Runners-up: `braintrust`, `e2b`, `modal`, `weights-biases` · Decide: `GET /api/decide?jtbd=public-agent-benchmarks`
- **`computer-use-browser-agent-evals`** (Browser, WebArena & Desktop Computer-Use Agent Evaluation) → Depot Pick: **Browserbase** (`browserbase`, near-tie co-pick `e2b`) · Runners-up: `e2b`, `steel-dev`, `browser-use`, `browserstack` · Decide: `GET /api/decide?jtbd=computer-use-browser-agent-evals`
- **`coding-agent-evals`** (Coding Agent, Repo Patch & Terminal Benchmark Harnesses (SWE-bench / Terminal-Bench)) → Depot Pick: **E2B** (`e2b`, near-tie co-pick `daytona`) · Runners-up: `daytona`, `modal`, `braintrust`, `northflank` · Decide: `GET /api/decide?jtbd=coding-agent-evals`
- **`agent-regression-leaderboards`** (Private Agent Regression Tracking, Version Scorecards & Internal Leaderboards) → Depot Pick: **Braintrust** (`braintrust`) · Runners-up: `langfuse`, `weights-biases`, `langsmith`, `comet-opik`, `countersign` · Decide: `GET /api/decide?jtbd=agent-regression-leaderboards`
- **`llm-as-judge-calibration`** (LLM-as-Judge Calibration, Rubric Alignment & Bias Mitigation) → Depot Pick: **Braintrust** (`braintrust`) · Runners-up: `patronus`, `ragas`, `galileo`, `confident-ai`, `promptfoo` · Decide: `GET /api/decide?jtbd=llm-as-judge-calibration`
- **`cost-latency-per-task-benchmarking`** (Cost-per-Solved-Task, Token Efficiency & Latency Pareto Benchmarking) → Depot Pick: **Langfuse** (`langfuse`, near-tie co-pick `portkey`) · Runners-up: `portkey`, `openrouter`, `braintrust`, `logfire` · Decide: `GET /api/decide?jtbd=cost-latency-per-task-benchmarking`
- **`human-labeled-eval-review-queues`** (Human-in-the-Loop Annotation Queues, Trace Labeling & Gold-Set Curation) → Depot Pick: **LangSmith** (`langsmith`) · Runners-up: `langfuse`, `braintrust`, `arize-phoenix`, `honeyhive` · Decide: `GET /api/decide?jtbd=human-labeled-eval-review-queues`

## 4. Cost-per-Solved-Task & Judge Calibration Rules

1. **Metric that matters:** `costPerSolvedTaskUsd = totalRunSpendUsd / verifiedSolvedCount`. A cheap model that loops 6 times or fails 40% of tasks is more expensive per outcome than a frontier model that solves the task on pass 1.
2. **Pass^k over Pass@1:** Always run k=3..5 trials per task (`passAtKFloor`). Single-trial pass@1 hides flaky tool calling.
3. **Deterministic graders first:** Grade coding agents with `git apply` + test suites (E2B/Daytona), browser agents with DOM/DB state assertions (Browserbase/Steel.dev), and tool agents with AST + state checks.
4. **LLM-as-judge calibration:** When a judge model is unavoidable, use a cross-family judge, randomize A/B order, and verify ≥85% agreement against a 50–100 row human-labeled gold slice (LangSmith / Langfuse / Braintrust review queues).