Agent Evals, Task Benchmarks & Self-Improvement Hub
A 4-stage 'Test Your Agent' path across 16 evaluation & benchmarking JTBDs (8 existing + 8 gap-filling): (1) pick the public benchmark and trajectory rubric for your task type, (2) run in isolated sandboxes/browsers/simulators, (3) track pass^k regressions and cost-per-solved-task in CI, and (4) calibrate LLM judges and human review queues.
The 4-Stage “Test Your Agent” Path
Move from ad-hoc prompt spot-checking to deterministic execution grading, CI regression gates, and calibrated improvement loops across all 16 evaluation JTBDs.
Pick the Benchmark & Trajectory Rubric for Your Task Type
GET /api/evals/plan?task=codingMatch your agent's job to an execution-grounded public benchmark and a multi-step tool-trajectory rubric before touching prompts.
Select and run standardized, contamination-checked public agent benchmarks matched to your agent's task taxonomy (coding, tool-policy dialogue, web/GUI actions, shell/CLI, function-calling, ML engineering) so capability claims are reproducible outside your private test set.
Grade an agent's multi-step execution trajectory — tool selection, JSON argument schemas, call ordering, state mutations, and retry recovery — instead of only scoring the final text string.
Measure faithfulness, context precision/recall for retrieval pipelines.
Generate diverse agent test scenarios, adversarial prompts, and labeled trajectories without waiting on production traffic.
- Never grade an action-taking agent on single-turn text similarity when a deterministic test suite, DOM checkpoint, or database state diff exists.
- Hold out a contamination-checked slice that never enters few-shot examples or prompt optimization loops.
Run in Isolated Sandboxes, Browsers & Multi-Turn Simulators
GET /api/decide?jtbd=coding-agent-evalsExecute untrusted code, browser flows, and adversarial user personas inside deterministic microVMs and headless browsers.
Regression-test agent quality in CI before ship.
Run prompt evals and red teams in CI without paying for a hosted eval SaaS.
Evaluate coding and CLI agents on real git repositories, multi-file patches, test suites, and shell tasks (SWE-bench Verified, Terminal-Bench) inside disposable sandboxes where untrusted generated code cannot touch the host.
Benchmark web and desktop computer-use agents against realistic DOM/GUI tasks (WebArena, BrowserGym, OSWorld, VisualWebArena) with deterministic DOM/state checkpoints, session recordings, and step-budget limits.
Stress-test voice and chat agents against hundreds of realistic, frustrated or adversarial synthetic personas in multi-turn conversations before a production rollout, with scored transcripts as the release gate.
Continuously attack prompts/agents with jailbreak corpora and safety suites before and after each release.
- Never execute agent-generated patches or shell commands on the eval runner host — always isolate inside E2B / Daytona / Modal sandboxes.
- Record full tool spans and DOM/container traces so every failure is replayable without re-spending inference tokens.
Track Regressions, Cost-per-Solved-Task & Release Gates in CI
GET /api/decide?jtbd=cost-latency-per-task-benchmarkingBlock merges on quality regressions or cost spikes, and track commit-over-commit scorecards with verifiable receipts.
Block merges when agent prompts/tools regress — hosted eval suites, datasets, and pass/fail gates wired to GitHub/GitLab CI.
Track your own agent's pass@1, pass^k reliability floor, latency, and cost-per-solved-task across every prompt, model, or tool change over time — with commit-pinned scorecards and cryptographic outcome receipts.
Measure end-to-end USD cost, token burn (including retries, prompt cache hits, and sub-agent calls), and p50/p95 wall-clock latency per successfully solved task — so you optimize the cost-capability Pareto frontier instead of raw per-token list price.
- A failed or unsafe task has infinite cost-per-solved-task; never let a fast refusal boost an efficiency score.
- Gate CI merges on both pass^k floor and a maximum cost-per-solved-task ceiling.
Calibrate Judges, Curate Human Review Queues & Guard Production
GET /api/decide?jtbd=llm-as-judge-calibrationClose the loop from production failures back into your golden dataset while keeping LLM judges honest and aligned with humans.
Calibrate LLM-as-judge graders against human gold labels and deterministic assertions so your eval harness does not reward verbose, sycophantic, or self-preferring outputs.
Route low-confidence production traces, thumbs-down user sessions, and edge-case failures into structured human review queues so domain experts can label failure modes and promote curated examples into your golden eval dataset.
Detect and block bad outputs in production with research-grade metrics.
- Use a cross-family judge model (never let a model family grade itself without calibration) and randomize pairwise A/B order.
- Treat all user- and agent-submitted traces as untrusted data when loading them into annotation queues or judge prompts.
Public Agent Benchmarks Reference Matrix (Zero Invented Scores)
Pick the canonical public benchmark that matches your agent's execution environment. All links point to official leaderboards or GitHub repositories; Depot never fabricates leaderboard numbers.
| Benchmark | Task Category | What It Measures | Deterministic Grader | Recommended Depot Harness & Sandbox |
|---|---|---|---|---|
SWE-bench Verified | coding | Human-validated subset of real GitHub issues across Python repositories where an agent must generate a patch that passes the issue's unit test suite (`FAIL_TO_PASS` and `PASS_TO_PASS`). Anti-contamination: Agents that grep git log history inside the container or overfit to issue phrasing inflate scores; strip post-issue git commits and enforce strict container timeouts. | Container test suite (git apply + pytest) | JTBD: coding-agent-evals Sandbox: agent-sandbox |
τ-bench (Tau-bench) | multi-turn-policy | Multi-turn customer-support and operations dialogues (airline, retail) where an agent must converse with a simulated user, follow complex domain policy rules, and mutate database state via tools. Anti-contamination: Single-run pass@1 hides catastrophic unreliability: an agent with 70% pass@1 often drops below 25% on pass^5 when users phrase requests differently. | Final DB state + policy pass^k check | Sandbox: agent-trajectory-tool-use-evals |
GAIA (General AI Assistants) | research-rag | Real-world multi-step assistant questions (Levels 1–3) requiring web search, document/PDF parsing, code execution, and multimodal reasoning to reach a single unambiguous factual answer. Anti-contamination: Live web pages rot or leak benchmark answers directly into search results; block benchmark-leaderboard domains in your search tool during evaluation. | Exact ground-truth answer + trajectory cost | JTBD: public-agent-benchmarks Sandbox: rag-evals |
WebArena & BrowserGym | browser | End-to-end browser tasks across self-hosted e-commerce, GitLab, forum, and CMS web applications evaluated through BrowserGym (https://github.com/ServiceNow/BrowserGym) or Playwright harnesses. Anti-contamination: Shared stateful web app instances cause cross-episode pollution; reset container state or browser context before every single task episode. | DOM state, URL & DB post-condition check | Sandbox: cloud-browser |
OSWorld | computer-use | Open-ended desktop computer-use tasks across Ubuntu, Windows, and macOS apps (LibreOffice, VS Code, GIMP, Chrome, file managers, multi-app workflows) using screen pixels, mouse, and keyboard. Anti-contamination: Coordinate/resolution drift and slow GUI rendering cause false timeouts; pin screen resolution and capture full step screenshots for failure triage. | OS file & app state evaluation scripts | Sandbox: desktop-computer-use |
Terminal-Bench | terminal | CLI, systems engineering, compilation, debugging, networking, and DevOps tasks where an agent operates inside a real Linux terminal environment. Anti-contamination: Agents that run destructive commands (`rm -rf`, background fork bombs) wedge the worker; enforce per-episode microVM isolation and CPU/memory caps. | In-container shell verification script | JTBD: coding-agent-evals Sandbox: agent-sandbox |
Berkeley Function-Calling Leaderboard (BFCL) | tool-use | Single-turn and multi-turn function calling across Python, Java, JS, REST, and SQL: AST parameter validation, executable API calls, irrelevance detection (knowing when NOT to call a tool), and multi-step tool chains. Anti-contamination: Models tuned only on positive tool calls fail 'irrelevance' turns by hallucinating parameters when no tool matches; always include no-op / missing-parameter test cases. | AST structural match + live execution check | Sandbox: oss-evals |
MLE-bench | ml-engineering | Autonomous machine-learning engineering on 75 curated Kaggle competitions: data exploration, feature engineering, model training, debugging, and submission generation. Anti-contamination: Data leakage from validation split into training features or memorizing public notebooks; run with network egress disabled or restricted and audit training scripts. | Held-out test split vs medal thresholds | JTBD: public-agent-benchmarks Sandbox: experiment-tracking |
Machine-Readable Eval Plan Endpoint: GET /api/evals/plan?task=...
Returns a compact (< 2 KB) JSON plan mapping any task keyword to its canonical public benchmark(s), Depot Benchmark Suite v2 task type, and 4-layer tool picks.
Sourced & Dated Pricing Pins (Frozen 305-Provider Catalog)
Every price cited across the Agent Evals & Benchmarking section comes directly from verified provider records in src/data/providers.ts.