Public Agent Capability Benchmarks (SWE-bench, τ-bench, GAIA, OSWorld, WebArena, Terminal-Bench, BFCL, MLE-bench)
Select and run standardized, contamination-checked public agent benchmarks matched to your agent's task taxonomy (coding, tool-policy dialogue, web/GUI actions, shell/CLI, function-calling, ML engineering) so capability claims are reproducible outside your private test set.
At a glance — Oct 2026
Updated 2026-10-28 · forward this page: /jtbds/public-agent-benchmarks#briefPromotion class: deterministic_source_reverified · 8.4 ± 0.5 (7.9–8.9 · editorial_prior) · Pricing verified 2026-10-25.
- Depot Pick
- Promptfoo · 8.44/10 for this job (curated estimate, high confidence)
Promptfoo runs local, declarative YAML/CLI evaluation matrices across providers and models (Apache-2.0, $0 OSS CLI) with custom Python/JS graders and Docker/sandbox hooks — ideal for wrapping public benchmark subsets (BFCL, GAIA, τ-bench, SWE-bench Verified slices) in repeatable runs without SaaS lock-in.
- Runner-up
- Braintrust · 8.22/10
Scores higher than the pick on: Capability fit, Latency / reliability, Maturity / track record.
- When not to use the pick
- Need isolated Docker/microVM sandboxes to execute SWE-bench Verified, Terminal-Bench, or MLE-bench container tasks — E2B (or Modal for bursty parallel container shards); hosted experiment & dataset artifact tracking across training/eval sweeps — Weights & Biases; hosted dataset + scorer UI for team review — Braintrust.
- Lock-in / migration
- The pick has lower stored lock-in risk than runner-up Braintrust (2.5 vs 5). Pick exit signals: open source (provider tag `oss`).
- Sourced price (no comparable unit)
Pick: no sourced price in this job's unit (no_sourced_price).
No cheapest SKU (no_comparable_unit_for_job). No price was invented.
- Free option
- No free resource lists this job. A cheap paid SKU is not a free option.
- Failure modes to watch
- No failure mode plainly applies to this job (no_failure_mode_plainly_applies).
Cite: Agent Provider Depot, Depot Pick for public-agent-benchmarks: promptfoo, scores as of 2026-10-28. Agents: GET /api/decide?jtbd=public-agent-benchmarks
Promptfoo
Promptfoo runs local, declarative YAML/CLI evaluation matrices across providers and models (Apache-2.0, $0 OSS CLI) with custom Python/JS graders and Docker/sandbox hooks — ideal for wrapping public benchmark subsets (BFCL, GAIA, τ-bench, SWE-bench Verified slices) in repeatable runs without SaaS lock-in.
Pricing (directional): As of 2026-10-24 (promptfoo.dev/pricing, fetched 2026-10-24): Community 'Free Forever' ('Our open-source tool, perfect for individual developers and small teams.') with 'Red teaming (10k probes/month)' and 'Run locally or self-host on your …
Top pick (8.44) and #2 runner-up (8.22) are separated by 0.22 points (<= 0.35 weight-sensitivity band). Treat them as co-picks and use the workload tie-breaker rule below rather than the raw composite decimal.
Workload Tie-Breaker Rule: Need isolated Docker/microVM sandboxes to execute SWE-bench Verified, Terminal-Bench, or MLE-bench container tasks — E2B (or Modal for bursty parallel container shards); hosted experiment & dataset artifact tracking across training/eval sweeps — Weights & Biases; hosted dataset + scorer UI for team review — Braintrust.
Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.
Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI control.
Quote for Public Agent Capability Benchmarks (SWE-bench, τ-bench, GAIA, OSWorld, WebArena, Terminal-Bench, BFCL, MLE-bench)
Depot Pick stays Promptfoo. Prices below are sourced SKU rows only (as of 2026-10-28). Input and output are not blended. GET /api/quote?jtbd=public-agent-benchmarks
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Sourced price unavailable for the Depot Pick. No row in the pricing index.
3 of 5 candidates have no sourced price row. 5 are excluded from the Matching price comparison (none). no comparable sourced unit for this job; chat input $/1M and other units were not borrowed
Runners-up
Braintrust
Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if…
E2B
Depot default code sandbox for agents. Use for untrusted code; Daytona when you need persistent…
Modal
Depot darling for Python agent backends and bursty GPU jobs. If you think in functions not VMs, start here.
Weights & Biases
Use for training/fine-tune discipline. For pure agent request tracing, Langfuse/Helicone are leaner.
When NOT to use the top pick
Need isolated Docker/microVM sandboxes to execute SWE-bench Verified, Terminal-Bench, or MLE-bench container tasks — E2B (or Modal for bursty parallel container shards); hosted experiment & dataset artifact tracking across training/eval sweeps — Weights & Biases; hosted dataset + scorer UI for team review — Braintrust.
Full ranking (score table)
Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.
| # | Provider | Overall | Fit | $/perf | Latency | DX | As of |
|---|---|---|---|---|---|---|---|
| 1 | E2B | 8.6 | 9.4 | 8.0 | 8.6 | 9.4 | 2026-10-28 |
| 2 | Modal | 8.5 | 9.0 | 8.2 | 8.6 | 9.6 | 2026-10-28 |
| 3 | PromptfooDepot Pick | 8.4 | 8.2 | 9.2 | 7.8 | 8.8 | 2026-10-28 |
| 4 | Braintrust | 8.2 | 9.2 | 7.5 | 8.6 | 8.8 | 2026-10-28 |
| 5 | Weights & Biases | 8.0 | 8.8 | 6.8 | 9.0 | 8.2 | 2026-10-28 |
Score methodology for this job
Maps agent task types to official public benchmarks (no invented scores): SWE-bench Verified (https://www.swebench.com/) for repo-level bug fixing; τ-bench (https://github.com/sierra-research/tau-bench) for multi-turn tool + policy compliance; GAIA (https://huggingface.co/spaces/gaia-benchmark/leaderboard) for tool-augmented general assistant reasoning; WebArena / BrowserGym (https://webarena.dev/ · https://github.com/ServiceNow/BrowserGym) for realistic web apps; OSWorld (https://os-world.github.io/) for desktop GUI computer-use; Terminal-Bench (https://tbench.ai/) for CLI/sysadmin tasks; Berkeley Function-Calling Leaderboard BFCL (https://gorilla.cs.berkeley.edu/leaderboard.html) for tool schema AST accuracy; MLE-bench (https://github.com/openai/mle-bench) for autonomous ML engineering.
capabilityFit: 26%pricePerformance: 18%latencyReliability: 14%developerDx: 18%privacyPosture: 10%lockInRisk: 8%maturity: 6%