AP
Agent Provider Depot
Opinionated picks for agents
OpenAPI
← All jobs

Coding Agent, Repo Patch & Terminal Benchmark Harnesses (SWE-bench / Terminal-Bench)

Evaluate coding and CLI agents on real git repositories, multi-file patches, test suites, and shell tasks (SWE-bench Verified, Terminal-Bench) inside disposable sandboxes where untrusted generated code cannot touch the host.

GET /api/decide?jtbd=coding-agent-evalsAPI embedE2B vs DaytonaE2B vs Modal

At a glance — Oct 2026

Updated 2026-10-28 · forward this page: /jtbds/coding-agent-evals#brief
VERIFIED OUTCOME BACKED · 8.7 ± 0.35 (8.3–9.0 · editorial_plus_verified_receipt)

Editorial prior (2026-10-28) + deterministic source check (2026-10-28) + Countersign receipts (cs-64391106, cs-2e116f7d).

Depot Pick
E2B · 8.65/10 for this job (curated estimate, high confidence)

E2B boots isolated Firecracker microVMs in ~150ms with per-second vCPU/GiB pricing ($0.000014/vCPU-s, $0.0000045/GiB-s as of 2026-10-28, up to 1,100 concurrent sandboxes on Pro++), making it the cleanest substrate for running SWE-bench Verified repo checkouts and Terminal-Bench test harnesses at scale.

Runner-up
Daytona · 8.31/10

Scores higher than the pick on: Price / performance, Lock-in risk (higher = worse).

When not to use the pick
Need persistent multi-container dev workspaces with sticky git worktrees across long horizons — Daytona; need bursty 500+ container parallelization or GPU-backed ML tasks (MLE-bench) — Modal; need the experiment/dataset scoreboard layer on top of sandbox runs — Braintrust; need a full PaaS with build pipelines — Northflank.
Lock-in / migration
Runner-up Daytona has lower stored lock-in risk (3 vs 4, 0–10, higher = more lock-in).
Sourced price (no comparable unit)

Pick: no sourced price in this job's unit (no_matching_sourced_price).

No cheapest SKU (no_comparable_unit_for_job). No price was invented.

Free option
No free resource lists this job. A cheap paid SKU is not a free option.
Failure modes to watch
No failure mode plainly applies to this job (no_failure_mode_plainly_applies).
Reported outcomes (submitted reports only)

This job: loading… JSON · reports never change scores · seed/test records excluded

Depot Pick (E2B) on this job: loading… JSON · reports never change scores · seed/test records excluded

Cite: Agent Provider Depot, Depot Pick for coding-agent-evals: e2b, scores as of 2026-10-28. Agents: GET /api/decide?jtbd=coding-agent-evals

Depot PickNear-Tie / Co-Pick (Δ 0.34 pts — check workload tie-breaker)

E2B

E2B boots isolated Firecracker microVMs in ~150ms with per-second vCPU/GiB pricing ($0.000014/vCPU-s, $0.0000045/GiB-s as of 2026-10-28, up to 1,100 concurrent sandboxes on Pro++), making it the cleanest substrate for running SWE-bench Verified repo checkouts and Terminal-Bench test harnesses at scale.

Pricing (directional): Firecracker microVM sandboxes billed per second; templates; code interpreter for agents.

Cite: Depot Pick for coding-agent-evals: e2b
Co-Pick (#2 within ≤ 0.35 weight-sensitivity band): Daytona (8.31/10 vs 8.65/10 · Δ 0.34 pts)
Edge: capabilityFit (9.4 vs 8.8)

Top pick (8.65) and #2 runner-up (8.31) are separated by 0.34 points (<= 0.35 weight-sensitivity band). Treat them as co-picks and use the workload tie-breaker rule below rather than the raw composite decimal.

Workload Tie-Breaker Rule: Need persistent multi-container dev workspaces with sticky git worktrees across long horizons — Daytona; need bursty 500+ container parallelization or GPU-backed ML tasks (MLE-bench) — Modal; need the experiment/dataset scoreboard layer on top of sandbox runs — Braintrust; need a full PaaS with build pipelines — Northflank.

Overall
8.7
curated_estimate±0.35–0.5 band
confidence: high · editorial judgment as of 2026-10-28
Capability fit
9.4
Price / performance
8.0
Latency / reliability
8.6
Developer & agent DX
9.4
Privacy / data retention
8.5
Lock-in risk (higher = worse)
4.0
Maturity / track record
8.0

Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.

Sources: E2B: https://e2b.dev · E2B computer use docs: https://www.e2b.dev/docs/use-cases/computer-use

Depot default code sandbox for agents. Use for untrusted code; Daytona when you need persistent devworkspaces. Also the depot default Linux desktop sandbox for computer-use (e2b-desktop + VNC stream) when you already standardize on E2B.

Quote for Coding Agent, Repo Patch & Terminal Benchmark Harnesses (SWE-bench / Terminal-Bench)

Depot Pick stays E2B. Prices below are sourced SKU rows only (as of 2026-10-28). Input and output are not blended. GET /api/quote?jtbd=coding-agent-evals

Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.

  • 1 GiB memory usd_per_gib_second $0.0000045 source as of 2026-10-28
  • 1 vCPU usd_per_vcpu_second $0.000014 source as of 2026-10-28
  • 2 GiB memory usd_per_sandbox_second_ram_charge $0.000009 source as of 2026-10-28
  • 2 vCPU usd_per_sandbox_second_vcpu_charge $0.000028 source as of 2026-10-28
  • 4 GiB memory usd_per_sandbox_second_ram_charge $0.000018 source as of 2026-10-28
  • 4 vCPU usd_per_sandbox_second_vcpu_charge $0.000056 source as of 2026-10-28

2 of 5 candidates have no sourced price row. 5 are excluded from the Matching price comparison (none). no comparable sourced unit for this job; chat input $/1M and other units were not borrowed

Runners-up

Daytona

Choose when agents need sticky filesystems and full dev environments. E2B for disposable code exec. Confirm…

8.3
sandboxworkspacesossagents

Modal

Depot darling for Python agent backends and bursty GPU jobs. If you think in functions not VMs, start here.

8.5
serverlessgpupythonsandbox

Braintrust

Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if…

8.3
evalstracingciprompts

Northflank

Choose when the agent needs a whole app platform, not just a code jail. E2B for pure sandboxes; Railway for…

7.6
paassandboxgpubyoc

When NOT to use the top pick

Need persistent multi-container dev workspaces with sticky git worktrees across long horizons — Daytona; need bursty 500+ container parallelization or GPU-backed ML tasks (MLE-bench) — Modal; need the experiment/dataset scoreboard layer on top of sandbox runs — Braintrust; need a full PaaS with build pipelines — Northflank.

Full ranking (score table)

Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.

#ProviderOverallFit$/perfLatencyDXAs of
1E2BDepot Pick8.79.48.08.69.42026-10-28
2Modal8.59.08.28.69.62026-10-28
3Braintrust8.39.27.58.68.82026-10-28
4Daytona8.38.88.28.28.82026-10-28
5Northflank7.68.07.47.88.02026-10-28

Score methodology for this job

Coding agent benchmarks are execution-bound: the grader is `git apply` + the repo's unit test suite inside a clean container. Never grade coding agents with LLM-as-judge alone when a deterministic test suite exists. Official benchmarks: SWE-bench Verified (https://www.swebench.com/) and Terminal-Bench (https://tbench.ai/).

  • capabilityFit: 28%
  • pricePerformance: 16%
  • latencyReliability: 18%
  • developerDx: 16%
  • privacyPosture: 10%
  • lockInRisk: 6%
  • maturity: 6%