Kit · 2026-10-03
Coding agent
coding
Edits a repository, runs commands in a workspace, and checks the result against tests or a diff.
Needs below are the JTBD ids already listed on this class, capped at 6. The pick is the current Depot Pick. The cheapest line is that job's matching sourced SKU, or unavailable. A free row is listed only when its jtbd ids include that job. This page does not rescore. Agents that cannot spend use ?cost=free and must not treat the paid pick as free.
GET /api/kit?class=coding · ?cost=free · all kits · class JSON
Depot Pick: anthropic · score as of 2026-10-19 · /api/jtbds/coding-agent
Depot Pick for coding-agent: anthropic (asOf 2026-10-19). See GET /api/jtbds/coding-agent + GET /api/methodology.
pickIsNotPriceWinner: true
Cheapest matching SKU: zai · GLM-4.6V-FlashX · 0.04 usd_per_million_input_tokens. min sourced input SKU among candidates; not quality-adjusted
Pick is not a price winner. anthropic sourced input $/1M is 0.1; cheapest among candidates is zai at 0.04.
Depot Pick: anthropic · score as of 2026-10-19 · /api/jtbds/diff-aware-pr-review
Depot Pick for diff-aware-pr-review: anthropic (asOf 2026-10-19). See GET /api/jtbds/diff-aware-pr-review + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: anthropic · Haiku 5.5 (prompts up to 100K tokens) · 0.1 usd_per_million_input_tokens. min sourced input SKU among candidates; not quality-adjusted
Pick is tied for the cheapest sourced input $/1M at 0.1 (also 1 other candidate).
Free resources
freeMatch: jtbd
- reviewdog · reviewdog · dev_tool · oss
Depot Pick: e2b · score as of 2026-09-14 · /api/jtbds/agent-sandbox
Depot Pick for agent-sandbox: e2b (asOf 2026-09-14). See GET /api/jtbds/agent-sandbox + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. Pick has no sourced usd_per_vcpu_hour row. GPU-hour, GiB-hour, and vCPU-second were not converted. Cheapest stays null. pickIsPriceWinner is null.
Nothing was compared. Pick has no sourced usd_per_vcpu_hour row. GPU-hour, GiB-hour, and vCPU-second were not converted. Cheapest stays null. pickIsPriceWinner is null.
Free resources
freeMatch: jtbd
- Bubblewrap · bubblewrap · free_sandbox · oss
- E2B SDK · e2b-sdk · dev_tool · oss
- Firecracker · firecracker · free_sandbox · oss
Depot Pick: e2b · score as of 2026-09-14 · /api/jtbds/code-interpreter
Depot Pick for code-interpreter: e2b (asOf 2026-09-14). See GET /api/jtbds/code-interpreter + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. no matching sourced usd_per_vcpu_hour among candidates; GPU-hour, GiB-hour, and vCPU-second were not converted; chat $/1M was not borrowed pickIsPriceWinner is null.
Nothing was compared. no matching sourced usd_per_vcpu_hour among candidates; GPU-hour, GiB-hour, and vCPU-second were not converted; chat $/1M was not borrowed pickIsPriceWinner is null.
Free resources
freeMatch: jtbd
- E2B SDK · e2b-sdk · dev_tool · oss
- Firecracker · firecracker · free_sandbox · oss
Depot Pick: modal · score as of 2026-10-19 · /api/jtbds/agent-ephemeral-ci
Depot Pick for agent-ephemeral-ci: modal (asOf 2026-10-19). See GET /api/jtbds/agent-ephemeral-ci + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. Pick has no sourced usd_per_vcpu_hour row. GPU-hour, GiB-hour, and vCPU-second were not converted. Cheapest stays null. pickIsPriceWinner is null.
Nothing was compared. Pick has no sourced usd_per_vcpu_hour row. GPU-hour, GiB-hour, and vCPU-second were not converted. Cheapest stays null. pickIsPriceWinner is null.
Free resources
freeMatch: jtbd
- Dagger · dagger · dev_tool · oss
Depot Pick: daytona · score as of 2026-09-14 · /api/jtbds/persistent-dev-env
Depot Pick for persistent-dev-env: daytona (asOf 2026-09-14). See GET /api/jtbds/persistent-dev-env + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: daytona · vCPU · 0.0504 usd_per_vcpu_hour · not a per-million-token rate. min sourced vCPU-hour among candidates; lowest vCPU-hour tier per provider; not quality-adjusted; GPU-hour, GiB-hour, and vCPU-second were not converted
Only 1 candidate has a matching USD per vCPU-hour; nothing was compared, so pickIsPriceWinner is null.
Failure modes
Tool loop tool-loop
Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.
Hallucinated tool arguments hallucinated-tool-args
Reject a call that fails the current schema or names an unknown id, and do that before the tool runs a side effect.
Partial tool success partial-tool-success
Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.
Collateral side effects collateral-side-effects
Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.
Weak verification weak-verification
Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Eval contamination eval-contamination
Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
Benchmark protocol mismatch benchmark-protocol-mismatch
Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
Runaway spend runaway-spend
Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
Silent truncation silent-truncation
Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
Non-determinism non-determinism
Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
Schema drift schema-drift
Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.
Context overflow context-overflow
If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
Instruction drift instruction-drift
Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.