Agent classes / Coding agent / Persistent agent dev environments
Persistent agent dev environments for a coding agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-09-14 (stale)
The class: Edits a repository, runs commands in a workspace, and checks the result against tests or a diff.
- A workspace that can be snapshotted
- Tests or a diff check that the agent does not grade itself
Depot Pick: Daytona
When not to use it: Disposable single-run code — E2B.
Runner-up: E2B.
Cheapest matching sourced SKU (usd_per_vcpu_hour): Daytona vCPU $0.0504 — source, asOf 2026-10-27. All sourced prices
Failure modes this class and job share (13)
- Context overflow: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
- Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
- Collateral side effects: Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Hallucinated tool arguments: Reject a call that fails the current schema or names an unknown id, and do that before the tool runs a side effect.
- Instruction drift: Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.
- Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
- Partial tool success: Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.
- Runaway spend: Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
- Schema drift: Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.
- Silent truncation: Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
- Tool loop: Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Free options this class uses (1)
- Development Container Specification (dev_tool, oss) — source, asOf 2026-10-25