Agent classes / Coding agent / Coding agent / code generation
Coding agent / code generation for a coding agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-10-19
The class: Edits a repository, runs commands in a workspace, and checks the result against tests or a diff.
- A workspace that can be snapshotted
- Tests or a diff check that the agent does not grade itself
Depot Pick: Anthropic
When not to use it: Compliance blocks Anthropic; or you need cheapest high-volume codegen (DeepSeek/open hosts) with human review.
Runner-up: OpenAI.
Cheapest matching sourced SKU (usd_per_million_input_tokens): Z.AI (Zhipu GLM) GLM-4.6V-FlashX $0.04 — source, asOf 2026-10-03. All sourced prices
Failure modes this class and job share (13)
- Runaway spend: Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
- Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
- Collateral side effects: Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.
- Context overflow: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Hallucinated tool arguments: Reject a call that fails the current schema or names an unknown id, and do that before the tool runs a side effect.
- Instruction drift: Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.
- Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
- Partial tool success: Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.
- Schema drift: Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.
- Silent truncation: Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
- Tool loop: Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.