Agent classes / Coding agent / PR review agents
PR review agents for a coding agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-10-19
The class: Edits a repository, runs commands in a workspace, and checks the result against tests or a diff.
- A workspace that can be snapshotted
- Tests or a diff check that the agent does not grade itself
Depot Pick: Anthropic
When not to use it: Bulk low-cost review of huge monorepos (DeepSeek/open hosts) with human confirm, or procurement locks you to Azure OpenAI.
Runner-up: OpenAI.
Cheapest matching sourced SKU (usd_per_million_input_tokens): Anthropic Haiku 5.5 (prompts up to 100K tokens) $0.1 — source, asOf 2026-10-27. All sourced prices
Failure modes this class and job share (13)
- Runaway spend: Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
- Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
- Collateral side effects: Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.
- Context overflow: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
- Hallucinated tool arguments: Reject a call that fails the current schema or names an unknown id, and do that before the tool runs a side effect.
- Instruction drift: Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.
- Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
- Partial tool success: Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.
- Schema drift: Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.
- Silent truncation: Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
- Tool loop: Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.
Free options this class uses (1)
- reviewdog (dev_tool, oss) — source, asOf 2026-10-25