Agent classes / Security review agent / PR review agents
PR review agents for a security review agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-10-19
The class: Reads a change or a trace for secrets, overly broad permissions, and untrusted text copied into a tool prompt. It does not produce an attack.
- A findings list with file and line, not a payload
- Secrets detected are redacted in the report
Depot Pick: Anthropic
When not to use it: Bulk low-cost review of huge monorepos (DeepSeek/open hosts) with human confirm, or procurement locks you to Azure OpenAI.
Runner-up: OpenAI.
Cheapest matching sourced SKU (usd_per_million_input_tokens): Anthropic Haiku 5.5 (prompts up to 100K tokens) $0.1 — source, asOf 2026-10-27. All sourced prices
Failure modes this class and job share (4)
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
- Over-refusal: Log refusals separately from successes. Review a sample against the written policy. Do not treat a refusal as a pass, and do not weaken a safety check to raise the pass rate.
- Prompt injection: Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.
Free options this class uses (1)
- reviewdog (dev_tool, oss) — source, asOf 2026-10-25