Agent classes / Security review agent / Prompt-injection / jailbreak firewall
Prompt-injection / jailbreak firewall for a security review agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-09-14 (stale)
The class: Reads a change or a trace for secrets, overly broad permissions, and untrusted text copied into a tool prompt. It does not produce an attack.
- A findings list with file and line, not a payload
- Secrets detected are redacted in the report
Depot Pick: Lakera Guard
When not to use it: Need programmable multi-step rails / topical policies — NeMo Guardrails; only need free hate/sexual/violence flags — OpenAI Moderation; already standardized on Palo Alto — Prisma AIRS; Cisco estate — cisco-ai-defense.
Runner-up: NVIDIA NeMo Guardrails.
No single-unit sourced price comparison for this job (no_comparable_unit_for_job).
Failure modes this class and job share (4)
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Over-refusal: Log refusals separately from successes. Review a sample against the written policy. Do not treat a refusal as a pass, and do not weaken a safety check to raise the pass rate.
- Prompt injection: Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Free options this class uses (1)
- LLM Guard (dev_tool, oss) — source, asOf 2026-10-25