Agent classes / Eval and judge agent / Production guardrails & hallucination mo…
Production guardrails & hallucination mo… for a eval and judge agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-09-14 (stale)
The class: Scores a trace or a patch against a rubric, without the gold label in its prompt, and records the harness version.
- Gold labels kept out of the judge prompt
- Harness version stored next to the score
Depot Pick: Galileo
When not to use it: Pre-release simulation focus — Maxim; lightweight CI — Promptfoo.
Runner-up: Braintrust.
No single-unit sourced price comparison for this job (no_comparable_unit_for_job).
Failure modes this class and job share (6)
- Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
- Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
- Observability gap: Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Free options this class uses (1)
- Guardrails AI (dev_tool, oss) — source, asOf 2026-10-25