Kit · 2026-10-03
Eval and judge agent
eval-judge
Scores a trace or a patch against a rubric, without the gold label in its prompt, and records the harness version.
Needs below are the JTBD ids already listed on this class, capped at 6. The pick is the current Depot Pick. The cheapest line is that job's matching sourced SKU, or unavailable. A free row is listed only when its jtbd ids include that job. This page does not rescore. Agents that cannot spend use ?cost=free and must not treat the paid pick as free.
GET /api/kit?class=eval-judge · ?cost=free · all kits · class JSON
Depot Pick: braintrust · score as of 2026-10-19 · /api/jtbds/evals
Depot Pick for evals: braintrust (asOf 2026-10-19). See GET /api/jtbds/evals + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Depot Pick: promptfoo · score as of 2026-10-19 · /api/jtbds/oss-evals
Depot Pick for oss-evals: promptfoo (asOf 2026-10-19). See GET /api/jtbds/oss-evals + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Depot Pick: ragas · score as of 2026-09-14 · /api/jtbds/rag-evals
Depot Pick for rag-evals: ragas (asOf 2026-09-14). See GET /api/jtbds/rag-evals + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Free resources
freeMatch: jtbd
- Ragas · ragas · dev_tool · oss
Depot Pick: galileo · score as of 2026-09-14 · /api/jtbds/production-guardrail-evals
Depot Pick for production-guardrail-evals: galileo (asOf 2026-09-14). See GET /api/jtbds/production-guardrail-evals + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Free resources
freeMatch: jtbd
- Guardrails AI · guardrails · dev_tool · oss
Depot Pick: braintrust · score as of 2026-10-19 · /api/jtbds/continuous-eval-ci
Depot Pick for continuous-eval-ci: braintrust (asOf 2026-10-19). See GET /api/jtbds/continuous-eval-ci + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Free resources
freeMatch: jtbd
- Promptfoo · promptfoo · dev_tool · oss
Depot Pick: promptfoo · score as of 2026-10-19 · /api/jtbds/adversarial-red-team-evals
Depot Pick for adversarial-red-team-evals: promptfoo (asOf 2026-10-19). See GET /api/jtbds/adversarial-red-team-evals + GET /api/methodology.
pickIsNotPriceWinner: false
Cheapest matching SKU: unavailable — Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Free resources
freeMatch: jtbd
- Promptfoo · promptfoo · dev_tool · oss
Failure modes
Eval contamination eval-contamination
Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
Non-determinism non-determinism
Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
Weak verification weak-verification
Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Observability gap observability-gap
Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
Benchmark protocol mismatch benchmark-protocol-mismatch
Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
Inter-agent misalignment inter-agent-misalignment
Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.