AP
Agent Provider Depot
Opinionated picks for agents
OpenAPI

Agent classes / Eval and judge agent / Prompt/agent eval harness

Prompt/agent eval harness for a eval and judge agent

Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-10-19

The class: Scores a trace or a patch against a rubric, without the gold label in its prompt, and records the harness version.

  • Gold labels kept out of the judge prompt
  • Harness version stored next to the score

Depot Pick: Braintrust

When not to use it: Budget OSS-only — Langfuse/Phoenix. Already standardized on LangSmith datasets. Humanloop is sunset (Anthropic acquihire) — do not pick.

Runner-up: Langfuse.

No single-unit sourced price comparison for this job (no_comparable_unit_for_job).

Failure modes this class and job share (6)

  • Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
  • Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
  • Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
  • Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
  • Observability gap: Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
  • Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.

Free options this class uses (20)

  • Promptfoo (dev_tool, oss) — source, asOf 2026-10-25
  • Inspect AI (dev_tool, oss) — source, asOf 2026-10-25
  • GAIA (free_data, free) — source, asOf 2026-10-25
  • MMLU (free_data, oss) — source, asOf 2026-10-25
  • MMLU-Pro (free_data, oss) — source, asOf 2026-10-25
  • Berkeley Function Calling Leaderboard (free_data, oss) — source, asOf 2026-10-25
  • τ-bench (free_data, oss) — source, asOf 2026-10-25
  • Terminal-Bench (free_data, oss) — source, asOf 2026-10-25
  • WebArena (free_data, oss) — source, asOf 2026-10-25
  • VisualWebArena (free_data, oss) — source, asOf 2026-10-25
  • OSWorld (free_data, oss) — source, asOf 2026-10-25
  • Mind2Web (free_data, oss) — source, asOf 2026-10-25
  • AgentBench (free_data, oss) — source, asOf 2026-10-25
  • SWE-bench Verified (free_data, free) — source, asOf 2026-10-25
  • BrowserGym (free_data, oss) — source, asOf 2026-10-25
  • HumanEval (free_data, oss) — source, asOf 2026-10-25
  • GSM8K (free_data, oss) — source, asOf 2026-10-25
  • OpenAI MRCR (free_data, oss) — source, asOf 2026-10-25
  • TheAgentCompany (free_data, oss) — source, asOf 2026-10-25
  • Multi-SWE-bench (free_data, oss) — source, asOf 2026-10-25