OSS / CI-native LLM evals

Scores as of 2026-10-19 (8 days old).

Depot Pick

Promptfoo · 8.43/10 for this job (curated estimate, high confidence)

pricePerf + privacy of local runs; Promptfoo wins general OSS+redteam.

Runner-up

Ragas · 8.29/10

Scores higher than the pick on: Price / performance, Privacy / data retention, Lock-in risk (higher = worse).

When not to use the pick

Need hosted collaboration/datasets for PMs — Braintrust; RAG-only metrics — Ragas.

Lock-in / migration

Runner-up Ragas has lower stored lock-in risk (2 vs 2.5, 0–10, higher = more lock-in). Pick exit signals: open source (provider tag `oss`). Self-host fallback listed for this job: Promptfoo, Inspect AI, SWE-bench.

Sourced price (no comparable unit)

Pick: no sourced price in this job's unit (no_sourced_price).

No cheapest SKU (no_comparable_unit_for_job). No price was invented.

Free option

• AgentBench (oss) — source, as of 2026-10-25
• Berkeley Function Calling Leaderboard (oss) — source, as of 2026-10-25
• GAIA (free) — source, as of 2026-10-25

Failure modes to watch

• Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
• Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
• Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
• Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
• Observability gap: Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
• Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.

Reported outcomes (submitted reports only)

This job: loading… JSON · reports never change scores · seed/test records excluded

Depot Pick (Promptfoo) on this job: loading… JSON · reports never change scores · seed/test records excluded

Cite: Agent Provider Depot, Depot Pick for oss-evals: promptfoo, scores as of 2026-10-19. Agents: GET /api/decide?jtbd=oss-evals