Private Agent Regression Tracking, Version Scorecards & Internal Leaderboards
Track your own agent's pass@1, pass^k reliability floor, latency, and cost-per-solved-task across every prompt, model, or tool change over time — with commit-pinned scorecards and cryptographic outcome receipts.
At a glance — Oct 2026
Updated 2026-10-28 · forward this page: /jtbds/agent-regression-leaderboards#briefEditorial prior (2026-10-19) + deterministic source check (verified) + Countersign receipts (cs-54379c97, cs-6eab5b5d, cs-316f2587).
- Depot Pick
- Braintrust · 8.29/10 for this job (curated estimate, high confidence)
Braintrust compares every experiment against a pinned baseline commit across accuracy, latency, token cost, and custom scorers, surfacing per-example regressions before merge; pair with Depot's `/api/benchmark` (85 seed tasks, pass^5 floor) and Countersign receipts for verifiable release claims.
- Runner-up
- Langfuse · 8.67/10
Scores higher than the pick on: Price / performance, Privacy / data retention, Lock-in risk (higher = worse).
- When not to use the pick
- Self-hosted / open-source observability + dataset runs on a budget — Langfuse; ML training & sweep-centric leaderboards — Weights & Biases; LangGraph-centric org standard — LangSmith; OSS Opik span + eval tracking — Comet Opik. Disclosure: Countersign (countersign.signalry.ai) is our sister protocol for hash-chained verification receipts and is listed as a runner-up rather than Depot Pick per our neutrality firewall.
- Lock-in / migration
- Runner-up Langfuse has lower stored lock-in risk (2.5 vs 5, 0–10, higher = more lock-in).
- Sourced price (no comparable unit)
Pick: no sourced price in this job's unit (no_sourced_price).
No cheapest SKU (no_comparable_unit_for_job). No price was invented.
- Free option
- No free resource lists this job. A cheap paid SKU is not a free option.
- Failure modes to watch
- No failure mode plainly applies to this job (no_failure_mode_plainly_applies).
Cite: Agent Provider Depot, Depot Pick for agent-regression-leaderboards: braintrust, scores as of 2026-10-28. Agents: GET /api/decide?jtbd=agent-regression-leaderboards
Braintrust
Braintrust compares every experiment against a pinned baseline commit across accuracy, latency, token cost, and custom scorers, surfacing per-example regressions before merge; pair with Depot's `/api/benchmark` (85 seed tasks, pass^5 floor) and Countersign receipts for verifiable release claims.
Pricing (directional): Eval-first platform; data-based pricing with unlimited seats on many plans; strong CI eval story.
Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.
Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if needed.
Quote for Private Agent Regression Tracking, Version Scorecards & Internal Leaderboards
Depot Pick stays Braintrust. Prices below are sourced SKU rows only (as of 2026-10-28). Input and output are not blended. GET /api/quote?jtbd=agent-regression-leaderboards
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Sourced price unavailable for the Depot Pick. No row in the pricing index.
5 of 6 candidates have no sourced price row. 6 are excluded from the Matching price comparison (none). no comparable sourced unit for this job; chat input $/1M and other units were not borrowed
Runners-up
Langfuse
Depot default observability for most agent teams: open, affordable, good enough to serious.
Weights & Biases
Use for training/fine-tune discipline. For pure agent request tracing, Langfuse/Helicone are leaner.
LangSmith
If you live in LangGraph, LangSmith is the obvious observatory. Otherwise compare Langfuse/Helicone first.
Comet Opik
Pick when you already live in Comet or want OSS Opik + optional cloud. Langfuse remains cheaper OSS default…
Countersign
Deterministic outcome verification ledger for AI agents with hash-chained receipts and live rechecks.
When NOT to use the top pick
Self-hosted / open-source observability + dataset runs on a budget — Langfuse; ML training & sweep-centric leaderboards — Weights & Biases; LangGraph-centric org standard — LangSmith; OSS Opik span + eval tracking — Comet Opik. Disclosure: Countersign (countersign.signalry.ai) is our sister protocol for hash-chained verification receipts and is listed as a runner-up rather than Depot Pick per our neutrality firewall.
Full ranking (score table)
Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.
| # | Provider | Overall | Fit | $/perf | Latency | DX | As of |
|---|---|---|---|---|---|---|---|
| 1 | Countersign | 8.7 | 8.8 | 9.4 | 8.8 | 8.9 | 2026-10-28 |
| 2 | Langfuse | 8.7 | 8.8 | 9.2 | 8.4 | 8.8 | 2026-10-28 |
| 3 | BraintrustDepot Pick | 8.3 | 9.2 | 7.5 | 8.6 | 8.8 | 2026-10-28 |
| 4 | LangSmith | 8.2 | 9.0 | 7.0 | 8.8 | 9.0 | 2026-10-28 |
| 5 | Weights & Biases | 8.1 | 8.8 | 6.8 | 9.0 | 8.2 | 2026-10-28 |
| 6 | Comet Opik | 8.1 | 8.2 | 8.5 | 8.0 | 8.4 | 2026-10-28 |
Score methodology for this job
A real agent leaderboard tracks four numbers per release: (1) pass@1 task success, (2) pass^k consistency across k=3..5 trials (since τ-bench shows pass^k drops sharply even when pass@1 looks high), (3) p95 wall-clock latency, and (4) cost per SOLVED task (total spend divided by verified successes).
capabilityFit: 26%pricePerformance: 16%latencyReliability: 16%developerDx: 16%privacyPosture: 12%lockInRisk: 6%maturity: 8%