RAG-specific evaluation metrics
Measure faithfulness, context precision/recall for retrieval pipelines.
At a glance — Oct 2026
Updated 2026-10-27 · forward this page: /jtbds/rag-evals#briefScores as of 2026-09-14 (43 days old; Depot marks scores stale after 30 days). The pick is still Depot's current editorial pick; it has not been re-reviewed since. Prices below carry their own dates.
- Depot Pick
- Ragas · 8.29/10 for this job (curated estimate, high confidence)
Capability fit for RAG metrics.
- Runner-up
- Promptfoo · 8.43/10
Scores higher than the pick on: Capability fit, Latency / reliability, Developer & agent DX, Maturity / track record.
- When not to use the pick
- General agent/tool evals — Promptfoo/Braintrust.
- Lock-in / migration
- The pick has lower stored lock-in risk than runner-up Promptfoo (2 vs 2.5). Pick exit signals: open source (provider tag `oss`). Self-host fallback listed for this job: Ragas.
- Sourced price (no comparable unit)
Pick: no sourced price in this job's unit (no_sourced_price).
No cheapest SKU (no_comparable_unit_for_job). No price was invented.
- Free option
- Ragas (oss) — source, as of 2026-10-25
- Failure modes to watch
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
- Context overflow: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
- Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
- Lost in the middle: Ask the agent to name the passage it used. Spot-check questions whose only support sits in the middle of the context, not only at the edges.
- Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
Cite: Agent Provider Depot, Depot Pick for rag-evals: ragas, scores as of 2026-09-14. Agents: GET /api/decide?jtbd=rag-evals
Free RAG-specific evaluation metrics options · for rag agents · for eval-judge agents
Ragas
Capability fit for RAG metrics.
Pricing (directional): As of 2026-09-14: Free open-source RAG evaluation library; cost is judge-LLM tokens only (~$0.50–$2 per 100 queries with cheap judges). No hosted SaaS required. Deep LangChain/LlamaIndex integration.
Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.
Default RAG metric library. Pair with Promptfoo/Braintrust for broader agent evals.
Quote for RAG-specific evaluation metrics
Depot Pick stays Ragas. Prices below are sourced SKU rows only (as of 2026-10-27). Input and output are not blended. GET /api/quote?jtbd=rag-evals
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Sourced price unavailable for the Depot Pick. No row in the pricing index.
3 of 4 candidates have no sourced price row. 4 are excluded from the Matching price comparison (none). no comparable sourced unit for this job; chat input $/1M and other units were not borrowed
Runners-up
Promptfoo
Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI…
Braintrust
Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if…
Galileo
Pick for production hallucination measurement + guardrails. Braintrust/Promptfoo for lighter eval loops.
When NOT to use the top pick
General agent/tool evals — Promptfoo/Braintrust.
Full ranking (score table)
Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.
| # | Provider | Overall | Fit | $/perf | Latency | DX | As of |
|---|---|---|---|---|---|---|---|
| 1 | Promptfoo | 8.4 | 8.2 | 9.2 | 7.8 | 8.8 | 2026-10-19 |
| 2 | RagasDepot Pick | 8.3 | 8.0 | 9.5 | 7.5 | 8.2 | 2026-09-14 |
| 3 | Braintrust | 8.3 | 9.2 | 7.5 | 8.6 | 8.8 | 2026-10-19 |
| 4 | Galileo | 7.5 | 8.4 | 6.2 | 8.0 | 7.8 | 2026-09-14 |
Score methodology for this job
Capability fit for RAG metrics.
capabilityFit: 26%pricePerformance: 12%latencyReliability: 12%developerDx: 22%privacyPosture: 12%lockInRisk: 8%maturity: 8%