{"jtbd":{"id":"oss-evals","slug":"oss-evals","name":"OSS / CI-native LLM evals","shortName":"OSS / CI-native LLM evals","problemStatement":"Run prompt evals and red teams in CI without paying for a hosted eval SaaS."},"pick":{"id":"promptfoo","slug":"promptfoo","name":"Promptfoo","tagline":"Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI…","website":"https://www.promptfoo.dev","overall":8.43,"scoreBand":{"low":7.8,"mid":8.4,"high":9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.8–9.0 (editorial prior)"},"scoreLayers":{"editorialPrior":8.43,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-10-19","pricingNotes":"As of 2026-10-24 (promptfoo.dev/pricing, fetched 2026-10-24): Community 'Free Forever' ('Our open-source tool, perfect for individual developers and small teams.') with 'Red teaming (10k probes/month)' and 'Run locally or self-host on your own infrastructure'; 'A probe is a single request made to your target system during red team testing.' Enterprise 'Custom' ('Contact us for a personalized quote.'; 'Enterprise customers can purchase additional probes for larger scale testing.'; 'Managed cloud deployment'). On-Premise 'Custom' ('Deployment on your own infrastructure'). No list price is published, so Enterprise price stays null.","scores":{"capabilityFit":8.2,"pricePerformance":9.2,"latencyReliability":7.8,"developerDx":8.8,"privacyPosture":9,"lockInRisk":2.5,"maturity":8,"overall":8.43,"kind":"curated_estimate","confidence":"high","asOf":"2026-10-19"}},"runnersUp":[{"id":"ragas","slug":"ragas","name":"Ragas","tagline":"Default RAG metric library. Pair with Promptfoo/Braintrust for broader agent evals.","website":"https://docs.ragas.io","overall":8.29,"scoreBand":{"low":7.7,"mid":8.3,"high":8.9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.7–8.9 (editorial prior)"},"scoreLayers":{"editorialPrior":8.29,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","pricingNotes":"As of 2026-09-14: Free open-source RAG evaluation library; cost is judge-LLM tokens only (~$0.50–$2 per 100 queries with cheap judges). No hosted SaaS required. Deep LangChain/LlamaIndex integration.","scores":{"capabilityFit":8,"pricePerformance":9.5,"latencyReliability":7.5,"developerDx":8.2,"privacyPosture":9.2,"lockInRisk":2,"maturity":7.8,"overall":8.29,"kind":"curated_estimate","confidence":"high","asOf":"2026-09-14"}},{"id":"braintrust","slug":"braintrust","name":"Braintrust","tagline":"Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if…","website":"https://www.braintrust.dev","overall":8.28,"scoreBand":{"low":7.7,"mid":8.3,"high":8.9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.7–8.9 (editorial prior)"},"scoreLayers":{"editorialPrior":8.28,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-10-19","pricingNotes":"Eval-first platform; data-based pricing with unlimited seats on many plans; strong CI eval story.","scores":{"capabilityFit":9.2,"pricePerformance":7.5,"latencyReliability":8.6,"developerDx":8.8,"privacyPosture":8,"lockInRisk":5,"maturity":8.2,"overall":8.28,"kind":"curated_estimate","confidence":"high","asOf":"2026-10-19"}},{"id":"langsmith","slug":"langsmith","name":"LangSmith","tagline":"If you live in LangGraph, LangSmith is the obvious observatory. Otherwise compare Langfuse/Helicone first.","website":"https://www.langchain.com/langsmith","overall":8.14,"scoreBand":{"low":7.5,"mid":8.1,"high":8.7,"halfWidth":0.6,"basis":"editorial_prior","display":"7.5–8.7 (editorial prior)"},"scoreLayers":{"editorialPrior":8.14,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","pricingNotes":"Seat + trace volume pricing; deep LangChain/LangGraph integration; Plus tiers from tens of $/mo.","scores":{"capabilityFit":9,"pricePerformance":7,"latencyReliability":8.8,"developerDx":9,"privacyPosture":7.8,"lockInRisk":6.5,"maturity":8.8,"overall":8.14,"kind":"curated_estimate","confidence":"high","asOf":"2026-09-14"}},{"id":"confident-ai","slug":"confident-ai","name":"Confident AI","tagline":"Best when your team already thinks in pytest/DeepEval. Braintrust remains Depot hosted-evals default…","website":"https://www.confident-ai.com","overall":8.06,"scoreBand":{"low":7.5,"mid":8.1,"high":8.7,"halfWidth":0.6,"basis":"editorial_prior","display":"7.5–8.7 (editorial prior)"},"scoreLayers":{"editorialPrior":8.06,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","pricingNotes":"As of 2026-09-14 (confident-ai.com/pricing): Free forever (2 seats, 1 project, 5 test runs/week, 1 GB-mo traces). Starter $200/mo (5 projects, 5 GB-mo then $1/GB-mo). Team $2,000/mo (unlimited projects, 75 GB-mo). Enterprise custom (on-prem, HIPAA, residency). Online eval tokens ~$0.05/$0.40 per 1M in/out.","scores":{"capabilityFit":8.6,"pricePerformance":7.8,"latencyReliability":8,"developerDx":9,"privacyPosture":7.8,"lockInRisk":4.5,"maturity":7.2,"overall":8.06,"kind":"curated_estimate","confidence":"high","asOf":"2026-09-14"}},{"id":"comet-opik","slug":"comet-opik","name":"Comet Opik","tagline":"Pick when you already live in Comet or want OSS Opik + optional cloud. Langfuse remains cheaper OSS default…","website":"https://www.comet.com/site/products/opik/","overall":8.02,"scoreBand":{"low":7.4,"mid":8,"high":8.6,"halfWidth":0.6,"basis":"editorial_prior","display":"7.4–8.6 (editorial prior)"},"scoreLayers":{"editorialPrior":8.02,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","pricingNotes":"As of 2026-09-14: Opik OSS self-host free. Comet Cloud free tier + Pro Cloud ~$19/mo class with ~100k spans / 60-day retention (confirm live comet.com/site/pricing). Fits teams already on Comet ML experiment tracking.","scores":{"capabilityFit":8.2,"pricePerformance":8.5,"latencyReliability":8,"developerDx":8.4,"privacyPosture":7.8,"lockInRisk":4,"maturity":8,"overall":8.02,"kind":"curated_estimate","confidence":"high","asOf":"2026-09-14"}},{"id":"langwatch","slug":"langwatch","name":"LangWatch","tagline":"Strong when you need multi-turn/voice agent simulations with EU self-host. Langfuse default tracing…","website":"https://langwatch.ai","overall":8.11,"scoreBand":{"low":7.5,"mid":8.1,"high":8.7,"halfWidth":0.6,"basis":"editorial_prior","display":"7.5–8.7 (editorial prior)"},"scoreLayers":{"editorialPrior":8.11,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","pricingNotes":"As of 2026-09-14 (langwatch.ai/pricing): Developer free (50k events/mo, 14-day access, 2 users). Growth €29/core-seat/mo + 200k events then €5/100k; 30-day retention then €3/GB. Enterprise hybrid/self-host/on-prem + marketplace billing. Scenario SDK OSS (Python+TS); voice agent simulation supported.","scores":{"capabilityFit":8.4,"pricePerformance":8,"latencyReliability":8,"developerDx":8.6,"privacyPosture":8.5,"lockInRisk":3.5,"maturity":7.2,"overall":8.11,"kind":"curated_estimate","confidence":"high","asOf":"2026-09-14"}}],"scoreBand":{"low":7.8,"mid":8.4,"high":9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.8–9.0 (editorial prior)"},"scoreLayers":{"editorialPrior":8.43,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"whyPickOverHigherScore":{"pickId":"promptfoo","pickName":"Promptfoo","pickScore":8.43,"highestScoredCandidateId":"promptfoo","highestScoredCandidateName":"Promptfoo","highestScore":8.43,"pickHasLowerCompositeScore":false,"delta":0,"pickLeadsOnDimensions":[],"explanation":"Promptfoo (8.43) holds both the highest composite score among ranked candidates and the editorial pick for OSS / CI-native LLM evals: pricePerf + privacy of local runs; Promptfoo wins general OSS+redteam."},"cheapest":null,"recommendedWorkloadQuote":null,"representativeQuote":null,"estimatedTaskCost":null,"freeAlternatives":[{"id":"agentbench","name":"AgentBench","kind":"free_data","jtbdIds":["evals","oss-evals"],"whatYouGet":"Public AgentBench multi-environment agent evaluation repository.","cost":"oss","license":"Apache-2.0","sourceUrl":"https://raw.githubusercontent.com/THUDM/AgentBench/main/LICENSE","notes":"LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical), no redirect from the THUDM path: 'Apache License Version 2.0, January 2004'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"bfcl","name":"Berkeley Function Calling Leaderboard","kind":"free_data","jtbdIds":["evals","oss-evals","agent-tools"],"whatYouGet":"Public function-calling eval dataset used by the Berkeley leaderboard.","cost":"oss","license":"Apache-2.0","sourceUrl":"https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard/raw/main/README.md","notes":"Dataset card fetched 2026-10-03 says license: apache-2.0. The gorilla GitHub LICENSE fetched the same day is also Apache-2.0. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: apache-2.0'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"gaia","name":"GAIA","kind":"free_data","jtbdIds":["evals","oss-evals","deep-research-agents"],"whatYouGet":"Gated public benchmark of tool-using assistant questions. Files require accepting a no-reshare condition. No license field was on the fetched card.","cost":"free","license":null,"sourceUrl":"https://huggingface.co/datasets/gaia-benchmark/GAIA/resolve/main/README.md","notes":"README fetched 2026-10-03. extra_gated_prompt says not to reshare outside a gated or private HF repo. No license key in the fetched card. Anonymous file download returned 401 earlier on the raw README path before this resolve URL. README re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter still has extra_gated_prompt 'To avoid contamination and data leakage, you agree to not reshare this dataset outside of a gated or private repository on the HF hub.' and still no license key, so license stays null.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"inspect-ai","name":"Inspect AI","kind":"dev_tool","jtbdIds":["oss-evals","evals"],"whatYouGet":"UK AISI eval framework. Model provider calls are billed by those providers.","cost":"oss","license":"MIT","sourceUrl":"https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE","notes":"Re-fetched twice 2026-10-25 (~10 s apart, plain curl, no login): GitHub blob page HTTP 200, no redirect (page chrome differs between fetches); raw LICENSE fetched twice too, HTTP 200, byte-identical: 'MIT License' / 'Copyright (c) 2024 UK AI Security Institute'. License unchanged.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"dev_tool","substitutable":true},{"id":"mmlu","name":"MMLU","kind":"free_data","jtbdIds":["evals","oss-evals"],"whatYouGet":"Public cais/mmlu multiple-choice eval dataset.","cost":"oss","license":"MIT","sourceUrl":"https://huggingface.co/datasets/cais/mmlu/raw/main/README.md","notes":"Dataset card fetched 2026-10-03 says license: mit. The hendrycks/test GitHub LICENSE fetched the same day is also MIT. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license:' / '- mit'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"mmlu-pro","name":"MMLU-Pro","kind":"free_data","jtbdIds":["evals","oss-evals"],"whatYouGet":"Public MMLU-Pro eval dataset, a harder multiple-choice successor set.","cost":"oss","license":"MIT","sourceUrl":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro/raw/main/README.md","notes":"Dataset card fetched 2026-10-03 says license: mit. The TIGER-AI-Lab/MMLU-Pro GitHub LICENSE fetched the same day is Apache-2.0 for the code repo; this row follows the dataset card, not the code license. Dataset card re-fetched twice 2026-10-25 (HTTP 200, byte-identical): front matter 'license: mit'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"promptfoo","name":"Promptfoo","kind":"dev_tool","providerId":"promptfoo","jtbdIds":["oss-evals","evals","adversarial-red-team-evals","continuous-eval-ci"],"whatYouGet":"Local eval CLI and library. No hosted quota is included.","cost":"oss","license":"MIT","sourceUrl":"https://github.com/promptfoo/promptfoo/blob/main/LICENSE","notes":"LICENSE fetched 2026-10-03 is MIT. Copyright line says Promptfoo 2025. README fetched 2026-10-03 says to automate checks in CI/CD. It does not say the check blocks a merge, and it does not name GitHub or GitLab. README fetched 2026-10-19 says promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, with red teaming and vulnerability scanning. Re-fetched twice 2026-10-25 (~10 s apart, plain curl, no login): GitHub blob page HTTP 200, no redirect (page chrome differs between fetches); raw LICENSE fetched twice too, HTTP 200, byte-identical: 'Copyright (c) Promptfoo 2025' / 'Permission is hereby granted, free of charge'. The file is the MIT permission text without an 'MIT License' heading. License unchanged.","limits":null,"asOf":"2026-10-25","freeType":"dev_tool","substitutable":true},{"id":"swe-bench","name":"SWE-bench","kind":"free_data","jtbdIds":["oss-evals","coding-agent"],"whatYouGet":"Public software-engineering benchmark instances. Download from the GitHub repo or the Hugging Face dataset princeton-nlp/SWE-bench.","cost":"oss","license":"MIT","sourceUrl":"https://github.com/SWE-bench/SWE-bench/blob/main/LICENSE","notes":"Dataset card: https://huggingface.co/datasets/princeton-nlp/SWE-bench. Instance text can include upstream repository code under other licenses. Re-fetched twice 2026-10-25: https://github.com/princeton-nlp/SWE-bench/blob/main/LICENSE now answers HTTP 301 to https://github.com/SWE-bench/SWE-bench/blob/main/LICENSE (HTTP 200; the GitHub blob page changes between fetches, page chrome), so sourceUrl moved to the redirect target; the raw file https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/LICENSE was also fetched twice (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2023 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R Narasimhan'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"swe-bench-verified","name":"SWE-bench Verified","kind":"free_data","jtbdIds":["evals","oss-evals","coding-agent"],"whatYouGet":"Public SWE-bench Verified subset. The dataset card fetched here does not state a license.","cost":"free","license":null,"sourceUrl":"https://huggingface.co/api/datasets/SWE-bench/SWE-bench_Verified","notes":"Hugging Face dataset API fetched 2026-10-03. cardData.license is null and gated is false. The existing swe-bench row is the MIT code license, not this subset card. License left null. Dataset API re-fetched twice 2026-10-25 (HTTP 200, byte-identical): cardData.license still null and gated false, so license stays null.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"tau-bench","name":"τ-bench","kind":"free_data","jtbdIds":["evals","oss-evals","agent-tools"],"whatYouGet":"Public tau-bench agent tool-use benchmark repository.","cost":"oss","license":"MIT","sourceUrl":"https://raw.githubusercontent.com/sierra-research/tau-bench/main/LICENSE","notes":"LICENSE fetched 2026-10-03 is MIT, copyright 2024 Sierra. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2024 Sierra'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"terminal-bench","name":"Terminal-Bench","kind":"free_data","jtbdIds":["evals","oss-evals","coding-agent"],"whatYouGet":"Public terminal-bench tasks for agents that use a shell.","cost":"oss","license":"Apache-2.0","sourceUrl":"https://raw.githubusercontent.com/laude-institute/terminal-bench/main/LICENSE","notes":"LICENSE fetched 2026-10-03 is Apache License 2.0. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'Apache License Version 2.0, January 2004'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true},{"id":"the-agent-company","name":"TheAgentCompany","kind":"free_data","jtbdIds":["evals","oss-evals"],"whatYouGet":"Public TheAgentCompany simulated-workplace agent benchmark.","cost":"oss","license":"MIT","sourceUrl":"https://raw.githubusercontent.com/TheAgentCompany/TheAgentCompany/main/LICENSE","notes":"LICENSE fetched 2026-10-03 is MIT, copyright 2024 TheAgentCompany. LICENSE re-fetched twice 2026-10-25 (HTTP 200, byte-identical): 'MIT License' / 'Copyright (c) 2024 TheAgentCompany'.","providerId":null,"limits":null,"asOf":"2026-10-25","freeType":"free_data","substitutable":true}],"referenceDocs":[],"falsification":{"jtbdId":"oss-evals","jtbdName":"OSS / CI-native LLM evals","currentPickId":"promptfoo","currentPickScore":8.43,"challengerId":"ragas","challengerScore":8.29,"currentScoreGap":0.14,"minVerifiedOutcomes":10,"requiredOutcomeMargin":0.29,"requiredSuccessRateDelta":0.15,"maxPickPriceIncreasePct":25,"whatWouldChangePick":["Ragas (ragas) outperforms Promptfoo (promptfoo) by >= 0.29 outcome points (or >= 15% worked-rate delta) across >= 10 Countersign-verified production outcomes.","A verified price increase > 25% on Promptfoo's representative SKU for oss-evals without a corresponding capability gain.","Live caller rail exhaustion (?exhausted=promptfoo) or sustained >= 30% degraded/failed rate in verified production outcomes."],"frozenHoldoutPolicy":{"holdoutSplit":"20% frozen benchmark holdout tasks + 80% live production outcome receipts","requireCountersignReceipt":true,"requiredEvidenceTiers":["outcome_verified","reference_checked"],"minDistinctContributors":2,"antiGamingRule":"Unverified self-reports and seed/editorial rows never mutate picks or scores."},"triggers":[{"kind":"verified_outcomes_outperformance","challengerId":"ragas","minVerifiedOutcomes":10,"requiredOutcomeMargin":0.29,"requiredWorkedRateDelta":0.15},{"kind":"verified_price_increase","providerId":"promptfoo","maxPriceIncreasePct":25},{"kind":"live_rail_exhaustion_or_degradation","providerId":"promptfoo","immediateOverrideUrl":"/api/decide?jtbd=oss-evals&exhausted=promptfoo"}],"submissionSpec":{"falsifyEndpoint":"/api/falsify?jtbd=oss-evals","outcomesEndpoint":"POST /api/feedback","correctionsEndpoint":"POST /api/corrections","payloadTemplate":{"agentIdentity":"agent:<your-agent-id>","jtbdId":"oss-evals","providerId":"ragas","outcome":"worked","detail":"Comparative production evaluation on oss-evals (n>=10 tasks) with Countersign receipt.","confidence":"high","receipt":{"countersign":{"claim_id":"cs-00000000","status":"verified","evidence_tier":"outcome_verified","counts_as_verified":true}}}}},"whatWouldChangePick":["Ragas (ragas) outperforms Promptfoo (promptfoo) by >= 0.29 outcome points (or >= 15% worked-rate delta) across >= 10 Countersign-verified production outcomes.","A verified price increase > 25% on Promptfoo's representative SKU for oss-evals without a corresponding capability gain.","Live caller rail exhaustion (?exhausted=promptfoo) or sustained >= 30% degraded/failed rate in verified production outcomes."],"routePlan":{"jtbdId":"oss-evals","primary":{"providerId":"promptfoo","name":"Promptfoo","overall":8.43,"scoreBand":{"low":7.8,"mid":8.4,"high":9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.8–9.0 (editorial prior)"},"scoreLayers":{"editorialPrior":8.43,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-10-19","role":"primary","reason":"Editorial Depot Pick for OSS / CI-native LLM evals; healthy/eligible under caller constraints."},"fallback":[{"providerId":"ragas","name":"Ragas","overall":8.29,"scoreBand":{"low":7.7,"mid":8.3,"high":8.9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.7–8.9 (editorial prior)"},"scoreLayers":{"editorialPrior":8.29,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","role":"fallback","reason":"Ranked fallback route for OSS / CI-native LLM evals."},{"providerId":"braintrust","name":"Braintrust","overall":8.28,"scoreBand":{"low":7.7,"mid":8.3,"high":8.9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.7–8.9 (editorial prior)"},"scoreLayers":{"editorialPrior":8.28,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-10-19","role":"fallback","reason":"Ranked fallback route for OSS / CI-native LLM evals."},{"providerId":"langsmith","name":"LangSmith","overall":8.14,"scoreBand":{"low":7.5,"mid":8.1,"high":8.7,"halfWidth":0.6,"basis":"editorial_prior","display":"7.5–8.7 (editorial prior)"},"scoreLayers":{"editorialPrior":8.14,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","role":"fallback","reason":"Ranked fallback route for OSS / CI-native LLM evals."},{"providerId":"confident-ai","name":"Confident AI","overall":8.06,"scoreBand":{"low":7.5,"mid":8.1,"high":8.7,"halfWidth":0.6,"basis":"editorial_prior","display":"7.5–8.7 (editorial prior)"},"scoreLayers":{"editorialPrior":8.06,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","role":"fallback","reason":"Ranked fallback route for OSS / CI-native LLM evals."},{"providerId":"comet-opik","name":"Comet Opik","overall":8.02,"scoreBand":{"low":7.4,"mid":8,"high":8.6,"halfWidth":0.6,"basis":"editorial_prior","display":"7.4–8.6 (editorial prior)"},"scoreLayers":{"editorialPrior":8.02,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","role":"fallback","reason":"Ranked fallback route for OSS / CI-native LLM evals."},{"providerId":"langwatch","name":"LangWatch","overall":8.11,"scoreBand":{"low":7.5,"mid":8.1,"high":8.7,"halfWidth":0.6,"basis":"editorial_prior","display":"7.5–8.7 (editorial prior)"},"scoreLayers":{"editorialPrior":8.11,"measuredBenchmark":null,"verifiedOutcomeScore":null,"verifiedOutcomeCount":0,"accountRuntimeState":"unprobed"},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-09-14","role":"fallback","reason":"Ranked fallback route for OSS / CI-native LLM evals."}],"disqualified":[],"expectedRegret":{"scoreDeltaVsUnconstrainedPick":0,"unconstrainedPickId":"promptfoo","activePrimaryId":"promptfoo","summary":"0.00 score regret: unconstrained editorial pick satisfies all caller constraints."},"evidenceAgeDays":{"editorialScoreAgeDays":8,"pricingVerifiedAgeDays":0,"editorialPriorAsOf":"2026-10-19","pricingLastVerifiedAt":"2026-10-27"},"confidenceBand":{"low":7.8,"mid":8.4,"high":9,"halfWidth":0.6,"basis":"editorial_prior","display":"7.8–9.0 (editorial prior)"},"constraintsApplied":false,"constraintBundle":{"task":null,"jtbd":"oss-evals","quotaExhausted":[],"degraded":[],"healthy":[],"budgetUsd":null,"maxLatencyMs":null,"region":null,"privacy":null,"exit":[],"existing":[]}},"pricingLastVerifiedAt":"2026-10-27","editorialPriorAsOf":"2026-10-19","tie":false,"scoreGap":0.14,"decidingFactor":null,"tieNote":null,"rationale":"pricePerf + privacy of local runs; Promptfoo wins general OSS+redteam.","whenNotTopPick":"Need hosted collaboration/datasets for PMs — Braintrust; RAG-only metrics — Ragas.","skipTopPick":false,"skipReason":null,"matchedConstraints":[],"pricingSummary":"As of 2026-10-24 (promptfoo.dev/pricing, fetched 2026-10-24): Community 'Free Forever' ('Our open-source tool, perfect for individual developers and small teams.') with 'Red teaming (10k probes/month)' and 'Run locally or self-host on your own infrastructure'; 'A probe is a singl…","cite":"Depot Pick for oss-evals: promptfoo (asOf 2026-10-19). See GET /api/jtbds/oss-evals + GET /api/methodology.","nextActions":[{"step":1,"title":"Inspect full JTBD + embeds","href":"/api/jtbds/oss-evals?embed=providers","why":"Confirm pick, runners-up, and methodology notes."},{"step":2,"title":"Quote sourced SKUs","href":"/api/quote?jtbd=oss-evals","why":"Depot Pick SKUs plus the cheapest sourced SKU whose unit matches this job. Does not change the pick. Does not blend input and output or borrow a chat rate for a non-chat job."},{"step":3,"title":"Open provider detail","href":"/api/providers/promptfoo","why":"Scores, pricingNotes, strengths/weaknesses, communitySignals (claims only)."},{"step":4,"title":"Cite methodology","href":"/api/methodology","why":"Required when publishing a Depot Pick citation."},{"step":5,"title":"Grab badge SVG","href":"/api/badge?type=depot-pick&jtbd=oss-evals","why":"Embed in README / agent passport."},{"step":6,"title":"Post production insight","method":"POST","href":"/api/insights","why":"Unverified until curated; never auto-mutates official scores."}],"honesty":["Do not invent scores or benches.","Insights are claims until curated/reviewed; they never silently mutate official scores.","lockInRisk higher = more lock-in (worse).","normalizedUsdPerMillionTokens is set only for a single sourced token row. Otherwise null, with a machine-readable reason. Do not coerce GPU, minute, or per-1k search units.","Runners-up are not Depot Picks.","Depot overall is editorial_weighted, not Arena Elo and not an Artificial Analysis index."],"asOf":"2026-10-19","depth":{"whenNotToUse":{"text":"Need hosted collaboration/datasets for PMs — Braintrust; RAG-only metrics — Ragas.","source":"editorial `whenNotTopPick` (data/jtbds.json)"},"whenNotToUseReason":null,"runnerUp":{"providerId":"ragas","name":"Ragas","overallForJob":8.29,"pickOverallForJob":8.43,"beatsPickOn":["pricePerformance","privacyPosture","lockInRisk"],"source":"first resolvable entry of the job's editorially ranked `runnersUpIds`; scores are stored job-weighted scores"},"runnerUpReason":null,"lockIn":{"pickLockInRisk":2.5,"runnerUpLockInRisk":2,"scale":"0–10 stored curated estimate; higher = more lock-in (worse)","pickExitSignals":["open source (provider tag `oss`)"],"runnerUpExitSignals":["open source (provider tag `oss`)"],"pickNotes":[],"selfHostFallback":[{"id":"promptfoo","name":"Promptfoo","license":"MIT","sourceUrl":"https://github.com/promptfoo/promptfoo/blob/main/LICENSE"},{"id":"inspect-ai","name":"Inspect AI","license":"MIT","sourceUrl":"https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE"},{"id":"swe-bench","name":"SWE-bench","license":"MIT","sourceUrl":"https://github.com/SWE-bench/SWE-bench/blob/main/LICENSE"}],"migration":"Runner-up Ragas has lower stored lock-in risk (2 vs 2.5, 0–10, higher = more lock-in). Pick exit signals: open source (provider tag `oss`). Self-host fallback listed for this job: Promptfoo, Inspect AI, SWE-bench.","migrationReason":null,"derivedFrom":"stored lockInRisk scores, provider tags (oss, self-host, openai-compat, open-models), free rows that list this job, and verbatim provider notes. No migration guide was invented."}},"citation":{"text":"Agent Provider Depot (v0.12.148), Depot Pick for oss-evals: promptfoo, scores as of 2026-10-19, retrieved 2026-10-27. https://getagentdepot.com/jtbds/oss-evals","markdown":"[Agent Provider Depot](https://getagentdepot.com/jtbds/oss-evals) (v0.12.148) — Depot Pick for `oss-evals`: **promptfoo** (scores as of 2026-10-19, retrieved 2026-10-27)","fields":{"publisher":"Agent Provider Depot","version":"0.12.148","role":"depot_pick","jtbdId":"oss-evals","providerId":"promptfoo","scoreAsOf":"2026-10-19","retrievedOn":"2026-10-27","page":"https://getagentdepot.com/jtbds/oss-evals","api":"https://getagentdepot.com/api/decide?jtbd=oss-evals"},"howToCite":"Cite Depot as an editorial, dated recommendation, not a benchmark: name the job, the Depot Pick (or runner-up), the score asOf date and the day you retrieved it, and link the job page. Format: “Agent Provider Depot (v{version}), Depot Pick for {jobId}: {providerId}, scores as of {scoreAsOf}, retrieved {retrievedOn}. {baseUrl}/jtbds/{jobId}”. Quote prices only with their own sourceUrl and asOf. If the pick's score is stale (freshness.pick.score.stale), say so."},"freshness":{"pick":{"score":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"lastReviewed":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"sourceCheck":{"checkedAt":"2026-10-27","sourceUrls":3,"sourcesReachable":3,"pricingRows":0,"pricingRowsReconfirmed":0,"note":"A source re-check confirms the page is reachable and, for pricing rows, that the stored amount and SKU name appear on it. It does not re-derive or change the score.","reason":null}},"scoreAsOf":"2026-10-19","editorialPriorAsOf":"2026-10-19","pricingLastVerifiedAt":"2026-10-27","pickLastReviewed":"2026-10-19","runnersUpScoreAsOf":["2026-09-14","2026-10-19","2026-09-14","2026-09-14","2026-09-14","2026-09-14"],"contractAsOf":"2026-10-03","pricingShape":"prose_notes_only","normalizedUsdPerMillionTokens":null,"normalizationStatus":"unavailable","normalizationReason":"no_sourced_price"},"pricingContract":{"normalizedUsdPerMillionTokens":null,"normalizationStatus":"unavailable","normalizationReason":"no_sourced_price","shape":"prose_notes_only","filterable":false,"priceSortEligible":false,"priceSortKeyUsdPerMillionInputTokens":null,"rows":[],"notesExcerpt":"As of 2026-10-24 (promptfoo.dev/pricing, fetched 2026-10-24): Community 'Free Forever' ('Our open-source tool, perfect for individual developers and small teams.') with 'Red teaming (10k probes/month)' and 'Run locally or self-host on your own infrastructure'; 'A probe is a singl…","notesCheck":{"status":"no_known_contradiction","pairs":[],"review":null,"note":null},"lastReviewed":"2026-10-19","scoreAsOf":"2026-10-19","website":"https://www.promptfoo.dev","doNotParse":"Do not parse pricingNotes. Use rows only. normalizedUsdPerMillionTokens is a single number only when this provider has exactly one sourced token row. Input/output, context tiers, and non-token units stay null at this level.","gap":"No sourced price row for this provider. pricingNotes stay prose. See research/gaps.md."},"links":{"jtbd":"/jtbds/oss-evals","jtbdApi":"/api/jtbds/oss-evals","leaderboard":"/api/leaderboards?jtbd=oss-evals","methodology":"/api/methodology","badge":"/api/badge?type=depot-pick&jtbd=oss-evals","decide":"/api/decide?jtbd=oss-evals","humanPage":"/jtbds/oss-evals","alternatives":"/api/alternatives?provider=promptfoo","canonical":"/api/canonical","quote":"/api/quote?jtbd=oss-evals","falsify":"/api/falsify?jtbd=oss-evals"},"constraintBundle":{"task":null,"jtbd":"oss-evals","quotaExhausted":[],"degraded":[],"healthy":[],"budgetUsd":null,"maxLatencyMs":null,"region":null,"privacy":null,"exit":[],"existing":[]},"stackContext":{"host":null,"framework":null,"repo":null,"lang":null,"existing":[],"volume":null,"sender":null,"detectedFromTask":[]},"keep":null,"contextAdvice":null,"communityFeedback":{"pickOnThisJob":{"failed":0,"degraded":0,"worked":0,"insights":0,"withVerifiedReceipt":0,"failureModes":{},"latestAt":null,"note":"Agent-reported claims (unverified unless a receipt is verified by Depot's re-check). They never change scores or picks."},"outcomes":{"job":{"n":0,"worked":0,"degraded":0,"failed":0,"failureModes":{},"verifiedReports":0,"contributors":0,"excluded":{"seed_editorial":0,"test":0,"notFeedback":0},"reason":"no_submitted_outcome_reports_yet"},"pick":{"providerId":"promptfoo","n":0,"worked":0,"degraded":0,"failed":0,"failureModes":{},"evidenceTiers":{"agent_claim":0,"self_declared":0,"reference_checked":0,"outcome_verified":0},"reason":"no_submitted_outcome_reports_yet"},"scoresMutated":false,"more":"/api/outcomes?jtbd=oss-evals"},"report":"POST /api/feedback {agentIdentity, outcome: failed|degraded|worked, detail, providerId, jtbdId, failureModeId?, receipt?}"}}