{"id":"continuous-eval-ci","slug":"continuous-eval-ci","name":"Continuous eval gates in CI/CD","shortName":"Eval CI gates","problemStatement":"Block merges when agent prompts/tools regress — hosted eval suites, datasets, and pass/fail gates wired to GitHub/GitLab CI.","whenNotTopPick":"Want OSS-first local gates (Promptfoo), DeepEval/Confident already standardized, or LangSmith is the org tracing system of record.","categoryIds":["cat-observability"],"depotPickProviderId":"braintrust","depotPickRationale":"Distinct from `evals` (general harness): CI gate DX + dataset versioning + flake control. Confidence: high for Braintrust/Promptfoo CI docs.","runnersUpIds":["promptfoo","confident-ai","langsmith","patronus"],"weights":{"capabilityFit":0.22,"pricePerformance":0.14,"latencyReliability":0.14,"developerDx":0.28,"privacyPosture":0.08,"lockInRisk":0.06,"maturity":0.08},"methodologyNotes":"Distinct from `evals` (general harness): CI gate DX + dataset versioning + flake control. Confidence: high for Braintrust/Promptfoo CI docs.","tags":["evals","ci","regression","agents"],"depth":{"whenNotToUse":{"text":"Want OSS-first local gates (Promptfoo), DeepEval/Confident already standardized, or LangSmith is the org tracing system of record.","source":"editorial `whenNotTopPick` (data/jtbds.json)"},"whenNotToUseReason":null,"runnerUp":{"providerId":"promptfoo","name":"Promptfoo","overallForJob":8.46,"pickOverallForJob":8.34,"beatsPickOn":["pricePerformance","privacyPosture","lockInRisk"],"source":"first resolvable entry of the job's editorially ranked `runnersUpIds`; scores are stored job-weighted scores"},"runnerUpReason":null,"lockIn":{"pickLockInRisk":5,"runnerUpLockInRisk":2.5,"scale":"0–10 stored curated estimate; higher = more lock-in (worse)","pickExitSignals":[],"runnerUpExitSignals":["open source (provider tag `oss`)"],"pickNotes":[],"selfHostFallback":[{"id":"promptfoo","name":"Promptfoo","license":"MIT","sourceUrl":"https://github.com/promptfoo/promptfoo/blob/main/LICENSE"}],"migration":"Runner-up Promptfoo has lower stored lock-in risk (2.5 vs 5, 0–10, higher = more lock-in). Self-host fallback listed for this job: Promptfoo.","migrationReason":null,"derivedFrom":"stored lockInRisk scores, provider tags (oss, self-host, openai-compat, open-models), free rows that list this job, and verbatim provider notes. No migration guide was invented."}},"freshness":{"pick":{"score":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"lastReviewed":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"sourceCheck":{"checkedAt":"2026-10-27","sourceUrls":1,"sourcesReachable":1,"pricingRows":0,"pricingRowsReconfirmed":0,"note":"A source re-check confirms the page is reachable and, for pricing rows, that the stored amount and SKU name appear on it. It does not re-derive or change the score.","reason":null}}},"depotPick":{"id":"braintrust","slug":"braintrust","name":"Braintrust","tagline":"Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if…","overall":8.34},"runnersUp":[{"id":"promptfoo","slug":"promptfoo","name":"Promptfoo","tagline":"Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI…","overall":8.46},{"id":"confident-ai","slug":"confident-ai","name":"Confident AI","tagline":"Best when your team already thinks in pytest/DeepEval. Braintrust remains Depot hosted-evals default…","overall":8.15},{"id":"langsmith","slug":"langsmith","name":"LangSmith","tagline":"If you live in LangGraph, LangSmith is the obvious observatory. Otherwise compare Langfuse/Helicone first.","overall":8.25},{"id":"patronus","slug":"patronus","name":"Patronus AI","tagline":"Layer on when regulated hallucination/safety gates matter more than general CI evals. Lakera for injection…","overall":7.3}]}