{"id":"evals","slug":"evals","name":"Prompt/agent eval harness","shortName":"Prompt/agent eval harness","problemStatement":"Regression-test agent quality in CI before ship.","whenNotTopPick":"Budget OSS-only — Langfuse/Phoenix. Already standardized on LangSmith datasets. Humanloop is sunset (Anthropic acquihire) — do not pick.","categoryIds":["cat-observability"],"depotPickProviderId":"braintrust","depotPickRationale":"capability eval workflows + dx.","runnersUpIds":["langfuse","langsmith","arize-phoenix","confident-ai","langwatch","comet-opik","honeyhive","logfire"],"weights":{"capabilityFit":0.26,"pricePerformance":0.12,"latencyReliability":0.12,"developerDx":0.22,"privacyPosture":0.12,"lockInRisk":0.08,"maturity":0.08},"methodologyNotes":"capability eval workflows + dx.","tags":["evals"],"depth":{"whenNotToUse":{"text":"Budget OSS-only — Langfuse/Phoenix. Already standardized on LangSmith datasets. Humanloop is sunset (Anthropic acquihire) — do not pick.","source":"editorial `whenNotTopPick` (data/jtbds.json)"},"whenNotToUseReason":null,"runnerUp":{"providerId":"langfuse","name":"Langfuse","overallForJob":8.65,"pickOverallForJob":8.28,"beatsPickOn":["pricePerformance","privacyPosture","lockInRisk"],"source":"first resolvable entry of the job's editorially ranked `runnersUpIds`; scores are stored job-weighted scores"},"runnerUpReason":null,"lockIn":{"pickLockInRisk":5,"runnerUpLockInRisk":2.5,"scale":"0–10 stored curated estimate; higher = more lock-in (worse)","pickExitSignals":[],"runnerUpExitSignals":["open source (provider tag `oss`)"],"pickNotes":[],"selfHostFallback":[{"id":"promptfoo","name":"Promptfoo","license":"MIT","sourceUrl":"https://github.com/promptfoo/promptfoo/blob/main/LICENSE"},{"id":"inspect-ai","name":"Inspect AI","license":"MIT","sourceUrl":"https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE"},{"id":"mmlu","name":"MMLU","license":"MIT","sourceUrl":"https://huggingface.co/datasets/cais/mmlu/raw/main/README.md"}],"migration":"Runner-up Langfuse has lower stored lock-in risk (2.5 vs 5, 0–10, higher = more lock-in). Self-host fallback listed for this job: Promptfoo, Inspect AI, MMLU.","migrationReason":null,"derivedFrom":"stored lockInRisk scores, provider tags (oss, self-host, openai-compat, open-models), free rows that list this job, and verbatim provider notes. No migration guide was invented."}},"freshness":{"pick":{"score":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"lastReviewed":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"sourceCheck":{"checkedAt":"2026-10-27","sourceUrls":1,"sourcesReachable":1,"pricingRows":0,"pricingRowsReconfirmed":0,"note":"A source re-check confirms the page is reachable and, for pricing rows, that the stored amount and SKU name appear on it. It does not re-derive or change the score.","reason":null}}},"depotPick":{"id":"braintrust","slug":"braintrust","name":"Braintrust","tagline":"Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if…","overall":8.28},"runnersUp":[{"id":"langfuse","slug":"langfuse","name":"Langfuse","tagline":"Depot default observability for most agent teams: open, affordable, good enough to serious.","overall":8.65},{"id":"langsmith","slug":"langsmith","name":"LangSmith","tagline":"If you live in LangGraph, LangSmith is the obvious observatory. Otherwise compare Langfuse/Helicone first.","overall":8.14},{"id":"arize-phoenix","slug":"arize-phoenix","name":"Arize Phoenix","tagline":"Choose Phoenix when you want open-source observability you can own. Cloud Arize when enterprise wants a…","overall":8.37},{"id":"confident-ai","slug":"confident-ai","name":"Confident AI","tagline":"Best when your team already thinks in pytest/DeepEval. Braintrust remains Depot hosted-evals default…","overall":8.06},{"id":"langwatch","slug":"langwatch","name":"LangWatch","tagline":"Strong when you need multi-turn/voice agent simulations with EU self-host. Langfuse default tracing…","overall":8.11},{"id":"comet-opik","slug":"comet-opik","name":"Comet Opik","tagline":"Pick when you already live in Comet or want OSS Opik + optional cloud. Langfuse remains cheaper OSS default…","overall":8.02},{"id":"honeyhive","slug":"honeyhive","name":"HoneyHive","tagline":"Use the free Developer tier to stand up agent evals/traces in a day; Enterprise when you need self-host…","overall":8.08},{"id":"logfire","slug":"logfire","name":"Pydantic Logfire","tagline":"Best when the agent is already PydanticAI/Python and you want app+LLM traces together. Langfuse stays Depot…","overall":8.37}]}