{"jtbd":{"id":"evals","slug":"evals","name":"Prompt/agent eval harness","shortName":"Prompt/agent eval harness"},"recommended":{"role":"depot_pick","id":"braintrust","slug":"braintrust","name":"Braintrust","website":"https://www.braintrust.dev","editorialOverall":8.28,"scoreKind":"curated_estimate","scoreAsOf":"2026-10-19","price":{"available":false,"unavailableReason":"no_sourced_price","skus":[],"normalizedUsdPerMillionTokens":null,"normalizationStatus":"unavailable","normalizationReason":"no_sourced_price"}},"priceComparison":{"skuKind":"none","unit":null,"comparison":"no comparable sourced unit for this job; chat input $/1M and other units were not borrowed","amountLabel":"Matching price","reportsUsdPerMillion":false,"matchedCandidates":0},"cheapestInput":null,"cheapest":null,"recommendedWorkloadQuote":null,"representativeQuote":null,"estimatedTaskCost":null,"cheapestGoodEnough":{"band":{"rule":"editorial overall >= Depot Pick overall - 0.5","minOverall":7.78,"pickOverall":8.28,"points":0.5},"pick":null,"reason":"no_matching_sourced_price","cheaperButBelowBand":[]},"dates":{"scoreAsOf":"2026-10-19","scoreKind":"curated_estimate","priceVerifiedAsOf":null,"note":"scoreAsOf is when the editorial score was last set; priceVerifiedAsOf is the newest fact-check of the pick's price rows. They are separate: a fresh price check does not re-score."},"sourcedInputs":[],"pickIsPriceWinner":null,"priceWinnerReason":"no_comparison_no_candidate_has_matching_sourced_price","priceWinnerNote":"Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.","excluded":18,"coverage":{"candidates":19,"withAnySourcedRow":1,"withSourcedInputUsdPerMillion":0,"withMatchingPrice":0,"excludedNoSourcedPrice":18,"excludedFromCheapestInput":19},"normalizedUsdPerMillionTokens":null,"normalizationReason":"input_and_output_are_not_blended","seo":{"published":false,"reason":"Only 0 candidate(s) have a matching Matching price (none). Need 3 before a crawlable price section. Quote JSON is still returned."},"cite":"Depot Pick for evals: braintrust (asOf 2026-10-19). See GET /api/jtbds/evals + GET /api/methodology.","asOf":"2026-10-27","honesty":["Recommended provider is the current Depot Pick from GET /api/decide. This quote does not change scores.","SKU rows are copied from data/pricing-rows.json via GET /api/pricing. No new pages were scraped for this response.","normalizedUsdPerMillionTokens on this quote is always null. Input and output stay on separate rows.","cheapestInput compares one native unit that matches this job. Chat and coding jobs use priceSortEligible input $/1M. Embedding jobs use embedding SKUs only. Speech-to-text uses a stored speech unit only (per minute when that unit exists; a voice-agent hosting minute and a speech-engine minute are not speech-to-text). Speech synthesis uses one stored TTS unit in preference order (per minute, then per 1k characters, then per million characters) and does not mix those units or a speech-engine minute. Hosted-browser jobs compare usd_per_browser_hour only (lowest tier per provider, not quality-adjusted); proxy per GB, CAPTCHA solves, monthly plans, and chat tokens are not included. Sandbox, code-interpreter, ephemeral CI, and persistent dev env compare usd_per_vcpu_hour only, and only when the Depot Pick has that row; GPU-hour, GiB-hour, and vCPU-second are not converted. Secret jobs compare identity-month or user-month, never both; if both are stored, cheapestInput is null (mixed_secret_units). Voice-agent platform jobs compare a voice-agent hosting minute only. Transactional email compares usd_per_1k_emails only. Marketing email compares usd_per_1k_emails when a non-transactional per-1k SKU exists; otherwise it ranks a marketing-labeled usd_per_email and does not convert that rate to per 1k or mix it with per-1k. Monthly plans are not a unit price (plan_not_a_unit_price). Outbound and inbound webhook jobs compare usd_per_message only. Queue jobs compare usd_per_100k_messages or usd_per_million_events, never both (mixed_units). Push does not rank MAU or monthly plans. Error observability ranks a unit only when every candidate shares it; per-error fractions are not ranked against monthly plans. MotherDuck CU-hours and flight hours are not vCPU-hours. Composio's unsplit sandbox token rate is not chat $/1M. CAPTCHA SKUs are not ranked. Other non-token jobs do not borrow a chat $/1M rate. If nothing matches, cheapestInput is null.","excluded counts candidates with zero sourced pricing rows. excludedFromCheapestInput also counts candidates whose rows are a different unit than this job.","A Depot Pick is not a price winner unless its matching sourced price is the lowest among candidates. The comparison is not quality-adjusted. pickIsPriceWinner is null (not false) when nothing was compared."],"links":{"decide":"/api/decide?jtbd=evals","quote":"/api/quote?jtbd=evals","pricing":"/api/pricing?provider=braintrust","leaderboardPrice":"/api/leaderboards?jtbd=evals&sort=price","humanPage":"/jtbds/evals#quote","methodology":"/api/methodology"},"sourceConflicts":[{"providerId":"langsmith","field":"calculator monthly total: static '$49 / mo' in the page HTML vs the calculator script's own default (5,000 traces, 1 seat → $39)","sourceUrl":"https://www.langchain.com/pricing","asOf":"2026-10-22","quotes":["'Your agent stack Monthly total LSU 0 $49 / mo Estimate only. Actual LSU consumption varies with the work your agents perform.' (https://www.langchain.com/pricing, server-rendered HTML; WebFetch renders the same figure as \"$49/ mo\")","'traceLsuPerTrace:0.005, seatPrice:39, tracesFreeIncluded:10000,' … defaults 'traces:5000,' 'seats:1,' … 'return {lsu:lsu,cost:(seats||1)*RATES.seatPrice+lsu*RATES.lsuPrice};' (page calculator script: 1 × $39 + max(0, 5,000 − 10,000) × 0.005 LSU = $39); plan table 'Add unlimited seats $39 per seat/month'"],"safeAssumption":"Use the published rates ($39 per Plus seat, 0.005 LSU per base trace, 1 LSU = $1), which the plan cards, the comparison table, the tooltip and the calculator script all agree on; the $49 static figure is not reproducible from any published rate and is not used."},{"providerId":"logfire","field":"Personal plan past the 10M-record allowance: pricing page 'ingestion pauses' vs billing docs 'keeps accepting and storing' for a while, then drops new data","sourceUrl":"https://pydantic.dev/pricing","asOf":"2026-10-22","quotes":["'Up to 10 million records This is a lot of telemetry for a solo developer! Once you hit the limit, ingestion pauses — so you can't owe us anything on the personal tier.' (https://pydantic.dev/pricing; the FAQ on the same page says 'if you go well over the free allowance, we'll eventually stop ingesting new data')","'As soon as you pass the included amount, some views are restricted.' … 'Logfire also keeps accepting and storing what you send, so you are not losing data at this point. Further past the allowance, new data stops being stored.' (https://pydantic.dev/docs/logfire/manage/logfire-costs/)"],"safeAssumption":"No price impact: both sources agree Personal never bills ('can't owe us anything', 'never bills you for overage'). Depot treats Personal as hard-capped at 10,000,000 records and does not cost it above that (exceeds_included_and_personal_plan_no_paid_overage); the grace ingestion past the allowance is not modelled."},{"providerId":"comet-opik","field":"Opik Cloud team-member limits: plan cards 'Up to 10 team members' (Free Cloud) / 'Up to 50 team members' (Pro Cloud) vs FAQ 'All plans include unlimited team members'","sourceUrl":"https://www.comet.com/site/pricing/","asOf":"2026-10-22","quotes":["'Free Cloud Perfect for individuals $0 Free plan Get Started Up to 10 team members 25k spans per month 60-day data retention' … 'Pro Cloud Popular Expanded usage for teams $19 Per month Start Free Trial Up to 50 team members 100k spans per month 60-day data retention' … 'Enterprise Security, compliance & flexible deployments Custom Contact Us Unlimited team members Custom usage plans' (https://www.comet.com/site/pricing/ plan cards)","'What are the differences between Opik Open Source, Opik Free, Opik Pro, and Opik Enterprise? The main differences are in usage limits and data retention. All plans include unlimited team members and the full suite of LLM observability and evaluation features.' (https://www.comet.com/site/pricing/ Questions & Answers)"],"safeAssumption":"No price impact: no per-member price is published on any plan. Depot uses the stricter card caps as plan ceilings (opik_team_members maxTotal 10 on Free Cloud, 50 on Pro Cloud), so more than 50 members leaves Pro uncosted rather than assuming unlimited; the FAQ wording is not used."}],"volumeEstimateHint":"Add &volume=<number> for a plan-level monthly cost (GET /api/estimate?jtbd=evals&workload=...)"}