Cost-per-Solved-Task, Token Efficiency & Latency Pareto Benchmarking
Measure end-to-end USD cost, token burn (including retries, prompt cache hits, and sub-agent calls), and p50/p95 wall-clock latency per successfully solved task — so you optimize the cost-capability Pareto frontier instead of raw per-token list price.
At a glance — Oct 2026
Updated 2026-10-28 · forward this page: /jtbds/cost-latency-per-task-benchmarking#briefPromotion class: deterministic_source_reverified · 8.7 ± 0.5 (8.2–9.2 · editorial_prior) · Pricing verified 2026-10-25.
- Depot Pick
- Langfuse · 8.69/10 for this job (curated estimate, high confidence)
Langfuse tracks model-aware token costs, cache-read discounts, retry loops, and span-level p50/p95 latencies across every trace (MIT OSS self-host or Cloud from $0 Hobby / $29 Core as of 2026-10-21), letting you join trace spend directly with task outcome scores to compute true cost-per-solved-task.
- Runner-up
- Portkey · 8.34/10
Scores higher than the pick on: Latency / reliability, Developer & agent DX.
- When not to use the pick
- Gateway-level A/B routing, virtual keys, and budget caps across providers — Portkey; one-API multi-model price/latency comparison across 60+ hosts — OpenRouter; eval-first experiment diffs — Braintrust; OpenTelemetry Python app+LLM latency profiling — Pydantic Logfire.
- Lock-in / migration
- The pick has lower stored lock-in risk than runner-up Portkey (2.5 vs 5). Pick exit signals: open source (provider tag `oss`).
- Sourced price (usd_per_million_input_tokens)
Pick: no sourced price in this job's unit (no_sourced_price).
Cheapest matching SKU: OpenRouter anthropic/claude-sonnet-5.5 $2 — source, as of 2026-10-28, re-checked 2026-10-28. Not the pick; not quality-adjusted.
Price history: first seen 2026-10-03 · last changed no change recorded · history JSON
- Free option
- No free resource lists this job. A cheap paid SKU is not a free option.
- Failure modes to watch
- Runaway spend: Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
Cite: Agent Provider Depot, Depot Pick for cost-latency-per-task-benchmarking: langfuse, scores as of 2026-10-28. Agents: GET /api/decide?jtbd=cost-latency-per-task-benchmarking
All sourced prices, cheapest first
Langfuse
Langfuse tracks model-aware token costs, cache-read discounts, retry loops, and span-level p50/p95 latencies across every trace (MIT OSS self-host or Cloud from $0 Hobby / $29 Core as of 2026-10-21), letting you join trace spend directly with task outcome scores to compute true cost-per-solved-task.
Pricing (directional): Open-source + cloud; generous self-host; cloud from ~$29–59/mo tiers; EU-friendly.
Top pick (8.69) and #2 runner-up (8.34) are separated by 0.35 points (<= 0.35 weight-sensitivity band). Treat them as co-picks and use the workload tie-breaker rule below rather than the raw composite decimal.
Workload Tie-Breaker Rule: Gateway-level A/B routing, virtual keys, and budget caps across providers — Portkey; one-API multi-model price/latency comparison across 60+ hosts — OpenRouter; eval-first experiment diffs — Braintrust; OpenTelemetry Python app+LLM latency profiling — Pydantic Logfire.
Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.
Depot default observability for most agent teams: open, affordable, good enough to serious.
Quote for Cost-per-Solved-Task, Token Efficiency & Latency Pareto Benchmarking
Depot Pick stays Langfuse. Prices below are sourced SKU rows only (as of 2026-10-28). Input and output are not blended. GET /api/quote?jtbd=cost-latency-per-task-benchmarking
Only 1 candidate has a sourced input $/1M; nothing was compared, so pickIsPriceWinner is null.
Sourced price unavailable for the Depot Pick. No row in the pricing index.
3 of 5 candidates have no sourced price row. 4 are excluded from the Input $/1M comparison (chat_completion_input). min sourced input SKU among candidates; not quality-adjusted
Runners-up
Portkey
Depot pick for production AI gateway (fallbacks, virtual keys, budgets). OpenRouter when you want a model…
OpenRouter
Best multi-model router for agents that need failover and experimentation. Pin providers in production; don’t…
Braintrust
Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if…
Pydantic Logfire
Best when the agent is already PydanticAI/Python and you want app+LLM traces together. Langfuse stays Depot…
When NOT to use the top pick
Gateway-level A/B routing, virtual keys, and budget caps across providers — Portkey; one-API multi-model price/latency comparison across 60+ hosts — OpenRouter; eval-first experiment diffs — Braintrust; OpenTelemetry Python app+LLM latency profiling — Pydantic Logfire.
Full ranking (score table)
Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.
| # | Provider | Overall | Fit | $/perf | Latency | DX | As of |
|---|---|---|---|---|---|---|---|
| 1 | LangfuseDepot Pick | 8.7 | 8.8 | 9.2 | 8.4 | 8.8 | 2026-10-28 |
| 2 | Pydantic Logfire | 8.4 | 8.4 | 8.6 | 8.2 | 9.2 | 2026-10-28 |
| 3 | Portkey | 8.3 | 8.8 | 8.2 | 8.6 | 9.0 | 2026-10-28 |
| 4 | Braintrust | 8.2 | 9.2 | 7.5 | 8.6 | 8.8 | 2026-10-28 |
| 5 | OpenRouter | 8.0 | 8.8 | 7.8 | 7.6 | 9.2 | 2026-10-28 |
Score methodology for this job
Per-1M-token price is a vanity metric when a cheap model takes 6 retry loops or fails 40% of tasks. Always compute `costPerSolvedTaskUsd = totalRunSpendUsd / verifiedSolvedCount` alongside `pass@1` and `p95LatencyMs`. A failed task has infinite cost-per-success.
capabilityFit: 24%pricePerformance: 22%latencyReliability: 18%developerDx: 14%privacyPosture: 8%lockInRisk: 6%maturity: 8%