Agent Trajectory, Tool-Call & Multi-Step Plan Evaluation
Grade an agent's multi-step execution trajectory — tool selection, JSON argument schemas, call ordering, state mutations, and retry recovery — instead of only scoring the final text string.
At a glance — Oct 2026
Updated 2026-10-28 · forward this page: /jtbds/agent-trajectory-tool-use-evals#briefEditorial prior (2026-10-19) + deterministic source check (verified) + Countersign receipts (cs-54379c97, cs-6eab5b5d, cs-316f2587).
- Depot Pick
- Braintrust · 8.33/10 for this job (curated estimate, high confidence)
Braintrust links nested span trees (tool calls, arguments, intermediate states) directly to dataset scorers and CI diff views, so you can grade trajectory efficiency, redundant tool loops, and final outcome together.
- Runner-up
- LangSmith · 8.22/10
Scores higher than the pick on: Latency / reliability, Developer & agent DX, Maturity / track record.
- When not to use the pick
- LangGraph-native node/edge trajectory assertions and state-graph replay — LangSmith; self-hosted OpenTelemetry span & tool-call evaluators — Arize Phoenix; open-source MIT tracing + dataset runs — Langfuse; local CLI tool-call assertions in CI without SaaS — Promptfoo.
- Lock-in / migration
- The pick has lower stored lock-in risk than runner-up LangSmith (5 vs 6.5).
- Sourced price (no comparable unit)
Pick: no sourced price in this job's unit (no_sourced_price).
No cheapest SKU (no_comparable_unit_for_job). No price was invented.
- Free option
- No free resource lists this job. A cheap paid SKU is not a free option.
- Failure modes to watch
- Observability gap: Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
Cite: Agent Provider Depot, Depot Pick for agent-trajectory-tool-use-evals: braintrust, scores as of 2026-10-28. Agents: GET /api/decide?jtbd=agent-trajectory-tool-use-evals
Braintrust
Braintrust links nested span trees (tool calls, arguments, intermediate states) directly to dataset scorers and CI diff views, so you can grade trajectory efficiency, redundant tool loops, and final outcome together.
Pricing (directional): Eval-first platform; data-based pricing with unlimited seats on many plans; strong CI eval story.
Top pick (8.33) and #2 runner-up (8.22) are separated by 0.11 points (<= 0.35 weight-sensitivity band). Treat them as co-picks and use the workload tie-breaker rule below rather than the raw composite decimal.
Workload Tie-Breaker Rule: LangGraph-native node/edge trajectory assertions and state-graph replay — LangSmith; self-hosted OpenTelemetry span & tool-call evaluators — Arize Phoenix; open-source MIT tracing + dataset runs — Langfuse; local CLI tool-call assertions in CI without SaaS — Promptfoo.
Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.
Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if needed.
Quote for Agent Trajectory, Tool-Call & Multi-Step Plan Evaluation
Depot Pick stays Braintrust. Prices below are sourced SKU rows only (as of 2026-10-28). Input and output are not blended. GET /api/quote?jtbd=agent-trajectory-tool-use-evals
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Sourced price unavailable for the Depot Pick. No row in the pricing index.
5 of 5 candidates have no sourced price row. 5 are excluded from the Matching price comparison (none). no comparable sourced unit for this job; chat input $/1M and other units were not borrowed
Runners-up
LangSmith
If you live in LangGraph, LangSmith is the obvious observatory. Otherwise compare Langfuse/Helicone first.
Arize Phoenix
Choose Phoenix when you want open-source observability you can own. Cloud Arize when enterprise wants a…
Langfuse
Depot default observability for most agent teams: open, affordable, good enough to serious.
Promptfoo
Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI…
When NOT to use the top pick
LangGraph-native node/edge trajectory assertions and state-graph replay — LangSmith; self-hosted OpenTelemetry span & tool-call evaluators — Arize Phoenix; open-source MIT tracing + dataset runs — Langfuse; local CLI tool-call assertions in CI without SaaS — Promptfoo.
Full ranking (score table)
Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.
| # | Provider | Overall | Fit | $/perf | Latency | DX | As of |
|---|---|---|---|---|---|---|---|
| 1 | Langfuse | 8.7 | 8.8 | 9.2 | 8.4 | 8.8 | 2026-10-28 |
| 2 | Arize Phoenix | 8.4 | 8.6 | 9.0 | 8.2 | 8.0 | 2026-10-28 |
| 3 | Promptfoo | 8.4 | 8.2 | 9.2 | 7.8 | 8.8 | 2026-10-28 |
| 4 | BraintrustDepot Pick | 8.3 | 9.2 | 7.5 | 8.6 | 8.8 | 2026-10-28 |
| 5 | LangSmith | 8.2 | 9.0 | 7.0 | 8.8 | 9.0 | 2026-10-28 |
Score methodology for this job
Weights capabilityFit and developerDx highest: trajectory evals must inspect intermediate tool spans, argument validity, and unnecessary retries alongside final task completion. Pair with BFCL (https://gorilla.cs.berkeley.edu/leaderboard.html) for function-calling schema accuracy and τ-bench (https://github.com/sierra-research/tau-bench) for policy-compliant tool sequences.
capabilityFit: 28%pricePerformance: 14%latencyReliability: 14%developerDx: 18%privacyPosture: 10%lockInRisk: 6%maturity: 10%