Synthetic evaluation & scenario data
Generate diverse agent test scenarios, adversarial prompts, and labeled trajectories without waiting on production traffic.
At a glance — Oct 2026
Updated 2026-10-27 · forward this page: /jtbds/synthetic-eval-data#briefScores as of 2026-09-14 (43 days old; Depot marks scores stale after 30 days). The pick is still Depot's current editorial pick; it has not been re-reviewed since. Prices below carry their own dates.
- Depot Pick
- OpenAI · 8.33/10 for this job (curated estimate, high confidence)
No Scale/Labelbox vendor in catalog — LLM-synthetic path only. Confidence: medium; human review still required for high-stakes evals.
- Runner-up
- Anthropic · 8.54/10
Scores higher than the pick on: Capability fit, Price / performance, Privacy / data retention, Lock-in risk (higher = worse).
- When not to use the pick
- Need open-model bulk generation (Fireworks), or Promptfoo red-team generators already cover the matrix.
- Lock-in / migration
- Runner-up Anthropic has lower stored lock-in risk (6.5 vs 7.2, 0–10, higher = more lock-in). Self-host fallback listed for this job: Toucan-1.5M.
- Sourced price (no comparable unit)
Pick: no sourced price in this job's unit (no_matching_sourced_price).
No cheapest SKU (no_comparable_unit_for_job). No price was invented.
- Free option
- Toucan-1.5M (oss) — source, as of 2026-10-25
- Failure modes to watch
- Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
- Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
- Observability gap: Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Cite: Agent Provider Depot, Depot Pick for synthetic-eval-data: openai, scores as of 2026-09-14. Agents: GET /api/decide?jtbd=synthetic-eval-data
Free Synthetic eval data options · for eval-judge agents
OpenAI
No Scale/Labelbox vendor in catalog — LLM-synthetic path only. Confidence: medium; human review still required for high-stakes evals.
Pricing (directional): GPT-5.6 tiers (Sol/Terra/Luna) roughly $5/$30 to $1/$6 per 1M tokens; prompt caching with explicit breakpoints; batch discounts. Media (images/audio) billed separately.
Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.
Default when you want maximum capability and ecosystem glue. Multi-home critical paths behind OpenAI-compat so Anthropic/Gemini remain exit ramps.
Quote for Synthetic evaluation & scenario data
Depot Pick stays OpenAI. Prices below are sourced SKU rows only (as of 2026-10-27). Input and output are not blended. GET /api/quote?jtbd=synthetic-eval-data
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
- gpt-6-astra usd_per_million_input_tokens $20 source as of 2026-10-03
- gpt-6-astra usd_per_million_input_tokens $10 source as of 2026-10-03
- gpt-6-luna usd_per_million_input_tokens $0.2 source as of 2026-10-03
- gpt-6-luna usd_per_million_input_tokens $0.1 source as of 2026-10-03
- gpt-6.1-sol usd_per_million_input_tokens $4 source as of 2026-10-03
- gpt-6.1-sol usd_per_million_input_tokens $2 source as of 2026-10-03
- gpt-6-astra usd_per_million_output_tokens $75 source as of 2026-10-03
- gpt-6-astra usd_per_million_output_tokens $50 source as of 2026-10-03
- gpt-6-luna usd_per_million_output_tokens $0.75 source as of 2026-10-03
- gpt-6-luna usd_per_million_output_tokens $0.5 source as of 2026-10-03
- gpt-6.1-sol usd_per_million_output_tokens $15 source as of 2026-10-03
- gpt-6.1-sol usd_per_million_output_tokens $10 source as of 2026-10-03
1 of 5 candidates have no sourced price row. 5 are excluded from the Matching price comparison (none). no comparable sourced unit for this job; chat input $/1M and other units were not borrowed
Runners-up
Anthropic
Depot default for serious coding and production agents. Prefer Sonnet for cost/quality; reserve Opus for hard…
Google AI (Gemini)
Best value frontier stack and multimodal king. Use Flash for high-volume agent loops; Pro when quality…
Fireworks AI
Top pick for production open-model serving with real DX. Prefer over random GPU DIY when you want tokens not…
Promptfoo
Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI…
When NOT to use the top pick
Need open-model bulk generation (Fireworks), or Promptfoo red-team generators already cover the matrix.
Full ranking (score table)
Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.
| # | Provider | Overall | Fit | $/perf | Latency | DX | As of |
|---|---|---|---|---|---|---|---|
| 1 | Anthropic | 8.5 | 9.7 | 6.8 | 8.7 | 9.2 | 2026-10-19 |
| 2 | Google AI (Gemini) | 8.4 | 9.2 | 8.8 | 8.5 | 8.2 | 2026-10-19 |
| 3 | Promptfoo | 8.4 | 8.2 | 9.2 | 7.8 | 8.8 | 2026-10-19 |
| 4 | OpenAIDepot Pick | 8.3 | 9.6 | 6.2 | 8.8 | 9.5 | 2026-09-14 |
| 5 | Fireworks AI | 8.3 | 8.6 | 8.8 | 8.4 | 9.0 | 2026-09-14 |
Score methodology for this job
No Scale/Labelbox vendor in catalog — LLM-synthetic path only. Confidence: medium; human review still required for high-stakes evals.
capabilityFit: 32%pricePerformance: 12%latencyReliability: 14%developerDx: 12%privacyPosture: 12%lockInRisk: 8%maturity: 10%