Automated Prompt, Few-Shot & Agent Program Optimization (DSPy / MIPROv2 / TextGrad)
Compile and optimize system instructions, tool descriptions, and multi-hop few-shot demonstrations against a deterministic metric and golden dataset — replacing manual prompt trial-and-error with versioned, eval-gated prompt compilation.
At a glance — Oct 2026
Updated 2026-10-28 · forward this page: /jtbds/prompt-program-self-optimization#briefPromotion class: deterministic_source_reverified · 8.3 ± 0.5 (7.8–8.8 · editorial_prior) · Pricing verified 2026-10-28.
- Depot Pick
- Braintrust · 8.27/10 for this job (curated estimate, high confidence)
Braintrust pairs versioned prompt/tool playgrounds directly with deterministic and LLM-judge dataset scorers, diffing every candidate instruction or few-shot bootstrap (including DSPy MIPROv2 / TextGrad compiled artifacts) against held-out regression slices before promotion.
- Runner-up
- Promptfoo · 8.40/10
Scores higher than the pick on: Price / performance, Privacy / data retention, Lock-in risk (higher = worse).
- When not to use the pick
- Local CLI / YAML matrix optimization and adversarial red-teaming in CI with zero SaaS dependency — Promptfoo; LangGraph node-level prompt versioning and hub playgrounds — LangSmith; open-source MIT prompt management + dataset experiment linkage — Langfuse.
- Lock-in / migration
- Runner-up Promptfoo has lower stored lock-in risk (2.5 vs 5, 0–10, higher = more lock-in).
- Sourced price (no comparable unit)
Pick: no sourced price in this job's unit (no_sourced_price).
No cheapest SKU (no_comparable_unit_for_job). No price was invented.
- Free option (runnable / substitutable)
- No runnable free tier lists this job. A cheap paid SKU or benchmark dataset is not a runnable free option.
- Failure modes to watch
- No failure mode plainly applies to this job (no_failure_mode_plainly_applies).
Cite: Agent Provider Depot, Depot Pick for prompt-program-self-optimization: braintrust (editorial prior, scores as of 2026-10-28). Agents: GET /api/decide?jtbd=prompt-program-self-optimization
Braintrust
Braintrust pairs versioned prompt/tool playgrounds directly with deterministic and LLM-judge dataset scorers, diffing every candidate instruction or few-shot bootstrap (including DSPy MIPROv2 / TextGrad compiled artifacts) against held-out regression slices before promotion.
Pricing (directional): Eval-first platform; data-based pricing with unlimited seats on many plans; strong CI eval story.
Top pick (8.27) and #2 runner-up (8.4) are separated by 0.13 points (<= 0.35 weight-sensitivity band). Treat them as co-picks and use the workload tie-breaker rule below rather than the raw composite decimal.
Workload Tie-Breaker Rule: Local CLI / YAML matrix optimization and adversarial red-teaming in CI with zero SaaS dependency — Promptfoo; LangGraph node-level prompt versioning and hub playgrounds — LangSmith; open-source MIT prompt management + dataset experiment linkage — Langfuse.
Lock-in risk: higher = more lock-in (worse). Overall already inverts this dimension.
Depot pick when evals are the product quality gate. Pair with Helicone/Langfuse for cheap request logs if needed.
Quote for Automated Prompt, Few-Shot & Agent Program Optimization (DSPy / MIPROv2 / TextGrad)
Depot Pick stays Braintrust. Prices below are sourced SKU rows only (as of 2026-10-28). Input and output are not blended. GET /api/quote?jtbd=prompt-program-self-optimization
Nothing was compared. no comparable sourced unit for this job; chat input $/1M and other units were not borrowed pickIsPriceWinner is null.
Sourced price unavailable for the Depot Pick. No row in the pricing index.
4 of 4 candidates have no sourced price row. 4 are excluded from the Matching price comparison (none). no comparable sourced unit for this job; chat input $/1M and other units were not borrowed
Runners-up
Promptfoo
Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI…
LangSmith
If you live in LangGraph, LangSmith is the obvious observatory. Otherwise compare Langfuse/Helicone first.
Langfuse
Depot default observability for most agent teams: open, affordable, good enough to serious.
When NOT to use the top pick
Local CLI / YAML matrix optimization and adversarial red-teaming in CI with zero SaaS dependency — Promptfoo; LangGraph node-level prompt versioning and hub playgrounds — LangSmith; open-source MIT prompt management + dataset experiment linkage — Langfuse.
Full ranking (score table)
Ranked with this JTBD's weights. Scores are curated estimates unless marked otherwise. Agents: prefer /api/decide then this table for tie-breaks.
| # | Provider | Overall | Fit | $/perf | Latency | DX | Lock-In Risk | As of |
|---|---|---|---|---|---|---|---|---|
| 1 | Langfuse | 8.7 | 8.8 | 9.2 | 8.4 | 8.8 | 2.5 | 2026-10-28 |
| 2 | Promptfoo | 8.4 | 8.2 | 9.2 | 7.8 | 8.8 | 2.5 | 2026-10-28 |
| 3 | BraintrustDepot Pick | 8.3 | 9.2 | 7.5 | 8.6 | 8.8 | 5.0 | 2026-10-28 |
| 4 | LangSmith | 8.1 | 9.0 | 7.0 | 8.8 | 9.0 | 6.5 | 2026-10-28 |
Score methodology for this job
Never optimize prompts on the same split you report scores on: require a 3-way train / validation / held-out test split and reject any candidate prompt that regresses existing golden assertions (`regressionCount === 0`). Reference frameworks: Stanford DSPy (`MIPROv2`, `BootstrapFewShotWithRandomSearch`) and TextGrad.
capabilityFit: 28%pricePerformance: 14%latencyReliability: 14%developerDx: 18%privacyPosture: 10%lockInRisk: 8%maturity: 8%