{"id":"adversarial-red-team-evals","slug":"adversarial-red-team-evals","name":"Adversarial red-team evaluation suites","shortName":"Red-team evals","problemStatement":"Continuously attack prompts/agents with jailbreak corpora and safety suites before and after each release.","whenNotTopPick":"Patronus safety SKUs preferred, or Confident/DeepEval already embedded.","categoryIds":["cat-observability"],"depotPickProviderId":"promptfoo","depotPickRationale":"Distinct from `continuous-eval-ci` (quality gates) — safety adversarial focus. Confidence: high for Promptfoo red-team.","runnersUpIds":["patronus","confident-ai","galileo","lakera"],"weights":{"capabilityFit":0.32,"pricePerformance":0.12,"latencyReliability":0.14,"developerDx":0.12,"privacyPosture":0.12,"lockInRisk":0.08,"maturity":0.1},"methodologyNotes":"Distinct from `continuous-eval-ci` (quality gates) — safety adversarial focus. Confidence: high for Promptfoo red-team.","tags":["red-team","safety","evals","security"],"depth":{"whenNotToUse":{"text":"Patronus safety SKUs preferred, or Confident/DeepEval already embedded.","source":"editorial `whenNotTopPick` (data/jtbds.json)"},"whenNotToUseReason":null,"runnerUp":{"providerId":"patronus","name":"Patronus AI","overallForJob":7.46,"pickOverallForJob":8.36,"beatsPickOn":["capabilityFit","latencyReliability"],"source":"first resolvable entry of the job's editorially ranked `runnersUpIds`; scores are stored job-weighted scores"},"runnerUpReason":null,"lockIn":{"pickLockInRisk":2.5,"runnerUpLockInRisk":5.5,"scale":"0–10 stored curated estimate; higher = more lock-in (worse)","pickExitSignals":["open source (provider tag `oss`)"],"runnerUpExitSignals":[],"pickNotes":[],"selfHostFallback":[{"id":"promptfoo","name":"Promptfoo","license":"MIT","sourceUrl":"https://github.com/promptfoo/promptfoo/blob/main/LICENSE"}],"migration":"The pick has lower stored lock-in risk than runner-up Patronus AI (2.5 vs 5.5). Pick exit signals: open source (provider tag `oss`). Self-host fallback listed for this job: Promptfoo.","migrationReason":null,"derivedFrom":"stored lockInRisk scores, provider tags (oss, self-host, openai-compat, open-models), free rows that list this job, and verbatim provider notes. No migration guide was invented."}},"freshness":{"pick":{"score":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"lastReviewed":{"asOf":"2026-10-19","ageDays":8,"stale":false,"staleAfterDays":30,"reason":null},"sourceCheck":{"checkedAt":"2026-10-27","sourceUrls":3,"sourcesReachable":3,"pricingRows":0,"pricingRowsReconfirmed":0,"note":"A source re-check confirms the page is reachable and, for pricing rows, that the stored amount and SKU name appear on it. It does not re-derive or change the score.","reason":null}}},"depotPick":{"id":"promptfoo","slug":"promptfoo","name":"Promptfoo","tagline":"Default OSS eval harness. Braintrust when you want hosted collaboration; Promptfoo when you want local/CI…","overall":8.36},"runnersUp":[{"id":"patronus","slug":"patronus","name":"Patronus AI","tagline":"Layer on when regulated hallucination/safety gates matter more than general CI evals. Lakera for injection…","overall":7.46},{"id":"confident-ai","slug":"confident-ai","name":"Confident AI","tagline":"Best when your team already thinks in pytest/DeepEval. Braintrust remains Depot hosted-evals default…","overall":7.98},{"id":"galileo","slug":"galileo","name":"Galileo","tagline":"Pick for production hallucination measurement + guardrails. Braintrust/Promptfoo for lighter eval loops.","overall":7.55},{"id":"lakera","slug":"lakera","name":"Lakera Guard","tagline":"Depot pick for a dedicated prompt-injection firewall in front of any LLM. NeMo when you need programmable…","overall":8.22}]}