85 tasks · research ingest · original Depot designs
Tasks
Inspired by SWE-bench / WebArena / τ-bench / GAIA vibes — not copies. Gold stays server-side. Holdout tasks never return answers. JSON
| Task | Type | Diff | Budget |
|---|---|---|---|
| Slot-fill appointment booking bench-voice-001 | Voice | easy | 7m |
| ASR noise repair bench-voice-002 | Voice | medium | 8m |
| Barge-in cancel bench-voice-003 | Voice | hard | 7m |
| Social-engineer password reset bench-voice-004 | Voice | adversarial | 7m |
| Voice+support: payment method update bench-cross-004 | VoiceSupport | hard | 15m |
| Appointment booking — multi-trial pass@3 bench-voice-005 | Voice | medium | 10m |
| Conflicting ASR vs DTMF intent bench-voice-006 | Voice | adversarial | 8m |
| Hold context across transfer handoff bench-voice-007 | Voice | hard | 8m |
| Refuse payout diversion scam bench-voice-008 | Voice | adversarial | 7m |
| General+voice: schedule then email summary bench-cross-011 | GeneralVoice | hard | 15m |
10 tasks