Agent taxonomy

Autonomous or semi-autonomous agent that reads/writes code, runs tests, navigates repos, and lands patches or PRs against executable success criteria.

Good looks like: Correct patches that pass hidden tests; minimal unrelated churn; recovers from failing tests; respects repo conventions; cites file:line evidence; does not invent APIs.

11 seed tasks mention this type · official coverage gate 7 · tasks · leaderboard

Agent that gathers, synthesizes, and cites evidence from web/docs/corpora to answer open-ended or multi-hop questions with traceable grounding.

Good looks like: Answers match ground truth or rubric; every non-trivial claim has a source; distinguishes unknown vs known; no fabricated citations; clear uncertainty.

11 seed tasks mention this type · official coverage gate 7 · tasks · leaderboard

Agent that diagnoses incidents, inspects logs/metrics/traces, proposes or applies runbook actions, and communicates blast-radius-aware remediations under change-control constraints.

Good looks like: Correct root-cause hypotheses with evidence; safe action ordering; respects freeze/change windows; clear rollback; no destructive commands without confirmation gates.

11 seed tasks mention this type · official coverage gate 7 · tasks · leaderboard

Multi-turn agent that resolves user issues using tools/policies (orders, refunds, account state) while staying on-policy, empathetic, and escalation-aware.

Good looks like: Final DB/state matches allowed resolution; policy never violated; clarifies ambiguity; consistent across trials (pass^k); clean handoff notes.

11 seed tasks mention this type · official coverage gate 7 · tasks · leaderboard

Agent that explores datasets, writes correct SQL/pandas/Spark, validates assumptions, and produces decision-ready analyses with reproducible steps.

Good looks like: Correct numbers vs gold queries; schema-aware; documents filters/joins; flags data quality issues; no silent row drops; reproducible query artifacts.

10 seed tasks mention this type · official coverage gate 6 · tasks · leaderboard

Agent that operates GUIs or browsers (click, type, navigate, upload) to complete end-to-end workflows in web or desktop environments.

Good looks like: Task end-state verified by execution scripts; stable selectors/grounding; recovers from popups/noise; avoids loops; respects site ToS in live mode.

10 seed tasks mention this type · official coverage gate 6 · tasks · leaderboard

Real-time or turn-based spoken agent handling ASR noise, barge-in, latency budgets, and spoken dialogue policy for booking, support, or IVR-like flows.

Good looks like: Goal completed within latency SLO; graceful repair on ASR errors; confirms critical slots; never invents confirmations; clean escalation to human.

10 seed tasks mention this type · official coverage gate 6 · tasks · leaderboard

Controller that decomposes work, routes to specialist agents/tools, merges results, resolves conflicts, and produces a coherent final artifact with clear ownership.

Good looks like: Correct merge without duplicated work; explicit handoffs; conflict detection; budget adherence; failure isolation; auditable assignment log.

10 seed tasks mention this type · official coverage gate 6 · tasks · leaderboard

Broad personal/work assistant spanning scheduling, light research, writing, and tool use without deep specialization—GAIA-style real-world questions.

Good looks like: High success on short-answer verifiable tasks; good tool selection under ambiguity; honest when stuck; efficient path length.

16 seed tasks mention this type · official coverage gate 10 · tasks · leaderboard