Mechanisms · 2026-10-03
Failure modes
What goes wrong, how to notice it, and which agent types it hits. Detection is a check on the trace, not a procedure for causing the failure. This catalog does not change a Depot Pick.
GET /api/failures · 28 modes · types · case studies
Prompt injection
prompt-injection
Untrusted text, including text from a page, file, or message the agent fetched, changes the agent's behavior away from the user's task. OWASP distinguishes a direct injection in the user's own prompt from an indirect injection in outside content. Neither form is fully removed by retrieval or fine-tuning, according to the OWASP LLM01:2025 page fetched today.
Detect: Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.
Classes: browser, computer-use, research, support, personal-assistant, security-review, sales-outreach
Studies for this mode
Tool loop
tool-loop
The agent calls the same tool, or a cycle of tools, without new information, so the task never reaches a stop condition.
Detect: Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.
Classes: coding, research, browser, computer-use, orchestrator, scheduled-ops
Studies for this mode
Stale retrieval
stale-retrieval
The answer is grounded in a passage, index, or policy page that is older than the source the operator now treats as canonical.
Detect: Store the source URL and the time it was fetched. If a newer canonical page exists, or two stored sources disagree, do not answer from the older one.
Classes: rag, research, support, data-etl, personal-assistant, sales-outreach, local-offline
Studies for this mode
Schema drift
schema-drift
A tool, table, or file changed shape, and the agent still sends the old arguments or reads the old columns.
Detect: Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.
Classes: coding, data-etl, local-offline, support
Studies for this mode
Partial tool success
partial-tool-success
A tool completes only part of a batch and the agent treats the whole batch as done.
Detect: Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.
Classes: coding, data-etl, browser, computer-use, voice, orchestrator, scheduled-ops
Studies for this mode
Runaway spend
runaway-spend
Retries, fan-out, or a loop keep calling paid models or tools after the task should have stopped.
Detect: Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
Classes: coding, research, orchestrator, scheduled-ops, sales-outreach, voice
Studies for this mode
Identity confusion
identity-confusion
The agent acts for the wrong user or tenant, or a principal treats the agent as a separate party that the operator does not have to answer for.
Detect: Bind every tool call to a user or tenant id. Refuse the call if that id changed mid-task. For a public bot, the operator owns the words the bot shows.
Classes: support, personal-assistant, sales-outreach, orchestrator, voice
Studies for this mode
Over-refusal
over-refusal
The agent declines a task the written policy allows, or declines so often that the job never finishes.
Detect: Log refusals separately from successes. Review a sample against the written policy. Do not treat a refusal as a pass, and do not weaken a safety check to raise the pass rate.
Classes: support, security-review, voice, browser
Studies for this mode
Silent truncation
silent-truncation
Tool output or a document is cut off without a marker, and the agent answers as if it saw the whole thing.
Detect: Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
Classes: rag, coding, data-etl, voice, local-offline, research
Studies for this mode
Non-determinism
non-determinism
The same task produces a different tool sequence or a different end state on another trial, so one success does not mean the agent is reliable.
Detect: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
Classes: support, coding, browser, scheduled-ops, eval-judge, voice
Studies for this mode
Eval contamination
eval-contamination
A score is not comparable to the one it is cited against: the gold label or the issue text leaked into the prompt, the harness rejects a valid fix, or a revised benchmark is treated as the old one.
Detect: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
Classes: eval-judge, coding, rag, security-review
Studies for this mode
Lost in the middle
lost-in-the-middle
The needed passage is in the prompt, but the model uses the beginning or the end and misses the middle of a long context.
Detect: Ask the agent to name the passage it used. Spot-check questions whose only support sits in the middle of the context, not only at the edges.
Classes: rag, research, local-offline
Studies for this mode
Hallucinated tool arguments
hallucinated-tool-args
The agent calls a tool with an id, path, or field that does not exist.
Detect: Reject a call that fails the current schema or names an unknown id, and do that before the tool runs a side effect.
Classes: coding, data-etl, browser, orchestrator
Studies for this mode
Collateral side effects
collateral-side-effects
The stated task ends in the desired state, and something outside the allowlist also changes: an unrelated file, row, or app.
Detect: Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.
Classes: coding, computer-use, data-etl, browser
Studies for this mode
Policy conflict
policy-conflict
Two sources the operator owns disagree, and the agent states one of them as the policy.
Detect: If cited sources contradict each other, return the conflict and the URLs. Do not pick a winner silently.
Classes: support, sales-outreach, rag, research
Studies for this mode
Inter-agent misalignment
inter-agent-misalignment
A worker follows a different task than the one the lead assigned, or the lead merges results that do not answer the assignment.
Detect: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
Classes: orchestrator, research, eval-judge
Studies for this mode
Weak verification
weak-verification
The run is marked successful because the agent said it was done, without an execution check, a test, or a second pass.
Detect: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Classes: coding, eval-judge, orchestrator, support, security-review
Studies for this mode
GUI grounding miss
gui-grounding-miss
A click, keystroke, or scroll lands on the wrong control, so later steps run against a screen the agent did not mean to open.
Detect: After each GUI action, compare the resulting screen or accessibility tree to the control the plan named. Stop when they differ.
Classes: computer-use, browser
Studies for this mode
Observability gap
observability-gap
Traces or run ingestion stop, and the failure is outside the agent's reasoning. A certificate or DNS fault can make the trace API fail while agents keep acting.
Detect: Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
Classes: eval-judge, scheduled-ops, orchestrator
Studies for this mode
Context overflow
context-overflow
The document or the session does not fit the context window, and the agent answers from whatever fragment was left in view.
Detect: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
Classes: rag, personal-assistant, research, local-offline, coding
Studies for this mode
Benchmark protocol mismatch
benchmark-protocol-mismatch
A score is cited next to another score that used a different setup: assisted versus unassisted editing, a subset versus the full set, or a revised harness versus the original.
Detect: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
Classes: eval-judge, coding, computer-use, browser
Studies for this mode
Confirmation skip
confirmation-skip
A consequential action, such as sending, paying, deleting, or submitting, runs without the confirmation the product policy requires.
Detect: Gate those tool names on an approval id recorded before the call. A missing approval is a failure even if the downstream tool returns success.
Classes: browser, sales-outreach, personal-assistant, support, computer-use, voice
Studies for this mode
Instruction drift
instruction-drift
Over a long trace the agent stops following the user's rule or the stop condition, including continuing after it already has enough information.
Detect: Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.
Classes: research, orchestrator, coding, browser, personal-assistant
Studies for this mode
Ungrounded answer
ungrounded-answer
The agent states a fact, a citation, or a policy with no retrieved support, or with a citation that does not contain the claim.
Detect: Require a source URL or passage id on factual sentences. Drop or mark any sentence whose source does not contain the claim.
Classes: rag, research, support, sales-outreach
Studies for this mode
Email authentication or sender requirements missing
email-auth-misconfigured
Mail is junked or rejected because the sending domain lacks SPF or DKIM, valid forward and reverse DNS (PTR) records, or a TLS connection. Google's sender guidelines (fetched 2026-10-21) require these for all senders to Gmail since February 1, 2024, ask senders to keep spam rates reported in Postmaster Tools below 0.3%, and add more requirements above 5,000 messages a day.
Detect: Before the first real send, confirm the provider shows the sending domain verified (SPF/DKIM found) and that a DMARC record exists. After sending, watch bounces, complaints and the Postmaster Tools spam rate; a send API returning 200 does not mean the mail reached the inbox.
Classes: sales-outreach, support, personal-assistant
Studies for this mode
Send quota or sandbox limit hit
send-quota-sandbox
A new sender hits a provider quota before the agent expects any error. Amazon SES places all new accounts in a sandbox: mail only to verified addresses or domains, a maximum of 200 messages per 24-hour period and 1 message per second. Resend Free has a daily limit of 100 emails; Postmark Free allows 100 emails a month with no overages.
Detect: Read the account's current quota before a batch (SES GetAccount / sending quota; provider dashboard) and treat quota errors as a stop, not a retry. Move out of the sandbox or upgrade before launch; a retry loop against a quota only adds failures.
Classes: sales-outreach, support, personal-assistant
Studies for this mode
Deploy or API credential expired or out of scope
deploy-credential-expired
A CI deploy or API call that worked before starts failing with 401/403 because the access token expired, was created with a scope that does not cover the target, or went inactive. Vercel tokens carry a chosen expiration (1 day to 1 year) and a scope (full account, team or project); a project-scoped token is denied for other projects and team resources, and a CLI login token unused for 10 days expires.
Detect: Treat 401/403 from the deploy step as a credential problem, not a code problem: check the token's expiration and scope on the provider's tokens page, pass teamId/slug when a full-account token targets a team, then rotate the CI secret. Retrying with the same token only repeats the failure.
Classes: coding, scheduled-ops
Studies for this mode
Home-made bot filter beaten by link scanners
homemade-bot-filter-beaten-by-scanners
A private link is shared by email or chat and a home-made rule (user-agent list, timing, header checks) decides human vs bot. Mail security scanners fetch links themselves: Microsoft's Safe Links does URL scanning of inbound mail during mail flow and time-of-click verification, and its recommended setting holds messages until URL scanning is finished. So a link can be fetched by a scanner, not only by the person it was sent to, and self-reported fields such as the user-agent are not proof of who is calling (Cloudflare treats a bot as verified only when it identifies itself by a signature, a published IP list or reverse DNS). Rules built on such fields get beaten by scanners and count them as visits.
Detect: Count visits that arrive within seconds of the message being sent, from many networks, with no follow-up interaction. If a 'human' signal is only a header or a GET request, treat it as unproven. Require a server-verified challenge token (Turnstile, Friendly Captcha or ALTCHA proof-of-work) or an explicit click on the page before counting a person.
Classes: personal-assistant, scheduled-ops, security-review
Studies for this mode