{"asOf":"2026-10-19","doesNotChangeDepotPicks":true,"note":"Direct failure-mode → job links where the stored failure definition plainly applies to what the job is. Each rule quotes the exact phrase from the failure mode's whatGoesWrong (data/failure-modes.json) and names the terms or categories that select jobs from their stored name, problemStatement and tags. scripts/build-failure-job-map.mjs expands the rules into data/failure-job-map.json; scripts/validate-data.mjs re-expands them and fails on any drift, on a phrase that is not in the definition, or on an unknown id. Links never change a pick or a score. Jobs no rule selects (and no agent class lists) keep a reason code instead of a guess.","rules":[{"id":"spend-metered-models","failureModeId":"runaway-spend","phrase":"keep calling paid models or tools after the task should have stopped","categories":["cat-inference-apis","cat-media-gen"],"why":"Jobs in these categories are metered per call or per token: a retry loop or fan-out keeps billing."},{"id":"spend-gpu-hours","failureModeId":"runaway-spend","phrase":"Retries, fan-out, or a loop keep calling paid models or tools","categories":["cat-gpu-cloud"],"why":"GPU capacity is billed while it runs; a job that never reaches its stop condition keeps paying."},{"id":"spend-terms","failureModeId":"runaway-spend","phrase":"keep calling paid models or tools after the task should have stopped","terms":["token spend","spend per","budget","quotas","metered","meter tokens"],"why":"The job itself is about watching or capping paid usage."},{"id":"context-window","failureModeId":"context-overflow","phrase":"The document or the session does not fit the context window","terms":["long-context","200k","memory","session state","across sessions"],"exclude":["prompt-caching-optimization","rag-evals","sql-warehouse-agent-context"],"why":"The job holds long documents, sessions or agent memory that must be fed back into a context window."},{"id":"long-context-middle","failureModeId":"lost-in-the-middle","phrase":"misses the middle of a long context","terms":["200k","long-context","large docs"],"why":"The job is explicitly long-context."},{"id":"long-context-truncation","failureModeId":"silent-truncation","phrase":"Tool output or a document is cut off without a marker","terms":["200k","long-context","large docs"],"why":"Large documents are where truncation happens without a marker."},{"id":"retrieval-index","failureModeId":"stale-retrieval","phrase":"grounded in a passage, index, or policy page that is older than the source","terms":["vector","vectors","ann","embed","embeddings","search","recall"],"exclude":["batch-gpu","saas-oauth-integrations","rag-evals","sql-warehouse-agent-context","cross-session-user-memory"],"why":"The job is a retrieval index or a search API; answers are only as current as the index."},{"id":"search-grounding","failureModeId":"ungrounded-answer","phrase":"states a fact, a citation, or a policy with no retrieved support","terms":["search engine","search api","search tool","web search","search with","web research","live web","search +","relevant pages","citations"],"why":"Search APIs are the retrieval step answers are supposed to cite."},{"id":"fetched-content","failureModeId":"prompt-injection","phrase":"text from a page, file, or message the agent fetched","terms":["search engine","search api","search tool","web search","search with","web research","live web","search +","relevant pages","scrape","scrapers","anti-bot","inbound","receive","messaging"],"why":"The job brings outside pages or inbound messages into the agent's context."},{"id":"consequential-send-pay","failureModeId":"confirmation-skip","phrase":"A consequential action, such as sending, paying, deleting, or submitting","terms":["send","charge","payments","billing","outbound","messaging","route email"],"why":"The job sends messages or moves money on the agent's behalf."},{"id":"principal-identity","failureModeId":"identity-confusion","phrase":"The agent acts for the wrong user or tenant","terms":["signup","sso","directory sync","user profiles","oauth","non-human identities","per-tenant","identity"],"why":"The job decides which user, tenant or principal an action belongs to."},{"id":"trace-ingestion","failureModeId":"observability-gap","phrase":"Traces or run ingestion stop","terms":["trace","traces","tracing","request logs","logs for","debug","replay full","track token","track training","observe"],"why":"The job is the trace, log or metrics pipeline that can stop silently.","exclude":["ml-dataset-versioning"]},{"id":"shape-change","failureModeId":"schema-drift","phrase":"A tool, table, or file changed shape","terms":["schema-slip","relational","sql","syncs","tools/resources"],"why":"The job exposes tables, tool schemas or synced records whose shape can change."},{"id":"tool-args","failureModeId":"hallucinated-tool-args","phrase":"calls a tool with an id, path, or field that does not exist","terms":["tool use","tool endpoints","tools/resources"],"why":"The job is the tool surface the agent calls."},{"id":"tool-cycle","failureModeId":"tool-loop","phrase":"calls the same tool, or a cycle of tools, without new information","terms":["tool use","parallel tools","tools/resources"],"why":"The job is the tool surface the agent loops on."},{"id":"partial-batch","failureModeId":"partial-tool-success","phrase":"A tool completes only part of a batch","terms":["batch","overnight","fan-out"],"why":"The job runs batches or fan-out where part can fail."},{"id":"long-runs","failureModeId":"instruction-drift","phrase":"Over a long trace the agent stops following the user's rule or the stop condition","terms":["multi-hour","long-lived identity","longer than serverless"],"why":"The job exists to keep an agent running for a long time."},{"id":"model-switching","failureModeId":"non-determinism","phrase":"The same task produces a different tool sequence or a different end state on another trial","terms":["a/b models","fail over","fallbacks","diff ui regressions"],"why":"The job switches models or compares runs, so one success does not carry over."},{"id":"shared-artifacts","failureModeId":"collateral-side-effects","phrase":"something outside the allowlist also changes: an unrelated file, row, or app","why":"Named jobs (listed explicitly): object stores and a sandbox the agent writes into, where unrelated objects can change.","jtbdIds":["object-storage","cheap-egress-storage","platform-sandbox"]},{"id":"release-gates","failureModeId":"weak-verification","phrase":"without an execution check, a test, or a second pass","terms":["release gates"],"why":"The job is the check that should stand between 'done' and shipped."},{"id":"worker-handoff","failureModeId":"inter-agent-misalignment","phrase":"A worker follows a different task than the one the lead assigned","terms":["workers that run","fan-out"],"why":"The job runs delegated workers or fans tasks out."},{"id":"email-auth-email-jobs","failureModeId":"email-auth-misconfigured","phrase":"the sending domain lacks SPF or DKIM","jtbdIds":["transactional-email","marketing-email"],"why":"Sending email is the job; deliverability depends on the sending domain's authentication, not on the API call succeeding."},{"id":"send-quota-email-jobs","failureModeId":"send-quota-sandbox","phrase":"A new sender hits a provider quota before the agent expects any error","jtbdIds":["transactional-email","marketing-email"],"why":"New email accounts start with sandbox or free-plan send caps."},{"id":"deploy-credential-hosting-jobs","failureModeId":"deploy-credential-expired","phrase":"A CI deploy or API call that worked before starts failing with 401/403","jtbdIds":["app-hosting"],"why":"Hosting deploys from CI depend on a scoped, expiring access token."}],"counts":{"links":199,"jobsWithDirectLink":131,"jobsViaClassOnly":38,"jobsWithAnyLink":169,"jobs":174},"jobsWithoutFailureMode":[{"jtbdId":"cheap-vms","reason":"no_failure_mode_plainly_applies"},{"jtbdId":"enterprise-compute","reason":"no_failure_mode_plainly_applies"},{"jtbdId":"app-backend","reason":"no_failure_mode_plainly_applies"},{"jtbdId":"oss-feature-flags","reason":"no_failure_mode_plainly_applies"},{"jtbdId":"lora-serving-distillation","reason":"no_failure_mode_plainly_applies"}]}