{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","id":"support","name":"Support agent","definition":"Answers a customer from the operator's policy and escalates when sources disagree or the action is consequential.","needs":{"jtbdIds":["hitl-approval-workflows","content-moderation","transactional-email","rag-enterprise","knowledge-graph-rag"],"freeKinds":["free_docs"],"capabilityTags":["enterprise","rag"],"plain":["One canonical policy page per topic","A human approval step for refunds and exceptions"]},"failureModeIds":["policy-conflict","ungrounded-answer","stale-retrieval","identity-confusion","prompt-injection","over-refusal","confirmation-skip","non-determinism","weak-verification","schema-drift"],"exampleTasks":["Answer a policy question by quoting the canonical page","Show both pages when they disagree instead of picking one","Do not send a refund without an approval id"],"failures":[{"id":"policy-conflict","name":"Policy conflict","whatGoesWrong":"Two sources the operator owns disagree, and the agent states one of them as the policy.","howToDetect":"If cited sources contradict each other, return the conflict and the URLs. Do not pick a winner silently.","classIds":["support","sales-outreach","rag","research"],"sourceUrls":["https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do"]},{"id":"ungrounded-answer","name":"Ungrounded answer","whatGoesWrong":"The agent states a fact, a citation, or a policy with no retrieved support, or with a citation that does not contain the claim.","howToDetect":"Require a source URL or passage id on factual sentences. Drop or mark any sentence whose source does not contain the claim.","classIds":["rag","research","support","sales-outreach"],"sourceUrls":["https://www.langchain.com/blog/rebuilding-chat-langchain"]},{"id":"stale-retrieval","name":"Stale retrieval","whatGoesWrong":"The answer is grounded in a passage, index, or policy page that is older than the source the operator now treats as canonical.","howToDetect":"Store the source URL and the time it was fetched. If a newer canonical page exists, or two stored sources disagree, do not answer from the older one.","classIds":["rag","research","support","data-etl","personal-assistant","sales-outreach","local-offline"],"sourceUrls":["https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do"]},{"id":"identity-confusion","name":"Identity confusion","whatGoesWrong":"The agent acts for the wrong user or tenant, or a principal treats the agent as a separate party that the operator does not have to answer for.","howToDetect":"Bind every tool call to a user or tenant id. Refuse the call if that id changed mid-task. For a public bot, the operator owns the words the bot shows.","classIds":["support","personal-assistant","sales-outreach","orchestrator","voice"],"sourceUrls":["https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do"]},{"id":"prompt-injection","name":"Prompt injection","whatGoesWrong":"Untrusted text, including text from a page, file, or message the agent fetched, changes the agent's behavior away from the user's task. OWASP distinguishes a direct injection in the user's own prompt from an indirect injection in outside content. Neither form is fully removed by retrieval or fine-tuning, according to the OWASP LLM01:2025 page fetched today.","howToDetect":"Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.","classIds":["browser","computer-use","research","support","personal-assistant","security-review","sales-outreach"],"sourceUrls":["https://genai.owasp.org/llmrisk/llm01-prompt-injection/","https://www.anthropic.com/news/developing-computer-use","https://simonwillison.net/2023/Apr/14/worst-that-can-happen/"]},{"id":"over-refusal","name":"Over-refusal","whatGoesWrong":"The agent declines a task the written policy allows, or declines so often that the job never finishes.","howToDetect":"Log refusals separately from successes. Review a sample against the written policy. Do not treat a refusal as a pass, and do not weaken a safety check to raise the pass rate.","classIds":["support","security-review","voice","browser"],"sourceUrls":["https://openai.com/index/introducing-operator/"]},{"id":"confirmation-skip","name":"Confirmation skip","whatGoesWrong":"A consequential action, such as sending, paying, deleting, or submitting, runs without the confirmation the product policy requires.","howToDetect":"Gate those tool names on an approval id recorded before the call. A missing approval is a failure even if the downstream tool returns success.","classIds":["browser","sales-outreach","personal-assistant","support","computer-use","voice"],"sourceUrls":["https://openai.com/index/introducing-operator/"]},{"id":"non-determinism","name":"Non-determinism","whatGoesWrong":"The same task produces a different tool sequence or a different end state on another trial, so one success does not mean the agent is reliable.","howToDetect":"Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.","classIds":["support","coding","browser","scheduled-ops","eval-judge","voice"],"sourceUrls":["https://arxiv.org/abs/2406.12045"]},{"id":"weak-verification","name":"Weak verification","whatGoesWrong":"The run is marked successful because the agent said it was done, without an execution check, a test, or a second pass.","howToDetect":"Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.","classIds":["coding","eval-judge","orchestrator","support","security-review"],"sourceUrls":["https://arxiv.org/abs/2503.13657","https://arxiv.org/abs/2310.06770"]},{"id":"schema-drift","name":"Schema drift","whatGoesWrong":"A tool, table, or file changed shape, and the agent still sends the old arguments or reads the old columns.","howToDetect":"Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.","classIds":["coding","data-etl","local-offline","support"],"sourceUrls":[]}],"caseStudies":[{"id":"tau-bench-2024","title":"τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains","year":2024,"studyKind":"capability_ceiling","classIds":["support","voice"],"failureModeIds":["non-determinism","policy-conflict","weak-verification"],"lesson":"The abstract says τ-bench checks whether an agent follows domain rules while talking to a simulated user, and grades the database at the end. It reports that even a function-calling agent such as gpt-4o succeeded on under 50% of tasks and was inconsistent, with pass^8 under 25% in the retail domain. One green trial is not reliability.","sourceUrl":"https://arxiv.org/abs/2406.12045","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 17 Jun 2024). Capability ceiling for policy-following tool use, not a named company's outage."},{"id":"workarena-2024","title":"WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?","year":2024,"studyKind":"capability_ceiling","classIds":["browser","support"],"failureModeIds":["weak-verification"],"lesson":"The abstract introduces WorkArena as 33 tasks on ServiceNow and BrowserGym as the environment. It says agents showed promise and that a considerable gap remained before full task automation, with a disparity between open and closed models. No success percentage is stored because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2403.07718","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 12 Mar 2024). Enterprise knowledge-work ceiling, not a ServiceNow incident."},{"id":"moffatt-v-air-canada-2024","title":"Moffatt v. Air Canada, 2024 BCCRT 149","year":2024,"studyKind":"tribunal_decision","classIds":["support"],"failureModeIds":["policy-conflict","identity-confusion","ungrounded-answer","stale-retrieval"],"lesson":"The decision says a customer used Air Canada's website chatbot while looking up a bereavement fare. The chatbot said a reduced rate could be requested within 90 days, including after travel. The linked bereavement page said the policy did not apply after travel was complete. An Air Canada representative later called the chatbot's words misleading. The tribunal found negligent misrepresentation, rejected the argument that the chatbot was a separate legal entity, and ordered Air Canada to pay $812.02. A support agent must not choose between two of the operator's pages, and the operator owns the answer.","sourceUrl":"https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do","asOf":"2026-10-03","notes":"Fetched the tribunal document (PDF) dated February 14, 2024, Tribunal Member Christopher C. Rivers. The order is $650.88 damages, $36.14 pre-judgment interest, and $125 in CRT fees. CanLII's HTML page returned a block and was not used."},{"id":"langchain-chat-rebuild-2025","title":"Why we rebuilt LangChain's chatbot","year":2025,"studyKind":"vendor_writeup","classIds":["support","rag"],"failureModeIds":["ungrounded-answer","stale-retrieval"],"lesson":"LangChain's post says support engineers were not using their own docs chatbot because a docs-only answer was not enough for the questions they actually had, and that citations and context were fragmented while the index had to be rebuilt as docs changed. The rebuild described separate paths for ordinary questions and harder ones, plus middleware for guardrails, retries, model fallback, and caching. No latency or accuracy figure is stored here.","sourceUrl":"https://www.langchain.com/blog/rebuilding-chat-langchain","asOf":"2026-10-03","notes":"Fetched page byline: The LangChain Team, November 5, 2025. Descriptive of their rebuild, not an audited outage."}]}