{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","id":"rag","name":"RAG agent","definition":"Answers from a retrieved corpus and says when the corpus does not support the claim.","needs":{"jtbdIds":["rag-enterprise","rag-memory","hybrid-search-rag","rag-rerank","prototype-rag","knowledge-graph-rag","late-interaction-retrieval","embeddings","managed-vector","rag-evals"],"freeKinds":["free_embedding","free_vector","free_rerank","free_data"],"capabilityTags":["rag","vector"],"plain":["Passage ids on every factual sentence"]},"failureModeIds":["stale-retrieval","lost-in-the-middle","silent-truncation","eval-contamination","ungrounded-answer","context-overflow","policy-conflict"],"exampleTasks":["Answer only from retrieved passages","Say the corpus has no support instead of filling the gap"],"failures":[{"id":"stale-retrieval","name":"Stale retrieval","whatGoesWrong":"The answer is grounded in a passage, index, or policy page that is older than the source the operator now treats as canonical.","howToDetect":"Store the source URL and the time it was fetched. If a newer canonical page exists, or two stored sources disagree, do not answer from the older one.","classIds":["rag","research","support","data-etl","personal-assistant","sales-outreach","local-offline"],"sourceUrls":["https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do"]},{"id":"lost-in-the-middle","name":"Lost in the middle","whatGoesWrong":"The needed passage is in the prompt, but the model uses the beginning or the end and misses the middle of a long context.","howToDetect":"Ask the agent to name the passage it used. Spot-check questions whose only support sits in the middle of the context, not only at the edges.","classIds":["rag","research","local-offline"],"sourceUrls":["https://arxiv.org/abs/2307.03172"]},{"id":"silent-truncation","name":"Silent truncation","whatGoesWrong":"Tool output or a document is cut off without a marker, and the agent answers as if it saw the whole thing.","howToDetect":"Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.","classIds":["rag","coding","data-etl","voice","local-offline","research"],"sourceUrls":["https://arxiv.org/abs/2310.08560"]},{"id":"eval-contamination","name":"Eval contamination","whatGoesWrong":"A score is not comparable to the one it is cited against: the gold label or the issue text leaked into the prompt, the harness rejects a valid fix, or a revised benchmark is treated as the old one.","howToDetect":"Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.","classIds":["eval-judge","coding","rag","security-review"],"sourceUrls":["https://openai.com/index/introducing-swe-bench-verified/","https://os-world.github.io/"]},{"id":"ungrounded-answer","name":"Ungrounded answer","whatGoesWrong":"The agent states a fact, a citation, or a policy with no retrieved support, or with a citation that does not contain the claim.","howToDetect":"Require a source URL or passage id on factual sentences. Drop or mark any sentence whose source does not contain the claim.","classIds":["rag","research","support","sales-outreach"],"sourceUrls":["https://www.langchain.com/blog/rebuilding-chat-langchain"]},{"id":"context-overflow","name":"Context overflow","whatGoesWrong":"The document or the session does not fit the context window, and the agent answers from whatever fragment was left in view.","howToDetect":"If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.","classIds":["rag","personal-assistant","research","local-offline","coding"],"sourceUrls":["https://arxiv.org/abs/2310.08560"]},{"id":"policy-conflict","name":"Policy conflict","whatGoesWrong":"Two sources the operator owns disagree, and the agent states one of them as the policy.","howToDetect":"If cited sources contradict each other, return the conflict and the URLs. Do not pick a winner silently.","classIds":["support","sales-outreach","rag","research"],"sourceUrls":["https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do"]}],"caseStudies":[{"id":"lost-in-the-middle-2023","title":"Lost in the Middle: How Language Models Use Long Contexts","year":2023,"studyKind":"mechanism_study","classIds":["rag","research"],"failureModeIds":["lost-in-the-middle"],"lesson":"The abstract says that on multi-document question answering and key-value retrieval, performance is often highest when the relevant information is at the beginning or the end of the context, and degrades when it is in the middle, including for models billed as long-context. Putting the best passage in the middle of a prompt is not the same as the model using it.","sourceUrl":"https://arxiv.org/abs/2307.03172","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment: accepted to TACL 2023. Not a vendor incident."},{"id":"memgpt-2023","title":"MemGPT: Towards LLMs as Operating Systems","year":2023,"studyKind":"mechanism_study","classIds":["personal-assistant","rag","local-offline"],"failureModeIds":["context-overflow","silent-truncation"],"lesson":"The abstract says limited context windows handicap document analysis and multi-session chat, and describes moving text between tiers so the model sees a working set rather than the whole history. An assistant that answers a long document or a long memory from a single truncated window is the problem this paper is about. No accuracy number is stored because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2310.08560","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. A system paper, not an incident."},{"id":"langchain-chat-rebuild-2025","title":"Why we rebuilt LangChain's chatbot","year":2025,"studyKind":"vendor_writeup","classIds":["support","rag"],"failureModeIds":["ungrounded-answer","stale-retrieval"],"lesson":"LangChain's post says support engineers were not using their own docs chatbot because a docs-only answer was not enough for the questions they actually had, and that citations and context were fragmented while the index had to be rebuilt as docs changed. The rebuild described separate paths for ordinary questions and harder ones, plus middleware for guardrails, retries, model fallback, and caching. No latency or accuracy figure is stored here.","sourceUrl":"https://www.langchain.com/blog/rebuilding-chat-langchain","asOf":"2026-10-03","notes":"Fetched page byline: The LangChain Team, November 5, 2025. Descriptive of their rebuild, not an audited outage."}]}