{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","id":"browser","name":"Browser agent","definition":"Completes a task by reading and operating a website in a browser, then stops before a consequential submit unless confirmation exists.","needs":{"jtbdIds":["cloud-browser","computer-use-browser","browser-agent-loop","web-agent","self-host-browser","web-scrape"],"freeKinds":["free_browser"],"capabilityTags":["browser","web"],"plain":["Confirmation before submit, pay, or send"]},"failureModeIds":["prompt-injection","tool-loop","gui-grounding-miss","partial-tool-success","non-determinism","confirmation-skip","hallucinated-tool-args","collateral-side-effects","over-refusal","instruction-drift","benchmark-protocol-mismatch"],"exampleTasks":["Find a stated price on a page and quote the URL","Fill a form on a test site and stop before submit","Refuse to continue when the page text tries to replace the user's task"],"failures":[{"id":"prompt-injection","name":"Prompt injection","whatGoesWrong":"Untrusted text, including text from a page, file, or message the agent fetched, changes the agent's behavior away from the user's task. OWASP distinguishes a direct injection in the user's own prompt from an indirect injection in outside content. Neither form is fully removed by retrieval or fine-tuning, according to the OWASP LLM01:2025 page fetched today.","howToDetect":"Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.","classIds":["browser","computer-use","research","support","personal-assistant","security-review","sales-outreach"],"sourceUrls":["https://genai.owasp.org/llmrisk/llm01-prompt-injection/","https://www.anthropic.com/news/developing-computer-use","https://simonwillison.net/2023/Apr/14/worst-that-can-happen/"]},{"id":"tool-loop","name":"Tool loop","whatGoesWrong":"The agent calls the same tool, or a cycle of tools, without new information, so the task never reaches a stop condition.","howToDetect":"Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.","classIds":["coding","research","browser","computer-use","orchestrator","scheduled-ops"],"sourceUrls":["https://www.anthropic.com/engineering/built-multi-agent-research-system"]},{"id":"gui-grounding-miss","name":"GUI grounding miss","whatGoesWrong":"A click, keystroke, or scroll lands on the wrong control, so later steps run against a screen the agent did not mean to open.","howToDetect":"After each GUI action, compare the resulting screen or accessibility tree to the control the plan named. Stop when they differ.","classIds":["computer-use","browser"],"sourceUrls":["https://arxiv.org/abs/2404.07972"]},{"id":"partial-tool-success","name":"Partial tool success","whatGoesWrong":"A tool completes only part of a batch and the agent treats the whole batch as done.","howToDetect":"Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.","classIds":["coding","data-etl","browser","computer-use","voice","orchestrator","scheduled-ops"],"sourceUrls":[]},{"id":"non-determinism","name":"Non-determinism","whatGoesWrong":"The same task produces a different tool sequence or a different end state on another trial, so one success does not mean the agent is reliable.","howToDetect":"Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.","classIds":["support","coding","browser","scheduled-ops","eval-judge","voice"],"sourceUrls":["https://arxiv.org/abs/2406.12045"]},{"id":"confirmation-skip","name":"Confirmation skip","whatGoesWrong":"A consequential action, such as sending, paying, deleting, or submitting, runs without the confirmation the product policy requires.","howToDetect":"Gate those tool names on an approval id recorded before the call. A missing approval is a failure even if the downstream tool returns success.","classIds":["browser","sales-outreach","personal-assistant","support","computer-use","voice"],"sourceUrls":["https://openai.com/index/introducing-operator/"]},{"id":"hallucinated-tool-args","name":"Hallucinated tool arguments","whatGoesWrong":"The agent calls a tool with an id, path, or field that does not exist.","howToDetect":"Reject a call that fails the current schema or names an unknown id, and do that before the tool runs a side effect.","classIds":["coding","data-etl","browser","orchestrator"],"sourceUrls":["https://arxiv.org/abs/2308.03688"]},{"id":"collateral-side-effects","name":"Collateral side effects","whatGoesWrong":"The stated task ends in the desired state, and something outside the allowlist also changes: an unrelated file, row, or app.","howToDetect":"Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.","classIds":["coding","computer-use","data-etl","browser"],"sourceUrls":["https://arxiv.org/abs/2407.18901"]},{"id":"over-refusal","name":"Over-refusal","whatGoesWrong":"The agent declines a task the written policy allows, or declines so often that the job never finishes.","howToDetect":"Log refusals separately from successes. Review a sample against the written policy. Do not treat a refusal as a pass, and do not weaken a safety check to raise the pass rate.","classIds":["support","security-review","voice","browser"],"sourceUrls":["https://openai.com/index/introducing-operator/"]},{"id":"instruction-drift","name":"Instruction drift","whatGoesWrong":"Over a long trace the agent stops following the user's rule or the stop condition, including continuing after it already has enough information.","howToDetect":"Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.","classIds":["research","orchestrator","coding","browser","personal-assistant"],"sourceUrls":["https://www.anthropic.com/engineering/built-multi-agent-research-system","https://arxiv.org/abs/2308.03688"]},{"id":"benchmark-protocol-mismatch","name":"Benchmark protocol mismatch","whatGoesWrong":"A score is cited next to another score that used a different setup: assisted versus unassisted editing, a subset versus the full set, or a revised harness versus the original.","howToDetect":"Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.","classIds":["eval-judge","coding","computer-use","browser"],"sourceUrls":["https://www.cognition.ai/blog/swe-bench-technical-report","https://openai.com/index/introducing-swe-bench-verified/","https://os-world.github.io/"]}],"caseStudies":[{"id":"webarena-2024","title":"WebArena: A Realistic Web Environment for Building Autonomous Agents","year":2024,"studyKind":"capability_ceiling","classIds":["browser"],"failureModeIds":["weak-verification","instruction-drift"],"lesson":"The abstract describes functional sites for e-commerce, forums, collaborative software development, and content management, plus long-horizon tasks. It reports a best GPT-4-based agent at 14.41% end-to-end success and human performance at 78.24%. Use it as a web-agent ceiling, not as evidence about a named vendor's production site.","sourceUrl":"https://arxiv.org/abs/2307.13854","asOf":"2026-10-03","notes":"Fetched arXiv v4, last revised 16 Apr 2024. Submitted 25 Jul 2023."},{"id":"visualwebarena-2024","title":"VisualWebArena","year":2024,"studyKind":"capability_ceiling","classIds":["browser","computer-use"],"failureModeIds":["gui-grounding-miss"],"lesson":"The abstract says many web tasks need visual information that text-only agents do not use well, and that the benchmark is visually grounded. A browser agent that reads only the DOM or the HTML is being measured on a different task than this one. No success rate from the abstract is repeated here because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2401.13649","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: accepted to ACL 2024."},{"id":"openai-cua-2025","title":"Computer-Using Agent","year":2025,"studyKind":"vendor_writeup","classIds":["computer-use","browser"],"failureModeIds":["benchmark-protocol-mismatch","gui-grounding-miss"],"lesson":"OpenAI's page says Computer-Using Agent drives a GUI from screenshots and reports 38.1% on OSWorld, 58.1% on WebArena, and 87% on WebVoyager. Those are the vendor's reported scores for that model. They are not a re-run of the original papers' baselines, and the page says the model is early and has limitations.","sourceUrl":"https://openai.com/index/computer-using-agent/","asOf":"2026-10-03","notes":"Fetched the OpenAI page dated January 23, 2025 in the article header. Depot did not re-run OSWorld, WebArena, or WebVoyager."},{"id":"openai-operator-2025","title":"Introducing Operator","year":2025,"studyKind":"vendor_writeup","classIds":["browser","personal-assistant"],"failureModeIds":["confirmation-skip","over-refusal"],"lesson":"The page describes a research-preview browser agent and says that before a significant action, such as submitting an order or sending an email, Operator should ask for approval. It also says the model is trained to decline some sensitive tasks, including banking transactions. A browser agent that submits or sends with no approval id is outside the behavior this writeup describes.","sourceUrl":"https://openai.com/index/introducing-operator/","asOf":"2026-10-03","notes":"Fetched page. A July 17, 2025 update on the same page says Operator was integrated into ChatGPT agent and that the standalone site would sunset. That update is a product change, not an incident report."},{"id":"the-agent-company-2025","title":"TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks","year":2025,"studyKind":"capability_ceiling","classIds":["coding","browser","research","orchestrator"],"failureModeIds":["weak-verification"],"lesson":"The abstract describes a simulated software company where agents browse, write code, run programs, and communicate with coworkers. It says the most competitive agent they tested completed 30% of tasks autonomously. Read that as a ceiling inside their simulation, not as a measurement of any employer's production team.","sourceUrl":"https://arxiv.org/abs/2412.14161","asOf":"2026-10-03","notes":"Fetched arXiv v3, last revised 10 Sep 2025. Submitted 18 Dec 2024."},{"id":"workarena-2024","title":"WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?","year":2024,"studyKind":"capability_ceiling","classIds":["browser","support"],"failureModeIds":["weak-verification"],"lesson":"The abstract introduces WorkArena as 33 tasks on ServiceNow and BrowserGym as the environment. It says agents showed promise and that a considerable gap remained before full task automation, with a disparity between open and closed models. No success percentage is stored because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2403.07718","asOf":"2026-10-03","notes":"Fetched the arXiv abstract (submitted 12 Mar 2024). Enterprise knowledge-work ceiling, not a ServiceNow incident."},{"id":"willison-prompt-injection-2023","title":"Prompt injection: What's the worst that can happen?","year":2023,"studyKind":"mechanism_study","classIds":["personal-assistant","browser","security-review"],"failureModeIds":["prompt-injection","confirmation-skip"],"lesson":"The post explains that a tool-using assistant which reads untrusted text, such as email or a web page, can be driven to take an action the user did not ask for, including sending mail or reaching data through a connected tool. The lesson for an assistant is to treat fetched text as untrusted and to require confirmation before a send or an export. Attack sketches from the post are not stored.","sourceUrl":"https://simonwillison.net/2023/Apr/14/worst-that-can-happen/","asOf":"2026-10-03","notes":"Fetched the 14 April 2023 post. Hypotheticals in the post are not reproduced and are not marked as incidents that Depot verified."}]}