{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","id":"computer-use","name":"Computer-use agent","definition":"Operates a desktop or mobile GUI, not only a single site API, and checks that the control it used is the one it intended.","needs":{"jtbdIds":["desktop-computer-use","mobile-computer-use","computer-use-browser","form-filling-rpa"],"freeKinds":["free_sandbox","free_browser"],"capabilityTags":["browser","agents"],"plain":["A screenshot or accessibility tree after each action","An allowlist of apps the task may change"]},"failureModeIds":["gui-grounding-miss","prompt-injection","collateral-side-effects","partial-tool-success","tool-loop","confirmation-skip","benchmark-protocol-mismatch"],"exampleTasks":["Change one setting and verify the resulting file","Stop if the click opened a different app than the plan named"],"failures":[{"id":"gui-grounding-miss","name":"GUI grounding miss","whatGoesWrong":"A click, keystroke, or scroll lands on the wrong control, so later steps run against a screen the agent did not mean to open.","howToDetect":"After each GUI action, compare the resulting screen or accessibility tree to the control the plan named. Stop when they differ.","classIds":["computer-use","browser"],"sourceUrls":["https://arxiv.org/abs/2404.07972"]},{"id":"prompt-injection","name":"Prompt injection","whatGoesWrong":"Untrusted text, including text from a page, file, or message the agent fetched, changes the agent's behavior away from the user's task. OWASP distinguishes a direct injection in the user's own prompt from an indirect injection in outside content. Neither form is fully removed by retrieval or fine-tuning, according to the OWASP LLM01:2025 page fetched today.","howToDetect":"Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.","classIds":["browser","computer-use","research","support","personal-assistant","security-review","sales-outreach"],"sourceUrls":["https://genai.owasp.org/llmrisk/llm01-prompt-injection/","https://www.anthropic.com/news/developing-computer-use","https://simonwillison.net/2023/Apr/14/worst-that-can-happen/"]},{"id":"collateral-side-effects","name":"Collateral side effects","whatGoesWrong":"The stated task ends in the desired state, and something outside the allowlist also changes: an unrelated file, row, or app.","howToDetect":"Snapshot the workspace or app state, then diff it against an allowlist. A passing check plus an unexpected diff is a failure.","classIds":["coding","computer-use","data-etl","browser"],"sourceUrls":["https://arxiv.org/abs/2407.18901"]},{"id":"partial-tool-success","name":"Partial tool success","whatGoesWrong":"A tool completes only part of a batch and the agent treats the whole batch as done.","howToDetect":"Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.","classIds":["coding","data-etl","browser","computer-use","voice","orchestrator","scheduled-ops"],"sourceUrls":[]},{"id":"tool-loop","name":"Tool loop","whatGoesWrong":"The agent calls the same tool, or a cycle of tools, without new information, so the task never reaches a stop condition.","howToDetect":"Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.","classIds":["coding","research","browser","computer-use","orchestrator","scheduled-ops"],"sourceUrls":["https://www.anthropic.com/engineering/built-multi-agent-research-system"]},{"id":"confirmation-skip","name":"Confirmation skip","whatGoesWrong":"A consequential action, such as sending, paying, deleting, or submitting, runs without the confirmation the product policy requires.","howToDetect":"Gate those tool names on an approval id recorded before the call. A missing approval is a failure even if the downstream tool returns success.","classIds":["browser","sales-outreach","personal-assistant","support","computer-use","voice"],"sourceUrls":["https://openai.com/index/introducing-operator/"]},{"id":"benchmark-protocol-mismatch","name":"Benchmark protocol mismatch","whatGoesWrong":"A score is cited next to another score that used a different setup: assisted versus unassisted editing, a subset versus the full set, or a revised harness versus the original.","howToDetect":"Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.","classIds":["eval-judge","coding","computer-use","browser"],"sourceUrls":["https://www.cognition.ai/blog/swe-bench-technical-report","https://openai.com/index/introducing-swe-bench-verified/","https://os-world.github.io/"]}],"caseStudies":[{"id":"visualwebarena-2024","title":"VisualWebArena","year":2024,"studyKind":"capability_ceiling","classIds":["browser","computer-use"],"failureModeIds":["gui-grounding-miss"],"lesson":"The abstract says many web tasks need visual information that text-only agents do not use well, and that the benchmark is visually grounded. A browser agent that reads only the DOM or the HTML is being measured on a different task than this one. No success rate from the abstract is repeated here because the abstract did not state one.","sourceUrl":"https://arxiv.org/abs/2401.13649","asOf":"2026-10-03","notes":"Fetched the arXiv abstract. Comment on the page: accepted to ACL 2024."},{"id":"osworld-2024","title":"OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments","year":2024,"studyKind":"capability_ceiling","classIds":["computer-use"],"failureModeIds":["gui-grounding-miss","weak-verification"],"lesson":"The abstract describes 369 real computer tasks across operating systems and apps, with execution-based checks. It says humans accomplish over 72.36% and the best model 12.24%, and that models struggle with GUI grounding and operational knowledge. This is a computer-use ceiling from the paper, not a production outage.","sourceUrl":"https://arxiv.org/abs/2404.07972","asOf":"2026-10-03","notes":"Fetched arXiv v2, last revised 30 May 2024. Later site notes about OSWorld-Verified are a separate study."},{"id":"osworld-verified-site-2025","title":"OSWorld project site: verified upgrade","year":2025,"studyKind":"capability_ceiling","classIds":["computer-use","eval-judge"],"failureModeIds":["benchmark-protocol-mismatch","eval-contamination"],"lesson":"The project site says that on 2025-07-28 OSWorld was enhanced as OSWorld-Verified, including fixed community-reported examples and updated benchmark results, and that new runs should be compared with those new results. A 2026-06-26 line says OSWorld 2.0 is available. A score from the 2024 paper is not the same number as a score on the later set.","sourceUrl":"https://os-world.github.io/","asOf":"2026-10-03","notes":"Fetched the project homepage. What the 2.0 task set contains was not copied because the homepage line only says it is available."},{"id":"openai-cua-2025","title":"Computer-Using Agent","year":2025,"studyKind":"vendor_writeup","classIds":["computer-use","browser"],"failureModeIds":["benchmark-protocol-mismatch","gui-grounding-miss"],"lesson":"OpenAI's page says Computer-Using Agent drives a GUI from screenshots and reports 38.1% on OSWorld, 58.1% on WebArena, and 87% on WebVoyager. Those are the vendor's reported scores for that model. They are not a re-run of the original papers' baselines, and the page says the model is early and has limitations.","sourceUrl":"https://openai.com/index/computer-using-agent/","asOf":"2026-10-03","notes":"Fetched the OpenAI page dated January 23, 2025 in the article header. Depot did not re-run OSWorld, WebArena, or WebVoyager."},{"id":"anthropic-computer-use-2024","title":"Developing a computer use model","year":2024,"studyKind":"vendor_writeup","classIds":["computer-use","security-review"],"failureModeIds":["prompt-injection"],"lesson":"Anthropic's post says computer use was in public beta and names prompt injection as a case where instructions in content can override prior directions or cause actions that were not the user's intent. It notes that a model reading screenshots from an internet-connected computer can be exposed to that content. The lesson stored here is to treat screen text as untrusted. The post's attack detail is not copied.","sourceUrl":"https://www.anthropic.com/news/developing-computer-use","asOf":"2026-10-03","notes":"datePublished on the fetched page is 2024-10-22. No procedure is stored."}]}