{"asOf":"2026-10-03","doesNotChangeDepotPicks":true,"doesNotChangePriceSort":true,"doesNotChangeFreeIndex":true,"note":"Classification and failure modes are mechanisms. Case studies are included only when the source URL was fetched on 2026-10-03. A capability ceiling is not a company incident. Vendor figures are labeled as that vendor's report.","id":"scheduled-ops","name":"Scheduled and ops agent","definition":"Runs a job on a queue or a clock, is safe to retry, and alerts when the job or its traces go dark.","needs":{"jtbdIds":["event-driven-agent-jobs","serverless-agent-queue","agent-dlq-retry","durable-idempotency","llm-cost-anomaly-alerts","agent-error-observability","progressive-agent-rollout"],"freeKinds":["dev_tool"],"capabilityTags":["serverless","cost"],"plain":["Idempotency key on every retry","An alert when trace ingestion drops"]},"failureModeIds":["runaway-spend","partial-tool-success","tool-loop","observability-gap","non-determinism"],"exampleTasks":["Retry a failed job without applying the side effect twice","Alert when the night job's trace count falls"],"failures":[{"id":"runaway-spend","name":"Runaway spend","whatGoesWrong":"Retries, fan-out, or a loop keep calling paid models or tools after the task should have stopped.","howToDetect":"Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.","classIds":["coding","research","orchestrator","scheduled-ops","sales-outreach","voice"],"sourceUrls":["https://www.anthropic.com/engineering/building-effective-agents"]},{"id":"partial-tool-success","name":"Partial tool success","whatGoesWrong":"A tool completes only part of a batch and the agent treats the whole batch as done.","howToDetect":"Require a per-item status and a failure count. Success for the task requires the failed count to be zero or explicitly handed to a retry queue.","classIds":["coding","data-etl","browser","computer-use","voice","orchestrator","scheduled-ops"],"sourceUrls":[]},{"id":"tool-loop","name":"Tool loop","whatGoesWrong":"The agent calls the same tool, or a cycle of tools, without new information, so the task never reaches a stop condition.","howToDetect":"Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.","classIds":["coding","research","browser","computer-use","orchestrator","scheduled-ops"],"sourceUrls":["https://www.anthropic.com/engineering/built-multi-agent-research-system"]},{"id":"observability-gap","name":"Observability gap","whatGoesWrong":"Traces or run ingestion stop, and the failure is outside the agent's reasoning. A certificate or DNS fault can make the trace API fail while agents keep acting.","howToDetect":"Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.","classIds":["eval-judge","scheduled-ops","orchestrator"],"sourceUrls":["https://www.langchain.com/blog/langsmith-incident-on-may-1-2025"]},{"id":"non-determinism","name":"Non-determinism","whatGoesWrong":"The same task produces a different tool sequence or a different end state on another trial, so one success does not mean the agent is reliable.","howToDetect":"Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.","classIds":["support","coding","browser","scheduled-ops","eval-judge","voice"],"sourceUrls":["https://arxiv.org/abs/2406.12045"]}],"caseStudies":[{"id":"langsmith-cert-incident-2025","title":"LangSmith incident on May 1, 2025","year":2025,"studyKind":"production_incident","classIds":["eval-judge","scheduled-ops"],"failureModeIds":["observability-gap"],"lesson":"LangChain's incident note says the US LangSmith API had an elevated error rate for 28 minutes on May 1, 2025, from 14:35 to 15:03 UTC, and that about 55% of API requests failed, including run ingestion. The cause they describe is a leftover DNS record that stopped certificate renewal until the certificate expired. Agent traces can disappear because the trace host's certificate failed, not because the agent reasoned badly.","sourceUrl":"https://www.langchain.com/blog/langsmith-incident-on-may-1-2025","asOf":"2026-10-03","notes":"Fetched LangChain's own incident writeup. This is an observability outage, not a model-behavior incident."}]}