Agent classes / Research agent / Web scraping / page fetch
Web scraping / page fetch for a research agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-10-19
The class: Gathers sources for an open-ended question and returns an answer that names those sources.
- A stop rule for when enough sources have been read
Depot Pick: Firecrawl
When not to use it: Hard anti-bot enterprise — Bright Data. Interactive multi-step — Browserbase.
Runner-up: ScrapingBee.
No single-unit sourced price comparison for this job (no_comparable_unit_for_job).
Failure modes this class and job share (11)
- Instruction drift: Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.
- Prompt injection: Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.
- Tool loop: Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.
- Context overflow: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
- Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
- Lost in the middle: Ask the agent to name the passage it used. Spot-check questions whose only support sits in the middle of the context, not only at the edges.
- Policy conflict: If cited sources contradict each other, return the conflict and the URLs. Do not pick a winner silently.
- Runaway spend: Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
- Silent truncation: Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
- Stale retrieval: Store the source URL and the time it was fetched. If a newer canonical page exists, or two stored sources disagree, do not answer from the older one.
- Ungrounded answer: Require a source URL or passage id on factual sentences. Drop or mark any sentence whose source does not contain the claim.
Free options this class uses (1)
- Crawl4AI (free_browser, oss) — source, asOf 2026-10-25