Agent classes / Research agent / LLM-ready web extraction
LLM-ready web extraction for a research agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-10-19
The class: Gathers sources for an open-ended question and returns an answer that names those sources.
- A stop rule for when enough sources have been read
Depot Pick: Firecrawl
When not to use it: Need search only without crawl — Exa/Tavily.
Runner-up: ScrapingBee.
No single-unit sourced price comparison for this job (no_comparable_unit_for_job).
Failure modes this class and job share (11)
- Context overflow: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
- Instruction drift: Restate the stop condition in the trace. End the run when it is met. Review long traces for steps that do not serve the original task.
- Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
- Lost in the middle: Ask the agent to name the passage it used. Spot-check questions whose only support sits in the middle of the context, not only at the edges.
- Policy conflict: If cited sources contradict each other, return the conflict and the URLs. Do not pick a winner silently.
- Prompt injection: Compare each tool call to the original user task. If the call only makes sense given text that arrived from a fetched page, file, or message, stop and ask. Record which source the instruction came from. This catalog does not store payloads or attack steps.
- Runaway spend: Set a budget and a tool-call cap per task. Alert when spend jumps. Stop the run at the cap instead of summarizing a partial trace as success.
- Silent truncation: Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
- Stale retrieval: Store the source URL and the time it was fetched. If a newer canonical page exists, or two stored sources disagree, do not answer from the older one.
- Tool loop: Hash the last several tool names and arguments. Stop when the same call repeats and the observation did not change. Cap tool calls per task.
- Ungrounded answer: Require a source URL or passage id on factual sentences. Drop or mark any sentence whose source does not contain the claim.
Free options this class uses (0)
No free row lists this job with a kind this class needs.