WAF / Cloudflare 1020 / 403 Forbidden / robots.txt Disallow on Web Fetch

Target origin blocks datacenter IPs, headless Chromium TLS fingerprints, or disallows raw bot user-agents.

1. Immediate Code & SKU Fix

Replace With SKU: firecrawl (web-scrape) or browserbase (cloud-browser)

Install Command: npm install @mendable/firecrawl-js

Required Env Vars: FIRECRAWL_API_KEY, BROWSERBASE_API_KEY

import FirecrawlApp from '@mendable/firecrawl-js';
const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });
const res = await app.scrapeUrl(targetUrl, { formats: ['markdown'] });

2. Verified Hot-Swap Fallback Route

JTBD: web-scrape

Primary: firecrawl → Fallback: bright-data

liveHealth: operational · autoActOk: true

Hard anti-bot enterprise — Bright Data

GET /api/rescue JSON (<2 KB)Falsify Pick (/api/falsify)

Matched Error String Signatures (9)

cloudflare 10201020 access denied403 forbiddenrobots.txtrobots.txtdisallowwafcaptchadatadomeblocked

curl -s "https://getagentdepot.com/api/rescue?q=cloudflare%201020"

Browse All 28 Agent Runtime Rescue Cards