Verified against ChatGPT · 2026-08-07
Build a web scraper that respects rate limits and fails predictably
A prompt for a scraping script that checks robots.txt, paces requests deliberately, retries transient failures without hammering the target, and extracts data into a validated structured schema — instead of a tight fetch loop that gets the IP blocked on day one.
The prompt
Ready to copy — highlighted parts are example details you can swap.
Build a web scraper for the target described below. This needs to behave like a responsible client, not a tight loop that fetches as fast as the network allows — that gets an IP blocked or rate-limited within minutes against most real sites, and the scraper stops working entirely rather than slowing down gracefully. TARGET SITE A public real-estate listings site, scraping individual listing detail pages linked from a search results page. DATA TO EXTRACT Listing price, square footage, address, and listed date from each detail page. SCALE AND FREQUENCY Roughly 2,000 listing pages, run once per day as a scheduled job — not a one-time pull. LEGAL/ToS CONTEXT robots.txt disallows /admin/ and /api/internal/ but allows /listings/; Terms of Service page has no explicit anti-scraping clause found. REQUIREMENTS 1. Check and respect robots.txt for A public real-estate listings site, scraping individual listing detail pages linked from a search results page. programmatically (using urllib.robotparser or an equivalent), not just as a one-time manual check — if a path this scraper needs is disallowed, say so explicitly and stop, rather than scraping it anyway and hoping it goes unnoticed. Note anything robots.txt disallows /admin/ and /api/internal/ but allows /listings/; Terms of Service page has no explicit anti-scraping clause found. says beyond robots.txt that also constrains what's being built here. 2. Rate-limit deliberately: a fixed minimum delay between requests to the same domain, ideally with jitter so requests don't arrive at suspiciously exact intervals, sized to Roughly 2,000 listing pages, run once per day as a scheduled job — not a one-time pull. and to whatever robots.txt's Crawl-delay (if present) specifies. State the requests-per-minute this design produces and whether that's actually reasonable for a site this scraper doesn't own or control. 3. Set a real, identifying User-Agent header naming what this is, not a spoofed browser User-Agent pretending to be a real person's browser — a scraper impersonating a browser to bypass basic bot detection is a materially different (and riskier) thing to build than one that identifies itself honestly and gets blocked if the target doesn't want it. 4. Retry transient failures (connection timeouts, 5xx responses) with backoff, matching the retry discipline of a well-built API client — but treat a 429 (Too Many Requests) or 403 as a signal to back off much further or stop entirely, never as just another transient error to retry through at the same pace, since retrying a 429 at the same rate is actively making the problem worse. 5. Extract Listing price, square footage, address, and listed date from each detail page. into a validated structured schema (a Pydantic model per extracted item), not raw dicts pulled straight from the parsed HTML — validate that required fields actually parsed correctly and log or quarantine a page that didn't match the expected structure, rather than silently producing a record with missing or garbage fields. 6. Design for the target site's HTML changing without notice: isolate every CSS selector or parsing rule in one place, not scattered through the extraction logic, so a layout change breaks one identifiable line instead of failing mysteriously somewhere in the middle of a long function. OUTPUT FORMAT 1. The robots.txt check. 2. The rate-limited fetch function with backoff and its 429/403 handling. 3. The extraction schema and parsing function. 4. The requests-per-minute this design produces, and confirmation it respects any Crawl-delay found.
Customize
Optional — swap in your own details for the highlighted parts above.
Why this works
Requiring a programmatic robots.txt check, not a one-time manual read, matters because robots.txt can change and because "I checked it once" doesn't scale to a scraper that runs daily for months — building the check into the scraper itself, via urllib.robotparser, means a future robots.txt change (a site adding a Disallow rule after noticing scraper traffic) is respected automatically on the very next run rather than requiring someone to remember to re-check it manually, which in practice nobody reliably does. The distinction between retrying a 5xx and backing off hard on a 429 targets a real and specific escalation risk: a 429 is the target server explicitly telling the client it's being rate-limited right now, and a naive retry-with-backoff implementation that treats every non-2xx response identically will keep hitting the same server at nearly the same rate while backing off only slightly, which from the target's perspective looks like a client that received the rate-limit signal and ignored it — this is exactly the behavior that turns a soft rate limit into a hard IP ban, so 429/403 need materially different handling than a transient timeout, not the same retry loop with the same parameters. Requiring an honest, identifying User-Agent rather than a spoofed browser one is a deliberate design choice with real consequences either way: identifying honestly means a site operator who doesn't want this traffic can block it cleanly by User-Agent, and the scraper fails in a way that surfaces immediately and unambiguously, while a spoofed browser User-Agent is specifically built to make the scraper's traffic indistinguishable from a real user's browser to bypass exactly that kind of detection — the difference matters for how {{tos_context}} should actually be read, since building a bot-detection-evasion tool is a fundamentally different thing to be asked to build than a scraper that plays by the rules and accepts being blocked if the target says no. Isolating every parsing rule in one place addresses the specific way scrapers actually fail in production: the target's HTML structure will change without any notice on the scraper's end, and a parsing rule embedded inline throughout a long extraction function means that change surfaces as a confusing partial failure somewhere in the middle of a run, while centralizing the selectors means the exact same change surfaces as one broken, easily locatable rule the next time the job runs.
Verified against
ChatGPT GPT-5.1 · 2026-08-07
Claude Code Sonnet 4.6 · 2026-08-08
Changelog
- 2026-08-08 — Initial publish, verified against ChatGPT (GPT-5.1) and Claude Code (Sonnet 4.6) using httpx 0.28.
Need this built into your business?
If a prompt isn't enough — custom software, built and maintained for you — that's Scult's day job.
EXPLORE CUSTOM SOFTWARE
