I needed to pull a few thousand product pages for a price-history side project. My first scraper hammered the site with 50 concurrent requests, got a 429, then kept retrying instantly and made it worse. Classic.
I rewrote the prompt to bake in politeness and resumability as non-negotiables: read robots.txt, cap concurrency, exponential backoff with jitter, and checkpoint progress so a crash doesn't restart from zero. The checkpoint file is the unsung hero, I can Ctrl-C any time and resume.
Still unsure about the etiquette default for crawl delay when robots.txt doesn't specify one. I went with 1 request/sec. Too aggressive?
Build a resumable Python scraper (requests + a small worker pool, no Scrapy) that fetches a list of URLs from `{INPUT_CSV}` (column `url`) and writes extracted fields to `{OUTPUT_JSONL}`.
Be a good citizen and be crash-safe:
- Fetch and honor `robots.txt` (use `urllib.robotparser`); skip disallowed URLs and log them.
- Set a descriptive `User-Agent` including a contact placeholder `{CONTACT_EMAIL}`.
- Rate limit to at most `{RPS}` requests/second per host; default 1. Add 0-300ms random jitter.
- Retry only on 429/500/502/503/504 and connection errors, with exponential backoff (base 1s, factor 2, cap 60s, max 5 tries) and full jitter. Respect the `Retry-After` header when present.
- Timeout every request (connect 5s, read 15s). Never retry on 4xx other than 429.
- Idempotent + resumable: append one JSON object per line to the output; before starting, read the output file and skip URLs already completed (key on a stable `id`/url hash). Write each record atomically (flush + fsync or write-then-rename).
- Extraction goes in a single `parse(html, url) -> dict | None` function so it is unit-testable; return None (and log) on unexpected structure instead of throwing.
- Print a final summary: fetched, skipped (robots), skipped (already done), failed, and elapsed.
Do not use headless browsers. Do not follow links beyond the input list. After writing the code, list every place the scraper could hang or spin forever and confirm each has a timeout or bounded retry.