Questions tagged [scraper]
Post a PromptPrompts that build scrapers and data extractors.
2 prompts
20votes
1answers
5.2kviews
Bounding concurrency in an asyncio scraper so I stop opening 5000 sockets at once
I asked for an async version of a fetcher and the model gleefully did await asyncio.gather([fetch(u) for u in urls]) over 5000 URLs. It opened every connection simultaneously, the event loop thrashed,
28votes
0answers
4.2kviews
One prompt for a polite, resumable web scraper that won't get me IP-banned
I needed to pull a few thousand product pages for a price-history side project. My first scraper hammered the site with 50 concurrent requests, got a 429, then kept retrying instantly and made it wors