I asked for an async version of a fetcher and the model gleefully did await asyncio.gather(*[fetch(u) for u in urls]) over 5000 URLs. It opened every connection simultaneously, the event loop thrashed, and the target rate-limited me into oblivion.
The fix in the prompt was to demand a bounded worker pattern (semaphore or a fixed queue of workers) and a single shared client session, plus per-host limits. I also make it explain why gather over an unbounded list is a footgun so it doesn't regress.
What's everyone's preferred pattern, asyncio.Semaphore around gather, or a fixed pool of workers draining a Queue? I keep going back and forth.
Write an asyncio scraper using `aiohttp` that fetches `{N}` URLs with STRICT concurrency control. Do not use `asyncio.gather` over an unbounded list of coroutines.
Requirements:
- One shared `aiohttp.ClientSession` for the whole run (created in `async with`), with a `TCPConnector(limit={TOTAL}, limit_per_host={PER_HOST})`.
- Cap in-flight requests to `{CONCURRENCY}` using either a fixed pool of worker tasks draining an `asyncio.Queue`, or a `Semaphore`. Pick one and justify it in a comment.
- Every request: `ClientTimeout(total=20, connect=5)`, retry 429/5xx with exponential backoff + jitter (max 4 tries), honor `Retry-After`.
- Handle `asyncio.CancelledError` cleanly so Ctrl-C cancels workers and closes the session (no 'unclosed session' warnings).
- Backpressure: results go to an `asyncio.Queue` consumed by a single writer task that appends JSONL, so disk I/O never blocks the fetchers and output lines never interleave.
- Graceful shutdown: on SIGINT, stop accepting new work, let in-flight requests finish or time out, flush the writer, then exit.
- Return counts of ok/failed/retried and total wall time.
Before finalizing, explain in 3-4 sentences why unbounded `gather` is dangerous here and confirm there is no code path that creates more than `{CONCURRENCY}` simultaneous requests.