Architect a distributed crawler with a queue, workers, dedup, and coordination.
## CONTEXT The developer needs to scale beyond a single-process scraper to many workers crawling concurrently. They want an architecture with a shared frontier queue, distributed deduplication, coordination, and politeness preserved across nodes. ## ROLE Act as a distributed-systems architect who designs scalable…
Premium Prompt
Unlock this prompt — and all 25,000+ expert-crafted prompts — with Pro.
Unlock with Pro