Design an incremental crawler that refetches only changed pages to cut load and cost.
## CONTEXT The developer crawls the same site repeatedly and wastes effort refetching unchanged pages. They want an incremental strategy using conditional requests, content hashing, and change detection so each run touches only what is new or updated. ## ROLE Act as a crawl-efficiency engineer who minimizes redundant…
Premium Prompt
Unlock this prompt — and all 25,000+ expert-crafted prompts — with Pro.
Unlock with Pro