FindPrompts

Sitemap-Driven Crawl Planner

Build an efficient crawl plan driven by sitemaps and freshness signals.

6/11/2026
Prompt
## CONTEXT
The developer wants to crawl a large site efficiently by leaning on its sitemaps rather than blindly following links. They need to parse sitemap indexes, prioritize by freshness, and build a coverage-aware crawl queue.

## ROLE
Act as a crawl-planning expert who uses sitemaps and lastmod data to maximize coverage while minimizing redundant fetches.

## RESPONSE GUIDELINES
- Parse sitemap index and child sitemaps.
- Use lastmod and changefreq to prioritize.
- Combine sitemap URLs with discovered links for coverage.
- Respect robots and disallow rules.
- Produce an ordered crawl queue.

## TASK CRITERIA

### Sitemap Parsing
- Fetch and parse the sitemap index.
- Recurse into nested sitemaps.
- Handle gzipped sitemap files.
- Tolerate malformed or partial sitemaps.

### Prioritization
- Order URLs by lastmod recency.
- Weight by changefreq and priority hints.
- Surface newly added URLs first.
- Deprioritize stale, unchanged URLs.

### Coverage
- Compare sitemap URLs to discovered links.
- Find pages missing from the sitemap.
- Detect orphan pages not linked anywhere.
- Estimate total crawl scope.

### Compliance
- Filter URLs against robots disallow rules.
- Honor crawl-delay declarations.
- Skip noindex pages where signaled.
- Keep within stated boundaries.

### Queue Construction
- Build a deduplicated, ordered URL queue.
- Persist the queue for resumable runs.
- Track fetched versus pending counts.
- Report planned coverage and freshness mix.

## ASK THE USER FOR
- The site domain and sitemap location.
- Which sections matter most.
- How fresh the data needs to be.
- Their crawl rate constraints.

Or press ⌘C to copy

Ready to use?

Copy and paste into your favorite AI tool

Test this prompt live

Try it in our AI Playground with Claude

Open Playground

Explore more Coding prompts

Browse Coding

Command Palette

Search for a command to run...