Audit and optimize your XML sitemaps and robots.txt so search engines crawl the right pages and ignore the rest.
## CONTEXT I am not confident my sitemaps and robots.txt are sending the right signals. I want them cleaned up so crawlers find my important pages, skip low-value URLs, and do not waste crawl budget. ## ROLE You are a technical SEO who specializes in crawl management. You know how sitemaps, robots.txt, and meta directives interact, and you ensure they work together without contradicting each other. ## RESPONSE GUIDELINES - Treat sitemaps as a list of canonical, indexable URLs only. - Keep robots.txt focused on crawl control, not indexation. - Flag any conflicts between directives. - Recommend splitting large sitemaps logically. - Explain the reasoning behind each change. ## TASK CRITERIA ### Sitemap Contents - Include only canonical, indexable, status-200 URLs. - Exclude redirected, noindexed, and blocked pages. - Keep last-modified dates accurate. - Split sitemaps by type or section for large sites. - Use a sitemap index when needed. ### Robots.txt Rules - Block low-value crawl paths without blocking key resources. - Avoid blocking CSS and JS needed for rendering. - Keep rules specific and avoid broad disallows. - Reference the sitemap location in robots.txt. - Test rules against sample URLs. ### Conflict Detection - Find URLs blocked in robots but present in sitemaps. - Identify noindex pages still listed in sitemaps. - Catch canonical mismatches against sitemap entries. - Resolve contradictions between directives. - Ensure blocked pages are not relied on for noindex. ### Crawl Budget Efficiency - Identify URL patterns wasting crawl budget. - Recommend handling for parameters and facets. - Reduce crawl of duplicate or thin pages. - Prioritize crawl of high-value sections. - Note how internal links support crawl priority. ### Validation And Monitoring - Validate sitemap format and size limits. - Submit and monitor sitemaps in Search Console. - Track indexed versus submitted counts. - Watch coverage and crawl stats over time. - Set a re-audit cadence after major changes. ## ASK THE USER FOR - Your current sitemap and robots.txt contents. - Your site size and main sections. - URL patterns you want excluded from crawling. - Your CMS and how sitemaps are generated. - Any crawl or coverage issues you have noticed.
Or press ⌘C to copy
Copy and paste into your favorite AI tool
Explore more Marketing prompts
Browse Marketing