The WebDoctor blog
Practical guides to sitemaps, crawling and technical site health, written to be acted on.
robots.txt for AI Crawlers: Allow and Deny Recipes
Three recipes and a bot-by-bot table for controlling GPTBot, ClaudeBot, PerplexityBot and the rest from one small text file.
SitemapsXML Sitemap Errors: Every Common Failure and Its Fix
Nine failures cause almost every broken sitemap. Here is how to recognize each one from its symptom and fix it for good.
SitemapsXML Sitemap Best Practices That Actually Matter
Half of the standard sitemap advice is folklore. These are the practices search engines actually respond to, and the fields they ignore.
SitemapsSitemap Index Files: When and How to Use Them
One file that lists your other sitemaps: the index is simple, but the rules about nesting, hosts and submission trip people up.
llms.txtWhat Is llms.txt and Does Your Site Need One?
A small markdown file at your site root tells AI systems what your site is and which pages matter. Here is exactly how it works and when it is worth adding.
AI searchAI Readiness: How to Get Cited by ChatGPT and Perplexity
AI assistants only recommend sites they can crawl, parse and trust. Those are three separate problems, and most sites fail at least one without knowing.