WebDoctor
Diagnostic tool

robots.txt tester

Load a site's robots.txt or paste your own, enter a URL and pick a crawler. You get a plain verdict, the exact line that decided it, and the same test run against twelve major crawlers at once.

Any crawler token works, so type one in if the crawler you need is not on the list.

Pasted rules are matched entirely in your browser. Fetching a robots.txt from a domain goes through our server to download that one public file, and nothing is stored.

How robots.txt matching actually works

A robots.txt file is a set of groups. Each group starts with one or more User-agent lines and holds the Allow and Disallow rules that apply to those crawlers. A crawler reads only the group that matches its own product token, plus the catch-all group if no named group fits, so rules written for Googlebot have no effect on GPTBot.

When several rules in a group match the same URL, the most specific one wins: the rule whose pattern matches the greatest number of characters. If an Allow and a Disallow tie on length, the Allow wins. That ordering comes from RFC 9309, the standardised Robots Exclusion Protocol, and it is why Disallow: /admin/ plus Allow: /admin/public/ leaves the public folder crawlable.

Two wildcards are supported in patterns. An asterisk stands for any run of characters, and a dollar sign anchors the pattern to the end of the URL. Paths are case sensitive, crawler names are not, and one robots.txt only governs its own scheme, host and port, so the file on the www subdomain does not cover the bare domain.

What the tester reports

  • Verdict for the crawler you pick, with the exact rule and line number behind it
  • The same URL tested against twelve crawlers at once, search bots and AI bots side by side
  • Which group matched, including whether it was a named group or the catch-all
  • Every Sitemap line declared in the file
  • Parse warnings: rules written before any User-agent line, unsupported directives, absolute URLs in Disallow, missing Sitemap

AI Readiness Checker

Can ChatGPT, Claude and Perplexity read and cite your site? Checks llms.txt, robots.txt AI-crawler rules, structured data and metadata.

Check AI readiness →

From the blog

Hreflang

Hreflang Mistakes That Kill International SEO

Ten errors account for almost every broken hreflang setup, and most of them fail silently. Here is how to spot each one and correct it.

30 July 2026 · 5 min read robots.txt

robots.txt for AI Crawlers: Allow and Deny Recipes

Three recipes and a bot-by-bot table for controlling GPTBot, ClaudeBot, PerplexityBot and the rest from one small text file.

22 July 2026 · 4 min read Sitemaps

XML Sitemap Errors: Every Common Failure and Its Fix

Nine failures cause almost every broken sitemap. Here is how to recognize each one from its symptom and fix it for good.

16 July 2026 · 4 min read

All guides →

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?
No. A Disallow rule stops crawling, not indexing. Google can still list a blocked URL it finds through links, showing it without a description. To keep a page out of results, allow crawling and use a noindex meta robots tag on the page itself.
Why does my Disallow rule not work?
Usually a longer rule outranks it. Matching is by pattern length, not file order, and Allow beats Disallow on a tie. Case matters too: /Blog/ and /blog/ are different paths. Test the exact URL here and read the rule the tester names as the winner.
Is crawl-delay respected?
Not by Google, which ignores the directive entirely and asks site owners to use crawl rate settings instead. Bing and Yandex have historically honoured it. Leaving it in your file costs nothing but do not rely on it to control Googlebot.
Can I put noindex in robots.txt?
It does nothing on Google. Support for an unofficial noindex directive in robots.txt ended on 1 September 2019. Use a noindex meta robots tag or an X-Robots-Tag HTTP header on the page you want kept out of search results.
Does one robots.txt cover my subdomains?
No. The file applies to one scheme, host and port only. A blog on blog.example.com needs its own robots.txt at blog.example.com/robots.txt, and the https version of a site is treated separately from the http version.