WebDoctor
Diagnostic tool

robots.txt tester

Load a site's robots.txt or paste your own, enter a URL and pick a crawler. You get a plain verdict, the exact line that decided it, and the same test run against twelve major crawlers at once.

Any crawler token works, so type one in if the crawler you need is not on the list.

Pasted rules are matched entirely in your browser. Fetching a robots.txt from a domain goes through our server to download that one public file, and nothing is stored.

How robots.txt matching actually works

A robots.txt file is a set of groups. Each group starts with one or more User-agent lines and holds the Allow and Disallow rules that apply to those crawlers. A crawler reads only the group that matches its own product token, plus the catch-all group if no named group fits, so rules written for Googlebot have no effect on GPTBot.

When several rules in a group match the same URL, the most specific one wins: the rule whose pattern matches the greatest number of characters. If an Allow and a Disallow tie on length, the Allow wins. That ordering comes from RFC 9309, the standardised Robots Exclusion Protocol, and it is why Disallow: /admin/ plus Allow: /admin/public/ leaves the public folder crawlable.

Two wildcards are supported in patterns. An asterisk stands for any run of characters, and a dollar sign anchors the pattern to the end of the URL. Paths are case sensitive, crawler names are not, and one robots.txt only governs its own scheme, host and port, so the file on the www subdomain does not cover the bare domain.

What the tester reports

  • Verdict for the crawler you pick, with the exact rule and line number behind it
  • The same URL tested against twelve crawlers at once, search bots and AI bots side by side
  • Which group matched, including whether it was a named group or the catch-all
  • Every Sitemap line declared in the file
  • Parse warnings: rules written before any User-agent line, unsupported directives, absolute URLs in Disallow, missing Sitemap

AI Readiness Checker

Can ChatGPT, Claude and Perplexity read and cite your site? Checks llms.txt, robots.txt AI-crawler rules, structured data and metadata.

Check AI readiness →

From the blog

Structured data

Schema Types That Actually Earn Rich Results in 2026

Nine types still earn a visual treatment. Two famous ones were retired. Here is the current list, from Google's own documentation.

15 September 2026 · 4 min read Core Web Vitals

Core Web Vitals in 2026: Thresholds and What Moved

The metrics did not change this year. Plenty of the tooling around them did. Here is what is actually true right now.

8 September 2026 · 4 min read Meta descriptions

Meta Descriptions in the AI Answer Era: What Works

The description tag still has one job, and it is smaller than it used to be. Here is what to write, and what matters more.

12 August 2026 · 4 min read

All guides →

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?
No. A Disallow rule stops crawling, not indexing. Google can still list a blocked URL it finds through links, showing it without a description. To keep a page out of results, allow crawling and use a noindex meta robots tag on the page itself.
Why does my Disallow rule not work?
Usually a longer rule outranks it. Matching is by pattern length, not file order, and Allow beats Disallow on a tie. Case matters too: /Blog/ and /blog/ are different paths. Test the exact URL here and read the rule the tester names as the winner.
Is crawl-delay respected?
Not by Google, which ignores the directive entirely and asks site owners to use crawl rate settings instead. Bing and Yandex have historically honoured it. Leaving it in your file costs nothing but do not rely on it to control Googlebot.
Can I put noindex in robots.txt?
It does nothing on Google. Support for an unofficial noindex directive in robots.txt ended on 1 September 2019. Use a noindex meta robots tag or an X-Robots-Tag HTTP header on the page you want kept out of search results.
Does one robots.txt cover my subdomains?
No. The file applies to one scheme, host and port only. A blog on blog.example.com needs its own robots.txt at blog.example.com/robots.txt, and the https version of a site is treated separately from the http version.