Robots.txt Tester

A robots.txt tester takes the rules in your robots.txt file and answers one precise question: for a given crawler and a given path, is that path allowed or blocked? Reading the rules by eye gets error prone fast once a file mixes several Disallow lines with Allow exceptions, so this tool applies the actual matching logic search engines use and gives a direct verdict.

How robots.txt matching actually works

Search engines first find the group of rules that applies to the crawler in question, matching the exact user-agent if one is listed, or falling back to the wildcard user-agent group written as User-agent: *. Within that group, every Allow and Disallow rule is compared against the path, and the rule with the longest matching pattern wins, not the first rule in the file. When an Allow and a Disallow rule are equally long, Allow wins. This is why a short Disallow: /admin/ can still let through a page under Disallow: /admin/public/, a common and correct pattern.

What robots.txt can and cannot do

Robots.txt controls crawling, not indexing. A Disallow rule tells well behaved crawlers not to fetch a page, which is useful for keeping crawlers out of admin areas, internal search results or duplicate parameter pages. But if a blocked URL is linked to from elsewhere, Google can still show it in search results without a description, since it never crawled the page to know what noindex says. If the goal is to keep a specific page out of search entirely, a noindex tag on a crawlable page is the correct tool, not a Disallow rule.

Testing before you ship a robots.txt change

Paste the exact file you are about to publish, then test every important path: pages you want indexed, admin or checkout paths you want blocked, and any Allow exceptions carved out of a broader Disallow. A single misplaced slash or an overly broad Disallow can silently deindex an entire section of a site, so testing the specific paths that matter before publishing catches the mistake while it still costs nothing. Auditing crawl access across a full site, and catching these issues before they ship, is standard practice in the technical SEO work Rankite runs for clients.