← Back to all tools

Robots.txt Checker

Inspect crawler rules, find issues and test which paths bots can crawl.

What we analyze

File availability

Fetch status and format.

Crawler groups

User-agent directives.

Crawl rules

Allow and Disallow patterns.

Path testing

Explain the matching rule.

Sitemap references

Declared sitemap URLs.

Syntax checks

Unsupported or malformed lines.

Understand crawler access

1
Enter your websiteProvide a domain (example.com) or full URL.
2
Review rules and warningsSee the parsed directives, detected issues and recommendations.
3
Test a bot and pathCheck if a specific user-agent can access a given URL path.

Simulated permissions do not confirm a URL exists.

Frequently asked questions

What does robots.txt control?

It publishes crawl rules for robots that choose to read it. It does not secure a page, and a rule does not decide whether a URL is indexed.

Can blocked URLs still appear in search?

A Disallow tells a cooperating crawler not to fetch the path. A search engine can still know about the URL from other links. This check does not measure indexing.

How are Allow and Disallow rules matched?

The checker uses the group whose user-agent matches the product token, or the * group when none does. The longest matching pattern wins. At the same length, Allow is used. Disallow: /admin/ does not block /admin.

Does this validate the sitemap itself?

Sitemap lines are recorded as declared. The file is not fetched or validated here. Open Sitemap Validator for that check.