robots.txt validator & tester
Paste your robots.txt to catch syntax errors and rules that quietly block your site — then test any URL against any crawler. Runs entirely in your browser.
Test a URL
Paste a robots.txt and enter a path to test.
Summary
- Groups
- –
- Rules
- –
- Sitemaps
- –
Findings
- Paste a robots.txt to validate it.
What this robots.txt validator checks
- Syntax errors — lines without a
directive: valueshape, rules that don't start with/or*. - Site-blocking rules — a stray
Disallow: /underUser-agent: *removes your entire site from search engines. It happens after staging configs leak to production more often than you'd think. - Orphan rules —
Allow/Disallowlines before anyUser-agentare ignored by crawlers. - Unsupported directives —
Crawl-delayandNoindexare ignored by Google; unknown directives are usually typos. - Missing Sitemap — a
Sitemap:line with an absolute URL helps every crawler find your pages.
How the URL tester works
The tester applies the same matching rules Google documents for its crawlers: the group is
selected by the most specific matching User-agent, then the
rule with the longest matching path wins, with
Allow beating Disallow on ties.
Wildcards * and the end anchor
$ are supported.
Your robots.txt never leaves the browser. To have it — plus your sitemap and every page — checked automatically after each deploy, try siteOS.
A bad robots.txt can deindex your whole site.
siteOS checks your robots.txt, sitemap, and every page on a schedule — and alerts you the moment a deploy blocks crawlers.