Robots.txt Tester

robots.txt is two lines of plain text with the power to remove a site from search results. The single most expensive mistake in web operations is a staging-site robots.txt containing Disallow: / that gets deployed to production along with everything else. Nothing breaks. The site loads normally, analytics keeps reporting, and traffic decays over the following weeks as search engines drop pages they have been told not to crawl. By the time someone connects the decline to a deploy, months of ranking can be gone.

There is also a persistent misunderstanding worth being precise about: Disallow does not mean 'do not index'. It means 'do not crawl'. A URL that is blocked in robots.txt can still appear in search results — typically as a bare link with no description, because the crawler was forbidden from fetching the page to see what is on it. Worse, blocking a page in robots.txt prevents crawlers from ever seeing a noindex tag on that page, so the block actively defeats the removal you were trying to achieve. If you want a page out of the index, allow crawling and serve noindex; if you want to save crawl budget on pages you do not care about, use Disallow.

The other half of the file is discovery. A Sitemap: line tells crawlers where your URL inventory lives, and unlike the rest of robots.txt it is not tied to a user-agent group — it applies globally. A sitemap that 404s, that lists URLs redirecting elsewhere, or that has gone stale after a restructure quietly wastes crawl budget on every visit.

This checker fetches robots.txt, parses every declared Sitemap: line, and specifically tests whether the wildcard User-agent: * group blocks the entire site with no matching Allow rule. It then fetches each declared sitemap — or falls back to /sitemap.xml when none is declared — and spot-checks up to 20 of the URLs inside to confirm they return 200 OK rather than errors or redirect chains.