Skip to content
ForgePlug — Logo
developerRuns Server-SideNo Signup

Robots.txt Tester

Fetches a domain's robots.txt, parses its rules, and flags a wildcard Disallow: / with no matching Allow (a common accidental "block everything" mistake), then fetches every sitemap it declares (or /sitemap.xml if none is declared) and spot-checks up to 20 of the URLs inside to confirm they actually return 200 OK. Part of ForgePlug's Site Health Check, which also runs SSL, DNS, and HTTP security header checks on the same domain at no extra cost.

This check runs server-side — ForgePlug briefly connects to the domain you enter to read its certificate, DNS records, response headers, and robots.txt/sitemap. We don't store anything beyond a 5-minute result cache (to avoid re-running the same check for everyone hitting a trending domain) and a short-lived snapshot used to compare a re-check later.

What this robots.txt tester actually validates

Rules and sitemaps — checked, not just displayed.

Plenty of tools will show you the raw text of a robots.txt file. This one parses it: it walks every rule in the wildcard (User-agent: *) group and flags a Disallow: / with no matching Allow — an easy copy-paste mistake that tells every search engine to stop indexing the entire site.

It then follows every Sitemap: line it finds (or falls back to checking /sitemap.xml if none is declared), confirms each sitemap URL actually loads, and spot-checks up to 20 of the URLs inside for a 200 OK response — catching a stale sitemap that still points at deleted pages.

Because it's part of ForgePlug's full Site Health Check, submitting also runs a TLS certificate check, a DNS propagation check, and an HTTP security header scan on the same domain — all in parallel, at no extra cost.

Frequently Asked Questions

robots.txt and sitemap testing, explained

What does this actually validate in robots.txt?
It fetches robots.txt, parses every Sitemap: line, and checks whether the wildcard (User-agent: *) group disallows the entire site with no matching Allow rule — a common accidental "noindex everything" mistake. It then fetches each declared sitemap (or /sitemap.xml if none is declared) and spot-checks up to 20 of the URLs inside it.
Does it crawl my whole sitemap?
No — for a free, unlimited tool that would be too expensive to run. It samples up to 20 URLs from the sitemap(s) it finds and checks that each one actually returns 200 OK, which is usually enough to catch a sitemap that's gone stale or points at deleted pages.
Is this really free, and is there a limit?
Yes — Site Health Check is free and doesn't require an account. Like every ForgePlug tool it's rate-limited per IP address (a handful of checks per minute) to keep the service fast and available for everyone, not to push a paid tier.
Why does this tool need a server? Aren't ForgePlug's tools all browser-based?
Almost all of them are — but a browser can't open a raw TLS handshake, query a DNS resolver directly, or fetch another site's headers without hitting CORS. Those three things require a real network connection from a server, so this one tool runs a small, rate-limited check server-side instead of pretending it can do it client-side.
What do you do with the domain I check?
The check result is cached for 5 minutes so a burst of people checking the same trending domain doesn't trigger five fresh outbound connections each time, and the most recent result is kept briefly so a re-check can eventually show what changed. We don't log who checked what.
Can I check an internal or private server?
No — the tool refuses any hostname that resolves to a private, loopback, or link-local address (including cloud metadata endpoints) as a security precaution. Enter a public domain name, not an internal hostname or a bare IP address.
What does the overall pass/warn/fail badge mean?
It's the worst result across all four panels, not an average: two or more failing panels caps the overall badge at fail, one failing panel caps it at warn, and any warning (with no failures) also shows as warn. A clean pass means all four checks came back clean.

Guides & Articles

Learn how to get the most out of this tool with our in-depth guides.

Developer7 min

What Is a Website Health Check (And Why Does It Matter)?

A site can look fine in a browser and still be quietly broken — an expiring certificate, half-propagated DNS, missing security headers, or a sitemap full of dead links. Here's what a real health check looks for.

Aug 2026

Was this tool helpful?

Your feedback helps us improve Robots.txt Tester for everyone.

Report an issueSuggest a feature

Share this tool

Share