What does this robots.txt checker do?

It fetches your robots.txt, validates directives, highlights common issues, and lets you test if a URL is allowed for a user-agent.

Does it support Allow/Disallow, wildcard (*) and ($) end anchors?

Yes. The tester uses simple pattern rules with * and $ anchors and the longest-match precedence with Allow over Disallow.

What if CORS blocks fetching robots.txt?

Use the 'Paste robots.txt' tab to validate content manually if your server disallows cross-origin fetches.

Coustom robots.txt Checker

Custom robots.txt Checker – Fast, Lightweight, SEO Tool

Robots.txt Checker & URL Tester

Fast-loading, mobile-friendly tool to fetch, validate, and test your robots.txt with actionable SEO suggestions.

Fetch robots.txt from a URL

Paste robots.txt

URL Tester (Allow/Disallow)

User-agent (e.g., Googlebot, Bingbot, *)

URL Path (e.g., /blog/post-1)

Results

Parsed Directives

Suggestions & Warnings

Raw robots.txt

What is robots.txt?

The robots.txt file tells search engine crawlers which parts of your site they can crawl. It supports directives like User-agent, Allow, Disallow, Crawl-delay, and Sitemap. While search engines may interpret rules slightly differently, this tool follows common conventions and longest-match precedence.