Hey Product Hunt, I’m Logan Mosby, an SEO professional with more than 10 years of experience across technical, local, eCommerce, national, international, franchise, and enterprise SEO. I built this robots.txt validator and testing tool because small mistakes in a robots.txt file can have major consequences, especially when a site is being redesigned, migrated, launched, or updated. The file may look simple, but crawler behavior depends on user-agent groups, path matching, wildcard rules, precedence, and the exact directive that wins for a given URL. The tool is designed to make that logic easier to understand before a change is published. You can use it to: • Paste a live robots.txt file or an unpublished draft • Analyze the file without fetching or uploading anything • Identify accidental full-site blocks • Find blocked CSS, JavaScript, and rendering resources • Flag deprecated or unsupported directives such as noindex, crawl-delay, and host • Check for missing sitemap references • Review AI crawler access policies • Test a specific URL against Googlebot, Bingbot, GPTBot, or a generic crawler • See the exact Allow or Disallow rule that wins • Review findings organized as Critical, Warning, Opportunity, or Passed The parser follows Google’s publicly documented robots.txt behavior, including specific user-agent matching, wildcard paths, rule precedence, and Allow winning when matching rules have equal length. There are a few important limitations. A robots.txt file controls crawling, not guaranteed indexing. A blocked URL may still appear in search results if Google discovers it through links. Robots.txt is also publicly accessible and should never be used to protect private or sensitive content. Other crawlers may interpret unsupported fields, wildcard behavior, or precedence differently. The tool therefore treats Google’s documented behavior as the primary model while still letting users test policies for other named crawlers. Nothing entered into the analyzer is sent to a server or stored. The entire analysis runs in the browser. I’d appreciate feedback on the parsing logic, severity classifications, and any additional robots.txt checks that would make the tool more useful during launches and migrations.

Robots.txt Validator & Testing Tool

Validate robots.txt rules before you publish

Validate robots.txt rules before you publish