Robots.txt Tester by DevHackes lets you check if your robots.txt file is blocking the pages you actually want crawled — or worse, letting bots crawl what you meant to hide. Paste your file or fetch it live from any domain, pick a user-agent (Googlebot, Bingbot, AI crawlers), enter your URLs, and instantly see Allowed vs Blocked with the exact rule responsible. No signup, no limits — built for developers and SEOs who want quick, accurate answers.
Hey Product Hunt! 👋 We built this after one too many "why isn't Google indexing my page" support tickets that turned out to be a bad robots.txt rule. It's completely free, works with any domain, and supports testing against multiple crawlers — including newer AI bots. Would love your feedback!