Web Crawler Directory & Bot Database - Identify, check, manage unwanted web crawlers and bots.

Hey Product Hunt! 👋 I’m Attila, a web developer and SEO from Transylvania. I built CrawlerCheck because I was tired of seeing random User-Agents in my server logs and asking myself: "What is 'Bytespider'? Is 'ClaudeBot' malicious? Should I block 'GPTBot'?" Googling them one by one was a pain, so I decided to build the resource I wish I had. The Crawler Directory is a free, searchable database of 150+ active web bots. What makes it different? It’s not just a list. It’s a tool. 🛡️ Safety Verdicts: Instantly see if a bot is "Safe" (Search Engines), "Caution" (AI Scrapers), or "Aggressive". ✂️ Copy-Paste Rules: Get the exact robots.txt syntax to block any specific bot instantly. ⚡ The Mini-Checker: This is my favorite feature. On every bot's profile page, you can run a Live Test against your own URL to see if that specific bot is currently allowed or blocked on your site. It’s 100% free and runs on a custom high-performance Go backend and Svelte frontend. I’d love to hear which bots are spamming your logs the most lately! 😅 Cheers, Attila