Best Products
Launches
Launch archive
Most-loved launches by the community
Launch Guide
Checklists and pro tips for launching
News
Newsletter
The best of Product Hunt, every day
Stories
Tech news, interviews, and tips from makers
Changelog
New Product Hunt features and releases
Forums
Forums
Ask questions, find support, and connect
Kitty Points Leaderboard
The highest scoring community members
Streaks
The most active community members
Events
Meet others online and in-person
Advertise
Subscribe
Sign in
p/ai-crawler-checker
See what your robots.txt says to 15 AI crawlers
•
0
reviews
•
1
follower
Start new thread
Unfollow
Following
Follow
trending
max
•
3h ago
AI Crawler Checker - See what your robots.txt says to 15 AI crawlers
Per-bot verdicts (allowed, partial, blocked, no rules) for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider, CCBot and 9 more, following RFC 9309. Includes a block snippet generator and an llms.txt checker. Free, no LLM calls.
1
0
Subscribe
Sign in