Launched this week

AI Crawler Access Checker

Launched this week

Check AI search and training rules in robots.txt

2 followers

Check AI search and training rules in robots.txt

2 followers

Visit website

Check robots.txt permissions for a specific page across ChatGPT search, Claude, Perplexity, Google and Bing. Compare search crawlers with training agents, including OAI-SearchBot and GPTBot. Fetch the public file or paste your own, then see the matching rule for each crawler. Free, with no signup or email required. This checks declared rules, not firewall access, indexing or citations.

Free

Launch tags:SEO

Launch Team

TinyFishSearch, fetch and browse for AI agents. Claim 30% bonus!

Promoted

Maker

📌

I'm with Firm Beacon. This free checker answers one narrow question: what does robots.txt permit for a specific page? Enter a page URL or paste robots.txt. It shows the matching rule for each supported crawler and keeps search and training permissions separate. For example, blocking GPTBot alone does not automatically block OAI-SearchBot. It does not test your firewall or prove that a page is indexed or cited. Those need separate evidence. If a result looks wrong, a public robots.txt example and the affected path would help us investigate. Please leave private pages and sensitive logs out of comments.

Report

2d ago

Forum Threads

p/ai-crawler-access-checker

•

1d ago

How do you separate AI search access from AI training access in robots.txt?

I keep seeing teams treat GPTBot, OAI-SearchBot, ClaudeBot and Google-Extended as if they had one purpose. They do not. A site may want to allow AI search retrieval while restricting training, or make a different choice for each crawler. How are you documenting that policy for your team? Do you keep a tested robots.txt example, or just edit rules when a new crawler appears? The tricky part is that robots.txt only expresses a declared preference. It does not prove that a firewall allows the request, that a page is indexed, or that an AI system will cite it. I would be interested in practical examples of how others verify the separate layers.

View all