Best Products
Launches
Launch archive
Most-loved launches by the community
Launch Guide
Checklists and pro tips for launching
News
Newsletter
The best of Product Hunt, every day
Stories
Tech news, interviews, and tips from makers
Changelog
New Product Hunt features and releases
Forums
Forums
Ask questions, find support, and connect
Kitty Points Leaderboard
The highest scoring community members
Streaks
The most active community members
Events
Meet others online and in-person
Advertise
Subscribe
Sign in
Alexander Hjalmarsson
#1551686
0 followers
0 following
Follow
About
Forums
Activity
Upvotes
1 Hunted
Collections
Stacks
Reviews
Badges
Veteran
Maker History
Robots.txt search engine
Most commonly disallowed paths, crawlers, etc
Sep 2019
🎉
Joined Product Hunt
December 2nd, 2018
Forums
Alexander Hjalmarsson
•
6yr ago
Robots.txt search engine - Most commonly disallowed paths, crawlers, etc
Search engine for robots.txt contents. Indexes the paths and access rules for which enabled you to find hosts that has specific disallow rules and see the most disallowed crawler agents.
1
3
View more
Report
Subscribe
Sign in
