Best Products
Launches
Launch archive
Most-loved launches by the community
Launch Guide
Checklists and pro tips for launching
News
Newsletter
The best of Product Hunt, every day
Stories
Tech news, interviews, and tips from makers
Changelog
New Product Hunt features and releases
Forums
Forums
Ask questions, find support, and connect
Streaks
The most active community members
Events
Meet others online and in-person
Advertise
Subscribe
Sign in
Alexander Hjalmarsson
#1551686
0 followers
0 following
Follow
About
Forums
Activity
Upvotes
1 Hunted
Collections
Stacks
Reviews
Badges
Veteran
Maker History
Robots.txt search engine
Most commonly disallowed paths, crawlers, etc
Sep 2019
🎉
Joined Product Hunt
December 2nd, 2018
Forums
Alexander Hjalmarsson
•
6yr ago
Robots.txt search engine - Most commonly disallowed paths, crawlers, etc
Search engine for robots.txt contents. Indexes the paths and access rules for which enabled you to find hosts that has specific disallow rules and see the most disallowed crawler agents.
1
3
View more
Report
Subscribe
Sign in
Top Product Categories
Engineering & Development
LLMs
Productivity
Marketing & Sales
Design & Creative
Social & Community
Finance
AI Agents
Engineering & Development
Vibe Coding Tools
AI Coding Agents
AI Code Editors
LLMs
AI Chatbots
AI Infrastructure Tools
Prompt Engineering Tools
Productivity
AI notetakers
Note and writing apps
Team collaboration software
Search
Marketing & Sales
Lead generation software
Marketing automation platforms
Design & Creative
Video editing
Design resources
Graphic design tools
AI Generative Media
Social & Community
Social Networking
Professional networking platforms
Community management
Finance
Accounting software
Fundraising resources
Investing
AI Agents
AI Voice Agents
AI Agent Automation
See All Categories >>
Trending categories
Top reviewed
Trending products
Top forum threads
Trending categories
Vibe Coding Tools
AI Dictation Apps
AI notetakers
Code Review Tools
No-code Platforms
Figma Plugins
Static site generators
Top reviewed
Lovable
n8n
Attio
PostHog
Vapi
Granola
Raycast
Supabase
Trending products
Lovable
Screen Studio
bolt.new
Wispr Flow
Framer
Replit
Vapi
Granola
Top forum threads
Cursor or Claude Code?
POLL: Domain or product first?
YC deadline in <2 weeks; Who's applying?
We Got into YC, Got Kicked Out, and Fought Our Way Back
How Wispr Flow found PMF through pivot
Best Vibe Coding tool so far?
Landing page roast - 48 hours only
Fix your tagline with the PH CEO
© 2026 Product Hunt
Newsletter
Apps
About
FAQ
Terms
Privacy & Cookies
Privacy Choices
Advertise
llms.txt
Contact us:
hello@producthunt.com
