LLMSDB
Find, verify, and compare AI usage policies across the web
Start new thread
Fatih Mert Doğancan

LLMSDB - Find, verify, and compare AI usage policies across the web

by•
LLMSDB is a searchable database of llms.txt files - the new standard for AI crawler policies. Discover how websites control AI bot access, compare policies by industry, track token counts, and add your site with dofollow SEO benefits.

Add a comment

Replies

Best
Fatih Mert Doğancan
Maker
📌
Hey Product Hunters! 👋 I'm excited to share LLMSDB with you all! As someone who's been closely watching the evolution of AI-web interactions, I noticed a gap: while llms.txt is becoming the standard for AI crawler policies (already adopted by major sites), there was no central place to explore and learn from these implementations. A few interesting insights from building this: 🎯 Over 40% of tech companies have already implemented llms.txt 📝 Average policy length is ~500 tokens, but some exceed 3,000! 🌐 Policies vary dramatically by industry (e-commerce vs news sites) What's next on our roadmap: API access for bulk analysis AI policy compliance checker Weekly trend reports Browser extension for quick llms.txt lookup I'd love to hear your thoughts! Have you implemented llms.txt on your sites? What challenges are you facing with AI crawlers? What features would make LLMSDB more valuable for your work?