Are ai.txt and llms.txt becoming the new robots.txt?
Over the past few months, I’ve been analyzing how AI answer engines discover and interpret websites.
We’re starting to see a pattern:
• robots.txt controls crawling
• sitemap.xml helps discovery
• structured data helps search engines
But AI systems introduce something new:
AI-ready artifacts like:
ai.txt
llms.txt
ai-feed.json
structured RAG indices
Some companies are already publishing these.
Most are not.
My questions to builders and founders here:
Are you exposing structured artifacts specifically for AI systems?
Have you tested whether ChatGPT / Copilot / Perplexity can actually discover them?
Do you think ai.txt / llms.txt will become a standard like robots.txt?
Feels like we’re early in defining the “AI web layer.”
Curious whether this is niche experimentation — or the beginning of a new standard.
– Billy
Replies