Self-Promotion
Show off what you're working on
Start new thread

Are ai.txt and llms.txt becoming the new robots.txt?

by•

Over the past few months, I’ve been analyzing how AI answer engines discover and interpret websites.

We’re starting to see a pattern:

• robots.txt controls crawling
• sitemap.xml helps discovery
• structured data helps search engines

But AI systems introduce something new:

AI-ready artifacts like:

  • ai.txt

  • llms.txt

  • ai-feed.json

  • structured RAG indices

Some companies are already publishing these.
Most are not.

My questions to builders and founders here:

  1. Are you exposing structured artifacts specifically for AI systems?

  2. Have you tested whether ChatGPT / Copilot / Perplexity can actually discover them?

  3. Do you think ai.txt / llms.txt will become a standard like robots.txt?

Feels like we’re early in defining the “AI web layer.”

Curious whether this is niche experimentation — or the beginning of a new standard.

– Billy

32 views

Add a comment

Replies

Best

Interesting early observation while testing this myself.

When I searched my own projects across different AI systems, the results were surprisingly inconsistent:

• Some AI answers cite Wikipedia or media sources
• Some summarize from company websites
• Some pull outdated information
• Some don't cite the company at all

Even when the official website exists.

That made me wonder if we're entering a phase similar to early SEO — where the structure and discoverability of information matters just as much as the content itself.

For example:

  • structured data

  • machine-readable artifacts

  • crawl accessibility for AI systems

Curious if anyone here has run similar experiments with their own product or startup.

Have you tried asking ChatGPT or Perplexity about your company recently?