AI assistants sent us readers from six countries last week

Here is what a week of AI traffic looks like on our site, measured rather than guessed.

In 24 hours, AI crawlers requested our pages 479 times. Applebot 179, meta-externalagent 180, ChatGPT-User 74, ClaudeBot 26, GPTBot 20.

Over seven days, assistants sent us real people: three visits from chatgpt.com , one from perplexity.ai

, two from Bing, arriving from Turkey, Greece, Hong Kong, Korea, the US and China. Small numbers. Real ones, from a browser beacon where bots cannot appear.

And here is the part I actually care about. Five of those six landed on a page built from original data, one of our teardowns where we read a few hundred real reviews and counted what people said. None landed on a page that sells anything.

That matches what we see in the crawl logs too: assistants fetch our data-built pages roughly nine times more often than our product pages. Same pattern, two different instruments.

So the headline finding is not that AI ignores you. It is that AI reads the pages where you did original work, and mostly ignores the pages where you describe yourself.

Then I checked which crawlers, one by one

This is where it got interesting.

OpenAI runs several bots and they do different jobs. GPTBot collects training data. ChatGPT-User fetches a page when a person asks about it. And OAI-SearchBot is the one that decides what shows up in ChatGPT's search answers.

The first two visited us. The third has never arrived.

Our robots.txt explicitly welcomes it. So this is not a block. Something else was going on, and I found the explanation on OpenAI's own help page, in a section most people never open:

"to be included it is important to allow OAI-Searchbot to crawl your site, and ensure your site host and/or content delivery network allows traffic from our published IP addresses"

There are two conditions there, not one.

The first is the one every tool in this space checks, ours included: does robots.txt allow the bot.

The second is that your host or CDN has to actually let those requests through. OpenAI publishes the IP ranges. I fetched the file: 35 address blocks, last updated in January. A firewall rule or an aggressive bot-fighting setting can refuse the crawler at your edge while your robots.txt sits there saying "welcome".

Anthropic says the same thing from the opposite direction. Their crawler page warns that blocking their bots by IP "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

Two companies, opposite intentions, same conclusion: what your edge does matters as much as what your robots file says.

Why one number cannot tell you this

If I had a single "AI visibility score", all of the above would have collapsed into one figure and I would have learned nothing.

Instead there are three separate things going on, and each has a different fix:

Our data pages are being read constantly, which means keep making them. Our product pages are not, which is a content problem I can act on this month. And one specific crawler has not arrived, which is a plumbing question, not a content one.

Same site, same week, three different answers. "Allowed" and "crawled" are not the same word either. One is a permission you wrote down, the other is an event that either happened or did not, and only one of them shows up in your logs.

One more line from that OpenAI page, since a fair number of people are currently selling the opposite:

"There is no way to guarantee top placement."

What we are building

We are turning this into a check inside Glotier that compares three signals instead of one:

what your robots.txt declares, which AI crawlers actually reached you according to your logs, and whether they arrived from the vendor's published IP ranges.

Declared, observed, verified. If your file says yes and your logs say nothing, you are not blocked and you are not fine either. You are in a third state, and the fix there is usually a link from somewhere the crawler already goes, because permission is not discovery.

We found ours by running it on ourselves, which is the only way I trust a tool. The teardowns are working. The crawler gap is not fixed yet. I will post the numbers either way.