Launching today
ai-crawler-honeypot
A public record of what AI crawlers actually do.
0 followers
A public record of what AI crawlers actually do.
0 followers
A public log of AI crawler requests to this domain. Real client IPs are never stored. Public log at /log/today. I built this honeypot as a public log to showcase how AI crawlers act in real time, using the robots.txt standard if they follow robots.txt or if they ignore it. I also built this to give anyone using AI crawlers an insight of how there crawler is behaving on a live site.
What do AI crawlers actually do when they visit a website?
Not what companies say they do. Not what the marketing claims. What they actually do — which paths they hit, whether they respect robots.txt, whether they lie about who they are.
Nobody was recording that. Not Google. Not Cloudflare. Not the AI companies themselves. The claims existed, but the evidence didn't.
So I built the evidence. A honeypot that logs every request, verifies claimed identity against published vendor IP ranges, distinguishes verified from claimed from mismatch, and publishes the record publicly.
What made it different
I didn't build it as a product. I built it as an instrument.
I spent my career building hardware — Amstrad-era machines, IBM and Motorola motherboards. In that world, you don't trust assumptions about how a system behaves. You put test points on the board and measure. The honeypot is the same instinct applied to the web.
When I found a request claiming to be GPTBot from outside OpenAI's published ranges, that wasn't a marketing claim. It was a measurement. When I found that robots_ignored_by_verified stayed at zero across thousands of requests, that wasn't an opinion. It was data.
The finding
The thing I actually built wasn't the honeypot. It was the record.
A public log of what AI crawlers do, timestamped, verified where possible, and published so anyone can check it. The log says what happened. It doesn't say what it means. That's the point.
I built it because the claims were everywhere and the evidence wasn't. Somebody had to record what was actually happening. I had the skills, the instinct, and the patience to do it.
That's why I built it.