What does an AI crawler access audit actually prove?
Teams sometimes say “AI crawlers can access our site” as if that closes the loop. It does not.
I find four evidence layers useful:
1. Policy — what robots.txt declares.
2. Delivery — the status, redirects, content type, and challenge behavior a real request receives.
3. Observation — whether verified crawler traffic appears in server or CDN logs.
4. Citation — whether an AI answer actually references the page.
A 200 response does not prove discovery. A crawler request does not prove citation.
For a weekly audit, I would hash robots.txt, test representative public URLs, review new 4xx/5xx patterns, and keep user-agent matching separate from verified crawler identity. Collection failures should remain failures instead of being reported as “brand absent.”
I published a practical implementation with TypeScript policy checks, curl probes, SQL log queries, and a review checklist:
Corank’s AI-visibility workflow covers the answer-measurement layer:
What evidence do you require before calling a page “AI-accessible”?
Replies