Waleed Arshad

Two runs of the same prompt can diverge because one is anonymous while the other inherits account history, location, or a selected search/grounding mode. If a dashboard blends those observations, the average looks precise but is not reproducible.

A practical split:

1. Controlled baseline fixed prompt version, market/language, engine and mode; a fresh session; timestamp; and the cited URLs.

2. Personalized diagnostic explicitly tagged as a returning session or profile, with only scenario-level interpretation.

Most analytics products are designed to always return a number. For AI-search visibility, I think a trustworthy product needs the opposite safeguard: the ability to withhold a score when the underlying evidence is not decision-grade.

At Corank, I am testing five publication gates:

1. Traceability every chart point must lead back to the exact prompt, raw model answer, citations, timestamp, collection status, and methodology version.

2. Denominator integrity failed and excluded runs must remain visible. A timeout cannot silently become brand absent, and an ineligible observation cannot quietly disappear.

A headline crawler-access score should disappear when the evidence cannot support it.

I would suppress the composite in four cases:

1. A critical public URL fails. A broken homepage, primary service page, or documentation entry point should not be averaged away by healthy low-priority samples.

2. The robots.txt artifact or policy inventory is missing. Without the exact policy input and expected assertions, a conformance percentage is not reproducible.