Verify LLM API
Check which model your AI agent is really using
Start new thread

Verify LLM API - Check which model your AI agent is really using

by•
Verify LLM API checks whether the model behind your AI agent or API matches its claim. Tell a Codex agent to run verifyllmapi.com/run, or test an OpenAI- or Anthropic-compatible endpoint. It reuses the Agent’s login in fresh sessions, samples one-token outputs, and matches their distribution with public fingerprints using Jensen–Shannon distance. Credentials and raw samples stay local. The library covers 161 exact model IDs; unknown or weak matches return INCONCLUSIVE.

Add a comment

Replies

Best
Maker
📌
Hey Product Hunt 👋 I built Verify LLM API after reading “One Token Is Enough,” a paper that shows how repeated single-token output distributions can act as behavioral model fingerprints. The smallest useful flow is one prompt: Run to verify this AI Agent’s model. For a Codex session, the Skill reuses the Agent’s host-managed login and starts fresh isolated sessions. It keeps credentials and raw samples local, then compares answer distributions with public references using Jensen–Shannon distance. The result is CONSISTENT, MISMATCH, or INCONCLUSIVE. It is behavioral evidence, not vendor attestation. The main limit today is coverage. The bundled library spans 161 exact model IDs, not every current model or routing layer. New releases, provider changes, and wrappers can shift a fingerprint. I hope other builders will use the same method to measure the newest models and share reproducible results. If you test a newer model fingerprint, please email me at . The same address is listed on verifyllmapi.com. Thank you to the authors of “One Token Is Enough” for publishing the paper, dataset, and code. This tool is a small practical build on top of their work. I’d value blunt feedback: where does the report feel clear, and where does it overstate or under-explain the evidence?

The privacy choice of keeping credentials and raw samples local is genuinely thoughtful, especially for a tool that has to handle API keys to do its job.

ran it on a couple of internal endpoints and it caught one Claude ID that was secretly routing to a smaller model, which honestly surprised me. The INCONCLUSIVE fallback is smart too.

the fingerprinting approach is clever, especially keeping credentials local. one thing i'd love to see is a way to schedule periodic verification on a cron or webhook basis, so i can get alerted if my api provider silently swaps the model under me. that would make this way more useful for production monitoring.