Can we build a shared benchmark for AI agent cost accuracy?
I've been onto something. As the project has grown, I've realised something that bothers me about this entire space.
We have dozens of tools tracking LLM usage, but how do we know their numbers are actually correct?
Two tools can show different costs for the same execution, and both might be internally consistent with their own assumptions.
I'm wondering whether the ecosystem needs something beyond another observability product.
Could we build an open benchmark and test suite for measuring AI costs?
Something that includes reproducible test cases, provider billing edge cases, reference calculations, and a common way to evaluate:
Cost accuracy and Token-usage accuracy
Missing or duplicated events
Attribution accuracy across agent hierarchies
How gracefully tools handle uncertain or incomplete data
I'd love to collaborate. Would anyone be interested in helping define the benchmark, contributing edge cases, or building a reference test suite?
p/raycast
p/kuc-kuc
p/runevr
p/google-cloud-platform
p/producthunt
p/offiai
p/lasso-11
p/many-displays
p/ai-context-flow
p/kodlo
p/reso
p/dex-by-exmergo
p/htmldrag
p/dino-kids
p/jev
p/supabase
p/jot-2-0
p/beamfy
p/nexupira
p/murror
Replies
tools disagreeing on the same run is wild, you'd think tokens are just tokens. is the gap mostly cache reads and retries being counted differently? a shared set of recorded runs would help a lot
@uinerd Exactly. Cache reads and retries are definitely two of the biggest sources of discrepancies, but the problem goes deeper.
Different providers report usage differently: some expose estimated usage while others return actual billing data, and factors like failed requests, partial streams, cache writes, and model aliases can introduce inconsistencies.
A shared set of recorded runs with expected outcomes would be a great starting point. Ideally, we'd have fixtures for these edge cases and a way for different tools to run against the same data and compare their results.
That's actually the direction I'm thinking of taking this. Would you be interested in contributing some real-world edge cases or helping define what the initial benchmark should cover?
yes please. cached and thinking tokens are where I'd expect the numbers to drift first. I'd make the provider's actual invoice the reference and check everything against it
@farzam_habibi Agreed. I think invoices should be the ultimate reference for financial accuracy, wherever the provider exposes enough granularity to reconcile individual requests.
Cached tokens and reasoning tokens are particularly interesting because two systems can report the same total token count but arrive at very different costs depending on how those tokens are classified and priced.
I think we could start small:
Recorded request/response fixtures covering cache reads, cache writes, reasoning tokens, retries, and partial failures.
Expected usage and cost calculations based on provider pricing and billing records.
A reproducible test suite that any cost-tracking tool could run against.
The tricky part will probably be reconciling request-level estimates with invoice-level billing, especially when providers aggregate charges.
If you're up for it, I'd love to collaborate on defining the first set of cases. We could start with the edge cases you've encountered yourself and build from there.
@kushagra_agrawal8 retries are a good one to start with. a call that fails halfway but still gets billed is the case I'd want covered first. happy to share edge cases as we run into them