Launching today
Inferock Bench
An independent receipt for every LLM API call
61 followers
An independent receipt for every LLM API call
61 followers
Inferock-bench is a local proxy that sits between your app and OpenAI, Anthropic, Gemini, or OpenRouter shaped calls. It captures per-call token usage, failures, and retries, then generates an independent receipt showing what you were billed and how much you're actually overpaying for.
Inferock Bench
Hey PH
We built inferock-bench because we kept paying for AI answers that died mid sentence, and nobody could tell us where the money went.
Providers give you totals. They don't give you the per call receipt you'd need to prove which answer broke, which retry ran, or which token count changed. The company that charges you also decides what counts as a failure and keeps the only detailed records.
inferock-bench runs locally as a proxy in front of OpenAI, Anthropic, Gemini, OpenRouter shaped calls. Point your existing SDK at it (change two settings: apiKey and baseURL), and it captures every call as an independent, per call record. Your provider key never touches our servers, it's used locally only, attached to provider requests.
What it catches:
- Answers cut off mid stream that still got billed
- Empty replies with billed tokens attached
- Token counts that don't match visible output
- Retries that may have silently doubled a charge
- Cache discounts you may be missing on your invoice
Every run reports a receipt: spend observed, bill-bounded money loss, time loss, and a separate "invoice-check exposure" line that never gets summed into money loss, because we don't want a louder headline at the cost of a weaker claim.
Run it in about a minute: npx inferock-bench
Its open source (FSL-1.1-Apache-2.0, converts to Apache-2.0 in 2 years).
Question for this community: has anyone here actually disputed an AI provider bill and gotten a credit? What worked?
This solves a problem I didn't know I could solve, I always assumed API billing discrepancies were just the cost of doing business. Turns out I was wrong.
Thanks @itohan_blessing_eigbadon , honestly, this is exactly what we told ourselves for months, until we actually looked at the per call receipts and couldn't unsee it. Really glad this clicked for you. Pls run it on your own traffic, come back and tell us what your receipts show, genuinely curious!
The independent receipt idea is genuinely useful. LLM costs are still surprisingly hard to audit at the individual request level.