Tokenwise
A smart LLM proxy that shows where you're overpaying
186 followers
A smart LLM proxy that shows where you're overpaying
186 followers
Tokenwise is a one-line LLM proxy (OpenAI-compatible baseURL) for makers and small teams. It learns from your real requests, shows exactly where you're overpaying, proven with quality checks on your own traffic, not public benchmark, and lets you apply the fix in one click while it verifies the savings in real dollars.
Tokenwise
@tofil congrats on the launch Theo, this is very useful (I can never match the advertised input/output costs to my work either). What's the overhead fo r this and how deep does it go reporting wise?
Tokenwise
Thanks @zolani_matebese , really appreciate it
On overhead: the proxy runs on Cloudflare Workers at the edge, so we add ~30-50ms p50 in most regions (the actual provider call dominates latency anyway).
On reporting depth, here's what you get per request:
Exact cost (we re-tokenize and apply current pricing tables, so the "I can never match the bill" problem you mentioned goes away)
Input/output token counts, latency (TTFT + total), status, error type if any
Full prompt/response payload if you opt-in per project (off by default for privacy)
Model + provider + project + custom tags you set
And on top of that, aggregations by prompt template (we cluster semantically), recommendations with quality proof on your own data, and a "saved this month" counter that tracks the impact of applied recos in real $.
@tofil Congratulations on the build! I imagine the more one builds, optimizing those token$ will become more important. Do you see this product as something primarily for heavy users/developers or do you see it benefiting those of us just getting started? I'm building my first app - would I likely see a savings?
Tokenwise
@anna_ludwinowski Honest answer: the dollar savings scale with how much you spend. At $1k/mo, cutting 30% is real money, so heavy users feel it most. But I think it's genuinely more useful when you're just starting out, for a different reason.
When you're building your first app, the problem usually isn't "I'm overpaying by 30%." It's the stuff you can't see yet. A retry loop firing 5 times, reaching for the biggest model when a cheaper one would've been fine, no caching on prompts you send over and over. You don't notice any of it until the bill shows up. Tokenwise plugs in with one line and just shows you where the money goes from your first request, so you catch that stuff early instead of learning the hard way.
It's free to start, no card needed, so there's no real downside to plugging your first app in and seeing what happens. Worst case you learn where your tokens actually go. Would love to hear what you're building!
@tofil That's very helpful, especially for us just starting out. Does it advise/warn you of any odd activity? I got robbed by a bot today so warnings would be helpful! Luckily I just happened to notice today but didn't notice the one from 2 days ago - ugh!
The "quality check on your own traffic, not public benchmarks" is the right frame, that's exactly the gap most LLM-cost tools wave at. Question for Théophile: when you replay a request on the cheaper model to verify quality, how do you score "same answer" without a human in the loop? Embedding similarity tends to be permissive and exact-match too strict.
Tokenwise
@fabriziowexare Yeah, you put your finger on exactly why we don't use either. Embedding similarity waves through answers that are subtly wrong, and exact match fails anything reworded. So we run an LLM judge against a rubric that's scoped to that one prompt template, and you tell it what "good" means for your case (correctness, format, whatever actually matters to you). It returns per-criterion scores, not one fuzzy number. We score the cheaper model on your own recent traffic, only switch if it clears your bar, then keep judging live so it rolls back automatically if quality slips. A judge isn't magic either, but scoring against your criteria on your traffic beats a public benchmark by a mile.
We've had seven AI agents running in production since last year, and token costs were a complete black box until the invoice arrived. We built a basic per-model logger ourselves -- took more engineering time than it should have. The edge case I'd push on: can you attribute spend to a specific workflow or user journey, not just a model? When you're debugging why a particular sequence of agent calls got expensive, model-level rollups aren't granular enough. That's where the real cost surprises live.
Tokenwise
@thekrew With you completely, model rollups are where the surprises hide, not where they get explained. We attribute three ways: by prompt template (grouped automatically on the system prompt, so each distinct call type is its own line), by tags you put on a workflow or journey, and per API key. So you can tag a whole agent sequence, watch what it costs end to end, then drill into which step blew up.
The shift from dashboard to apply is the right call. Every observability tool I have run hits the same wall: you have the numbers but no path from chart to fix without writing a migration script yourself. The A/B split on the apply step is the part that actually earns trust. Twenty four hours of quality scores on real traffic before a ramp is the difference between this and a vendor that just suggests a cheaper model and walks away. Question on the semantic clustering of prompt templates: how do you handle the case where two templates look semantically identical but have different system prompts that meaningfully change output? Do you cluster on the user message only or the full request shape?
Tokenwise
@zimasilevuyo That's the precise case we built the grouping around. We don't cluster on the user message, that's the variable part. We key on the system prompt plus the model, so two templates that read alike but ship different system prompts land in separate groups, because the system prompt is what actually steers the output. If there's no system prompt we fall back to the request shape (roles and message structure) instead. The bar is then measured per group, so a switch is always scoped to that exact template and never smeared across ones that just look similar. And yeah, the 24h of real-traffic scores before a ramp is the whole point for us, suggesting a cheaper model and walking away is the easy part that helps nobody.
Observe-only is probably where I’d start, especially for Claude Code spend. The scary part is the “apply” step.
Before swapping a model, does Tokenwise show exactly which traffic it will touch, and is there an easy rollback?
Tokenwise
@novamaker01
Here's how it works:
Before apply, you see exactly:
The prompt template(s) affected (with a sample of recent requests)
The estimated traffic % (e.g. "this rule will route ~12% of your project's requests")
Optional scoping: limit to a tag, a project, exclude certain endpoints
The "apply" doesn't blindly cutover. By default it runs as an A/B split, say 10% of matching traffic on the new model and you watch the quality scores + latency + cost for 24h before deciding to ramp to 100%. You can also choose immediate cutover if you prefer.
This looks awesome. I use a load balancer but I can probably use that OpenAI key and output to tokenwise but would you ever be interested in developing a built in load balancer for multi-account setups? I imagine there's far more savings to be had if your also load balancing time-based limits but I can also see where it might be out-of-scope for this project and better chained.
Tokenwise
@nohj Yeah I love this. You can already chain it today, just point your load balancer's output at Tokenwise (or put Tokenwise in front) and it still tracks cost and savings on top. A built in multi-account balancer that rotates keys to dodge per-account rate limits is something I keep coming back to, and you're right that the time-based limits are where a lot of the extra savings hide. Would love to see how your setup looks, it'd actually help me decide whether to build it in or keep it chained.
Are you talking about query optimization/compression? Something like what Google recently did with an algorithm that compresses prompts by 7x without losing quality?
Tokenwise
@natalia_iankovych A bit of both, but not only compression. There's a compress option you can switch on per route, and it runs a quality check first so it doesn't quietly wreck your outputs. The bigger wins usually come from routing the cheap calls to a smaller model and caching repeats though. Nothing as aggressive as that Google 7x thing, I keep it safe and measurable so the savings are actually trustworthy.