How are you handling rising token costs?
Hey builders 👋
Token bills are creeping up as we ship more AI features, and I'm curious how the community is dealing with it.
A few things I'd love your take on:
Model switching: Do you actually bother routing between models (e.g. a cheap/fast model for simple stuff, a premium one for hard tasks)? Or is the engineering overhead not worth the savings? Would love to hear real numbers if you've measured it.
Monitoring: How do you track consumption, and do you separate customer-facing usage from internal/employee usage? Feels like these two get lumped together and it makes it hard to know what's actually driving cost. What's your setup? homegrown dashboards, a third-party tool, tagging by request?
Please give me your lessons below ... Trying to build a mental model of what "good cost discipline" looks like right now.
Replies
one thing that's helped is keeping a smaller model as the default and only switching to a agree one when the task actually needs it. it wasn't perfect at first but after a few iterations it strated making a noticeable difference. still curious how ithers decide when to route request.
@evelynmariil Thanks for your insight. I've tried the same but often find myself juggling between different models and it's quite time-consuming
i've realized prompt quality matters just as much as model choice. cleaning up prompts and removing unnecessary context cut more tokens than i expected. it's ne of those optimizations that's easy to overlook until costs start adding up.
@garrettadak57i Thanks, Garrett. Agreed, cleaning up the context definitely helps, and I agree that's true from a system-building perspective.
But from the perspective of my customers using my product, how can I reduce their LLM usage and, ultimately, my LLM costs?
one habit that's helped is reviewing AI costs every week instead of waiting for the monthly invoice.
even small changes in prompts or news feature can move the needle faster than i exected. catching those early has saved us a few surproses.
@george_rosevear Great advice, thanks a lot George
@robin_de_lacroix I have asked this query multiple times to Claude, and every time it says to set up a local LLM server when API costs go higher than the cost of the server. I haven't gone live, so I will rely on APIs right now.
Also, you need to have a backup model. My engine was running Gemini Flash 2.5 Lite, to be shut down in Oct 2026. On July 6 it was offline for a couple of hours due to some wrong config. I was doing testing and none of the systems worked. I couldn't figure out what went wrong. Can't rely on just 1 model. Now I have backup 3.1 Flash Lite in case of any issue. It's 4x more expensive. I am planning to roll out free early access for 1-2 months to check what the average per-user API cost will be like and then decide the model.
@roopesh_donde Thanks Roopesh, this is gold 🙏 local server tip makes sense and good to keep for when I'm at scale.
The backup model thing is really worth it. Do you failover automatically or switch by hand?
And measuring real per-user cost during free access is smart and that's the number I'm chasing too. Would love to hear where you land once you've got data
@robin_de_lacroix Really no point spending 5-10$k on local llm when idea is not market tested.
Failover is an automatic switch; I have asked to be notified once such a switch happens.
Per-user number is really tricky. We need to be cognizant of power users but aware of the median user. I will update the post after early access.
Two that haven't come up yet.
Not every AI feature needs an API. Our product speaks questions out loud and that runs on the browser's built in speech synthesis. Costs zero, no round trip. Worth auditing which pieces the browser already ships for free before you meter them.
And gate your AI endpoints. We found ours could be hit by anything with the URL, a scraper could have quietly run the meter for weeks. Per-session tokens fixed it. If the bill creeps while real usage doesn't, look there before touching models.
Also plus one on forced tool calls for structured output. Malformed JSON means silent retries, and retries are the most boring way to pay twice.
@venkat_a2 Great points, thanks for sharing.
Makes sense to avoid leveraging browser-native capabilities to avoid API calls, and the reminder to secure AI endpoints is a good one. Also completely agree on structured outputs, silent retries are such an easy way to waste tokens.
@robin_de_lacroix Glad it helped. The endpoint one took us embarrassingly long to think of, nobody treats their AI routes as a budget line until a stranger does.
Good luck with the mental model. Honestly this thread already sketched it: cache first, route second, evals always.
Turning off thinking_budget (reasoning) on workloads that don't require reasoning/thinking and just need simple token generation has helped me reduce cost ~10x.
Keep iterating on prompts. Preferably have a prompt manager which tracks different prompts in your system. Do analytics with that, you will come to a prompt which costs less and does similar or even better job for some workloads.
Keep your skills thin.
@shivambats Really interesting, thanks for sharing.
The point about disabling reasoning for simpler workloads makes a lot of sense, and I like the emphasis on continuously iterating on prompts rather than treating them as fixed :)
I think many teams focus heavily on switching models, but inefficient prompts often end up being the bigger source of wasted tokens.
@amard_sonal True, in the end it's still on us to learn how to prompt as efficiently as possible.
I wonder if we'll eventually see AI cost management become as standard as could infrastructure monitoring.
Order of operations is everything. We got the biggest drop from prompt caching, not from switching models, because most of our spend was the same context sent over and over. After that, confidence-based routing to a cheap model for simple calls got most of the rest. Just do not ship any of it without evals in place. Cheaper output that nobody checked is how you lose trust fast.
@shivangit26 Spot on, thanks 🙏
Prompt caching being the biggest lever is a great callout. Makes total sense that repeated context is where the money quietly leaks out, so caching it before even touching models is clearly the right order.
And "cheaper output nobody checked is how you lose trust fast" is going straight in my notes. Too easy to chase cost and forget the evals are the thing keeping it all honest.
On your second question, the customer side one, the reframe that helped me was to stop treating tokens as an infrastructure cost and start treating them as a per feature unit cost, the way you would COGS. Tag every LLM call with the feature and the user id, push it into whatever analytics you already run, and you stop staring at one scary invoice and start seeing cost per feature per user. That tells you which feature and which cohort is actually driving the bill, and it is almost never spread evenly.
Once you can see it, the biggest customer side lever is usually not the model, it is the surface you give them. Free text in means unbounded context and unbounded output. Structured inputs, sensible defaults, and a cap on how much context a request can pull cut spend before a token is ever priced. Output usually costs more than input, so anywhere you can return a short structured answer instead of prose is a saving the customer never notices.
And watch the median user, not the mean. A few power users drag the average up and panic you into changes that punish the 80 percent who cost almost nothing. Optimise for the median, handle the heavy tail on its own.
@oshylabs Thanks, this is a really interesting way to look at it.
I especially like the idea of treating tokens as feature-level COGS instead of just an infrastructure cost, and the point about optimizing for the median user rather than the mean really makes sense. Appreciate it, thanks :)