Throttle - Smart routing for LLM inference, save thousands on API costs
by•
We built Throttle because teams are bleeding money on LLM inference. Your bill keeps climbing even though you're not using better models. Throttle sits between your code and Claude/OpenAI, automatically routes requests to cheaper models when quality isn't sacrificed, caches intelligently, and batches efficiently. Result: 70% cost cuts, same performance. Built for startups and companies already paying $1000+/month on inference.
Replies
Hey Product Hunt 👋
I'm Kushagra. I built Throttle because I kept changing my LLM server's config and could never tell if it actually got cheaper, or if I just measured a quiet minute.
Measure. Change. Know.
- Measure what your model really costs per million tokens, on any OpenAI-compatible endpoint (vLLM, SGLang, Ollama, TGI).
- Change anything: a flag, a quantization, a new GPU.
- Know if it worked: CHEAPER, MORE EXPENSIVE or NO WINNER. It won't call a winner until repeat runs have measured the noise.
How it works:
Throttle runs repeated measurement blocks with 95% confidence intervals, fingerprints your serving config, and compares each run with your history.
On a real A100, one vLLM flag took cost from $0.746 to $0.234 per million tokens, verified with a six-run counterbalanced protocol.
Open source and local: `pipx install throttle-pro`. Add the GitHub Action and a deploy that costs more fails CI.