We replaced a $5k/year LLM observability tool with our own 🙂
PH Community!
Last week we built us a tool to see exactly what our LLM calls cost.
We were paying ~$5k/year for a different tool that did this, but it was not working so well for us. So we just built it.
Now we can see:
→ cost per LLM call
→ which user costs the most
→ latency
→ where it fails
→ limit per user
→ down to a single request, up to any timeframe
We might open source it because it's super helpful. Basically I want to make sure no user will ever bankrupt us just by using our product. And I want to be able to debug any LLM issue a user had in a few minutes.
Who wants to try it??
25 views
Replies
The ability to quickly trace an issue back to one request is a really practical feature. 👏
This is a smart internal build, espeially with cost visibility down to each rquest. How are you handling shared infrastructure costs when multiple users trigger the same LLM workflow?
Pretty Prompt
@sitara_ismail Hey! we've optimized the product to handle millions of requests, and we're processing millions of tokens every month! We use Supabase and fly.io if that helps! (We were on Railway before fly, but moved to fly.io a few months back and it's been very reliable)