I was paying $5k/year to watch our token bill. So we built our own LLM observability tool 🫠

Hands up who's worried with token costs? Yep, me too.

We were actually paying ~$5/year to keep track of our LLM costs for @Helicone AI. After the @Mintlify acquisition it wasn't the same 🥲.

So last week we built (vibe coded) our own LLM observability tool to keep track of this.

Now we can see:

→ cost per LLM call

→ which user costs the most

→ latency

→ where it fails

→ down to a single request, up to any timeframe

No name yet, we're still focused on building our own startup (@Pretty Prompt) but I'm pushing to make it available for others as it's really really helpful.

That's the nice thing about bootstrapping, you just build stuff to solve your problems, never ask for permission, and it might end up making you money too.

Anyone here also dealing with token costs and observability? If you want to give it a try let me know! 🙌