Build industrial-grade LLM applications: one API for every LLM, observability, optimization (prompts, models, etc.), evaluations, and A/B testing — all open source. Turn metrics and human feedback into smarter, faster, and cheaper LLMs. Get started in minutes.
Free
Launch Team / Built With
Render Sandboxes: a computer for every agentGive every agent its own machine. Starts in under a second
Promoted
Hi Product Hunt - we're the team behind TensorZero, an open-source LLM infrastructure project.
What is TensorZero?
TensorZero is an open-source stack for industrial-grade LLM applications:
Gateway: access every LLM provider through a unified API, built for performance (<1ms p99 latency)
Observability: store inferences and feedback in your database, available programmatically or in the UI
Optimization: collect metrics and human feedback to optimize prompts, models, and inference strategies
Evaluation: benchmark individual inferences or end-to-end workflows using heuristics, LLM judges, etc.
Experimentation: ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.
Take what you need, adopt incrementally, and complement with other tools.
Why should you use a tool like this?
Over time, these components enable you to set up a principled feedback loop for your LLM application. The data you collect is tied to your KPIs, ports across model providers, and compounds into a competitive advantage for your business.
Here are some recent blog posts we wrote that illustrate some of the benefits:
Reverse Engineering Cursor's LLM Client
Fine-tuned Small LLMs Can Beat Large Ones at 5-30x Lower Cost with Programmatic Data Curation
We hope TensorZero will be useful to many of you Hunters!
How is TensorZero different from other tools?
1. TensorZero enables you to optimize complex LLM applications based on production metrics and human feedback.
2. TensorZero supports the needs of industrial-grade LLM applications: low latency (thanks to Rust 🦀), high throughput, type safety, self-hosted, GitOps, customizability, etc.
3. TensorZero unifies the entire LLMOps stack, creating compounding benefits. For example, LLM evaluations can be used for fine-tuning models alongside AI judges.
And it's all open source!
How much does TensorZero cost?
Nothing. TensorZero is 100% self-hosted and open-source (Apache 2.0). There are no paid features.
("But really, how do you plan to make money?" PH sneak peek: next year, we're planning to launch an optional, complementary paid service focused on automated LLM optimization, abstracting away all the GPUs needed to handle that. The developer tool we're working on today will remain open source.)
How can I help?
We'd love to get your feedback: features you like, features that are missing, anything confusing in the docs, etc.
TensorZero is 100% open source, so feedback from the builder community helps us prioritize the roadmap, improve the developer experience, fill any gaps in the docs, and so on.
Thank you! Please let us know if you have any questions or feedback.
Really impressive open-source stack — the unified Gateway plus observability and A/B testing feels very production-ready. Curious: how easy is it to plug TensorZero into an existing CI/CD/GitOps pipeline, and do you provide examples for Kubernetes Helm or Argo workflows?
@jaredl Thank you!
It should be straightforward. Here's an example for Kubernetes + Helm:
https://github.com/tensorzero/tensorzero/tree/main/examples/production-deployment-k8s-helm
There are multiple companies using Kubernetes/Helm/Argo.
Congrats on launch! Can I swap model providers per request and get latency/cost dashboards out of the box?
@anwarlaksir You can swap model providers per request by changing the `model_name` during inference!
We're about to ship a latency/cost dashboard as well! Latency should come out very soon, we have an internal version already. Thanks!
Congrats on the launch! It's cool how easy TensorZero makes fine tuning - I think a lot of people skip fine tuning today because setting up data collection/curation/evaluation is such a headache.
@k_kelleher Thank you! Yes, we often hear people want to fine-tune but struggle to do it. TensorZero makes it super easy. And results can be very good!
Fine-tuned Small LLMs Can Beat Large Ones at 5-30x Lower Cost with Programmatic Data Curation
@cacti We already support fine-tuning! We provide fine-tuning in the UI, programmatically, and in Jupyter notebooks. We also support RLHF and other techniques programmatically (& planning to bring them to the UI soon!).
@cacti Thanks! I recently built new implementations for supervised fine-tuning (SFT) with OpenAI, Google, Fireworks AI, and Together AI. I'm currently working on reinforcement fine-tuning (RFT) with a couple of providers as well, with a lot more on the way! The experiments in our recent blog post utilized these implementations: Fine-tuned Small LLMs Can Beat Large Ones at 5-30x Lower Cost with Programmatic Data Curation
@mattturck Thanks Matt! Appreciate you supporting TensorZero!
We use T0 in production at Vest and it's changed the way that we interact with LLMs: now, instead of having the SDKs of multiple providers tightly coupled to our backend -- or having to maintain our own abstractions -- we just use the T0 gateway. The gateway is just the beginning, however, and what really excites us is how organizing data from our inference calls on T0 unlocks downstream parameter tuning, prompt optimization, and research into the performance of our LLM applications. This is a fantastic project, and I can't wait to see where these guys go!
@aaron_polhamus1 Thanks Aaron! It's been an honor to work with you & team. Excited to see what you guys build with TensorZero.