Subscribe
Sign in
p/helicone-aiOpen-source LLM Observability for Developers
Start new thread
Michael Seibel

Helicone AI - Open-source LLM Observability for Developers

by
The LLM observability platform for monitoring, debugging and improving your AI apps. Helicone is an open-source observability platform that provides a 1-line integration to access cost tracking, agent tracing, prompt management and more - get started for free.

Add a comment

Replies

Best
Justin Torre
Maker
📌
💡 Hi Product Hunt! 👋👋 Justin here, CEO of Helicone with my co-founder Cole (@cole_gottdank). We are extremely excited to launch Helicone on Product Hunt! 🧊 Background story Before launching Helicone, we developed several projects with GPT-3/GPT-J, including valyr chat, airapbattle.com, tabletalk.ai, dreamsubmarine.com, and debateai.org. For each project, we used a beta version of Helicone which gave us instant visibility into user engagement and result quality issues. As we talked to more builders and companies, we realized they were spending too much time building in-house solutions like this and that existing analytics products were not tailored for LLM workloads. 🧊 Helicone’s differentiators Other startups (LangChain) are working on similar products like LangSmith with high learning curves, rigid pricing structure and closed source, or are slow-moving giants like Datadog which isn’t exactly purpose-built for Large Language Models. 🧊 Our goals Our goal is simple. As a team of developers, we realized that we wanted a tool that “just works”, and was dead easy to start. We focus a ton on the developer experience, by allowing developers to access all observability and monitoring features with just a header. We aim to help developers surface insights or errors, so they can focus on problem-solving faster. 🧊 What’s new with this launch ⭐ Improved dashboard & request table: we make it faster to segment your data on the dashboard, and view responses and relevant monitoring metrics such as cost, latency, and time to first token (TTFT). ⭐ Introducing Sessions: trace your AI agent and RAG chatbot, debug them or refine your responses in different context in our Playground. ⭐ Introducing Prompts & Experiments: iterate on your prompts, experiment with a different dataset or model, and evaluate your responses with Scores. ⭐ Scalability & Reliability: since the beginning, we’ve made significant progress towards 100% log coverage, so all your logs will be captured. Thanks for supporting us on the launch! We are always excited to hear your feedback. Feel free to message us or “submit an issue” on GitHub, and we will respond/build it. Github: github.com/Helicone/helione Docs: docs.helicone.ai Happy developing! ✌️ And thank you @mwseibel for hunting us!
Max Savonin
@cole_gottdank @mwseibel @justintorre Helicone's focus on simplifying LLM observability and monitoring sounds like a valuable solution for developers working with large language models (LLMs). The emphasis on ease of use and developer experience aligns well with the current needs in this field. Here are some questions for further exploration: - Which LLMs (e.g., GPT-3, Jurassic-1 Jumbo) currently does Helicone support for integration? - How does Helicone ensure the security and privacy of sensitive user data processed by LLMs? - Does Helicone offer integrations with popular LLM development frameworks or platforms? As a developer with experience in LLMs and data analysis, I'm interested in Helicone's technology. Here are some potential areas for collaboration: - Contributing to expanding Helicone's support for a wider range of LLMs. - Developing functionalities for advanced error detection and performance analysis of LLM outputs. - Collaborating on further security features and data anonymization techniques for LLM integrations. I'd love to learn more about Helicone's technical architecture and explore opportunities to contribute to its development. Additionally, I'll check out the updated Helicone features and explore the GitHub repository!
Joseph Abraham
@cole_gottdank @mwseibel @justintorre Cheers to the team behind this! Wishing you all the best on this new adventure. What are you most excited about with this launch?
Cole Gottdank
Maker
@mwseibel @justintorre @kjosephabraham Thanks, Joseph! We're thrilled to have your support. For this launch, I'm excited about sharing Helicone with the open-source community and our new Sessions feature. I believe the future of AI workflows is agentic, so having the ability to track your AI agent through its entire lifecycle is a necessity. The approach we took to building Sessions is unlike others, but it follows our ethos of focusing on a seamless developer experience. How about you? Any feature that caught your eye?
Cole Gottdank
Maker
@mwseibel @justintorre @max_savonin1 Hi Max, thank you for the kind words! To answer your questions: - We have support for all LLMs! To fully integrate with Helicone, all that is required is changing your baseURL to proxy through us! All of the docs to connect to the LLM providers are available here: https://docs.helicone.ai/getting... - Helicone is fully SOC2 compliant and we follow the best security practices. For use cases where users may have strict compliance, we offer the ability to omit request and response bodies. We also have product-grade self hosting support through our Helm chart or Docker solutions. - Yes! We have tons of integrations with platforms like LangChain, LlamaIndex, LiteLLM, DeepInfra, OpenWebUI, DifyAI and many more. All of these are available on our docs! To learn about Helicone's architecture, you can read my blog I did on it here: https://upstash.com/blog/impleme... All contributions to our open-source repository would be greatly appreciated! Thanks for your question Max!
Nathaniel Finger
@cole_gottdank @mwseibel @justintorre The product looks great, cant wait to use it! Congrats on the launch, best of luck going forward.
Hahnbee Lee
Helicone is a team that delivers a product that is reliable, fast, and scalable. So impressive! Helicone to the moon 🚀
Justin Torre
Maker
@hahnbeelee Thank you Hahnbee! 🚀
Ammaramaya
@hahnbeelee I need back linkss
Han Wang
Huge fan of Helicone and what the team have built!
Justin Torre
Maker
@han_wang6 Thanks Han! Feeling is mutual with Mintlify
Ragav Yarasi
Been using Helicone for a while for our startup and it's been wonderful. Very useful. I've also met with the founders and they are so talented, helpful and humble. On many occasions they have helped me gain some perspective, especially in the founder-to-founder context. As a product, Helicone is very easy to use, intuitive and the active Discord community is quite helpful in resolving issues. Great to see it succeed!
Justin Torre
Maker
@ragavyarasi Hi Ragav. I really appreciate this message. It touched my heart <3 I would love to have more founder-to-founder chats. It is really hard to be a founder. I am super happy you are having a great experience with the product! We are super dedicated to making sure we respond to every message. As we scaled to over 10k+ users it has been a bit hard to keep up, but we are super dedicated to our community.
Cole Gottdank
Maker
@ragavyarasi, thank you so much for this! We are huge fans of you and what you're working on!
Cole Gottdank
Maker
@ragavyarasi, Ragav thanks for the support. We appreciate all of your support of Helicone and being a user for quite some time. Always reach out if you need anything!
Ragav Yarasi
@cole_gottdank haha, I am not sure how confident I am about you even remembering our conversations or products haha. But I'll be reaching out to you guys once we have our product out. Looking forward to share the journeys!
Ragav Yarasi
@justintorre yeah, we need more candid, heartfelt interactions between our kind. I'll be reaching out to you guys once I have got some of our products out this year. Looking forward to it!
flo merian
definitely upvoted for one more pushup — and the great docs! let's go, ?makers!!1
Knarik Avanesyan
@fmerian I can't wait to see it 😂
Cole Gottdank
Maker
@fmerian Thank you for the support! We're all going to be jacked after this!
flo merian
@cole_gottdank hope you've planned a team retreat or time-off after the workout session 😸
Cole Gottdank
Maker
@fmerian, the grind never stops!
Max
Amazing team and sleek product 🔥 Congrats on the launch @justintorre @cole_gottdank & team Helicone 🫶
Justin Torre
Maker
@cole_gottdank @mxcrbn Thank you Max. It was great seeing you! Appreciate you making the time to come support us on Product Hunt.
Sreejith PP
This looks super useful, especially the LiteLLM integration & Posthog exporting! Definitely trying this out at our company (MinusX)!
Justin Torre
Maker
@ppsreejith MinusX is an amazing product! I just saw your demo, and having co-pilot built directly into notebooks makes so much sense. We are built to scale and we are developer focused. I would love to discuss a possible integration with you and your team if you'd be interested feel free to reach out to me justin at helicone.ai Helicone is extremely developer-focused and would pair well with a tool like MinusX, we have also been around for over 18 months and are extremely reliable and our dashboards are incredibly fast.
Cole Gottdank
Maker
@ppsreejith, yes, we're so happy to have an integration with an awesome platform like LiteLLM! And our PostHog integration let's you build any dashboard you want with the data we collect on your LLM usage! Please share more about your company MinusX! I'd love to learn more!
Lina Lam
Maker
@ppsreejith hey Sreejith!! Thanks so much for hopping on and supporting Helicone! Congratulations on your launch earlier! Please do share more about MinusX - curious to know how you folks are using LLMs!
Kote Mushegiani
upvoting for an awesome product and a +1 pushup https://x.com/linathelam/status/...
Justin Torre
Maker
@kmushegi Kote? More like the GOAT!! thank you so much for your support! We are super stoked about this launch, and we are super excited to bring it to a brand new audience today.
Cole Gottdank
Maker
@kmushegi, thank you Kote! You're the best!
Cole Gottdank
Maker
@kmushegi, we will be jacked tomorrow!
Jose Vazquez
Excited to see what Helicone has been working on! Keep it up guys!
Justin Torre
Maker
@jose_vazquez10 Thanks Jose!
Tony Han
An open source product that streamlines the whole LLM observability ops workflow! LET'S GO! I love how you can manage prompts and versioning here - does this sync from Github? Love how you can break down the cost and things like that - are you able to get an alert for abnormality just like Sentry or Datadog? Congrats on launching this product @justintorre and team!
Cole Gottdank
Maker
@justintorre @tonyhanded Thanks for the comment and kind words Tony! We don't have a sync with Github yet, but that is on the roadmap. Ideally, we want to add Github actions support to ensure evaluations are run and pass a certain threshold before allowing a prompt change merge into main! Regarding alerts, we will alert you on cost and error rates! If it passes a specific threshold for n number of requests, you will be alerted and a cooldown period begins. It must then fall below the threshold for the alert to resolve. We send alerts via email! Thank you!!
Tony Han
@justintorre @cole_gottdank - def sharing this product with our team ASAP!
Sarmad Qadri
Helicone is awesome! I've been a fan for a while, and it's amazing to see the progress the team has made by relentlessly iterating. Great work and congrats on the launch!
Justin Torre
Maker
@saqadri Thanks Sarmad! We are super excited to be working with you and your team at Last Mile and cannot wait until we have a Helicone <> Last Mile integration. See you guys soon!
Marcel Tan
We use Helicone at Tusk to easily monitor our AI coding agent's LLM requests and debug performance issues. Very helpful for agentic products like ours where one run comprises many requests chained together.
Justin Torre
Maker
@marceltan Tusk gave us that magic "wow" moment. We are super happy and lucky to have Tusk as a customer and we are excited to grow together 🚀. I look forward to working together in the future!
Yori
Looks so awesome! @justintorre 🙌 Congrats and can't wait to use 🚀
Cole Gottdank
Maker
@justintorre @yoriii, we appreciate the support! Can you share what you're working on?
Knarik Avanesyan
Wow! Congrats with the launch 💙🚀
Justin Torre
Maker
@knarik Thank you! We are super excited for what's to come 🚀
Patricia Harris
Hey @justintorre this sounds like exactly what devs need! One quick question tho, does Helicone require users to have prior experience with LLMs or is it pretty intuitive? As a developer, I’d love to know if there’s a learning curve before diving in.
Justin Torre
Maker
@patriciaharris Hi Patricia, thanks for your message. You're right! This is the reason we built Helicone in the first place. Helicone works by hooking into an existing LLM that you are using within some AI app. However, it does not require experience with LLMs, and as a developer, you should have no problem picking it up! :) We built our platform for developers who have no AI experience.
Kyrylo Silin
Hey Justin, Do you have plans to support other LLM providers beyond OpenAI in the future? Congrats on the launch!
Justin Torre
Maker
@kyrylosilin thank you for your support! Right now Helicone supports almost every provider with our gateway integration. https://docs.helicone.ai/getting... For a full list of supported providers you can see them there, and within the side bar on our docs!
Mr. Ånand
Congratulations on the launch🎉 Excited to see what Helicone is building!
Justin Torre
Maker
@astrodevil thank you!
Jonny Miles
This looks like such a professionally made tool, and super useful for those of us hacking on AI projects. Congrats on the launch! Great work 🚀
Cole Gottdank
Maker
@jonnymiles Thanks for the kind wishes Jonny! We have been heads down for over a year engineering on Helicone, so hearing comments like this make it all feel worth it. Thank you!
Andrew Birt
Great product, great team, happy customer! Super easy to integrate and start working with. Excited to see how this continues to evolve but already the best observability platform have used so far.
Cole Gottdank
Maker
@andrewbirt Thank you so much Andrew! I may be biased, but I agree :)
Izu
There's 2 products that have been the bedrock of my founding journey: GPT, and Helicone to track our GPT use. Thank you for building such a killer product, @justintorre & @cole_gottdank!
Cole Gottdank
Maker
@justintorre @izuchukwu Thanks for the comment & kind words Izu! We love working with you and your team!