Deepchecks Monitoring
Open Source Monitoring for AI & ML
Start new thread
Shir Chorev

Deepchecks Monitoring - Open source monitoring for AI & ML

by
Deepchecks Monitoring
•
Deepchecks Monitoring takes the open source testing experience all the way to production: enabling you to send data over time, explore system status and receive alerts on problems that arise over time.

Add a comment

Replies

Best
Shir Chorev
Maker
📌
Hi Product Hunt! 🚀 I'm Shir Chorev, Co-founder and CTO of Deepchecks, and I’m thrilled to introduce you to our open-source monitoring solution for AI systems in production! When we launched our open-source testing package last year, we quickly received an overwhelming response with over 2.5K stars ⭐️ and more than 600K downloads! The success of our approach has motivated us to level up the Deepchecks experience for the AI ecosystem. We originally intended for our monitoring component to be closed, with a focus on companies already using Deepchecks testing. However, we quickly discovered that there were significant needs in the community for open-source monitoring. Here's why we decided to open source it: 📊 An open-source package enables teams of any size or budget to access state-of-the-art ML monitoring features, with a customizable stay-for-free-forever solution. 🔒 Users can try before committing, without sharing sensitive data with any third parties. 💪 It empowers creative, proactive individuals to quickly ideate and prototype with a powerful solution over a matter of days. So, here it is, our open-source monitoring solution! It offers a comprehensive set of features, including: ➡️ Root cause analysis capabilities that integrate seamlessly with our UI and Jupyter notebooks ➡️ Tracking checks results over time and setting alert rules to be triggered upon certain conditions ➡️ Monitoring of one model per deployment & basic user management As an open-core strategy, we also offer premium features catering to advanced teams, including advanced security, scalable deployment options, a centralized dashboard, and audit/compliance testing templates. By keeping our commitment to the open-source community, we strive to build a robust ecosystem of tools for “Continuous Validation” of AI pipelines from research to production. It’s never been easier or more accessible to monitor the health of your models & data. Star us ⭐️ on Github: https://www.github.com/deepchecks/deepchecks Useful links to dive right in: • Get Started with Deepchecks Monitoring • Join Our Slack Community for more updates and info about ML validation and monitoring
@shirch wow, sounds interesting! Congrats on your launch!
Muritala Yusuf Oluwaseyi
@shirch Fantastic tool just looking forward to reviewing it on the Easy Save AI toolbox
Agam More
Hey, congrats on the launch! I was just wondering about some real world examples of your product, care to share? I feel like it's a bit too vauge to me at the moment 🙏
Shir Chorev
Maker
Sure @agamm, happy to share a few examples: As an overview - Deepchecks Monitoring tracks metrics (overtime, in production) such as your model's performance, distribution (detecting things such as feature drift, prediction drift), and data integrity, enables receiving alerts & notification when your model's behavior deviates from the expected, and quickly understanding the problem. As for the types of use cases this is used for: it can really include any type of ML model. A few examples of popular scenarios for monitoring really cross industries, and include: - Credit risk - Pricing models - Fraud - Classification - Ranking & Recommenders Currently the open source monitoring includes monitoring of tabular models, so the data & predictions should be structured.
Agam More
@shirch awesome! Thanks for the reply! How does it integrate to those pipenllines?
Shir Chorev
Maker
@agamm The central integration for data is with our Python SDK (for uploading samples and for updating labels). In the commercial offering we have additional options (e.g. reading the data from S3). For alerts we currently support e-mail, webhooks and slack (in the commercial tiers)
Matan Mishan
Looks awesome! Love the open-core motion here...
Shir Chorev
Maker
Thanks @matan_mishan! It's great fun to be able to contribute to this amazing community, and to be able to utilize the feedback for improving the product for all 😃
Steve Lewis
Congrats Shir & team on your launch!
Shir Chorev
Maker
@steve_lw thanks for your support!
Maria
Absolutely amazing! 🤩 I'll be happy to join the crew of users, I love Deepchecks features! Congratulations on the launch to the team! 🤗🥳💪✨
philip tannor
Maker
@maria_brm glad tp hear!
Shir Chorev
Maker
@maria_brm Thanks so much for your support!
Dor Sasson
Awesome to be able to use the same checks and configuration from research testing also for monitoring! Anything planned for pre-deployment testing?
Shir Chorev
Maker
@dor_sasson1 great, happy you liked our approach :) For your question: Yes, we have also a CI offering! You can check out the docs to see how to use the open source version of it (link at the end of this comment), or alternatively reach out to us (info@deepchecks.com / via our website https://www.deepchecks.com) if you'd like to hear about the managed version, with a UI for collaborating over test results etc. Link to CI docs: https://docs.deepchecks.com/stab...
Max T.Pham
Congrats Shir & team on your launch! Open-sourcing Deepchecks Monitoring is a tremendous step towards democratizing AI and ML. As a fellow tech enthusiast, I admire your contribution to the community!
Shir Chorev
Maker
@maxtpham thanks so much for your support!
On Freund
Looks amazing! Are you planning to support the use case of GenAI/LLMs down the road?
philip tannor
Maker
@on indeed we are! It's a really interesting use case, and we're already offering a solution for it in a closed beta. Feel free to ping me if you'd like to try it out.
On Freund
@ptannor sweet! I'll reach out in private
Itamar Friedman
Congrats! Exciting! Please share more details about the data processing: Is it done locally or on a remote server? So, I am asking about data privacy if to be direct. I loved other Deepchecks tools, so I can't wait to try this one 🚀
Shir Chorev
Maker
@maritamar In the open source deployment all of the components (including all processing steps, data storage etc.) running on your premise, so you can rest assured that none of your data is sent externally! Happy to hear and looking forward for your feedback 😃
Nilay Jayswal
Congrats on the launch @shirch
Shir Chorev
Maker
@nilay1101 thanks so much
123
Next
Last