Giskard - Open-source testing framework for LLMs & ML models

Hello Product Hunt, I'm Alex, here with Jean-Marie, Andrey, and the rest of the Giskard team. We're thrilled and slightly nervous to present Giskard 2.0. This has been 2 years in the making, involving a group of passionate ML engineers, ethicists, and researchers, in partnerships with leading standard organizations, such as AFNOR and ISO. So, why Giskard? Because we understand the dilemma you face. Manually creating test cases, crafting reports, building dashboards, and enduring endless review meetings - testing ML models can take weeks, even months! With the new wave of Large Language Models (LLMs), testing models becomes an even more impossible mission. The questions keep coming: Where to start? What issues to focus on? How to implement the tests? 🫠 Meanwhile, the pressure to deploy quickly is constant, often pushing models into production with unseen vulnerabilities. The bottleneck? ML Testing systems. Our experience includes leading ML Engineering at Dataiku and years of research in AI Ethics. We saw many ML teams struggling with the same issues: slowed down by inefficient testing, allowing critical errors and biases to slip into production. Current MLOps tools fall short. They lack transparency and don’t cover the full range of AI risks: robustness, fairness, security, efficiency, you name it. Add to this compliance to AI regulations, some of which could be punitive, costing up to 6% of your revenue (EU AI Act). Enter Giskard: 📦 A comprehensive ML Testing framework for Data Scientists, ML Engineers, and Quality specialists. It offers automated vulnerability detection, customizable tests, CI/CD integration, and collaborative dashboards. 🔎 An open-source Python library for automatically detecting hidden vulnerabilities in ML and LLMs, tackling issues from robustness to ethical biases. 📊 An enterprise-ready Testing Hub application with dashboards and visual debugging, built to enable collaborative AI Quality Assurance and compliance at scale. ∭ Compatibility with the Python ML ecosystem, including Hugging Face, MLFlow, Weights & Biases, PyTorch, Tensorflow, and Langchain. ↕️ A model-agnostic approach that serves tabular models, NLP, and LLMs. Soon, we'll also support Computer Vision, Recommender Systems, and Time Series. Equip yourself with Giskard to defeat your AI Quality issues! 🐢🛡️ We build in the open, so we’re welcoming your feedback, feature requests and questions. For further information: Website: https://www.giskard.ai/ GitHub: https://github.com/Giskard-AI/gi... Discord Community: https://gisk.ar/discord Best, Alex & the Giskard Team

Artificial intelligence is expanding rapidly, and product teams are under pressure to integrate new AI features into their products quickly. While it may be easy to put together a prototype for a demonstration, releasing it for production comes with many other considerations. In particular, LLMs can produce hallucinations (misinformation) and show biases. These errors can harm both product quality and the trust users place in our technology.

Hey Alex and the Giskard Team! 👋 I'm amazed at the comprehensive approach Giskard 2.0 is providing to tackle ML Testing. The fact this is open-source and compatible with multiple platforms makes it even more appealing. 🚀 Could you shed some light on how Giskard manages AI risk range and the mitigation process? Also, it's impressive how you aim to cover different model types. 🙌 Looking forward to seeing Giskard revolutionize ML Testing!

Exciting! Does Giskard detect data drift (when production data significantly changes from the training set) and how?

Congrats on the launch 🚀 Loved it's open source 👌

Hey Giskard team, big congrats on the launch! Giskard 2.0 sounds like a real game-changer. Just one question: How does Giskard handle the detection of ethical biases in ML models? And a suggestion - how about a feature simulating the impacts of detected vulnerabilities? It could help teams make more informed decisions. Keep up the great work!

When you see what current ML models do, Giskard is definitely the product that help improving their quality

Impressive, congrats on your launch!

Congratulations on the launch !! Really great to see such great products being in open-source

In our company, we face a serious problem when creating our internal LLMs for data extraction. Our biggest problems are hallucinations and model biases. With Giskard we can finally monitor our models automatically instead of doing a lot of manual work and running several hand-crafted non-inclusive tests Highly advise you to check Giskard out!

Congrats guys, you’ve come quite a long way with this! As a ML engineer I will definitely need this! Kudos to the team for the hard work!

Congratulations on the launch. you've indeed journeyed impressively far with this!

This is probably one of the most exciting products in the MLOPs space. Ensuring the production quality of your models and reduce your technical debt is for sure a needed step but mostly not implemented by ML teams around the world. Of course, testing an LLM model is definitely a daunting task on top of the pile of work needed to train and deploy your model and Giskard makes it for you! co-author of CML

Congratulations on the launch team! I'm very curious to know about how giskard detects biases and what is the LLM behind it?

2 years to develop an entire product. You guys are very diligent and that’s commendable. This deserves respect, so I will definitely support your product. Congratulations on the launch from the Giskard team

Awesome product, awesome team! Congrats for the launch :)

Terrific product for anyone working with ML models and LLMs! Congrats on the launch team!

Congrats team ! Giskard is really impressive and a real game changer in ML world!

Huge contribution to the AI space, congrats @a1x and the Giskard team 👏👏

😃🔥 wow! Congrats.