Openlayer is a powerful testing and observability platform for ML. It lets you collaborate with others on finding issues in models and data, debugging them, and committing new versions.
👋 Hey PH!
Firstly, thanks for hunting us @mwseibel!
We're excited to introduce Openlayer, a cutting-edge ML testing, evaluation, and observability platform built by ex-Apple ML engineers. Our mission is to help ML teams efficiently identify and resolve issues in their models.
📊 The problem
Error analysis is essential for ensuring fairness and establishing guardrails in AI. Traditional software testing platforms are designed for deterministic systems, but ML models are probabilistic, making testing a challenge.
🔧 The solution
Openlayer tackles this by offering an end-to-end solution for both pre- and post-deployment stages of the ML pipeline. We focus on data integrity, data consistency, performance, fairness, and robustness, which traditional monitoring tools often overlook.
🎁 What you get with Openlayer
1. Define goals / guardrails across categories like data integrity, performance, fairness, and robustness.
2. Add a hook in your training / data ingestion pipeline to automatically test your models against your goals.
3. Easily discover problematic subpopulations within your datasets and establish performance benchmarks.
4. Diagnose issues, gain insights into model predictions (e.g. SHAP values), and receive actionable recommendations for enhancing model performance or addressing concerns.
5. Track progress over time across all your ML projects and versions.
Our user-friendly dashboard offers a comprehensive view of your project's progress, with aggregate metrics and a timeline of committed changes. It’s easy enough that even non-technical team members can help establish guardrails and quality criteria for models.
🏖️ Sandbox version
We're happy to share that we have a sandbox version available for you to test out, and we'll be adding support for more tasks, including LLM-based ones, in the very near future.
🙌 Give Openlayer a spin and join us in revolutionizing ML development for greater efficiency and success. We'd love to hear your input, feedback, and any questions you might have in the comments!
Rishab, Vikas, Gabriel
Report
At my last job, I found myself going crazy working with so many product regressions whenever we updated our models. I’m so happy that someone is finally making a user-friendly way to better manage this workflow! Looking forward to following Openlayer and trying out more demos as you guys grow!
Just curious - I’m often limited by the types of ML tools that we can use because we’re working with sensitive PII. How does Openlayer handle the privacy and security of datasets uploaded to the platform?
@christinahuang13 Thanks for the support!
Good question! Security is our highest priority - here's a few key things we've done to help ensure the privacy of customer data:
- We offer on-prem deployments on all cloud platforms, so your data never needs to leave your infrastructure
- We're on the cusp of receiving our SOC2 Type 2 certification
- All data is encrypted in transit and at rest, and lives in an isolated private subnet
If you have more questions, definitely reach out to me at rishab@openlayer.com :)
Hey, software dev here. A niche, but so huge product at the same time. There are so many variables and in most of them you cannot hit 100%, I like that you can set goals, all in a very nice and clean UI. Amazing!
@igorpavlov Hey Igor, thanks! That is true, and exactly why we decided to build this. We used to work on ML models that shipped to 100s of millions at Siri, and there it was extremely difficult to uncover and derisk all the possible things that could go wrong for users around the world.
Appreciate the compliments about the UI - we've got an amazing design lead @ericaluzzi on our team to thank for that!
Congratulations on the launch of Openlayer!
It's great to see a cutting-edge ML testing and evaluation platform that addresses the unique challenges of probabilistic ML models. The focus on data integrity, performance, fairness, and robustness is crucial for ensuring the reliability and ethical use of AI. I'm particularly impressed by the ability to define goals, diagnose issues, and track progress over time. The user-friendly dashboard is a definite plus.
Wishing you all the best in revolutionizing ML development with Openlayer!
Very excited to see this launch - I've watched Vikas, Gabe, Rish, Erica and the team build this over the last couple years. They've been through many iterations to nail down a thoughtful, beautiful product and workflow. We're big fans of Openlayer at Dream3D!
Wooow! As a product strategist, I'm excited to see the launch of OpenLayer on Product Hunt. The most attractive feature of OpenLayer is its powerful and flexible API that allows developers to create customized and interactive maps and visualizations for their projects. This feature will enable users to easily integrate their data and create beautiful and insightful maps for various use cases.
What steps did you take to ensure the API was user-friendly and accessible for developers with varying levels of expertise?
Additionally, how do you plan to continue improving and expanding the API to meet the evolving needs of your users in the future?
@anastasiia_veselova Hey Anastasiia thanks for the kind words!
The API was one of the components of our platform that we started with, actually. Our whole thesis centers around bringing in more than just engineers in on the model development and QA process. If you have more eyes on it, and more diverse perspectives, then you're likely to find more biases or insights about the model.
The UI is designed in a way to be clear and friendly to all sorts of users, and enables collaboration through comments. But we wanted to make sure that even onboarding models and data to the platform through our API was also simple, so we built a framework that allows people to get started with only a few lines of Python.
In terms of expanding the API, this is actually something we are focusing on in the next few weeks. We will be publishing some easy-to-follow guides on how to use the API to integrate into more advanced ML pipelines that auto-train models on changes to the data / code. Additionally, we'll be opening up more endpoints to allow teams to get alerts whenever they make changes to their model if the newest version does not pass all your goals in Openlayer.
If you have more ideas, we'd love to hear them! Send me a note at vikas@openlayer.com
@vikasnair Vikas, looks like sooo good! Congrats once again! It would be great to support each other during the launch process :)
Stay tuned. We'll roll out Lumos' launch at Product Hunt very soon. You may test it out or use Notify me button for future updates! I appreciate your support, and we can’t wait for you to join our Lumos community!
Replies
Openlayer
Openlayer
Quilgo
Openlayer
Openlayer
Openlayer
Openlayer
Openlayer
Alloy Credit Underwriting
Openlayer
Openlayer
Lumos
Openlayer
Lumos