Deepmark AI

LLM benchmarking tool for task-specific metrics on your data

85 followers

LLM benchmarking tool for task-specific metrics on your data

85 followers

Visit website

No-code Platforms

•

Automation tools

•

Testing and QA software

Deepmark AI is a benchmarking tool that enables assessment of several large language models (LLM) on various extrinsic (task-specific) metrics (e.g. accuracy, relevance, failure rate, latency, etc) on your own data, so your AI apps have reliable performance.

Free

Launch tags:Developer Tools•Artificial Intelligence•GitHub

Launch Team

AWS Startup AdvisorPersonalized AWS guidance that meets you where you’re already building

Promoted

Deepmark AI

Maker

📌

Hey Everybody 👋 , We are excited to open source an amazing tool that we've been using internally for some time and which helped us a lot in most of our AI projects - Deepmark AI! Deepmark AI is a benchmarking tool for GenAI builders that enables assessment of several large language models (LLM) on various extrinsic (task-specific) metrics (e.g. accuracy, relevance, failure rate, latency, etc) on your own data, so your AI applications have predictable and reliable performance. 🎯 Why we building this ? Based on IngestAI Labs' 40'000 users and dozens of AI projects we completed, we clearly see that GenAI builders and organizations need to be able to assess large languag models (LLM) on their own data to deliver predictable and reliable results that balance accuracy, precision, recall (the model’s ability to correctly identify positive cases within a given dataset), as LLM models can produce different answers to the same prompts, impeding the user’s ability to assess the accuracy of outputs. To address this challenge of reliability, we (IngestAI Labs) have developed and open sourced Deepmark AI - a benchmarking tool, that enables assessment of several large language models (LLM) on various extrinsic (task-specific) metrics on your own data. It has pre-built integration with leading Generative AI APIs such as GPT-4, Anthropic, GPT-3.5 Turbo, Cohere, AI21, and others. 🚀 DeepMark AI is a tool specifically designed for AI builders.This solution focuses on iterative assessment of extrinsic metrics to identify predictable, reliable, and cost-effective Generative AI models based on the unique needs of a particular use case and data. Deepmark AI offers comprehensive assessment of various important performance metrics such as: ⚡Question answering accuracy ⚡Text classification accuracy ⚡PII recognition accuracy ⚡Named entity recognition (NER) accuracy ⚡Summarisation quality (Relevance) ⚡Sentiment analysis accuracy ⚡Cost analysis ⚡Failure rate ⚡Accuracy ⚡Latency Deepmark AI empowers organizations to make informed decisions when navigating through the most important performance metrics of Large Language Models. ❤️ Our Mission : Our mission is to make Generative AI more accessible, organised, and useful for anyone and for any size of organisations. 🎯 Follow our roadmap on Github : Star us / watch us / fork us: https://github.com/IngestAI/deep... And join our community at : 📣 Discord : https://discord.gg/kMpbueJMtQ 📣 Twitter : https://twitter.com/ingestai_io Would love to hear your usecases & feedbacks. 🙌 Hope you loved it. ❤️ Team IngestAI Labs / Deepmark AI

Report

3yr ago

Deepmark AI

Maker

@stanislav_bakhariev thanks a lot for your feedback! Really appreciate it!

Report

3yr ago

Hey there! Your product, Deepmark AI, sounds like a fantastic benchmarking tool for large language models. I'm really excited to see it launch soon! As someone who is also preparing to launch their own product, I would love to hear any advice you have for a successful launch. Additionally, I would greatly appreciate your feedback once my product goes live. Feel free to click on the "Notify" button to receive a notification when it's ready. Thank you in advance!

Report

3yr ago

Maker

@cen_xu thank you so much for your support! We can discuss your questions in DM

Report

3yr ago

Having an AI benchmarking tool like Deepmark available to measure task-specific metrics on your data can be a game-changer. @vasyl_r_ should be proud of themselves for creating something that could potentially revolutionise how metrics are measured. Well done!

Report

3yr ago

Maker

@rep_eat thank you 🙏

Report

3yr ago