Open-source eval framework for AI agents

aligned to the OWASP Agentic Security Initiative Top 10

3 followers

aligned to the OWASP Agentic Security Initiative Top 10

3 followers

I just published an open-source framework for red-teaming AI agents. Not LLM chatbots — agents. The kind built on LangChain, CrewAI, AutoGPT-style architectures that use tools, call APIs, and take multi-step actions in the world. GitHub: https://lnkd.in/eCSea5ak If you're building agents and you've hit unexpected failure modes — I'd like to hear about them.

Open-source eval framework for AI agents gallery image

Free

Launch tags:Open Source•Artificial Intelligence•GitHub

Launch Team / Built With

Wispr Flow: Dictation That Works EverywhereStop typing. Start speaking. 4x faster.

Promoted

Maker

📌

Hey Product Hunt! 👋 I built safelabs-eval because I kept seeing the same pattern: teams moving fast with AI agents, shipping to production, and only discovering the failure modes after something went wrong. The problem isn't that people don't care about safety — it's that there was no practical, framework-agnostic tool to systematically test AI agents against known attack vectors before deployment. OWASP published their LLM Top 10 but there was nothing open-source that actually operationalized it for agents specifically. So I built it to work with the frameworks teams are already using — LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, and Google ADK — so you don't have to change your stack to add safety evaluation. What I'd love to hear from you: Are you currently red-teaming your AI agents before shipping? What attack vectors worry you most in production? Happy to answer any questions about the framework, the OWASP alignment, or where we're taking this next. 🙏

Report

3mo ago

Forum Threads

p/open-source-eval-framework-for-ai-agents

•

3mo ago

How are you currently testing AI agents for security vulnerabilities before shipping to production?

I've been building safelabs-eval an open-source framework for red-teaming and evaluating AI agents aligned to the OWASP LLM Top 10.

Before I launch, I'm genuinely curious about the community's current approach to agent safety:

Are you doing any adversarial testing on your agents before deploying them?
Which attack vectors concern you most prompt injection, tool misuse, privilege escalation, data exfiltration?
Are you using any existing tools, or mostly manual testing?

I built this because I couldn't find anything open-source that worked across LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, and Google ADK without forcing you to change your stack.

View all