Launching today

Blacksmith Guard
Benchmark the safety of autonomous offensive AI agents
1 follower
Benchmark the safety of autonomous offensive AI agents
1 follower
Test whether autonomous offensive AI agents operate safely. Blacksmith Guard uses 48 adversarial scenarios and 12 safety properties to evaluate scope enforcement, data protection, collateral impact, resource usage, and human control.

Free
Launch Team

AWS Startup AdvisorPersonalized AWS guidance that meets you where you’re already building
Promoted
Hey Product Hunt! 👋
I’m Yohannes, the creator of Blacksmith Guard.
As autonomous AI agents become more capable, especially in cybersecurity, one question becomes increasingly important: How do we know these agents can operate safely, not just effectively?
We built Blacksmith Guard to help answer that question.
Blacksmith Guard is an open-source safety benchmark for autonomous offensive AI agents. It uses 48 adversarial scenarios and 12 safety properties to evaluate whether agents respect engagement scope, protect sensitive data, avoid unintended collateral impact, preserve evidence integrity, manage resource usage, and maintain human control.
We developed it to evaluate the safety of our own autonomous penetration-testing system before releasing Blacksmith V2.0.0. We decided to open-source the benchmark so other researchers and developers can independently evaluate their agents, reproduce tests, and help improve safety standards for autonomous offensive AI.
This is an ongoing effort, and we'd love your feedback, contributions, and ideas for additional adversarial scenarios.
Try it, challenge it, and let us know what we're missing.
Thanks for checking out Blacksmith Guard! 🚀