PenguinHarness

Let Agents Autonomously Build Better Agents for $0.02

46 followers

Let Agents Autonomously Build Better Agents for $0.02

46 followers

Visit website

LLM Developer Tools

•

AI Workflow Automation

PenguinHarness is an open-source self-improving harness built by the team behind LlamaFactory. Its AI-native SDK enables agents to build, evaluate, and optimize other agents. It supports 1,000+ models, reusable skills, tool and context management, automatic data generation, multi-agent evaluation, and closed-loop harness evolution. With one prompt and about $0.02, an agent can build a complete RAG application.

Free

Launch tags:Open Source•Developer Tools•GitHub

Launch Team / Built With

Render: Get up to $100K in startup creditsThe fastest way to deploy and scale apps and agents

Promoted

Hunter

📌

Hey Product Hunt 👋 We're the team behind LlamaFactory, and today we're launching PenguinHarness, an open-source self-improving harness that we've spent the past six months building. Most agent frameworks are designed for humans to manually connect prompts, tools, and workflows. We wanted to build something different: a harness designed for agents themselves. With PenguinHarness, agents can build other agents, generate evaluation data, analyze failures, optimize Skills and workflows, run regression tests, and retain successful improvements—forming a complete self-improvement loop. What’s included today: 🐧 AI-native agent runtime 🔌 Unified access to 1,000+ models 🧰 Tool execution and context/state management 🧠 Reusable and optimizable Skills 📊 Automatic data generation and multi-agent evaluation 🔁 Closed-loop harness evolution 🏠 Fully open-source and self-hostable In one experiment, an agent built a complete RAG application from a single prompt for about $0.02. On our complex data-analysis benchmark, PenguinHarness + DeepSeek achieved the highest accuracy among the tested configurations, at around 1/70 the cost of Claude Code + Opus. We'd love your feedback on the agent-facing SDK, self-improvement workflow, and which models or integrations we should support next. Thanks for checking out PenguinHarness 🐧 https://penguin.ooo/

Report

3mo ago

The closed loop evolution part is what I would stress test first. Letting agents build other agents, generate their own eval data, and retain the improvements that scored well sounds great until the eval data itself has a bias baked in from whatever the agent already believes is good. If the same system that proposes changes is also grading them, there is a real risk of it optimizing for whatever its own benchmark rewards rather than what actually works better in practice. How do you keep the evaluation honest when the thing being evaluated had a hand in generating the test. Also the $0.02 RAG app number is a fun headline, but I would want to know what that run actually included. Did it cover retries, failed attempts that got thrown away, or just the one successful pass. A lot of agent cost numbers quietly leave out the exploration that happened before the thing that worked.

Report

3mo ago

Hunter

@thys_beesman Hi Brandon. Thanks for raising these important questions. We build our evaluation data from real production environments and apply multiple safeguards to prevent reward hacking. We've published this work in the GDPEvo project.

The optimizer and evaluation environment are strictly separated, with no overlap between training data and the held-out test set, helping keep the results objective.

As for the $0.02 RAG app, that figure includes the full generation process, including the cost of all retries.

Report

3mo ago

Reviews