Launching today
Prefactor
Evaluate your AI Agents in real-time
643 followers
Evaluate your AI Agents in real-time
643 followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
Can I use Prefactor for my fleet of Hermes and OpenClaw agents ?
Prefactor
@nathan_ngz we officially support integrating with OpenClaw, and I have it on good authority that Hermes is very easy to instrument also. When in doubt, just point your agent at our docs, and SDKs on Github and let it cook!
Prefactor
@nathan_ngz You can indeed -- there's an OpenClaw plugin. There's no Hermes plugin at this point but we have seen someone get Hermes to integrate itself with Prefactor to great effect!
Prefactor
Hey Product Hunt, I'm Mukil — an engineer at Prefactor. 🔥
People always ask me:
"Mukil, you work with AI all day, isn't it crazy what it can do now?"
Sure, it's pretty cool. But I spend most of my time on the mostly ambiguous half of that sentence. Yes, it did something but was it the right thing? Did it actually do it or did it just say it did? Did it just hand someone the account details for the completely wrong account?
When an agent screws up, it often doesn't like to stop. It keeps going, fully confident and you find out hours or days later. And at that point its often when a customer's already upset or a refund went out that absolutely should not have. As an engineer this made me a little annoyed. The problem was never that I couldn’t see what the agent did. Dashboards will happily show me — right after it’s already done something I didn't want it to do. It's already leaked that PII, emailed the wrong guy or somehow spun up 15 subagents to run a database query.
With Prefactor you can see runs happening live, stopping the agent in its tracks. Every run gets scored live and lets the human know to step in if they need to. Instead of waiting 30 minutes to see this failed run on your custom dashboard (it's very pretty, I'm sure), you can step in and kill it while its executing.
In short, almost anyone can build an AI agent these days. And as the barrier to building them gets lower, the standard for running them well needs to get much higher.
We obsess over that second part, so you can spend less time investigating what your agent did at 3:14 a.m. and more time letting it do actual work.
Would love for you to try it and tell us your experience with it!
Jinna.ai
Congrats on the launch! Does your tool make agents more smart over time by building right context around them, or is it still a responsibility of an agent?
Prefactor
@nikitaeverywhere Hey Nik. Great question. The self-improving loop is something within its capabilities but not something we currently have as native.
I ran a POC for a customer where they wanted to have their rag database self improve. I did that by setting up a HITL when a conversation went badly via sentiment analysis running over every span. The rag database would then update with the answers drawn from the agent. So the rag db was self improving.
Everything is configurable so you can literally go in and do all sorts. I am passionate about working out how to use non token evals to allow 100% coverage and only use the token coverage for the triaged cases.
Would love to talk more when you've given it a go.
kind of worried this just becomes the new green checkmark people stop questioning. same problem, one layer up?
Prefactor
@kellyops Thanks for the question.
I see a world where we are able to leverage multiple layers of evaluation framework to reduce chance of it going wrong. It's still software though, and even the best softtware breaks. But the key thing is knowing why it broke, when it broke, and who is the person responsible. It's remarkable how those simple questions can't be answered by so many companies we speak to.
Would love to understand what else you think is needed to fix that? Matt
Prefactor
Hi folks, I'm Simon - CTO and other co-founder of Prefactor. The real challenge for companies now is not building agents, but managing the infrastructure and process around them. A lot of the lessons learnt from deploying traditional software at scale still apply, but agents pose new and unfamiliar challenges. It's a combination of traditional devops and HR. The approaches to risk and quality that have worked in the past need to be updated.
Prefactor makes this approachable for any engineering team. Closing the loop on turning feedback into improvements to the agent; ensuring consistency over model and prompt updates; simple ways to understand and contain risk; ways to monitor and control the actions of agents across frameworks and deployment environments.
Prefactor
Hey guys Ethan here, I'm part of the team at Prefactor.
My main focus is on the go-to-market side which means I pretty much spend my week on calls with teams and engineers running agents in production.
I usually hear the same stuff all the time.
For example: The agent worked in testing and POC, it went live, but now nobody can answer a basic question: is it doing what we told it to?
That’s where we’d come in, Prefactor gives those teams visibility into what their agents are actually doing in production, so accountability doesn't stop the moment it's in production.
If you've got an agent running live right now, what does your monitoring actually look like? Keen to know if anyone here has actually solved this properly and if so how.
Cheers!
Prefactor
Hi everyone, Joseph here. I’m a designer at Prefactor.
Something I’m particularly interested in that’s easy to overlook: the moment Prefactor flags something mid-run, a human has to look at a screen and decide whether to step in or let it ride. That decision is only as good as the interface it happens on.
Agents generate an enormous amount of data, and most tools just show you all of it. My job is making sure that when something's going wrong, you can tell in seconds, not after scrolling through a wall of spans. Real-time control needs real-time legibility.
If you try Prefactor, I’d love to know: did you know where to look, or did you have to dig?