Prefactor
Evaluate your AI Agents in real-time
1.5K followers
Evaluate your AI Agents in real-time
1.5K followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
That's exactly the gap that matters. An agent can produce a perfect transcript and still fail the task in the real world. We've seen jobs report "success" while nothing actually changed. That's why the real evaluation isn't what the agent said—it's what actually happened. Can Prefactor validate external outcomes, like confirming a page was updated, a record was written, or an API state changed, instead of only grading the execution log?
Prefactor
@md_khayruzzaman Absolutely -- we've designed it so that those outcomes can be recorded alongside the agent activity. They are a vital part of understanding the full outcome of an agent run.
Prefactor
@md_khayruzzaman The way you'd handle that would be either connecting Prefactor directly into the MCP of your product and then associating the actions with spans inside the run, meaning you would be able to connect outcome to evaluation/ run quality.
Other ways might be to send completed outcomes directly into Prefactor so you top and tail the process. Would love to talk to you about how you would want that to work. The idea of self improving agents only works if you have the outcome connected to drive next steps.
Real-time eval is the piece most agent stacks skip until something breaks in prod. Curious how you handle scoring when there's no ground-truth label — is it rubric-based, or are you comparing against a reference trace? Congrats on the launch.
Prefactor
@lucasjpols Hey Lucas! Appreciate the support!
Both, plus a third. You define rubrics for LLM-as-judge, technical checks handle the deterministic stuff like schema conformance, and custom spans let you pull real external state into the run so the eval grounds against what actually happened rather than a reference trace.
Is this right@joshgillies?
Prefactor
@lucasjpols @ethan_lee8 Correct! Lucas if you're interested in learning more, we'd be happy to jump on a call at some point to go over it. And thanks again for the support!
Prefactor
What kinds of failure modes or edge cases have you seen most often when an agent passes evaluation but fails in production, and how would real‑time scoring and drift alerts need to look for your team to trust them enough to act automatically?
Prefactor
@swati_paliwal Thanks for your comment Swati.
We are not just focused on edge cases. The first thing is are the agents being actually assessed against whether they are doing their job.
That contract you've built for the agent is key. Eg, what tools are they allowed to call (or not in the case of OpenAIs agent), is it looping? Are parts of the flow failing causing excessive token usage.
Evals are point in time reference points for the agents themselves. When models change, when databases get different answers, drift happens. When sampling involved 1% of all agent runs or less (and given the cost of tokens, it is increasingly less), its not the 1% that will kill you. Its the 1% of the 99% you don't eval.
Can I use Prefactor for my fleet of Hermes and OpenClaw agents ?
Prefactor
@nathan_ngz we officially support integrating with OpenClaw, and I have it on good authority that Hermes is very easy to instrument also. When in doubt, just point your agent at our docs, and SDKs on Github and let it cook!
Prefactor
@nathan_ngz You can indeed -- there's an OpenClaw plugin. There's no Hermes plugin at this point but we have seen someone get Hermes to integrate itself with Prefactor to great effect!
Prefactor
@nathan_ngz OpenClaw yes, Hermes not yet
Congrats on launching! The "did it actually do it, or just say it did" framing will ring true for anyone who has put an agent in front of real users. The 25k spans a month free tier makes it easy for a small team to give this a proper try. How do the custom spans that pull context from Linear or GitHub work in practice, do you write those checks yourself or pick from templates?
Prefactor
Adapt
How are you handling latency? I imagine Prefactor is scanning through all of my agent traces and flagging things. Is it flagging proactively, or retrospectively? If it's proactively, I'd imagine latency and speed is a problem. I wouldn't want having to wait for Prefactor to finish scanning before my agent can continue its run.
Prefactor
@matteo8p hey mate. Thanks for the note.
Neither latency or speed is a problem and we don't sit on the agent run slowing it down.
We sit inside the agent via the SDK, and as a result we watch everything as it happens rather than sitting inside the agent workflow.
The tracking/ scoring happens instantaneously.
kind of worried this just becomes the new green checkmark people stop questioning. same problem, one layer up?
Prefactor
@kellyops Thanks for the question.
I see a world where we are able to leverage multiple layers of evaluation framework to reduce chance of it going wrong. It's still software though, and even the best softtware breaks. But the key thing is knowing why it broke, when it broke, and who is the person responsible. It's remarkable how those simple questions can't be answered by so many companies we speak to.
Would love to understand what else you think is needed to fix that? Matt
Prefactor
@kellyops Very valid concern!
Any scoring layer can turn into a checkmark people stop questioning, so what matters imo is whether you can see why a run scored the way it did and disagree with it, not just that a number exists.
We would rather be wrong loudly than right silently. What would make you actually trust a score rather than just accept it?