Launching today
Prefactor
Evaluate your AI Agents in real-time
598 followers
Evaluate your AI Agents in real-time
598 followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
'no honest way to say which agents still do their job' nails it. when the llm-judge itself drifts, what keeps the eval honest?
Prefactor
@andrewzakonov exactly right, who judges the judge, ad infinitum. How are you managing this at scale currently?
Prefactor
@andrewzakonov Our core approach is tokenless. We/ you use the agent run to spot problems, risks, abnormal behaviour. When an LLM is needed, then you bring it in on demand rather than sampling. The idea is that when an LLM is needed, its being brought in for a specific task to confirm what the tokenless approach has already found.
You could apply the same logic to bringing in a HITL when a customer support conversation is going off the rails which is discovered through sentiment analysis (tokenless)
Prefactor
@andrewzakonov That's actually a great point. Making a good judge LLM agent is actually a big challenge too - it needs just as much engineering as the thing it's judging. Luckily you can instrument the LLM-as-judge agent with Prefactor as well. (I guess you could instrument that recursively but you might run into diminishing returns at some point!)
One thing I'd love to see: does the scoring account for cost and latency alongside quality or is this purely about correctness? Teams I've talked to care about all three together, not in isolation.
Prefactor
@ayla_reynolds Hey Ayla, Thanks for the comment.
The risk scoring doesnt take it into account but there are 16 or so different categories you can choose from which incorporate some of that.
The cost embedded into each agent could be used as a way to throttle an agent if they were going over their budget though. Thats just a threshold you run over the agent, using the cost in Prefactor.
Latency is already included in Prefactors quality scoring, so all you'd do is build an evaluation which takes that into account. If you're open to it, I'd love to build that in an environment and show it to you? Both are pretty straightforward inside our API/ platform.
Prefactor
@ayla_reynolds All of those things. As you say, they're all important -- and often it's a question of trading off between them.
IMO the 'act inside the same run' is what actually separates this from the dashboards that just score runs after the fact. Charting a bad run three days later never stopped anything. QQ - once an eval can hold or block at runtime, that eval is sitting in the critical path - how much latency does the inline check add before it lets a step through? congrats on the launch!
Prefactor
@artstavenka1 generally the approach is to keep everything async in the runtime, this is to ensure minimal latency is introduced into production agents. Would you favor a different approach for the agents you manage?
Prefactor
@artstavenka1 Great question. There are a couple of approaches, depending on your needs. You can choose to make your agent wait for permission to continue (synchronous) or just log the spans, and then that termination/feedback signal can come at a later stage when it's ready (asynchronous). Obviously that isn't the right approach in all scenarios but it can remove any possibility of latency. The other part of it is choosing how you're judging quality at different stages.
I like that Perfector focuses on real production behavior instead of only passing evaluations. Many teams discover issues after release. How do you help teams trace the exact reason behind a quality drop across different agent workflow?
Prefactor
Prefactor
@john_michael31 There are various approaches, but the core of it is being able to store quality metrics against runs, and then extract the full information about those runs. Depending on the sort of agent you're building, your coding agent can use that directly to improve things.
That's exactly the gap that matters. An agent can produce a perfect transcript and still fail the task in the real world. We've seen jobs report "success" while nothing actually changed. That's why the real evaluation isn't what the agent said—it's what actually happened. Can Prefactor validate external outcomes, like confirming a page was updated, a record was written, or an API state changed, instead of only grading the execution log?
Prefactor
@md_khayruzzaman Absolutely -- we've designed it so that those outcomes can be recorded alongside the agent activity. They are a vital part of understanding the full outcome of an agent run.
Prefactor
@md_khayruzzaman The way you'd handle that would be either connecting Prefactor directly into the MCP of your product and then associating the actions with spans inside the run, meaning you would be able to connect outcome to evaluation/ run quality.
Other ways might be to send completed outcomes directly into Prefactor so you top and tail the process. Would love to talk to you about how you would want that to work. The idea of self improving agents only works if you have the outcome connected to drive next steps.
The runtime controls are the strongest part here. Being able to hold or block an agent when an eval fails turns evaluation into an actual safety layer, not another dashboard.
How do teams prevent a noisy or misconfigured eval from repeatedly stopping healthy production runs?
Prefactor
@adityaharish2002 Absolutely. Thanks for the comment.
The beauty of the architecture, @simon_russell1 is being able to hold, insert a hitl/ agent, kill an agent mid flow. Really we would be encouraging anyone who inserts a killswitch to be absolutely sure that its the right approach.
It becomes a tiered model, where you only employ the strongest action once it's already gone through the other steps to avoid healthy runs being stopped.
Eg, How many loops have happened, how many tool runs have occurred against the norm, how long has the run gone on.
There is an argument you could have a healthy run and it just cost a lot, so you need to be clear on what those boundaries are.
What kinds of failure modes or edge cases have you seen most often when an agent passes evaluation but fails in production, and how would real‑time scoring and drift alerts need to look for your team to trust them enough to act automatically?
Prefactor
@swati_paliwal Thanks for your comment Swati.
We are not just focused on edge cases. The first thing is are the agents being actually assessed against whether they are doing their job.
That contract you've built for the agent is key. Eg, what tools are they allowed to call (or not in the case of OpenAIs agent), is it looping? Are parts of the flow failing causing excessive token usage.
Evals are point in time reference points for the agents themselves. When models change, when databases get different answers, drift happens. When sampling involved 1% of all agent runs or less (and given the cost of tokens, it is increasingly less), its not the 1% that will kill you. Its the 1% of the 99% you don't eval.