Prefactor
Catch your agents' mistakes before they reach customers
1.4K followers
Catch your agents' mistakes before they reach customers
1.4K followers
We evaluate every step your agents take, live. Errors get caught as they happen.
Not in a Slack thread at 1am on a Tuesday
Free
Launch Team / Built With
Framer AI AgentsDesign and publish professional sites with AI
Promoted
'no honest way to say which agents still do their job' nails it. when the llm-judge itself drifts, what keeps the eval honest?
@andrewzakonov exactly right, who judges the judge, ad infinitum. How are you managing this at scale currently?
@andrewzakonov Our core approach is tokenless. We/ you use the agent run to spot problems, risks, abnormal behaviour. When an LLM is needed, then you bring it in on demand rather than sampling. The idea is that when an LLM is needed, its being brought in for a specific task to confirm what the tokenless approach has already found.
You could apply the same logic to bringing in a HITL when a customer support conversation is going off the rails which is discovered through sentiment analysis (tokenless)
@andrewzakonov That's actually a great point. Making a good judge LLM agent is actually a big challenge too - it needs just as much engineering as the thing it's judging. Luckily you can instrument the LLM-as-judge agent with Prefactor as well. (I guess you could instrument that recursively but you might run into diminishing returns at some point!)
@andrewzakonov 100% agree with this! What agents are you currently building?
IMO the 'act inside the same run' is what actually separates this from the dashboards that just score runs after the fact. Charting a bad run three days later never stopped anything. QQ - once an eval can hold or block at runtime, that eval is sitting in the critical path - how much latency does the inline check add before it lets a step through? congrats on the launch!
@artstavenka1 generally the approach is to keep everything async in the runtime, this is to ensure minimal latency is introduced into production agents. Would you favor a different approach for the agents you manage?
@artstavenka1 Great question. There are a couple of approaches, depending on your needs. You can choose to make your agent wait for permission to continue (synchronous) or just log the spans, and then that termination/feedback signal can come at a later stage when it's ready (asynchronous). Obviously that isn't the right approach in all scenarios but it can remove any possibility of latency. The other part of it is choosing how you're judging quality at different stages.
@artstavenka1 Hey Art thanks for the question! I'll let @simonru or @matt_doughty answer this :)
Congrats on the launch.
When teams try Prefactor for the first time, how do teams catch the first bad run before users notice?
@dmitrii_volosatov hey mate, thanks for the question. Every run is risk assessed, so if you see something in the run representing a high risk, you can insert a hold into the agent via the spans. Which stops the agent continuing. Or you can either manually or automatically insert a killlswitch, so the short answer is the customer should never see a bad run.
How are you handling this today?
@dmitrii_volosatov Thanks Dmitrii! Short version is we score runs as they happen rather than sampling after the fact, so a bad run gets flagged while it is still executing instead of surfacing in a dashboard the next morning.
Most teams start with loose thresholds, watch what actually trips in week one, then tighten.
Are you in production already or still at POC?
The runtime controls are the strongest part here. Being able to hold or block an agent when an eval fails turns evaluation into an actual safety layer, not another dashboard.
How do teams prevent a noisy or misconfigured eval from repeatedly stopping healthy production runs?
@adityaharish2002 Absolutely. Thanks for the comment.
The beauty of the architecture, @simon_russell1 is being able to hold, insert a hitl/ agent, kill an agent mid flow. Really we would be encouraging anyone who inserts a killswitch to be absolutely sure that its the right approach.
It becomes a tiered model, where you only employ the strongest action once it's already gone through the other steps to avoid healthy runs being stopped.
Eg, How many loops have happened, how many tool runs have occurred against the norm, how long has the run gone on.
There is an argument you could have a healthy run and it just cost a lot, so you need to be clear on what those boundaries are.
That's exactly the gap that matters. An agent can produce a perfect transcript and still fail the task in the real world. We've seen jobs report "success" while nothing actually changed. That's why the real evaluation isn't what the agent said—it's what actually happened. Can Prefactor validate external outcomes, like confirming a page was updated, a record was written, or an API state changed, instead of only grading the execution log?
@md_khayruzzaman Absolutely -- we've designed it so that those outcomes can be recorded alongside the agent activity. They are a vital part of understanding the full outcome of an agent run.
@md_khayruzzaman The way you'd handle that would be either connecting Prefactor directly into the MCP of your product and then associating the actions with spans inside the run, meaning you would be able to connect outcome to evaluation/ run quality.
Other ways might be to send completed outcomes directly into Prefactor so you top and tail the process. Would love to talk to you about how you would want that to work. The idea of self improving agents only works if you have the outcome connected to drive next steps.
Real-time eval is the piece most agent stacks skip until something breaks in prod. Curious how you handle scoring when there's no ground-truth label — is it rubric-based, or are you comparing against a reference trace? Congrats on the launch.
@lucasjpols Hey Lucas! Appreciate the support!
Both, plus a third. You define rubrics for LLM-as-judge, technical checks handle the deterministic stuff like schema conformance, and custom spans let you pull real external state into the run so the eval grounds against what actually happened rather than a reference trace.
Is this right@joshgillies?
@lucasjpols @ethan_lee8 Correct! Lucas if you're interested in learning more, we'd be happy to jump on a call at some point to go over it. And thanks again for the support!
What kinds of failure modes or edge cases have you seen most often when an agent passes evaluation but fails in production, and how would real‑time scoring and drift alerts need to look for your team to trust them enough to act automatically?
@swati_paliwal Thanks for your comment Swati.
We are not just focused on edge cases. The first thing is are the agents being actually assessed against whether they are doing their job.
That contract you've built for the agent is key. Eg, what tools are they allowed to call (or not in the case of OpenAIs agent), is it looping? Are parts of the flow failing causing excessive token usage.
Evals are point in time reference points for the agents themselves. When models change, when databases get different answers, drift happens. When sampling involved 1% of all agent runs or less (and given the cost of tokens, it is increasingly less), its not the 1% that will kill you. Its the 1% of the 99% you don't eval.