Prefactor
Evaluate your AI Agents in real-time
1.5K followers
Evaluate your AI Agents in real-time
1.5K followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
Genuinely curious what "scoring every run" looks like at scale, is that sampled or are you actually evaluating 100% of production traffic? That distinction matters a lot for cost and for trust in the numbers.
Prefactor
@yolanda_c_schneider All traffic. The 100% element is sitting alongside the agents, seeing everything they do and being able to articulate what they are actually doing. Eg, what tools are they calling. Are the actions they are making write actions vs read actions. How is the pattern of the agent different from what you know to be the case. If an agent takes 3 turns to resolve a problem, why is it taking 8 (insert HITL).
The beauty of our approach is that the cost is 0 until you introduce an LLM as judge, but by using the no token approach it means you can actually target the problematic runs/ spans/ actions rather than running random samples which could catch nothing when there is something.
Prefactor
@yolanda_c_schneider right now, we score every run. This may change in the future, and if you have a preference we'd love to hear your thoughts on the matter! From a cost perspective, the data that goes into the scoring is entirely up to you. We score what you send us about your production agents, and generally the rule of thumb is something like, trace the parts of your agent required to answer the question of "did it do good job".
Prefactor
@yolanda_c_schneider There can be different approaches depending on the application. Obviously running full LLM-based evals on a high volume chatbot might not make sense. But that's when being able to filter down the conversations you do choose to run deep analysis over makes sense. You can use heuristics to do that, then use that to guide things.
Prefactor
@yolanda_c_schneider Hey Yolanda! The capture and sampling are configurable per integration, so it's your call rather than ours.
What makes full capture practical is that not every eval needs a model.
Schema conformance and risk classification are derived from the span types that ran, so most of what you want to know costs nothing per run, and LLM-as-judge only fires where you've defined it.
What volume are you at currently in terms of runs for your agents?
How do you define "quality" here? That word does a lot of work and I imagine it means something different for a support agent than a coding agent.
Prefactor
@sheikh_umair1 Absolutely right. SO we see risk and quality as two sides of the same coin. We break them out because if an agent is making a write action on a tool call, then its both a quality problem if it goes wrong and a risk problem if it doesn't land properly and affects teams reliant on the agents.
Quality to us is technical metrics, agent patterns - literally is it calling too many tools or looping too frequently but the real end game, and why we exist, is to answer the question, how do you know what your agent is doing? And more importantly, how can you tell if it doesn't do what you expect it to.
Prefactor
@sheikh_umair1 That's very true, and it's why we've designed the system to be flexible -- so you can decide how you want to approach it. The diversity in evaluation frameworks just for coding agents is a sign of how complex (and fast evolving) it is.
Prefactor
@sheikh_umair1 Very good question, quality is doing a lot of work in that sentence.
In practice you define what good looks like for your specific agent rather than us handing you a universal definition, because you are right that a support agent and a coding agent share almost nothing.
The common layer I'd say is the scoring and enforcement, not the criteria.
What would you want it measuring for the agent you are running?
'no honest way to say which agents still do their job' nails it. when the llm-judge itself drifts, what keeps the eval honest?
Prefactor
@andrewzakonov exactly right, who judges the judge, ad infinitum. How are you managing this at scale currently?
Prefactor
@andrewzakonov Our core approach is tokenless. We/ you use the agent run to spot problems, risks, abnormal behaviour. When an LLM is needed, then you bring it in on demand rather than sampling. The idea is that when an LLM is needed, its being brought in for a specific task to confirm what the tokenless approach has already found.
You could apply the same logic to bringing in a HITL when a customer support conversation is going off the rails which is discovered through sentiment analysis (tokenless)
Prefactor
@andrewzakonov That's actually a great point. Making a good judge LLM agent is actually a big challenge too - it needs just as much engineering as the thing it's judging. Luckily you can instrument the LLM-as-judge agent with Prefactor as well. (I guess you could instrument that recursively but you might run into diminishing returns at some point!)
Prefactor
@andrewzakonov 100% agree with this! What agents are you currently building?
IMO the 'act inside the same run' is what actually separates this from the dashboards that just score runs after the fact. Charting a bad run three days later never stopped anything. QQ - once an eval can hold or block at runtime, that eval is sitting in the critical path - how much latency does the inline check add before it lets a step through? congrats on the launch!
Prefactor
@artstavenka1 generally the approach is to keep everything async in the runtime, this is to ensure minimal latency is introduced into production agents. Would you favor a different approach for the agents you manage?
Prefactor
@artstavenka1 Great question. There are a couple of approaches, depending on your needs. You can choose to make your agent wait for permission to continue (synchronous) or just log the spans, and then that termination/feedback signal can come at a later stage when it's ready (asynchronous). Obviously that isn't the right approach in all scenarios but it can remove any possibility of latency. The other part of it is choosing how you're judging quality at different stages.
Prefactor
@artstavenka1 Hey Art thanks for the question! I'll let @simonru or @matt_doughty answer this :)
Congrats on the launch.
When teams try Prefactor for the first time, how do teams catch the first bad run before users notice?
Prefactor
@dmitrii_volosatov hey mate, thanks for the question. Every run is risk assessed, so if you see something in the run representing a high risk, you can insert a hold into the agent via the spans. Which stops the agent continuing. Or you can either manually or automatically insert a killlswitch, so the short answer is the customer should never see a bad run.
How are you handling this today?
Prefactor
@dmitrii_volosatov Thanks Dmitrii! Short version is we score runs as they happen rather than sampling after the fact, so a bad run gets flagged while it is still executing instead of surfacing in a dashboard the next morning.
Most teams start with loose thresholds, watch what actually trips in week one, then tighten.
Are you in production already or still at POC?
I like that Perfector focuses on real production behavior instead of only passing evaluations. Many teams discover issues after release. How do you help teams trace the exact reason behind a quality drop across different agent workflow?
Prefactor
Prefactor
@john_michael31 There are various approaches, but the core of it is being able to store quality metrics against runs, and then extract the full information about those runs. Depending on the sort of agent you're building, your coding agent can use that directly to improve things.
The runtime controls are the strongest part here. Being able to hold or block an agent when an eval fails turns evaluation into an actual safety layer, not another dashboard.
How do teams prevent a noisy or misconfigured eval from repeatedly stopping healthy production runs?
Prefactor
@adityaharish2002 Absolutely. Thanks for the comment.
The beauty of the architecture, @simon_russell1 is being able to hold, insert a hitl/ agent, kill an agent mid flow. Really we would be encouraging anyone who inserts a killswitch to be absolutely sure that its the right approach.
It becomes a tiered model, where you only employ the strongest action once it's already gone through the other steps to avoid healthy runs being stopped.
Eg, How many loops have happened, how many tool runs have occurred against the norm, how long has the run gone on.
There is an argument you could have a healthy run and it just cost a lot, so you need to be clear on what those boundaries are.