Prefactor
Evaluate your AI Agents in real-time
1.5K followers
Evaluate your AI Agents in real-time
1.5K followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
the re-read-and-compare pattern people are describing here (rereading the page, querying the row) works because the outcome leaves a durable artifact you can go check after the fact. what about agents whose whole job is a live voice call - once the call ends there's no row to reread, the only ground truth is whatever got captured live during the call itself. curious if Prefactor's eval model assumes there's always some external state to verify against, or if it can score purely off the conversation transcript/audio when that's genuinely the only signal that ever existed.
Prefactor
@omri_ben_shoham1 Hey Omri! No, it doesn't assume durable external state.
LiveKit is a native SDK integration for us and voice is one of our case studies, so the transcript and per-turn behaviour can be the signal on its own.
Custom spans are there when external state exists, not required when it doesn't.
Are you building many voice agents? Keen to have you try us out and get your feedback!
Prefactor
"Most agents pass their evals and fail in production" matches what we saw, and the reason it happened to us was not that our evals were too easy. It was that evals measure the average and production punishes the tail.
We run an AI that answers and places real phone calls. Our offline numbers looked fine the entire time the product felt unreliable to users, because a handful of bad calls a day outweighed a few hundred good ones. A 95% agent that fails loudly loses to a 90% agent that fails safely and says "I wasn't sure, so I didn't." Nothing in an averaged eval score expresses that, so if real-time scoring gives me a distribution and a worst-case rather than a mean, that is the part I would care about most.
The other thing that cost us, and it is a scoring-design question rather than an infrastructure one: we let a cheap model judge its own capability, and it started refusing intents it genuinely supported. Refusal rate climbed while capability stayed flat. That is close to the worst failure mode available, because it looks like caution from the inside and reads as the product being broken from the outside, and no accuracy metric flags it. Judgement had to move to a separate leg with no stake in the answer.
So the question I would ask of any evaluation layer: does the scorer share context with the agent it is scoring, or is it genuinely independent? An evaluator that sees the agent's own reasoning tends to find it persuasive, which is exactly when you need it not to be.
Prefactor
@getosmo There's no pre-defined structure requiring you to use the same agent scoring itself, or a separate scoring agent. Everything you've highlighted rings true and we've heard it from other people. Building a good eval dataset is really difficult, and building it up from the real world interactions that didn't go how you wanted is the only approach that works long term.
Prefactor
@getosmo I'll let @matt_doughty and @joshgillies answer this :)
Prefactor
@getosmo @matt_doughty @ethan_lee8 only to mirror what @simonrussell_au said above.
Aside from that if there's anything you'd like to learn about the platform, we're always available to take a call and discuss.
Thanks again for the support!
"Most agents pass their evals and fail in production" is almost word for word what we say about backtests — they look great in-sample and melt the moment they trade live. The failure mode I keep running into is that the eval set slowly becomes the thing you tune against, so the score stops measuring generalization and starts measuring fit.
Does Prefactor do anything to preserve a holdout — scoring against runs the team hasn't looked at, or splitting eval traffic by time rather than at random? Real-time scoring on every run sounds exactly right for catching drift; I'm just curious how you keep the scorer from becoming the target.
Prefactor
@matt_doughty
Thanks — that clears up the dataset hygiene, which is honestly the part I'd have gotten wrong first. The failure I had in mind is slower though, and I don't think run-level isolation catches it: over weeks, people tune the agent until it scores well on the eval set you happen to have, and that set stops being a sample of production and becomes the thing that's been fit to. Nothing is ever mixed — the number just quietly stops meaning what it used to.
On the PoC → production hump: in our domain the honest answer is that we gave up closing it offline. Walk-forward and out-of-sample splits are good at telling us which strategies to discard, not which ones will work. What actually bridges it is running the thing for real with nothing at risk — the same strategy gets exported to Pine Script, fires alerts into a broker paper account, and we replay that live equity curve against what the backtest predicted for the same window. The gap between predicted and actual is the only number I really trust.
The transferable part for agents might be exactly that: a shadow deployment where you diff production behaviour against what the eval predicted, rather than a better offline eval. Curious how close your real-time scoring already is to that.
Prefactor
@yoshiaki_sakae Thats a pretty sophisticated approach, kudos! What we have tried to do with the risk profiles, which we are in the process of replicating for quality, is that you define up front what your agent should do. What tools it can access, what capabilities it has. Then we cross reference the theoretical with the actual, and that diff is what we look at - similar to your model.
For quality, the logic is the same. You define the quality schema not on the basis of an eval, but in terns of what good actually looks like to begin with even if it is in a closed env. Eg, I expect this agent to have 10 turns on average and close, I expect this agent to use 2 tools and use write in 10% of cases. Etc. To get that baseline you need to run the agent, as you say in a risk free online env. But the problem is that if it's risk free, then you aren't actually testing out the agent at all. You're doing a variation of what you said you didn't want to do.
I'd love to get you in as a design partner to help us shape the next phase of that because what you've got is ahead of the curve.
@matt_doughty You're right, and that's the sharpest version of the objection. A risk-free env is itself a distribution shift — paper fills are optimistic in ways that flatter everything: no queue position, no partial fills, no market impact. So I'd put it this way: paper trading isn't validating the edge. It's validating the plumbing.
And the plumbing turns out to be most of what kills you between PoC and production. The bugs paper mode caught for us had nothing to do with whether the strategy was any good — orders sitting unfilled because of an order-type default, position desync when the signal source and the broker disagreed about current size, a dedup key that silently dropped every symbol after the first while still returning 200 to the sender. None of those are reachable from an offline eval, and every one of them would have been real money.
The edge question does stay open until capital is at risk, and I've made peace with that: out-of-sample to discard, paper to prove the wiring, then accept the last mile is unfalsifiable in advance. For agents I'd guess the split is similar — a shadow env won't tell you the agent is good, but it'll surface tool failures, auth expiry, retries and malformed output, which is a big share of the hump.
Your "define what good looks like up front rather than deriving it from an eval" is the part I want to steal, by the way. That's a prior, not a dataset — much harder to overfit.
On the design partner offer — thank you, genuinely, that means a lot coming from today's #1. I have to pass though. I'm a solo maker with a day job and this launch is already more than I can carry, and I'd rather give you a clean no now than be the partner who goes quiet in three weeks. That said, this has been the most useful conversation I've had all week, so if you ever want a second pair of eyes on the quality-schema design, just ping me — no commitment on either side.
Prefactor
Matt, following on from the forum thread where I said the zero-token layer does most of the work: the thing I keep bumping into is that every deterministic check I own exists because something already went wrong once. Banned phrases, schema conformance, the contradicts-its-own-inputs assertion, all of it got written the morning after a bad output shipped. So my free layer's coverage is really a map of my incident history, and the failure it structurally cannot see is the one I haven't met yet.
That interacts awkwardly with scoring 100% of traffic. Full coverage on rules-someone-wrote reads as all-green on the dashboard, and all-green on 100% is a much stronger-sounding claim than all-green on a 30% sample, which is the version everyone already knows to distrust.
You told Yolanda the pattern layer asks how the agent's behaviour differs "from what you know to be the case", and that phrase is where my question lands, because that's still a declared baseline. Is there anything that flags a run as unusual without someone having said what usual is, off span shape, tool sequence, turn count? That's the piece that catches the first instance rather than the second, and it's the only part of this I've never managed to build for myself.
Prefactor
The evals pass then production drifts problem has a close cousin in ordinary end to end testing, where a suite goes green on a dev laptop and red on a slower CI box. Same root cause both times: the check was written against conditions that only held in one environment.
What I would want to know here is how you handle the scoring being non deterministic itself. If the judge is a model, its verdict drifts too, so a regression alert could mean the agent changed or the scorer changed. Do you pin the scorer version and replay old runs through new scorers to separate those two?
Congrats on the launch.
Prefactor
Hey @matt_doughty — the observe → evaluate → act loop closing all inside one run is the part I keep coming back to. Most eval tooling only gets you the first two, then hands you a dashboard and a black box. I've been digging into agent-initiated payment flows lately, and "catch it live and block before it commits" is exactly the primitive you'd want on any payment-adjacent tool call, not a retro three days later.
Also really like that you open-sourced the deterministic core (no LLM in the loop by construction) as prefactor-evals — most eval startups keep that bit proprietary as the moat. Good trust signal for a category people are rightly skeptical of right now.
Prefactor
@akbar_b Thanks Akbar, the ability to have the act loop is an intended consequence of the architecture. By not just relying on OTEL, and sitting inside the agent, we are able to see data, in real time, and then, through our SDKs, makes changes - insert HITL, kill an agent, stop a run. We're scratching the surface on what multi agent workflows involving different types of these actions can do.
@matt_doughty Appreciate the detail. Curious on the multi-agent side — when one agent gets killed or held for HITL mid-run, does that propagate to agents waiting on its output downstream, or is that on the user's orchestration layer to catch?
Prefactor
@akbar_b it's a good question.
I've not used it with subagents but assuming you set up the subagents to only activate when the prior agent has had a successful run, then it would just use Prefactors action in the same way as a failed run
@matt_doughty Appreciate you being straight about not having tested that config yet — that kind of honesty is rare on a launch thread.
The case I'm actually curious about is a bit narrower than the sequential trigger: what if the subagents are already fired off in parallel before the parent's HITL hold kicks in, so they're mid-run rather than waiting to activate? In that setup the gate you described wouldn't have anything to gate. Is that something you've thought through, or is parallel fan-out still mostly sequential in how teams are using Prefactor today?