Launching today
Prefactor
Evaluate your AI Agents in real-time
363 followers
Evaluate your AI Agents in real-time
363 followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
Prefactor
Hi everyone, Joseph here. I’m a designer at Prefactor.
Something I’m particularly interested in that’s easy to overlook: the moment Prefactor flags something mid-run, a human has to look at a screen and decide whether to step in or let it ride. That decision is only as good as the interface it happens on.
Agents generate an enormous amount of data, and most tools just show you all of it. My job is making sure that when something's going wrong, you can tell in seconds, not after scrolling through a wall of spans. Real-time control needs real-time legibility.
If you try Prefactor, I’d love to know: did you know where to look, or did you have to dig?
How does Prefactor decide what parts of the code need attention without creating unnecessary changes? Congrats @ethan_lee8 & team!
Prefactor
@ethan_lee8 @hamza_afzal_butt We dont claim to know every agent and how they work. We give you the tools to track things as they go wrong, whether thats through meta data, llm as judge evals etc.
We help the unnecessary change bit by version controlling each agent, enabling different environments so you can track the problems as they occur in dev and when you're ready push them to prod.
Prefactor
Friends, Josh here. AI Engineer @ Prefactor. I built the agent instance tracking that scores quality and flags data risk while your agent runs, plus the SDKs and docs that get teams from zero to live without guessing.
Here's the thing I keep coming back to. You can't have confidence in something you can't see. An agent in production is a bucking bronco; it'll throw you the moment you stop paying attention. I've talked to too many teams who deployed, watched it work for a week, then realised they had no idea what it was actually doing. No visibility, no guardrails, just hope.
That's the piece Prefactor owns: the quality and data risk scoring that runs in-process, flagging misbehaviour mid-gallop rather than in a trace you read after the damage is done.
If you've got agents live right now, how do you actually know they're behaving? Or are you just holding on and hoping?
I spent months building internal scripts to catch exactly this kind of drift. Would've saved me a lot of late nights to just plug into something like this instead.
Prefactor
@malka_parveen many such cases unfortunately... Which is the reason we have built Prefactor, to save your engineering team the time and burden of building, and maintaining a platform like this. We should compare notes at some point, would love to know if we missed anything you'd consider essential for keeping your agents on the straight and narrow.
How do you define "quality" here? That word does a lot of work and I imagine it means something different for a support agent than a coding agent.
'no honest way to say which agents still do their job' nails it. when the llm-judge itself drifts, what keeps the eval honest?
The runtime controls are the strongest part here. Being able to hold or block an agent when an eval fails turns evaluation into an actual safety layer, not another dashboard.
How do teams prevent a noisy or misconfigured eval from repeatedly stopping healthy production runs?