Prefactor
Evaluate your AI Agents in real-time
1.5K followers
Evaluate your AI Agents in real-time
1.5K followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
I spent months building internal scripts to catch exactly this kind of drift. Would've saved me a lot of late nights to just plug into something like this instead.
Prefactor
@malka_parveen many such cases unfortunately... Which is the reason we have built Prefactor, to save your engineering team the time and burden of building, and maintaining a platform like this. We should compare notes at some point, would love to know if we missed anything you'd consider essential for keeping your agents on the straight and narrow.
Prefactor
@malka_parveen Thanks Malka! Would love for you to try it out and get your feedback!
What sort of agents have you built?
How does Prefactor decide what parts of the code need attention without creating unnecessary changes? Congrats @ethan_lee8 & team!
Prefactor
@ethan_lee8 @hamza_afzal_butt We dont claim to know every agent and how they work. We give you the tools to track things as they go wrong, whether thats through meta data, llm as judge evals etc.
We help the unnecessary change bit by version controlling each agent, enabling different environments so you can track the problems as they occur in dev and when you're ready push them to prod.
Prefactor
@hamza_afzal_butt Hey Hamza thanks so much for the support!
Slight clarification, we don't touch your code. Prefactor sits at runtime and records what the agent actually does on every run, then optional guardrails can hold a high-risk action before it executes.
So it's less "which code needs changing" and more "which agent stopped doing its job, and stop it now."
Are you running any agents in production at the moment btw?
scoring the run live instead of charting it three days later is the part that actually matters. catching a bad agent after it already shipped the damage is just reporting.
Prefactor
@alex_watson2110 Totally agree -- evaluation needs to be part of the control loop for an agent to really be trustworthy.
Prefactor
@alex_watson2110 100%, are you building many agents yourself?
Creatium
This is such a valuable tool. As someone who ships agents to clients, an automated testing/scoring tool would really scale my QA processes.
Prefactor
Prefactor
@kelly_king3 Thanks Kelly! Try it out and let us know how you go!
Feel free to reach out to any of us if you have any questions, always keen for feedback :)
@joshgillies is a good person to ask!
Prefactor
@kelly_king3 @ethan_lee8 for sure, if there's something specific you'd like to understand about the platform and how it can apply to automated testing/quality scoring of your agents, we're available to take a call and discuss anytime!
Otherwise we hope to capture enough of what's able to be done with the platform in our documentation, so feel free to point a coding agent at it and our public repos on github to get a taste for what's possible.
Thanks again for the support!
Using this currently to secure our agents, absolutely shocking to me that people give the keys to the kingdom to their AI without considering how this all plays out. Love this product.
Prefactor
@joshua_farr Great to have you on board!
Prefactor
@joshua_farr Look how farr we have come since we met you 2 years ago. Such a farr sighted view on what we can achieve together.
Prefactor
@joshua_farr Thanks Joshua! Always keen for feedback. Feel free to reach out directly to any of us if you need any help :)
The line about real-time control needing real-time legibility stuck with me. If someone has to scroll to find the problem, the alert may as well not exist. Refreshing to see design treated as part of the safety story.
Prefactor
@rohanmanthani thanks mate. You absolutely need real time control because it's not even finding the alerts that's the problem, it's that they are retrospective and once they've happened, it's likely your reviewing your updates minutes, hours or days later when the damage has been done.
TimeToCoda
Working in high-trust environments, I don’t think the future is humans approving every AI action. It’s agents operating independently 95% of the time, with enough visibility and confidence that when they drift outside the guardrails, a human can intervene before it matters. Looks like you’re tackling that problem head on. Congrats @matt_doughty @simon_russell1 and team!
Send me my 1M spans to give it a solid rev up! ;)
Prefactor
@simon_russell1 @emotf Done. You've got to set up an agent first and then you get them in your account.
You were one of our first conversations wayyyy back. LFG!
TimeToCoda
@simon_russell1 @matt_doughty Big fan of your work. Following your journey all the way... I love seeing how this idea has progressed, and now PH launch 🤘
Prefactor
@matt_doughty @emotf Thanks! Those 1M are yours if you sign up now :D