Prefactor
Catch your agents' mistakes before they reach customers
1.4K followers
Catch your agents' mistakes before they reach customers
1.4K followers
We evaluate every step your agents take, live. Errors get caught as they happen.
Not in a Slack thread at 1am on a Tuesday
Free
Launch Team / Built With
Dropbox SignSimple, secure eSignatures for the way your team works.
Promoted
The live scoring plus action piece is the part that clicks for me. If an agent drifts in production, finding out three days later is already too late.
@ari_shin completely agree, too many instances where an agent goes off the rails only it to be discovered days later. A control loop for agents is essential infrastructure for taking agents into production. Let us know if you manage to give the platform a spin, we'd love your feedback!
@ari_shin Yeah, it shouldn't be a reporting question -- it's part of agent operations.
@ari_shin Yes exactly! Are you building an agent at the moment yourself?
Friends, Josh here. AI Engineer @ Prefactor. I built the agent instance tracking that scores quality and flags data risk while your agent runs, plus the SDKs and docs that get teams from zero to live without guessing.
Here's the thing I keep coming back to. You can't have confidence in something you can't see. An agent in production is a bucking bronco; it'll throw you the moment you stop paying attention. I've talked to too many teams who deployed, watched it work for a week, then realised they had no idea what it was actually doing. No visibility, no guardrails, just hope.
That's the piece Prefactor owns: the quality and data risk scoring that runs in-process, flagging misbehaviour mid-gallop rather than in a trace you read after the damage is done.
If you've got agents live right now, how do you actually know they're behaving? Or are you just holding on and hoping?
How does Prefactor decide what parts of the code need attention without creating unnecessary changes? Congrats @ethan_lee8 & team!
@ethan_lee8 @hamza_afzal_butt We dont claim to know every agent and how they work. We give you the tools to track things as they go wrong, whether thats through meta data, llm as judge evals etc.
We help the unnecessary change bit by version controlling each agent, enabling different environments so you can track the problems as they occur in dev and when you're ready push them to prod.
@hamza_afzal_butt Hey Hamza thanks so much for the support!
Slight clarification, we don't touch your code. Prefactor sits at runtime and records what the agent actually does on every run, then optional guardrails can hold a high-risk action before it executes.
So it's less "which code needs changing" and more "which agent stopped doing its job, and stop it now."
Are you running any agents in production at the moment btw?
scoring the run live instead of charting it three days later is the part that actually matters. catching a bad agent after it already shipped the damage is just reporting.
@alex_watson2110 Totally agree -- evaluation needs to be part of the control loop for an agent to really be trustworthy.
@alex_watson2110 100%, are you building many agents yourself?
This is such a valuable tool. As someone who ships agents to clients, an automated testing/scoring tool would really scale my QA processes.
@kelly_king3 Thanks Kelly! Try it out and let us know how you go!
Feel free to reach out to any of us if you have any questions, always keen for feedback :)
@joshgillies is a good person to ask!
@kelly_king3 @ethan_lee8 for sure, if there's something specific you'd like to understand about the platform and how it can apply to automated testing/quality scoring of your agents, we're available to take a call and discuss anytime!
Otherwise we hope to capture enough of what's able to be done with the platform in our documentation, so feel free to point a coding agent at it and our public repos on github to get a taste for what's possible.
Thanks again for the support!
Using this currently to secure our agents, absolutely shocking to me that people give the keys to the kingdom to their AI without considering how this all plays out. Love this product.
@joshua_farr Great to have you on board!
@joshua_farr Look how farr we have come since we met you 2 years ago. Such a farr sighted view on what we can achieve together.
@joshua_farr Thanks Joshua! Always keen for feedback. Feel free to reach out directly to any of us if you need any help :)
The line about real-time control needing real-time legibility stuck with me. If someone has to scroll to find the problem, the alert may as well not exist. Refreshing to see design treated as part of the safety story.
@rohanmanthani thanks mate. You absolutely need real time control because it's not even finding the alerts that's the problem, it's that they are retrospective and once they've happened, it's likely your reviewing your updates minutes, hours or days later when the damage has been done.