Prefactor
Catch your agents' mistakes before they reach customers
1.4K followers
Catch your agents' mistakes before they reach customers
1.4K followers
We evaluate every step your agents take, live. Errors get caught as they happen.
Not in a Slack thread at 1am on a Tuesday
How does Prefactor decide what parts of the code need attention without creating unnecessary changes? Congrats @ethan_lee8 & team!
Prefactor
@ethan_lee8 @hamza_afzal_butt We dont claim to know every agent and how they work. We give you the tools to track things as they go wrong, whether thats through meta data, llm as judge evals etc.
We help the unnecessary change bit by version controlling each agent, enabling different environments so you can track the problems as they occur in dev and when you're ready push them to prod.
Prefactor
@hamza_afzal_butt Hey Hamza thanks so much for the support!
Slight clarification, we don't touch your code. Prefactor sits at runtime and records what the agent actually does on every run, then optional guardrails can hold a high-risk action before it executes.
So it's less "which code needs changing" and more "which agent stopped doing its job, and stop it now."
Are you running any agents in production at the moment btw?
scoring the run live instead of charting it three days later is the part that actually matters. catching a bad agent after it already shipped the damage is just reporting.
Prefactor
@alex_watson2110 Totally agree -- evaluation needs to be part of the control loop for an agent to really be trustworthy.
Prefactor
@alex_watson2110 100%, are you building many agents yourself?
Creatium
This is such a valuable tool. As someone who ships agents to clients, an automated testing/scoring tool would really scale my QA processes.
Prefactor
Prefactor
@kelly_king3 Thanks Kelly! Try it out and let us know how you go!
Feel free to reach out to any of us if you have any questions, always keen for feedback :)
@joshgillies is a good person to ask!
Prefactor
@kelly_king3 @ethan_lee8 for sure, if there's something specific you'd like to understand about the platform and how it can apply to automated testing/quality scoring of your agents, we're available to take a call and discuss anytime!
Otherwise we hope to capture enough of what's able to be done with the platform in our documentation, so feel free to point a coding agent at it and our public repos on github to get a taste for what's possible.
Thanks again for the support!
Using this currently to secure our agents, absolutely shocking to me that people give the keys to the kingdom to their AI without considering how this all plays out. Love this product.
Prefactor
@joshua_farr Great to have you on board!
Prefactor
@joshua_farr Look how farr we have come since we met you 2 years ago. Such a farr sighted view on what we can achieve together.
Prefactor
@joshua_farr Thanks Joshua! Always keen for feedback. Feel free to reach out directly to any of us if you need any help :)
The line about real-time control needing real-time legibility stuck with me. If someone has to scroll to find the problem, the alert may as well not exist. Refreshing to see design treated as part of the safety story.
Prefactor
@rohanmanthani thanks mate. You absolutely need real time control because it's not even finding the alerts that's the problem, it's that they are retrospective and once they've happened, it's likely your reviewing your updates minutes, hours or days later when the damage has been done.
TimeToCoda
Working in high-trust environments, I don’t think the future is humans approving every AI action. It’s agents operating independently 95% of the time, with enough visibility and confidence that when they drift outside the guardrails, a human can intervene before it matters. Looks like you’re tackling that problem head on. Congrats @matt_doughty @simon_russell1 and team!
Send me my 1M spans to give it a solid rev up! ;)
Prefactor
@simon_russell1 @emotf Done. You've got to set up an agent first and then you get them in your account.
You were one of our first conversations wayyyy back. LFG!
TimeToCoda
@simon_russell1 @matt_doughty Big fan of your work. Following your journey all the way... I love seeing how this idea has progressed, and now PH launch 🤘
Prefactor
@matt_doughty @emotf Thanks! Those 1M are yours if you sign up now :D
I really like the idea of tracking how the agent is working, as it is performing actions. Being more proactive is better than finding out something went wrong several days ago. It's not to be overlooked that the interruption isn't just killing the agent run, but handing it to a human too. I am going to have to play around with this and my own agents.
Prefactor
@philnash Thanks for the comment Phil. Would be happy to share some of the examples Ive built too, I'm building an eval triaging tool. 3 phases.
First phase: Tokenless Evals/ technical reviews
Second phase: pattern management against prior activity
Third phase: LLM as judge
All live. Let me know where you get to.
Prefactor
@philnash Thanks Phil, exactly! Killing a bad run is only half of it, the handoff to a human is where the actual value sits, otherwise you have just built a more expensive way to fail.
Would love to hear what breaks when you try it on your own agents, that is the feedback we learn the most from.
What are you running them on at the moment?