Prefactor
Catch your agents' mistakes before they reach customers
1.4K followers
Catch your agents' mistakes before they reach customers
1.4K followers
We evaluate every step your agents take, live. Errors get caught as they happen.
Not in a Slack thread at 1am on a Tuesday
Can I use Prefactor for my fleet of Hermes and OpenClaw agents ?
@nathan_ngz we officially support integrating with OpenClaw, and I have it on good authority that Hermes is very easy to instrument also. When in doubt, just point your agent at our docs, and SDKs on Github and let it cook!
@nathan_ngz You can indeed -- there's an OpenClaw plugin. There's no Hermes plugin at this point but we have seen someone get Hermes to integrate itself with Prefactor to great effect!
@nathan_ngz OpenClaw yes, Hermes not yet
Congrats on launching! The "did it actually do it, or just say it did" framing will ring true for anyone who has put an agent in front of real users. The 25k spans a month free tier makes it easy for a small team to give this a proper try. How do the custom spans that pull context from Linear or GitHub work in practice, do you write those checks yourself or pick from templates?
How are you handling latency? I imagine Prefactor is scanning through all of my agent traces and flagging things. Is it flagging proactively, or retrospectively? If it's proactively, I'd imagine latency and speed is a problem. I wouldn't want having to wait for Prefactor to finish scanning before my agent can continue its run.
@matteo8p hey mate. Thanks for the note.
Neither latency or speed is a problem and we don't sit on the agent run slowing it down.
We sit inside the agent via the SDK, and as a result we watch everything as it happens rather than sitting inside the agent workflow.
The tracking/ scoring happens instantaneously.
Congrats Matt and team! may I ask for the original story of why do you build this?
@nathan_tran2 Thanks Nathan! We were building in the MCP security space and we realised that the speed of adoption of MCP and thus the need for security was going to take too long to build an ecosystem around. Concurrently, customers were telling us their biggest concern was how to take agents from POC to Prod and ensure reliability. Which is where Prefactor came from!
@nathan_tran2 Thanks for the support Nathan! I'll let @simon_russell1 or @matt_doughty answer this as they're the founders :)
The product looks great! What agents I can try it on? Does new Kimi K3 supported?
@roman_matusevich Thanks Roman. We work at the trace level rather than being tied to a specific model, so it is less about which model and more about whether we can see the runs.
Let me confirm the Kimi K3 but I'm pretty sure it should be fine.
@joshgillies
@roman_matusevich @ethan_lee8 Kimi K3 should be fine, we've used the K2.5 and K2.6 varieties quite a bit. Let us know if you have any issues though!
Congrats on the launch! How do you evaluate the AI Agents is it deterministic or using LLMs by providing the truth table?
@arokamal_sethy Thanks mate, and thanks for the question. We enable both, at the moment our first class evaluation is deterministic using a risk profiling strategy plus meta data evaluation plus span patterns/ looping etc.
Prefactor enables you to run any eval you want against the runs in real time though, which allows you to use your existing evals and store them against the run direclly inside Prefactor.
Congrats on the launch! It would be really interesting to see how well it integrates with the other softwares, especially the agent orchestrators as there are multiple agents running in parallel and such insights can help improve the orchestration.
@sachinsharma Thanks Sachin. A huge part of our vision is to ensure that you can combine multiple systems/ frameworks into Prefactor so you have a true overview of what an agent is doing at any one time.
@sachinsharma Yep we aim to be framework agnostic, already have quite a number of integrations actually!
What sort of agents are you working on?