Prefactor
Evaluate your AI Agents in real-time
1.5K followers
Evaluate your AI Agents in real-time
1.5K followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
Uselink
Congrats Matt and team! may I ask for the original story of why do you build this?
Prefactor
@nathan_tran2 Thanks Nathan! We were building in the MCP security space and we realised that the speed of adoption of MCP and thus the need for security was going to take too long to build an ecosystem around. Concurrently, customers were telling us their biggest concern was how to take agents from POC to Prod and ensure reliability. Which is where Prefactor came from!
Prefactor
@nathan_tran2 Thanks for the support Nathan! I'll let @simon_russell1 or @matt_doughty answer this as they're the founders :)
The product looks great! What agents I can try it on? Does new Kimi K3 supported?
Prefactor
@roman_matusevich Thanks Roman. We work at the trace level rather than being tied to a specific model, so it is less about which model and more about whether we can see the runs.
Let me confirm the Kimi K3 but I'm pretty sure it should be fine.
@joshgillies
Prefactor
@roman_matusevich @ethan_lee8 Kimi K3 should be fine, we've used the K2.5 and K2.6 varieties quite a bit. Let us know if you have any issues though!
Prefactor
Congrats on the launch! How do you evaluate the AI Agents is it deterministic or using LLMs by providing the truth table?
Prefactor
Prefactor
@arokamal_sethy Thanks mate, and thanks for the question. We enable both, at the moment our first class evaluation is deterministic using a risk profiling strategy plus meta data evaluation plus span patterns/ looping etc.
Prefactor enables you to run any eval you want against the runs in real time though, which allows you to use your existing evals and store them against the run direclly inside Prefactor.
MakersClaw
Congrats on the launch! It would be really interesting to see how well it integrates with the other softwares, especially the agent orchestrators as there are multiple agents running in parallel and such insights can help improve the orchestration.
Prefactor
@sachinsharma Thanks Sachin. A huge part of our vision is to ensure that you can combine multiple systems/ frameworks into Prefactor so you have a true overview of what an agent is doing at any one time.
Prefactor
@sachinsharma Yep we aim to be framework agnostic, already have quite a number of integrations actually!
What sort of agents are you working on?
Prefactor
Hey Product Hunt, I'm Mukil — an engineer at Prefactor. 🔥
People always ask me:
"Mukil, you work with AI all day, isn't it crazy what it can do now?"
Sure, it's pretty cool. But I spend most of my time on the mostly ambiguous half of that sentence. Yes, it did something but was it the right thing? Did it actually do it or did it just say it did? Did it just hand someone the account details for the completely wrong account?
When an agent screws up, it often doesn't like to stop. It keeps going, fully confident and you find out hours or days later. And at that point its often when a customer's already upset or a refund went out that absolutely should not have. As an engineer this made me a little annoyed. The problem was never that I couldn’t see what the agent did. Dashboards will happily show me — right after it’s already done something I didn't want it to do. It's already leaked that PII, emailed the wrong guy or somehow spun up 15 subagents to run a database query.
With Prefactor you can see runs happening live, stopping the agent in its tracks. Every run gets scored live and lets the human know to step in if they need to. Instead of waiting 30 minutes to see this failed run on your custom dashboard (it's very pretty, I'm sure), you can step in and kill it while its executing.
In short, almost anyone can build an AI agent these days. And as the barrier to building them gets lower, the standard for running them well needs to get much higher.
We obsess over that second part, so you can spend less time investigating what your agent did at 3:14 a.m. and more time letting it do actual work.
Would love for you to try it and tell us your experience with it!
Jinna.ai
Congrats on the launch! Does your tool make agents more smart over time by building right context around them, or is it still a responsibility of an agent?
Prefactor
@nikitaeverywhere Hey Nik. Great question. The self-improving loop is something within its capabilities but not something we currently have as native.
I ran a POC for a customer where they wanted to have their rag database self improve. I did that by setting up a HITL when a conversation went badly via sentiment analysis running over every span. The rag database would then update with the answers drawn from the agent. So the rag db was self improving.
Everything is configurable so you can literally go in and do all sorts. I am passionate about working out how to use non token evals to allow 100% coverage and only use the token coverage for the triaged cases.
Would love to talk more when you've given it a go.
congrats on the launch! curious as to what metrics they evalaute the other AI agents on
Prefactor