Launching today
Prefactor
Evaluate your AI Agents in real-time
407 followers
Evaluate your AI Agents in real-time
407 followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
Ethan asked how people keep tabs on live agents, so here is my honest answer. I run scheduled agent jobs daily for SEO reporting and roughly one in three submits used to report success while the page never actually changed, so every job now ends by rereading the rendered result and comparing it against what the agent claimed. Mukil's question of did it actually do it or did it just say it did is the exact failure I kept measuring. Can an eval in Prefactor check an external side effect like that, the page actually updated or the row actually written, instead of scoring the run transcript?
Prefactor
@abdullah_javaid3 Thanks for your comment. Short answer: yes, and it's basically the loop you already built.
The re-read-and-compare step becomes the eval signal. Your job verifies the real outcome (re-reads the rendered page, or queries the row) and emits that as a span, and the eval scores against that ground truth instead of the agent's self-reported "success." So Prefactor never has to trust "I did it", it scores what actually landed.
That's exactly Mukil's line: the transcript is the claim, the emitted outcome is the verification, and the eval only counts the second one. Your 1-in-3 false-success rate is precisely what surfaces once you're scoring outcomes instead of runs.
Prefactor
@abdullah_javaid3 Interesting question. Alongside the LLM activity you can store whatever you like; also you can store quality assessments against an individual run. Usually the goal is to understand what the success rate is, then use that information to improve the prompting and harness. For your situation a hybrid of prompting and deterministic assessment makes sense.
Live mode stopping a run mid-breach is the feature I'd actually want. What happens to the agent's state/context at the moment it's stopped — does the next run pick up where it left off, or does it start clean? If a breach happens 20 steps into a long task, being able to see (or roll back to) exactly what the agent had built up right before the stop seems almost as valuable as the stop itself.
Prefactor
@rashad_builds Hey mate, thanks for the comment. We stop via the span itself, and you can insert a HITL/ killswitch or another eval. So in that 20 step example, you see the whole flow which is stored as the instance (conversation/ run/ workflow) and at any point in that workflow, you can jump in.
Prefactor
@rashad_builds There are a few approaches -- but at the core of it all the actions are in Prefactor. So it would be entirely possible to reconstruct the context from that. (That's assuming you want it to continue -- usually it's more of a post-mortem thing)
Cursor
Congrats Ethan and team!
Prefactor
@benln appreciate the support 🙏
Prefactor
@benln Thanks Ben!
Prefactor
Prefactor
Hi Product Hunt - Matt here, co-founder of Prefactor with Simon.
We've been heads-down on this one for a while, so finally getting to show you is a real thrill.
Let me start with the question this whole thing is built around: do you actually know what your agents are doing in production right now?
For most teams we talk to, the honest answer is "…not really." Your evals pass, everything's green, you ship - and then every real run vanishes into a black box. Quality quietly drifts. Risk creeps in. Costs climb. And eventually someone asks which of your agents are still doing their job, and the room goes quiet.
That silence is the entire reason Prefactor exists. Gartner reckons 40%+ of agentic AI projects get scrapped by 2027 - and honestly, this gap is a big part of why.
So we built the thing we kept wishing we had: Prefactor scores every run in production the moment it happens - quality, drift, risk - then wires those scores straight into action. A failing agent gets caught live, not charted three days later.
How it actually works
prefactor init - one command connects your workspace and discovers your agents across your runtimes. First traced run in under 5 minutes.
Drop in the SDK (TypeScript or Python) - native for LangChain, Claude, Vercel AI, OpenClaw and LiveKit. Every call becomes a span, streaming in live with cost and data risk attached.
Run the evals you define on every run - LLM-as-judge, technical checks, qualitative metrics. Custom spans pull context from GitHub, Linear, Jira or your database, so every eval is grounded in what actually happened, not a guess.
Act. Hold, approve or block the second a run crosses a line - automatically at runtime, or routed to a human. Every decision logged and enforced through the SDK or API.
The payoff: you get to ship agents like real software — versioned, staged, promoted through dev → staging → prod only when evals pass, with instant rollback when they don't.
Why it's different
Most tools observe and score, then hand you the problem. Prefactor closes the loop - observe, evaluate, act, all inside the same run. A risky agent gets caught, not just charted.
My favourite bit of feedback so far: a customer with 40 agents in production and, in their words, "no honest way to say which ones were still doing their job." We gave them that answer - and the brake pedal for when one wasn't.
Who it's for
Engineering teams shipping agents to real customers, on any stack. Agent frameworks work natively; everything else plugs in through OpenTelemetry or the core SDK.
A little something for the PH community
Sign up today and you get 1,000,000 free agent steps. Every span counts, so that's a serious amount of live production evaluation on us. Valid until Friday 11:59pm PT, once you've set up your first agent.
Our ask
If you're running agents in production, tell us how you keep tabs on them today - even if the honest answer is "we're mostly hoping." We'll be in the comments all day and we'd genuinely love to hear what's working, what's breaking, and what you'd want a tool like this to do next.
Get started free at prefactor.tech. First 25,000 spans a month free, no card needed.
@simon_russell1 @joshgillies @ethan_lee8 @rheu @joeys and I will be here all day.
Big thanks to @rohanrecommends @rohanchaubey4 hunting us.
@simon_russell1 @joshgillies @ethan_lee8 @rheu @joeys @rohanrecommends @rohanchaubey4 @matt_doughty Love this. I wouldn't leave my toddler home alone so why would I deploy an agent and not checkup on it. You get it. Cut through the agentic AI hype and solve the issues so we can actually get ROI on AI.
Prefactor
@simon_russell1 @joshgillies @ethan_lee8 @rheu @joeys @rohanrecommends @rohanchaubey4 @stefan_knight Thanks Stefan. This is awesome to hear.
How do you answer the question of what your agents are actually doing?
Genuinely curious what "scoring every run" looks like at scale, is that sampled or are you actually evaluating 100% of production traffic? That distinction matters a lot for cost and for trust in the numbers.
Prefactor
@yolanda_c_schneider All traffic. The 100% element is sitting alongside the agents, seeing everything they do and being able to articulate what they are actually doing. Eg, what tools are they calling. Are the actions they are making write actions vs read actions. How is the pattern of the agent different from what you know to be the case. If an agent takes 3 turns to resolve a problem, why is it taking 8 (insert HITL).
The beauty of our approach is that the cost is 0 until you introduce an LLM as judge, but by using the no token approach it means you can actually target the problematic runs/ spans/ actions rather than running random samples which could catch nothing when there is something.
Prefactor
@yolanda_c_schneider right now, we score every run. This may change in the future, and if you have a preference we'd love to hear your thoughts on the matter! From a cost perspective, the data that goes into the scoring is entirely up to you. We score what you send us about your production agents, and generally the rule of thumb is something like, trace the parts of your agent required to answer the question of "did it do good job".
Prefactor
@yolanda_c_schneider There can be different approaches depending on the application. Obviously running full LLM-based evals on a high volume chatbot might not make sense. But that's when being able to filter down the conversations you do choose to run deep analysis over makes sense. You can use heuristics to do that, then use that to guide things.
'no honest way to say which agents still do their job' nails it. when the llm-judge itself drifts, what keeps the eval honest?
Prefactor
@andrewzakonov exactly right, who judges the judge, ad infinitum. How are you managing this at scale currently?
Prefactor
@andrewzakonov Our core approach is tokenless. We/ you use the agent run to spot problems, risks, abnormal behaviour. When an LLM is needed, then you bring it in on demand rather than sampling. The idea is that when an LLM is needed, its being brought in for a specific task to confirm what the tokenless approach has already found.
You could apply the same logic to bringing in a HITL when a customer support conversation is going off the rails which is discovered through sentiment analysis (tokenless)
Prefactor
@andrewzakonov That's actually a great point. Making a good judge LLM agent is actually a big challenge too - it needs just as much engineering as the thing it's judging. Luckily you can instrument the LLM-as-judge agent with Prefactor as well. (I guess you could instrument that recursively but you might run into diminishing returns at some point!)
How do you define "quality" here? That word does a lot of work and I imagine it means something different for a support agent than a coding agent.
Prefactor
@sheikh_umair1 Absolutely right. SO we see risk and quality as two sides of the same coin. We break them out because if an agent is making a write action on a tool call, then its both a quality problem if it goes wrong and a risk problem if it doesn't land properly and affects teams reliant on the agents.
Quality to us is technical metrics, agent patterns - literally is it calling too many tools or looping too frequently but the real end game, and why we exist, is to answer the question, how do you know what your agent is doing? And more importantly, how can you tell if it doesn't do what you expect it to.
Prefactor
@sheikh_umair1 That's very true, and it's why we've designed the system to be flexible -- so you can decide how you want to approach it. The diversity in evaluation frameworks just for coding agents is a sign of how complex (and fast evolving) it is.