Launching today
Prefactor
Evaluate your AI Agents in real-time
266 followers
Evaluate your AI Agents in real-time
266 followers
Most agents pass their evals and fail in production. Prefactor is the evaluation layer that closes the gap. We score every agent run in real time, surface quality regressions and drift as they happen, and show engineering teams exactly how their agents are performing at scale. Built for the teams shipping agents to customers.
Ethan asked how people keep tabs on live agents, so here is my honest answer. I run scheduled agent jobs daily for SEO reporting and roughly one in three submits used to report success while the page never actually changed, so every job now ends by rereading the rendered result and comparing it against what the agent claimed. Mukil's question of did it actually do it or did it just say it did is the exact failure I kept measuring. Can an eval in Prefactor check an external side effect like that, the page actually updated or the row actually written, instead of scoring the run transcript?
Prefactor
@abdullah_javaid3 Thanks for your comment. Short answer: yes, and it's basically the loop you already built.
The re-read-and-compare step becomes the eval signal. Your job verifies the real outcome (re-reads the rendered page, or queries the row) and emits that as a span, and the eval scores against that ground truth instead of the agent's self-reported "success." So Prefactor never has to trust "I did it", it scores what actually landed.
That's exactly Mukil's line: the transcript is the claim, the emitted outcome is the verification, and the eval only counts the second one. Your 1-in-3 false-success rate is precisely what surfaces once you're scoring outcomes instead of runs.
Live mode stopping a run mid-breach is the feature I'd actually want. What happens to the agent's state/context at the moment it's stopped — does the next run pick up where it left off, or does it start clean? If a breach happens 20 steps into a long task, being able to see (or roll back to) exactly what the agent had built up right before the stop seems almost as valuable as the stop itself.
Prefactor
@rashad_builds Hey mate, thanks for the comment. We stop via the span itself, and you can insert a HITL/ killswitch or another eval. So in that 20 step example, you see the whole flow which is stored as the instance (conversation/ run/ workflow) and at any point in that workflow, you can jump in.
Cursor
Congrats Ethan and team!
Prefactor
@benln appreciate the support 🙏
Prefactor
@benln Thanks Ben!
Prefactor
Prefactor
Hi Product Hunt - Matt here, co-founder of Prefactor with Simon.
We've been heads-down on this one for a while, so finally getting to show you is a real thrill.
Let me start with the question this whole thing is built around: do you actually know what your agents are doing in production right now?
For most teams we talk to, the honest answer is "…not really." Your evals pass, everything's green, you ship - and then every real run vanishes into a black box. Quality quietly drifts. Risk creeps in. Costs climb. And eventually someone asks which of your agents are still doing their job, and the room goes quiet.
That silence is the entire reason Prefactor exists. Gartner reckons 40%+ of agentic AI projects get scrapped by 2027 - and honestly, this gap is a big part of why.
So we built the thing we kept wishing we had: Prefactor scores every run in production the moment it happens - quality, drift, risk - then wires those scores straight into action. A failing agent gets caught live, not charted three days later.
How it actually works
prefactor init - one command connects your workspace and discovers your agents across your runtimes. First traced run in under 5 minutes.
Drop in the SDK (TypeScript or Python) - native for LangChain, Claude, Vercel AI, OpenClaw and LiveKit. Every call becomes a span, streaming in live with cost and data risk attached.
Run the evals you define on every run - LLM-as-judge, technical checks, qualitative metrics. Custom spans pull context from GitHub, Linear, Jira or your database, so every eval is grounded in what actually happened, not a guess.
Act. Hold, approve or block the second a run crosses a line - automatically at runtime, or routed to a human. Every decision logged and enforced through the SDK or API.
The payoff: you get to ship agents like real software — versioned, staged, promoted through dev → staging → prod only when evals pass, with instant rollback when they don't.
Why it's different
Most tools observe and score, then hand you the problem. Prefactor closes the loop - observe, evaluate, act, all inside the same run. A risky agent gets caught, not just charted.
My favourite bit of feedback so far: a customer with 40 agents in production and, in their words, "no honest way to say which ones were still doing their job." We gave them that answer - and the brake pedal for when one wasn't.
Who it's for
Engineering teams shipping agents to real customers, on any stack. Agent frameworks work natively; everything else plugs in through OpenTelemetry or the core SDK.
A little something for the PH community
Sign up today and you get 1,000,000 free agent steps. Every span counts, so that's a serious amount of live production evaluation on us. Valid until Friday 11:59pm PT, once you've set up your first agent.
Our ask
If you're running agents in production, tell us how you keep tabs on them today - even if the honest answer is "we're mostly hoping." We'll be in the comments all day and we'd genuinely love to hear what's working, what's breaking, and what you'd want a tool like this to do next.
Get started free at prefactor.tech. First 25,000 spans a month free, no card needed.
@simon_russell1 @joshgillies @ethan_lee8 @rheu @joeys and I will be here all day.
Big thanks to @rohanrecommends @rohanchaubey4 hunting us.
@simon_russell1 @joshgillies @ethan_lee8 @rheu @joeys @rohanrecommends @rohanchaubey4 @matt_doughty Love this. I wouldn't leave my toddler home alone so why would I deploy an agent and not checkup on it. You get it. Cut through the agentic AI hype and solve the issues so we can actually get ROI on AI.
Prefactor
@simon_russell1 @joshgillies @ethan_lee8 @rheu @joeys @rohanrecommends @rohanchaubey4 @stefan_knight Thanks Stefan. This is awesome to hear.
How do you answer the question of what your agents are actually doing?
What kinds of failure modes or edge cases have you seen most often when an agent passes evaluation but fails in production, and how would real‑time scoring and drift alerts need to look for your team to trust them enough to act automatically?
Prefactor
@swati_paliwal Thanks for your comment Swati.
We are not just focused on edge cases. The first thing is are the agents being actually assessed against whether they are doing their job.
That contract you've built for the agent is key. Eg, what tools are they allowed to call (or not in the case of OpenAIs agent), is it looping? Are parts of the flow failing causing excessive token usage.
Evals are point in time reference points for the agents themselves. When models change, when databases get different answers, drift happens. When sampling involved 1% of all agent runs or less (and given the cost of tokens, it is increasingly less), its not the 1% that will kill you. Its the 1% of the 99% you don't eval.
Jinna.ai
Congrats on the launch! Does your tool make agents more smart over time by building right context around them, or is it still a responsibility of an agent?
Prefactor
@nikitaeverywhere Hey Nik. Great question. The self-improving loop is something within its capabilities but not something we currently have as native.
I ran a POC for a customer where they wanted to have their rag database self improve. I did that by setting up a HITL when a conversation went badly via sentiment analysis running over every span. The rag database would then update with the answers drawn from the agent. So the rag db was self improving.
Everything is configurable so you can literally go in and do all sorts. I am passionate about working out how to use non token evals to allow 100% coverage and only use the token coverage for the triaged cases.
Would love to talk more when you've given it a go.
Can I use Prefactor for my fleet of Hermes and OpenClaw agents ?
Prefactor
@nathan_ngz we officially support integrating with OpenClaw, and I have it on good authority that Hermes is very easy to instrument also. When in doubt, just point your agent at our docs, and SDKs on Github and let it cook!