Argmin AI - Test AI agent workflows without ML expertise
by•
For teams and founders shipping agentic AI workflows and features. Generic metrics don't speak your product's language, so a new prompt, model, or RAG change can pass them and still ship real issues. A quick manual check catches even less. You don't need annotated data, an ML team, or a month to be safe from regression. Drop in your agent's task, business rules, docs, and examples, and Argmin AI builds an evaluation you run before every change.
Replies
Quorini
@dmitrii_konyrev Really well explained. Quick question: how heavy is the initial setup for a new agentic workflow?
Do we need to provide any initial examples or can Argmin bootstrap the evaluation metrics mostly from our description of success?
@alex_tartach Thanks Alex. Setup is light: Argmin bootstraps the eval and the metrics mostly from your description of success plus your docs, so you don't need to bring examples to get started. A handful helps with precision though, dropping in 10-20 (especially the failure cases you actually worry about) sharpens the judge and shortens calibration. One honest caveat for agentic specifically: the final-output eval bootstraps cleanly from the description, while step and trajectory-level checks are more hands-on right now. When you say a new agentic workflow, is "success" the final answer, or the whole path it took to get there?
would love to see a github action or CI hook that runs the eval automatically on every PR and comments the diff in the checks section, that would make it actually part of the loop without anyone having to remember to trigger it
@berke9f56 This is exactly where we want it to live, Berke. An eval that only runs when someone remembers isn't really a gate, so a PR-triggered run that posts the diff into the checks section is the right shape. Honest status: it's on the roadmap, not shipped yet. Design question while you're here, do you want it to just comment the diff, or block the merge on a regression past a threshold? Those are two very different levels of trust in the eval, and I'm curious where teams want to start.
Would love a way to compare eval runs side by side so we can see exactly where a new prompt version regressed before merging. A diff view across runs would save a lot of guesswork.
@yaz2029848 This is on the money, Yağız. Catching the regression before you merge is exactly where evals earn their keep, and the part that makes it actionable is a per-example diff across versions, not just an aggregate score nudging up or down. It's high on our list. One question: when a run regresses, do you care more about the aggregate delta, or about seeing the specific examples that flipped? That changes how we'd build the diff.
the fact that it generates evals from just a task description and a few examples is genuinely impressive. most eval tools assume you already have a labeled dataset sitting around, so skipping that whole step is a real unlock for smaller teams shipping fast.
@esma144191 Thanks Esma. The labeled-dataset assumption always felt backwards to me: the feature you most need to eval is the new one, which is exactly the one you have no labels for yet. Starting from the task description plus a couple of examples gets you a first eval on day one, then you tighten it as real cases come in. Anything you're shipping right now where that would've saved you?
@levington27 Straight answer: it doesn't keep pace on its own yet. As scope grows (more tools, longer chains) you regenerate from the updated task description and drop in a few of the new traces, and the judge has to be re-checked against your experts for the new behavior, because one calibrated on the old scope will quietly grade the new steps wrong. That refresh is a deliberate step today, not automatic. Making the eval track scope, especially at the trajectory level and not just the final output, is exactly the hard part we're building toward. Is your scope creep more new tools, or longer chains?
what caught my attention is the focus on evaluation data rather than the evaluator itself. in many ML projects, preparing a reliable evaluation set is where a considerable amount of time disappears. if this can simplify that step, it would be genuinely useful.
@anastasia_gromova1 You're pointing right at the time sink, Anastasia. Building a reliable eval set by hand is where whole sprints quietly disappear. One thing we ran into: the set and the evaluator are coupled. A clean eval set only matters if whatever grades it actually agrees with your experts, so we bootstrap both from your task and docs, then calibrate the judge against a handful of your own labels.
Quorini
Seems to be great tool for AI agentic at scale. Looking forward to give it a try! Is there relevant documentation or guide on how to start using it?
@alex_tartach Appreciate it Alex. Getting started is quick, you go from your task description to a first eval in a few minutes. Happy to send you the getting-started guide and help you get your first agentic eval running. Want me to DM it over with a couple of tips for the agentic setup?
Finally an eval tool that gets agentic workflows without forcing us to label datasets. Dropped in our task description and a few examples and got a usable eval suite the same day.
@hayrettin988209 This is the one I'll be re-reading all day, thank you Hayrettin. Same-day usable was exactly the bar we set for ourselves. The real test now: run it for a week and see if it catches a regression you'd otherwise have shipped, that's when it earns its place. What kind of agentic workflow did you point it at, and did anything in the generated suite surprise you?
love the direction. most teams don’t have perfectly labeled datasets, and building them from scratch usually turns into a project of its own. starting from production traces instead feels much closer to how things actually work. gonna give it a try with one of our internal workflows and see how it does.
@dmitrii_manuev Appreciate it Dmitrii. If you're starting from traces, pick 15-20 that include a couple of the failures you actually worry about, not just the clean happy-path ones, that's what makes the generated eval catch something real on the first run.
love how you cut straight to the eval problem instead of padding with generic AI agent dashboards. the "drop in your docs and get evals" angle feels like exactly what shipping teams have been cobbling together with brittle scripts for too long.
@nazls86733 You nailed why we built it, Nazlı. The brittle-scripts phase is real: everyone reinvents a fragile grader, it slowly drifts from what their experts would actually say, and then nobody trusts the number. We'd rather own that one hard part, a judge that agrees with your domain experts, than ship another dashboard. What were you stitching together before?