AgentX - Evaluate AI agent, pinpoint issues, and fix with one click.

@james_carter35 You're right, evals are almost always the last thing teams think about, until something breaks in production.

Setup is faster than most people expect. There's a library of ready-made eval suites so you're not starting from a blank page, and once you've built one you can reuse it across agents and versions. Most teams have their first suite running in minutes.

The part that surprised early users most: after the eval runs, the built-in AI doesn't just show you the score - it analyzes the results and suggests concrete fixes, both to the eval itself and to your agent's instructions. So it's not just "here's what's broken," it's closer to having a senior reviewer tell you exactly what to change.

@james_carter35 
Great question!

For a first eval suite, we try to keep it lightweight: start with a few critical scenarios, expected outcomes, required fields, and tool behaviors the agent must get right.

The goal is not to evaluate everything on day one - it’s to cover the workflows where failure would matter most, then expand the suite as real edge cases appear.

Most teams should be able to start small and get a useful signal quickly

@dominic_cruz Exactly this. The prompt is usually fine in isolation - it's the lack of a feedback loop before deployment that causes failures. Teams ship, something breaks, they patch the prompt, ship again, hope for the best. That's not a workflow, that's trial and error in production with real users as the test suite.

The eval layer is what turns that loop into something systematic. Run it before you ship, catch the drift early, fix with confidence. That's the shift we're going for.

Appreciate you framing it so well, that's exactly the problem we wake up thinking about.

@dominic_cruz 
Yes, that’s the gap!

A lot of agent failures are not because the prompt is terrible - they happen because teams change prompts, models, tools, or workflows without a reliable way to test behavior before shipping.

“Ship with confidence” means evals become part of the release process, not an afterthought

@elena_fischer1 

Exactly, and that mental model gap is underrated. Flat swarms feel powerful in demos but fall apart the moment someone has to debug them, because there's no clear ownership.

The hierarchy solves that by design. Every task has a clear delegation path, so when something goes wrong in eval you can trace it back to a specific agent and a specific instruction rather than shrugging at the final output.

Teams also onboard faster because the structure maps to how they already think. You don't have to teach a new mental model - you just ask "who in your team would handle this?" and that becomes an agent.

@elena_fischer1 
That’s exactly the idea.

Flat swarms can become hard to debug and reason about very quickly. A hierarchy gives teams a clearer mental model: who owns the task, who is responsible for each step, where handoffs happen, and where things failed.

We think agent teams need structure, not just more agents!

@emily_carter18 Thank you! Designed to be as lightweight as possible.

If you're already running an agent in Python it's a few lines of code - wrap your existing agent function, point it at a dataset, and the eval runs against whatever you've built. No changes to your agent code itself. If your agent is already deployed somewhere, the HTTP adapter works directly against the endpoint.

For teams not on Python there's the AgentX builder - create and run evals from the dashboard without touching code at all. What's your current stack?

@emily_carter18 
Thank you!

The goal is to make it easy to plug into an existing workflow, not force teams to rebuild their stack.

You define the scenarios you care about, connect the agent/tool execution path, and run evals before changes go live - prompt updates, model swaps, tool changes, or workflow edits.

So the CI/CD idea is: test the agent like software before shipping it.

@joshua_martinez7 Honestly this one took us by surprise too when we first started running real evals internally.

You set your criteria, run it, and instead of digging through traces for hours you just... get told what's wrong and what to change. Specific instruction, specific reason, suggested rewrite. The first time it pinpointed a tool call our agent was silently skipping - output looked fine, nobody would have caught it manually. That was the moment the team looked at each other and went "okay this is the thing."

The tool usage check is what gets people. Most teams assume if the output looks right the agent did the right thing. It often didn't.

Bigger sample = sharper analysis. Three cases gives you directional signal, a few hundred gives you something you'd actually bet a production deploy on.

What are you evaluating — single agent or a chain?

@joshua_martinez7 
We treat suggested fixes as guidance, not something to apply blindly.

The key is finding the failure point: wrong tool, bad parameters, missing context, weak instructions, or a downstream issue.

Then you can replay the scenario and verify whether the fix actually improved the result.

@elsa_williams Exactly this - and you've described the wall better than most.

Single-purpose bots are a great start until they need to talk to each other, and then the duct tape appears fast. The hierarchy is what replaces the duct tape, and once you add eval on top you actually know the whole system is working, not just the parts you tested in isolation.

Appreciate you getting it so clearly, this is exactly the pain we're going after.

@elsa_williams 
Yes, that’s exactly the pain we’re focused on.

Single-purpose bots can work well in isolation, but once the workflow needs coordination, handoffs, shared context, and different responsibilities, things get messy fast.

We believe agent teams need structure: clear roles, manager agents, specialized agents, tool access, and visibility into how the work moves across the system.

That’s where coordination starts becoming a real productivity layer, not just a collection of disconnected bots.

@joshua_cooper2 Every stage hits a different version of the same problem.

• Solo builders and indie makers benefit earliest - the moment you've shipped your first agent and you're not sure if it's actually doing what you think, that's when eval changes everything. You stop guessing and start knowing.
  

• Teams building agents for their clients hit it even harder, because now reliability isn't just your problem - it's in someone else's production environment with your name on it. Running evals before you hand over a workflow is the difference between "we deliver working agents" and "we deliver and hope."
  

• And enterprise? That's where eval becomes a genuine competitive weapon. Complex multi-agent workflows at scale, real business outcomes riding on them — the teams that run rigorous evaluation programs end up with agents that work on a level nobody else's do. Big datasets, multi-LLM judge panels, per-agent tracing across the whole chain. That's not a small advantage. That's the kind of thing that makes your AI operation the one people talk about.

What's your current situation: building for yourself or for clients?

@joshua_cooper2 
Great question!

I think teams benefit most once they move beyond prototype/demo stage and start putting agents into real workflows.

That usually means:

• agencies building agents for clients

• startups shipping agentic products

• internal AI teams moving pilots into production

• companies with agents using tools, APIs, or multi-step workflows

The more autonomy an agent has, the more important evaluation and visibility become.

@emilia_novak Haha this one stings a bit - but it's fair.

Most "multi-agent" systems are exactly what you described. A few LLM calls in a trenchcoat. The manager routes to the same prompt with slightly different context and someone puts a diagram on the landing page.

The difference is what happens when it breaks. Real architecture gives you a clear delegation path, scoped roles, structured handoffs — so when something goes wrong you can actually find it. "Multiple LLM calls dressed up as agents" just gives you a mystery.

The eval layer is partly how we keep ourselves honest on this. If the architecture is real, the tracing shows it. Per-agent steps, tool calls, handoffs - everything observable, not just the final output.

Appreciate you calling it out. The hype is loud right now and the bar for what counts as "multi-agent" is embarrassingly low.

@emilia_novak 
Thank you, really appreciate that.

That’s been one of our core beliefs from the beginning: multi-agent systems only become useful when there is a clear structure, responsibility split, and coordination layer.

Otherwise, it quickly becomes just multiple LLM calls without accountability.

We think the real value is in designing agent teams more like systems: manager agents, specialized agents, clear handoffs, tool access, evaluation, and visibility into how decisions are made across the workflow.

@daniel_harris11 Visibility is core to how we built this - and it goes deeper than most people expect. Visibility is core to how we built this, and it goes deeper than most people expect. Every run captures the full execution path: each agent step, tool call, handoff, routing decision, and the thinking process between agents.
You're not reconstructing what happened from the final output, you're reading the actual sequence of decisions that got you there, step by step. When something breaks the eval doesn't just flag it - it pinpoints exactly where in the chain it went wrong, why it went wrong, and what to change. Then it automatically surfaces suggested fixes directly to the agent's instructions and orchestration setup. Not "something drifted here," but here's the specific edit, ready to apply. For teams debugging multi-agent failures that used to take hours of trace-reading, that's a pretty different world.

@daniel_harris11 
Great question.

Yes, replay is a key part of closing the loop. Visibility alone is useful, but teams also need to reproduce what happened, inspect the path, adjust the prompt / model / tools, and rerun the same scenario to see if the issue is fixed.

That’s where evals and observability connect: failed executions from production should become test cases, so the same mistake does not keep repeating.

@luz_bidelspach Based on what we see in practice, it clusters around a few things.

Average score and consistency across runs tend to be the starting point - not just "did it score well once" but does it score well reliably. Score variance is what tells you if you have an agent or a coin flip.

Instruction adherence is the one that surprises people. Teams assume their agent follows the instructions they wrote. It often doesn't, not fully, and seeing that measured explicitly is usually a wake-up call.

Tool usage is the silent killer metric. An agent can return a perfectly plausible answer while having skipped the tool call it was supposed to make. Nobody catches that without explicit tracking.

And then at the analysis level - judge agreement. If multiple LLM judges disagree significantly on a response, that's a signal the behavior is borderline and needs a human call, not just an automated pass.

The combination of those tells you whether your agent is actually reliable or just looks reliable in demos. What's the metric you're most nervous about right now?

@luz_bidelspach Great question.

The metrics we see teams care about most are usually:

• task completion rate

• tool / function call accuracy

• consistency across runs

• hallucination or missing-info rate

• performance across different LLMs

• regression rate after prompt, model, or workflow changes

For production agents, the key metric is often not just “was the answer correct?” but “did the agent reliably complete the business task end-to-end?”