AgentX - Evaluate AI agent, pinpoint issues, and fix with one click.

@elena_fischer1 

Exactly, and that mental model gap is underrated. Flat swarms feel powerful in demos but fall apart the moment someone has to debug them, because there's no clear ownership.

The hierarchy solves that by design. Every task has a clear delegation path, so when something goes wrong in eval you can trace it back to a specific agent and a specific instruction rather than shrugging at the final output.

Teams also onboard faster because the structure maps to how they already think. You don't have to teach a new mental model - you just ask "who in your team would handle this?" and that becomes an agent.

@elena_fischer1 
That’s exactly the idea.

Flat swarms can become hard to debug and reason about very quickly. A hierarchy gives teams a clearer mental model: who owns the task, who is responsible for each step, where handoffs happen, and where things failed.

We think agent teams need structure, not just more agents!

@emily_carter18 Thank you! Designed to be as lightweight as possible.

If you're already running an agent in Python it's a few lines of code - wrap your existing agent function, point it at a dataset, and the eval runs against whatever you've built. No changes to your agent code itself. If your agent is already deployed somewhere, the HTTP adapter works directly against the endpoint.

For teams not on Python there's the AgentX builder - create and run evals from the dashboard without touching code at all. What's your current stack?

@emily_carter18 
Thank you!

The goal is to make it easy to plug into an existing workflow, not force teams to rebuild their stack.

You define the scenarios you care about, connect the agent/tool execution path, and run evals before changes go live - prompt updates, model swaps, tool changes, or workflow edits.

So the CI/CD idea is: test the agent like software before shipping it.

@elsa_williams Exactly this - and you've described the wall better than most.

Single-purpose bots are a great start until they need to talk to each other, and then the duct tape appears fast. The hierarchy is what replaces the duct tape, and once you add eval on top you actually know the whole system is working, not just the parts you tested in isolation.

Appreciate you getting it so clearly, this is exactly the pain we're going after.

@elsa_williams 
Yes, that’s exactly the pain we’re focused on.

Single-purpose bots can work well in isolation, but once the workflow needs coordination, handoffs, shared context, and different responsibilities, things get messy fast.

We believe agent teams need structure: clear roles, manager agents, specialized agents, tool access, and visibility into how the work moves across the system.

That’s where coordination starts becoming a real productivity layer, not just a collection of disconnected bots.

@joshua_cooper2 Every stage hits a different version of the same problem.

• Solo builders and indie makers benefit earliest - the moment you've shipped your first agent and you're not sure if it's actually doing what you think, that's when eval changes everything. You stop guessing and start knowing.
  

• Teams building agents for their clients hit it even harder, because now reliability isn't just your problem - it's in someone else's production environment with your name on it. Running evals before you hand over a workflow is the difference between "we deliver working agents" and "we deliver and hope."
  

• And enterprise? That's where eval becomes a genuine competitive weapon. Complex multi-agent workflows at scale, real business outcomes riding on them — the teams that run rigorous evaluation programs end up with agents that work on a level nobody else's do. Big datasets, multi-LLM judge panels, per-agent tracing across the whole chain. That's not a small advantage. That's the kind of thing that makes your AI operation the one people talk about.

What's your current situation: building for yourself or for clients?

@joshua_cooper2 
Great question!

I think teams benefit most once they move beyond prototype/demo stage and start putting agents into real workflows.

That usually means:

• agencies building agents for clients

• startups shipping agentic products

• internal AI teams moving pilots into production

• companies with agents using tools, APIs, or multi-step workflows

The more autonomy an agent has, the more important evaluation and visibility become.

@emilia_novak Haha this one stings a bit - but it's fair.

Most "multi-agent" systems are exactly what you described. A few LLM calls in a trenchcoat. The manager routes to the same prompt with slightly different context and someone puts a diagram on the landing page.

The difference is what happens when it breaks. Real architecture gives you a clear delegation path, scoped roles, structured handoffs — so when something goes wrong you can actually find it. "Multiple LLM calls dressed up as agents" just gives you a mystery.

The eval layer is partly how we keep ourselves honest on this. If the architecture is real, the tracing shows it. Per-agent steps, tool calls, handoffs - everything observable, not just the final output.

Appreciate you calling it out. The hype is loud right now and the bar for what counts as "multi-agent" is embarrassingly low.

@emilia_novak 
Thank you, really appreciate that.

That’s been one of our core beliefs from the beginning: multi-agent systems only become useful when there is a clear structure, responsibility split, and coordination layer.

Otherwise, it quickly becomes just multiple LLM calls without accountability.

We think the real value is in designing agent teams more like systems: manager agents, specialized agents, clear handoffs, tool access, evaluation, and visibility into how decisions are made across the workflow.

@daniel_harris11 Visibility is core to how we built this - and it goes deeper than most people expect. Visibility is core to how we built this, and it goes deeper than most people expect. Every run captures the full execution path: each agent step, tool call, handoff, routing decision, and the thinking process between agents.
You're not reconstructing what happened from the final output, you're reading the actual sequence of decisions that got you there, step by step. When something breaks the eval doesn't just flag it - it pinpoints exactly where in the chain it went wrong, why it went wrong, and what to change. Then it automatically surfaces suggested fixes directly to the agent's instructions and orchestration setup. Not "something drifted here," but here's the specific edit, ready to apply. For teams debugging multi-agent failures that used to take hours of trace-reading, that's a pretty different world.

@daniel_harris11 
Great question.

Yes, replay is a key part of closing the loop. Visibility alone is useful, but teams also need to reproduce what happened, inspect the path, adjust the prompt / model / tools, and rerun the same scenario to see if the issue is fixed.

That’s where evals and observability connect: failed executions from production should become test cases, so the same mistake does not keep repeating.

@luz_bidelspach Based on what we see in practice, it clusters around a few things.

Average score and consistency across runs tend to be the starting point - not just "did it score well once" but does it score well reliably. Score variance is what tells you if you have an agent or a coin flip.

Instruction adherence is the one that surprises people. Teams assume their agent follows the instructions they wrote. It often doesn't, not fully, and seeing that measured explicitly is usually a wake-up call.

Tool usage is the silent killer metric. An agent can return a perfectly plausible answer while having skipped the tool call it was supposed to make. Nobody catches that without explicit tracking.

And then at the analysis level - judge agreement. If multiple LLM judges disagree significantly on a response, that's a signal the behavior is borderline and needs a human call, not just an automated pass.

The combination of those tells you whether your agent is actually reliable or just looks reliable in demos. What's the metric you're most nervous about right now?

@luz_bidelspach Great question.

The metrics we see teams care about most are usually:

• task completion rate

• tool / function call accuracy

• consistency across runs

• hallucination or missing-info rate

• performance across different LLMs

• regression rate after prompt, model, or workflow changes

For production agents, the key metric is often not just “was the answer correct?” but “did the agent reliably complete the business task end-to-end?”

@easton_carter The root cause analysis with suggested fixes - consistently, and it's not close. People expect a score.
They don't expect to be told exactly which instruction caused the drift, why it caused it, and get a concrete rewrite suggestion applied directly to their agent. The first time someone sees that land on a failure they'd been chasing manually for hours, that's usually the moment it clicks.

The second one is tool usage validation. Teams assume if the output looks right the agent did the right thing - it often didn't. You can set tool usage as an explicit eval criterion, so a run only passes if the agent actually called the right tool, not just returned a plausible answer. That's caught a lot of silent failures nobody knew were happening.

Both tie together: observability shows you what happened, root cause analysis tells you what to do about it. One without the other is half the loop.

@easton_carter 
Great question!

The strongest feedback is usually around eval-before-deploy: teams want confidence before pushing prompt, model, tool, or workflow changes to production.

Observability helps you understand what happened after the fact, but evaluation gives you a way to catch regressions earlier.

The combination is important, but the “can I safely ship this agent version?” question seems to resonate the most.

@fletcher_oliver Thank you! Traditional monitoring tells you something broke. We tell you why, where in the chain it broke, and what to change to fix it.

Most monitoring tools are built around infrastructure: latency, error rates, uptime. Useful, but when an agent returns a confidently wrong answer with 200ms response time and zero errors, traditional monitoring gives you a green dashboard while your users get bad outputs.

The difference is we evaluate behavior, not just availability. Every run captures the full decision chain, thinking process between agents, tool calls, handoffs, routing decisions - scored by a panel of LLM judges against criteria you define. When something's off you get a specific diagnosis, not a spike on a graph.

And the loop closes automatically: suggested instruction changes applied directly to your agents, not a ticket for someone to investigate later. What does your current monitoring setup look like?

@fletcher_oliver 
Thank you!

Traditional monitoring is usually great at system signals: latency, errors, uptime, logs, costs.

Agent observability needs a different layer: what the agent decided, which tools it used, whether the steps made sense, whether it stayed aligned with the task, and whether the final outcome was actually useful.

For agents, “200 OK” does not mean the work was done correctly.

That’s where we focus: connecting traces, evaluation, and outcome quality so teams can debug and improve the agent itself.

@mikebrandswarm Hey! You've described exactly the gap we set out to close. You're right that most tools drop you after deploy.

Post-deploy every conversation is logged with full traces - per-agent steps, tool calls, the decision chain between agents. It's not sampled, it's everything, so you're not hoping a bad response happened to get captured.

Bad responses can be flagged and replayed step by step. You see the full thinking process, where the chain diverged from what you expected, and what specifically went wrong. From there the same AI analysis that runs in pre-deploy evals kicks in: root cause, suggested instruction changes, applied directly.

The feedback loop is the point. Pre-deploy evals catch known failure modes before they ship. Post-deploy monitoring catches what real users actually trigger. Both feed back into the same eval framework, so your agent gets measurably better over time rather than just getting patched when someone complains.

Demo to production-grade is exactly the gap we're targeting.

@mikebrandswarm 

Totally agree - the real challenge starts after launch!

We don’t see evaluation as something that should be left entirely to the builder. The feedback loop needs to include traces, bad response review, replay, and comparison against expected behavior so teams can understand what went wrong and whether a fix actually improved the agent.

The goal is to connect pre-deploy evals with post-deploy learning: catch regressions before release, then use real production failures to improve the eval suite over time.

That’s what turns agent QA into an ongoing system, not a one-time checklist.