AgentX - Evaluate AI agent, pinpoint issues, and fix with one click.

@roopreddy one word: FULLY ;)

@roopreddy It is supported via our API. See our developer doc here: https://docs.agentx.so/docs/getting-started

@alyssoncm Thank you Alysson! Keep me posted and would love to help.

@carter_son Good one, this is what kills most "agents in a room" setups.

We handle it by architecture: it's hub-and-spoke. A single Manager reads the task and assigns it to specific specialists, so agents don't self-assign and grab the same job. Each agent has its own scoped role, permissions and knowledge - clear separation keeps them in their lane and stops context explosion. They hand off context and escalate rather than all broadcasting at once.

And since this is the eval launch: we trace both the individual agents and the overall workflow, so if two start stepping on each other it surfaces before production, not in a user complaint.

@chen_hao3 Fast to get started - you don't have to hand-define every agent.

There's a template library to jump from, and chat-to-build Agent available: describe what you want in plain language and it spins up the agents and wires the orchestration for you. Manual role definition is there if you want full control, but it's the optional path, not the default.

Then you tighten it with evals. Run evaluations against real cases, check the behaviour, refine the instructions until it's dialled in. That's what takes you from "works in a demo" to "I actually trust this in production."

Most people have a first agent running in a few minutes. What are you trying to automate first?

@avery_thompson2 

Thanks! And yes - non-determinism is the reason hard pass/fail per sample doesn't work for agents.

We handle it on a few levels. Each case runs multiple times so we're scoring a distribution, not a single shot. The scoring itself is graded rather than binary: LLM-as-a-judge gives every response a 0–10 rating, with optional cosine and Jaccard similarity against the expected output, all averaged across runs. The quality threshold then sits on that aggregate, so one off-sample nudges the score instead of failing the whole run.

It ends up feeling a lot like setting a test coverage threshold in CI: you define what "good enough" looks like, and a regression has to actually move the average past that bar to block the deploy.

@imogen_wallace Thank you! Reliability at scale is exactly what we built this for.

The most surprising thing we keep seeing: the failure is rarely the LLM itself - it's the prompt behaving differently once it's inside a chain. Instructions that look perfect in isolation start drifting when an agent is receiving output from another agent instead of a human. Teams had no idea because they were testing agents individually, never as a team under real load.

What's your setup: early deployments or already in production?

@naor_sabag Good question. Short version: LangSmith and Langfuse are observability and eval layers you point at an agent you built somewhere else. AgentX is where you build, evaluate, and run the agent. Eval is one part of a loop, not the whole product.

LangSmith lives in the LangChain world, Langfuse is open-source, framework-agnostic tracing plus eval. Both are good at capturing traces and scoring runs. But you bring your own agent and orchestration. They observe and grade, they don't build or run anything for you.

AgentX is agent-native end to end. You build the agent team here, evaluate it against real cases with a panel of LLM judges, and deploy to production from the same place with versioning and rollback. Because the eval sees the actual orchestration, it traces the thinking and handoffs between agents, not just LLM spans. And when a score is bad it tells you why and proposes the fix, applied straight to the agent's instructions, instead of handing you a dashboard to interpret.

If you already have an agent, our eval can score it via an HTTP endpoint or precomputed outputs too. The difference is you can also just build and run the whole thing here.

What are you using today?

@naor_sabag Great question!

Langfuse / LangSmith are strong for tracing, observability, debugging, and evaluating LLM apps.

Where we’re positioning AgentX Evaluation is more around agent QA before deployment: running structured scenarios against full agent workflows, comparing performance across different LLMs, checking tool-call behavior, multi-step execution, consistency, and whether the agent actually completes the business task.

So less “what happened in this trace?” and more “is this agent version ready to ship, and will it still work if we change the model, prompt, tools, or workflow?”

We see it as complementary in many stacks, especially for teams moving agents from prototype to production.

@david_marko A hundred percent! If we only compare input and output, the hidden issues will never surface!

@david_marko That’s a really good example of the kind of failure that standard observability often misses.

The main agent can complete the task correctly, but a later AI step - summary, extraction, classification, handoff note - can silently drop important information.

This is why evals need to cover the full pipeline, not only the primary agent response. The source conversation, intermediate steps, and final structured output should all be checked against each other to catch missing fields, dropped context, or mismatches.

“Looks fine” is often the most dangerous failure mode in production

@codeamesh_consultancy Great observation. There are many metrics we offer for various of scenarios.
For example cosine score for the semantic closeness, Jaccard score for the text overlapping rate. And most useful is the overall score of the whole eval from the multiple LLM-as-a-judge.