AgentX - Evaluate AI agent, pinpoint issues, and fix with one click.
by•
Evaluate AI agents before they fail. Create test suites, run evaluations, and pinpoint issues before they reach production.
AgentX provides full observability and traceability for your AI agents. AI analysis not only identifies problems but also suggests fixes-like an AI doctor for your agents.
Simulate run your agents across multiple LLM providers to compare performance, cost, and latency, helping you make better decisions about which LLM to go.
Run eval before deploy. Like CI/CD for AI agents.
Replies
Nice launch! Comparing performance, latency, and cost across providers from one place sounds incredibly useful :))
@himani_sah1
Thank you! :))
That’s one of the areas we’re most excited about. Model choice should not be based on hype or guesswork.
Teams need to see how each provider performs on their actual agent workflows - quality, latency, cost, and reliability - before making a switch.
Have you considered adding automated regression testing whenever prompts, tools, or workflows change?
@zerotox
We already do!
Try section “recommended prompt and tools changes” :)
how AgentX handles non-deterministic agent behavior across repeated evaluation runs. Is there a way to measure consistency?
@iamanantgupta
Yes - repeated runs help measure consistency directly: pass rate, output variance, tool-call consistency, and where behavior drifts.
One good run is not enough for agents
@iamanantgupta
Yes - repeated runs help measure consistency directly: pass rate, output variance, tool-call consistency, and where behavior drifts.
One good run is not enough for agents
Can AgentX evaluate multi-agent workflows where several agents collaborate and hand tasks between each other?
@nuseir_yassin1
Yes - that’s one of the main use cases :)
AgentX can evaluate the full multi-agent workflow: handoffs, role boundaries, tool usage, context transfer, intermediate steps, and whether the team completed the intended task.
Multi-agent systems need evals at the coordination layer, not just per-agent outputs
Congrats on the launch :) The screenshots look clean and the workflow appears straightforward for developers.
@syed_shayanur_rahman
Yep! Developers and non-techies can find themselves pretty easily ;)
How does AgentX integrate with existing frameworks like LangGraph, CrewAI, AutoGen, or custom agent architectures?
@roopreddy one word: FULLY ;)
@roopreddy It is supported via our API. See our developer doc here: https://docs.agentx.so/docs/getting-started
That's extremely helpful. My big pain point today is to build a effective continuous improvement process for my agents. I will give a try, definitely . Congrats on the Launch and count on me as a customer !
@alyssoncm Thank you Alysson! Keep me posted and would love to help.
congrats on the launch! agent evaluation is such a pain point right now.how do you handle non deterministic outputs during evaluations?
@avery_thompson2
Thanks! And yes - non-determinism is the reason hard pass/fail per sample doesn't work for agents.
We handle it on a few levels. Each case runs multiple times so we're scoring a distribution, not a single shot. The scoring itself is graded rather than binary: LLM-as-a-judge gives every response a 0–10 rating, with optional cosine and Jaccard similarity against the expected output, all averaged across runs. The quality threshold then sits on that aggregate, so one off-sample nudges the score instead of failing the whole run.
It ends up feeling a lot like setting a test coverage threshold in CI: you define what "good enough" looks like, and a regression has to actually move the average past that bar to block the deploy.
congratulations on the launch! agent reliability is becoming a major concern as deployments grow.what has been the most surprising issue you have uncovered using AgentX?
@imogen_wallace Thank you! Reliability at scale is exactly what we built this for.
The most surprising thing we keep seeing: the failure is rarely the LLM itself - it's the prompt behaving differently once it's inside a chain. Instructions that look perfect in isolation start drifting when an agent is receiving output from another agent instead of a human. Teams had no idea because they were testing agents individually, never as a team under real load.
What's your setup: early deployments or already in production?
Just curious, how is it different from Langfuse/smith?
@naor_sabag Good question. Short version: LangSmith and Langfuse are observability and eval layers you point at an agent you built somewhere else. AgentX is where you build, evaluate, and run the agent. Eval is one part of a loop, not the whole product.
LangSmith lives in the LangChain world, Langfuse is open-source, framework-agnostic tracing plus eval. Both are good at capturing traces and scoring runs. But you bring your own agent and orchestration. They observe and grade, they don't build or run anything for you.
AgentX is agent-native end to end. You build the agent team here, evaluate it against real cases with a panel of LLM judges, and deploy to production from the same place with versioning and rollback. Because the eval sees the actual orchestration, it traces the thinking and handoffs between agents, not just LLM spans. And when a score is bad it tells you why and proposes the fix, applied straight to the agent's instructions, instead of handing you a dashboard to interpret.
If you already have an agent, our eval can score it via an HTTP endpoint or precomputed outputs too. The difference is you can also just build and run the whole thing here.
What are you using today?
@naor_sabag Great question!
Langfuse / LangSmith are strong for tracing, observability, debugging, and evaluating LLM apps.
Where we’re positioning AgentX Evaluation is more around agent QA before deployment: running structured scenarios against full agent workflows, comparing performance across different LLMs, checking tool-call behavior, multi-step execution, consistency, and whether the agent actually completes the business task.
So less “what happened in this trace?” and more “is this agent version ready to ship, and will it still work if we change the model, prompt, tools, or workflow?”
We see it as complementary in many stacks, especially for teams moving agents from prototype to production.