AgentX - Evaluate AI agent, pinpoint issues, and fix with one click.
by•
Evaluate AI agents before they fail. Create test suites, run evaluations, and pinpoint issues before they reach production.
AgentX provides full observability and traceability for your AI agents. AI analysis not only identifies problems but also suggests fixes-like an AI doctor for your agents.
Simulate run your agents across multiple LLM providers to compare performance, cost, and latency, helping you make better decisions about which LLM to go.
Run eval before deploy. Like CI/CD for AI agents.
Replies
The eval suite plus multi-provider simulate-run (basically CI/CD for agents) is the part I'd wire in first — pre-prod agent debugging is exactly where I lose the most time. Where do the eval suites and traces actually live: stored per-project in AgentX's hosted backend, or can I export/version them in my own repo so they run in my CI? And when you simulate across LLM providers, do I bring my own keys per provider or does AgentX proxy those calls?
AgentX - Multi-agent and eval framework
@noctis06
Today, the main eval suites and traces live per project in AgentX, so teams can inspect runs, compare results, and debug failures in one place.
Where we want this to go is exactly what you described: portable/versionable eval artifacts that can run as part of CI, not just live in a UI.
For provider simulation, we’re designing around flexibility: teams should be able to test across providers without being locked into one setup, whether that means BYO keys or a managed/proxied flow depending on their workflow
That flexibility is exactly the right call — BYO keys would be my default so cost and rate limits stay on my own provider accounts. When the portable eval artifacts land, are you picturing them as plain diffable files (JSON/YAML committed in-repo) so a changed eval shows up in a normal PR review, or more of an export/import bundle? The in-repo route is what would actually get this into my CI.
Congrats on the launch! 🎉
for multi-agent setups specifically, does it trace failures at the individual agent level or do you just see the whole chain break down as one blob?
AgentX - Multi-agent and eval framework
@boyuan_deng1 Thank you!
We do both! We look at each individual agent detail process to determine if there is any issue. And at same time we look at the overall workflow and output.
Curious how this handles non-deterministic failures like an agent that passes 9 times out of 10.
Does the eval framework catch flaky behavior or only hard fails?
AgentX - Multi-agent and eval framework
@alexander_gray3 Great question Alexander! We actually have a setting that allows you to choose the repetition amount for each test set. Because we also realized that, for same question, the natural of non-deterministic may lead to different answer each time with same exactly setup. So you can totally put say 10 repetition for certain test case to make sure there will be no consistency failure.
How much setup is needed to wire this into an existing agent stack?
Is it a quick SDK drop-in or more of a real integration project?
AgentX - Multi-agent and eval framework
@amna9
It's designed to be a lightweight integration. If you already have an agent, you can connect it using our official AgentX Python SDK and start evaluating it without major changes to your existing stack. For most teams, it's more of an SDK drop-in than a large integration project.
The official SDK is available via AgentX Python SDK GitHub.
How do you benchmark 'productivity gains' here is there a before/after metric you're tracking with early users?
AgentX - Multi-agent and eval framework
@benjamin_harris3
Fair question! And an honest one, because "productivity" gets thrown around a lot without a definition.
For the eval side specifically, we don't try to claim a vague "X% more productive." We track concrete before/after signals with early users:
• Debugging time - how long it takes to find the root cause of a bad agent response. Our AI root-cause analysis is aimed squarely at collapsing this from hours of reading traces to minutes.
• Issues caught pre-deploy vs. in production - the whole point is moving failures left, so we look at what evals catch before release vs. what would've reached users.
• Eval-to-deploy cycle time - how fast a team goes from "I changed a prompt" to "confident enough to ship."
• Quality trend across versions - the LLM-as-a-judge score (0–10) tracked version over version, so "better" is a number, not a vibe.
Curious how you measure agent quality today? Most teams we talk to are still doing it by eyeball, which is kind of the problem we're attacking.
We use a mix of OpenAI, Anthropic, and Gemini models across our agent stack. Can AgentX help us decide which model works best?
AgentX - Multi-agent and eval framework
@pawel_dmochewicz Yes, Paweł, tanks for this question! That's one of the core use cases. AgentX lets you simulate and evaluate your agents across multiple LLM providers, then compare performance, cost, and latency side by side. Instead of guessing which model to use, you can make data-driven decisions before deploying changes to production.
Timbal AI
The "CI/CD for agents" framing is the right one. How does AgentX handle non-deterministic outputs? Curious whether you can set tolerance bands on an eval so a single off-sample doesn't fail the whole run, or if it's pass/fail per case.
AgentX - Multi-agent and eval framework
@david_vilalta Glad the framing lands - and you've put your finger on exactly why we don't do hard pass/fail on a single sample.
Three pieces handle the non-determinism:
1. Multiple runs per case. Every case has a number_of_requests setting, so we run it N times and score the distribution rather than a single shot. One unlucky sample doesn't get to define the case.
2. Graded scoring, not binary. Each response gets an LLM-as-a-judge rating (0–10), with optional cosine and Jaccard similarity (0–1) against the expected output. Those average across runs and across cases - so an off-sample nudges the average instead of flipping a red/green bit.
3. Thresholds on the aggregate. Teams define their own eval criteria and quality thresholds, and a regression has to push the averaged score past that bar to fail the run — which then blocks the deploy, the same way automated tests stop a bad release. So yes: it's a tolerance band on the distribution, not pass/fail per sample.
How are you handling it on your side today - straight averaging, or a "k out of n must pass" rule?
What stands out here is pairing the orchestration layer with an evaluation framework. A lot of agent platforms ship the automation half and skip the how do you actually trust this in production half.
AgentX - Multi-agent and eval framework
@new_user___10520260379921a76fc2d64
That's exactly the gap we set out to close. Most platforms stop at "here's your agent, good luck" - and teams end up finding out if it works the way every software team used to before CI existed: in production, from users.
Build without eval is just a prototype you haven't stress-tested yet. We wanted the whole loop in one place - build, evaluate, deploy, so shipping with confidence isn't a separate project on top of building the agent.
Appreciate you putting it so cleanly.
congratulations team! AI powered root cause analysis sounds like a huge time saver.what kind of feedback does the system provide for failures?
AgentX - Multi-agent and eval framework
@olivia_bennett7 Thank you! Here's what the feedback actually looks like in practice.
You get a top-level score summary first: average rating, score range, variance and consistency across all runs. Under that, the AI Analysis runs a panel of multiple LLM judges and tracks agreement between them: how close their scores were, mean absolute error, inter-judge kappa. If judges strongly disagree on a case it gets flagged for human review.
The breakdown then goes section by section: overall assessment, instruction adherence, response patterns, reasoning analysis, tool usage. Each surfaces a different failure mode rather than bundling everything into one number.
At the bottom is Suggested Instruction Changes - concrete edits to your agent's instructions based on what the analysis found. Not "something's wrong," but specifically what to fix and why. What kind of agent are you evaluating?
AgentX - Multi-agent and eval framework
@olivia_bennett7
Thank you!
For failures, the goal is to show more than “this failed.”
We want to surface where it failed, why it likely failed, and what to check next - for example wrong tool choice, bad parameters, missing context, weak instructions, dropped fields, or a downstream step issue.
Then teams can replay the case and verify the fix instead of guessing.
congratulations! the debugging workflow sounds incredibly valuable.can AgentX evaluate multi agent systems as well?
AgentX - Multi-agent and eval framework
@barnaby_lloyd Yes - that's actually where it matters most.
Single agent evals are straightforward. With multi-agent teams the hard part is knowing which agent in the chain introduced the failure, not just that the final output was wrong. We trace both the individual agent steps and the overall workflow, so you get a per-agent breakdown rather than one opaque score at the end.
The AI analysis then surfaces where exactly the delegation broke down: wrong routing, instruction drift, tool call failure, and suggests the specific instruction changes to fix it.
What kind of multi-agent setup are you working with?
AgentX - Multi-agent and eval framework
@barnaby_lloyd
Thank you!
Yes - multi-agent systems are one of the main areas we care about.
AgentX can evaluate not only the final answer, but also how agents coordinate: handoffs, tool usage, role boundaries, intermediate steps, and whether the full team completed the intended workflow.
That’s where debugging gets especially important.