AgentX - Evaluate AI agent, pinpoint issues, and fix with one click.

@robin_xw Looks good! Congrats

@robin_xw this is honestly one of the most needed pieces in the whole AI agent stack right now!!!

building agents is getting easier but trusting them in production is still kinda scary because failures are usually invisible until a user hits them

one thing I’m curious about: in your experience what is the hardest thing to evaluate properly today — multi step reasoning tool usage accuracy long running state consistency or just reproducing real world scenarios reliably

also love the CI/CD analogy for agents that feels exactly right

excited to see where this goes 🔥

@robin_xw One-click fixes for pinpointed issues is solid. What was the hardest part of building the detection layer? Did you train on specific agent failure patterns or go more general?

@blah_mad 
That’s a great point!

For us, eval failures should point to the action boundary, not just a score: which tool call, write, state change, or approval step caused the risk.

That’s where agent evals become useful for real workflows, not just message quality.

@polman_trudo Exactly. Teams can define evaluation criteria and quality thresholds. If a change causes performance regressions, the evaluation can fail before deployment, similar to how software teams use automated tests to prevent bad releases.

@abod_rehman Thank you Abdul. We currently support all major LLM vendors out of box (Claude, GPT, Gemini, Llama, Grok). You can also use custom LLM to provide your own base url that point to any other LLM that is not listed here.

@yagnaveena Hey, thanks for the question! This is the failure mode that's hardest to catch and most common in practice. No alerts fire, no errors surface, the agent just quietly becomes worse across versions.

The way we handle it is version-tracked scoring. Every eval run is tied to a version, so you're not just getting a score in isolation - you're seeing the trend line across releases. A drop from 8.2 to 7.9 to 7.4 is invisible in any single run but obvious when you're tracking it over time.

The consistency metric is what catches gradual drift specifically. An agent can maintain its average score while its variance increases, meaning it's getting less predictable even if the mean looks fine. That's early warning before the average itself starts moving.

And because the eval criteria stay fixed across versions, you're comparing apples to apples. The definition of "good" doesn't drift with the agent - so when outputs start becoming less useful, the score reflects it even if they're technically still correct.

@yagnaveena 
Gradual drift is often harder to catch than a clear failure.

That’s exactly the kind of thing evals should monitor across releases: not only whether the answer is technically correct, but whether it is still complete, useful, consistent, and aligned with the expected outcome.

The idea is to compare agent versions over the same scenarios, track quality signals over time, and flag regressions before they reach users.

So yes - quality drift, not just pass/fail, is an important part of how we think about agent evaluation.

@artstavenka1 Great challenge! And you're right that naive CI/CD breaks down if you treat each agent run as a binary test. We don't.
The gate sits on the aggregate, not the individual run. Each case runs multiple times, gets graded 0-10 by a panel of LLM judges, and the threshold is set on the averaged score across runs. So one off-sample nudges the average instead of flipping the gate.

You can also track consistency explicitly - it's one of the core metrics in the report. An agent that scores 8.0 average with low variance ships very differently from one that swings between 5 and 10. Both might have the same average but only one is actually reliable.

So yes, closer to "9/10 runs must score above X" than a hard pass/fail, but applied to a distribution rather than a count.

@artstavenka1 
Great question, and yes - determinism is tricky with agents.

We don’t think about gating only as a single pass/fail run. For agent workflows, it usually needs repeated runs and thresholds: pass rate, consistency, tool-call accuracy, and severity of failures.

So a team might gate on something like “9/10 runs pass for critical scenarios,” but also block deploys immediately for certain high-severity failures, like wrong tool usage, missing required fields, or unsafe outputs.

The goal is not to pretend agents are deterministic - it’s to measure the variance before production.

They're really solving two different problems, and the honest comparison lands in our favor. n8n is a great workflow automation engine - fair-code, self-hostable, integrations, and since 2.0 you can drop a LangChain-powered AI Agent node into a workflow. If your problem is connecting a lot of services into a reliable pipeline, it's genuinely good. But its center of gravity is the workflow, with the agent as one node inside it, and it stops at "built." It doesn't answer the question that matters once you ship: is this agent actually doing the right thing?

AgentX is agent-native, and the build experience reflects that. You're not hand-wiring every branch on a canvas. The core object is a team of agents- a Manager that reads intent and delegates to specialists, with scoped roles, knowledge and structured handoffs. You get a library of ready-made agents to start from, and you can build new ones just by describing what you want in plain language.

Tooling is just as flexible: connect via MCP, use the built-in tools and integrations, or have an agent build a custom tool for you on the spot. Deterministic workflows are there too when you want hard logic - so you're never forced into one paradigm.

Then comes the part that's our whole reason for existing: evaluation. Build datasets from real cases, run your agent against them, and get graded scoring from a panel of LLM judges (0–10, plus optional cosine and Jaccard similarity), with consistency and variance so you know whether you've got a reliable agent or a coin flip. Every run is version-tracked, so you catch quality drift across releases — the slow degradation where outputs are still technically correct but quietly getting worse, which no execution log will ever surface.

You can read the full thinking process between agents, every step, every tool call and handoff, and when something breaks the AI analysis pinpoints where, why, and what to change - surfacing suggested fixes straight to the agent's instructions and orchestration. You can even gate deploys on a quality threshold, so a regression fails before it ships. Real CI/CD for agents, not a metaphor.

And the honest kicker: it's not even either/or. AgentX evals can score an agent you built in n8n too — point our eval at your n8n webhook and we call it per case, or feed us the outputs you've already generated. Either way you get the full judge panel and root-cause analysis on top.

So the one-liner: n8n builds the agent, AgentX builds it, evaluates it, and makes it one you can trust in production.

@barath_kanna_bk 
Great question, thank you!

n8n is excellent for workflow automation - connecting apps, APIs, triggers, and deterministic flows.

AgentX is focused more on AI agent workflows: agents that can reason, use tools, collaborate with other agents, and be evaluated before deployment.

So I’d say n8n is great when you know the exact workflow upfront. AgentX is useful when the workflow needs more decision-making, natural language handling, multi-agent coordination, or testing how the agent performs across models and edge cases.

They can also be complementary, depending on the stack.

@itsluo We agree. Production data is usually the best source of truth. Our focus is helping teams build evals from real traces and failure cases, while also supporting synthetic generation when coverage gaps exist. The best results come from combining both approaches.

@harini_mukesh 
Great question! The most common first team structure users build is a simple 3-agent hub and spoke setup - one Manager agent that reads user intent and routes to the right specialist, and 2-3 sub-agents handling specific roles like Support, Sales, or Lead Collection. 

The Manager is the key - users don't need to know which agent to talk to, the Manager figures it out and delegates automatically. 

But teams can scale way beyond 3 agents. Many organizations end up running 10-agent teams covering full department: Sales, Support, HR, Finance, Operation, all under one orchestrating Manager. We've even built accounting workflows with 15+ specialized agents handling everything from invoice processing and payroll to tax calculations and compliance checks. 

The architecture stays the same regardless of size. You start small, then just keep adding specialist agents as your needs grow. 

Most popular starting points we see: 

• Customer Support + Sales triage team 

• Content pipeline (Researcher, Writer, Editor) 

• Internal ops hub (HR, Finance, IT under one Manager) 

Happy to walk through how any of these are structured if helpful! 

@margharitha 
The difference in our approach is that AgentX is focused on agent teams and full workflow QA, not only traces or model-level evals. We care a lot about how agents coordinate, use tools, hand off work, complete multi-step business tasks, and perform across different LLMs.

So in many stacks this can be complementary. Our focus is the question: “is this agent workflow ready to ship, and will it still work after prompt, model, tool, or workflow changes?”