AI data agents are converging on the same lesson: accuracy is a context problem, not a code problem
I've been watching the "AI data analyst" space for a while, and something clicked after reading Anthropic's internal write-up on how they run analytics with Claude. I wanted to lay out the pattern I'm seeing, because the whole category is quietly pivoting from a speed story to a trust story, and most of the marketing hasn't caught up to that yet.
Disclosure up front: I'm a co-maker of ada.im (and chartgen.ai and clawteams.ai), so I'm biased. Posting this in Self-Promotion for that reason, but I've tried to keep it analytical rather than a pitch.
THREE GENERATIONS OF "SELF-SERVICE ANALYTICS"
Traditional BI (Tableau, Power BI, Looker) made data visible but pushed all the hard work onto humans. Someone still had to model the data, write queries, and build the dashboard. "Self-service" really meant "self-service if you already know how to model data."
Then came text-to-SQL. The promise was that natural language would replace the query editor. In practice, a lot of these tools produced syntactically perfect SQL against a schema they didn't actually understand, confidently answering the wrong question because they guessed which table "revenue" lived in.
Now we're in the agentic phase: systems that plan, retrieve context, run analysis, check their own work, and hand back a narrative answer. A bunch of products sit here now, including ada.im, various "chat with your data" startups, and the analyst-agent features being bolted onto the incumbents. What's interesting is that fast-moving startups and a frontier lab independently landed on nearly the same architecture.
WHAT ANTHROPIC'S WRITE-UP ACTUALLY SAYS
Their post ("How Anthropic enables self-service data analytics with Claude," June 2026) reports that ~95% of their business analytics queries are automated at ~95% accuracy in aggregate. But the useful part isn't the headline number, it's the diagnosis.
Their core claim: analytics accuracy is a context and verification problem, not a code-generation problem. Modern models already write excellent SQL. What breaks is everything around the SQL. They note the same agent sat around 21% accuracy without the right scaffolding, and that curated context ("skills") is what drove the jump toward 95%. The intelligence wasn't the bottleneck.
They break failures into three modes that generalize to basically every tool in the category:
Concept-to-entity ambiguity: a vague word like "revenue" or "active users" maps to many plausible tables, and the agent picks a reasonable-looking wrong one.
2. Data staleness: schemas and definitions drift constantly, so the agent's knowledge silently rots.
3. Retrieval failure: the right answer is documented somewhere, but the agent never finds it.
Their fix is a four-layer stack: clean data foundations to shrink the space of plausible answers; a governed semantic layer as the mandatory default path (so metrics are computed the same way the official BI tool computes them); "skills" as procedural runbooks; and a validation layer of offline evals plus live monitoring. The phrase I keep coming back to is their description of the real risk: confidently generating beautifully formatted but incorrect output. A wrong answer that looks completely plausible is the failure mode none of the guardrails fully catches, and it's the one that quietly kills trust.
THE CONVERGENCE
Look at how these startups describe themselves and you see the same lessons arrived at independently. ada.im (my own product) frames its defense against hallucination as architectural: an auto-generated semantic layer so the model only touches verified, structured data. That's the same "sources of truth" idea Anthropic puts at the center of its stack. It also has a human-in-the-loop editing step, which maps to Anthropic's adversarial-review and online-validation layers. I'm not claiming it's solved; the honest open question for all these tools is how they hold up on chaotic real-world data.
What I find notable is the architectural agreement. When independent teams keep reaching for a semantic layer plus a human checkpoint plus continuous validation, that's a signal the shape of the solution is stabilizing.
ON "100% PRECISE": A REALITY CHECK
You'll see marketing claims about perfectly accurate AI analysis. I'd push back on that framing, including on my own side. No analytics system, human or AI, is perfectly precise on every real-world question; Anthropic's own best number is ~95%, not 100%.
What you can engineer toward 100% is accountability, not precision. Precision on the questions that matter, and full traceability on all of them. Concretely: constrain the model to a governed semantic layer instead of raw table access (this kills most entity-mapping errors); treat context as a maintained product, not one-time setup; separate observation ("the data shows X") from interpretation ("this suggests Y"); attach provenance to every answer (source tier, freshness, whether it was reviewed, who owns it) so "trust me" becomes "verify me"; keep a human checkpoint between generation and decision; and measure accuracy continuously with pinned evals.
DISCUSSION
The category is shifting its center of gravity from speed (now table stakes) to accountability. Anthropic's contribution was naming the failure modes publicly; the startups are betting the same architecture can be packaged for teams without a dedicated data-engineering org.
Curious what others think: for those who've deployed one of these agents on real internal data, did the semantic-layer approach hold up, or did it break the moment someone asked something slightly off-schema? And has anyone found provenance footers actually change how much stakeholders trust the output?
Replies
Taking only your second question, because I have not deployed one of these on real internal data and will not pretend otherwise.
The provenance footer is generated by the same run that produced the answer, and that is the problem with it. Take your third failure mode, the one where the right answer is documented somewhere and the agent never finds it. The footer does not go blank in that case. It cites whatever the agent did find, correctly formatted and confident, and the answer now looks sourced. So the footer raises trust hardest in exactly the case where trust is least warranted.
I have been measuring a much smaller version of this on my own agents for a week, just whether something reported as done had actually happened. Roughly one in three reports did not match the real state, and none of them looked wrong. Accountability is the right word for what you are after. A footer written by the run being held accountable is the wrong mechanism for it.
the semantic layer answer holds until the schema question changes faster than someone updates the layer. we've felt this on our own side - a semantic layer is itself a piece of context that goes stale the same way raw schemas do, just one level removed. someone still has to notice when a metric definition shifted and go update the mapping, and that job tends to fall through the cracks because it feels like maintenance work, not shipping work. so the real dependency isn't "do you have a semantic layer" its "who owns keeping it current and how fast do they find out its wrong." the off-schema case Tony asked about is usually where that ownership gap shows up first
ClawTeams
Both of you are circling the same crack from different sides, and I think you're right that it's the load-bearing weakness in the whole architecture.
Abdullah - your point about the footer being written by the same run that produced the answer is the sharpest version of this I've heard. On the retrieval-failure mode especially, the footer doesn't go blank, it cites whatever was found and now the wrong answer looks sourced. That's exactly the "confidently plausible but wrong" failure that none of the guardrails fully catch, and you're right that a self-generated provenance footer can make it worse, not better. The one-in-three mismatch you measured (and none looking wrong) is the whole problem in miniature. The fix has to be an independent check, not a self-report - provenance is only worth anything if the thing attesting isn't the same process being held accountable.
Gal - same root cause, one level removed. A semantic layer is just context, and context rots. Agreed the real question isn't "do you have a layer" but "who notices when a metric definition drifts, and how fast." That ownership gap is where the off-schema case breaks first, exactly like you said. It tends to fall through because it reads as maintenance, not shipping.
So I think the two of you together point at the actual dependency: not the semantic layer or the footer as artifacts, but an external staleness/verification signal that lives outside the run - something that flags "this mapping is N days behind" or "this answer's sources were never reviewed" before a stakeholder ever sees a clean-looking output. Accountability only means something if the checker isn't the doer. Fair to say that's the unsolved part.