The benchmark score that sold you your model can't tell you if your product works

Frontier models now cluster above 90% on the public benchmarks, and Gartner says over 40% of agentic projects get killed by 2027. Those two facts are closer to the same fact than they look — here's the eval that actually predicts whether your thing ships.

There's a number that decided which model your product runs on. You saw it on a leaderboard or in a launch post — 94% on this, state-of-the-art on that — and it did its job: it made you feel safe picking. I've done exactly this. That number is close to useless for the thing you actually care about, and this year it got worse, not better.

Two facts that are really one fact. The first: frontier models have saturated the benchmarks we leaned on. When everyone clusters above 88–90% on the standard tests, the gaps between them stop meaning anything — you're reading noise and calling it a decision. The second: Gartner is projecting that over 40% of agentic AI projects get canceled by the end of 2027, and the reasons it lists aren't "the model wasn't smart enough." They're escalating cost, unclear value, and weak risk controls. Put those together and you get the uncomfortable version: the score you picked your model on and whether your product survives contact with real users are almost unrelated quantities.

Here's why, concretely. A benchmark asks one question: given this input, did the model produce the right final answer? Your product almost never lives or dies on that. It lives on the path. An agent that lands on the correct answer by calling the wrong tool, backtracking twice, leaking its reasoning into the user's face, and taking nine seconds to do a one-second job scores as a pass and ships as a failure. The eval that actually correlates with "does this work" is the trajectory: the sequence it took, the tools it reached for, what it did when a call failed, whether it looped before it landed. The final answer is the last frame of a movie you didn't watch.

The good news for makers is that the fix is smaller than the enterprise tooling market wants you to believe. You do not need an eval platform to start. You need about twenty real cases. Pull them from your own traffic — the actual messages, the actual weird inputs, the three that made you wince — and save them. When you change a prompt or swap a model, rerun those twenty and read the paths, not just the outputs. That's it. Twenty hand-picked real traces will tell you more about your product than any leaderboard, because they're your distribution, not someone's held-out academic set. Most of the "our AI got worse after the update" stories are just teams who had no such set and were flying on vibes.

One trap on the way, because it's fashionable and it bites: don't outsource the judgment to another model and walk away. "Ask a big model whether this output is good" is a real technique, but it has known, documented biases — it rates longer answers higher, it prefers whichever option it saw first, and it quietly prefers text written in its own style. Use it to triage, never as the final word on anything that matters. A judge with a length bias will happily tell you your more verbose, worse version is the better one.

I'll say the harder part, because I build on the wrong side of this. Murror is an AI companion for understanding your own emotions, and there is no benchmark for it. None. There's no golden answer to "did this help someone sit with a hard feeling for ten minutes." The trajectory framing doesn't fully rescue me either — the path that matters is happening inside a person, not in a tool-call log. So my eval is the least scalable one there is: I read what real people actually wrote, and whether they came back and went deeper or bailed. It's slow, it doesn't fit in a dashboard, and it's the only signal that has ever been right. If your product's value is subjective, make peace with this early: your eval is going to be a human reading real sessions, and no leaderboard is coming to save you.

The makers who make it through the 2027 cull Gartner is describing won't be the ones who picked the highest-scoring model. That was never the hard part, and this year it's noise. They'll be the ones who built a small, honest, embarrassingly manual way to see what their thing actually does to real people — and looked at it often enough to catch it drifting.