Evals without tokens
by•
Hey all. I'm really interested in how you evaluate your agents in production.
The standard approach is sample, LLM as judge and possibly a golden dataset. That seems to be the standard setup.
But the challenges include: model drift, who is reviewing the llm and it's not live...
I'm looking for who is doing world class evaluation on agents, would love to hear how the community is doing it.
Matt
7 views
Replies
Most of what catches regressions for me costs zero tokens, and the judge only comes out after that layer is clean.
Concretely, per generated response: does it parse against the schema, are the required fields non-empty, is any field secretly the fallback string, does it contain anything from a banned-phrase list, does it contradict its own inputs. That last one earns its keep — if the request says the kid is a girl, "he" anywhere in the output is a hard fail, and that slipped past several rounds of human reading while a two-line assertion catches it every time. Length and structure bounds catch drift cheaply too. All deterministic, milliseconds, and it never disagrees with itself between runs.
The judge is worth paying for on one thing: tone. Nothing cheap tells you the output got subtly colder. The trap there is your "who reviews the llm" point — a judge built on the same stack as the generator inherits its taste. Mine flagged structural problems instantly and stayed blind to voice regressions for weeks, because both of them thought the new voice was fine.
Two additions to the standard setup I'd argue for. Build the corpus out of real production inputs rather than a hand-written golden set, since hand-written ones drift toward the cases you already thought of. And track variance across repeated runs on the same input, not just the mean — a moving mean is a signal, a widening spread is usually the earlier one.
ClawTeams
Being early here so: the setup you described (sample + LLM-as-judge + golden set) is where most people stop, and I think it quietly under-measures the thing that actually hurts in production.
Two things that have moved accuracy more for us than a better judge:
Pin ground truth so it can't drift. An eval written against live data goes stale the moment the underlying number moves — you end up "failing" on a correct answer. Anchor each eval to a frozen expected value, not to whatever the query returns today.
Track the path, not just the output. For us, the strongest online signal isn't a judge score — it's the share of answers that resolved through the trusted/governed source vs. improvised. When that share drops, accuracy is about to drop, and you see it before a user does.
On model drift specifically: LLM-as-judge drifts with the same models you're evaluating, so we treat the judge as a suggestion and the frozen ground-truth comparison as the actual pass/fail. Curious what "world-class" looks like to you here — continuous, or release-gated?