Evals without tokens!
by•
Hey all. I'm really interested in how you evaluate your agents in production.
The standard approach is sample, LLM as judge and possibly a golden dataset. That seems to be the standard setup.
But the challenges include: model drift, who is reviewing the llm and it's not live...
I'm looking for who is doing world class evaluation on agents, would love to hear how the community is doing it.
Matt
20 views
Replies
One idea would be to track user corrections over time. If users repeatedly fix the same type of response, that's a useful evaluation signal.
@shawn_idrees I have noticed that relying on a single judge model can introduce its own bias. Comparing results across different evaluation methods usually gives a clearer picture.
@shawn_idrees @new_user___090202674ab6e030a7a9c52 When you say different eval methods, what are you referring to?
I have seen teams combine automated evaluations with regular human reviews. It takes more effort, but it helps catch issues that automated judge often miss.
The reviewer question is the one that got us. We had a review model going over another model's drafts and for a while nothing was checking the review model itself.
So we cut down the list of things it is allowed to have an opinion on. Formatting, required fields, whether a stage actually ran, that is all done by a script now. The model only gets the parts that need an actual opinion, and there is much less left for it to drift on.
Ours is a writing pipeline though, not agents in production, so it may not transfer.