Mathew Chang

Generation tools are low-stakes when they're wrong draft an email, get a bad draft, edit it. Judgment tools are different: grading, flagging, scoring. If the output looks confident and it's actually a guess, someone might act on it without knowing that.

Working on letsflw's exam-checker made this concrete objective questions are easy (exact match, no ambiguity), but for open-ended answers we made a call to surface the AI's mark as a suggestion to review, not a final score, because presenting it as settled felt dishonest.

I've been thinking about something that surprised me while building AI products.

The first version of a product almost never fails because of poor engineering. It fails because it's answering the wrong question.

Hey all. I'm really interested in how you evaluate your agents in production.

The standard approach is sample, LLM as judge and possibly a golden dataset. That seems to be the standard setup.