... common, easy kind. A made-up example : nearly every sample is a normal refund request, and only two are odd ones, like a refund split across two currencies. Why it s hard, as far as I can tell: real datalooks the same way. Common cases show up all day, so they fill the test set without anyone choosing them. The rare ones that cost the most show up once a month, so nobody has enough examples to test them ...

... where the model filled in missing fields ran a search and verified the result except it verified the result against its own generated text rather than the original input. All my basic tests passed because the output datalooked super clean and accurate on paper. It made me realize how dangerous it is to let LLMs modify source