How do you unit test AI outputs when the results "look" completely correct?

The hardest bugs to spot in LLM-based apps are the ones where the output looks totally convincing at first glance. I had a bug where the model filled in missing fields ran a search and verified the result except it verified the result against its own generated text rather than the original input.

All my basic tests passed because the output data looked super clean and accurate on paper. It made me realize how dangerous it is to let LLMs modify source data instead of just using them for quick search suggestions.

What kind of checks or boundaries do you write in your code to catch these silent logic bugs?