Our data test set is mostly easy cases. How do you add the rare ones that cost most?

Before each release we check our AI feature with a list of sample questions. Looking at it again, almost all of them are the common, easy kind.

A made-up example: nearly every sample is a normal refund request, and only two are odd ones, like a refund split across two currencies.

Why it’s hard, as far as I can tell: real data looks the same way. Common cases show up all day, so they fill the test set without anyone choosing them. The rare ones that cost the most show up once a month, so nobody has enough examples to test them.

What we’re trying (early, could be wrong): asking support for the three worst cases of the quarter and adding them by hand, even though the set no longer looks like real traffic.

"So, How do you decide which rare cases earn a spot in the test set?"