Three input cases I want to test in an on-device AI app

What should an on-device relationship-reflection app do when the input barely says anything?

I’m the maker of 他爱我吗 (Does He Love Me?), an iOS app that uses Laya through Core ML to compare candidate answers locally. Here is a small test set I’d like to discuss—not benchmark results or examples from real users.

Question: Does he like me?

A. ‘He replied.’

B. ‘He replied quickly once, but has declined every invitation to meet.’

C. ‘He regularly makes plans, follows through, and checks in without prompting.’

These inputs contain very different amounts of evidence. For A, I’d want uncertainty to be prominent. For B, I’d want the conflicting signals to remain visible. For C, I’d still avoid presenting a model score as a measured probability of someone’s feelings. Those are evaluation goals, not claims that the current model reliably does all three.

A second check would keep the situation identical and change only the wording or order of the candidate answers. If the ranking moves a lot, that’s something to investigate rather than hide behind a polished percentage.

For builders working on local AI: which failure case would you add first—too little context, contradictory evidence, or sensitivity to answer wording? A concrete synthetic example would be especially helpful. Please don’t share private messages or identifying details.

The current app has a Chinese interface and offline Chinese speech input. It is a one-time paid download. This discussion is about evaluating the interaction, so there’s no need to buy it to participate.

Disclosure: I’m the maker; this post was drafted with AI assistance.