My safety classifier flagged my own button. How do you test UI strings against a content classifier?
I run a 3-level classifier (0 normal, 1 distress, 2 crisis) in front of a text generator. At Level 1 or 2 the app skips the generation, shows a support page, and pauses the feature. The final level is the max of three signals: a phrase filter, the classifier, and a level returned by the generation call itself. I tested it across several sets of typed messages: crisis, sarcasm, jokes, other languages.
Then, while testing on production, I tapped one of the three preset answer buttons (“I failed again”) and got the support page and a lockout. At the time, a Level 1 flag meant a 72-hour pause.
The cause: the preset labels went through the classifier exactly like typed text. Run alone, the classifier returned Level 1 on that label in 5 of 5 runs, while the grading call returned 0. Max of signals meant Level 1 won.
Why my evals missed it: every test row was something a person might type. None of them were the strings the app itself sends.
The fix: an exact, trimmed match against the three preset labels skips the classifier. The phrase filter and the grading call still run on them. I added 11 tests (66 to 77). After the deploy, I re-tested on production: each button returns a normal response and writes no flag row, while typed distress still gets flagged at Level 2 and written.
Known gap: if someone hand-types that same sentence, it’s still flagged, because the classifier sees the text without knowing which question was asked. The fix is to pass it a server-stored copy of the question. I haven’t done that yet.
Tradeoff I’m unsure about: any exemption is a bypass surface. Mine is an exact match on three fixed strings, and the other signals still run, but I’d like to hear how others handle it.
Do you run canned UI strings through your safety pipeline, or exempt them? And how do you test that the exemption can’t be abused?
Replies
Great write up, and "none of the test rows were strings the app itself sends" is a lesson worth keeping. One way to remove the bypass surface entirely: do not send the preset as text. Send its id, and let the server look up the label and the question from its own copy. Then the exemption is keyed on an id that the typed path can never carry, so pasting the same sentence by hand goes through the full check, and there is nothing to match against. It also gives you your missing context for free, because the server knows which question the preset answered. My app has the same split between preset texts and typed ones going into a strict filter, and keeping presets as server side records rather than strings from the client is what I would do there too.