Safety classifier in the same LLM call as the reply, or a separate call?
I’m designing a content-safety check for an app that generates personal feedback text, and I’m stuck on one tradeoff.
Option A: one call. The model returns JSON with the label as the first key, then the generated text. This saves a round trip and one failure point. The cost is that the classification and the generation share one prompt, so a long or adversarial user message could hurt both at once.
Option B: a separate call. It’s cleaner, and you can run it on a smaller, stricter prompt. The cost is added latency and a second thing that can time out.
Whichever I pick, these rules seem right to me:
Put the label first in the JSON, so it’s produced before the text.
Let cheap rules (regex) only raise the level, never lower it.
If the model is unsure between two levels, take the higher one.
Fail closed: if the check errors or times out, generate nothing.
Check the generated text again before showing it.
I haven’t measured the accuracy gap between A and B yet. Has anyone compared the two on real traffic? Does a separate call catch enough extra to be worth the latency?
Replies
Hold on. If you put the label first in the JSON how do you stream text back to the end?
I'd replay a labeled set of real and adversarial prompts through both end-to-end paths, then compare unsafe outputs that slip through, false positives, timeouts, and p95 latency. Classifier accuracy alone misses what the post-generation check catches.
The label-first field still comes from the same generation step, so I wouldn’t treat it as an independent safety gate.
Have the app enforce an allow or block decision on the exact final text before it reaches the user, even if generation passed its own label check.
I went with B, but run in parallel instead of before. My app generates songs, and the music engine sings whatever it's handed, so every request is screened by a separate small model call that runs at the same time as the generation. The user waits no longer, and if the verdict is "prohibited" the result is thrown away once both are back. That wastes a generation now and then, but it's rare. One place I chose differently: I fail open, a judge that errors lets the song through, because a wrong block on someone's birthday song costs us more than a rare miss. Also worth calibrating on real examples: angry breakup songs and roasts with swearing should pass, hate shouldn't, and a strict prompt blocks the first two more often than you'd expect.