How do you stop an AI feature from making unauthorized promises to users?
by•
When AI drafts replies to real customer messages, one hallucinated discount or freebie turns into an instant headache. Right now I keep strict negative rules in the system prompt and force a manual human approval step before anything gets sent. For those shipping customer-facing AI in production, do you trust prompt guardrails on their own or still require human sign-off?
104 views
p/runevr
p/producthunt
p/google-cloud-platform
p/jot-2-0
p/beamfy
p/ai-context-flow
p/refs
p/dex-by-exmergo
p/reso
p/kodlo
p/duckfightclub
p/seshka
p/talanoa-2
p/nudgeforme
p/nexupira
p/many-displays
p/dino-kids
p/murror
p/operum
p/tabstack
Replies
We keep the human sign off too, and also moved the promise itself out of the model's reach. Our support drafts come from a model that works from a facts file about the product, in the language the user wrote in. Credits are not something the draft can invent: the person answering sets the amount in a separate field, the draft is generated with that number passed in, and the grant happens in the same transaction that records the sent reply, keyed by a reply id so a retried send never grants twice. So a made up freebie can only exist as words in the draft, and a person edits those before anything goes out.
@gbasovv Moving the actual commitment out of the model's reach—and tying the grant idempotently to the sent reply_id—is a super clean architectural pattern.
Grounding the draft strictly in a structured facts file (and matching the customer's language while letting the human review and edit the exact text) hits the sweet spot: AI handles the repetitive phrasing, while the human owns both the financial commitment and the final send. Thanks for breaking down how you structured that transaction!
Mid-task the agent lost earlier context and rebuilt a helper that already existed two files away with a different name. When models drop context, they never ask, they invent a plausible duplicate to bridge the gap. That silent amnesia is where unauthorized commitments come from.
@konstantin_tikhaev "Silent amnesia" is the exact right term for it. When context gets diluted across a big prompt, the model defaults to being overly helpful—inventing a plausible discount or refund just to smooth things over.
That’s actually why I architected a 7th layer in my context injection that sits completely isolated from everything else. While the first 6 layers handle tone, SEO, and business identity, this standalone layer holds only the owner's priority forbidden rules (custom banned words, phrases, and off-limit offers). Keeping it strictly separated ensures those hard boundaries override everything else and never get lost in the noise—backed up by mandatory human sign-off before anything hits the Google API.
@fab_lp An isolated prompt layer still relies on attention weights holding under load. If the boundary is business critical, put it in deterministic code or an API interceptor, not in the prompt.
@konstantin_tikhaev You’re 100% right—a prompt layer is still probabilistic at the end of the day, and attention weights can drift.
Combining your point with our 7th layer actually gives us a super clean deterministic loop: we can run a code-level interceptor against the owner's forbidden-words list right after generation, and if a violation is caught, automatically bounce the draft back to the LLM for a targeted rewrite before it ever reaches the review UI—while still keeping mandatory human sign-off before hitting the Google API.
Definitely building that interceptor into the next sprint. Appreciate the push!
@fab_lp Keyword interceptors miss semantic commitments. A model can promise a concession with completely different phrasing, and bouncing it back just rerolls the dice.
Keep the human sign-off. Prompt rules lower the rate of bad promises, they don't remove them. What helped us: the AI drafts, then stops at a clear "not sent" state, and the person approves the exact text that goes out. Checking the recipient against what the user actually typed caught a few surprises too.
@geekynaman 100% agree on that—prompt rules reduce the frequency, not the worst-case risk.
That "hard stop before sending" is the exact pattern I settled on too: the AI drafts the response (with a live translation back into the owner's language when needed), and nothing ever goes live until a human explicitly approves the final text.
Good point on verifying the recipient/context match too—subtle mismatches are often way harder to catch than obvious hallucinations. Appreciate you sharing what worked for your team!
I still require sign off, and I don't trust the prompt on its own. Customer emails my agents write stay as drafts until I send them myself. On top of that, hooks check banned words and dates against the real calendar before a draft even reaches me.
@hisashispace Adding deterministic pre-checks before the draft even reaches your screen—especially verifying dates against a real calendar and catching banned words—is such a smart layer.
It saves the human reviewer from having to catch obvious factual slips manually, while still keeping the final send strictly in human hands. Great defense-in-depth setup!
we took prices out of the model's hands completely. it can write the whole page but the price field stays empty until a person types it. prompt rules alone never got us to zero
@farzam_habibi That’s a really smart pattern. Separating creative generation from deterministic human-filled fields is probably the cleanest way to guarantee zero hallucinations on critical numbers.
In our case (drafting Google review replies), the risk isn't a single price field—it's the AI casually slipping a "come back for a free coffee" into a paragraph. Since we can't leave a blank slot for tone, we keep human sign-off mandatory AND built a feedback loop: whenever an owner catches an unwanted phrase or hallucination during review, they add it directly to their isolated "priority forbidden rules" layer so the AI never repeats that mistake again.
Love the "never got us to zero" takeaway—that matches what I'm seeing in production too.
@fab_lp a forbidden list that grows from real catches is a nice touch. curious if owners keep adding to it after the first week or if it settles down
moving commitments into code instead of the prompt is the part that convinced me. prompts drift, code doesn't. same thing happened in my ui work, rules in a file plus a check that fails is what stopped the agent going off system. does your interceptor block the reply or rewrite it?
@uinerd It blocks automatic publishing and hands it to the human—no silent auto-rewrite loop.
As @konstantin_tikhaev rightly pointed out above, auto-rewriting just rerolls the probabilistic dice and can still miss subtle semantic phrasing. So any automated check only acts as an early flag, while the actual hard stop is in code: the draft stays locked in a pending state until a human reviews (and edits if needed) the exact text—paired with a native-language back-translation—before triggering the Google API call.
Love your pattern of "rules in a file + a check that fails"—keeping constraints deterministic in code instead of relying on prompt discipline is the only way we've seen agents stay predictable in production!
The agent lost context mid-task and improvised instead of failing, silently rebuilding a helper that already existed two files away with a different name. Injecting an immutable symbol index into each step fixed it rather than letting it rely on conversation history.