our assistant agreed to something we do not sell, and the customer sent it to print
one of the shops we support does photo prints. someone asked if they also print diplomas. they do not. the assistant gave a vague answer, the customer asked again more directly, and the second time it said yes. they sent the file to print.
we sorted it out and nobody lost money, but the second answer is the part i keep thinking about. the first one was hedged. it only became a yes after the person pushed. so it did not really invent a product, it agreed with a confident customer.
what we changed is that the assistant can only confirm a service if it exists in the shop data. if it is not there it says it will check instead of guessing. that killed the whole class, but only because a printed product is something we can look up. tone or policy questions have no record to check against.
has anyone found a way to stop the agreeing under pressure part in general? that is the one that scares me, because every word of the reply reads fine.
Replies
Pushary
The second answer is the right thing to stare at, and I think the first one is why it happened.
Your hedge was not neutral. It went into the context, so the conversation then held a question plus an answer that visibly did not satisfy. When the customer asked again more directly, that re-ask reads as a report that the previous turn failed. The learned repair for an unsatisfying answer is to be more decisive, and with nothing to check against, more decisive has only one direction available. So the hedge did not delay the yes, it set it up. Which is uncomfortable, because the instinct after an incident like this is to make it hedge harder, and that just manufactures more of the same gradient.
That is also why I do not think this is fixable by instruction, and I would be sceptical of anyone claiming otherwise. A system prompt is just more context competing against live pressure in the same window. The instruction is old and general, the pressure is recent and specific, and over enough turns recent and specific wins.
Your fix is the right shape though, and it generalises further than you are giving it credit for. What you did was turn a generative decision into a retrieval one. The general version is not look everything up, it is separate information from commitment. Turnaround time is information. Yes we can print your diploma is a commitment, because it creates an obligation someone has to honour. Commitments are a small enumerable class and exactly the class where being wrong costs money. Tone and policy feel unfixable because they have no record, but most of them create no obligation either, so they are less dangerous than they feel.
On the general question, there is one signal that needs no record, and you already have the artifact. It hedged at turn one and affirmed at turn three, on the same question, in the same conversation. You do not need to know anything about diplomas to notice that. A position change on a repeated question, moving toward agreement, is machine checkable, and it works for tone and policy too, because it compares the assistant against its own earlier self rather than against a source. Same move you already made, one level up: stop asking the model to judge, and give something outside it a comparison to make.
It is also detectable before the file moves, because there is a turn boundary between the yes and the customer acting on it. Which is the thing I would want to know: when it says it will check, where does that actually go right now?
@aadilghani on where it goes: knowledge base first, then any tool that could actually answer it, and if neither resolves it the conversation is marked needs human. so the promise does leave state behind, it is not just a sentence in the thread.
separate information from commitment is a better way to put it than i had. turnaround time is a fact about the world, yes we can print that is an obligation someone has to honour, and the second list is short enough to write down.
the position change one is what i am going to steal. cheapest version i can see is classifying each turn on the same question as hedge, yes or no, and firing when it moves toward yes. still a model judgement, but a much smaller one than is this true, and it compares the assistant to itself so it needs no source. needs human is exactly where it should fire.
what i cannot tell yet is the false positive rate, because hedge then yes is also what a correct answer looks like when the model finally reads the right record.
Pushary
@jernej_jan_kocica
Needs human being real state rather than a sentence in the thread is what makes the rest of this possible, so that answers my question.
On the false positive, I think it has a clean discriminator and it is yours. A legitimate hedge then yes happens because something entered the context between the two turns, a knowledge base hit or a tool that finally resolved. A sycophantic hedge then yes happens because nothing entered except the customer asking again. So the two cases are not actually alike. They differ by whether the grounding changed, and that is a log check rather than a judgement. Position moved and nothing was retrieved in between is the sycophantic case with almost none of the legitimate one in it. Position moved and a lookup landed in between is your system working, and you suppress it.
Which is your own line from the other thread doing the work. The reviewable unit is the action plus what it was grounded in. You already had the discriminator, it was just pointed at review instead of detection.
It also shrinks the classifier twice, which matters more than its accuracy. It only runs on commitment class turns, and only on the ones where the grounding delta is empty. Everything else never reaches a model judgement at all. So the small judgement you were willing to accept gets smaller and cheaper, and the part that actually decides is mechanical.
The other thing I would change is the question. You are asking for a false positive rate, but the costs here are wildly lopsided. A false positive routes to needs human, which you already have and already staff, and the customer has already been told you will check. A false negative is a diploma at the printer. When the asymmetry is that steep you do not tune for precision, you tune for recall and let people absorb it. So the number that binds is not the rate, it is false positives per day against how many a person can actually look at. Different number, and far easier to measure than accuracy.
Honest limit: all of this needs retrievals logged and attributable to a turn. If the lookup happens inside an opaque call, or the framework writes one flat log without saying which turn caused it, the delta is not computable and you are back to judging.
So does yours attribute a knowledge base hit to the turn that triggered it, or is it flat?
@aadilghani Flat for the knowledge base, attributed for tools.
Tool calls carry the message id of the turn, so that delta is a join we can run today. The knowledge base has no row at all. Retrieval runs inline, top 3 chunks over cosine 0.5, pasted onto the customer's message in the request and never written down. When nothing clears 0.5 it returns an empty string, which is your empty delta, and we throw it away.
So half of it is computable now and the half the diploma came from is blind.
The part I had wrong: I filed this as a logging project. It is one write at one call site.
How often does a real lookup land on your side? If most turns retrieve nothing anyway, empty is everywhere and only the commitment filter is doing the work.
The interesting failure here is not hallucination but compliance. The assistant optimized for satisfying the customer instead of protecting the business. I think trustworthy agents need explicit “permission boundaries” and a habit of saying “I’ll check” when the answer affects a real-world action. Knowing when not to answer may become as important as knowing how to answer.
@stephanie_t0317 Compliance rather than hallucination is the right split, and it is why reading the reply does not catch it. Nothing in the text looks wrong, it is just confident about the wrong thing.
The I'll check version works for us because the list is short. Turnaround, stock, whether we do a thing at all. You can enumerate the questions that create an obligation, you cannot enumerate the ones that need a good tone.
Jernej, on the false positive rate you said you can't tell yet: you can, and without shipping anything. The discriminator Aadil landed on is computable backwards. Take the last few months of conversations, find the ones where the same question got asked twice, and label each pair by whether a retrieval landed between the turns. That gives you both numbers before any of this touches production: how often position moves with an empty grounding delta, and how many of those a person would have had to look at per day.
The second number is the one that decides whether you can actually run it, and it's the one people usually discover after they've built the thing. Backfilling is a day against logs you already have.
On the limit Aadil named, retrievals being attributable to a turn: real, but smaller than a logging project. You only compute the delta on commitment-class turns, and you already have to enumerate those. So what you need is a turn id stamped on lookups in that one path, not a coherent trace across the whole assistant. I've deferred "fix the logging properly" for a year at a time and shipped the narrow version in an afternoon, and the narrow version is the one that gets used.
One thing I'd add to the backfill: also count the cases where the customer asked twice and the assistant held its position. That's your baseline, and nobody has it. If it turns out the model almost never holds under a re-ask, then you're not missing a detector, agreement is just the default, and the volume going to needs-human will be much larger than the diploma case makes it look.
@narek_keshishyan Only half of that backfill runs here. Tool calls are logged against the message id, KB retrieval is not logged at all, so those pairs are unknown rather than empty. Lower bound, and one that flatters us.
Your second one needs no logging though, and I will run it first. Same question twice, did the answer hold. That is in the messages already.
You are right that nobody has it. I have been designing a detector without knowing if the thing is rare or is the default.
If it turns out the assistant almost never holds under a re-ask, is that still a detector problem to you?
Yes! but if the assistant almost never holds under a re-ask, it stops looking like a rare detector case and starts looking like a product boundary. A useful artifact would be the gap between the original question, any new grounding, and the commitment that followed.
That’s close in spirit to why we built Zentrik around keeping the source, decision, intended outcome, and return evidence connected: a plausible sentence isn’t proof.
Yet, if the model moves toward "yes" with no new grounding, route it; if a lookup changed the context, preserve that path. I’d treat the hold-rate as a regression test as you change models or policies, not only as a one-time detector metric.
the commitment-vs-information framing in the reply above is the clearest version of this I've seen, and it maps directly onto a problem we have running voice agents. in chat there's at least a turn boundary between the yes and the customer acting on it, the file still has to get sent to print. on a live call there often isn't one - the customer says "great, see you Tuesday at 3" and hangs up, and the commitment and the action are the same utterance. by the time you'd detect the position-change pattern (hedge at turn one, confident yes at turn three) the call is already over and the thing is already booked. so the detector has to fire mid-utterance, before the agent finishes speaking the affirming turn, not after the conversation ends. has anyone actually tried scoring that kind of pattern in real time on audio rather than post-call on a transcript, or is the latency just not there yet to interrupt an agent mid-sentence based on "this looks like it's about to become a commitment"
@galdayan We run voice on the same pipeline as chat. I would not score the audio.
The text exists before the audio does. We cut the reply into chunks and hand each to TTS as it completes, first sentence terminator, clause break after ~12 chars, hard wrap at 140. So "yes, we can do that," goes to TTS on its own while the rest is still generating. That boundary is your seam, and there is already a gate on it.
One shot though, you cannot un-say a streamed chunk, so it can only sit on the first chunk of a commitment turn. And it needs a bounded worst case. I did real time control before this, and a check with no bound is not a check, it is a new failure. A stalled scorer is dead air.
Do you get a chunk boundary you can hold, or is TTS streaming straight off the token stream?
@jernej_jan_kocica it's asymmetric, not held everywhere. most of the reply streams token to TTS as soon as it's generated, latency wins by default. the exception is anything that reads like a commitment, price, availability, "yes we can do that": those get pulled into a hold where the full clause has to finish generating before it goes to TTS, even though that costs a couple hundred ms in that turn. so the delay isn't global, it's applied by classifying what's about to be said before it's said. which loops back to your point above, if that classifier misses and tags a commitment as a normal turn, you're back to the same seam, just one layer earlier.
@galdayan Classify the question, not the prefix. The answer is half generated when you have to decide, the question was whole and it arrived earlier. Do you also print diplomas is commitment class before the agent says a word. Arm the hold there and a miss costs a couple hundred ms on a safe turn instead of a booked Tuesday.
It misses the agent volunteering something nobody asked for. That one still needs your output side check.
Do you get the final transcript early enough to arm before generation starts?
@jernej_jan_kocica we get partials continuously but arming on a partial is risky, "do you also print dip" reads differently than "do you also print diplomas" and the classifier can flip mid-word. what we actually do is arm provisionally once the partial has held stable across two ASR updates in a row, generation only starts after that. if the eventual final transcript disagrees with what was armed, we discard whatever's generated so far and restart the turn. so the transcript is early enough to arm on, just not early enough to trust outright, the trust comes from a stability check on the partials rather than waiting for the finalized version.
The re-ask reading as a report that the previous turn failed is the sharpest description of this I've seen. It didn't invent a product, it read dissatisfaction and corrected toward whatever would settle it.
Two things I'd add from running one on support.
Retrieval fixes what it knows. It doesn't fix what it's willing to promise. So I'd check on the way out as well as on the way in. Commitments being a small enumerable class cuts both ways: if you can enumerate them, you can spot them in the draft. Anything that reads as yes we can, we'll have it by, we do offer, gets held for a human even when the retrieval looked clean. Cheap to build, and it catches the case where the lookup was confidently about the wrong document.
Second, the same question asked twice looks like a routing signal in its own right. Not more effort on answer two, less. Handing over on a re-ask rather than trying harder seems the safer default, because on the evidence answer two is the one most likely to be wrong.
Glad it cost you nothing but a reprint.