How do you find the hallucinations you didn’t know to test for?

by•

I have been having this conversation several times last couple weeks. How do you catch hallucinations or drifts you don't think can happen?

Evals are easy when failure options are known, but what if you don't know - is the only way waiting for the output?

Spec-driven also help but again this is before and not during when the control already is handover to the AI

I also see this on the vibe coded apps - while I think is great to be able to spin up something quick, many times some basic product construction frameworks are left behind (e.g. testing)

46 views

Add a comment

Replies

Best

Mid-task an agent lost context and rebuilt a helper that already existed two files away, just with a different name. Unit tests passed because both worked. It was only caught by checking git diffs for structural duplication across the repo, because functional tests were blind to it.

 yep! I have seen that also on end-user applications for example, creating google docs - let the tokens burn :/ we are still on this token maxxing but doesn't seem sustainable atm

 The problem is not the wasted compute. Traditional tests cannot detect an agent quietly bloating the architecture with parallel implementations.

The ones that got me hid in data I had assumed was ground truth. Our music model returns lyrics alongside the audio, and I showed that text as what was sung. When we checked three songs against the audio, every one had drifted: a line dropped, a line sung twice, two chorus lines merged with a word lost. The text was the plan, not the performance. Now a second model listens to the finished track and maps each sung line to a planned line or writes what it actually heard. That check had its own surprise later: a tester saw the wrong name in their song, because a blurry name got written as heard. So the rule became that a person's name stays as requested unless the singer clearly sang a different one. Neither would have been on an eval list. Both were found by comparing the output to something independent of the model, the audio itself and a person reading the result.

 Would have had a significant impact to find it while the agent is listening to the track instead of at the end? time, tokens?

 Good question. In our case there is no "during": the music engines hand back the finished track in one piece, so the check can only start once the audio exists. It runs alongside the title and cover, which are made after the song anyway, so it adds little to the wait. Tokens were the bigger lever: capping the model's thinking took it from about 80 s to about 24 s. And if the check fails, the planned text stays, so it never blocks the song.

what kinda works for us is logging every output in prod and having a 2nd cheap model flag stuff that doesnt match the input, then reading those by hand. you still find the weird ones after the fact but its days not months

 for sure, the time to find them reduces, but the amount of them increases as the speed of production/building increases, I wonder if compounded this is still weeks (: