Kopai
The Cloud for AI Agents
Start new thread

How to stop your Agents from Hallucinating?

by
•

One of the main goals of Kopai, is to make sure that the LLMs do not hallucinate. That is to stay , the human and the AI are always in Sync. Kopai has a feature called Chat Orientation where the user is shown the goals and actions taken by the LLM. If they LLM drifts you can manually update to realign the LLM.

Let us know what you would like to incorporate in this and how we can make this better?

139 views

Add a comment

Replies

Best

the panel review + human sign-off before publish makes sense for catching a bad agent at birth, but what happens after that. if the creator doesn't touch the config for a few months but the underlying model provider ships an update, or the knowledge base source content quietly goes stale, does Chat Orientation catch that drift on its own, or does it only surface goal-state disagreement when the creator happens to be watching a live chat? asking because the failure mode I'd worry about isn't a bad agent at launch, it's a good agent that quietly stops being the agent that got the score.

 The drift to be honest was something we were hoping to keep the human in the loop, because the end goal was for the humans domain expertise to be showcased.However, the stale agent or as was talking about a decay score is something that we would want to address as soon as possible and that was honestly something we did not think about.

   glad it's on the radar. if it helps prioritize: the decay score and rabnoor's delta-view feel like the same fix from two angles - one is "flag it automatically over time," the other is "make the flag legible when a human is actually looking." you probably want both eventually, but the delta-view is the cheaper one to ship first since it doesn't need new scoring infra, and it'd make whatever decay signal you build later actually actionable instead of just another number on a dashboard nobody checks.

Chat Orientation puts a human in the loop as the drift detector, and that only pays if the human actually notices. A plausible drift reads exactly like correct work to anyone skimming. Showing the goal list gets you a nod, not a catch. So I would not show goals. Show the delta. You asked for X, I am now optimising for Y, and here is the step where that changed. A person has nothing to react to in a list and something to react to in a diff. Separate thing on LLM as judge, since you mentioned it. If the judge and the producer share a base model, they fail in correlated ways. The judge does not catch the hallucination because it finds the same thing plausible, for the same reasons. Worth running the judge on a different model family than whatever produced the answer, even if the judge is the weaker model. Independence beats capability for that particular job. What is the judge running on today?

 It's a panel, GPT and DeepSeek, deliberately cross-provider for exactly the reason you're pointing at. Your correlated-failure case only bites when the judge shares the producer's base and misses the same thing the same way. With two families on the bench, whatever the producer runs on, at least one panelist is off-family and reading it from a different failure surface , the independent one catches what a same-family judge would wave through. However, you do raise an important question. Currently it does clear by majority but maybe in the future a flag from that off-family judge on the classes block the agent regardless of the other votes. That's the part that matters, independence is worthless if the outvoted-but-correct judge can't actually stop the thing. So yes, independence beats capability here, and that's the reason the bench is cross-family instead of one stronger model.

   the "majority vote by default, escalate on off-family dissent" design is the right shape for this. one thing worth stress-testing before it ships: does the off-family panelist actually get to see the full trace, or just the final answer? if it's only scoring the output, you can still get correlated blind spots on anything that looks right on the surface but is wrong for a reason baked into how the question was framed upstream, before any model even touched it.

   The trace point is the one that matters, and it goes further than the panel does. If the framing upstream is wrong, every panelist inherits it, off-family or not. Cross-provider buys independence in how they reason, not independence in what they were asked. The only panelist that catches a bad frame is one that sees the original request rather than the cleaned-up question the system built from it. That is awkward to wire, because the raw request is the messiest thing in the pipeline and the first thing that gets normalised away. Also, thanks for the sequencing call. Cheaper one first, and it makes the later signal actionable instead of another number on a dashboard. I had those as two separate ideas and you turned them into one plan.

 Two families on the bench does answer it, and you had already built the thing I was worried about, so that one is on me for not asking first. The flag idea is the important half though. Majority is the rule that quietly cancels the panel. Two same-family panelists outvote the off-family one in exactly the case you assembled the panel to catch, so off-family dissent is not one vote against two, it is the only reading you did not already have. Building on what Gal raised, that panelist also needs the trace and ideally the original request, because a bad frame upstream gets inherited by everyone on the bench regardless of provider. What does the off-family panelist see today, the full trace or just the final answer?

   the cheap version of "raw request" doesn't require reconstructing anything after the fact, it just means never letting the cleaned-up version overwrite the original. attach the verbatim input as an immutable field alongside the parsed intent instead of replacing it, and the off-family panelist can diff its own reading of the raw text against the frame the pipeline settled on. the two only need to disagree occasionally for this to earn its keep, most of the time they'll converge and it's free. the expensive version is teaching the panelist to notice a frame is wrong when it reads perfectly reasonable on its own, which is a harder problem than access.

   Never letting the cleaned version overwrite the original is the right primitive and it is cheaper than what I was describing. Keep the verbatim input immutable alongside the parsed intent, and the disagreement becomes a diff rather than a reconstruction. The part I would underline for anyone implementing it: the value is entirely in the rare disagreement, so the temptation will be to drop the raw field once it agrees ninety-eight times running. That is exactly when it stops being insurance. Same shape as keeping an as-of timestamp on a conclusion, which someone made to me on another thread today and which I have not stopped thinking about. The general rule seems to be: never let a derived value be the only surviving copy of the thing it was derived from.