How to prune training data to fix AI agent hallucinations

We used to add new product documents to our vector database every time a customer asked a tricky question. We thought more information would make our support agent smarter.. What happened was the model started confusing old refund policies with current ones. During checkout it gave users answers that didn’t match the terms—sometimes even contradicting itself.

When we looked into the knowledge base we found something. Three different versions of the shipping guide were active, at the same time. The retrieval system pulled all of them in. The language model made up a hybrid policy based on those conflicting sources. It wasn't just wrong—it cost us a refunds.

We fixed it by setting up a deprecation rule. Now every document must have a review date and a freshness score before it gets ingested. We automated the score to check how recent the content is and we block anything that’s too old or hasn’t been reviewed in time.

After two weeks of cleanup the hallucination rate dropped significantly. The agent started giving consistent answers again.

How often do you audit your agents knowledge sources?. Do you let it ingest everything automatically?