My agent writes the words. A script builds every date, path and JSON field
The weekday kept coming out wrong in my agent. Reading the time from the system was never the issue. It would take that time, compute the day on its own, and get it wrong. So now a hook feeds it the current time along with a two week calendar on every message I send. If a reply contains a date that doesn't line up, the agent has to rewrite it.
From then on, I stopped letting it generate anything that a program would read. Frontmatter, JSON, config values, file paths, URLs, dates. A script handles all of those, and it also decides pass or fail. The model's only job is writing the plain text.
When I genuinely can't avoid handing a decision to a model, I don't trust a single one. Gemini will review whatever Claude came up with. For the larger tasks, Codex and Grok get pulled in too.
Are you still letting your agent pick dates or decide pass or fail anywhere?
Replies
An agent wrote the narrative around pipeline data, and the report ended up with a confident metric nobody had measured. It took a day to realize the model had invented a placeholder to make the prose flow. Splitting words from numbers doesn't stop the text from faking its own facts.
@konstantin_tikhaev Yeah, that's the part a script can't build for you. What helped me was a hook that reads every reply before I see it. If the agent says something is fixed or done without having checked, it has to mark it as unverified and send it again.
@hisashispace That works for task claims. The issue is the model hallucinating plausible background numbers just to make a sentence sound complete.
no.. pass or fail moved out of the model here too, and the same rule for the record of what happened: trial emails get a sent-at stamp on the row, so "did it send" is a fact in the data, not a line the model wrote. the agent never writes the thing it's graded on. when gemini and claude disagree on a review, who breaks the tie, you or a third model?
@neelagiri_chettiyar Not by vote. When Gemini and Claude disagree I pull in Codex and Grok too, and I go by the reasons they give, not how many agree. If the four still split, that usually means it's a real judgement call rather than a bug, and those ones come to me. Same idea as yours really, the model never gets the last word on its own work.
@hisashispace going by the reasons makes sense.. three models agreeing can still be the same mistake three times. same here for the splits, anything the agent can't settle sits held until i look, it can't wave itself through. do you keep the four reviews somewhere or only the verdict?