Arash Rahimi

We have been building since late 2024. In that time we were a gen AI agency, then a voice AI company, then a procurement tool, before landing on the thing that actually worked. Every one of those pivots was decided by the same wrong diagnosis. Something was not selling, we concluded the product was not good enough yet, and we went and built more. It is a very comfortable conclusion because building is the part you control and the part you enjoy. It also has a satisfying story attached: not there yet, nearly there, one more release. What was actually happening both times was that nobody knew we existed, and the people who did know had no reason to try it this week rather than some other week. That is not a capability problem and no amount of shipping fixes it. The way we eventually found the real thing was not analysis. We ran the catalogue on ourselves, and the one item we kept reaching for was the cheapest thing on the list, the one we had been treating as a feature to sell the expensive stuff around. Two things I would tell myself at the start. First, if a customer signs and then does not use it, that is not a customer, that is a very expensive compliment, and it should be counted as churn on the day it happens rather than revenue. Second, when something is not selling, force yourself to write down the distribution version of the explanation before you are allowed to write down the product version. The product version will still be there afterwards. It just should not get to go first. Curious how many people here have made the same call. When something stalled, did you build or did you go and find out why nobody came back?

LLM as judge has quietly become the default way people check agent output, and almost nobody states which model is doing the judging. In most setups it is the same one that produced the answer, or a sibling from the same family. That arrangement fails in a specific and predictable way. The judge does not catch the hallucination because it finds the same thing plausible, for the same reasons, out of the same training. You have not added a check. You have asked the same mind twice and written down that it agreed with itself. Errors that are correlated between producer and judge are exactly the errors that survive, and they are also the ones you most need caught, because a plausible wrong answer is the only kind that ships. There is a second version of this that shows up inside agent loops. A builder agent grading its own work grades against its own model of what "done" means, which is the same model that generated the work, so it cannot see the gap by construction. And the moment your tester can read the builder's own claim of success, it anchors on it. Two things that seem to hold. Run the judge on a different model family than the producer, even when the judge is the weaker model, because independence beats capability for that job. And give the judge the artifact and the original spec only. No producer reasoning, no summary, no "I have verified this works". The reason people do not do this is cost and latency, and those are real. But a judge that agrees with the producer for free is not cheaper than no judge. It is more expensive, because it manufactures confidence you did not have before. What is your judge running on, and did you pick it on purpose?

There is a good discussion running elsewhere on this platform about when an agent should stop and ask permission. The consensus has moved from "can this be undone" to "how far does this reach", which is a real improvement. Irreversibility over-fires on writing a log line and under-fires on a migration you can only roll back with downtime. Reach is the better axis. But every gate in that conversation, and every gate I have seen shipped, evaluates a single action in isolation. Sending one email to one person is low reach. It passes. Four hundred of them inside twenty minutes is a mass email to your entire list, assembled one approved step at a time, and not one individual step ever looked like the thing you were guarding against. The reach was never in the action. It was in the rate. So the gate wants a running total inside a window, not a verdict on the action in front of it. And approvals need a clock on them, because right now an approval is treated as a standing licence. You approved "send this email". The agent reads that as permission to send email. If it comes back to the same signed-off action ten minutes later it should have to ask again, because what it is doing now is a different act from the one you looked at. The uncomfortable version: most of the agent incidents I have heard described were not one bad decision. They were one acceptable decision executed a few hundred times while everybody watched the individual approvals go past and felt fine about each one. Has anyone here actually shipped a rate-aware gate rather than a per-action one? Curious what you keyed the window on, and whether it survived contact with a legitimately bursty workload.