Mathew Chang

We asked a coding agent fifteen times to make one button blue on a page built to tempt it: Add to Cart and Checkout shared a class, a badge and the nav link shared the accent variable. Every run added an id-scoped rule. Checkout stayed green. Appending "Do not change anything else" changed one thing: the five runs with it wrote the hex inline where nine of the other ten had added two CSS variables. A path rule fencing off every file but the stylesheet never fired.

Real repositories look different. One benchmark pre-applied the fix to 200 issues and reopened them for five recent models in their vendors' harnesses. The right patch is empty; 35 to 65% of instances got an edit to executable code anyway, and in the authors' analysis of one model's failed traces, 87.1% had modified code unrelated to the issue. Task wording moved that figure a long way in both directions. Telling the agent to "edit the codebase" dragged one model's correct-abstention rate to 36.5%; reproduce first, then fix or abstain if nothing was wrong, lifted it to 88.5%. Their explanation: the agent acts on whatever it believes success means.

The harness side: switch on auto-accepted edits in both CLIs whose docs we read and the grant is the whole working directory. A path rule brings that down to files; one vendor's docs state that it applies to the built-in edit tools and recognized shell commands, and that a script opening a file itself is outside it. A selector inside a permitted file sits below every layer's resolution. Whether the blue landed on one button or on every primary button is decided by the sentence you typed and by whoever reads the diff.

When your agent overshoots, where does the boundary live in your setup today: the prompt, a rule file, a path rule, a sandbox, or the person reading the diff? Which of those has caught a change for you inside a file the agent was allowed to edit?

I was testing how LLMs rank products in our space by asking vague buyer questions like "what's the best tool for X." We showed up on Gemini, but ChatGPT kept listing older competitors that haven't updated in years.

It made me realize how inconsistent these AI recommendations are compared to traditional Google SEO.

If you added comparison pages, structured data, or an llms.txt file recently, did you notice any real difference in how models cite you?

One day, we got an email from a user confused as to why our application displayed "Error 403" after trying to invite a teammate. That was all he could tell us. I had written that particular error message almost eight months ago, so I had never even thought about it again since then because I knew what 403 meant.

I decided to go through every single one of the error messages in the product and re-write them to inform the user what had gone wrong and how he could proceed. Support emails about "The app is not working" decreased significantly, despite the bugs in question remaining exactly the same. I knew what the problems were. I just did not see them anymore.