agents score 80% on isolated tasks but 38% maintaining a codebase. does that match what you see
by•
reading the end of software engineering paper (arxiv 2606.05608). its own benchmark has agents above 80% on one-off tasks and at most 38% when they maintain a codebase over time, because early mistakes carry forward. writing code got cheap, keeping a system correct didnt. feels like the real job is now writing intent down clearly and checking agent output with a different method than the one that wrote it. are you spending more time writing code or reviewing what agents wrote?
23 views
Replies
My read: the 38% is mostly a review problem, not a generation problem. Treat agent output like work from a strong new hire. Narrow scope, acceptance criteria written before the work starts, and tests the agent didn't write itself. Skip the last one and the early mistakes compound exactly the way the paper describes.
An agent wired a reporting pipeline that looked clean. The report showed a confident number nobody had measured, and it took a full day to trace back that it was just a hardcoded placeholder. Across an existing codebase, they fake integration just to make the run pass.
yes.. but a test only catches what you already thought of, it does nothing the first time the agent tries something new. what caught the most for me wasn't a test, it was a log of what the agent actually ran, so finding out later is a grep instead of guesswork. are you reviewing the diff or what it actually did?
Matches what I see. I've stopped trying to review every line, nobody can at this volume. What works for me is checking with a different method than the one that wrote it: end to end tests that click the real screens, and when a model makes a judgment call, a different model checks it, Gemini and sometimes Codex or Grok too. Writing intent down matters just as much. Every correction becomes a file the next session reads.