I had a model write the plan, then had a second one execute it. Ask me what broke!

Small experiment I ran this week on an open source project of mine. I gave the newest model a real task:

read the project's docs and write me a pre launch plan plus a step by step checklist, where each step has a Findings section to fill in as it goes. It produced 268 lines of plan and 684 lines of checklist, thirteen steps.Then I handed that checklist to a workflow of ten agents on a cheaper model and told them to actually do it.

What happened, honestly:

The first run did nothing. One agent, journal said "started" and that was the whole log. Killed it and resumed, second run finished all ten. It pinned a dependency at version 1.16.0. That version does not exist. Nobody had published it yet, including me.

It left my verification gate failing and wrote that both failures were pre existing bugs in my checkers. That was half true and the half that was false mattered. One of the two failures was a broken link the run itself had introduced, and one of the checkers it blamed had been written during the same run.

It also found two real functional bugs and a contradiction in my own documentation, which I would not have found that week.So the tally is: genuinely useful, and confidently wrong about its own work in a way that would have shipped if I had trusted the summary instead of reading the diff. The failure mode is not that it cannot do the work. It is that it grades itself.

Happy to answer anything about the setup, what I would do differently, or the specific prompts. Also interested if anyone has found a way to make a model's self report on its own run actually trustworthy, because I have not.