Comparing two agent prompts fairly: how do you keep the starting state identical?
When you compare two prompts, two models or two agent configs, the result is only as fair as the starting environment. If attempt A ran on a machine that attempt B had already touched (a warm cache, a half-applied migration, a leftover temp file), you're measuring the mess, not the agent.
What we do: build one base sandbox with the project and its dependencies ready, take a snapshot, and fork it so every variant starts from exactly the same running machine. Then each variant runs, gets scored, and the losers are thrown away. kosa8 can fork a live sandbox into several independent copies (about half a second per clone on an Apple M-series Mac in our benchmark harness).
How do you handle this today? Fresh containers per run, git reset between attempts, or just accept some drift?
Replies
Be the first to reply
Have a question or a thought to share? Add a comment above to start the conversation.