When you compare two prompts, two models or two agent configs, the result is only as fair as the starting environment. If attempt A ran on a machine that attempt B had already touched (a warm cache, a half-applied migration, a leftover temp file), you're measuring the mess, not the agent.
What we do: build one base sandbox with the project and its dependencies ready, take a snapshot, and fork it so every variant starts from exactly the same running machine. Then each variant runs, gets scored, and the losers are thrown away. kosa8 can fork a live sandbox into several independent copies (about half a second per clone on an Apple M-series Mac in our benchmark harness).
While building kosa8 we kept coming back to the same short list of controls that make it reasonable to let Claude Code, Codex or Cursor run without someone watching every command:
One VM per agent, not a shared container. Containers share the host kernel; code an LLM wrote a minute ago is untrusted by definition.
Network closed by default. Allow-list the model API and your package registries, enforced outside the VM so root inside can't lift it.
No long-lived credentials in reach. If ~/.aws or a .env is mounted, the agent has it.
Undo for the whole machine. Checkpoint before the risky step and rewind afterwards, processes and all.
Sandbox the tools too. Every MCP server is code running with your permissions.
A log you can verify. What did the agent actually run, and can you prove nobody edited the record?
Which of these do you actually do today, and what would you add? Honest answers welcome, including "none of them".
Hey everyone, I'm Drew, building kosa8. We launch here on Tuesday 20 October.
The question behind it: AI coding agents now run with your keys, your data and a shell. Most of us either babysit them or give them less to do than they could handle, because one wrong command or one poisoned web page is all it takes.
