Connect with the Product Hunt community

Start a thread, join a discussion, stay in the loop.

Start new thread
trending
General

p/general

by

A harness feature that expires when the model improves

In August one coding-agent CLI's release note took its task-tracking tools away from its newest models and kept them for the older ones, with an environment variable to reverse it. The note gives no reason. A component study posted in mid-September is the nearest thing to the arithmetic behind a call like that: a harness part is built to cover for something the model does badly, and the same part does a different job once the model stops doing that thing badly.

The study held an agent loop fixed and swapped parts across four models of different size. Its planning scaffold, a plan re-injected every turn, added 11.6 points on the repository benchmark for the smallest model, whose median run otherwise ended after five turns, two runs in three without an edit. For the largest it trimmed cost by about 30% with a two-point drop in success, because what it cut was verification the model would otherwise keep doing. A predefined tool set showed the same crossover: worth 15 points to the small model on the repository benchmark, while the large one did better there and 53% cheaper with bash alone, bundling several edits into one script.

Our reading: whether a part helps is a question about the model, the task and the budget; what the part costs, in tokens on every turn, is charged whatever the answer. A part that was right when it shipped can be wrong a model generation later without anyone touching it.

We tried the smallest version on our own setup: one small task, one model, fifteen runs, five under each of three tool lists. Every run passed the same four checks. With the task-tracking tools switched back on, the model was offered them in every run and called them zero. With bash alone it finished in a median of 10 turns instead of 17, and the first request carried a quarter of the context. We kept the typed file tools on anyway, because path permission rules attach to them and cannot see inside a shell script.

0

First
Previous
•••
456
•••
Next