Your agent's skills are procedures with the standing of instructions, and that is where they bite

by

Agent skills look like a knowledge feature. Mechanically, they are something else: a paragraph of description that decides whether a block of instructions enters the model's context, and once it enters, it stays for every following turn with the same authority as anything else you told the agent. That design is fine when the skill fits the task. It has a specific failure shape when it almost fits.

An August study put a number on that shape. Researchers paired runs of the same tasks with and without a matched skill, same model, same harness, same verifier, and confirmed by hand 307 cases where the skill made the agent worse. The surprise was where the harm came from. Of 125 functional failures, only 2 were a skill firing on a task it did not suit. 86 were an on-topic skill whose method, default value or example quietly displaced something the task required. One case: the task asked for net exports as a percent of GDP, the skill-guided run delivered the ratio, the reference run delivered the ratio times 100.

The other 182 cases were efficiency regressions, both runs passing but the skill run costing more than double in tokens or time. Length of the skill body explained the smaller share. The larger share was procedure: extra exploration, heavier build pipelines, and 67 cases of the agent continuing to verify after it had already passed, because the skill said to.

The counterweight matters. The study used one harness and one model, and the aggregate benchmark it drew from still shows curated skills lifting pass rates by around 16 points on average. Skills help. The claim here is narrower: a skill's description is the only part the model sees before deciding, and its body is a colleague's runbook that can outrank your ticket.

So read the body before the agent does. Count the steps it prescribes before you count its lines. And when a run passes with a skill loaded, ask what the skill saved, since a run with no skill passes too.

1 view

Add a comment

Replies

Be the first to comment