If your agent checks its own work by looking at a screenshot and reading the tool's status line, a recent CAD experiment by ModelRift is worth 3 minutes of your time. ModelRift gave six agents the same three printable parts, three in CadQuery and three in OpenSCAD, one agent per cell, unattended, capped at twelve versions, and then checked every exported mesh with a parser that took neither tool's word.

All six parts came out printable. The clearest case on the way there was a threaded adapter whose core cylinder a boolean union silently dropped. The agent read four renders and saw a finished thread. The kernel reported valid=True, solids=1. A volume of 7,065 mm against an expected 10,323 caught it. That was one of sixteen failures across the six runs. Nine of them the tool never mentioned, and the renders caught only the coarse ones, like four mounting posts deleted by a cavity subtraction. Every defect that would have ruined a print was found by a number.

Loosening a tolerance produced a negative-volume solid that also reported valid. The other tool certified an export as error-free while it carried four non-manifold edges and sixty zero-area triangles.

This is really a coding-agent problem, just with a 3D mesh instead of code. An agent gets its feedback as a picture, as the tool's verdict, or as a number produced by something that did not build the artifact. A picture checks what a picture can carry, and the agent picks the picture. The tool's verdict is the maker grading its own work. The number from outside was the one that failed the broken part.

Two turns, same model, same afternoon. In one, the agent rewrites a 400-line test file and the text pours out. In the other, it changes a single line in a config file and nothing appears for ten seconds. Most of us file the second one under "the model is slow today". It usually isn't the model.

A turn has more than one clock. Before the first character, the model reads the whole context again: system prompt, tools, transcript, every pasted file. For a coding agent that context grows every turn, so the pause before the first character is mostly about whether the prompt cache held. A compaction, an edited system prompt, a changed tool list, or a gap longer than the cache's lifetime means the prefix is read from the top.

Once the typing starts, generation is serial: one forward pass per token. That's why a long rewrite is expensive whatever its difficulty, and why both major API vendors now sell a faster lane, around 2.5x output tokens per second, one of them explicitly not faster to the first token, the other explicitly "more consistent". The standard lane's speed depends on load.

Then there's the part that surprised us. A write-up on the vLLM blog measured speculative decoding, where a small draft component guesses the next few tokens and the real model verifies them in one pass. Acceptance falls with position and sits lower on code: 94% for the first drafted token on math problems, 89% on Python, 66% and 49% by the fifth. The same model types faster on text it can guess. Boilerplate is guessable. New logic is not. Their numbers, not ours, and whether the hosted lanes work this way is undisclosed.

Agent skills look like a knowledge feature. Mechanically, they are something else: a paragraph of description that decides whether a block of instructions enters the model's context, and once it enters, it stays for every following turn with the same authority as anything else you told the agent. That design is fine when the skill fits the task. It has a specific failure shape when it almost fits.

An August study put a number on that shape. Researchers paired runs of the same tasks with and without a matched skill, same model, same harness, same verifier, and confirmed by hand 307 cases where the skill made the agent worse. The surprise was where the harm came from. Of 125 functional failures, only 2 were a skill firing on a task it did not suit. 86 were an on-topic skill whose method, default value or example quietly displaced something the task required. One case: the task asked for net exports as a percent of GDP, the skill-guided run delivered the ratio, the reference run delivered the ratio times 100.

The other 182 cases were efficiency regressions, both runs passing but the skill run costing more than double in tokens or time. Length of the skill body explained the smaller share. The larger share was procedure: extra exploration, heavier build pipelines, and 67 cases of the agent continuing to verify after it had already passed, because the skill said to.

The counterweight matters. The study used one harness and one model, and the aggregate benchmark it drew from still shows curated skills lifting pass rates by around 16 points on average. Skills help. The claim here is narrower: a skill's description is the only part the model sees before deciding, and its body is a colleague's runbook that can outrank your ticket.

The API underneath is stateless. Every single turn, the tool you're using compiles a fresh document and sends the whole thing: its own system prompt, every tool definition, your project rules, an edited version of the session history, and your newest message at the very bottom. The chat window you scroll is a rendering for your benefit. The model reads the compiled document, top to bottom, every time it answers.

"Edited" is the important word. As sessions grow, old tool results get cleared and replaced with placeholders. When the context fills up, the whole history gets replaced by a summary of itself, and what survives that summary is a judgment call. Meanwhile your client keeps the full, unmodified history. So the transcript you see and the context the model reads are two different documents, and they drift further apart the longer you work.

Once you know this, a few habits follow. When the agent acts like it never saw something you said, check whether that sentence survived into the current context before blaming the model; often it got summarized away. Put anything load-bearing in a rules file, because rules files get re-sent near the top of every compile, while a chat message is one compaction away from becoming a clause in a summary. And after a compaction, re-state the constraints that matter in one line. It feels redundant. It works.

The uncomfortable part: the model can't audit its own briefing. It has no original to compare against. The only one holding the unabridged session is you.

Here's a debugging pattern worth stealing: when your AI coding agent suddenly feels different, before blaming the model, ask which of its two version numbers actually moved.

Because what you call "the agent" is a pair. The model itself never executes anything it reads what it's given and proposes. The program wrapped around it (the harness) does everything else: assembles what the model sees each turn (its own system prompt, every tool definition, your rules files, the trimmed conversation), runs whatever the model proposes, handles the errors and retries, manages what stays in the context window as the session grows, and decides when the loop stops. One message from you can be forty laps around that loop, and every design decision inside it was made by the tool, before you typed a word.

That layer moves real outcomes. One major provider's blog on agent harnesses says plainly that even a frontier model, run in a loop with nothing but a high-level prompt, falls short of production-quality work; the difference is the structure around it. And academic work on agent scaffolds keeps finding the same thing: identical model weights land at meaningfully different resolve rates and wildly different token bills depending on the loop they're wearing.

Two habits that follow. First, log both versions: harnesses ship far more often than models, so when behavior drifts, the tool is usually the prime suspect: a changelog line like "improved tool descriptions" quietly rewrote what your model reads every turn. Second, only compare models inside the same tool. A model that looks smarter elsewhere may just be wearing a better loop, and cross-tool comparisons measure the pair.

An agent's competence isn't a single number. It's a map of where the internet wrote the most code, and the terrain drops fast. The same agent that produces a clean React table on the first try will confidently call a method on your niche framework that has never existed in any version. No hesitation, no hedging. The confidence gauge barely moves while the competence falls off a cliff.

The research backs the shape. Benchmark work that translated coding tests across 19 languages was built exactly to probe how language frequency in training data drives performance, and a survey in ACM's TOSEM journal calls the low-resource gap a significant challenge affecting millions of developers. There's a second, sneakier edge too: time. Even the most popular framework is a niche framework at the version boundary, because the model knows your stack as of its training cutoff. One benchmark (CodeUpdateArena) found that for the open models it tested, even pasting the docs of an API change didn't reliably make the model reason with the change.

What actually pulls the floor up, in rough order: a working example in your repo (models imitate a real snippet more reliably than they apply a doc page). Version pins stated as hard rules in the prompt, because the model's prior leans toward whatever version dominated its training data. Docs in context, dosed carefully they help with facts and cost tokens every turn. And tests that fail loudly, because a red test is the only channel through which the agent ever learns it invented an API.

Should this pick your stack? It's a real input now for greenfield projects with heavy agent use, where boring mainstream choices pay off twice. But domain fit still beats corpus fit: a framework that halves your problem's complexity is worth the agent tax. Scaffolding recovers a lot of the gap. Nothing recovers a wrong abstraction.

As of this month, the flagship coding models ship a million-token context window by default, and at least one major vendor now prices it flat: no beta flag, no surcharge above 200k a 900k request bills at the same per-token rate as a 9k one. Which raises a fair question. My repo is 700k tokens, so why is this thing still running grep like it's 2024?

Two reasons, and both are measurable.

First, the meter. An agent re-sends its whole history as input on every turn: system prompt, tools, conversation, and anything you pasted. A 700k-token repo at $5 per million input tokens costs about $3.50 per turn. Forty turns into a working session, the paste alone has billed around $140, before the model has written a line of code. Prompt caching cuts the re-read to a tenth of the price, but it just makes those tokens cheaper they still fill the window.

Second, reading quality drops long before the window is full. The lost-in-the-middle effect still holds: models read the start and end of a long context better than the center. The NoLiMa benchmark (ICML 2025) removed literal word overlap between question and content and 10 of 12 models fell below half their short-context baseline by 32k tokens. One model advertising 2M tokens of context held its quality bar to about 2k. Newer models do better. The direction hasn't changed. Even the vendor's own docs now warn that "more context isn't automatically better" and give the failure mode a name, context rot. The company charging per token is telling you to send fewer tokens.

there's a bug that sat in flask for 16 years: template extension matching was case sensitive, so page.HTML silently skipped autoescaping. the fix that finally landed this year is one line. no test upstream figured it was too small to need one.

that made it a perfect lab rat. i reverted the fix in a clone and handed the bug to a coding agent 9 times, three prompt styles, three runs each: a proper bug report, the same report plus "keep the change minimal, don't touch anything unrelated", and a vague one ("some of my templates arent getting autoescaped, can you find and fix it"). headless, auto-approve everything, measure the diff against the base commit afterwards and run the full 491-test suite.

first attempt got thrown out entirely btw. the clone still had git history, and the vague run just... diffed against main, found the upstream fix one commit ahead, and copied it. word for word, docstring included. had to delete the remote, the branches and the reflog and gc the object store before the runs meant anything.

clean-room results: 9/9 fixed it correctly, full suite green every time, and six runs produced character-for-character the same line upstream wrote. no drive-by refactoring anywhere, which honestly wasn't what i expected going in.

I ran a small experiment this week that changed how I think about instruction files (AGENTS.md, CLAUDE.md, whatever your tool reads).

Setup: fresh clone of Flask, one question a real session asks all the time: "what exact command does CI use to run tests, and what runs type checks." The true answer has three non-default flags plus an env var, so the agent either digs through the CI workflow or gets told. Five configs, two runs each: no file, a lean ~950-byte file with the commands, an 86KB file with the same commands buried under a long architecture overview, then the lean and bloated versions again under the harness's own preferred filename instead of AGENTS.md.

Finding 1: my harness silently ignored AGENTS.md. The AGENTS.md lanes were identical to having no file, differences smaller than run-to-run noise. The vendor is listed on the format's site as a supporting tool. I double-checked with a no-tools probe (ask what the instruction file says, forbid file reads): standard name got "UNKNOWN", the tool's own filename got the file quoted back verbatim. So before you polish a single line, check your tool actually loads the thing.

Finding 2: when the lean file WAS read, it saved a third of total input tokens. The 950 bytes themselves are basically free; the saving came from a whole turn never happening: the agent answered from the file instead of excavating CI config, and an avoided turn means the entire conversation-so-far doesn't get re-sent again. The unit of savings is the turn.