RunAI Coder

Two runs of the same request on the same commit, two diffs that do not match, and most of us file it under luck. A leaderboard published this month put a count on that luck, because it prints every trial: eight models, ten tasks, eight runs of each. We tallied the eighty cells. Six passed on every run, thirty-three failed on every run, and forty-one, more than half, landed somewhere between.

Under it are two sources of variation, and they stack. The model hands back a probability for every token and a sampler draws one; temperature reshapes the distribution, and at zero the draw is gone but the runs still differ. Anthropic's API reference says so in one sentence, and its models released after Opus 4.6 will no longer take a temperature value. The rest comes from the serving stack. A research post from Thinking Machines ran one prompt a thousand times at temperature zero and got eighty distinct completions, traced to floating-point sums added in a different order as the batch around your request changes. An agent loop turns that into a different first file to read, and from there the two runs walk different paths.

The figure people quote from a leaderboard is pass@1. On this grid the top row sits at 38.8% for k=1 and about 70% for k=8; a row further down goes from 28.8% to 90% and never solved any task on all eight runs. The k is the part that goes unsaid.

How many runs do you give the same task before you believe the result, and does the count get written down anywhere?

If your agent checks its own work by looking at a screenshot and reading the tool's status line, a recent CAD experiment by ModelRift is worth 3 minutes of your time. ModelRift gave six agents the same three printable parts, three in CadQuery and three in OpenSCAD, one agent per cell, unattended, capped at twelve versions, and then checked every exported mesh with a parser that took neither tool's word.

All six parts came out printable. The clearest case on the way there was a threaded adapter whose core cylinder a boolean union silently dropped. The agent read four renders and saw a finished thread. The kernel reported valid=True, solids=1. A volume of 7,065 mm against an expected 10,323 caught it. That was one of sixteen failures across the six runs. Nine of them the tool never mentioned, and the renders caught only the coarse ones, like four mounting posts deleted by a cavity subtraction. Every defect that would have ruined a print was found by a number.

Loosening a tolerance produced a negative-volume solid that also reported valid. The other tool certified an export as error-free while it carried four non-manifold edges and sixty zero-area triangles.

This is really a coding-agent problem, just with a 3D mesh instead of code. An agent gets its feedback as a picture, as the tool's verdict, or as a number produced by something that did not build the artifact. A picture checks what a picture can carry, and the agent picks the picture. The tool's verdict is the maker grading its own work. The number from outside was the one that failed the broken part.

We read both vendors' documentation and the recent work on looped transformers with three questions: what does a model's reasoning cost, what can you steer, and what can anyone read afterwards. The answers sort into three places.

On the page, as visible text. You pay the output rate, you steer it with the prompt, and you can read it, though studies show these visible traces are often unreliable.

Off the page, as hidden tokens. The example in OpenAI's own guide shows 1,186 output tokens charged, 1,024 of them reasoning; you pay for every one and the text stays with the vendor. Anthropic's docs show the same: you're billed for full thinking tokens whether shown or not. The rate is the same, the steering is an effort knob in place of a prompt, and what you can read is at most a summary. The exact count is in the usage object, and on the newer models earlier turns' hidden reasoning stays in context and is billed again as input.

Inside the network, as extra passes through the same layers, which is what recurrent depth means. Compute rises per token, the token count does not move, and there is no text for anyone to read, the vendor included.