Run the same task eight times and "can it do this" turns into "sometimes"

Two runs of the same request on the same commit, two diffs that do not match, and most of us file it under luck. A leaderboard published this month put a count on that luck, because it prints every trial: eight models, ten tasks, eight runs of each. We tallied the eighty cells. Six passed on every run, thirty-three failed on every run, and forty-one, more than half, landed somewhere between.

Under it are two sources of variation, and they stack. The model hands back a probability for every token and a sampler draws one; temperature reshapes the distribution, and at zero the draw is gone but the runs still differ. Anthropic's API reference says so in one sentence, and its models released after Opus 4.6 will no longer take a temperature value. The rest comes from the serving stack. A research post from Thinking Machines ran one prompt a thousand times at temperature zero and got eighty distinct completions, traced to floating-point sums added in a different order as the batch around your request changes. An agent loop turns that into a different first file to read, and from there the two runs walk different paths.

The figure people quote from a leaderboard is pass@1. On this grid the top row sits at 38.8% for k=1 and about 70% for k=8; a row further down goes from 28.8% to 90% and never solved any task on all eight runs. The k is the part that goes unsaid.

How many runs do you give the same task before you believe the result, and does the count get written down anywhere?

Our own version of this is small. In one A/B on compression, a single instance disagreed with the rest; five reruns per arm gave 3 of 5 against 3 of 5, and one run per arm would have shipped a difference that was not there. We are moving our comparisons to counts: three runs per side per task, one harness, both version numbers recorded, then the counts side by side. When they come out close, the comparison is not finished.