Benchmarks should report the cost of verification

by

A model can produce an answer quickly while moving the real work into checking it.

For any benchmark meant to represent useful work, I want two numbers: time to first answer and time until a human can accept the result with confidence. The gap includes source checking, reruns, manual tests, cleanup, and reversals.

A system that is 30 percent faster at generation but doubles verification cost is not more productive. It has just moved latency out of the headline metric.

What would you include in a practical verification-cost benchmark?

1 view

Add a comment

Replies

Be the first to comment