If you benchmark your own product against competitors, what would make you believe it?

I built a benchmark that runs nine WordPress backup plugins head to head on identical sites. One of the nine is mine, and it currently sits second. That is an obviously compromised position, so I have spent more time on what makes a vendor-run benchmark believable than on the measuring.

Three things I landed on, and I am not sure any of them are enough.

Publish the runs you threw away. 581 completed runs, 180 scored, 401 excluded, and every excluded one is browsable with its reason attached. Most are boring reruns rather than embarrassing failures, and saying so is part of the point: 360 superseded, 41 excluded for cause. The honest number includes the boring ones.

Say when a lead is not a lead. Mine sits five points above third place with plus or minus 113 error bars. That is a tie, and the page says tie rather than letting me quote the higher number.

Do not report one score. Speed and archive size move against each other, so ranking by one is quietly ranking by the inverse of the other. Both are shown and neither is collapsed into a verdict.

What I am genuinely unsure about is whether any of that is enough, or whether a benchmark run by someone selling one of the entries is simply not credible no matter what it discloses, and the only real answer is handing the method to someone with no stake in it.

So: if you have run or read one of these, what would you need to see before you believed a number published by a vendor? And has anyone here found a disclosure that actually changed their mind?