One A/B test, 41 days, 38,000 visitors.

We ran a single A/B test on our pricing page from April 8 to May 19. 38,412 total visitors, split 50/50.

Control: three plans side by side, middle plan highlighted with a "Most popular" badge.
Variant: same three plans, but the middle plan was pre-selected with a toggle and the other two were collapsed into "Compare all plans" links.

Results:

• Control conversion (visit to trial start): 2.8%

• Variant conversion: 4.1%

• Statistical significance: p = 0.003 at day 41

But the number that mattered more: trial-to-paid conversion.

• Control: 19%

• Variant: 11%

The variant got more trials from people who were less qualified. Overall revenue per visitor was actually 6% lower in the variant despite the higher trial rate.

We kept the control. The lesson was that we had been optimizing for the wrong metric for weeks. We now run every pricing test with revenue per visitor as the primary metric, not trial starts.

What test have you run where the "winning" variant lost on the metric that actually mattered?