How do you evaluate AI models for production use?

While building an AI-powered product, we tested several leading models. We quickly realized that benchmark scores don't always reflect real-world performance.

Some models were better at reasoning, others at multilingual tasks, and others at following structured instructions. Choosing the right model became one of the biggest product decisions we made.

I'm curious how other founders approach this.

Do you rely on benchmarks or your own testing?

What metrics matter most to you—accuracy, latency, cost, or consistency?

Has one model surprised you recently?

I'd love to hear how others are making these decisions.