Hey Product Hunt! 👋🚀

I'm so excited to finally share BENCHillionaire with
you all! 🎉💰

Here's the thing 🤔 — everyone's talking about AI Decision models (like JEV!), but when you need a model that makes fast, cheap, confident decisions, how do you actually choose one? Vendor benchmarks? Vibes? 🫠 We knew there had to be a better way.

So we built an independent benchmark for System One models 🧠⚡ — the models that take a question and a set of options and return one choice with a confidence for each.

And what better proving ground than a well-known quiz show where humans can play along too? 🎤💸

🔍 What we test:
- 📚 80,000k real questions from 12 English-language versions of the show
- 🪜 Full 15-question games, with progressive cash prizes.
- 🚪 Whether a model knows when to walk away with cash under uncertainty (spoiler:
knowing the answer is only half of it!)
- 🆘 Deciding to use "lifelines" to improve their odds
- ⏱ Accuracy, calibration, speed and cost — measured the same way for every vendor

💡 It's not just about being right — it's about knowing when you're right. A model that knows 93% of the answers but walks away from half of them wins far less than one that commits. That's calibration, and it matters for your product too. 🎯

💼 The site is free to browse and free to play. But if you're actually choosing a model for your product, the Early supporter plan is where the real data lives:

- ✅ Every single answer from every model, with its confidence — not just the headline rankings
- 📈 Calibration curves and results broken down by version and ladder step, so you see exactly where each model falls apart
- 📦 CSV and Parquet exports — the raw data, straight into your notebooks and evals
- 🧪 Your own test set run across every System One model — bring your questions, get every model's answers back
- 🔒 A price locked for as long as you stay subscribed — sign up early, never see a price increase

That's the difference between "a model scored 71%" and knowing which model to ship, at which confidence threshold, for your use case. One ranking tells you who won; the full dataset tells you what to buy. 🎯

🏢 Need more? Enterprise covers evaluation on your own data, pre-release testing on our private held-out set, and hands-on advice on picking the right model and provider.

🛡 And yes, we're truly independent: no vendor owns or sponsors us, and nobody can pay for a better ranking. Ever.

🎮 Oh, and one more thing... you can play the game yourself! Answer real questions against the clock, use your lifelines, and even phone an AI model for help 📞🤖 — then see how you stack up against the models on the exact same questions. 👀

I promise: play for a little while and you'll get a feel for which questions the models handle well — and which ones they're terrible at! Then imagine having every one of those answers, from every model, exported to a CSV. That's the Early supporter plan.

We'd love your feedback 🙏💬 Which models should we benchmark next? What would make this more useful for your team? Drop a comment below 👇

Thanks for checking us out — now go see if you can beat the machines! 🏆✨

BENCHillionaire

Compare AI decision models on accuracy, speed and cost

Compare AI decision models on accuracy, speed and cost