Cli Modelarium
Compare LLMs with real statistics, right from your terminal
5 followers
Compare LLMs with real statistics, right from your terminal
5 followers
CLI tool for comparing AI language models with statistical rigor. Supports 10 cloud
providers (OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Groq, OpenRouter,
Alibaba/DashScope, Z.AI) plus local models. Bootstrap confidence intervals, paired
significance tests, hallucination detection, LLM-as-judge panels, cost tracking with
hard caps. One pip install, no infrastructure. Available on Linux, macOS, and Windows.
Python 3.11+. Apache 2.0. pip install cli-modelarium
Free
Launch Team
Customer.ioAutomate Messaging Everywhere — Startups Get 12 Months Free
Promoted
Quick update for anyone following along. Cli Modelarium 0.1.4 just shipped, and the headline is two new providers.
It is 10 cloud providers now, up from 8. You can put Alibaba's Qwen models (via DashScope) and Z.AI's GLM models head to head against the usual lineup (OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Groq, OpenRouter), plus your local models. If you have wanted to benchmark the open-weight models against the frontier ones on your own prompts, that is now a single command.
Also in this release:
Refreshed all the pricing to current provider rates
Wired Qwen and GLM into the model groups (all-flagship, all-budget, all-fast, all-cheap, all-reasoning), so you can pull them in by group instead of one at a time
Added Python 3.14 support
A few model id updates to track provider renames
Everything else works the same: side-by-side output with cost and latency, real statistical tests (bootstrap confidence intervals, McNemar's, paired significance), hallucination detection, LLM-as-judge, and the --max-cost flag so you do not burn through API credits by accident. Still one pip install, no infrastructure, Apache 2.0.
Upgrade with: pip install --upgrade cli-modelarium
Would love to hear how the new providers work for your use case.
Quick update for anyone following along. Cli Modelarium 0.1.5 just shipped.
The latest frontier models are now callable. Anthropic and OpenAI both deprecated the temperature parameter on their newer models, returning a 400 if the parameter is present, so the tool omits it where the provider requires that. If you have been trying to benchmark Opus 5 or the gpt-5.6 line against the rest of your lineup, that now works in a single command.
Also in this release:
--output-format json|csv is pipeable. Progress and status moved to stderr and the payload writes straight through, so | jq works at any terminal width
CSV values are escaped against spreadsheet formula evaluation. A response beginning with = is text, not a formula
Retired model IDs now fail with the replacement named and the retirement date, rather than resolving to whatever the provider redirects to. Some providers redirect a retired slug and bill at the new rate, so the run looks fine and the reported cost is quietly wrong
Pricing re-verified against first-party provider pages
Worth knowing before you upgrade a pipeline: CI exit codes shift once assertions actually run, human output moves to stderr for machine formats, CSV values may carry a leading apostrophe, and all-open-weight now needs GROQ_API_KEY.
Everything else works the same: side-by-side output with cost and latency, real statistical tests, hallucination detection, LLM-as-judge, and --max-cost so you do not burn through credits by accident. Still one pip install and no infrastructure.
Upgrade with: pip install --upgrade cli-modelarium