Launching today

Cli Modelarium

Launching today

Compare LLMs with real statistics, right from your terminal

4 followers

Compare LLMs with real statistics, right from your terminal

4 followers

Visit website

CLI tool for comparing AI language models with statistical rigor. Supports 8 cloud providers (OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Groq, OpenRouter) plus local models. Bootstrap confidence intervals, paired significance tests, hallucination detection, LLM-as-judge panels, cost tracking with hard caps. One pip install, no infrastructure. Available on Linux, macOS, and Windows. Python 3.11+. Apache 2.0. pip install cli-modelarium

Free

Launch tags:Open Source•Developer Tools•Artificial Intelligence

Launch Team

FramerLaunch websites with enterprise needs at startup speeds.

Promoted

Maker

📌

I built Cli Modelarium because every time I wanted to compare two LLMs, I had to pick between eyeballing outputs in a chat window or spinning up an entire evaluation platform. The CLI does it from the terminal. You give it a prompt, pick your models, and get a side-by-side with cost tracking, latency, and actual statistical tests. Bootstrap confidence intervals, McNemar's test, hallucination detection, the works. 8 cloud providers, local model support, and a --max-cost flag so you don't burn through API credits by accident. 917 tests across 9 OS/Python combinations. Apache 2.0. Would love to hear what you think.

Report

1d ago