Viktor.comAn AI coworker that actually does the work
Launching today
Inferbench
Benchmarks local LLM engines on your hardware
1 follower
Benchmarks local LLM engines on your hardware
1 follower
InferBench is a vendor-neutral CLI tool that benchmarks local LLM inference engines like omlx and llama.cpp directly on your own hardware. Instead of relying on external metrics, it reports real, measured tokens per second. InferBench auto-detects engines, runs a fixed prompt set, and recommends the fastest configuration for your setup. It is self-hosted and Apache 2.0 licensed.
The inspiration for InferBench stemmed from the constant need for absolute certainty regarding local LLM performance. Developers often have to rely on benchmark numbers generated on completely different hardware configurations, which rarely reflect actual, real-world performance. The primary problem to solve was this lack of reliable, local benchmarking. While existing solutions offer theoretical memory fit estimates or isolated, single-engine metrics, there was a distinct need for a tool that reports real, measured tokens per second directly on a user's own machine.
To address this, the approach evolved into creating a solution that is entirely vendor-neutral and cross-engine. Instead of just running simple tests, InferBench was engineered to automatically detect installed engines (like omlx and llama.cpp) and process a fixed prompt set through one shared HTTP harness. This evolution ensured that the tool not only benchmarks but actively recommends the absolute fastest configuration for any specific hardware setup.
Repo: https://github.com/RudrenduPaul/InferBench
MCP Servers:
https://mcpservers.org/servers/rudrendupaul/inferbench
https://glama.ai/mcp/servers/RudrenduPaul/inferbench
NPM: https://www.npmjs.com/package/inferbench-cli
PyPI: https://pypi.org/project/inferbench-cli