Launching today
jitllm
An operating system for LLM inference
1 follower
An operating system for LLM inference
1 follower
One binary. No dependencies. Drop it in and run. Up to 4.62x faster prefill than llama.cpp and 5.53x faster batched decoding than vLLM. jitllm detects your hardware and optimizes kernels and model placement across CPUs and GPUs. Run open-weight flagships, even larger than memory. OpenAI- and Anthropic-compatible APIs, desktop and terminal apps. Linux, macOS and Windows.
Hey Product Hunters,
Inference engines are bulky and rigid. There’s a lot to set up, and even after you get a model running, getting the most out of your hardware is another job.
I wanted to build something easier to live with: one binary you can drop onto a machine and start running models. Something that does more of the work of figuring out the hardware itself.
The inspiration came from operating systems and JIT compilers, two things I've worked on a lot. An OS detects hardware, uses the right drivers, schedules processes, and manages memory. A JIT compiler generates code for the machine it's running on. jitllm brings those ideas together to make inference faster, easier to run, and more flexible.
That's why I call it an operating system for LLM inference. It detects your CPUs and GPUs, generates optimized kernels, and orchestrates model placement at runtime. Models larger than memory page from disk. Layers can move between devices while keeping their conversation state, so changing placement doesn't mean starting over.
jitllm is easy to set up, deploy, and use, and it's mighty fast! You can find the benchmarks and get started at https://jitllm.org