Hey Product Hunt 👋 I built VeloxQuant because I kept hitting the same wall running local models on my own Mac: the KV cache — not the model weights — is what actually kills your context length and your memory budget as generation goes on. Every "run an LLM locally" tutorial stops at loading a quantized model and calling `generate()`; almost none of them touch the thing that grows unbounded with every token you generate. So I built the piece that was missing: a compression engine purpose-built for the KV cache, with 43 methods pulled from the research literature (quantization, token eviction, cross-layer merging) behind one interface, backed by hand-written Metal kernels so the compression itself doesn't become the new bottleneck. A few things I'd love feedback on: - If you're already running local models (Ollama, LM Studio, raw MLX/llama.cpp), what's actually stopping you from going to longer contexts today — memory, speed, or tooling? - Anyone building offline-first or on-device products (healthcare, legal, fintech, defense) — what would make you trust a local inference stack enough to ship it, versus falling back to a cloud API? - I'm building this out into SDKs beyond Python (TypeScript, Go, Rust) plus a native macOS app and Android support — which of those would unlock something for you first? This is solo-built and open source (MIT), and it's an early step toward something I care about a lot: making private, on-device AI a default technical property of the hardware people already own, not something you have to opt into by giving up convenience. Happy to answer anything about the internals, the Metal kernels, or the roadmap.

VeloxQuant-MLX

Run bigger local LLMs in less memory, on Mac

Run bigger local LLMs in less memory, on Mac