@sap_uy M5 Pro, but it runs optimised for all M series chips
Report
Took a few minutes to set up on my M2 and the speed genuinely surprised me compared to other runtimes I've tried. Love that everything stays on device, no tokens flying off to some server.
@lukas_base Just a quick question how does it compare to LiteRT? It is no secret that MLX and llama.cpp are falling behind but we already have a great architecture from Google.
Tuning specifically for the M5's new tensor cores through the Metal 4 API, instead of just optimizing existing kernels, is a genuinely sharp bet given how far the hardware has pulled ahead of the software stack. Your published numbers highlight prefill speed specifically, so how much does decode/token-generation throughput improve on M5 versus llama.cpp and MLX, since that's usually the bottleneck for longer, more conversational or agentic workloads?
@lukas_base Benchmarks you posted seems to be prefill numbers. do you have decode or token generation throughput comparisons against llama.cpp and MLX too, or is most of the gain concentrated in prompt processing rather than generation speed? Thanks!
Report
the prefill numbers are impressive, but prefill isn't usually where I feel the wait on a laptop - it's token-by-token decode speed during a long generation. is the decode-side speedup in the same range as prefill, or is that gap smaller since decode is more memory-bandwidth bound than compute bound?
Report
the no-per-token part matters as much as the speed imo, it's what makes always-on background agents affordable to run. is that the direction you're headed?
6.4x faster than llama.cpp is a big claim — what's the tradeoff? Curious whether it's fast enough to run a real-time voice pipeline on-device (where every 100ms of latency is felt), or if the win is mainly on batch/throughput. Running local with no token cost + full privacy is a huge deal for small-business use cases. Following.
6.4x over llama.cpp is a big claim for a field this optimized — where does the speedup actually come from? Custom Metal kernels, better KV-cache layout, speculative decoding, or quantization formats? And does the gap hold at batch-1 decode on long contexts, or is it mostly a prefill win? Genuinely curious — we run local models for dev work and llama.cpp has been the default for so long that a 6x claim deserves a look.
Report
Is it also on huggingface? I don't want to install it via bash command.
@danielherman huggingface just hosts models, bash command would the easiest way to install. Do you prefer a UI?
Report
@lukas_base So, in general, I am avoiding installing via bash command, unless the community support is large. I am familiar with HF and ollama and I test new models locally via ollama or HF as preferred ways of doing so.
Replies
Product Hunt
BaseRT
@sap_uy M5 Pro, but it runs optimised for all M series chips
Took a few minutes to set up on my M2 and the speed genuinely surprised me compared to other runtimes I've tried. Love that everything stays on device, no tokens flying off to some server.
BaseRT
@irmakmemik thanks, love the feedback!
@lukas_base Just a quick question how does it compare to LiteRT? It is no secret that MLX and llama.cpp are falling behind but we already have a great architecture from Google.
HarnessRouter
Tuning specifically for the M5's new tensor cores through the Metal 4 API, instead of just optimizing existing kernels, is a genuinely sharp bet given how far the hardware has pulled ahead of the software stack. Your published numbers highlight prefill speed specifically, so how much does decode/token-generation throughput improve on M5 versus llama.cpp and MLX, since that's usually the bottleneck for longer, more conversational or agentic workloads?
BaseRT
@renchu_song decode ist also up to 33% faster, numbers here https://www.basecompute.co/getbasert
@lukas_base Benchmarks you posted seems to be prefill numbers. do you have decode or token generation throughput comparisons against llama.cpp and MLX too, or is most of the gain concentrated in prompt processing rather than generation speed? Thanks!
the prefill numbers are impressive, but prefill isn't usually where I feel the wait on a laptop - it's token-by-token decode speed during a long generation. is the decode-side speedup in the same range as prefill, or is that gap smaller since decode is more memory-bandwidth bound than compute bound?
the no-per-token part matters as much as the speed imo, it's what makes always-on background agents affordable to run. is that the direction you're headed?
BaseRT
@andrewzakonov yes! All you can eat tokens :)
6.4x faster than llama.cpp is a big claim — what's the tradeoff? Curious whether it's fast enough to run a real-time voice pipeline on-device (where every 100ms of latency is felt), or if the win is mainly on batch/throughput. Running local with no token cost + full privacy is a huge deal for small-business use cases. Following.
BaseRT
@david_marko Check out our technical papers, we compare for all M series chips and a whole suite of models https://www.basecompute.co/research
6.4x over llama.cpp is a big claim for a field this optimized — where does the speedup actually come from? Custom Metal kernels, better KV-cache layout, speculative decoding, or quantization formats? And does the gap hold at batch-1 decode on long contexts, or is it mostly a prefill win? Genuinely curious — we run local models for dev work and llama.cpp has been the default for so long that a 6x claim deserves a look.
Is it also on huggingface? I don't want to install it via bash command.
BaseRT
@danielherman huggingface just hosts models, bash command would the easiest way to install. Do you prefer a UI?
@lukas_base So, in general, I am avoiding installing via bash command, unless the community support is large. I am familiar with HF and ollama and I test new models locally via ollama or HF as preferred ways of doing so.