BaseRT
Fastest LLM runtime on Apple Silicon
265 followers
Fastest LLM runtime on Apple Silicon
265 followers
Running AI locally means no per token cost + full privacy. BaseRT is the fastest LLM runtime on Apple Silicon. Install it with one command and run local models on your own device.
This is the 2nd launch from BaseRT. View more
BaseRT
Launched this week
BaseRT is the fastest LLM runtime on Apple Silicon. Install it with one command and run local models on your own device.
Free
Launch Team
Epsilla (YC S23)
Tuning specifically for the M5's new tensor cores through the Metal 4 API, instead of just optimizing existing kernels, is a genuinely sharp bet given how far the hardware has pulled ahead of the software stack. Your published numbers highlight prefill speed specifically, so how much does decode/token-generation throughput improve on M5 versus llama.cpp and MLX, since that's usually the bottleneck for longer, more conversational or agentic workloads?
BaseRT
@renchu_song decode ist also up to 33% faster, numbers here https://www.basecompute.co/getbasert
Took a few minutes to set up on my M2 and the speed genuinely surprised me compared to other runtimes I've tried. Love that everything stays on device, no tokens flying off to some server.
BaseRT
@irmakmemik thanks, love the feedback!
@lukas_base Just a quick question how does it compare to LiteRT? It is no secret that MLX and llama.cpp are falling behind but we already have a great architecture from Google.
@lukas_base Benchmarks you posted seems to be prefill numbers. do you have decode or token generation throughput comparisons against llama.cpp and MLX too, or is most of the gain concentrated in prompt processing rather than generation speed? Thanks!
the prefill numbers are impressive, but prefill isn't usually where I feel the wait on a laptop - it's token-by-token decode speed during a long generation. is the decode-side speedup in the same range as prefill, or is that gap smaller since decode is more memory-bandwidth bound than compute bound?
AISA AI Skills Test
the Metal 4 tensor core optimization is the interesting part here. most local inference tools treat Apple Silicon as a nice-to-have but you're actually building for it as the primary target. curious how the performance scales with context length — prefill speed is great but does it hold up at 32k+ tokens?
The speed numbers are impressive, especially on Apple hardware. For teams deciding between llama.cpp, MLX, and BaseRT, what is the tradeoff in model coverage or quantization support? I would love to know where you see BaseRT fitting in a production local-inference stack.