BaseRT
Fastest LLM runtime on Apple Silicon
269 followers
Fastest LLM runtime on Apple Silicon
269 followers
Running AI locally means no per token cost + full privacy. BaseRT is the fastest LLM runtime on Apple Silicon. Install it with one command and run local models on your own device.
This is the 2nd launch from BaseRT. View more
BaseRT
Launched this week
BaseRT is the fastest LLM runtime on Apple Silicon. Install it with one command and run local models on your own device.
Free
Launch Team
Installed it on my M2 and pulled a 7B model in under a minute, felt snappy even on battery. Nice to skip the per-token bill without cooking my laptop.
BaseRT
@zcanhasanoiulk thanks Özcan, yes 7B models are a sweet spot for local LLMs. Glad you liked it
Epsilla (YC S23)
Tuning specifically for the M5's new tensor cores through the Metal 4 API, instead of just optimizing existing kernels, is a genuinely sharp bet given how far the hardware has pulled ahead of the software stack. Your published numbers highlight prefill speed specifically, so how much does decode/token-generation throughput improve on M5 versus llama.cpp and MLX, since that's usually the bottleneck for longer, more conversational or agentic workloads?
BaseRT
@renchu_song decode ist also up to 33% faster, numbers here https://www.basecompute.co/getbasert
Took a few minutes to set up on my M2 and the speed genuinely surprised me compared to other runtimes I've tried. Love that everything stays on device, no tokens flying off to some server.
BaseRT
@irmakmemik thanks, love the feedback!
@lukas_base Just a quick question how does it compare to LiteRT? It is no secret that MLX and llama.cpp are falling behind but we already have a great architecture from Google.
@lukas_base Benchmarks you posted seems to be prefill numbers. do you have decode or token generation throughput comparisons against llama.cpp and MLX too, or is most of the gain concentrated in prompt processing rather than generation speed? Thanks!
the prefill numbers are impressive, but prefill isn't usually where I feel the wait on a laptop - it's token-by-token decode speed during a long generation. is the decode-side speedup in the same range as prefill, or is that gap smaller since decode is more memory-bandwidth bound than compute bound?
AISA AI Skills Test
the Metal 4 tensor core optimization is the interesting part here. most local inference tools treat Apple Silicon as a nice-to-have but you're actually building for it as the primary target. curious how the performance scales with context length — prefill speed is great but does it hold up at 32k+ tokens?