BaseRT
Fastest LLM runtime on Apple Silicon
265 followers
Fastest LLM runtime on Apple Silicon
265 followers
Running AI locally means no per token cost + full privacy. BaseRT is the fastest LLM runtime on Apple Silicon. Install it with one command and run local models on your own device.
This is the 2nd launch from BaseRT. View more
BaseRT
Launched this week
BaseRT is the fastest LLM runtime on Apple Silicon. Install it with one command and run local models on your own device.
Free
Launch Team
On-device inference is exactly the direction I keep hoping more indie apps can go — no per-token bill and privacy by default is a big deal. Is BaseRT meant to be embedded inside a shipping Mac/iOS app, or is it more of a dev/CLI runtime for now? That'd decide whether I could actually bundle it into my own app.
BaseRT
@lennoxbeflying super keen for builders to bundle it into their app, can you share a bit more about what you want to build with it and we'll make sure to help out? Easiest will be our discord https://discord.gg/wQF5vQ3jpe
Is it also on huggingface? I don't want to install it via bash command.
BaseRT
@danielherman huggingface just hosts models, bash command would the easiest way to install. Do you prefer a UI?
@lukas_base So, in general, I am avoiding installing via bash command, unless the community support is large. I am familiar with HF and ollama and I test new models locally via ollama or HF as preferred ways of doing so.
6.4x over llama.cpp is a big claim for a field this optimized — where does the speedup actually come from? Custom Metal kernels, better KV-cache layout, speculative decoding, or quantization formats? And does the gap hold at batch-1 decode on long contexts, or is it mostly a prefill win? Genuinely curious — we run local models for dev work and llama.cpp has been the default for so long that a 6x claim deserves a look.
The discussion's focused heavily on prefill vs. decode, but there's a third axis worth thinking about for Apple Silicon specifically: unified memory's effect on how many models you can keep resident simultaneously. llama.cpp and MLX both require fairly explicit load/unload, and on a machine where you might want a small coding model, a larger reasoning model, and an embedding model all warm at once, the round-trip cost of loading from disk breaks the local-first experience more than raw token throughput does. Is BaseRT doing anything with model-weight sharing or partial preloading, or is that a separate problem from what you've optimized?
Product Hunt
BaseRT
@sap_uy M5 Pro, but it runs optimised for all M series chips
curious how it handles model swapping on the fly, like dropping into a bigger LLM only when a prompt clearly needs it. A simple toggle or auto-routing between something like Llama 3 8B and 70B based on the task would be a really nice touch for battery life on MacBooks.
BaseRT
@brahimn9oo we’re currently looking into model routing for cost optimisation, latency. What are you looking to optimise for
Installed it on my M2 and pulled a 7B model in under a minute, felt snappy even on battery. Nice to skip the per-token bill without cooking my laptop.
BaseRT
@zcanhasanoiulk thanks Özcan, yes 7B models are a sweet spot for local LLMs. Glad you liked it