BaseRT is the fastest LLM runtime on Apple Silicon. Install it with one command and run local models on your own device.
Replies
Best
On-device inference is exactly the direction I keep hoping more indie apps can go — no per-token bill and privacy by default is a big deal. Is BaseRT meant to be embedded inside a shipping Mac/iOS app, or is it more of a dev/CLI runtime for now? That'd decide whether I could actually bundle it into my own app.
@lennoxbeflying super keen for builders to bundle it into their app, can you share a bit more about what you want to build with it and we'll make sure to help out? Easiest will be our discord https://discord.gg/wQF5vQ3jpe
Report
The discussion's focused heavily on prefill vs. decode, but there's a third axis worth thinking about for Apple Silicon specifically: unified memory's effect on how many models you can keep resident simultaneously. llama.cpp and MLX both require fairly explicit load/unload, and on a machine where you might want a small coding model, a larger reasoning model, and an embedding model all warm at once, the round-trip cost of loading from disk breaks the local-first experience more than raw token throughput does. Is BaseRT doing anything with model-weight sharing or partial preloading, or is that a separate problem from what you've optimized?
the Metal 4 tensor core optimization is the interesting part here. most local inference tools treat Apple Silicon as a nice-to-have but you're actually building for it as the primary target. curious how the performance scales with context length — prefill speed is great but does it hold up at 32k+ tokens?
Report
The speed numbers are impressive, especially on Apple hardware. For teams deciding between llama.cpp, MLX, and BaseRT, what is the tradeoff in model coverage or quantization support? I would love to know where you see BaseRT fitting in a production local-inference stack.
The 3.9x over MLX claim caught my eye — is that mostly custom Metal kernels or smarter KV-cache handling? And does the speedup hold for long-context prompt processing, or mainly decode? Running models locally is fast becoming my default for anything privacy-sensitive, so this is timely.
Replies
On-device inference is exactly the direction I keep hoping more indie apps can go — no per-token bill and privacy by default is a big deal. Is BaseRT meant to be embedded inside a shipping Mac/iOS app, or is it more of a dev/CLI runtime for now? That'd decide whether I could actually bundle it into my own app.
BaseRT
@lennoxbeflying super keen for builders to bundle it into their app, can you share a bit more about what you want to build with it and we'll make sure to help out? Easiest will be our discord https://discord.gg/wQF5vQ3jpe
The discussion's focused heavily on prefill vs. decode, but there's a third axis worth thinking about for Apple Silicon specifically: unified memory's effect on how many models you can keep resident simultaneously. llama.cpp and MLX both require fairly explicit load/unload, and on a machine where you might want a small coding model, a larger reasoning model, and an embedding model all warm at once, the round-trip cost of loading from disk breaks the local-first experience more than raw token throughput does. Is BaseRT doing anything with model-weight sharing or partial preloading, or is that a separate problem from what you've optimized?
AISA AI Skills Test
the Metal 4 tensor core optimization is the interesting part here. most local inference tools treat Apple Silicon as a nice-to-have but you're actually building for it as the primary target. curious how the performance scales with context length — prefill speed is great but does it hold up at 32k+ tokens?
The speed numbers are impressive, especially on Apple hardware. For teams deciding between llama.cpp, MLX, and BaseRT, what is the tradeoff in model coverage or quantization support? I would love to know where you see BaseRT fitting in a production local-inference stack.
LottieFiles
The 3.9x over MLX claim caught my eye — is that mostly custom Metal kernels or smarter KV-cache handling? And does the speedup hold for long-context prompt processing, or mainly decode? Running models locally is fast becoming my default for anything privacy-sensitive, so this is timely.
Tindog