Hey Product Hunt! Lukas here, co-founder of BaseRT.
TLDR: a new performance ceiling for LLMs on Apple's M5 Pro.
The M5 introduces a new tensor core architecture, exposed through the Metal 4 tensor API. We tuned BaseRT to use all of it.
The results:
Up to 6.3× faster prompt processing (prefill) than llama.cpp
Up to 3.9× faster prefill than MLX
Tested on 10 configs across Qwen3/3.5/3.6, Llama 3.2, and Gemma 4, from 0.6B to 35B params
We built BaseRT because we think on-device inference is about to matter a lot, and Apple silicon is the best consumer hardware to run it on.
The software just hasn't been keeping up with the chips, and we're trying to close that gap. Happy to answer anything about the benchmarks, the Metal 4 tensor API, or where we're headed next. Would love your feedback, and if you've got an M5, try it and tell us what numbers you get.
Report
curious how much of this gain is specific to M5's tensor cores vs the general Metal 4 tensor API. I'm running an M4 Pro, plenty of people doing local inference haven't upgraded yet - does BaseRT still beat llama.cpp/MLX meaningfully on M4, just with a smaller multiplier, or is most of the win locked to the new hardware?
@galdayan we’ve optimised for all M series chips. Check out our technical reports, the first one has the benchmarks for previous M series chips you’re looking for. Here https://www.basecompute.co/research
Report
@lukas_base will do, thanks - checking the report now. good to know the mini isn't left behind on this one, appreciate the quick answer
@hafizezdztabm thanks that’s great feedback, we made one line install a prio
Report
The one-command install is such a thoughtful touch, especially for folks who just want to experiment without dealing with setup headaches. Nice work making local LLMs feel that accessible on Apple Silicon.
@oykuener77260 awesome, thanks :) curious what model did you run
Report
Love how simple the install is and the speed on my M2 is genuinely impressive. One thing that would make this even better is a built-in model benchmark tool so you can see tokens/sec for different models on your specific hardware right from the CLI.
One thing I'd love to see is a simple built-in model downloader so I can grab and switch between popular open weights from the CLI without hunting for GGUF files manually. That would make testing new models way smoother.
curious how it handles model swapping on the fly, like dropping into a bigger LLM only when a prompt clearly needs it. A simple toggle or auto-routing between something like Llama 3 8B and 70B based on the task would be a really nice touch for battery life on MacBooks.
@zcanhasanoiulk thanks Özcan, yes 7B models are a sweet spot for local LLMs. Glad you liked it
Report
Very interesting work. It feels like we're finally seeing software catch up to what Apple Silicon has been capable of for a while.
One thing I'm curious about: if an application needs to switch between different models depending on latency or quality requirements, does BaseRT help manage that orchestration, or is that intentionally left to the application layer?
@nextmark that feature is in the making :) But you can already load multiple models
Report
@lukas_base Great! Glad to hear that's in progress.
Realizing that you can already load multiple models opens up interesting possibilities, though I imagine the real value unlock is when BaseRT makes intelligent decisions about when to switch between them. The orchestration logic living in the app layer is fine for power users, but having BaseRT recommend or auto-suggest model switches based on the query would be a killer feature for mainstream adoption.
Replies
BaseRT
curious how much of this gain is specific to M5's tensor cores vs the general Metal 4 tensor API. I'm running an M4 Pro, plenty of people doing local inference haven't upgraded yet - does BaseRT still beat llama.cpp/MLX meaningfully on M4, just with a smaller multiplier, or is most of the win locked to the new hardware?
BaseRT
@galdayan we’ve optimised for all M series chips. Check out our technical reports, the first one has the benchmarks for previous M series chips you’re looking for. Here https://www.basecompute.co/research
@lukas_base will do, thanks - checking the report now. good to know the mini isn't left behind on this one, appreciate the quick answer
BaseRT
@galdayan you're welcome! We running a discord for on-device and open-source AI if you have more questions https://discord.gg/wQF5vQ3jpe
@lukas_base will hop in there, thanks. good way to keep the questions out of the comment section for anything more detailed
Running Llama on my M2 was genuinely instant, no setup headaches. Loving that it just works without Docker or weird config files.
BaseRT
@hafizezdztabm thanks that’s great feedback, we made one line install a prio
The one-command install is such a thoughtful touch, especially for folks who just want to experiment without dealing with setup headaches. Nice work making local LLMs feel that accessible on Apple Silicon.
BaseRT
@aleyna710492 thanks! Just curious, would you want a UI or were you happy with the terminal setup
honestly the install was just one command and my m2 was running llama models in like a minute, kind of wild how fast it feels compared to ollama
BaseRT
@oykuener77260 awesome, thanks :) curious what model did you run
Love how simple the install is and the speed on my M2 is genuinely impressive. One thing that would make this even better is a built-in model benchmark tool so you can see tokens/sec for different models on your specific hardware right from the CLI.
BaseRT
@ag_rbas9241 it's already included, check out https://docs.basecompute.co/
One thing I'd love to see is a simple built-in model downloader so I can grab and switch between popular open weights from the CLI without hunting for GGUF files manually. That would make testing new models way smoother.
BaseRT
@adem1343093 it's already included! have a look at the download section in the docs https://docs.basecompute.co/
curious how it handles model swapping on the fly, like dropping into a bigger LLM only when a prompt clearly needs it. A simple toggle or auto-routing between something like Llama 3 8B and 70B based on the task would be a really nice touch for battery life on MacBooks.
BaseRT
@brahimn9oo we’re currently looking into model routing for cost optimisation, latency. What are you looking to optimise for
Installed it on my M2 and pulled a 7B model in under a minute, felt snappy even on battery. Nice to skip the per-token bill without cooking my laptop.
BaseRT
@zcanhasanoiulk thanks Özcan, yes 7B models are a sweet spot for local LLMs. Glad you liked it
Very interesting work. It feels like we're finally seeing software catch up to what Apple Silicon has been capable of for a while.
One thing I'm curious about: if an application needs to switch between different models depending on latency or quality requirements, does BaseRT help manage that orchestration, or is that intentionally left to the application layer?
BaseRT
@nextmark that feature is in the making :) But you can already load multiple models
@lukas_base Great! Glad to hear that's in progress.
Realizing that you can already load multiple models opens up interesting possibilities, though I imagine the real value unlock is when BaseRT makes intelligent decisions about when to switch between them. The orchestration logic living in the app layer is fine for power users, but having BaseRT recommend or auto-suggest model switches based on the query would be a killer feature for mainstream adoption.
Looking forward to seeing how that evolves!