Meet BrunoSend a request, commit the collection. No account, no cloud.
Mobile LLM Server
Run local LLMs with Android. llama.cpp & LiteRT-LM
9 followers
Run local LLMs with Android. llama.cpp & LiteRT-LM
9 followers
AI Node brings local LLM inference to Android with llama.cpp and GGUF model support. It turns your phone into a private AI server with OpenAI-compatible API endpoints. Unlike cloud-based AI apps, all models run fully on-device for privacy and low latency.
How well does it handle longer contexts on mid-range phones, and is there any noticeable battery drain during extended inference sessions?
@bedriyehhql If your phone has 12GB of RAM, I recommend configuring a maximum context size of 16K. My test device has an 8 Gen2 processor, and the first character reply takes over 20 seconds. The most comfortable context size is below 8K. Regarding battery consumption, it's generally assumed that the phone will run at maximum load, which will inevitably lead to overheating.
Finally got a working llama.cpp setup on my phone without rooting it. The OpenAI-compatible endpoint just worked with my existing scripts, which saved me a ton of hassle.