Launching today
Reame
Self-hosted LLM inference on the hardware you already have
7 followers
Self-hosted LLM inference on the hardware you already have
7 followers
Reame is a CPU-first LLM inference server on llama.cpp with an OpenAI-compatible API. Built for narrow, repetitive workloads on cheap hardware — a €5 VPS, a free tier, a 2-core ARM box. Its memory layer caches prompts, prefixes and past generations to disk, so request #100 costs a fraction of request #1. MIT.
finally something for tiny boxes like my old VPS. The disk cache thing actually works, like the second request comes back way faster and barely touches CPU.
One thing that would make this way more useful for me: built-in support for batched inference across multiple cached prompts. Right now I assume requests still queue one at a time, but if the memory layer already knows the prefixes, batching the matching ones together could really cut down latency on bursty workloads.
Memory layer caching is a clever way to keep costs down on tiny boxes. One thing that would help me adopt it faster is a built-in warmup script that pre-decodes common prompt prefixes after a restart, so the first real request after deploy doesn't eat the cold-cache penalty and wreck latency on smaller instances.
Honestly impressed it actually runs on a 5 euro VPS without choking, and the cached prefixes made repeat calls nearly instant on my end.