Reame
Self-hosted LLM inference on the hardware you already have
6 followers
Self-hosted LLM inference on the hardware you already have
6 followers
Reame is a CPU-first LLM inference server on llama.cpp with an OpenAI-compatible API. Built for narrow, repetitive workloads on cheap hardware — a €5 VPS, a free tier, a 2-core ARM box. Its memory layer caches prompts, prefixes and past generations to disk, so request #100 costs a fraction of request #1. MIT.
Memory layer caching is a clever way to keep costs down on tiny boxes. One thing that would help me adopt it faster is a built-in warmup script that pre-decodes common prompt prefixes after a restart, so the first real request after deploy doesn't eat the cold-cache penalty and wreck latency on smaller instances.
@semihkrcayf3f
You’re absolutely right that avoiding the first-request penalty is important on small instances.
One useful detail is that Reame’s prefix cache is persistent: prefixes already stored on disk survive normal process restarts, so they shouldn’t need to be computed again simply because the server restarted.
For a new deployment, an empty cache or newly introduced prompts, Reame also provides POST /v1/warm, which pre-fills a prompt into the cache and can already be called from a deployment or startup hook.
What is still missing is a more convenient built-in workflow for automatically loading a predefined list of common prefixes. I agree that this would make production deployments cleaner and easier, so it’s a valuable suggestion. Thank you!
finally something for tiny boxes like my old VPS. The disk cache thing actually works, like the second request comes back way faster and barely touches CPU.
@hacer358337
Thank you! That’s exactly the kind of workload I built Reame for. On a small CPU, paying the same prefill cost twice makes no sense, so seeing the second request return much faster is the best possible validation.
If you don’t mind sharing the VPS specifications and model you tested, it would be a valuable real-world data point for the project. And if Reame has been useful to you, sharing it with someone using similar hardware would really help more people discover it.
Honestly impressed it actually runs on a 5 euro VPS without choking, and the cached prefixes made repeat calls nearly instant on my end.
@nazlcan1332465
I’m really glad to hear it worked well on a €5 VPS—that is exactly the type of hardware Reame is designed for. The goal is to make useful inference possible without requiring an expensive GPU or paying repeatedly for the same computation.
If you can share the VPS CPU, RAM, model and approximate cold-versus-warm timings, I’d love to include that configuration in the project’s real-world test data. If you know someone else experimenting with LLMs on inexpensive hardware, sharing Reame with them would also help the project grow.