infai
Run local AI with profiles, not flags
Start new thread

infai - Run local AI with profiles, not flags

by•
Its an open-source terminal control plane for local AI inference. Find or import models, save reusable launch profiles for llama.cpp and vLLM, check hardware fit, and run multiple models with live logs, ports, tokens-per-second, CPU, RAM, and VRAM telemetry. One Go binary, one SQLite file, no cloud or telemetry.

Add a comment

Replies

Best
Maker
📌

👋 Hey Product Hunt! I'm the maker of infai.

The problem: Every local inference engine — llama.cpp, vLLM — has its own dialect of flags. You end up scrolling through shell history, copying flags from READMEs, and praying you didn't forget --flash-attn . Your shell history is not a configuration system.

The fix: infai gives every engine one shape: the profile (model + engine + config). Define it once, launch it with one keypress, run multiple models concurrently, and watch live telemetry — all from a single terminal.

What's already shipped:

  1. Support popular inference engine: llama.cpp and vLLM backends under the same workflow

  2. Runs dashboard: with per-model status, uptime, and live TPS

  3. Resource Telemetry: System + per-model CPU, RAM, VRAM telemetry

  4. Hugging Face model imports: model search & download with resumable transfers

  5. Hardware-fit checks before launch

  6. 11 terminal themes

  7. One SQLite file, one Go binary, Apache-2.0

It's fully open source:

I'd love to hear what engines or features you'd want next. Fire away! 🔥