Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-compatible API. - rwilliamspbg-ops/Ghostlink

Hey Product Hunt! I built Ghostlink to solve a problem I kept hitting while running local LLMs: my compute was scattered across different devices (CPUs, GPUs, and NPUs), but existing orchestration stacks were either too heavy or hard to route across mixed hardware. Ghostlink is a high-performance, open-source distributed inference fabric that routes workloads across local, heterogeneous clusters. Key Highlights: Zero-Copy SPSC Ring Buffers: Sub-microsecond handoff primitives for low latency. Hardware-Aware Placement: Auto-detects and routes across CPU, CUDA, DirectML, Metal, and NPUs. OpenAI-Compatible API: Plug-and-play drop-in replacement for existing LLM tools and workflows. Ghostlink Studio GUI: Modern frontend with real-time hardware telemetry and built-in MCP tool integration. It’s 100% open source under the MIT license. Check out the GitHub repo or run launch.bat / launch.sh to try it out locally. Would love to hear your feedback, thoughts on distributed local inference, or feature requests!

Pushed a small model through Ghostlink on a mixed rig and was surprised how little config it actually needed to hit decent throughput. The zero-copy ring buffers seem to do real work instead of just sounding fast.

@nelson_gao Thank you for trying Ghostlink, I would be interested in your equipment setup, I have been limited to a few options.

Ghostlink

Run local LLMs across heterogeneous clusters

Run local LLMs across heterogeneous clusters