Agnos LLM gateway - a gateway agnostic control plane that contains LiteLLM, Bifrost and Portkey
Hi folks,
I'm excited to share Agnos Proxy - an open-source (MIT) LLM governance gateway I've been working on.
Repo/Website Link:
Website: https://agnos-llm-gateway.site
Live demo (playground): https://agnos-llm-gateway.site/app
Description:
Agnos is a self-hosted, gateway-agnostic control plane for LLM traffic. Your apps call one OpenAI-compatible endpoint; behind it, the gateways you already know (LiteLLM, Bifrost, Portkey) plus a built-in engine plug in as swappable "stateless translator" adapters. You own the control plane - keys, budgets, guardrails, audit - and the gateways only translate.
It's built around three ideas:
Contain - Already running LiteLLM / Bifrost / Portkey? Put Agnos in front and use only the stateless translate path of that gateway, never its control plane. Provider keys live in Agnos's encrypted vault and are injected per request; the contained gateway stores nothing. You get one place for keys, spend limits, guardrails and an audit log without handing any of that to the underlying gateway.
Swap - Change the engine live (for the whole gateway or per provider) with no client changes. If a contained gateway ships a regression, quarantine it and evacuate its traffic to another engine in seconds. A shadow/parity mode lets you compare a new engine against the current one before you cut over.
Route among them - Send each request to the gateway that fits, from a single endpoint and one key set:
High-volume chat -> Bifrost (fast Go translator)
A self-hosted model that must stay in-VPC -> built-in engine (in-process, no sidecar)
A long-tail model only one gateway speaks -> LiteLLM (100+ providers via LiteLLM)
Azure OpenAI you already run through Portkey -> keep Portkey's translate path You can even run two gateways at once, e.g. Bifrost for the fast path and LiteLLM for reach, behind the same endpoint.
Why it's useful: you get vendor-style governance (keys, spend, guardrails, reliability, audit) over whatever gateway(s) you already trust, and you're never locked into one of them. Overhead is ~1ms p50 for the full control plane.
Deployment:
Docker: docker run -p 8090:8090 ghcr.io/siva010928/agnos-proxy:latest
pipx: pipx install agnos-proxy-llm-gateway
docker-compose example, an install script, and a self-host guide are in the docs: https://agnos-llm-gateway.site/app/docs/install
MIT licensed and fully self-hostable. The live demo above is the same app running in a read-only playground, so you can click through before installing anything.
Feedback, issues, contributions welcome! Would love to see what engines and providers the community wants to contain next.
Replies