Agnos LLM gateway - a gateway agnostic control plane that contains LiteLLM, Bifrost and Portkey

Hi folks,

I'm excited to share Agnos Proxy - an open-source (MIT) LLM governance gateway I've been working on.

Repo/Website Link:

• Website: https://agnos-llm-gateway.site

• GitHub: https://github.com/siva010928/agnos-proxy-oss

• Live demo (playground): https://agnos-llm-gateway.site/app

• Repo: https://github.com/siva010928/agnos-proxy-oss

• Youtube: https://www.youtube.com/watch?v=cGCz44h_D-k

Description:

Agnos is a self-hosted, gateway-agnostic control plane for LLM traffic. Your apps call one OpenAI-compatible endpoint; behind it, the gateways you already know (LiteLLM, Bifrost, Portkey) plus a built-in engine plug in as swappable "stateless translator" adapters. You own the control plane - keys, budgets, guardrails, audit - and the gateways only translate.

It's built around three ideas:

• Contain - Already running LiteLLM / Bifrost / Portkey? Put Agnos in front and use only the stateless translate path of that gateway, never its control plane. Provider keys live in Agnos's encrypted vault and are injected per request; the contained gateway stores nothing. You get one place for keys, spend limits, guardrails and an audit log without handing any of that to the underlying gateway.

• Swap - Change the engine live (for the whole gateway or per provider) with no client changes. If a contained gateway ships a regression, quarantine it and evacuate its traffic to another engine in seconds. A shadow/parity mode lets you compare a new engine against the current one before you cut over.

• Route among them - Send each request to the gateway that fits, from a single endpoint and one key set:

  • High-volume chat -> Bifrost (fast Go translator)

  • A self-hosted model that must stay in-VPC -> built-in engine (in-process, no sidecar)

  • A long-tail model only one gateway speaks -> LiteLLM (100+ providers via LiteLLM)

  • Azure OpenAI you already run through Portkey -> keep Portkey's translate path You can even run two gateways at once, e.g. Bifrost for the fast path and LiteLLM for reach, behind the same endpoint.

Why it's useful: you get vendor-style governance (keys, spend, guardrails, reliability, audit) over whatever gateway(s) you already trust, and you're never locked into one of them. Overhead is ~1ms p50 for the full control plane.

Deployment:

• Docker: docker run -p 8090:8090 ghcr.io/siva010928/agnos-proxy:latest

• pipx: pipx install agnos-proxy-llm-gateway

• docker-compose example, an install script, and a self-host guide are in the docs: https://agnos-llm-gateway.site/app/docs/install

• MIT licensed and fully self-hostable. The live demo above is the same app running in a read-only playground, so you can click through before installing anything.

Feedback, issues, contributions welcome! Would love to see what engines and providers the community wants to contain next.