Agent Gateway - Lossless token compression reduces costs by 50%‑90%.

Last month I built a multi-agent collaborative local automation script. It ran smoothly, but when I checked the OpenAI and Claude bills at month-end, I was utterly silenced. Agent systems are essentially built on "continuous reflection + iterative tool-calling loops." With each additional round, the history context snowballs—cluttered with repetitive system prompts, oversized tool schemas, and malformed JSON responses. Most reverse proxies and gateways on the market only handle routing and rate limiting; they're largely powerless against the token explosion caused by long contexts. After the sticker shock, I rolled my own LLM gateway tailored specifically for agent scenarios. The core is squeezing redundancy at both I/O ends: 1. Static/Dynamic Layered Context Caching Naive prompt hashing doesn't work here, since every agent invocation carries tiny dynamic variables (timestamps, local state). I split the request payload into three layers: Static layer: `System Prompt` and `Tool Definitions` Semi-static layer: multi-turn history recall and global context Dynamic layer: current step-specific input By applying prefix decoupling and semantic alignment to the static and semi-static layers, cache hit rates stabilize at 90%–99% in multi-turn loop tasks. The savings aren't just monetary—TTFT (Time-To-First-Token) drops significantly as well. 2. Lossless Context Compression Raw JSON returned from tools often contains redundant fields, null values, and overly verbose descriptions. The gateway performs lightweight AST-level pruning and semantically lossless compression before forwarding—filtering out noise while preserving the complete reasoning chain. In practice, this shows no negative impact on model judgment accuracy. 3. Output-Side Constraints & Truncation In many cases, we only need the agent to return a short structured action, but models tend to over-explain with lengthy filler. The gateway enforces strict stop sequences and dynamic token truncation at the protocol layer, cutting over 50% of ineffective output tokens. On Data Privacy & Cost Verification I personally detest third-party relays that store intermediate data, so from day one this gateway was designed with a hardline principle: Pure Stateless: streaming and in-memory transformations only—never persists any prompts or business data to disk, released immediately after each run. Bring Your Own Key: directly configure your official API keys; requests go straight to providers. To verify token savings, just compare usage curves and historical billing in your provider dashboards. Currently, unified definitions are wrapped with adapters for 8 mainstream LLMs(seamless switching across popular closed-source and open-source APIs). Across the entire pipeline, real-world agent token costs are typically reduced by 50%–90%. Since this is a personal project with limited bandwidth, I'd love to bring in a few fellow engineers who are also heavily into agent development and have felt the sting of inflated bills—to help test extreme concurrency and long-context edge cases. If you uncover critical bugs or provide actionable architectural feedback, I'll grant permanent access afterward. Drop a comment if interested, and I'll share the setup config and test keys.