shad0w watches your LLM answer the same small questions (which intent, which team, spam or not), learns the answers, and gives them back in microseconds. It only answers what it's proven to agree on; the rest goes to your LLM. Open source 🐍

How did you use Cloud Run for this launch?

The plan is a public decision API on Cloud Run: the real 77-intent BANKING77 table from the playground, scaling to zero between requests, so anyone can try shad0w over HTTP without installing anything. A second service will run the proxy in front of a team's own LLM, so the repeat questions it learns are answered by the container in microseconds instead of being billed as model calls.

Hey Product Hunt 👋 I'm Gaurav. For months one number bugged me: 1.4 seconds. That's how long our LLM took to answer "which intent is this message?" Not a hard question. The same one it had answered thousands of times. 🔁 So I built shad0w, my LLM's shadow. It sits in front of your LLM and quietly watches its answers. Once it has seen enough (about 1,000), it compiles them into a tiny table and starts answering the repeats itself, in microseconds, on a plain CPU. Anything it isn't sure about still goes to your LLM, and that answer gets logged so it keeps learning. The moment that made me grin: 🐍 I let it learn Snake just by watching an LLM play. At 10 moves a second the LLM can't keep up and crashes into the wall. The 2 KB table it learned from that same LLM keeps eating. You can watch the real recordings here: https://2796gaurav.github.io/sha... Or try to beat it at Pong, right in your browser (it decides every frame in 1.5 µs): https://2796gaurav.github.io/sha... The part I care about most: it knows when it doesn't know. Before it answers anything, it's tested on answers it never saw, and it only answers where it agrees with your LLM 95%+ of the time. On fuzzy stuff like movie-review sentiment it answers 0% and hands everything back. That's on purpose. I call it System 0: System 2 thinks, System 1 decides, System 0 remembers. It's free and open source (Python, JavaScript, C). Using the OpenAI Decisions API? It's one URL change. 🎬 2-minute explainer: https://2796gaurav.github.io/sha... 💻 Code: https://github.com/2796gaurav/sh... I'd love to know: what small question does your app ask its LLM over and over? Atraffic, tell me where it breaks. I'm here all day. 🙏

How did you use Cloud Run for this launch?

shad0w

System 0 for your LLM: repeat answers in microseconds

System 0 for your LLM: repeat answers in microseconds