Framer AI AgentsDesign and publish professional sites with AI
Promoted
ngrok AI Gateway
One private gateway for every AI model
658 followers
One private gateway for every AI model
658 followers
ngrok AI Gateway provides one hosted gateway for every model: public providers, custom endpoints, and the models you run yourself. Use one key and one URL to route across OpenAI, Anthropic, and self-hosted models with observability, access control, and fallbacks built in. Your private models connect through ngrok’s network, so they sit beside hosted providers without being exposed to the public internet.
Payment Required
Launch Team / Built With
Ngrok
Hey Product Hunt 👋
I'm Niji, a product manager at ngrok. Today we're launching ngrok.ai, ngrok's AI Gateway.
For years, ngrok has helped developers connect their applications and services in minutes instead of days.
As I started building with AI, I ran into similar infrastructure problems at the application layer.
An application might start with OpenAI, then Claude for another use case. As newer, faster, or more affordable models became available, I would create more accounts and update my code just to try them. Eventually, more specialized needs would lead me to run fine-tuned or task-specific models on my laptop, private GPUs, or internal cloud infrastructure.
Before long, I was managing multiple gateways and SDKs, sharing provider keys across configuration files and vaults, checking usage in several dashboards, maintaining complicated fallback logic, and accidentally exposing models that were supposed to remain private.
If any of this sounds familiar, it is why we built ngrok.ai. It gives you one hosted gateway for managing models across providers, private infrastructure, and your own hardware.
One URL for every model
Getting started is simple. Point your SDK at https://gateway.ngrok.ai with your ngrok.ai access key, and begin routing requests to public providers, custom endpoints, and models you run yourself.
It works with all popular SDKs like OpenAI, Anthropic, and Vercel AI, so you can easily swap models and providers without rebuilding your entire application.
Aside from being a hosted AI Gateway, we enable you to:
Connect self-hosted models privately
Route to a model running on your laptop, local GPU, or private network without complex networking, opening inbound ports or dealing with IPs.
Build fallbacks into the gateway
Define a list of models and when a model or key fails, we will make another attempt or route the request to a healthy alternative.
Use credits to make requests
Leverage ngrok.ai to make requests against OpenAI, Anthropic, z.ai and more without having to create your own accounts with each provider.
Use your existing provider keys
Don't want to use our accounts? No worries, you can bring your own OpenAI, Anthropic, or custom provider keys that you already.
Control access by application or developer
Create separate access keys and decide which providers and models each one is allowed to call, and which keys each model should use, whether ours or yours.
See usage across your entire model stack
Track tokens, latency, errors, models, providers, and estimated cost in one place instead of piecing together several provider dashboards.
Manage everything through the dashboard or API
Set up gateways, keys, providers, access rules, and routing from your own tooling using our API or directly in the ngrok.ai dashboard.
Who we're building this for
ngrok.ai is for developers and platform teams that want the freedom to use the right model for each job without worrying about how to scale and maintain an ai gateway themselves and or taking on another infrastructure project every time their model strategy changes.
We're especially interested in hearing:
How are you routing between models today?
Are you running any models on your own infrastructure?
Which gateway features would make your AI stack easier to manage?
We'll be here throughout the launch to answer questions and hear what you think. Thanks for checking it out.
Mastra
@ngrok is so back.
If you've ever needed to share a local app or demo something live, there's one thing that strikes about this product. Developers love it. I mean millions of developers - look at this wall of love - including people at @GitHub, @OpenAI, @Vercel, and much, much more.
@cassidoo put it simply: "ngrok saves me so much time during development. What a tool."
pumped to see them back [on Product Hunt]. S/O ?makers, already looking for your future launches.
Vamah.ai
@cassidoo @fmerian ngrok has been core in my dev setup for a very long time to test real world app scenarios during development. one gateway for every hosted model sounds interesting.
Mastra
@gokuljd lfg! if you get a chance to give ngrok.ai a spin, make sure to add your review here: producthunt.com/products/ngrok-ai-gateway/reviews/new
ngrok AI Gateway
@gokuljd Thanks so much for using ngrok over the long haul! If there's anything you'd like to see fixed or have us add (aside from this AI Gateway, of course!), I'd love to hear about it.
the fallback is the feature i would think hardest about, because for anything that produces text a fallback is a silent quality change.
if a code call falls back you find out, something breaks or the tests go red. if a customer facing reply falls back to a weaker model, nothing breaks. the reply still reads fine, it is just slightly worse, and you find out from a complaint two days later while looking at the wrong model in your logs. the gateway is the only component that knows which model actually answered, so i would want that on the response itself rather than only in a dashboard.
the other one is retries. a fallback triggered by a timeout is not the same as one triggered by an error, because on a timeout the first call may well have completed on the provider side. harmless for a plain completion, not harmless once a tool call is attached to it, which is most agent traffic now. does the gateway treat timeouts as retryable by default, and is there a way to mark a request as do not retry?
ngrok AI Gateway
@jernej_jan_kocica This is really great feedback.
For your first point—I've always found that responses from the APIs of public providers like OpenAI and Anthropic contain some kind of `model` key that you could build logic around in your apps. It'll get flagged in the dashboard, just like you asked for, but you could build some error handling/logs around what happens when that fallback model is the one that responds. We'll dig into this one some more and see if we can add some headers at the gateway your app could also consume.
On your second question, timeouts only trigger retries if you've specified fallback models though the `model` key. If there isn't one, you just get an error, which is kind of akin to marking it as do not retry. This does feel like something we should look into implementing as a very clear attribute of any provider setup.
@joel_hans2 both of those are good answers, thanks for actually digging in.
one refinement on the do not retry attribute: i would put it on the request rather than on the provider setup. whether a call is safe to retry depends on what the call does, not on where it goes. the same model at the same provider will serve an idempotent summarise in one code path and a tool calling turn in another, and if the flag lives on the provider config you end up duplicating the whole provider just to express that difference. a header on the request costs nothing and lets the caller say it, and the caller is the only one who knows.
the model header on the response is the one i would ship first though. it is the difference between debugging what you configured and debugging what actually happened.
What strikes me most is the self-hosted piece. I've run models on my own infrastructure before and connecting them through it's network instead of exposing a public endpoint solves a real security headache I've dealt with firsthand. That alone makes this worth testing on my end.
ngrok AI Gateway
@itohan_blessing_eigbadon Thanks for the comment! We have people interested in exactly what you're saying, production apps needing private connectivity to self-hosted inference, but we're also seeing a newer trend of people wanting that same simple infra for connecting their coding agents to self-hosted models, which is pretty dope. It's kinda the panacea if you care about software that's open, flexible, and "ownable."
We created a guide for Cursor, OpenCode, Pi, and Zed: https://ngrok.com/docs/ai-gateway/guides/use-with-coding-agents
Would love thoughts on what else we should cover!
Mastra
yes, and make sure to add your review here: producthunt.com/products/ngrok-ai-gateway/reviews/new
ngrok AI Gateway
@itohan_blessing_eigbadon
a big factor that drove us (ngrok) to decide to build an AI Gateway is that we thought that by provided a hosted offering paired with our already widely adopted and battle tested tunnels/endpoints, we could give folks an easy way to securely connect any compute resources they have hosting local models, whether that's a bunch of small machines running on your desk like the mac mini, or other cloud based hosted compute environments. we're still hard at work making the existing features better / easier to use, and and adding new ones each week :)
Serand
How easy is it to switch traffic between providers if one model has highest latency or goes offline?
Mastra
good q. @ngrok AI Gateway is designed for automatic failover. if one provider/model fails, times out, returns an HTTP error, or has a connection issue, the gateway automatically tries the next candidate instead of requiring your app to switch providers itself.
read the docs for more details on how it works: ngrok.com/docs/ai-gateway/how-it-works
hope it clarifies!
ngrok AI Gateway
@anthony_adams_ Thanks for the question! Well, I think it's pretty darn easy. @fmerian got the gist of it, but here's a bit more specificity. In your app (I'll use a JS example), you already specify a model, but now you can add a models key to list the order in which you'd like to fall back to other providers and models.
In this case, the AI Gateway will try Fable 5 first, then retry the same request with Luna if Fable isn't working (kinda like right now, with Anthropic's API on the struggle bus). Here's the doc: https://ngrok.com/docs/ai-gateway/guides/configure-fallback-models
You can also fail over between providers or models if your key stop working, like if you run out of precious, precious credit with one: https://ngrok.com/docs/ai-gateway/guides/key-selection-failover
ngrok AI Gateway
@anthony_adams_
we've got options to per model timeouts to limit how long you wait for one of them to respond before moving on to the next one, options to configure fallback behaviour with models and/or keys if you're bringing your own keys.
on our roadmap is a way to let users define their own model groups or pick an automatic group of models by category. for example:
- the most responsive models
- models with the highest fidelity output
- models good for a specific task such as programming / research / etc.
if you've got a particular pain that's not being solved or a feature that you'd like to see added, we'd love the feedback.
ngrok AI Gateway
@anthony_adams_ If one goes offline, it'll fail over automatically and retry the request if you've set both a default `model` and one or more other `models` in your request through an SDK. Here's the full doc on how it works: https://ngrok.com/docs/ai-gateway/guides/configure-fallback-models
Voquill
I like that you can start with your own provider keys and still get a single place to track usage, latency, and costs. Jumping between multiple dashboards gets old fast, so having that consolidated is a practical improvement.
ngrok AI Gateway
@henry_habib Yep yep, you have complete control over whether you bring your existing providers keys (BYOK) or use ngrok-managed ones. Either way, it all ends up on the same request/usage logs.
We actually just added more detailed breakdowns of usage data we were already collecting, so now it's really easy to see things like the average cost per request or per-provider failure rate. This is me just playing around with a few requests this morning (and having forgotten to bring my local Ollama model online before sending the request):
I've been looking for a better way to experiment across OpenAI and Anthropic without constantly rewriting integrations. This feels like it could make testing significantly faster.
ngrok AI Gateway
@sitara_ismail Totally, even a "simple" switch like Open AI<>Anthropic also means using different SDKs and dealing with different API shapes, and none of that's trivial to just make happen. It's all cognitive load and more code. There's also a playground in the dashboard that would let you do some freeform testing without having to edit your app at all until you're ready.
ngrok AI Gateway
@sitara_ismail Thanks for the support! Would love to hear what your testing for and how we could layer in some new features to make that dead simple.
Mastra
yes! make sure to add your review here: producthunt.com/products/ngrok-ai-gateway/reviews/new