Launched this week
ngrok AI Gateway
One private gateway for every AI model
777 followers
One private gateway for every AI model
777 followers
ngrok AI Gateway provides one hosted gateway for every model: public providers, custom endpoints, and the models you run yourself. Use one key and one URL to route across OpenAI, Anthropic, and self-hosted models with observability, access control, and fallbacks built in. Your private models connect through ngrok’s network, so they sit beside hosted providers without being exposed to the public internet.
Payment Required
Launch Team / Built With
Hedy AI
I like that private models can stay off the public internet while still working alongside hosted ones. That's a thoughtful approach.
ngrok AI Gateway
@kate_sleeman Yup yup. Just knowing that no one random can access a self-hosted model on a URL like https://my-precious-model.internal is a huge W, and then you still get to configure access a bunch of different ways.
ngrok AI Gateway
@kate_sleeman as a bonus, you can monitor usage and requests of a self-hosted model using the same ui as the hosted alternative, providing a one-stop shop for all your models and inference!
I can definitely see this being for teams experimenting with different models. Have one plaace to manage everything sounds much easier. 👍
ngrok AI Gateway
@zeeshan_aslam2 Yeah, you can definitely experiment by changing the `model` key in your app where you're calling the inference providers, checking the output, and seeing what feels right.
We also have a playground where you can send the same prompt to multiple models and see the differences right away (plus status on TTFT and overall speed!): https://app.ngrok.ai/playground
TrackerJam
I'm curious how smooth the setup is for someone already using multiple AI providers. Is migration pretty straightforward?
ngrok AI Gateway
@maklyen_may we worked hard to make the setup as seamless as possible. we support the inference apis from anthropic and openai, so it should be a drop in replacement for most sdks.
here's our docs page on how to set that up: https://ngrok.com/docs/ai-gateway/overview#quick-example
ngrok AI Gateway
@maklyen_may @brenden_ehlers
native gemini API support and cross-API translation are some other things on our near term roadmap, so stay tuned for those features :)
Kilo Code
yes, and it's worth mentioning this quick, 5-step migration guide to move from a good ol' endpoint-based setup to app.ngrok.ai and gateway.ngrok.ai.
read this: ngrok.com/docs/ai-gateway/guides/migrating-from-endpoints
hope it helps!
How does ngrok AI Gateway help developers handle these issues compared to building their own layer? congrats team!
Kilo Code
ngrok AI Gateway
@hamza_afzal_butt
as with most software products, if you know how to write software and have the budget to host/distribute it, you can totally roll your own solution if your needs are simple.
similar to a traditional API gateway, if all you need to do is proxy requests, that's really easy to build your own solution. once you start getting into solving for things like rate limiting, observability & reporting, request manipulation, authentication, authorization, and role-based permissions, caching, building all these things in-house compared to picking an off the shelf solution is a decision that most teams have to decide if they'd rather pay the cost of a product or the cost of maintaining their own solution. we do the same thing internally with other tools. sometimes we write our own, other times we pay for a solution.
our goal with ngrok.ai is to provide one unified interface for any kind of LLM inference need and make it super easy to use while offering the depth to solve as many of the pain/friction points as possible. we hope that the benefits that people get from using it and the problems they don't have to think about solving themselves make it worth the cost to them. we're always on the lookout though for ways to keep improving it each week :)
What I appreciate is that this isn't just about hosted providers. I've worked on internal tools where the model had to stay inside our network for compliance reasons and every gateway I tried assumed everything sat behind a public API.
ngrok AI Gateway
@morgan_nabors We fully expect some kind of AI compliance reckoning, especially in Europe, as people start to hear more about the open models and neoclouds like RunPod or Lambda AI get more appealing. We hope to be right there with folks as they search for simple ways to try those out without changing how their apps behave!
Many companies struggle with switching models during outages. The built-in fallbacks seem useful, especially if users can define custom failover rules.
ngrok AI Gateway
@khaildnaseem A big +1 on custom failover rules! Seems like that's the big undercurrent of what people are looking for here. We had a version of this before, but it was too complex and we wanted to look at it again from first principles. I'll pass this to the team... any other thoughts on how we could design these custom rules to be most useful to you?
Kilo Code
@khaildnaseem thanks for the support! if you get a chance to give ngrok.ai a spin, make sure to add your review here: producthunt.com/products/ngrok-ai-gateway/reviews/new
Managing different AI providers can get messy pretty quickly. A single endpoint with build in routing makes a lot of sense.
Kilo Code
what should they build/improve/fix from your perspective? cc ?makers