General
Share and discuss tech, products, business, startups, or product recommendations
Start new thread

Anyone else building AI features without relying on external LLM APIs?

by•

Hi all, there's a distinction I keep thinking about: apps built with AI (Cursor, Claude, Copilot writing the code) vs apps that depend on AI APIs at runtime to function.

I'm in the first camp. I used AI-assisted coding heavily to build Briskly (a cleaning app that scans your room and generates a timed task list), but the app itself makes zero LLM API calls. The scanning runs on Apple's Vision framework, entirely on-device. No tokens, no per-user inference costs, no latency, works offline, and my privacy label is basically empty.

Part of that was a deliberate bet: token costs for API-dependent apps feel unpredictable, and as a solo dev I didn't want my margins tied to a provider's pricing page. Every user costs me nothing to serve. I also hate the idea of sending images externally to an LLM.

But I'm aware I'm trading away capability. A cloud LLM could do far smarter things than my on-device pipeline can.

So, questions for anyone further down this road:

  1. If your app depends on external LLM APIs, how are you thinking about cost as you scale? Has pricing volatility bitten anyone yet?

  2. Anyone shipped with local/on-device models (Apple's Foundation Models, Core ML, quantised open models)? How was the quality gap in practice?

  3. Where's the line for you? Which features genuinely need a frontier model vs which are fine with something small and local?

And mostly: I'd love examples. What are the best shipped apps you've seen running LLMs locally or on-device? I want to study a few before I build my own local AI features, so drop names, even your own app. Especially your own app, honestly, self-promo welcome if it's actually running local inference.

Curious what people have found, especially other solo devs where inference cost comes straight out of your own pocket.

104 views

Add a comment

Replies

Best

Not running local, so can't speak to on-device but pricing volatility has bitten me for real. Cerebras announced deprecating a model I had in my main chain, on a deadline, same week they were also killing their free tier. Wasn't a "maybe someday" scare, it was "this breaks in 30 days."

Ended up walking away from that provider entirely rather than adding a card, and built a small internal gateway that reads model configs from a JSON file with deprecation dates baked in — so a provider killing a model doesn't take down a live feature, it just silently routes around it. Also added hard budget caps per provider so a traffic spike can't turn into a bill I didn't see coming.

Doesn't solve the margin question the way on-device does I'm still fully exposed to whatever providers charge but it turned "provider does something and my app breaks" into a non-event. If you do end up API-dependent for anything, that resilience layer is worth building before you need it, not after.

 oh wow the 30 day deadline thing is exactly the scenario that scared me off - I was worried about prices creeping up, hadn't even considered a provider just killing a model with a deadline. that's worse. The gateway idea is really smart tbh, deprecation dates baked into the config so it routes around dead models automatically. hard budget caps too - although a traffic spike turning into a surprise bill is one problem I'm fairly safe from, my users could triple overnight and the invoice would still be £0 (the number of users also helps 😄) saving all of this for if I ever do need an API for something - thanks for sharing it properly and not just saying "it depends"

 The deadline part really is the underrated risk price creep you can see coming and budget around; a provider just pulling a model out from under you with a countdown is a different kind of scramble. And fair, zero infra cost when it's £0 either way is a genuinely nice place to be no bill to protect against is its own kind of resilience.

Good luck with Briskly, and if you ever do need an API for something specific, the config-with-deprecation-dates pattern scales down fine too even a single provider benefits from not hardcoding a model name you'll have to hunt down and change everywhere the day it disappears.

Thank you! I will keep that in mind!
Hey Maria, this is the first Product Hunt post I read after creating my account, and the focus on on-device inference is exactly what sparked my interest. We built Talking Friday, a full speech-to-speech AI that runs entirely on the iPhone Neural Engine. No external LLM APIs at all. We ship three custom quantized models inside the app so the whole loop (listening, thinking, speaking) stays local. Privacy by design was a hard requirement for us: the voice never leaves the device, even for a second. The biggest constraint we keep running into is Apple’s memory limit for a single app. As conversations get longer the memory requirement climbs, and that has been our main engineering challenge so far. We’re planning a UX update next. If anyone wants to try the current version it’s at talkingfriday.com Curious how others are handling similar memory or size limits with local models!

 this sounds really cool! Also, three models running locally on the phone is pretty impressive. And yeah, the privacy thing is why I avoid voice input in the first place. I don't have the same problem to be fair. Mine's Vision framework doing image classification, so it's a quick burst with nothing building up over a conversation, just the accuracy isn't always the best. Your usecase sounds like a much harder version of it than anything I'm dealing with.

Bit outside my lane on the memory side, but I was wondering about the summarise-and-reset approach where you compress the conversation so far to clear the cache. Probably already on your radar. If it does mean a pause, that feels like a design problem as much as an engineering one - a voice assistant going quiet for a couple of seconds reads as broken unless something tells the user why. Might be less painful than trying to squeeze the memory down.

I like that you made the architecture decision based on long term sustainability, mot just what's easiet to build today. That's refreshing.

 it was more of a happy accident tbh. I knew it would cost me even to test (very little but still) - the accuracy of the scan isnt brilliant, and ive tried to improve it with different workaround and experimented with the system prompts but my plan next is to utilise Apple's newest Foundation Models (iOS 26) to make the task planning smarter - still on device so still free. we'll see if it works 😄