Anyone else building AI features without relying on external LLM APIs?
Hi all, there's a distinction I keep thinking about: apps built with AI (Cursor, Claude, Copilot writing the code) vs apps that depend on AI APIs at runtime to function.
I'm in the first camp. I used AI-assisted coding heavily to build Briskly (a cleaning app that scans your room and generates a timed task list), but the app itself makes zero LLM API calls. The scanning runs on Apple's Vision framework, entirely on-device. No tokens, no per-user inference costs, no latency, works offline, and my privacy label is basically empty.
Part of that was a deliberate bet: token costs for API-dependent apps feel unpredictable, and as a solo dev I didn't want my margins tied to a provider's pricing page. Every user costs me nothing to serve. I also hate the idea of sending images externally to an LLM.
But I'm aware I'm trading away capability. A cloud LLM could do far smarter things than my on-device pipeline can.
So, questions for anyone further down this road:
If your app depends on external LLM APIs, how are you thinking about cost as you scale? Has pricing volatility bitten anyone yet?
Anyone shipped with local/on-device models (Apple's Foundation Models, Core ML, quantised open models)? How was the quality gap in practice?
Where's the line for you? Which features genuinely need a frontier model vs which are fine with something small and local?
And mostly: I'd love examples. What are the best shipped apps you've seen running LLMs locally or on-device? I want to study a few before I build my own local AI features, so drop names, even your own app. Especially your own app, honestly, self-promo welcome if it's actually running local inference.
Curious what people have found, especially other solo devs where inference cost comes straight out of your own pocket.
Replies
I use external APIs today because they are easier to get started with, but I keep wondering if moving some features locally would make more sense over time.
@yosun_negi same reason I went the other way tbh - APIs are way easier to start with, I was just too scared of the bills 😄 if it helps, local was less painful than I expected. Accuracy and changing the system prompt on the other hand took most of the work.
@mariabuildsbriskly Not running local, so can't speak to on-device but pricing volatility has bitten me for real. Cerebras announced deprecating a model I had in my main chain, on a deadline, same week they were also killing their free tier. Wasn't a "maybe someday" scare, it was "this breaks in 30 days."
Ended up walking away from that provider entirely rather than adding a card, and built a small internal gateway that reads model configs from a JSON file with deprecation dates baked in — so a provider killing a model doesn't take down a live feature, it just silently routes around it. Also added hard budget caps per provider so a traffic spike can't turn into a bill I didn't see coming.
Doesn't solve the margin question the way on-device does I'm still fully exposed to whatever providers charge but it turned "provider does something and my app breaks" into a non-event. If you do end up API-dependent for anything, that resilience layer is worth building before you need it, not after.
@farrukh_ahmed8877 oh wow the 30 day deadline thing is exactly the scenario that scared me off - I was worried about prices creeping up, hadn't even considered a provider just killing a model with a deadline. that's worse. The gateway idea is really smart tbh, deprecation dates baked into the config so it routes around dead models automatically. hard budget caps too - although a traffic spike turning into a surprise bill is one problem I'm fairly safe from, my users could triple overnight and the invoice would still be £0 (the number of users also helps 😄) saving all of this for if I ever do need an API for something - thanks for sharing it properly and not just saying "it depends"
@mariabuildsbriskly The deadline part really is the underrated risk price creep you can see coming and budget around; a provider just pulling a model out from under you with a countdown is a different kind of scramble. And fair, zero infra cost when it's £0 either way is a genuinely nice place to be no bill to protect against is its own kind of resilience.
Good luck with Briskly, and if you ever do need an API for something specific, the config-with-deprecation-dates pattern scales down fine too even a single provider benefits from not hardcoding a model name you'll have to hunt down and change everywhere the day it disappears.
I like that you made the architecture decision based on long term sustainability, mot just what's easiet to build today. That's refreshing.
@yahya_rogers it was more of a happy accident tbh. I knew it would cost me even to test (very little but still) - the accuracy of the scan isnt brilliant, and ive tried to improve it with different workaround and experimented with the system prompts but my plan next is to utilise Apple's newest Foundation Models (iOS 26) to make the task planning smarter - still on device so still free. we'll see if it works 😄