Empromptu AI - Train Fine Tuned Models With AI Apps You're Already Building
by•
Most AI apps launch on someone else’s model and stay there forever. Empromptu AI turns live AI features into custom models you own. As your app runs, Empromptu AI captures real-world usage, human corrections, and edge cases from live AI workflows, then uses that signal to train a custom model you own. Improve accuracy, lower inference costs, and stop depending forever on rented intelligence from the same providers moving into your category.
Replies
The idea of turning your own app's usage into fine tuning data is genuinely clever but how do you handle the cold start problem for teams whose apps don't yet have enough interaction volume to generate meaningful training signal?
@andrew_paul11 that's the beautiful thing about the app creating the data. The very first time you run the app it creates data!
@andrew_paul11 The part that makes that work under the hood is that even a small number of high quality runs beats a large volume of synthetic data. Because the signal comes from real interactions against your actual eval, even early data is already shaped around your domain. You're not waiting for volume, you're waiting for signal and those are very different thresholds.
@andrew_paul11 From a true cold start you generally will want to use the SOTA frontier model to get the best results you can for first use cases. Then once you've built up a small corpus you can use that to fine tune your own model and get improved results for lower cost at volume compared to always using the off the shelf frontier models.
@andrew_paul11 There are a few ways, including generating simulated responses, or actual responses using calls that mimic an array of user intents to create that volume, and in each case we incorporate the way the user interacts with the training data set to help us understand what "good" looks like in terms of training directionality.
@hellovidya thanks! We all know that data in today's world is gold! We feel that everyone should be able to turn data into gold 😁
@hellovidya @shanealeven yes, that name is gold!
@hellovidya Thank you!
@hellovidya Thanks!
Can you walk us through what AI apps you're already building means in practice are you integrating with specific frameworks like LangChain or LlamaIndex?
@ana_popescu2 no. We built our own proprietary frameworks specifically designed to get highly accurate AI outputs. So many of our users build their entire products on our platform like we have had users build Soc 2 platforms, AI tax products, full healthcare products for their communities. We have a grant program winner who is building AI workflows for airlines! This is sophisticated tools that ml and ai engineers use made accessible for everyone
@ana_popescu2 To double click on Shanea's point, the proprietary architecture is what makes those wildly different use cases possible on the same platform. The fine tuning loop needs to close reliably whether you're building a healthcare product or an airline workflow, and that's a very different design requirement than general purpose orchestration frameworks are built for.
@ana_popescu2 No - we have a full builder, supported by function-focused templates in our new Studio area in the platform (studio.empromptu.ai), the planner, and the Builder, all of which dramatically accelerate time-to-market.
We also enable people coming from other builders like Lovable or Claude Code or Codex, where you can / probably do have a Github Repo connected, to just simply connect that repo to Empromptu and begin integrating their functions to our vertical-integration & optimization stack.
@jordan_hanson1 congrats guys, about the 98% figure. Is that something customers typically achieve after the model has learned from their application data, or is that the starting point?
It would be interesting to understand what the baseline was before the learning process.
@jordan_hanson1 @xavair yes! It is. Whats unique about our platform is that we can agenticly get really high accuracy rates as we built a custom model ourselves to make corrections in real time dynamically. But the last percent you can correct responses. So because these tools are integrated into one platform you really get very high accuracy rates
@jordan_hanson1 @xavair Building on what Shanea said, the 98% is what the system converges toward over time, not where it starts. The baseline depends on how far the foundation model is from your specific domain out of the box. The more specialized your use case, the bigger the delta and honestly the more dramatic the improvement curve once your own production data starts feeding the loop.
@xavair This data comes from a few different examples, but I think instead it's better to simply reframe: think about the accuracy gains in terms of opportunity cost, where 30% is always relatively better. Our platform optimizes toward the outcomes the user specifies, and the training data helps us understand and contextualize the decisions and edge cases that make each individual model characteristically unique.
In my experience, I've found that the foundation builder assistants sort of lose things by contrast to the degree that now when I'm building outside Empromptu, I feel like my personal environment is missing things that I try to build, but they're always worse off and they always break. The contrast, again, is that the functions I build on the platform work, because they're built and deployed at production-grade standards, and that extends to the way the model is being refined in real time, as I help train it on edge cases, because I find it needs me to nudge and correct it less and less.
Fine tuning from live app behavior raises an interesting data quality challenge what mechanisms does Empromptu use to filter out bad or edge case interactions that could quietly degrade model performance over time?
@antonio_manuel1 this is really a New paradigm. Subject matter experts can self-correct data to always ensure performance is high for the first time alongside a tool that feels like you're simply vibe coding
@antonio_manuel1 This is exactly the right question to ask and honestly one of the harder engineering problems we solved. Raw production data is noisy by nature so we never let it flow directly into training.
Every interaction gets scored against the eval you defined upfront. That eval is your ground truth and anything that doesn't meet the quality threshold gets filtered before it touches the training pipeline. Edge cases don't get discarded though, they get flagged for SME review because edge cases are often where the most valuable signal lives. The difference is they go through a human in the loop step before they become training data.
The other layer is the SME override architecture Shanea mentioned earlier. When experts disagree on a correction, the eval arbitrates. You're never training on conflicting signal, you're training on verified ground truth.
The result is a labeled dataset that stays small and clean over time rather than growing noisy with volume.
@antonio_manuel1 The most common way is that the way you train the model gives us some sense of directionality, but we also have some proprietary systems that help us optimize toward 'good' automatically.
Woo, love seeing this ship! Already mulling through some of the fun stuff I could add to my companies in terms of being able to fine tune some models, hah.
@holman amazing! Would be super curious to see what you build!!
@holman let us know how we can support you!
@holman Woo! We're so excited too!
the feedback loop approach is smart. the part that usually trips teams up isnt the training pipeline though, its the quality of the corrections feeding it. if the humans correcting the AI output dont have a systematic way to evaluate whats actually wrong you end up fine-tuning on noise. curious how you handle that signal quality problem
@ozandag The correction itself is not the signal, the correction scored against a defined expected outcome is the signal. Without that layer you are just fine tuning on whoever had an opinion that day.
The eval is defined upfront by the SMEs who know what correct looks like. Every correction gets scored against it before it touches training, filtered if it doesn't meet the bar, flagged for review if it's ambiguous. When experts disagree the eval arbitrates. You never train on conflicting signal, only verified ground truth.
@ozandag +1 to what Sean said. We allow the user to first define what good looks like. and we actually remove the hard parts so they can define it in natural language. Often with a single statement. No configs or files so anyone can do it. Then we measure accuracy towards that goal.
@ozandag The pipeline's the easy half. We score corrections before they hit fine-tuning, so a sloppy "this is wrong" doesn't weigh the same as a structured one. Bad corrections are their own failure mode and most teams don't instrument for it. What surfaced this for you?
I keep thinking about how much institutional knowledge disappears when someone leaves a company. Most organizations have years of expertise locked inside conversations, corrections, and unwritten rules. The idea of turning those signals into a continuously improving system feels like a much bigger opportunity than no-code app building itself.
@mehmet_s_taskesen This is exactly the framing that drove the original architecture decision. The problem was never that foundation models were bad. It was that every organization was essentially starting from zero every time because there was no infrastructure to capture what their best people actually knew.
The accountant who knows the exception. The support lead who knows when an escalation is real. That knowledge compounds inside a person over years and then walks out the door. Alchemy is fundamentally an infrastructure problem solved, not an AI feature added. The signal was always there in the corrections, the edge cases, the judgment calls. It just had nowhere to land permanently.
The no code angle is actually secondary to us. What we are really building is the layer that turns institutional knowledge into a durable asset that survives the people who created it.
@mehmet_s_taskesen we totally and completely agree. Better yet. What if you could monetize it as well?
@mehmet_s_taskesen Right? this is the "what if [any person you rely on] gets hit by a bus problem" -- just SOLVED by AI systems that can basically build / learn / do anything you can plan and train.
One concern enterprises always raise around fine tuning pipelines is data residency can you speak to how Empromptu isolates customer training data and whether it ever touches shared infrastructure?
@carlos_leonardo1 Data isolation is non negotiable for the customer segments we serve. Healthcare organizations and financial workflows were in production on Empromptu before we ever talked to a general market so the architecture was built around those requirements from day one, not retrofitted later.
Every customer's training data is fully isolated. Your corrections, your edge cases, your labeled dataset never touch shared infrastructure and they never inform another customer's model. The model you train is trained exclusively on your data and the weights are yours.
The broader point is that data residency is actually core to the product thesis. The reason Alchemy exists is that your data should build your asset, not someone else's. Letting customer training data bleed across tenants would contradict the entire value proposition.
@carlos_leonardo1 we take privacy and data very seriously
@carlos_leonardo1 your data is your data! we don't mix up those things, and we believe it so strongly that we have created a pathway for you to train weights that you own (and even export them / host them somewhere else, if you needed to, for whatever reason).
How does Empromptu approach the tricky intersection of user privacy and training data collection specifically how do you help developers stay compliant when end users haven't explicitly consented to having their interactions used for model training?
@new_user___10520260379921a76fc2d64 Great question!! We've actually built in a data anonymizer! to randomize any PII before it goes into model training.
@new_user___10520260379921a76fc2d64 @shanealeven Building on that, the anonymizer runs automatically before any interaction touches the training pipeline so PII never reaches the labeled dataset in the first place. The compliance burden on the developer side is significantly reduced because the infrastructure handles it at the platform level rather than requiring custom data handling logic in every application.
@new_user___10520260379921a76fc2d64 we have several mechanisms to securely protect end-user data, and also offer the ability to mechanically substitute the 'data' from the 'identifying components' through 'john doe' substitutes that makes PII attribution impossible, especially in training applications.