We know what our users are feeling. That's the most dangerous thing about building an AI product.
When someone journals on Murror, our AI doesn't just process text. It reads emotional weight. It picks up on patterns the user might not see yet -- the way their language shifts when they talk about work versus family, the recurring themes they circle back to every few weeks, the gradual change in tone that might signal something deeper.
We built this because it makes the product better. The AI can ask more relevant questions, create more meaningful reflections, and know when to give space versus when to gently prompt.
But here's what keeps me up at night: we know things about our users that their closest friends don't know.
This isn't like a social media company knowing your shopping habits. This is an AI that has read someone's rawest, most vulnerable thoughts -- their fears about their marriage, their struggles with self-worth, their grief they haven't told anyone about.
The temptation for any product team would be to use that depth of understanding to maximize engagement. You could send perfectly timed notifications when someone is most likely to journal. You could surface insights that create emotional dependency. You could make the AI so attuned that users feel like it understands them better than any human does.
We made a deliberate decision not to do any of that.
Instead, we built what we internally call "ethical forgetting." The AI uses emotional context in the moment to be helpful, but it doesn't build a persistent emotional profile to optimize against. It doesn't use vulnerability patterns to drive engagement metrics. And when someone is processing something really heavy, we deliberately reduce how "smart" the AI appears -- because the last thing someone in pain needs is to feel like they're being analyzed.
The hardest product conversations we have aren't about features or growth. They're about the line between "understanding users to help them" and "understanding users to keep them."
I think every AI product that deals with personal data is going to face this question eventually. The companies that get it right won't be the ones with the best AI -- they'll be the ones who chose not to use everything their AI could do.
Anyone else wrestling with the ethics of what your AI knows about your users?
Replies
This is one of the most honest thing I've read from a founder about AI ethics.I build an inventory management tool so the stakes are much lower but even there I think about what I do with store data. Sales patterns, revenue, which products are struggling. That data could theoretically be used in ways the store owner never intended.
The "ethical forgetting"concept ia interesting. The default in most products is to accumulate and use everything. Choosing not to is a deliberate product decision that cost you something. The line between "understanding users to help them'' versus "understanding users to keep them" is the clearest way I've heard that distinction explained.
@debnix I think that's exactly where the responsibility gets real. If I can understand how users feel, I can help them better but I also need to make sure that understanding is never used to manipulate them.
@malani_willa Exactly. And what makes it harder is that most teams would never frame it as a choice at all.
They would just call it personalisation and move on. Naming it clearly — this is manipulation versus this is help — is the harder step.The fact that you named it and chose the harder path is the interesting part. Most products never even have that conversation internally.
Murror
@debnix Thank you for seeing that distinction so clearly. You're right that the default in most products is to accumulate everything -- and honestly, even with inventory data, those patterns can reveal more about store owners than they'd expect. The fact that you're already thinking about it at lower stakes means you'll be ahead when the stakes get higher.
The cost of "ethical forgetting" is real for us. Our AI sometimes seems less impressive than it could be because we're deliberately holding back. But we'd rather build something people trust in 5 years than something that impresses them today.
PopShort.Al -Stream Short Drama
As builders, I don't think the scary part is that AI knows more. The scary part is "What incentives are we giving the people who control it?"
Murror
@elara_thorn That's a great reframe. The question isn't really "what does the AI know" -- it's "what are the incentives of the people building it." We try to align our incentives by making Murror work better when it respects boundaries, not despite them. If our product only succeeds by exploiting emotional data, then we've built the wrong product. The restraint has to be structural, not just policy.
This is a strong framing. The phrase that stands out to me is “chose not to use everything their AI could do.”
One practical version of that might be making the restraint legible to the user without turning it into a scary privacy dashboard. For example: “used for this reflection only,” “not saved as a profile,” “not used to time prompts,” or “intentionally simplified because this topic looks sensitive.”
The hard part is that invisible restraint builds ethical integrity but not always user trust, because users can’t see what didn’t happen. So the product question becomes: how do you show enough of the boundary to make people feel protected, without making them feel watched?
Murror
@jim_jeffers You're naming the exact tension we live in. We've been experimenting with light contextual cues -- things like "this reflection won't be used to adjust your prompts" or "this entry stays in this session only." Small signals that make the boundary visible without turning the whole experience into a privacy dashboard.
The goal is for users to feel safe, not monitored. It's a subtle but important difference. We've found that when we over-explain the restraint, it actually backfires -- people start wondering what we could be doing with their data. The best version is when the product just feels trustworthy by default and the transparency is there if they go looking for it.