We know what our users are feeling. That's the most dangerous thing about building an AI product.
When someone journals on Murror, our AI doesn't just process text. It reads emotional weight. It picks up on patterns the user might not see yet -- the way their language shifts when they talk about work versus family, the recurring themes they circle back to every few weeks, the gradual change in tone that might signal something deeper.
We built this because it makes the product better. The AI can ask more relevant questions, create more meaningful reflections, and know when to give space versus when to gently prompt.
But here's what keeps me up at night: we know things about our users that their closest friends don't know.
This isn't like a social media company knowing your shopping habits. This is an AI that has read someone's rawest, most vulnerable thoughts -- their fears about their marriage, their struggles with self-worth, their grief they haven't told anyone about.
The temptation for any product team would be to use that depth of understanding to maximize engagement. You could send perfectly timed notifications when someone is most likely to journal. You could surface insights that create emotional dependency. You could make the AI so attuned that users feel like it understands them better than any human does.
We made a deliberate decision not to do any of that.
Instead, we built what we internally call "ethical forgetting." The AI uses emotional context in the moment to be helpful, but it doesn't build a persistent emotional profile to optimize against. It doesn't use vulnerability patterns to drive engagement metrics. And when someone is processing something really heavy, we deliberately reduce how "smart" the AI appears -- because the last thing someone in pain needs is to feel like they're being analyzed.
The hardest product conversations we have aren't about features or growth. They're about the line between "understanding users to help them" and "understanding users to keep them."
I think every AI product that deals with personal data is going to face this question eventually. The companies that get it right won't be the ones with the best AI -- they'll be the ones who chose not to use everything their AI could do.
Anyone else wrestling with the ethics of what your AI knows about your users?
Replies
This is a strong framing. The phrase that stands out to me is “chose not to use everything their AI could do.”
One practical version of that might be making the restraint legible to the user without turning it into a scary privacy dashboard. For example: “used for this reflection only,” “not saved as a profile,” “not used to time prompts,” or “intentionally simplified because this topic looks sensitive.”
The hard part is that invisible restraint builds ethical integrity but not always user trust, because users can’t see what didn’t happen. So the product question becomes: how do you show enough of the boundary to make people feel protected, without making them feel watched?
This is one of the most honest thing I've read from a founder about AI ethics.I build an inventory management tool so the stakes are much lower but even there I think about what I do with store data. Sales patterns, revenue, which products are struggling. That data could theoretically be used in ways the store owner never intended.
The "ethical forgetting"concept ia interesting. The default in most products is to accumulate and use everything. Choosing not to is a deliberate product decision that cost you something. The line between "understanding users to help them'' versus "understanding users to keep them" is the clearest way I've heard that distinction explained.