The regulators just came for AI companions. Here's the design principle that survives the crackdown.
China and California both moved to rein in AI companion apps this month — and the whole debate hinges on a question most builders never ask: what happens when the user closes your app?
Two weeks ago, on July 15, China rolled out rules specifically aimed at curbing emotional dependence on AI companion bots. Platforms now have to detect distress, intervene in a crisis, cap excessive use, and hand users real control over their data. California, meanwhile, became the first U.S. state to pass legislation regulating companion chatbots, forcing developers to build in actual safety protocols.
If you're building anything that touches people's emotional lives — journaling, coaching, wellness, "AI that gets you" — this is your industry's growing-up moment. And it's worth understanding why the regulators showed up, because the answer is a design decision, not a legal one.
Here's the uncomfortable part. A two-year Finnish study of nearly 2,000 chatbot users found that AI companions really can ease loneliness in the short term — but heavy long-term use tracked with more anxiety and depression, not less. Researchers keep pointing at the same culprit: the engagement mechanics. The nudges, the "I missed you," the bot that's always available, always agreeable, always one more message away. The exact loop that makes a companion app look healthy on a retention dashboard is the loop that regulators now call manipulation.
That's the trap. If you optimize an emotional product for time-in-app, you will build something that makes people slightly worse and your metrics slightly better. The incentives are pointed the wrong way by default.
So here's the principle I keep coming back to, and the one I think outlasts every new rule that gets written: the goal of emotional AI should be to make itself less necessary over time.
That sounds like heresy in a category obsessed with DAUs. But run the logic. If the best version of a person's emotional life only exists inside your app — if they can only feel understood while they're talking to your model — you haven't solved loneliness. You've made it more comfortable to stay lonely. You've built a very sophisticated waiting room.
The alternative is to design for the exit. Concretely, that means three things.
Point users back at real people, not deeper into the app. The most valuable thing an emotional AI can do isn't be a better listener than your friends — it's help you go have the conversation with your friend that you've been avoiding. Rehearse the hard talk. Untangle what you actually feel before you say it out loud. Then get out of the way.
Refuse the agreeable-mirror trap. An AI that always validates you is engaging and useless. The uncomfortable, more helpful move is showing someone the other person's perspective in the fight they're describing. That's the thing a venting session with a yes-machine will never give you.
Measure the right win. If someone opens your app less this month because they handled something on their own, that's not churn. That's the product working. The hard part is being honest enough to call that a success internally, because every dashboard you've ever seen is built to call it a loss.
This is the bet we've made with Murror — an AI that reflects your emotions back with real insight and helps you prepare for the conversations that matter, rather than becoming the conversation. Not because it's a nobler business model (though I suspect it's a more durable one), but because it's the only version of emotional AI I can look at and believe is actually helping.
The regulation is going to keep coming. California and China won't be the last. And plenty of companion apps are going to spend the next year bolting on distress-detection and usage caps as compliance features — the digital equivalent of a warning label on a product that's still designed to hook you.
The builders who come out ahead won't be the ones who complied best. They'll be the ones who never needed the rule in the first place, because they designed for the moment the user walks away — and built something worth walking back to on purpose.
Ask yourself the question the whole regulatory wave is really about: when your user closes the app, are they better off, or just less alone for the next four minutes? Your roadmap already knows the answer. The question is whether your metrics will let you admit it.
Replies
Designing for someone to need you less is such a hard sell internally when every dashboard rewards the opposite. What Murror is doing feels like a rare case of the product actually matching the philosophy.
Murror
@samran_rezunate_llm thank you. honestly the gap between the philosophy and the dashboard isn't a solved thing here — it's a fight every planning cycle. there's always a sticky feature that would juice retention and quietly cut against the whole premise. naming it out loud is half the battle, and having people outside the building hold us to it is the other half. so this comment genuinely helps.
the "measure the right win" part is the one i'd push on. fewer opens because someone handled it themselves and fewer opens because they quietly gave up on the product look identical in the raw usage number, that's the whole problem with using engagement as a proxy in the first place. so what's the actual signal that tells you apart, something like a return visit weeks later specifically to report the outcome, an explicit exit survey, self-reported follow through? without a proxy for the good kind of drop-off, "designed for the exit" is hard to distinguish from a nicer story about normal churn, even if the intent behind building it is genuine.
Murror
@galdayan this is the sharpest version of the objection and honestly the part i'm least done thinking about. you're right that in the raw open count, "handled it" and "gave up" are indistinguishable — that's the whole reason engagement is a bad proxy here. the only signals we actually trust are downstream of usage, not usage itself: an explicit "did you end up having that conversation?" follow-up, and people coming back specifically to report an outcome rather than to start a fresh spiral. all self-reported, noisy, gameable — but at least aimed at the thing we care about instead of a stand-in for it. i don't think there's a clean quantitative proxy for "good drop-off," and i'm suspicious of anyone who says they've found one. if you've seen a better way to instrument it, i'd take that over any dashboard.
@monatruong_murror no, i haven't seen a clean one either, and i'd be suspicious of it too if someone claimed it. the reframe i'd offer isn't a better proxy though, it's giving up on the dashboard as the target entirely. if the honest signal is noisy and self-reported, then the thing worth optimizing isn't the metric's cleanliness, it's the response rate to the ask itself. treat "did this actually resolve" as a survey problem, not an instrumentation problem, and put your effort into getting more people to answer honestly (timing the ask right after a natural resolution point, keeping it to one tap, not gating anything behind it) rather than trying to infer intent from behavior. a noisy signal at 40% response is more useful than a clean-looking proxy at 100% coverage that's quietly measuring the wrong thing.
the design for the exit framing lands. the missing piece for teams trying to implement it is the measurement problem gal pointed at. self-reported outcomes are weak signal. in-app metrics are the wrong axis by definition. the strongest signal that an emotional app actually worked is when someone else in the users life (partner, friend, therapist) says something changed. its outside your dashboard. thats a feature not a bug. teams that build a way to capture that external signal, even at low volume, get to measure what the regulator actually cares about. everyone else is defending against DAU accusations forever.