The metrics I track that might be lying to me.
by•
My dashboard is beautiful. Signups, activation, weekly actives, retention curves, the whole thing. And I've started to suspect it's telling me a story I want to hear.
I chose the definitions. I chose the date range. I chose which cohorts to compare. Every one of those choices was defensible on its own, and together they might add up to a portrait of a business that doesn't exist.
The uncomfortable question is whether I'd be able to tell if the numbers were wrong — or whether I've built a mirror instead of a window.
Have you ever caught your own metrics lying to you? What did it take to notice?
3 views
Replies
Mine lied through the plumbing, not the definitions. My web funnel sends buyers to the app with a code, so I can tell which visitor became which install. On Android one source kept answering "not found" for a funnel buyer, which on its own would have counted them as organic. Following that one person by hand showed why: they tapped the store badge late at night and opened the app nine hours later, and that lookup only searches back 15 minutes. A second source had matched them fine. That's my test now for any number I care about: pick one real user and check that the data tells their story correctly. A mirror fails that test fast.