Surveys are bad at initial learning

Alexey Shashkov
14 replies
When asked to do the tiniest thing to learn from customers, many product people’s first instinct is to conduct surveys or focus groups. While running surveys and focus groups may seem more efficient than interviewing customers, starting there is usually a bad idea. 1. Surveys assume you know the right questions to ask. It is hard to script a survey that hits all the right questions to ask because you don’t yet know what those questions are. During a customer interview, you can ask for clarification and explore areas outside your initial understanding. Customer interviews are about exploring what you don’t know you don’t know. 2. Surveys assume you know the correct answers, too. In a survey, not only do you have to ask the right questions, but you also have to provide the customer with the right choice of answers. How many times have you answered [Other] to a survey question? The best initial learning comes from open-ended questions. 3. You can’t see the customer during a survey. Body language cues are as much an indicator of Problem/Solution Fit as the answers themselves. 4. Focus groups are just plain wrong. The problem with focus groups is that they quickly devolve to groupthink, which is wrong for most products. 5. Surveys can be pretty effective at verifying what you learn from customer interviews. A customer interview is a form of qualitative validation that is quite effective in uncovering strong signals for or against hypotheses using a reasonably small sample size. Once you have preliminary validation on your hypotheses, you can then use what you have learned to craft a survey and verify your findings quantitatively. The goal is no longer learning, but demonstrating scalability or statistical significance of the results. Inspired by the book: «Running Lean: Iterate from Plan A to a Plan That Works» Originally published at Startup Summary

Replies

Jennifer
Hey Alex, really interesting! I have started using open-ended questions in the last few years when I realized customers have a lot of great ideas and feedback when they are able to express them freely, without a defined direction usually found in multiple response sets. Will definitely look into the book, thanks for sharing
Alexey Shashkov
@jennifer00 Hey, Jenifer. It's so cool you use open-ended questions! It's a measure of your maturity as a product maker. Thank you for the comment. Nice to meet you. 🙌🙂
Jennifer
@shashcoffe Thanks Alexey!! it can sometimes be a hard sell in corporate settings but was happy to see you and others share the same viewpoint. Great to meet you too :)
Alexey Shashkov
@jennifer00 That's what we have in common here at Product Hunt. =)
Mohsen Kamrani
Awesome as always. Thanks Alexey, just would be good if you could post them on a blog too instead of Telegram.
Alexey Shashkov
@m_kamrani Hi, Mohsen. I'm so glad you appreciated it. Thank you, man! I published it on Revu and Medium: Revue: https://shashkoff.com/issues/sur... Medium: https://shashcoffe.medium.com/su... Don't you think that's enough and need a separate blog?
Mohsen Kamrani
@shashcoffe That's awesome. Nope that's it, no need for separate blog.
NotesbyHugh
I've always had similar thoughts! Going to give this book a read - thanks for sharing! If not surveys, what're the alternative options for initial learnings?
NotesbyHugh
@shashcoffe interviews are great but hard to scale like surveys. Did the book recommend any other options?
Alexey Shashkov
@hugh_dawkins Hey, Hugh! Customer interviews are the best way for initial learning.
Alexey Shashkov
@hugh_dawkins There are only about customer interviews and observing (reviews etc)
Lior Galante Cohen (Vaza)
Great insights, thanks for sharing Alexey!
Alexey Shashkov
@lior_galante_cohen Thanks for your appreciation, Lior!