What to Listen for in a Solution Test Interview
Interviewers often miss important user reactions.
I’ll hear users say “wow” when they encounter a concept but the interviewer just continues with their prepared script.
We need to stop and ask the user when we hear a reaction like that.
Here are more tips on what to listen for, how to stimulate great conversation and how to react to what users say:
The user is probably just filling time and trying to give you some response about what they’re seeing.
In fact, the user might as well be saying “Meh”.
This happens because most users want to be nice to their interviewers (most are getting paid after all).
And as interviewers, we don’t push back or dig deeper because we don’t know any better. I’ve seen over 600 user interviews so I can tell the difference between “like” and “love”.
Instead, use a couple “Why” questions to learn more. And be ready to accept that users just may not value your solution.
I usually interrupt the interview at that moment and ask the user to explain this reaction.
These types of exclamations are emotional reactions and they are the gold of user interviews. They are rare and they are valuable.
They are important because they happen quickly and are the unfiltered thoughts of a user.
This is when I get some of my best feedback on the value of an idea (and my best user quotes).
These are other emotional reactions that indicate you might be on to something good:
“I love this”
“Oooh”
"When will this be available?"
[big smile]
[eyes widen]
"Huh"
"Hmmm"
"Oh"
[frown]
[eyes narrow]
You should interrupt at this moment and say something like “I'm sensing some hesitation. You just said 'oh'. Can you tell me more?"
“Creepy”
“Spammy”
“That’s overwhelming”
"Cumbersome"
After interviewing for so many years, it seems that negative reactions are harder to overcome and are a big blocker to adoption. Remember that most users will settle for giving you a neutral reaction like “that’s interesting” so when you hear very negative comments this is a big red flag.
Make sure to pause and talk to the user about why they feel so negatively.
…ask them right then and there to have them show it to you. You can ask to take a screenshot or get a copy yourself.
As a Product team, you want to understand what workarounds and current solutions exist that users take advantage of. it saves you time and potentially gives you innovative approaches.
Instead of answering, say…“How would YOU do that?”
I usually don’t explain confusing concepts. I just mark it down and move on. The team just needs to improve it before the next interview.
If the user really wants to know, I ask “Is it okay if I explain it to you at the end of the interview?”
This saves time and saying nothing helps the Product team from trying to explain themselves.
Write down the EXACT phrase.
DON’T paraphrase.
For example, this quote:
"Using a list is demoralizing"
is more powerful than writing down:
[user says they didn't like the list concept]
Your colleagues will connect better with the user’s exact words
“I could see my friends using this”
“I think people would find this interesting”
This is hearsay...second hand information. As an interviewer, you are seeking to learn about THIS user’s opinions and thoughts.
Go ahead and interrupt the user politely. Then, redirect the user to talk ONLY about THEIR experience and opinion.
"Ok. Tell me about your experience..."
You can ask the user who they are referencing and maybe that would be a great user to interview next.
Visual design feedback is commenting on the colors, button placements, and other visual elements.
It’s a natural reaction by users since it’s more familiar territory than the new concept that you’re showing them.
Sometimes this happens when you ask “What do you see?” (avoid this question) or by presenting visuals that are overwhelming to the point of distraction.
Ask yourself if the feedback is a visual design opinion (they really like a slightly bigger button) or a user experience opinion (they always do Y before X).
Their user experience opinion is the good feedback we're looking for. The visual design feedback is usually less actionable since teams have design guidelines they follow.
Sometimes you discover a bad design guideline that should be changed but most times, you will prefer the user to spend this valuable time talking about whether they would or would not adopt the solution.
A reliable way to get them back on track without introducing bias is to refocus them on the task to complete or remind them of the mindset described at the beginning of the interview.
Summary
Whether you're an experienced interviewer or a novice, these tips will help you get better at knowing what to listen for, how to stimulate great conversation and how to react to what users say.
Jim is a coach for Product Management leaders and teams in early stage startups, tech companies and Fortune 100 corporations.
Previously, Jim was an engineer, product manager and leader at startups where he developed raw ideas into successful products several times. He co-founded PowerReviews which grew to 1,200+ clients and sold for $168 million. He product-managed and architected one of the first ecommerce systems at Fogdog.com which had a $450 million IPO.
Jim is based in San Francisco and helps clients engage their customers to test and validate ideas in ecommerce, machine learning, reporting/analysis, API development, computer vision, online payments, digital health, marketplaces, and more.
He graduated from Stanford University with a BS in Computer Science.
