How to Kickoff a Successful Solution Test Interview
Kicking off an interview with a user can be simple and stress free.
For most users, it’s the first or second time they are participating in a user test.
So at the start of the interview, I make the user feel comfortable and do a few things to make the interview as actionable and efficient as possible.
Here are the keys to starting off a successful solution test interview:
Use a templated script to ensure you cover what’s important.
A live event like a user interview can cause you to forget important details such as reminding the user to think out loud and be honest or simple logistics like turning on the recording.
Reading from a script may seem strange but many of my clients are first-time interviewers themselves and appreciate having a friendly-sounding, well-tested template to get a solution test started.
Take the time to make the user comfortable with giving feedback.
Most folks are not used to giving critical feedback to strangers so take the time and explicitly give them permission to think out loud, be honest and be critical.
We want them to feel like they are doing us a favor by telling us what they like or don’t like.
One example is to remind users that this is a good time to give feedback since it’s better to learn from users now than after months of development.
Start the interview by having the user reflect on a real experience.
Teresa Torres often refers to this as “collecting a specific story.”
This story should include how the user has/had the problem or opportunity that you are solving for.
We don’t expect users to have experience with our potential solutions, but we do expect them to have experienced the pain point. Otherwise, they won’t have any context for giving useful feedback.
After telling their story, the user will be in the Right Mindset to reflect on new solutions that the team has created.
You can stimulate storytelling like this:
Consumer example: “Think back to a time in the past 60 days when you purchased an iPhone accessory. Tell us about that time.”
Business example: “Think back to a time when you had to set up benefits for an employee in the HR system. Describe that to me.”
Avoid starting your prompt with “Imagine…” since that might bias users to make up scenarios that wouldn’t normally exist.
And make sure that the user’s story has enough detail to be memorable and believable:
Too-generic example: “I went online and searched for home contractors and then selected one for a project we had at our house.”
Right-amount-of-detail example: “I went to Yelp and searched by typing in ‘handymen’ and then I looked through the first 3 or 4 that showed up. I checked their ratings, read some reviews and then called a couple until I found one that had a reasonable hourly rate and would come by to give an estimate.”
Having the user tell us a specific story ensures that we get insights from an authentic source.
I don’t collect deep background information up front.
We can get actionable feedback without getting a user's life story.
Of all of my advice, this is the most shocking to other UX researchers since they tend to collect a lot of background information about the individual up front.
Instead, collect only the need-to-know background information during the interview when it’s relevant.
We still ask “Why” to learn more but it’s always in the context of what we are testing.
For example, over 50% of my user testing these days is in digital health. We don’t pry into an entire patient’s health history when only a piece of it is relevant to us.
This need-to-know approach works for other personally sensitive information and government classified subjects as well.
You could think of this approach as focused on a user’s behavior rather than the entire user…more of a Jobs to be Done mentality than a Persona approach.
Note that this is different than generative interviews where we are more focused on finding problems and might explore more of the user's background.
When working with new product teams and students, I realize how much jargon we use in our industry.
Reducing jargon can make interviewees feel less confused and thus feel more comfortable.
For example, the phrase "user test" is not great. Nobody likes a test.
Instead, try saying "In this session, we'll have a conversation where we'll show you some concepts that we'd like your feedback on."
To avoid setting the wrong tone, I don't explain that the prototype is fake and I avoid using the word "prototype."
I explain that this is a “new concept” that is not fully built out…and leave it at that.
Often, users end up believing they are using a real app or website.
This less-is-more approach results in more realistic feedback than over explaining it.
Many solution test interviews look like a powerpoint presentation where the product team describes the product on screen and then advances the prototype screen by screen without the user interacting at all.
This hands-off style is to be avoided at all costs. When the product team controls the flow, they don’t encounter situations or make “mistakes” that only a user would.
It’s much better to transfer control of the prototype to the user (usually by sending them the link) and have them share their screen.
Then the user can move around the prototype freely, click anywhere and find themselves in situations that you didn’t anticipate. For example, they might click on something that they think is clickable but you didn’t.
Transferring control to the user helps you find the “unknown unknowns” as well as reduce the bias you might create by inadvertently leading the user.
This usually requires making a clickable prototype so the user can advance through the journey by tapping or clicking themselves. Note that even rough prototypes can be made clickable and most all users can find their way through a rough clickable prototype.
Modern design tools like Figma are so easy to use that even non-designers like myself can make a bunch of screenshots clickable in a few minutes.
Of course in a prototype like this, the users will encounter dead ends. When that happens, let them know that you haven’t built out that section yet and encourage them to continue towards their goal using a different path.
Where possible, personalize the prototype to connect with the user more deeply.
For example, if we are testing a movie rental application, we might ask the user to tell us 3 of their favorite movies during the interview introduction. While we’re finishing the introductions, the prototype maker can personalize the prototype with those movie names. In addition, if you use the user’s name in the prototype then change those references to the user’s actual name.
These are simple but effective ways to increase the belief a user has in the reality of the prototype even when it’s not a real, working product.
Increased belief in reality makes the user forget they are using a prototype and leads to more authentic feedback.
Using these simple steps can reduce the anxiety of the interviewee (and the interviewer) and improve the interviewee feedback thus getting you off to a great start in a solution test interview.
Jim is a coach for Product Management leaders and teams in early stage startups, tech companies and Fortune 100 corporations.
Previously, Jim was an engineer, product manager and leader at startups where he developed raw ideas into successful products several times. He co-founded PowerReviews which grew to 1,200+ clients and sold for $168 million. He product-managed and architected one of the first ecommerce systems at Fogdog.com which had a $450 million IPO.
Jim is based in San Francisco and helps clients engage their customers to test and validate ideas in ecommerce, machine learning, reporting/analysis, API development, computer vision, online payments, digital health, marketplaces, and more.
He graduated from Stanford University with a BS in Computer Science.
