The Solution Test Interview Explained

Solution Tests - Recruiting & Interviewing

Feb 24

Written By Jim Morris

600+ Users Interviewed

6 Years

80+ Product Teams

The solution test interview is a valuable and efficient method for validating product ideas.

It tests your potential solutions while at the same time verifying your assumptions about the problems or opportunities that your target customer might be having.

Yet, most Product teams don’t often validate solutions with users, if at all.

These are my practical lessons learned from 6+ years of recruiting and interviewing over 600 users.

First, set up the right solution experiment and plan how you’re going to run the solution test.

As you’re creating your solution experiments, recruit and screen for your target customers.

Once you have a user in front of you (on Zoom or in person), make sure you kickoff the interview correctly and avoid saying these no-no phrases.

In the solution test interview, you will listen, observe, and take proactive steps to make the most of the time you have with the user.

Even though you’ve collected a lot of data, you can speed up your analysis, determine whether you’ve got a winning solution, and decide on your next steps sometimes in 30 minutes or less.

The full list of articles is listed below.

Featured

The Product Discovery Valley of Death

Many Product teams only listen to users in the beginning of the product development process.

After this initial research, they build their solution but don’t test it as they create it.

This gap in interacting with users (sometimes lasting months) is the Product Discovery Valley of Death.

Design to Learn. Beware the Completionist Approach.

Has your Product team created a “pixel perfect” design before even talking to users?

Does the initial design have more than 10 screens?

If so, the designs are overbuilt and too big to learn from.

How to Recruit Users for a Successful Solution Test Interview

Read these quick tips for recruiting and screening users

Based on 500+ users and 60+ Product teams over 5+ years of interviewing

How to Plan a Successful Solution Test Interview

When I first started interviewing users, I struggled to analyze the results.

We weren’t organized in how we asked for user feedback. And so analysis took too long.

Don’t Say These Things During User Interviews

User interviewers often talk too much, ask overly-specific questions or just ask the wrong type of question.

And when you lead interviews like this, the responses won’t be actionable.

How to Kickoff a Successful Solution Test Interview

Kicking off an interview with a user can be simple and stress free.

For most users, it’s the first or second time they are participating in a user test.

So at the start of the interview, I make the user feel comfortable and do a few things to make the interview as actionable and efficient as possible.

What to Listen for in a Solution Test Interview

Interviewers often miss important user reactions.

I’ll hear users say “wow” when they encounter a concept but the interviewer will continue with their prepared script.

We need to stop and ask the user when we hear a reaction like that.

What to Watch for in a Solution Test Interview

Many interviewers try to keep the user on the happy path rather than watching where the user wants to go.

Instead of talking when a screen first comes up…start watching and waiting.

Silence and frustration are necessary for teams to see what they need to fix.

When to be Proactive in a Solution Test Interview

Though interviewers should primarily listen and observe, there are moments when they need to interrupt or take an action.

Being proactive during a solution test interview is fine as long as it’s rare.

How to Speed Up Analysis of Solution Test Interviews

The emotional high of conducting user interviews is often met with the spiritual low of having to analyze the results.

User testing creates a lot of unstructured data: interview notes, video recordings, screener questionnaires and so on.

Teams can easily get overwhelmed with the amount of data they need to analyze.

How to Determine a Winner in Solution Test Interviews

Don’t use your gut to figure out which solution users preferred the most.

This introduces bias and reduces morale as different team members often have conflicting gut reactions.

Use a quantitative approach.

Now What? Making Decisions After Testing Solutions with Users

Everybody hopes that Product Discovery will yield clear answers in a fixed time frame.

In reality, user feedback is not unanimous and it often takes multiple rounds of solution testing to determine a result.

So when are you done? And did you find a winning solution?

Jim is a coach for Product Management leaders and teams in early stage startups, tech companies and Fortune 100 corporations.

Jim co-founded PowerReviews which grew to 1,200+ clients and sold for $168 million. He product-managed and architected one of the Internet's first ecommerce systems at Fogdog.com that went IPO at a $450 million valuation.

These days, he coaches companies to find product-market fit and accelerate growth in digital health, financial services, ecommerce, internal platforms, machine learning, computer vision, energy infrastructure and more.

He graduated from Stanford University with a BS in Computer Science. He lectures in Product Management at UC Berkeley.

HomepageTopPost

Jim Morris