Now What? Making Decisions After Testing Solutions with Users
This is the FINAL installment in my series about Solution Test Interviews.
Don’t get attached to an idea.
Yes, a Product team must believe there is a solution to the user’s problem or opportunity.
But fixating on a solution because you like it or an executive has demanded it could lead to wasted months of engineering time and opportunity cost in the market.
Use the advice below to know when you’ve got an idea worth building, iterating or discarding.
Nobody gets an idea right the first time.
As you invest in a concept by prototyping and discussing it, you can develop a strong opinion that it must be valuable to users.
Yes, product teams need to develop an opinion…a point of view…about the user experience but they can’t let that opinion outweigh negative feedback.
It’s okay to retreat…to iterate and try again…and move on to something different.
Iterating and pivoting are the most common outcomes of solution testing.
A common form of iteration is when part of the solution received a positive consensus. The team can then pivot the solution and re-test after re-orienting the solution around the successful part.
Letting the users influence your strong opinion is the first step in making a good decision.
You do not have infinite resources or time to bring a product to market.
When users don’t get excited about an idea…when their feedback is neutral or clearly negative…it’s time to discard it.
Fixating on a bad idea comes at the cost of finding that great idea.
A decision to “delete” a bad idea creates the time and space to create a winning idea.
To counter the bias of optimism, organize user feedback.
Then, turn your qualitative feedback into quantitative data to determine if an idea is winning or losing.
After each set of 5 solution tests, calculate the consensus of the users’ feedback.
Most concepts need to be iterated at least once so the first decision is usually to revise and retest.
If you can determine a winner, you might be ready to send the concept to engineering.
This is a great time to revisit the four risks of a product:
Value risk - Will they use it?
Usability risk - Can they use it?
Feasibility risk - Can we build it?
Viability risk - Should we build it?
If you’ve been able to reduce risk and feel confident about the remaining risk, then you are ready for engineering.
Since most of my clients’ solution tests are focused on the Value risk, they are made quickly and not pixel perfect. So there is usually a final round of testing focused on usability that tests a better designed version of the winning prototype.
There’s no hard and fast rule for the number of users to interview.
Small, simple solutions have less risk and need less user input than larger, more complex solutions.
So when possible, simplify a solution test to address one hypothesis at a time to isolate the risks and make your solution tests more targeted.
Use this convention to match the risk with the number of users to interview (experienced teams can deviate as they see fit):
Near zero risk: Don’t do discovery at all
Low risk: 5 to 8 users
Medium risk: 12 to 15 users
High risk: 25+ users
If you test, test with at least 5 users. If a scheduled user doesn’t show up, fill that slot with someone else or reschedule.
Note that as you learn from users, you might adjust the definition of the problem, scope of your solution, and definition of your target audience…all of which increase or decrease the level of risk and the number of users you should test with.
There are other issues to consider when making a decision.
Wrong users…You might have recruited the wrong users. Adjust the criteria for your target users and then recruit and test again.
Wrong questions…Did you ask users “What do you see?” Did you engage in a design review rather than a solution test? Read Don’t Say These Things During User Interviews.
Wrong prototype…When you created multiple solutions, did they end up being too similar to each other? Increase the differences between the options you are showing to users so they can make a clearer demarcation of what they prefer and don’t prefer.
As you work with users more frequently, you’ll be able to make better decisions, more quickly. Your skills of recruiting, prototype making, and interviewing will improve rapidly.
And remember...if you always validate an idea, then take more risk and test a wider variety of solutions.
Good luck!
Jim is a coach for Product Management leaders and teams in early stage startups, tech companies and Fortune 100 corporations.
Jim co-founded PowerReviews which grew to 1,200+ clients and sold for $168 million. He product-managed and architected one of the Internet's first ecommerce systems at Fogdog.com that went IPO at a $450 million valuation.
These days, he coaches companies to find product-market fit and accelerate growth in digital health, financial services, ecommerce, internal platforms, machine learning, computer vision, energy infrastructure and more.
He graduated from Stanford University with a BS in Computer Science. He lectures in Product Management at the University of California at Berkeley.
