Make data-driven decisions by turning your vision into a math equation
Many Product teams struggle about what to do next. Learn to prioritize everyday work using business outcomes. It made our website profitable and grew our traffic to over 5 million monthly users.
For 4 years, I was the General Manager of a shopping website that peaked at 5.5 million monthly active users. It was a one-stop research destination with 100 million product reviews on 20 million products...think of it as the “Yelp for product reviews.”
We were an unprofitable startup and this website was the monetization slide in our VC deck.
The main caveat to this article is that all quantitative efforts should consistently be paired with and balanced with qualitative customer outreach. Over optimization can mask deeper user experience problems that can be fatal in the long run. If the teams that I coach spent as much time working with customers as they do in analytics dashboards, they would create much better products. That said, all Product leaders need to think like business owners and this article turns business into a science.
Data-driven decisions: Start with your outcome
Our goal was for our customers to find the right product for them. If they purchased then we would make a commission.
To manage our priorities and allocate our team resources we started with our desired business outcome: Revenue.
The KPI Pyramid model helped us map the steps from the desired business outcome to our delivered user experience.
Data-driven decisions: Choose your math equation
To achieve our desired business outcome, we chose our business model.
Obtain licensing rights to product reviews and product catalog data
Gather them into one, easy-to-use website
Drive consumers to the site to consume and learn from content
Provide links for consumers to continue their research onto ecommerce sites where they could buy
Earn commission from consumer purchases that originated from our site
(eventually) Drive enough traffic to attract the budgets of display ad advertisers
We broke down our business model into component parts and turned it into a math equation.
(Website Visitors x Click thru Rate x Revenue per Click) + (Display Ads Revenue per Impression) = Revenue
We had a dedicated analyst who pulled data from dozens of internal and external systems to calculate our equation every week.
Data-driven decisions: Fit your equation to a KPI Pyramid
To make the math equation actionable, we used a KPI Pyramid. (KPI = Key Performance Indicator).
Using a KPI Pyramid helped us:
Provide a starting point for drilling down on problems
Create a visual model for prioritizing investment opportunities
The KPI Pyramid became our weekly data-driven decision tree.
The components of the math equation became our supporting KPIs that led to our desired business outcome.
Data-driven decisions: Where to drill down
We were new at running a website at scale. A huge benefit from the KPI Pyramid was solving problems.
When a KPI decreased unexpectedly, we quickly investigated the part of the site that was related to that KPI.
Over time, we learned which parts of the site most related to each KPI and filled out the lower levels of the KPI Pyramid. We became optimization experts. (There is a downside to over optimization...more on that another day).
Problem Scenario: Visitor counts would go up but revenue stayed the same. Not all products led to users buying. We learned that low rated products caused users to go straight back to Google to search for something else. So we built a recommendation module to suggest highly rated products in the same category in order to retain that traffic.
Problem Scenario: Pageviews would go up but Display Ads Revenue stayed the same. We learned that advertisers didn’t care about every pageview on our site. They preferred the first pageview for any given user. They didn’t want to show more than one set of high value ads to any given user in one 24 hour period. And they preferred certain categories of content (Electronics, Health & Beauty) over others (Sporting Goods, Books).
Problem Scenario. Visitors remained constant but revenue went down. From time to time, we launched the website and removed a feature accidentally or deemphasized a monetization module. We did do visual QA but did not have programmatic QA at the time. So these subtle changes only showed up in lost revenue due to fewer clicks (lower CTR).
Each scenario led to a root cause analysis journey that helped us fill out the lower levels of the KPI Pyramid. After a while, we had a rich set of drill down options to solve any problems that arose.
Also, when you have multiple ways to measure your app or website, you have a backup measurement system in case one of the analytics systems gets removed accidentally.
Data-driven decisions: Where to invest
With a detailed KPI Pyramid, we were able to move from debugging problems to prioritizing investment.
When we wanted to increase revenue, we looked to KPI Pyramid for areas where we could develop opportunities for any of the supporting KPIs.
In this fashion, we frequently used metrics to choose where to invest.
Investment Scenario: Boost Click Thru Rate before the Holiday season. During Holidays, we knew we’d get more traffic on Black Friday and Cyber Monday so we invested in increasing our Click Thru Rate (CTR) to optimize every visit. We conducted a massive multivariate test and increased our CTR by 70%.
Investment Scenario: Respond to our advertisers. Our display advertisers wanted more pageviews to show electronics related ads. So we focused our business development efforts on licensing more electronics products and product review content. This boosted our electronics pageview count (via Google “sending” more free visitors) and our advertisers filled that inventory with high value display ads.
Investment Scenario: Boost Revenue per Click. To earn money on purchases, we showed relevant links from hundreds of websites to our visitors. We relied on those websites to convert the user from a browser into a buyer. We noticed that certain websites converted our users better than others. Higher conversion typically led to higher Revenue per Click (RPC). So we started to emphasize higher RPC websites over low RPC sites. To do this for 20 million products, we had to connect our hand gathered RPC numbers to be incorporated automatically into our website algorithms.
Investment Scenario: Boost website visitors. Our website was created during the era of content. During this time, Google heavily preferred sites that created content that matched what users were searching for. So we curated and created shopping research materials based off of our review content. Over time, Google added hundreds of additional criteria for rankings which eventually deemphasized some forms of content. (more on this another day…)
Data-driven decisions: Caveats
Even though Revenue is so important, it is a trailing indicator. Once you know what your revenue will be, it’s too late to change it.
The supporting KPIs listed above are also trailing indicators. Though there is an aspect to website traffic patterns that I was able to use as a leading indicator.
Every leader needs to find reliable leading indicators. They give a heads up to whats going to happen. The further into the future the better.
Read about how leading indicators can help Product leaders:
Apply the KPI Pyramid to your product.
Jim knows how to build successful teams and products from scratch.
He co-founded PowerReviews, a B2B2C product reviews SaaS platform, that had an exit in 2012 for $168 million at a 13x multiple. In the early days of the web, he product managed and architected one of the original ecommerce sites that had a $66 million IPO in 1999, online sporting goods retailer Fogdog.com.
For his Product teams, he’s created a curriculum and training program that pulls from his 20+ years of experience and the best minds in Product Management. In addition, he relies on his software engineering background and experience to bridge the gap between their Product and Engineering teams. He graduated from Stanford University with a degree in Computer Science.
Jim is based in San Francisco and works with clients from 2 to 20,000 employees in a variety of industries and business models. Previous clients include VSP Global, PagerDuty, Dictionary.com, and Hallmark. He’s also worked with startups in machine learning, API development, computer vision, payments and digital health.
