How do you know you're successful today?
It’s a simple question.
But often there’s an unsatisfying answer.
These are the most common answers to “How do you know if you’re successful today?”:
We launched on-time
No defects were reported
The stakeholders (“the business”) approved
We received positive customer feedback
These answers give an impression of success but aren’t quantitatively reliable enough to represent true success.
The first step towards understanding whether you're successful today starts with recognizing (and using less of) the following types of data:
“Anecdata”
Reactive measurements
Internal metrics
“Anecdata”
“Anecdata” sounds like data but is really one or more anecdotes pretending to be data.
Examples of “anecdata”:
In an interview, a customer tells us the new app is helping them keep track of their calories and lose weight
A sales rep tells us the new product features are generating follow up calls
A customer success manager describes a client who’s doesn’t understand how to use the new features
Recent App Store reviews are positive in sentiment
"Anecdata" is real but it’s usually limited to a small time range and a small sample of customers.
There are several reasons why people use "anecdata", even though it is not a reliable form of evidence:
Confirmation bias: People tend to seek out information that confirms their existing beliefs, and they are more likely to remember and share anecdotes that support their point of view.
Availability bias: People tend to rely on information that is easily accessible.
Illusion of validity: Anecdotes can seem more convincing than statistical data.
Lack of statistical skills: Many people do not have a solid understanding of statistics.
It is important to be aware of the limitations of "anecdata" and to avoid using it as the definition of success.
Reactive measurements
Some teams only pay attention when there’s a problem.
Teams get busy. Leaders focus on output, output, output.
As a result, teams don’t add analytics technology to their products and services.
They rely on reactive measurements to evaluate "success".
"Success" is when they don’t hear about problems.
They look the other way and hope for the best.
These are common reactive measurements:
Customer complaints
Defects found
The assumption is that a functional product is a valuable product.
This is short-sighted. Working technology is just the first step to having valuable technology.
Is anybody using the technology?
Does it provide enough value to renew? To recommend to others?
A team with a healthy skepticism about success embraces doubt.
A proactive team takes the time to install measurement technology into their products and services.
This gives them advance notice of problems and tracks customer success proactively.
Internal metrics
Internal metrics are a representation of how well or poorly your company is running.
They include process metrics, productivity metrics, quality metrics, team health metrics and more.
These metrics are internally focused and don’t represent the value your customers receive from your products and services.
Examples of internal metrics:
Process metrics: Hitting a deadline
Productivity metrics: Number of points the development team completed last sprint
Quality metrics: Number of defects in the last development cycle
Team health metrics: Employee satisfaction
Internal metrics can be useful but even the most well-run companies lose customers, lose revenue and lose momentum if customers don’t get value from their well-oiled machine.
Summary
“Anecdata”, passive measurements, and internal metrics are not a reliable representation of success.
In my upcoming course, I help individuals and teams measure success by creating and measuring useful, actionable and quantitatively reliable analytics.
Join me. It’s a 5 session class starting weekly on May 22nd. We work through example material and you practice these new techniques on your own product.
Early Bird pricing applies until April 22nd.
Jim coaches Product Management organizations in startups, growth stage companies and Fortune 100s.
He's a Silicon Valley founder with over two decades of experience including an IPO ($450 million) and a buyout ($168 million). These days, he coaches Product leaders and teams to find product-market fit and accelerate growth across a variety of industries and business models.
Jim graduated from Stanford University with a BS in Computer Science and currently lectures at University of California, Berkeley in Product Management.
