Andreas Soller
Outcome Metrics
Why is it important to measure outcomes and not only output or impact?
This article discusses the importance of outcome metrics, differentiating them from output and impact metrics in the context of agile software development.
3 min read (720 words)
Feb 13, 2023 – Updated Nov 12, 2023, 05:59
METRICS
IMPACT
OUTPUT
OUTCOME
What are outcomes?
To measure success, it is important to understand those user behaviors that drive the desired business results:
“(…) an outcome is a change in human behavior that drives business results. (…) outcomes are the changes in customer, user, employee behavior that lead to good things for your company, your organization, or whomever is the focus of your work.” – Joshua Seiden 2019:12
Definining the right metrics
When we think about agile software development we deal in essence with three types of metrics:
- Output metrics
- Impact metrics
- Outcome metrics
Output metrics
Output metrics are important to measure the performance of our activities.
Examples:
- To measure delivery speed or if stories where implemented in an efficient order we use story breakdown charts.
- To improve the work process we do team retros and agree on concrete action points.
But those metrics cannot tell us anything about the usage of the working software.
Takeaway
Output metrics measure the performance of activities, like delivery speed or process improvements. However, they don't indicate the usage or effect of the software on user behavior. Output itself is the concrete working software that has been developed. Just building software efficiently does not tell us if the software is also used by the assumed target segment.
Impact metrics
Impact is about understanding what will drive business results.
Think about: increasing the revenue, decreasing costs, increasing market share / customer adoption, increasing revenue from existing customers, increasing shareholder revenue, increasing service delivery / productivity, strengthening the brand, increasing life-time value, decrease cost of aquisition (marketing, sales people), increase monthly return in revenue (MRR), etc.
An impact metric is a generic way to measure business success and provide us with comparable – although high level – quantifiable data (increase / decrease).
“(…) at the highest level of a business, leaders are concerned with the overall performance of the organization, and the performance numbers they watch tend to come down to these factors – which, in our langauge are high-level or “impact” metrics.” – Joshua Seiden 2019:24
Examples:
- Cost / Benefit ratio: compare the investiment (efforts spent) to the expected value. To give a high level example, building a certain feature might cost 80 person days and 5 person days per year for maintenance. Then you can compare those efforts to the expected return.
- Business Impact potential: additionally you can calculate the potential return and how likely is it to be in the lower / middle / high ranges?
Often impact metrics as success metrics are too abstract.
They cannot easily be translated into concrete features. Think about increase revenue: how should this be translated to something particular?
Therefore, impact metrics serve only as a basis to define specific outcome metrics that can then be translated to output (working software). Impact metrics are more like the north star but to travel there, we must first understand what we can do in our own solar system. It is all about making things concrete and applicable.
Key takeaways
Impact metrics measure business success, focusing on factors that drive business results like revenue, costs, market share, brand strength, etc. They provide quantifiable data but are most of the time too abstract to translate into specific features.
Outcome metrics
Outcome or user metrics include impact and output metrics:
- Understand what customer or user behaviors drive business results.
- Understand how we can increase or decrease those behaviors.
- Measure the outcomes to make necessary course corrections.
Example (cf. Seiden 2019:32):
Impact: decreasing costs
Output: improve usability of confusing software features
Outcome: fewer people call tech support for product A
As not everything that brings value to users brings also value to your organization we can also get blindfolded by looking at outcomes alone.
Therefore, it is important to understand how output delivers outcomes that in turn impact your company or organization.
Key takeaways
Outcome metrics combine both output and impact metrics. They focus on understanding and measuring user behaviors that drive business results and how changes in these behaviors affect the outcomes.
References and recommended readings
- Patton, Jeff (2022): Passionate Product Leadership. Certified Scrum Product Owner course. URL (training promotion): https://www.jpattonassociates.com/services/ppl-online-course/ (26 Nov 2022)
- Seiden, Joshua (2019): Outcomes over Output. Why customer behavior is the key metric for business success. United States (On demand): Sense & Respond Press
Thank you for your feedback.