Touching the unknown

When you launch an innovative new product or feature there is lack of data. You will do research to validate your value assumptions. The truth is as long as your product doesn't hit the market you can never be sure if and how it will be used in reality.

To overcome this problem, we started delivering software in increments. We follow patters such as the build – measure – learn cycle or talk about minimum (= next) viable (= best) products (= test). We include continuous customer feedback loops (Voice of Customer, Friends and Family feedback, etc.) to mitigate risk and build up levels of confidence.

Still, it happens that the real usage is not what we validated over and over. Our KPIs are not met, and we wonder where we took a wrong turn. In the worst case this is where the invention of narratives starts and we identify hypothetical reasons (or scapegoats) why a certain result was not achieved in order to keep going. Don't start spinning a yarn! Even if it saves us in inconvenient situations, it will lead to disaster. If not today, then tomorrow or the day after. It always fires back.

In the beginning you are more of an explorer who set out to discover unchartered territories. Therefore, you become a cartographer who maps out those new regions and transfers the unknown to the known.

There is nothing that will prevent unexpected realities from happening. But the good news is that there are metrics and processes that help us to focus on success and failure from start to end, from the idea or initial concept to the mature product or service.

In need of a navigator

Becoming a cartographer is not enough. We also need a navigator as we cannot trust our own gut feelings and assumptions. We need a navigator that is objective, always.

To go there, let us first talk about a navigational device with a certain superpower: a device that provides evidence, be it convenient or inconvenient news. Real data that is hard to deny or to fantasize about.

A device everyone has agreed upon to be able to steer us in troubled waters. Be it business stakeholders, product owners or agile teams. A device so powerful that it does not only have they buy-in of everyone once it is put to use but that is additionally a shapeshifter. It is able to be adjusted to almost any new situation.

Well, don't stop reading. I am not making something up. This device truly exists. I am also quite convinced you already know about it the one or other way and the whole magic is only about putting attention to it.

By the way, an Astrolabe (Greek: ἀστρολάβος; literally: “star taker”) is an astronomic device that served as physical model of heavily bodies. For the ancient astronomer it was almost a Swiss Army Knife or the famous “eierlegende Wollmilchsau” in German colloquialism that refers to a mythical creature that can do everything. Just think of Chuck Norris, but way, way back. As a precursor to the Sextant it allowed to determinate latitude and the explorer of worlds would utilize the mariner's astrolobe in ancient times – an early spinoff product.

What are outcomes?

The navigational device I am talking about are outcome metrics. To make use of it, we must first understand the difference between output, outcome and impact.

Output

We have ideas (opportunities) that are transformed into concrete working software.

In an agile team, costs (human resources, infrastructure, etc.) and time (sprint duration) are fixed. Hence, the scope (deliverables) becomes the zone of influence.

Output is the very concrete working software. Output is great, because we can predict it and after each sprint, we can re-check the effort that was needed. We know everything about time and cost spent. There is nothing magical about it – it's simple elementary school math.

Additionally, we know how we build the software. Therefore, the activities – the way how we created the working software – is well known to us. In an agile setup we have a scrum master role that helps the team to improve on the activities. Retrospectives are meetings where process optimizations, etc. are discussed and concrete actions are agreed.

Outcome and impact

While we know everything about output, initially we don't know anything about outcome. Because outcome happens outside our organization (Unless we build working software for internal users but even then, the users are usually sitting in another division, department, etc.).

Additionally, there is a time difference: we estimate output in sprint refinements and adjust the estimation after each sprint. Therefore, a team knows precisely the effort spent. Compared to output it takes as a long – sometimes a very long – time to fully understand outcome.

When we think about idea bias – the simple fact that we overvalue our own ideas or the ideas of the social group (team, initiative, project…) we belong to creates a real problem: If we do not continuously validate the outcome, we might keep delivering working software that is actually not used as imagined or – worse – results in sunk costs.

To avoid outcomes becoming our blind spot we need to

• understand what customer or user behaviors drive business results.
• understand how we can increase or decrease those behaviors.
• measure the outcomes to make necessary course corrections.

The right navigational device

Well, Columbus failed. He never discovered the intended trade route to India going westwards. He was convinced until his death he had found a route to “Las Indias,” believing he had sailed down the coast of Asia. He could not have known that another ocean lay ahead to cross.

If the organization or entrepreneur is capable of truly understanding the full impact of one's finding is another topic worth thinking about.

We remain on the level of the navigational device. Our target is to design and build the right digital product or service. Our goal is to stay on course to make those discoveries in the first place.

As the ship's boy turned the sandglass every half-hour to keep track of time, the nocturnal was used to determine midnight and the quadrant supported orientation, we will utilize outcome metrics to be able to navigate towards our goal.

A word on metrics

When we think about agile software development we deal in essence with three main types of metrics:

1. Output metrics
2. Impact metrics
3. Outcome metrics

All of those metrics matter but each has its own qualities. We must be aware of the limitations and power of each metric to make use of them.

Output metrics

Output metrics are important to measure the performance of our activities.

• For example, to measure delivery speed or if stories were implemented in an efficient order we use story breakdown charts,
• or we improve the work process with team retros and agreed action points.

Those metrics will not tell us anything about the usage of the working software. Just building software efficiently does not tell us if and how the software is used by our users.

Impact metrics

Impact is about understanding what will drive business results.

Think about: increasing the revenue, decreasing costs, increasing market share / customer adoption, increasing revenue from existing customers, increasing shareholder revenue, increasing service delivery / productivity, strengthening the brand, increasing life-time value, decrease cost of acquisition (marketing, sales people), increase monthly return in revenue (MRR), etc.

Examples for impact metrics:

• Cost / Benefit ratio: compare the investment (efforts spent) to the expected value. To give a high-level example, building a certain feature might cost 80 person days and 5 person days per year for maintenance. Then you can compare those efforts to the expected return.
• Business Impact potential: additionally, you can calculate the potential return and how likely is it to be in the lower / middle / high ranges?

An impact metric is a generic way to measure business success and provide us with comparable – although high level – quantifiable data (increase / decrease).

Most of the time impact metrics as success metrics are too abstract. They cannot easily be translated into concrete features.

Think about increase revenue: how should this be translated to something particular?

Therefore, impact metrics can serve as a basis to define specific outcome metrics that can then be translated to output (working software). Impact metrics can act as guiding stars but to travel, we must first understand how to read them and what concrete actions we must take to follow their path.

It is all about making things concrete and applicable.

Outcome metrics

Outcome metrics measure concrete changes in user behavior.

Below a simple example from Josuah Seiden (cf. Seiden 2019:32):

Measuring the number of people calling tech support is benchmarking your output against concrete user behavior. You know exactly what effect your output had on customers. Additionally, you can translate those outcomes to impact (business metrics). This is the full loop.

Another example:

Be aware that not everything that brings value to users brings also value to your organization. We could also get blindfolded by looking at outcomes alone.

The goal is to understand how output delivers outcomes that in turn impact your company or organization.

Make metrics concrete

Let us assume the initial hypothesis is already drafted.

Usually, things still look foggy. Hence, it is important to understand right from the beginning – when a (business) opportunity is identified – what benchmark can be used to measure success or failure along the whole journey, from the initial idea to the mature product. The crucial part is to make it as concrete as possible. Of course, our metrics will be adapted along the journey. I am not saying that one and the same metric will be the benchmark in all phases but the focus on outcome metrics will remain during the whole product cycle.

In a nutshell you specify in detail what user behavior you can measure to continuously monitor if solving this problem is on the right track.

There is no specific rule what metric framework to use. Personally, I prefer OKRs as their structure to first think about objectives and subsequent key results serves our purpose well. What matters is that the key results are behaviors / actions on your user side that you can monitor on a regular basis.

1. Set clear Objectives: they should be ambitious, inspiring, and aligned with your organization's goals. Objectives should be qualitative and provide a clear direction.
2. Define Key Results: they are specific, time-bound, and measurable. Key results indicate progress towards the objective. Typically, two to five key results are set for each objective.

When I talk about data, I do not only refer to quantitative data but also consider qualitative data that you gather. Be aware that at certain stages during your product development you will have to rely on data that is not yet as accurate as you wish for. Obviously, the level of evidence of explorative research where you gather validation of your assumption is lower than the data collected in first usability tests or later data coming from A/B testing on your live application or concrete sales figures. The first challenge will be to find the proper data sources to validate your key results.

It's a collective endeavour

New territories lurk everywhere: It took 48 years until the particle described by British physicist Peter Higgs was confirmed at the CERN Large Hadron Collider. More than 3000 physicists collaborate nonstop around the world to make such discoveries possible.

A whole village of physicists was needed to validate or invalidate this particle. As a captain is in need of his crew or innovative ideas feed on centuries of thought, we are standing on the shoulders of our predecessors. Discoveries never happen out of the blue.

The benefits of outcome metrics

Many navigators

The metrics itself are just the navigational device. Additionally, we need navigators who take decisions based on these metrics.

In large companies a lot of different roles work together. There will be sponsors, product managers, delivery managers, stakeholders of many different flavors, agile teams or even teams of teams, etc.

Each of the roles will have to face these risks in one or the other way:

• Value risk: will users use / buy this product?
• Usability risk: will users be able to use this product?
• Feasibility risk: will we be able to build this product with our given resources (time, skills, technology…)?
• Business viability risk: will we have a return of investment?
• Product specific risks such as regulatory risks or ethical risks and so forth…

Many navigators but only one navigation device

Outcome metrics bring a lot of power to the organization and each individual:

Using outcome metrics will create a shared benchmark across the organizational structure to focus on the same end-goals. Understanding what user behaviors drive what business impact will create a customer centered focus on all involved levels.

• Alignment: They align the efforts of different departments and teams on the same end-goals.
• Clarity: They provide clarity on what defines success or failure to make the shared objectives tangible.
• Decision-making: they inform decision-making by highlighting success and failure
• Accountability: The hold individuals and teams accountable for their contribution to shared goals.

Shifting from abstract KPIs or process-oriented metrics alone to outcome-based measurements shifts the focus how we collaborate and work. A shared benchmark across organizational structures fosters cross-team collaboration and streamlines the whole product development process.

• Greater Flexibility: Teams can adjust and tailor their methods to best achieve the desired outcomes, fostering a flexible work environment.
• Cultural / Mindset Shift: There is a shift towards innovation / experimentation and shared ownership rather than fulfilling given requirements.
• Efficiency: Focusing on outcomes can streamline operations and create processes for gaining and including continuous feedback loops (customers and process) on all levels of the organization.

Empowerment happens on each individual role. If there is clarity on what output creates the desired outcomes it fosters transparency. Each individual and team can immediately see their contribution towards the objective. It fosters a shared responsibility. Outcome metrics shift the focus from merely following prescribed tasks to achieving defined results, which naturally distributes power and control.

• Empowerment: By focusing on outcome, employees at all levels are empowered to make decisions that contribute to the strategic goals, rather than just following processes or requirements.
• Communication: They facilitate better communication about goals and strategies.
• Responsibility: They assign responsibility for specific outcomes to teams or individuals, regardless of their position in the hierarchy, fostering a sense of ownership.
• Innovation: With a clear understanding of desired outcomes, individuals at all levels are encouraged to innovate and experiment to achieve those outcomes.

Of course, if not done properly, it can easily happen that metrics just become “some numbers” that are communicated on a regular basis until they are not, and some people will not even remember that there were ever “numbers” shared in the first place.

What's next?

In itself, outcome metrics are a simple concept. There is nothing magical about it but they have the power to be a game-changer. Applied on an organizational level it might be groundbreaking. You can apply outcome metrics to any context or situation. Start with a simple use case in your team and experiment how it benefits your situation.