This is the first co-authored piece from Pooja Bharat and Jason Schreuder.

We (Pooja and Jason) have both worked in organizations where measuring the value of an agile approach to working and delivery is a hot topic.  We love the objectives and key results (OKR) framework as a method to bring discipline to measurement into an organization. This is a two-part series, and before we dive deep into OKRs in Part 2, this first part explores concepts like measurement and agile metrics and sets a baseline of differentiation between outcomes and output.  We hope this will lay a foundation for a better understanding of this nuanced topic.

The Importance of Measurement in an Agile Approach

There are two critical concepts that undergird an agile approach to work. They are empirical process control and continuous improvement. Essentially, we use tight feedback loops to inspect or approach and then adapt (tune and adjust) based on what we know.  In an increasingly complex environment, where the future is unpredictable, we rely on this sensemaking approach of empirical process control to refine and course-correct, plan and replan, forecast and then evaluate so that we know we are headed in the right direction. 

When you layer in a dedication to continuous improvement, you get a “process for improving the process” as you go.  This is how organizations around the world are approaching complex endeavors like product development in an agile way.  

Now, to be able to use empirical process control and implement a process of continuous improvement, we have to have some way to measure what we inspect, or what we see happening, in order to 1) make sense of it and then 2) make a decision (and act) on it. 

Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.

H. James Harrington

Metrics in Agile 

Conversations about metrics and measurement in the agile community have been raging for years.  Leaders in organizations that are turning their heavy, bureaucratic approaches into more lightweight, adaptive approaches that are better suited for the complexities of our modern world, are looking for answers to the question “how will I know this is working?” When we change from promised dates and watermelon status (reported as green [on the outside] but really red all over [the inside]), organizations want an elegant way to control, inspect, and adapt. So we toil to find the “right” metrics for an agile approach. 

A metric is defined as:

• A system or standard of measurement. 
• A set of figures or statistics that measure results.  
• Measure or something used to Measure.

Now, to measure is to estimate or assess the extent, quality, value, or effect of (something). We need good measures to run a business. Really, you need measures to gain understanding and insight about any endeavor, whether it is training for a marathon or baking a cake.  In an agile organization, one that is navigating the complex realities of our ever changing world, we have to not only measure, but also use metrics, or a system of measurement, to help us “steer the ship through a fog” — we know we want to sail to someplace awesome, but we don’t have an exact map to get there AND the environment is complex (we will return to this nautical metaphor later). 

Common Approach to Strategic Business Management: KPI

Many organizations use key performance indicators (KPIs) to understand how things are going from a performance perspective, and then use them to make strategic decisions. 

The components of a KPI are:

Metric: something to measure (could be a discrete value, or a rate of change)

Current Value: where you are now, your baseline. 

Target Value : Is the Minimum or Maximum Value of the metric that you want to have (# of contracts signed per period, min-max revenue, # of engaged qualified leads in sales funnel, # production support items closed, etc.). 

If something is designated as a key area of business then it is likely something you constantly need to work on and monitor. For example: infrastructure support or claims processing & service. It would be useful to have a metric to understand how the business is doing. Here, a KPI looks at the business, as usual and allows leaders to monitor performance. KPIs are great if your business operates in a space that is fairly stable, and humming along with business, as usual is the norm — perhaps your key customer is pretty static, and their preferences don’t change much over time. And perhaps the competitive landscape provides enough stability that the work of your organization can be defined as complicated, and not too complex. Increasingly, though, as companies make the shift to fully digital AND the competitive landscape is ever-changing, they must turn to something more than just a measure of success with lagging indicators like KPIs. They may have been fine to “course correct” and adjust as needed before but what got you here won’t get you there. 

Returning to our nautical metaphor, where the environment you operate in is a bit foggy (complex), and you perhaps don’t have a perfect map to tell you where you are, BUT you know of some key objectives you want to succeed in along the way (learn to sail, not break any equipment or tip the boat, find someplace awesome to stop and have a picnic). You will need a different system of measurement. 

A system that measures the right thing, and allows us to maximize the why, and produce value, and then tune and adjust accordingly. Too often, we see organizations focused on activities, or at best outputs, rather than outcomes.  

Outcomes, Outputs, and Activities  

Organizations have systems of measurement in place in order to make decisions, and then tune and adjust as described above. To measure is to estimate or assess the extent, quality, value or effect of (something). 

If a measurement matters at all, it is because it must have some conceivable effect on decisions and behaviour. If we can’t identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value.

Douglas W. Hubbard, How to Measure Anything: Finding the Value of “Intangibles” in Business

We must be careful, though, to understand WHAT we are measuring: activities, outputs, or outcomes. 

Outcomes are the consequence of doing something.  In many cases, it is a change in the satisfaction or experience of a customer or user. To use an all-too-real example of a virus outbreak, an outcome would be an improvement in the infection rate of a group of people. This is hard to measure, but is driven by purpose and the “why” behind an intended impact. 

Let’s take a look at two other concepts that are easy to measure, and driven by “how” and “what” questions.  These are easier for us to intuitively grasp and thus orient toward. 

Outputs are things that people produce. For our virus outbreak scenario, an output is the number of doses of a vaccine that have been administered. 

Activities are things that people do, a task to be undertaken. For example, the number of tests that have been run.

So if outcomes help us understand the “why” and outputs and activities only address the “how” and “what”, shouldn’t we Start with Why  (a la Simon Sinek) and get a better understanding of the consequences and impact? 

Dude’s Law

Finally, we get to the header image on this article. 

Dude’s law states : The Value of anything equals  the “Why” over the”How”. 

If you really want to make an impact, and thus have value, you have to maximize the “why” over the “how”. 

Organizations fall into the productivity trap and “get busy” with activities and output, before they clearly articulate the “why”.  So, focus on outcomes, and maximize their use — that is the path to value, and a primary drive in an agile organization. 

Be Mindful of What and How you Measure 

Just a little bit of a word of caution here, on what you measure.  

Tell me how you measure me, and I will tell you how I will behave.  If you measure me in an illogical way do not complain about illogical behavior.

Dr. Eli Goldratt

Sometimes measures can create a perverse incentive.  Take a look at the cobra effect in India while it was under british rule.  To reduce the cobra population in Delhi, the british government offered a bounty for every dead cobra. At first it worked, until enterprising people began to breed them as a source of income. Eventually the bounty program stopped, and thousands of cobra breeders set their now worthless snakes free, thereby increasing the cobra population.  

As another example, take a look at social media where you see the same effect. The metric, and unfortunately, goal for many is to get attention via likes and upvotes. We see what sort of trouble THAT has caused. 

So, be mindful of how you measure. People will find a way to game the system for better or worse. 

A Better Framework: OKR 

OKRs provide agile organizations with a metric, or a system of measurement, that can provide a lot more clarity in a complex environment than the lagging indicator of a KPI can. 

The OKR (objective and key results) framework provides us an effective way to start with why, and to clarify the value of outcomes we want the organization to orient on. 

The objective is the goal — simply put, it is what you want to achieve. It should be clear, compelling and aspirational.  Key results will link  to objectives, and they aid us on two fronts:  First, they remove ambiguity by clarifying and quantifying what success for an objective should look like. Secondly, they provide you a measurement of progress towards that Objective.  

You have to get the organizational habit of aspirational goal setting in addition to measuring success as you go. Clarifying your objectives is a way to unlock the “why” for people, and find untapped potential in an organization. And this type of a goal setting is a basic component of a coaching approach to help people perform better. People don’t excel when it is BAU only. 

It also gives a method that is iterative and incremental, just like an agile approach. 

Because strategy can only capture a company’s best thinking at a given point in time, strategy (like a software program) needs to be refined and improved as people gain and distribute new experience and knowledge. Given this reality, sound strategy development processes should enable a company to create and adapt strategy quickly and iteratively… and allocate resources in changing environments.

Keith R. McFarland, Should You BuildStrategy Like You Build Software?,  MITSloan Management Review

So although many organizations are adopting lean and agile practices and principles for delivery, they still use a waterfall mindset and approach for goal setting.

Stay tuned for part 2 of this series where we will do a deeper dive on OKR.

Until the Next Iteration . . .

Pooja Bharat & Jason Schreuder