A Lean journey in Software Development - Visualize your performance

I think very highly of my mentor and friend Régis Médina. However, I couldn't help feeling a little suspicious when he enjoined me to set up indicators and measure our performance everyday.
Tedious experience with a former high level manager came back to my mind: he used to patronize the development teams, bluntly exhorting people to improve his metrics of the project. I can't remember the exact nature of those metrics and to be honest, very few of my development team cared to look at them. Why? Because we felt no connection between these measurements and our daily activity.

I'm now convinced we were wrong in our attitude. True enough, this manager failed to involve the team when designing his key metrics. But more importantly, WE failed to understand that a good scoreboard, updated daily, could help us relate our work to measurable effects, hence, direct our efforts more effectively.

Completed with a clear and shared direction, few, simple operational indicators represent the best (if not only) way to know for sure whether or not we are progressing, and by how much.

Michael Ballé (french Lean Sensei) states :

"Lean starts at the gemba, there’s no way around it." 

Gemba is the actual place where the value gets created: for software projects, you typically find it at the team workplace where people interactions occur, and on the computer screens of developers, system engineers, and support functions.
When looking closely at your gemba, tons of problems should call for your attention, provided the environment is informative enough (more on this later): few, good performance indicators help direct our attention to the right, most important problems.

What characterizes a GOOD performance indicator?

• Performance as seen from the client perspective (Is his/her problem solved? Quality? Delay?),
• Displayed on the wall, for everyone to see,
• Simple to understand,
• With a visible target (the gap with actual measurement makes the problem explicit),
• Updated everyday, in less than 5 minutes,
• Graphed over time (to see progress and trends),
• With annotations explaining peaks and valleys,
• Reviewed frequently: key questions should be "where do we stand? what have we improved since the last time we met?" instead of "how much 
did we make last week?"

A Lean Sensei might argue that this list of key points isn't accurate nor exhaustive... well, this is a reflection of my current knowledge, and I have to say we unfortunately have not even managed to satisfy these criteria in our past attempts.

Performance in the Outcome or in the Process?

It is crucial to clearly distinguish outcome metrics (client, or end user centered) from process metrics ("internal", meaningless to the client).

Here are a few examples of outcome metrics applicable to software development:

• Lead Time from idea to working software in production
• Number of Production Incidents
• Right First Time newly introduced features

Let's now illustrate the "process metrics" concept to see the difference:

• Volume of Work In Progress
• Success of automated tests in the Continous Integration System
• Code coverage with automated tests
• Amount of time in Rework
• Mean time from code ready to working software in production

The outcome metrics only matter in the end: properly defined, they form the true performance indicators of the project. The process metrics focus on a particular aspect in the value stream, and may or may not help to see a problem early in the value creation process (before it deteriorates into a problem for the client).

A reference point for Leading Improvement

Performance indicators should support the Plan and Check phases of high level PDCA improvement cycles. We'll explore PDCA in another article to come, but let's illustrate the last point with two imaginary problems (notice the pattern in the description, clear evidence of the underlying scientific approach).

1- Performance indicator: Lead time for solving a user request => objective : less than a day 

We want to address every customer request within the day it was issued,
Our Mean Time addressing a customer demand is currently 10 days,
We have observed waste X in 12 occurences out of 15 from initial demand to final answer,
We think that waste X mainly stems from ...,
We are now experimenting action Z,
And expect the average Lead Time to drop to 4 days in 2 weeks,
Because A and B

2- Performance indicator: Number of incidents associated to production deployments => objective : 0

We want ZERO defects discovered after production deployments,
Our average number of quality defects per deployment is 2 over the last 2 months,
We have observed ...,
We are experimenting action Z for a month,
And expect the mean number of defects to drop by 1,
Because A and B

In both cases, the indicators reveal the problem (a gap between a desired performance and an actual outcome) AND are used to make predictions about the effect of the countermeasure in the future (objective: test the accuracy of our current understanding, and grow our knowledge in case we were wrong).

Real World Experiment : Performance Indicators for a Software Project

In the context of our large legacy project, we had to refocus on producing quality, and stop the devastating pattern of regressions and rollbacks following production deployments. We thus set up 2 indicators, targeted at these 2 aspects (I unfortunately have lost the photo for the second):

Ratio of Rollbacks in Production

Number of Production Incidents

We also set up an indicator on the accuracy of our estimates: at the end of each sprint (2 week iteration), we graphed the percentage of finished work compared to what we engaged at the beginning of the sprint.

Sprint reliability in terms of stories engaged

Retrospectively, I can say we largely missed the point with those indicators: we looked at them as a report of our past performance, not for testing PDCA experiment and driving improvement.
I believe the lack of momentum around them was partly caused by the fact that they got updated only once every 2 weeks, and have too rarely been the primary focus of attention in our team retrospectives (mea culpa).

On top of that, I am only now slowly realizing that we may have missed the most important indicator of all, directly inspired from my current reading, the excellent book of Eric Ries, "The Lean Startup": our speed (or cyle time) for making experiments and testing predictions toward total client satisfaction. As Eric states in his book:

Success is not delivering features; success is learning how to solve the customer's problem.

Well... albeit incomplete, our measurements looked like a random suite of numbers. Instead of questioning again the pertinence of those performance indicators, we decided to understand the current working conditions and stabilize the process outcome by tackling concrete problems at the gemba.