A Lean journey in Software Development - Seeing problems when and where they occur

This article is the fourth of a short series, describing few aspects of one concrete attempt to follow a Lean Approach in a Agile Software Development project. Each post of the series follows the same structure: I first explain my current understanding of one concept, and then present how we applied it in the last project I was involved in.

Like in most other complex working fields, unforeseen difficulties sooner or later show up in a Software Development project,  that hinder the effectiveness of your daily activities.

In the first of his QSM volumes, Jerry Weinberg has written:

It's not the event that counts, it's your reaction to the event.

Indeed, we are free to choose among a large set of options for reacting to those prickly events. Here are what I believe to be 2 extremes of the spectrum:

1. Do nothing:  accept them as "necessary inconveniences". If absolutely necessary, design a  workaround to the problem, and resume immediately to "real, productive work".
2. Make Problem Solving a daily activity for everyone, at every level, starting by detecting problems as early as possible in the process (when they make no or little harm, and the environment still provides contextual data for investigation).

The first option seems to represent the path of least resistance, which is why we so often find it in companies. In isolation, most problems cause a seemingly negligible additional complexity to achieve the same outcome, so people quickly get used to them, and stop paying attention. Do you see the deathly spiral? In the sacred name of 'responsiveness to business needs', we build sure foundations for mediocre performance: over time this mindset leads to clumsy, fragile environments and permanent firefighting.

At the opposite of the spectrum, the exploration of the second mindset aims at generating solid organisations, that don't fear "crisis situations". Steve Spear, respected author of "The High Velocity Edge", explains:

When you are seeing and solving problems everyday, there are no crisis per se; there are just problems that are bigger and more demanding than others.

"Everyone at every level adopts a focused, almost obsessive, attention to discovering, solving and capitalizing on problems." I think this mental model alone guarantees long term success: the organization gets results faster than its competitors because behind in each problem lies a true opportunity for improvement.

Seeing problems "where and when they occur" requires the habit of "Genshi Gembutsu" (go and see for yourself):

• interview end users and see them in action as often as you can, for knowing (not assuming) what are the problems you need to work on first, and fixing early your false assumptions,
• understand the actual working place, where value-adding activities take place, for grasping the working conditions and spotting (not imagining) the true obtacles to attaining your goals.

This article focuses on the second item. I highly recomment reading "The Lean Startup", of Eric Ries, for those who need guidance on the first.

Look at this picture:

Perhaps this team and their boss believe the project is on track. This may be true... or not. Perhaps the situation deteriorates a little everyday, but nobody notices the trend because the physical environment simply gives no clue. This is a typical problem in software development: one key challenge is to design the team environment to help them realize problems without effort as they arise.

In his book "The High Velocity Edge", Steven Spear emphasizes the crucial importance of seeing problems early:

(...) detect abnormality in things that one cannot see by default (...) if they go wrong and the problem is not detected, they cause trouble. If they are seen as they start to go wrong, their effects may be mitigated quickly and the organisation may learn from their occurrence.

Real World Experiments : Environments that sustain Problem Awareness

• Value Stream Map from initial idea to software in production

We felt, but couldn't prove, that our backlog was exclusively filled with "urgent stories", at the expense of important subjects for the middle and long term client satisfaction.
The outcome (value) of Software development consists in solutions to actual needs and problems, faced by the target users. With that definition in mind, we decided to map on the wall the flow of value from raw idea in the head of the client to features available and used in production:

Flow from raw Needs to Development

Flow from Development to Production

The goals:

1. develop a shared, constant awareness of where we stand and where we go,
2. avoid discrepancies between client wishes and actual developments,
3. see immediately when an idea gets lost or poorly addressed,
4. notice and address bottlenecks early, before they cause any harm,
5. limit WIP and slowly develop a pull system

• Standardized daily meetings

Informative and short daily meetings bring many concrete benefits to projects.  Make them too long or disorganized, and they become a huge waste of time for everyone.

Besides timeboxing the activity to 15 minutes, we decided to describe and publish on the wall the fixed sequence of questions that would define, for us, a "good" daily meeting.

Standard Daily Meetings

This simple technique proved very useful for bringing difficulties early to everyone attention, and organize our working day accordingly: who's taking care of the broken build? Are we late? why? Let's  agree on a critical working path. Is there a need for a demo or a quick design session before starting the day?

• Continuous Integration : Unit, Acceptance, Performance and Platform Tests

We considered very important to be warned as soon as something that used to work suddenly stopped working. Our Continuous Integration System played perfectly that role: 

- each VCS commit triggered the complete build, Unit and Acceptance Tests, 

- every night the system was deployed on a Bench platform and we tested a few performance metrics, 

- every hour, the different platforms were compared to a predefined standard (symbolic links, file system sructure, file permissions...)

Platform Tests

• Standardized Deployment process

Several applications composed the solution under development, and deploying a release was far from a simple task. We were working hard to slowly automate the full deployment process, heading towards a push-button deployment.
Howebver, as long as deploying the application required more than 1 step, we used to print our best known procedure before each deployment, and log any anomaly as we were progressing through the operation:

Deployment Standard

After each deployment, we systematically explored the conditions of each encountered difficulty, and applied a rigorous problem-solving process to define and test a countermeasure for the next round:

Tracking Anomalies after Production deployments

• Pair Programming

I've experienced pair-programming for more than 10 years in many very different projects. I am a strong proponent of this XP practice for many reasons.
Among other benefits, pair-programming helps detect and fix problems in the code as early as possible: the co-pilot reads and analyses the code as it is written, warns his mate about potential risks, raises questions and brings his perspective to produce a solid emerging design.
From my experience, this practice is much more effective at producing bug-free programs than a formal code review, executed days (or weeks) after the code has been checked in.

Conclusion
The examples provided above represent only a subset of what we have done in our context. Listing more would serve no purpose, as I believe every professional development project should customize its own system for detecting problems as they occur, for their specific value stream.
Good news: Problem Solving begins with Problem Awareness. Once your working environment triggers the attention of the right people on the right problems, the most trivial disappear by themselves, and the situation gets a little better.
I now think the next question should be: what is the thinking process of the best performing organisations for solving their problems? How can we apply it to our situation, in software development?

Waiting for the next article, I wish you a happy new year. See you in 2012!