A Lean journey in Software Development - Seeing problems when and where they occur

This article is the fourth of a short series, describing few aspects of one concrete attempt to follow a Lean Approach in a Agile Software Development project. Each post of the series follows the same structure: I first explain my current understanding of one concept, and then present how we applied it in the last project I was involved in.

Like in most other complex working fields, unforeseen difficulties sooner or later show up in a Software Development project,  that hinder the effectiveness of your daily activities.

In the first of his QSM volumes, Jerry Weinberg has written:

It's not the event that counts, it's your reaction to the event.

Indeed, we are free to choose among a large set of options for reacting to those prickly events. Here are what I believe to be 2 extremes of the spectrum:

1. Do nothing:  accept them as "necessary inconveniences". If absolutely necessary, design a  workaround to the problem, and resume immediately to "real, productive work".
2. Make Problem Solving a daily activity for everyone, at every level, starting by detecting problems as early as possible in the process (when they make no or little harm, and the environment still provides contextual data for investigation).

The first option seems to represent the path of least resistance, which is why we so often find it in companies. In isolation, most problems cause a seemingly negligible additional complexity to achieve the same outcome, so people quickly get used to them, and stop paying attention. Do you see the deathly spiral? In the sacred name of 'responsiveness to business needs', we build sure foundations for mediocre performance: over time this mindset leads to clumsy, fragile environments and permanent firefighting.

At the opposite of the spectrum, the exploration of the second mindset aims at generating solid organisations, that don't fear "crisis situations". Steve Spear, respected author of "The High Velocity Edge", explains:

When you are seeing and solving problems everyday, there are no crisis per se; there are just problems that are bigger and more demanding than others.

"Everyone at every level adopts a focused, almost obsessive, attention to discovering, solving and capitalizing on problems." I think this mental model alone guarantees long term success: the organization gets results faster than its competitors because behind in each problem lies a true opportunity for improvement.

Seeing problems "where and when they occur" requires the habit of "Genshi Gembutsu" (go and see for yourself):

• interview end users and see them in action as often as you can, for knowing (not assuming) what are the problems you need to work on first, and fixing early your false assumptions,
• understand the actual working place, where value-adding activities take place, for grasping the working conditions and spotting (not imagining) the true obtacles to attaining your goals.

This article focuses on the second item. I highly recomment reading "The Lean Startup", of Eric Ries, for those who need guidance on the first.

Look at this picture:

Perhaps this team and their boss believe the project is on track. This may be true... or not. Perhaps the situation deteriorates a little everyday, but nobody notices the trend because the physical environment simply gives no clue. This is a typical problem in software development: one key challenge is to design the team environment to help them realize problems without effort as they arise.

In his book "The High Velocity Edge", Steven Spear emphasizes the crucial importance of seeing problems early:

(...) detect abnormality in things that one cannot see by default (...) if they go wrong and the problem is not detected, they cause trouble. If they are seen as they start to go wrong, their effects may be mitigated quickly and the organisation may learn from their occurrence.

Real World Experiments : Environments that sustain Problem Awareness

• Value Stream Map from initial idea to software in production

We felt, but couldn't prove, that our backlog was exclusively filled with "urgent stories", at the expense of important subjects for the middle and long term client satisfaction.
The outcome (value) of Software development consists in solutions to actual needs and problems, faced by the target users. With that definition in mind, we decided to map on the wall the flow of value from raw idea in the head of the client to features available and used in production:

Flow from raw Needs to Development

Flow from Development to Production

The goals:

1. develop a shared, constant awareness of where we stand and where we go,
2. avoid discrepancies between client wishes and actual developments,
3. see immediately when an idea gets lost or poorly addressed,
4. notice and address bottlenecks early, before they cause any harm,
5. limit WIP and slowly develop a pull system

• Standardized daily meetings

Informative and short daily meetings bring many concrete benefits to projects.  Make them too long or disorganized, and they become a huge waste of time for everyone.

Besides timeboxing the activity to 15 minutes, we decided to describe and publish on the wall the fixed sequence of questions that would define, for us, a "good" daily meeting.

Standard Daily Meetings

This simple technique proved very useful for bringing difficulties early to everyone attention, and organize our working day accordingly: who's taking care of the broken build? Are we late? why? Let's  agree on a critical working path. Is there a need for a demo or a quick design session before starting the day?

• Continuous Integration : Unit, Acceptance, Performance and Platform Tests

We considered very important to be warned as soon as something that used to work suddenly stopped working. Our Continuous Integration System played perfectly that role: 

- each VCS commit triggered the complete build, Unit and Acceptance Tests, 

- every night the system was deployed on a Bench platform and we tested a few performance metrics, 

- every hour, the different platforms were compared to a predefined standard (symbolic links, file system sructure, file permissions...)

Platform Tests

• Standardized Deployment process

Several applications composed the solution under development, and deploying a release was far from a simple task. We were working hard to slowly automate the full deployment process, heading towards a push-button deployment.
Howebver, as long as deploying the application required more than 1 step, we used to print our best known procedure before each deployment, and log any anomaly as we were progressing through the operation:

Deployment Standard

After each deployment, we systematically explored the conditions of each encountered difficulty, and applied a rigorous problem-solving process to define and test a countermeasure for the next round:

Tracking Anomalies after Production deployments

• Pair Programming

I've experienced pair-programming for more than 10 years in many very different projects. I am a strong proponent of this XP practice for many reasons.
Among other benefits, pair-programming helps detect and fix problems in the code as early as possible: the co-pilot reads and analyses the code as it is written, warns his mate about potential risks, raises questions and brings his perspective to produce a solid emerging design.
From my experience, this practice is much more effective at producing bug-free programs than a formal code review, executed days (or weeks) after the code has been checked in.

Conclusion
The examples provided above represent only a subset of what we have done in our context. Listing more would serve no purpose, as I believe every professional development project should customize its own system for detecting problems as they occur, for their specific value stream.
Good news: Problem Solving begins with Problem Awareness. Once your working environment triggers the attention of the right people on the right problems, the most trivial disappear by themselves, and the situation gets a little better.
I now think the next question should be: what is the thinking process of the best performing organisations for solving their problems? How can we apply it to our situation, in software development?

Waiting for the next article, I wish you a happy new year. See you in 2012!

A Lean journey in Software Development - Visualize your performance

I think very highly of my mentor and friend Régis Médina. However, I couldn't help feeling a little suspicious when he enjoined me to set up indicators and measure our performance everyday.
Tedious experience with a former high level manager came back to my mind: he used to patronize the development teams, bluntly exhorting people to improve his metrics of the project. I can't remember the exact nature of those metrics and to be honest, very few of my development team cared to look at them. Why? Because we felt no connection between these measurements and our daily activity.

I'm now convinced we were wrong in our attitude. True enough, this manager failed to involve the team when designing his key metrics. But more importantly, WE failed to understand that a good scoreboard, updated daily, could help us relate our work to measurable effects, hence, direct our efforts more effectively.

Completed with a clear and shared direction, few, simple operational indicators represent the best (if not only) way to know for sure whether or not we are progressing, and by how much.

Michael Ballé (french Lean Sensei) states :

"Lean starts at the gemba, there’s no way around it." 

Gemba is the actual place where the value gets created: for software projects, you typically find it at the team workplace where people interactions occur, and on the computer screens of developers, system engineers, and support functions.
When looking closely at your gemba, tons of problems should call for your attention, provided the environment is informative enough (more on this later): few, good performance indicators help direct our attention to the right, most important problems.

What characterizes a GOOD performance indicator?

• Performance as seen from the client perspective (Is his/her problem solved? Quality? Delay?),
• Displayed on the wall, for everyone to see,
• Simple to understand,
• With a visible target (the gap with actual measurement makes the problem explicit),
• Updated everyday, in less than 5 minutes,
• Graphed over time (to see progress and trends),
• With annotations explaining peaks and valleys,
• Reviewed frequently: key questions should be "where do we stand? what have we improved since the last time we met?" instead of "how much 
did we make last week?"

A Lean Sensei might argue that this list of key points isn't accurate nor exhaustive... well, this is a reflection of my current knowledge, and I have to say we unfortunately have not even managed to satisfy these criteria in our past attempts.

Performance in the Outcome or in the Process?

It is crucial to clearly distinguish outcome metrics (client, or end user centered) from process metrics ("internal", meaningless to the client).

Here are a few examples of outcome metrics applicable to software development:

• Lead Time from idea to working software in production
• Number of Production Incidents
• Right First Time newly introduced features

Let's now illustrate the "process metrics" concept to see the difference:

• Volume of Work In Progress
• Success of automated tests in the Continous Integration System
• Code coverage with automated tests
• Amount of time in Rework
• Mean time from code ready to working software in production

The outcome metrics only matter in the end: properly defined, they form the true performance indicators of the project. The process metrics focus on a particular aspect in the value stream, and may or may not help to see a problem early in the value creation process (before it deteriorates into a problem for the client).

A reference point for Leading Improvement

Performance indicators should support the Plan and Check phases of high level PDCA improvement cycles. We'll explore PDCA in another article to come, but let's illustrate the last point with two imaginary problems (notice the pattern in the description, clear evidence of the underlying scientific approach).

1- Performance indicator: Lead time for solving a user request => objective : less than a day 

We want to address every customer request within the day it was issued,
Our Mean Time addressing a customer demand is currently 10 days,
We have observed waste X in 12 occurences out of 15 from initial demand to final answer,
We think that waste X mainly stems from ...,
We are now experimenting action Z,
And expect the average Lead Time to drop to 4 days in 2 weeks,
Because A and B

2- Performance indicator: Number of incidents associated to production deployments => objective : 0

We want ZERO defects discovered after production deployments,
Our average number of quality defects per deployment is 2 over the last 2 months,
We have observed ...,
We are experimenting action Z for a month,
And expect the mean number of defects to drop by 1,
Because A and B

In both cases, the indicators reveal the problem (a gap between a desired performance and an actual outcome) AND are used to make predictions about the effect of the countermeasure in the future (objective: test the accuracy of our current understanding, and grow our knowledge in case we were wrong).

Real World Experiment : Performance Indicators for a Software Project

In the context of our large legacy project, we had to refocus on producing quality, and stop the devastating pattern of regressions and rollbacks following production deployments. We thus set up 2 indicators, targeted at these 2 aspects (I unfortunately have lost the photo for the second):

Ratio of Rollbacks in Production

Number of Production Incidents

We also set up an indicator on the accuracy of our estimates: at the end of each sprint (2 week iteration), we graphed the percentage of finished work compared to what we engaged at the beginning of the sprint.

Sprint reliability in terms of stories engaged

Retrospectively, I can say we largely missed the point with those indicators: we looked at them as a report of our past performance, not for testing PDCA experiment and driving improvement.
I believe the lack of momentum around them was partly caused by the fact that they got updated only once every 2 weeks, and have too rarely been the primary focus of attention in our team retrospectives (mea culpa).

On top of that, I am only now slowly realizing that we may have missed the most important indicator of all, directly inspired from my current reading, the excellent book of Eric Ries, "The Lean Startup": our speed (or cyle time) for making experiments and testing predictions toward total client satisfaction. As Eric states in his book:

Success is not delivering features; success is learning how to solve the customer's problem.

Well... albeit incomplete, our measurements looked like a random suite of numbers. Instead of questioning again the pertinence of those performance indicators, we decided to understand the current working conditions and stabilize the process outcome by tackling concrete problems at the gemba.

A Lean journey in Software Development: first, be clear about the direction

This article is the second of a short series on the definition of Lean and its application to the Software Development field.

The next 3 or 4 posts will follow the same structure: I first explain my current understanding of the concept, and then present what we did in the last project I was involved in.

Short disclaimer: I may be making a huge mistake trying to catch the Lean Approach with bullet-points, but I can't resist the temptation of such a convenient way for me to articulate what I learned in distinct articles.

As far as I understand Lean Thinking today, if you (individual, team, organization) want to consistently outperform the competition in the long term for any complex field, you should address 4 key aspects as rigorously as you can (if possible with the support of a coach):

1. Define your 'True North', the long-term direction you will commit to pursue for years,
2. Visualize everyday your actual Performance and its evolution over time,
3. Identify clearly what problem(s) you need to work on now,
4. Progress by applying the Scientific Method for Problem Solving.

Define your 'True North': what for? How?

I attended last week to the 1st European Lean IT Summit in Paris. Gee! My little brain really enjoyed those 2 days!

Dan Jones (co-author of 'The Machine that changed the World' with James Womack, and top-notch thought Leader of the Lean community) shared with us his definition of 'Lean':

"Lean is the Practice of using the Scientific Method to solve Business Problems in order to create Value"

Wow! There is absolutely no waste in this sentence. Almost every word deserves a careful attention.

In particular, Dan emphasized that the 'Value' had to be considered along 3 axis:

• for the customers: help them solve problems in their lives,
• for the people experimenting the process: they become experts in their fields and grow everyday their problem solving skills,
• for the organization to grow and prosper.

This may at first look obvious, but have you met any business organization truly value-oriented, considering this entire definition? I have not.

Anyway, we certainly need to find words more tangible than "Value Creation" for making a good Vision statement for a project, a product or a company.

What about following these guidelines?

• The 'True North' must state in very few words the ideal success of your project in the middle or long term, as you see it today,
• It will serve as the direction for ALL your efforts,
• As a direction it does NOT have to be precise, so avoid delving into much details, or spending months to tailor it,
• Your activity ultimately must bring value to someone : start by identifying your client(s), and (if possible) ask him to articulate your conditions of success,
• The statement needs to combine two qualities : it has to be CLEAR to everyone, and SHARED by everyone involved in the value chain.

Aha! Interesting, isn't it? Let's now examine how we attempted to apply these guidelines to a concrete case in the Software Development field.

Real World Experiment : defining a True North for a Software product

For you to understand certain of our choices, get a look to this quick overview of our context when I joined the company 2 years ago:

• The product: public video portal (live and on-demand) for mobile devices,
• 1 unique customer: the historical and largest french Telecom Company (our customer is not the end user of the product),
• over 1 million unique visitors each month, growing, with peaks of traffic during sport events,
• heavy-weight project generating revenue, and critical for the short and middle-term cashflow of the company,
• 6-year old project, with loads of features dumped on top of a 3-month prototype,
• ecosystem of 15 applications (front apps, Web interfaces and batches, taking care of provisioning, content management, customer support, statistics...),
• 450 thousands lines of code, mostly in Java,
• many layers of technical debt (unused features, complicated code, different technologies for the same needs),
• 70 production servers,
• about 15 people directly involved: 4 system engineers, 1 functional expert, 8 developers, a network expert and the project manager,
• several critical incidents in the past months, resulting in Low Quality of Service (slow response time) and potential loss of statistics

This of course depicts a very incomplete presentation of the situation. I could add more to it, but prefer to stop here to focus on the core of our current subject.

The development team created a first version of our True North, and asked a feedback from 2 client representatives, that led to a few minor changes.

We identified our client to be the 2 distinct departments we were dealing with on a regular basis:

• one was affiliated to the Marketing department, and expressed the needs for the next-generation capabilities of the product,
• the other focused exclusively at the quality of the overall system running in production (our part formed a portion of a more global solution)

Although we found many reasons to object to this bicephalous mode of organisation, we had no choice but to adapt.

We then attempted to state what problem we had to solve, from the client perspective: "Do what I want, when and where I want it, be reliable and do not make me waste my time".

We published the result on the wall, for everyone to see and remember:

The team forged a few key performance metrics out of it, and accepted it for a while.

After about 8 months, we noticed however the low level of energy this vision statement created in the organisation. We formulated the hypothesis this was due to the poor clarity of the message (post-its scattered all over the place), and the fact that it was not formulated by the client himself.

We want our client to be totally satisfied. We thus directly questioned the client:

"What are our conditions of success? What will make you say, one year from now, 'I am happy at what you have accomplished'?"

We then tried to capture the core vision in very few key points to make them stick in our memories (we published underneath goals more specific to each client division).

We found this last version more satisfying (even though less pretty):

We could proudly say to anyone getting into our room:

"Here is our direction, the Graal we are after"

I remember situations where this post-it directly helped deciding between several choices for action:

"This idea may cost a little more than this other one... but is it the right question, really? Look at our direction (pointing at the wall): which option do you think will provide the best user experience?"

More pratically, we used this clear vision statement to determine how to measure our performance in a less ambiguous manner... but this story is for the next post!

A Lean mindset for software development: why does it matter?

This article is the first in a short series on the definition of Lean and its application to the Software Development field.

I often hear people say: "Software Development is a craft"... well sure, I agree with that.

Each good developer requires many years of experience to develop his/her craftsmanship in forging and maintaining high-quality, long-lasting code. She/he needs to learn from training, books, seminars, conferences, but mostly by doing day after day, advised by more experienced mentors... (and of course making mistakes)

Yet, projects in Software Development share common characteristics with other types of projects:

1- structurally, they are defined by the presence of stakeholders, a team, limited financial means and goals to be attained in a finite amount of time,

2- dynamically, they all follow (consciously or not) some kind of working process, and encounter unforeseen obstacles along their way.

The Software field is still very young compared to other businesses.

This is I think why any serious software organisation looking for more-than-average results must look at what top performers in other fields do to stay well ahead of all their competitors.

Some recent studies demonstrate that there concretely exists a common mindset and type of behaviour in the face of difficulties encountered by all those organisations: read "The High Velocity Edge" (Steve Spear) for thorough examples applying to several, very different domains.

Let's review a few common false assumptions before moving on.

Lean IS NOT :

• one more trick or control tool invented by managers to exploit workers,
• about cost reduction and layoffs (as some want to make believe),
• the rigid application of standards defined by some Quality Department,
• Kanban (but, Kanban may sometimes be used as a tool, following Lean Thinking under a certain set of circumstances)

Where does the term "Lean" come from?

The expression "Lean Manufacturing" was first used in 1990 by James Womack, in a book called "The machine that changed the world", that describes the history of the automobile industry and in particular the very successful Toyota TPS (Toyota Production System). According to Wikipedia, Lean Production is "a practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful, and thus a target for elimination".

How do I define Lean today?

I've been thinking about this for the 2 past years, reading and experimenting in the context of a heavy project, with a mix of both very interesting and poor results.

In the meantime, I applied last year (2010) to a 6 day training course called "Lean IT Project Management", broken down into 3 chunks of 2 days for 3 consecutive months. The trainer was my former XP mentor, Régis Médina. This course offered me great insight on the core of "Lean Thinking", and ways to apply it to the IT world.

I'm definitely not a Lean expert, but actively want to undestand its nature. I personally prefer to talk about "Lean Thinking", or the "Lean Mindset".

First, let's be clear about one thing: true Lean Thinking is about developing the true asset of the organisation: its people. If you can't feel this quality in a company, then it is definitely NOT a Lean organisation (in spite of anything they claim or say).

From that assumption, I would define "Lean Thinking" as a rigorous, generic, adaptable and repeatable thinking process for targeting operational excellence and maximizing the value (thus, the satisfaction) of the end customer in any complex organisation.

What does it mean for a software development project?

From the former section, I hope you understand why the "Lean mindset" does matter.

In the next article, I'll be more specific about what it meant for my past experience in a large development and maintenance project for the Telecom industry, starting by what should be the very first step : defining the "True North" of the project team.

--------------

References :

I recommend you the reading of those books on the subject, as they have profoundly affected the way I think and work, all for the better :

• The Gold Mine, by Mickael Ballé
• The Lean Manager, by Mickael Ballé
• Toyota Kata, by Mike Rothers
• The High Velocity Edge, by Steve Spears

You can also find many interesting information online. I'll just mention 2 resources I find just great:

• the Lean Edge
• the Gemba Coach

10 years of Agile... and?

This year, the Agile 2011 conference will be held at Salt Lake City. A very symbolic choice for a symbolic year: the Agile manifesto got signed exactly 10 years ago very close by.

For this occasion, I’m thrilled and honored to having accepted Esther Derby’s invitation, and become member of the review team for the stage "Collaboration, Culture and Teams".

2011 means much to me too, as I discovered XP at the beginning of 2001 (exactly 10 years ago, what a coincidence!) in a large telecom manufacturer... The project was one of the very first Agile projects in France (second or third, I don’t know for sure).

Let’s try to quickly recap my last 10 years as an Agile practitioner:

• 2001, year 1 : a “short” rewrite project in C++ with predefined, fixed requirements (no user interface, but an old and mature communication protocol with other subsystems). I joined a set of 3 talented freelance developers, now friends for a long time. A frank success: we released the ultimate version of this pilot project on time, and with extremely few defects.
• Years 2 to 6 : a much bigger project for the same client (telecom company) with plenty of uncertainty along the way. Our incremental and adaptive approach allowed for an organic growth from scratch to a solid (“state-of-the-art”, would even write a customer, important telecom carrier) configuration tool for large network carriers. We all agree, none of us would have been able to predict and design ahead of time what we ended with. After 1 year, a natural split emerged in two subprojects (a kernel & GUI team that I was part of, and a plugin team for domain checks and wizards). After 3 years, the product met such success that the company assembled several other plugin development teams around the globe (US, Russia, China) for other domains, and a multi-user, collaborative version got released few months before I left for new adventures.
  
• Years 7 to 9 : an Agile offshore experience (team of 3 in France and 4 in India) as an XP coach, with a high turnover in the first year. Aside from showing the way of the engineering practices and driving the XP rituals (stand-ups, iteration planning games and retrospectives), my assignment consisted in giving rise to a genuine team spirit in a team divided by over 7000 kms of lands and seas. Concretely, we pair-coached for 3 years with the project manager to make it real (I often wore the technical lead hat), and we all traveled several times to physically get to know and trust each other.
  
• Year 10 : management through a mix of Agile and Lean approaches of a large project flooded with quality problems inherited from history. The development team was familiar with Scrum and XP engineering practices, which allowed us to experiment a (limited) Lean journey for Problem Solving on top of Agile, and progress steadily on the path of excellence.

This has been a truly exciting human adventure so far, thanks to the numerous great people I had the chance to work with... I am learning very much every passing year and realizing with a mix of fear and pleasure that the journey will never end.

What about the software community as a whole? Where does stand the Agile movement after 10 years?

Some Agile proponents publicly say, “Agile has been widely accepted as a mainstream way for building software ; the success of any project is only a matter of implementing correctly its tried and tested techniques!”

Paradoxically, I think that the stakes have never been so high for the Agile movement.

First, the vast majority of so-called agilists deliberately ignore the roots and motivations under the practices: any fellow having built a product backlog, worked iteratively and attended a retrospective will consider himself an expert, and ruin the image of Agile by his blathering. Not true? Just look and listen around you... “Agile” has become a revered buzzword, the temptation is just too high for people seeking quick money.

To XP core values of Communication, Simplicity, Feedback, Courage, Respect... we need to add Humility today: the good practitioner needs to keep the beginner’s mind, in the sense of “Shoshin” (http://en.wikipedia.org/wiki/Shoshin).

Motivated persons might devour many books on any subject, but that will not make them proficient or experts... Practice only does. Getting involved in real projects for years, testing, coding, refactoring, pairing... and the most important, making mistakes.

I don’t know for the rest of the world, but I believe there are scarcely anybody (a handful at most) that I would call an Agile expert in France. I’ve never claimed to be one, notwithstanding my double-digit experience.

The second source of fragility comes from inside the movement.

For the past three years, a dangerous rift have been threatening the Agile movement unity: in response to the neat predominance of Scrum over the other Agile flavours, Bob Martin has launched and popularized the “Software Craftmanship” movement.

The two approaches basically differ in what they consider should be the primary focus of attention for a software organisation:

• Organisation and communication (soft skills) for Scrum
  
• Coding as a craft (technical skills) for Software Craftmanship

This creeping internal conflict is now discussed openly on many top-notch blogs (Dan North, Robert Martin, and Martin Fowler among others). See Martin’s article for a summary at http://martinfowler.com/bliki/CraftmanshipAndTheCrevasse.html.

How sad Scrum has prevailed over XP. Aside from this certification folly, I think the system advocated by XP more powerful than Scrum or Software Craftmanship, as it devises a synergetic work on both the two aspects.

Finally, Agile, Scrum, XP, Software Craftmanship, Lean... who cares? These are mere labels. We professional software developers are employed to help our clients and users by delivering the best possible tool for their specific, real-life needs, on time. That only matters. The rest is crap.

And happy birthday, sir Agile!

Increase the odds of a lasting, healthy software project

Standard project life today: same as 15 years ago... with iterations

I have too often witnessed (and heard about) the same dramatic pattern of behavior over the course of software development projects:

1. Startup: Zeal, pleasure, exploratory thinking, excitement and optimism in the first few months, typically lead by a very small team of focused people with a pioneer mentality. They get acquainted with the domain, form a shared vision of the project with stakeholders, produce a product backlog of user stories to attain that goal, and happily engage in the battle (after optionally having spiked few technical solutions),
2. Expansion: the team expands (often too much and/or too quickly), in order to increase its “story production capacity”. The shared vision and momentum from the first months begin to falter. Yet, most team members still feel motivated and involved in what appears to be a great adventure. Code base grows at light speed.
3. Cruise control: The team less and less considers the long term aspects, and enters in a routine mode, continuously stacking up features, patching here and there the old design to make it fit, without questioning it in depth. In this context, project stakeholders consider the project mature and focus on team velocity. The project’s no longer appealing to innovators, that flee towards more juicy, creative horizons. Technical and functional debts become the norm, but there is very little time and money to address them: let’s be serious, clients pay for getting features now, or they will leave for competition, right?
4. Great Depression: The project has become a huge, fat mammoth, deadly wounded with numerous patches, obsolete code, and technical debts. Adding features turns difficult as heck, and stakeholders just don’t understand why. The team, almost exclusively composed of recent members now, suffers the consequences of past lousy project management. They ask the pace of demands to slow down for some time in order to cut the fat off the code. However, who are these guys anyway? They get slower and slower every month, and would like to slow down even more? They must be either lazy, incompetent, or, more likely, dumb idealists striving for perfection everywhere... Let’s push for quick and dirty for the moment ; we’ll invest time in beauty later, promised!
5. Collapse: Adding a very basic feature takes a month or two. The management decides we need to redesign from scratch, or the client stop to pay altogether and go for competition. The project slowly dies, a ghost team manages the transition to the new system with a mixed feeling of bitterness, rage and lack of interest.

Does it look even vaguely familiar to your experience? Of course, this “tale” depicts a very simplistic and tragic view of the reality. Yet, some projects succeed! How come? With courage and support from management, there is room for action in steps 2, 3 and even 4.

My three cents of advice

If I had only 3 meta-advices to give for software projects, these would be today:

1/ Alignment first :

Teamates strive to be aligned at all times on standards and processes to move forward and respond to change. 
Very small teams (from 4 to 6 members) achieve this with very little effort. By my experience, the difficulty is simply exponential above 8 members.
However remember that alignment applies outside the team boundaries as well :

• build a ubiquitous language from the code to the user stories
• manage stakeholders expectations, to build trust (read the instructive book from Naomi Karten on the subject, or at least this introductory article)
• maintain and update a shared vision, a common understanding of where you stand and where you would like to be a year or two from now (this kind of product ownership develops a genuine motivation for everybody as a side effect).
  
  

2/ Expertise within: 

Nobody should be indispensable, I agree with that. Yet, if the project’s technically complex, a healthy team needs at least 2 proficient or expert members full time (see Dreyfus model on Wikipedia for more details on this skill distribution model).
On living (features under development) projects in Cruise Control phase, the day-to-day activities may require less innovation. However, this certainly does not mean the coding activity has become trivial ; you always need technical leaders on your team, attached to high quality built-into the code, and that other members will trust. Their resignation could represent the visible sign of the future collapse, unless you make sure the best elements in your team are more than just “competent”.

3/ Beginner’s mind and Continuous Exploration:

Extend (or revive) the high energy momentum of step 1, by encouraging focused exploration activities at every iterations (functional, technical or organisational, decided in iteration retrospectives).  

Let me share with you this piece of Zen wisdom from a blog (http://zenhabits.net/how-to-live-life-to-the-max-with-beginners-mind/): “Zen philosophy encourages us to keep our beginner’s mind. What does this mean? Beginner's Mind is about using the spirit of enquiry – without getting stuck in preconceived ideas.”

Invest part-time on learning, broaden and stretch your mind beyond the mainstream thinking, listen to your intuition, experiment new techniques and share with your peers. Reject those silver-bullet dogmas said to be applicable everywhere.

Hey, what about the current mainstream Agile dogmas then?
I reckon Agile may not be applicable everywhere. Values and principles advocated by Agile methodologies will help only if applied matter-of-factly by people who deeply understand their underlying reasons and motivations. Same thing for Lean thinking process: start your journey with a Lean coach that also happens to know intimately what software development is all about (this is a rare gem today).

Exploration requires a large enough Time Horizon

Investing on middle and long-term future in a context of strong market pressure... how many organisations dare to do that? Aren’t they aware of the risks?

In his 4th volume of the QSM series, “Anticipating Changes”, Jerry Weinberg introduces the concept of Time Horizon: “Any change seems more expensive than the status quo, and it is, if you look on a short enough time horizon”.

Striving to convince managers to invest in costly activities that (you think) will guarantee a high ROI potential in the long run makes little sense, if you don’t first make sure they value this Time Horizon... the bad news is that they often don’t so much, or giving in because of too much market pressure.

I read from a blog post (http://tinyurl.com/252r4wx) : “At the operational level one has a time horizon of days and weeks, while at the strategic level one has a time horizon of several years”

Well, I believe any responsible, competent, developer should work at both levels. Stakeholders seldom understand this, which makes our job so terribly interesting and difficult at the same time!

Indeed, it is our responsibility, as developers, to develop a strategic attention for the project inner quality. The growing, insidious decay of code over time is completely invisible to stakeholders (unlike in manufacturing, where any external attentive eyes can notice the inventory stacks, the mess on the floor impeding movements or the bottleneck machine on the production line...).

A team without this level of attention can develop relatively fast for months, in some cases even a couple years, before bad effects start to show up. However, then it is often too late, or at least very expensive, to turn the system into a healthy, productive, state again. (See the story at the beginning of this post) I like to often recall my coworkers that code is both the product we ship to end users and our working environment. Are we ready to mess it up? (unless I know my project will be killed in less than a month, I’m not)

What can we do at our level?

There are several scales of actions :

- At a fine-grained level, the Refactoring automatism before every commit represents the only sustainable way to clean code, and allows best designs to emerge naturally over time through the coding activity flow itself.

- Pair-Programming and Retrospectives (after every iteration and quaterly) are also strong allies, as they offer room for tactical and strategic exchanges between peers, to build up alignment and shared vision. Retrospectives also make it easy to share problems and develop corresponding action plans as a team.

- Slack time ensures the long term success of a project. The key is to put some distance between you and your project from time to time and:

• explore design ideas
• take control of yet “enemy” territories (harness legacy with tests, and redesign parts)
• perfect the testing DSL (invest in your production tools)
• share your knowledge (dojos, presentations...)
• expand your mind with new techniques, new paradigms (invest in yourself)

- Finally, develop Awareness for you and stakeholders, with visible data and charts:
Gather data evidence before irreparable damages surface, in order to effectively communicate the problem with stakeholders.
Be factual, and treat stakeholders with much respect to gain their trust (they don’t know, or don’t remember, what software development is... and what? This is our area of expertise, not theirs, right?).

Of course, those measures will not guarantee the success of your project, but I feel ready to bet they will significantly increase the odds!

Dynamics of project success and failures

Let’s examine one aspect the complex nature of software projects, that lately has given me a hard time.

How do well-intentioned stakeholders sabotage  technical projects?

So many large technical projects suffer from stakeholders misunderstanding of complex, underground dynamics. In period of stress, or when the project releases turns out to be late, they sometimes hastily take decisions that dramatically lowers the odds of success, although they genuinely believe they act for the good.

Imagine this common case-scenario:

• A 5 year-old, pretty big, software project, with very few automated tests, and having slowly accumulated heavy technical and functional debts over the years,
  
• Early team members have nearly all quit (current members have all less than 2 year-experience on the project),
  
• The team is the only true asset of the project: technically experienced, they’re used to work extremely well together for several years with Agile methods,
  
• The required investments to clean up the project are huge, 
  
• The client, (rightly) focused on the roadmap of the next functional features, does not understand the poor team velocity, and every sprint trust less and less the team abilities to respond to demand,
  
• A vicious circle installs : the client requires to monitor and pilot the detailed team activities, with no understanding of the software development complexity... precipitating the death of the project.
  

I have attempted to draw a diagram of effects of this project, following roughly the notations of Jerry Weinberg (see next chapter for some theory).

Disclaimer: As many people before me have thought and written about this, it is very possible that other authors (or Jerry himself) already contributed similar diagrams.

Theory: Seeing the larger system through “Diagram of effects”

Some years ago, my friend and mentor suggested me to read a book from Peter M Senge, “The fifth discipline”, in order to learn “System Thinking”. Later on, I read further on the topic in Jerry Weinberg’s first volume of the “Quality Software Management” series. (I highly recommend these 2 books for anyone interested in explaining the dynamics behind a tough situation)

Don Gray provides an interesting first explanation for people unfamiliar with diagram of effects.

Jerry Weinberg also defines the following notations :

• arrows with squares stand for open choice for action from the management
• arrows with double pipes ( ----| |----> ) represent a time delay between a change and its consequence.

The concepts of delay and non-linearity may at first seem secondary when we look at or design a diagram of effects: balancing and re-inforcing systems indeed provides an interesting level of understanding, that may sometimes proves sufficient for acting wisely.

However, the combination of delays and non-linearities often explain the inability of some people to manage even simple systems :

• delays hide the effects of decisions : the consequences (positive or negative) of any action occur weeks (or months) later,
• non-linearities take people by surprise, by the dramatic magnitude of change (again, positive or negative) a minor change can produce

If your environment seems to run out of control, I highly recommend you take a sheet and a pen, and brainstorm for producing a diagram of effects, reflecting your system dynamics (read Senge and Weinberg excellent books for guidance). This analysis alone certainly won’t alone solve your problem, but it stands as a solid starting point before choosing your next actions.