Lean and Agile principles in software development

You are currently viewing Lean and Agile principles in software development

As we know, software development is about creating products and IT applications. The process for project delivery most commonly followed is the waterfall model Software Development Lifecycle (SDLC), in which requirements are defined all at once prior to project initiation. In waterfall, project delivery is about performing sets of operations for various stake holders. Historically with this model, the typical target project duration is around 9 months. In my experience however, I have never seen a single project executed successfully within this timframe. Invariably, there have to be compromises in the scope, cost, or quality of the project in order to ensure the delivery schedule doesn’t change. The most common reasons I have seen for these changes, which represent genuine failures, are:

  1. Scope changes
  2. Poor requirements understanding
  3. Incorrect estimations
  4. Tangled team dependencies
  5. Change in business process by the time project delivered.

In any project the standard set of operations during delivery are significant contributors towards delivery time. Thus, the focus for successful application of lean principles should be to eliminate waste by removing operational overheads and increasing process efficiency.  In order to identify forms of waste the team needs to come up with a meaningful value stream and subsequent process for identifying priorities. The biggest problem for software development organizations is to create this value stream for outcomes that are both digital and largely intangible. 

I have a theory on the history of Agile: The Agile development method has evolved as framework / process / principle by applying Lean principles to the standard (and inefficient ) waterfall development process.

Lean development can be summarized by seven principles which are very close in concept to lean manufacturing principles:

  1. Eliminate Waste
  2. Amplify Learning 
  3. Decide as late as possible 
  4. Deliver as fast as possible 
  5. Empower the team 
  6. Build integrity 
  7. Work Holistically

All these principles are seen being directly and indirectly implemented in Agile. As such, Agile principles basically “follow the waterfall process but in shorter cycles”.

There are multiple frameworks that are complaint with Agile principles, the most popular of which is the Scrum framework. Based on Scrum, sprints are time bounded ( recommendation is 2-4 weeks). Product Owners, Scrum masters, and Scrum teams work closely together to complete the sprint.

The Product Owner is the person who is responsible for consolidating requirements into a Product Backlog. The Scrum Master looks into the product backlog and decides on priorities, which are codified into a collection of stories that team will commit to implement, or burn down, within the sprint duration. Teams have daily standups that cover what they did the day before and what are they planning to perform that day to ensure that the sprint is on track. At the end of the sprint, during retrospection meetings, teams check what improvements can be added in future sprints. As such, the release plan is similar to the traditional project plan but without the same extensive efforts. It is a very high level plan with tentative dates which change based on the progress of the sprints. Because a release contains multiple sprints, as progress, setbacks, and information accrue, the release plan is updated and provides a real-time view of the project health. For teams the focus rests more heavily on sprint planning than release planning, as the release plan exists to provide high level visibility to management and product owners rather than direction to teams.

Now that I have introduced Lean Principles and Agile terminology, some interesting artifacts of the evolution contained in my hypothesis become clear:

  1. Eliminate Waste: The product manager works with team in breaking stories and coming up with business value. He also works with the engineering team to get the time cost for completing each story. Story priority is decided mostly on these two values. The Scrum team takes up high priority stories, which effectively eliminates waste.
  2. Amplify Learning: The agile retrospection meeting is a valuable resource to carry over learning to the next sprint plan.
  3. Decide as late as possible: Sprints with 2/4 weeks will give the necessary flexibility to make decisions later.
  4. Deliver as fast as possible: Once the story is moved from the product backlog to the sprint backlog, the turnaround time for delivery is short.
  5. Empower team: Since the Scrum team is selecting which stories will be pulled from the product backlog to the sprint backlog using story priorities, management is not making decisions on what should be part of the sprint backlog. This empowers the actual team to make decisions without those further from the work interfering. 
  6. Build integrity: Agile core values ensure team member integrity by making them truly necessary parts of the team, producing buy in and pride.
  7. Work Holistically: The release plan makes the Scrum team approach projects holistically because it provides a bird’s eye view of the value of each activity, while still maintaining focus on the sprint.

The success of the Agile implementation depends a lot on organization mindset, especially leadership maturity. Using enabling tools is also necessary ( eg: JIRA Agile + Confluence ) in order to record all necessary artifacts. Continuous Integration, Continuous Delivery (eg: Stash + Bamboo) and Test Automation complement the value Agile creates. Effectively Agile Scrum framework reduces the failures highlighted above to the maximum extent. However, there are still problems with an Agile Scrum framework:

  1. It is focused on Engineering Team : Though the product owner is from the Product Marketing team there is no established framework on how the product backlog should be groomed. Usually there are multiple product owners with requests that need to be included in the product backlog. For product log grooming, I have recommended Kanban process to many customers and they are very happy with the solution. This way the input from Service Engineering to product backlog is still an open area.
  2. Adaptability of Matrix / Larger Engineering Project Teams : Agile recommends 6-8 member to a team. In enterprises, the projects I have worked on usually number around 20 – 50 members at most global and matrix organizations. Some principles of Scrum work to some extent within Engineering, but when representing an engineering team in cross functional project meetings (Marketing, Engineering, Systems, Service Engineering and Operations, Sourcing, QA, RA, Program / Project Manager) agile and scrum process aren’t understood. In these situations adaptability is a key factor. You have to prepare yourself differently for different meetings.
  3. Bigger Platform Change : Sometimes an OS announces End Of Line or Strategic decisions that drive changes and product platform migrations. To the customer it doesn’t make any difference since they only care about application functionality. As per Agile, we need to focus on changes that make a difference to the customer. Can these projects be neglected?
  4. Design Focus : There are side effects of doing design in the sprint, especially for the products which need to be scalable in nature. Platform design is very critical in the longer run and in the rush of delivering shorter cycles this should not be compromised as it can prove very costly in the long term.
  5. Products incremental release to multiple instances: Continuous Delivery works really well for a single instance eg: Facebook (push model). Releasing to external teams after every sprint makes sense, but imagine updating a high volume install base. Some argue that companies like Microsoft release patches / service packs very frequently. However, an Agile framework doesn’t cover this. Instead this is left as part of continuous delivery scope. Upgrading systems which are under regulatory surveillance is different ball game.

Software development Enterprise needs are addressed by Scaled Agile Framework (http://scaledagileframework.com) and deeper look is needed. In the next blog entry, I will elaborate on this.  As lean emphasizes making improvements on existing processes to satisfy different challenges and refinements to existing or new frame works, it will keep evolving, and I’ll keep writing. 

Leave a Reply