Continuous Delivery: The Ultimate Challenge for Software Development Managers

By Kevin Aguanno, PMP, MAPM, IPMA-B, Cert.APM, CSM, CSP
and Mike Bowler, CSM

In a recent article titled “Agile Methods and The Need for Speed”, the author notes that many people are adopting agile development methods in hopes that they can deliver their project faster. The article goes on to state that, while speed is a possible outcome of agile methods, it is not guaranteed. Agile projects are more likely to achieve other benefits first, such as reduced technical risk, higher quality, greater likelihood of meeting true requirements, and more. Yet there are still those elusive speed-related benefits that people strive to achieve in their first agile projects.

To truly deliver software faster, one must look towards cutting down the timespan of all processes in the software development lifecycle from requirements gathering to deployment. Many who seek faster delivery use agile methods to improve the requirements gathering, design and development processes but are frustrated in their attempts to get a speedier deployment of the new software. These people often see deployment activities as unnecessarily cumbersome and often without much perceived value.

Perhaps the agile community has contributed to this perception. We have talked about concepts such as “continuous deployment” for years as if it were just one of the many agile techniques we can employ on our projects. Yet, this particular technique stands apart from many of the other basic agile techniques such as holding daily stand-up meetings, managing requirements using backlogs, and breaking a project down into iterations which culminate in a demonstration to stakeholders. Continuous deployment is among the most difficult of agile techniques to employ successfully and requires a very high level of agile maturity and discipline in the team.

Generally, to be successful at continuously deploying software into a production environment, a number of preconditions must be met:

• Automated Builds – The team must have tooling in place to automate their software builds. In addition, the team must follow consistent coding standards to help facilitate the automated builds. For continuous deployment, there is no time to waste manually integrating source code from various developers into the official build.
   
• Automated Regression Tests – Every time an automated build takes place, there should be software in place to execute a suite of automated regression tests to make sure that the existing deployed software is not impacted negatively by the new code. Some automated build tools can execute “smoke tests” or a partial suite of lightweight regression tests to verify basic functionality. For full regression testing passes, however, you will likely want to take advantage of a separate, dedicated automated testing solution.
   
• Automated Compliance Scans – In some environments, after the software development team determines that a high-quality package of software is ready for deployment to production, the software must still go through external compliance scans. One example of this is found in web applications that capture credit card information as part of a financial transaction. Credit card merchants are now required to have their applications go through a Payment Card Industry (PCI) compliance scan by an independent, external provider. If one of these external compliance scans is required for an application before it can go to production, then the initiation and execution of such scans also should be automated as much as possible. We know of one company with a PCI compliance scanning requirement who is integrating HP Fortify (security) checks into their ongoing builds so that the developers are notified as early as possible about possible security issues. It is not yet clear to the team whether a full HP Fortify scan can be done in less than three or four days so this can be a real bottleneck to rapid deployments. If more rapid deployments are required, there has to be a discussion with the external compliance verification agency of whether PCI compliance can be maintained with only a subset of a full scan.
   
• “One-Click” Deployment – While not achievable in many organizations due to regulatory or policy restrictions, in the ideal world, teams would have the ability to select a software build for deployment to production and with the click of a mouse promote the software. Executing typical governance functions such as user acceptance testing, independent systems integration testing, and scheduled deployment windows all slows down the release of software to production, often forcing days – or even weeks – into the timeline between code completion and live deployment. For truly continuous deployment, teams need to be able to bypass these quality assurance and management processes. Obviously, this requires very high-quality code to consistently come out of the development phase – something many organizations have difficulty achieving.
   
• Automated Database Migrations – Data schema changes may occur during an agile development project. Such changes complicate deploying code to production as database updates also need to be coordinated. In continuous deployment environments, these changes need to be automated. Automated database migration is a concept that became popular through Ruby On Rails several years ago and is now implemented in a wide range of languages such as SQL, Java, or C#. Migrations are typically written in pairs: one to modify the database forward and one to roll back the same change in case of mistakes. A typical project will have a database in several environments that all use the same schema (for example: Development, Staging, UAT, and Production) and automated migration ensures that they are all consistent and reproducible.
   
• Automated Software Alerts and Monitoring – If code and database changes are being deployed at the click of a button, one should monitor the solution closely once it has been deployed to a production environment. Code should be written with monitoring “hooks” for application monitoring tools such as IBM Tivoli Monitoring, HP Application Performance Management, or even Open Source tools such as the Nagios package. Software applications should generate alerts that these monitoring tools can access to identify errors at a granular level. In a perfect world, these monitoring tools would not be necessary as we should be generating perfect, high-quality code from our development phase; however, in the real world, these tools become a necessity. Because extensive (and time-consuming) independent testing has not occurred at the end of the development phase, teams attempting continuous deployment need a way to identify granular issues in real time and monitoring tools are an ideal solution to this problem.
   
• Flags to Enable or Disable Features – Finally, once we automatically deploy software with the click of a button, and our automated monitoring tools sense a problem, we need a way to at least disable the offending features and at most quickly revert code back to a previous version. Externalizing flags to enable or disable features is a simple programming technique that can allow monitoring tools (or administrators) to turn off troublesome areas quickly, keeping a production environment running while the problem is investigated and fixed in a development environment. This also allows us to deploy an incomplete feature into a production environment; with the flag for the incomplete feature turned off, no one will ever see the feature and the code will not execute. Having the ability to do this means that software can be promoted to production at any time, not just when features are complete. Similar to feature flags is the concept mentioned earlier of backing out database schema changes if they are found to cause problems. Development managers must take care in maintaining traceability between code changes and data schema changes so that related items can be turned off or rolled back in parallel to avoid causing further issues.

All of these prerequisites mean that continuous deployment is very difficult to achieve and to execute well. It is not just a fancy term for debugging code in the production environment – which, although unfortunately common, is clearly a very bad practice. Continuous deployment requires clearly-defined standards, very strict discipline among team members, a high level of skill, a maniacal focus on quality, and unwavering professionalism. In short, it is achievable for mature agile development teams but is certainly out of reach for those new to agile development practices.

Many companies have been able to achieve this gold standard of agile development:

1. Etsy, an online marketplace for handmade items, deploys code to production an average of 50 times per day. In order for such a large number of daily production deployments to succeed, their code quality is forced to be exceptionally high.
    
2. Facebook has institutionalized continuous code deployment with no explicit human governance checkpoints. They have incorporated extensive application monitoring, however, to identify when they have introduced an error. Facebook mandates that its developers create automatic rollbacks to address any errors. In some cases, the error-handling routines in the Facebook source code turn off offending features, creating a self-modifying code base.
    
3. Finally, most impressive of all, Google has an amazing continuous build system that runs 75 million automated regression tests per day across their codebase. Google has taken automation to the extreme to manage their 5,500 source code check-ins per day.

As we have shown, faster releases to production are possible. What business sponsors must realize, however, is that to continuously deploy code requires a number of development environment, application design, production monitoring, coding practice prerequisites. While those may seem simple to achieve, the agile process maturity, high level of team discipline, and extreme focus on quality may be harder to achieve. Organizations should start with obtaining some of the agile benefits that are easier to achieve and leave continuous deployment – faster delivery – for later when they have achieved increased maturity.

Kevin Aguanno is a principal consultant with GenXus Management Consulting, a specialist in project and programme management strategy consulting with deep expertise in agile delivery frameworks. Author of over 20 books, audiobooks, and DVDs on agile topics, Aguanno teaches agile methods at several universities and at conferences around the world. Find out more at www.GenXus.com/aguanno.

Mike Bowler is an experienced agile consultant and coach with a strong emphasis on both process and technical practices. He is known in the agile community for his strong technical skills such as Test Driven Development (TDD), refactoring, pair programming, and continuous delivery. He regularly coaches teams on the use of these skills using languages such as Java, Ruby and C#. Find out more at www.GenXus.com/bowler.