§Continuous delivery

1. Antipatterns

2. Proposed model

3. Common delivery issues

Agenda

§Antipatterns

Deploying software manually• production of extensive, detailed documentation that describes the

steps to be taken and the ways in which the steps may go wrong

• reliance on manual testing to confirm that the application is running

correctly

• frequent calls to the development team to explain why a deployment is

going wrong on a release day

• frequent corrections to the release process during the course of a

release

• environments in a cluster that differ in their configuration

• releases that take more than a few minutes to perform

• unpredictable releases that often have to be rolled back or run into

unforeseen problems

Deploying to a production-like environment only after development complete

• testers, if involved by this stage, have tested the system on

development machines

• releasing into staging is the first time that operations people interact

with the new release

• dev team assembles the correct installers, configuration files, db

migrations and deployment documentation to pass to the deployment

team without having tested them in an environment that looks like

production or staging

• little/no collaboration between the dev and deployment teams

Manual configuration management of production environments• having deployed successfully many times to staging, the deployment

into production fails

• different members of a cluster behave differently

• operations team take a long time to prepare an environment for a

release

• cannot step back to an earlier configuration of your system

• servers in clusters have, unintentionally, different versions of OS, 3rd

party infrastructure, libraries or patch levels

§Proposed model

Software delivery principles

Create a repeatable, reliable release process

Automate almost everything

Keep everything in version control

If it hurts, do it more frequently and bring the pain forward

Build quality in

Done means released

Everybody is responsible for delivery process

Continuous improvement

The deployment pipeline

- deployment pipeline = an automated implementation of your application’s build, deploy, test, and release process

- makes every part of the process of building, deploying, testing, and releasing software visible to everybody involved

- improves feedback so that problems are identified and resolved as early as possible- enables teams to deploy and release any version of their software to any environment

at will through a fully automated process

Blue-Green Deployments

- blue and green = 2 identical versions of production environment- deploy new version to the blue environment- when ready, change the router to point to blue – blue becomes production- if something goes wrong, switch the router back to green- careful with the db!- if budget allows, blue and green can be completely separate replicas- if not, use 2 copies of your application running side by side on the same environment

as blue and green

Canary Releasing

§Common delivery

issues

Infrequent/buggy deployments - symptomsProblem - long time to deploy the build, brittle deployment process

Symptoms

• long time for bugs to be closed by testers

• long time for stories to be tested or signed off by the customer

• testers are finding bugs that developers fixed a long time ago

• nobody trusts the UAT, performance, or CI environments, and

people are skeptical as to when a release will be available

• showcases rarely happen

• the application can rarely be demonstrated to be working

• the team’s velocity (rate of progress) is slower than expected

Infrequent/buggy deployments - causes• deployment process not automated

• not enough hardware available

• hardware and OS configuration not managed correctly

• deployment process depends on systems outside the team’s control

• not enough people understand the build and deployment process

• testers, developers, analysts, and operations personnel not collaborating

sufficiently during development

• developers not disciplined about keeping the application working by

making small, incremental changes, and so frequently break existing

functionality

Poor application quality - symptomsProblem - Delivery teams failing to implement an effective testing

strategy

Symptoms

• regression bugs keep popping up

• the defect number keeps increasing even when your team spends

most of its time fixing them

• customers complain of a poor-quality product

• developers groan and look horrified whenever a new feature request

arrives

• developers complain about the maintainability of the code

• an ever-increasing amount of time to implement new functionality, the

team starts falling behind

Poor application quality - causes• testers do not collaborate with developers during feature development

• stories or features “DONE” without comprehensive automated tests,

being signed off by testers, or being showcased to users from a production-

like environment

• defects routinely entered into a backlog without being fixed on the spot

with an automated test to detect regression problems

• developers or testers without enough experience developing automated

test suites

• team doesn’t understand the most effective types of tests to write for the

technology or platform

• developers working without enough test coverage

Poorly managed CI - symptomsProblem - build process not properly managed

Symptoms

• developers don’t check in often enough (at least once a day)

• commit stage permanently broken

• high number of defects

• long integration phase before each release

Poorly managed CI - causes• automated tests take too long to run

• commit stage takes too long to run (ideally under 5 minutes, more than

10 minutes unacceptable)

• automated tests fail intermittently, giving false positives

• nobody empowered to revert check-ins

• not enough people understand and can change the CI process

Poor configuration management - symptomsProblem - environments can’t be commissioned, and applications

installed reliably, using an automated process

Symptoms

• mysterious failures in production environments

• new deployments are tense, scary events

• large teams dedicated to environment configuration and

management

• deployments to production often have to be rolled back or patched

• unacceptable downtime of production environment

Poor configuration management - causes• UAT and production environments are different

• poor or badly enforced change management process for making

changes to production and staging environments

• insufficient collaboration between operations, data management and

delivery teams

• ineffective monitoring of production and staging environments to detect

incidents

• insufficient instrumentation and logging built into applications

• insufficient testing of the non-functional requirements of applications

§Thanks!