yet another continuous integration story

Yet Another Continuous Integration Story

Who I Am● Anton Serdyuk

● Technical Leader at Startup Labs, Inc.

● anton.serdyuk@gmail.com

● anton.serdyuk@itstartuplabs.com

● @anton_serdyuk

● Skype: serdyuk.anton

Disclaimer

Disclaimer

Projects are different● Technical requirements

● Business requirements

● Length

● Budget

● Load

● Data size

● etc

Disclaimer

Teams are different● Level

● Size

● Structure

● etc

Disclaimer

There is no silver bullet● Worked for us != will work for you

● Adopt patterns and principles rather than the full

process

Who We Are

Who We Are

Project● b2b system

● Unexpected downtime is very expensive

● Bugs in production are also very expensive because of

the reputation loss

● Small amount of traffic (But expensive)

● 1.5 years

● 4-15 developers (+ layout, + qa)

● 15 Gb Database (Grows not very fast)

Who We Are

Technologies● PHP 5.4

● Java 7

● PostgreSQL

● RabbitMQ

Who We Are

Frameworks● Hibernate

● JUnit

● Symfony2

● Silex

● Behat

● PHPUnit

Goals and Principles

Goals and Principles

Agile● Small changes

● Frequent releases

● Early feedback

What Is NOT Our Way

What Is NOT Our Way

Git Flow

What Is NOT Our Way

Feature Branches

master

feature2

feature1

What Is NOT Our Way

Feature BranchesActive feature branches usage example

What Is NOT Our Way

Release Branches

master

1.0.x

1.1.x

What Is NOT Our Way

master/staging/develop branches

master

develop

What Is NOT Our Way

Release Tags

master

v1.1.2v1.1.1

What IS Our Way

What IS Our Way

Continuous Integration● Push every day

● Build every commit

● Run all tests on every build

● Fix red builds quickly

What IS Our Way

One Branch

master

What IS Our Way

Branch by Abstraction● Commit switch-off for this feature

● Develop, integrate

● … your code in production now, but switched-off

● Develop, integrate

● Test

● Switch-on this feature in production

What IS Our Way

Autotests● QAs write behat scenarios

● Developers implement

● Every feature covered by behat scenarios

● Great regression tests suite - easy refactoring, less

bugs

● + ~100% development cost

What IS Our Way

Every Build Is Release Candidate● Do not push if state after commit is not suitable for

production

What IS Our Way

Some Numbers● ~1000 behat scenarios (unit tests are not counted)

● ~15 minutes build duration

● 2 build servers (Intel Xeon, 32 Gb RAM, SSD)

● 5-6 build agents for tests (VMs)

● 3 build agents for build phase (VMs)

● ~15 builds per day

● 2 releases per week

Build Process

Build Process

Overview

build automatic tests manual tests load and prod migrate tests production

Build Process

Overview

testagentstest

agentstest

agentsbuild agentgit

repository

push

bamboo

pull

build agents

build

aptrepository

artifacts

testagents

puppetmasterdevelopers

release engineer (architect/admin)

tune

test

QAserver

preprod prod

Build Process

Use The Same Artifacts

testagentstest

agentstest

agentsbuild agentbuild

agents

aptrepository

artifacts

testagents

QAserver

preprod prod

Build Process

Autoconfigure All Environments

testagentstest

agentstest

agentstest

agents

puppetmaster

release engineer (architect/admin)

tune

QAserver

preprod prod

Build Process

Deploy by Pressing a Button

testagentstest

agentstest

agents

bamboo

testagents

QAserver

preprod prod

Build Phase

Build Phase

Goals● Create artifacts which will be used in all phases

● Run unit tests

Build Phase

Technologies● Phing

● Maven

● deb-packages (dpkg-buildpackage)

● PHPUnit

● JUnit

Build Phase

Example

Automatic Test Phase

Automatic Test Phase

Goals● Run functional tests

Automatic Test Phase

Deploy to Completely Clean System

agent 1

Automatic Test Phase

Run Tests in Parallel

behat process 1 application 1

...

agent 2

behat process 2 application 2

agent 5

behat process 5 application 5

Automatic Test Phase

Run Tests in Parallelfeature 1

feature 2

feature 3

feature 4

feature 5

feature 6

feature 7

feature 8

feature 9

feature 50

feature 51

...

job 01

job 02

job 03

Job 15

agent 1

agent 2

agent 3...

Automatic Test Phase

Run Tests in Parallelgithub: StartupLabs/behat-partial-runner

● bin/behat --count 15 --current 1

● bin/behat --count 15 --current 2

● bin/behat --count 15 --current 3

● bin/behat --count 15 --current 4

● bin/behat --count 15 --current 5

● …

● bin/behat --count 15 --current 15

Automatic Test Phase

Write Fast Tests● Continuously speed-up your tests

● Use curl instead of Selenium

● Put initial data directly to DB

● etc

Automatic Test Phase

Write Fast TestsMerge scenarios which do not modify data

Feature: Posts pagination

Background:

Given I am logged in as admin

And there is many posts

Scenario: Pagination works

Given I am at the posts list

Then I should see pagination

Scenario: Link “last” works

Given I am at the posts list

When I click at the “last” link

Then I should see last page

Feature: Posts pagination

Background:

Given I am logged in as admin

And there is many posts

Scenario: All pagination tests

# Scenario: Pagination works

Given I am at the posts list

Then I should see pagination

# Scenario: Link “last” works

When I click at the “last” link

Then I should see last page

Automatic Test Phase

Write Fast TestsTranslate outline to single scenarios with TableNode

Feature: Post validation

Background:

Given I am logged in as admin

And I am on post add page

Scenario Outline: Validation works

When I enter <value> into <field>

Then I should see error <error>

Examples:

| field | value | error |

| title | | Can not be empty |

| tags | %%%% | Should be letters |

Feature: Post validation

Background:

Given I am logged in as admin

And I am on post add page

Scenario: Validation works

Then I should see such errors with such input

| field | value | error |

| title | | Can not be empty |

| tags | %%%% | Should be letters |

server

Automatic Test Phase

Write Fast Tests

@parallel

Feature: Login

Background:

Given I am registered user

| username | alex |

| password | 123 |

Scenario: I can login

Given I am at the login page

When I fill form with

| username | alex |

| password | 123 |

Then I should be logged in

Parallel test execution

● Many problems with race

conditions

● Not worked for us :(

behat process 1

behat process 2application

alex = alex1

alex = alex2

Manual Test Phase

Manual Test Phase

Goals● Run manual tests

● Place where QA see new functionality

Manual Test Phase● Deploy to completely clean system

Load and Prod Migrate Tests

Load and Prod Migrate Tests

Load Tests Goals● Catch race conditions

● Catch serious performance degradation when database

grows or after adding new functionality

Load and Prod Migrate Tests

Prod Migrate Tests Goals● Test migrations on real data

● Test migrations time on real data

Load and Prod Migrate Tests● Migrate load test env to new version

● Rollback migrate test env to production snapshot

Deploy to Production

Deploy to Production● nb-prod - small production copy used by other teams for

integration purposes

Problems

Problems

Red Builds for a Long Time● QA can not deploy new version to staging

● New commits make new tests red like a snowball

● Nobody knows who are responsible for all those failed

tests

Solutions:

● If build can not be fixed in 15 minutes - revert commit

● Autoreverts on red builds (?)

Problems

Long Builds● Nobody wants to wait green build

● Nobody cares about green build after commit

● Red builds for a long time

Solutions:

● Speedup tests

● Parallel build

● Add more build servers

Problems

Blinking Failures● Failures are trustless - “it is not my fault - it is blinking,

just do rebuild”

● Time consuming

Solutions:

● Eliminate them ASAP

● Do not just run “Rebuild failed jobs”

Key Points, Again

Key Points, Again

Autotests● There is no sense in CI without tests

● Easy refactoring

● Less bugs

Key Points, Again

Fast Builds

● Fast tests

● Parallel run

● Hacks in tests

● etc

● Long builds => More changes => Red builds

Key Points, Again

One branch● There are no builds from other branches

● There are no hotfixes

● Easy deploy and build pipeline

Todo

Todo● Rollback strategy

● Use Bamboo deployment

Inspired By● Continuous Delivery: Reliable Software Releases

through Build, Test, and Deployment Automation (Jez

Humble, David Farley)

● http://timothyfitz.com/

Questions?