continuous deployment-at-flipkart

Continuous deployment at flipkart

Pankaj Kaushal

Photo credit: http://www.flickr.com/photos/bomb_tea/

Old school deployment

Check out svn

Point docroot to new svn

Place copies on all servers

One server at a time

Check if it works

Fear of the tar

Site breaks unexpectedly

many downtimes everyday

most related to deployments

Deployments

Module X is no longer working.

Foostat widget is now displaying php instead of rendering html.

All images are gone

Blah blah tab shows sometimes and disappears sometimes

I get blank screens

I'm sure I deployed on all four servers

Forgot to restart foo backend service

If anything can go wrong, it will.

Discourage deployments in the interest of stability?

or

Allow deployments to happen as fast as they need to?

Flipkart releases

Deployment should be a non-event

Flipkart deployment procedure

Step1: svn copy -m deploying to prod trunk/ prod/2011-07-08-10:00/

Step2: there is no Step 2

How?

You don't need magic

1. Continuous integration

1. Continuous integration

integrate frequently

against a controlled source code repository

Multiple integrations per day

2. Multiple Environments

2. Multiple Environments

Integration environments are replicas of production

All components of the system are available

Duplicate the production environment as much as you can

3. One step build

4. One step build and deploy

Continuous integration

Is not a tool, neither is it the result of a tool

It's a philosophy

You don't need lots of money or fancy tools

We use a bunch of shell scripts to do this

Use existing Free softwareDebian

Subversion

SSH

Puppet


Integration Environments

Packaging

Configuration

Automated Builds and Deployments

Building with Continuous Integration

Step1: check out trunk/mainline

Step 2: build feature/ fix bug

Step 3: Commit to mainline/trunk

Step 4: Commit to eng/Integration environment builds automatically



Packaging

Configuration



We have two integration environmentsENG (Engineering)

SB (Sandbox)

Each with it's own svn branch

ENG is on virtual machines

Each environment acts as a monitor to the repository

Every time there is a commit against the repository Build server automatically checks out the sources

initiates a build

notifies the committer of the result of the build



Packaging

Configuration


@flipkart apt-get install whatever

We leverage the debian package mangaer

All software is installed using apt-get


Automated builds make .deb

.deb pushed to a repository (flipkart repo)


Software dependencies

Version information

Allows rollbacks and prevents software mismatches

Apt Repository

Apt Repository



Package management

Configuration


Where is the booboo file?

Many things together build a productEverything should be in the repositorySource Code, Configuration, Database schemas

Can't find booboo

Put everything in the repository not just code

Everything you need to do a build should be in theretest scripts

properties files

database schema

install scripts

third party libraries

Static ConfigurationsPackages

Dynamic configurationsDatabase

System ConfigurationsPuppet



Packaging

Configuration


Automated Builds

Turning source code to a running system is complicated Typing strange commands is a waste of time

Automated deployments

Multiple environmentsOne to run commit tests (eng)

One or more to run secondary tests (sandbox)

Production

Scripts that allow you to deploy the application into any environment easily

Deployment procedure is the same everywhere

@flipkart fk-ops-build

One Package thatHas a config to describe environments

Contains Build scripts for all packages for all env

Monitors the svn tree for changes

Build new packages automatically (or manually)

Deploys new packages automatically (or manually)

Continuous Deployment

Changes trickle down environments automatically

Builds succeed in all test environments

Elaborate test cases

Automatically hit production

experiments

Some teams use automated deployments till Sandbox

Some teams use Teamcity to do integrated builds on every commit

We can do continuous deployments till prod but we do not practice it in every project

Way forward

Much tighter integration with test cases

Environments as RingsENG to do unit tests

Sandbox to do smoke tests

Functional tests before push to production

Resources

Wikipedia

martinfowler.com

moo

@spo0nman

Spo0nman on freenode #linux-india #perl #debian

1Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level

| Internal use only

continuous deployment-at-flipkart

Technology