applied lean startup ideas: continuous deployment at kaching

Continuous Deployment at kaChingApplied Lean Startup Ideas:

#leanstartup

&

Pascal-Louis Perez, kaChing Group Inc.

Twenty Deployments a Day Keeps the Nasty Bugs Away

10-May

11-May

12-May

13-May

14-May

15-May

16-May

17-May

18-May

19-May

20-May

21-May

0

5

10

15

20

25

30


“Connect investors with outstandinginvestment managers”


Release is a Marketing Concern

Reducing Code Inventory is what Engineering Focuses On


Lean Startup at kaChing

• Methodical approach to eliminating waste• No Quality Assurance engineers, no Operations

team• Empower everybody to drive their own projects

“Human institution designed to create something new under conditions of extreme uncertainty” – Eric Ries


What is Continuous Deployment?

• Continuous, successful and repeatable methodology to deploying code

• Automates every steps of taking checked in code and making it run on production servers, in front of customers


True Story

• Investment managers calls, comments about unintuitive trading flow.

• Improvements are made, and deployed within the next 20 minutes.

• We call him back “What do you think of the improvement?”


Benefits of Continuous Deployment

• It allows quick iterations• Obsoletes processes, e.g. “cutting a release”• Reduces risk• Everyone is aware of production• No one throws code over the wall• Exposes 24x7 operational requirements• Trunk stable


ContinuousDeployment

Immune System

Making Continuous Deployment a Reality

Test Driven Development

Culture


#architecture


Our architecture

• Service oriented system• Vertical sharding• Everything uses the same platform kawala• Coordination using ZooKeeper• Data interchange using JSON and Protobufs


Typical Stack

• Clustered services, multiple instances• Replicated databases (e.g. MySQL or NoSQL)• Caching (e.g. memcached)• Denormalized data (e.g. Voldemort)


kawala

• Command pattern. A command is a query• Dynamic Inversion of Control• Queries produce a value


Sample Query

class GetUser(Id<User> userId) { @Inject UserRepository repository; public User process() { return repository.load(userId); }}


What We do with Queries

RPC Persist Queries

Push into MQ

Command Line Tools

CPS Render Pages

Serve BLOBS

Lifecycle

• 1. Commit• 2. Testing• 3. Deployment• 4. Production Validation• 5. Lifetime Monitoring

Development

• Trunk stable• Small, frequent commits• Unreleased features live behind experiments• Forward & Backward compatibility• Responsive code review• Pair difficult problems• Trivial rollbacks

#testing

Testing Philosophy

• Only automated testing matters• If it isn’t tested, it isn’t finished or correct• Write testable code• Embrace abstractions• Testing is cross functional, we all own quality


Benefits of TDD

• It allows quick iterations• It empowers engineers to change anything,

and as such helps in scaling the team• It is more cost effective than debugging• It obsoletes the need for functional QA• It facilitates continuous refactoring, allowing

the code to get better with age• It attracts the right kind of engineers

Types of Testing

• Unit Testing - does the code work?• Integration Testing - does the code work

together?• Regression Testing - learn from your

mistakes• Frontend Testing - a whole different

ballgame

Defensive Testing

• Nightmare scenarios• A common conversation at lunch:– Alice: What would happen if X blew up?– Bob: uh... the site would go down.

• Fix it, test for it

Implementation

• Frameworks: junit, dbunit• CI: Hudson• Frontend: Selenium

Production

• Process, Infrastructure, Deployment, Monitoring

Process

• Everyone: writes code and tests, releases to production, adds monitoring– Specialists help out when needed

• New hires push code the first day• Problems, issues, errors, bugs, oversights...

Stability

• Cluster everything• As little global state as possible• Maintain global state through ZooKeeper• Monitor everything

ZooKeeper

• Reliable Distributed Synchronization• For:– Service Discovery– Service Status– Coarse locking


Life of a Deployment

DM

ZK

PMPM PM



DM

ZK

PMPM



DM

ZK

PMPMPM

Deployment

• Exponentially increasing group sizes• Increased monitoring during deployment• Self Test• Automated rollbacks

Implementation

• Building: ant• Deploying: rpm, yum• Monitoring: nagios, jcollectd, RRDtool, jmx• Custom automation

#monitoring

Monitoring Philosophy

• Prefer business metrics• Monitor statistical deviations not absolute

values• Automatically annotate graphs

Monitoring Errors

• False negatives - errors of omission• False positives - errors of implementation

End to End Monitoring

• Run Selenium on production• Accessibility, Speed• Ad-hoc Keynote - customized for our flows• Must control data creation, analytics impact


#future-developments


Quarantining

• Isolate new releases• Flexible partitioning of requests• Gradually shift load to fresh services


Describing Infrastructure

• Many moving parts– nagios, collectd, backups, services, databases, …

• Consistency is key• Adding new tools should be easy and

thorough• Standardize best practices


What we Covered Today

• Lean Startup and Continuous Deployment• Anatomy of a Release– Commit, Testing, Deployment, Monitoring

• Future Developments– Quarantining, DSL for Infrastructure


References• We’re Recruiting [email protected]• kaChing’s blog http://eng.kaching.com• kawala http://bit.ly/kawala• Deployment Infrastructure

http://eng.kaching.com/2010/05/deployment-infrastructure-for.html

• Extreme Testing http://bit.ly/9bOFaA• Writing Testable Code

http://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-decided-to.html

• ZooKeeper http://bit.ly/kc-zookeeper• Eric Ries http://startuplessonslearned.com

applied lean startup ideas: continuous deployment at kaching

Technology

kaching group

deploymentpascallouis

realitypascallouis perez

valuepascallouis perez

protobufspascallouis

voldemortpascallouis

customerspascallouis

projectspascallouis