retrofitting continuous delivery
TRANSCRIPT
Retrofitting Continuous
DeliveryA DevOps Tale
Hi. I’m Alan.
betterment :
investing::
devops : engineering
Phase 1: Snowflakes
Betterment Core Applications
“brochure”
wordpressapache
phpmysql
batch(trading)
cronjava
springhibernate
mysql
webapp
apachetomcat
javaspring
hibernatemysql
Rackspace Loadbalancer
pre-prodapachetomcatmysqlhudsonjiraconfluence“warehouse”sendmailfile lockerbackup
Look at the Mess You’ve Made
● Overburdened, inflexible infrastructure from Rackspace● Only one pre-prod environment
○ And it runs five other things including Jira/Confluence/etc.● Building and deploying artifacts manually, ad-hoc, as needed
○ “Why is your trunk .war 23 bytes larger than mine?”● No Automated Functional Testing ● Deployments require downtime.● Deployments happen at 3am ET● Manual Provisioning on persistent servers
○ “This wordpress plugin needs php5.3 but we have 5.1” ● Branching (and merging) with Subversion requires significant
effort● …
yuck. so what’s the good
news?
we shipped a ton of functionality.
delivery speed faster than
innovation speed
Phase 2: Addicted to Automation
stop building features.fix what’s painful.
DevOps
fix onboarding.
boxen.
github // @jbarnette + @wfarr
mac osx+ homebrew
+ puppet+ facter
github.com/boxen/puppet-*
fix tooling.
>
$ time ant clean test deployment19m 22s
$ time gradle clean test intTest war zipJar2m 36s
(~builds/day · Δt · devs · man_days/mo)/min =
(6 · 17 · 19 · 20) / 60 = 646 hrs/mo
fix notifications & logging.
Whoa.
fix parallel workflows.
everybody loves jenkins
build open pull requests* build develop/stage/mastercobertura reports coverage
* https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin
fix crowded infrastructure.
AWS + Rackspace(sitting in a tree, N-E-T-W-O-R-K-I-N-G)
<10ms !
“canary” ec2 webapps exposed via route53
weighting
fix provisioning.
Ansible (not manual) AMIs
● Playbooks are YML that execute over SSH● Playbooks define necessary machine config
o Results in AMI creation in AWSo Returns AMI_ID
● Commit playbooks and current AMI_ID● Jenkins reads AMI_ID from build artifact
o Spins-up necessary EC2 instanceso Runs deploy playbook, unrelated to provisioning playbook
● If you need to modify a playbooko Run playbook to create new AMI_IDo Commit new playbook and AMI_ID to repo.
no more snowflakes.
everybody loves jenkins again
deploys to rackspacedeploys to ec2
bash or ansible playbooks
fix database migrations.
Flyway Database Migrations
● Looks a lot like rake db:migrateo core/src/main/resources/db/migrationso core/src/main/resources/db/cleanups
● Kept track of in a schema_versions table.
● Cleanups are promoted to migrations when develop moves to stage.
● Run as part of weekly releases.● http://flywaydb.org/
Database Subsetting with Jailer
● Walks the constraint graph from the mysql schema.
● Requires some manual intervention to cut off circular or “eager” relations.
● Exports a referentially valid subset of data.o User, Account(s), Trade(s), Etc.
● http://jailer.sourceforge.net/
Users. Duh.
+ = freshdb
subsets live in s3. download and run flyway.instant db on any branch.
eos:better-core(develop)$ ./fresh -fForcing download...Downloading fresh data... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 21.0M 100 21.0M 0 0 1004k 0 0:00:21 0:00:21 --:--:-- 1365k:setupFreshDb:setupFreshDb:complete
__ _ / _| |_ ___ ____ _ _ _| |_| | | | \ \ /\ / / _` | | | || _| | |_| |\ V V / (_| | |_| ||_| |_|\__, | \_/\_/ \__,_|\__, | |___/ by fido |___/
[stderr] INFO AbstractCLI.doMain(287) | cli_started[stderr] INFO FlywayDbMigrationCLI.executeBeforeSpring(164) | jdbc:mysql://127.0.0.1:3306/bettermentdbfresh [stderr] INFO FlywayWrapper.migrateTo(154) | Running ALL pending migrations ...[stderr] INFO ApacheCommonsLog.info(43) | Validated 146 migrations (execution time 00:00.616s)[stderr] INFO ApacheCommonsLog.info(43) | Current version of schema `bettermentdbfresh`: 20140617123653[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132438[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132439[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132440[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132441[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140625091709...
ah. that’s better.now what?
Phase 3: Innovation That Scales
innovating is easy.demoing is easy.
product teams move faster than
infrastructure teams.
jruby on railsfactory girl for Javamultiple schemas
webservices
delivery speed slower than
innovation speed
DevOps
DevOps
Solution?Undercover DevOps
every team builds product features with
delivery in mind
Questions?
first one: can you start at phase 3?
[email protected]/twitter: @nonrational