Vagrant for Effective DevOps Culture

Vaidik Kapoor @vaidikkapoor

Me?

• I work at Wingify

• Software Engineer for first few months

• Infrastructure and Operations after that

• github.com/vaidik

• twitter.com/vaidikkapoor

Wingify EngineeringBack when I joined:

• ~12 people

• Two teams: backend and frontend

• Multiple services, but only a few with active development

• Random bash scripts everywhere to setup things

Wingify EngineeringToday:

• ~25 people

• Multiple teams: backend, frontend, JS library, mobile, operations

• Lots of overlap between teams

• Multiple services ~ very SOAish

Problems• Things work fine locally but not when deployed

• Things work fine in testing / staging environment but not in production

• All the services are not available locally while development is on going

• Too many services

• Too difficult to setup because everyone has a different dev environment

• Testing end-to-end was not possible for developers

• Releases take a lot of time

Similar Environments

Similar Environments

• Closer to production

• Staging should be as similar as possible

• Use similar procedures for setting up every environment

Configuration Management• Formalize your infrastructure

• Use same modules / recipes / playbooks / whatever everywhere

• Many out there:

• Puppet!

• Chef

• CFEngine

• Ansible

• Salt Stack

Problems Solved

• Issues that were caught in production started getting caught early on

• End-to-end testing was made possible

• More confident releases

• Faster releases

New Problems?

New Problems• Only Ops people writing / maintaining Puppet

code base i.e. only two guys

• Dependency and bottleneck

• Ops and devs working in isolation - details shared at a very high level

• Slow feedback, slow releases and slow resolutions

• Not scalable

Changes Were Required

Devs Ops OpsDevs

Shared ResponsibilityIsolation

How to get there?

For Developers

• Give developers environments for development which closely resemble production infrastructure.

• Same OS as used in production

• Possible to re-use configuration management for dev environments

Virtual Machines• As dev environments

• Distributable

• Re-usable

• Cheap

• Develop -> Test -> Deploy -> Destroy… Repeat!

• We did this for a while!

Drawbacks with VMs

• Lack of flexibility to choose your own tools, IDEs, etc.

• Was still too much manual work for setting these up - lack of tooling

• Intimidating at first

vagrant

vagrant• A super cool wrapper around:

• Virtual Machine providers (VirtualBox, VMWare, etc.)

• Docker

• Support for provisioning using configuration management tools

• Bunch of awesome tooling to make life easy

• synced folders

vagrant upto start a VM and provision it if not already done

vagrant sshto ssh into the VM

vagrant provisionto run your configuration management scripts again

vagrant destroyto throw away your VM

vagrant packageto package an existing VM and distribute

For Developers• Give developers environments for development which

closely resemble production infrastructure.

• Same OS as used in production

• Possible to re-use configuration management for dev environments

• synced folders make it possible for you to use the tools you love

• Vagrant’s tooling makes management of dev environment really simple

For Ops?

For Ops

• Use vagrant for testing out configuration management setup

• From small modules to complete scripts

• Faster feedback loop between developers and operations

Devs and Ops Together• Ops get control of tooling for infrastructure and

managing configuration

• Ops’ work gets tested out first in dev environments

• Developers get a closer and more up-to-date replica of production systems

• End-to-end testing is possible because all or most important services are setup using configuration management

Devs and Ops Together• Tighter feedback loop between devs and ops

• Reasons of failures can be pin pointed faster

• Resolutions can be made faster

• Helps with better testing of applications - what works locally has higher chances of working everywhere

• Helps with better testing of configuration management - strives for a more modular formalization of infrastructure

More Stability

New Possibilities?

• For developers to actually know how things work in production

• Slowly have developers own parts of configuration management

• Start giving developers a share of responsibility to handle infrastructure

Did I mention Multi-Machine?

Vagrant Multi-Machine• Capable of setting up multiple machines in a cluster

and controlling them.

!

$ vagrant up webserver

$ vagrant ssh dbserver

!

Vagrant Multi-Machine• Have all your services running across multiple VMs

• Example: web application and memcached on one VM, postgres and redis on another, elasticsearch on another.

• Test REAL ISSUES with infrastructure on your machine

• Simulate network issues in distributed systems

• Jepsen?

• Your playground!

VagrantfileRuby DSL - very flexible and programmable

It’s a long journey• Get Vagrant in place to give common playgrounds to both

developers and operations

• Have infrastructure formalized in some way - choose configuration management of your choice and have everyone use it

• Take small steps

• Do it for small projects and show success stories

• Replicate across projects

• Improve along the way

• Try to get developers also contribute

Thank youQuestions?