Rise of the MachinesContinuous Delivery at SEEK

CO PRESENTED BY:

CD @ SEEK“To the cloud”

Scorched Earth Deployments

● Create a pristine environment for every release.● No need to migrate state of environment.

○ ∴ Configuration Management tools not required.

● State should not be changed at runtime.○ ∴ Immutable Server

● Blue/Green deployment strategy:○ Monitor health during deployment.○ Auto-rollback if new environment is unhealthy.

Automate all the things!

● Delivery team is responsible for the automation of:○ Server installation○ Infrastructure creation○ Deployment

● Automation is impossible without team managed infrastructure.

Infrastructure as Code● SEEK use AWS CloudFormation to automate their infrastructure.● Operations are responsible for the network-level infrastructure,

including:○ VPC○ Routing○ Users and Roles○ Shared services

● Delivery team is responsible for infrastructure within the VPC, such as:○ Elastic Load Balancer○ Security Group○ Auto Scaling Group○ Launch Configuration○ Instance Profile

Server automation

● Provisioning Linux:○ SSH ⇒ Easy!

● Provisioning Windows:○ WinRM ⇒ Hard!○ AWS makes bootstrapping easier:

■ EC2 User Data■ cfn-init

○ Packer Windows Plugins can be used to create Windows Amazon Machine Images

● Combination of Packer and CloudFormation can be used regardless of target platform

Packaging● Applications should be self-installable.● Application installation should be separate from server automation.● Installation should be independent of Continuous Deployment tool.● Native package manager is ideal solution:

○ Linux: apt-get or yum○ Windows:

■ One-Get (available in next version of Windows)■ Chocolatey (available now!)

● Package manager is platform independent pattern○ <package-manager> install <package-name>

● Windows Coding Horror:○ Interacting with Windows resources is not trivial, custom

PowerShell modules or PowerShell DSC Resources make this easier.

Pre-Baked Images

● Install a package on a base image.○ But… don’t include configuration.

● Then, create an image (AMI).● AMI is our environment independent deployable

artifact.

● Future is containerised...○ Docker is the next evolution of our approach.

Late-binding of context

● Keep your artifact environment agnostic:○ Frameworks want you to apply configuration at build or

package-time.○ Deploy tools want to apply configuration at install-time.○ When deploying to the cloud, you should apply configuration at

launch-time.● When AMI is launched it doesn’t know where it is:

○ Need to inject environment-specific context at launch-time.○ How do you do this?

Environment VariablesA platform independent way of specifying environment-specific config that’s been around since 1979● Use environment variables for:

○ DB connection parameters.○ Location of service endpoints.

● Don’t use them for:○ Altering runtime behaviour.

● Overriding .NET configuration with environment variables:○ .NET 4.5.x: write your own Configuration Provider.○ .NET 4.6: works out of the box.○ Azure + .NET 4.5.x: works out of the box.

● Service discovery is on the horizon:○ SEEK is investigating.○ Externalised config is first step.

Confidence to Release

● Put enough steps and checkpoints in pipeline to get enough confidence to release.

● Too many checks will result in a slow pipeline and kill the benefits of Continuous Delivery.

● Take things away until you cry.○ Then... compensate some other way○ For example, consumer-driven contract testing makes sure we

honour API contracts without the need to integrate.

Ship It!

● Distinction between Deployment and release:○ Deploy: create Blue environment and deploy application.○ Release: cut-over from Green to Blue environment.

● Smoke test in Blue environment before release using synthetic transaction.

● Same smoke test can be used to monitor health after release.● Choose to enable manual release if change set is risky and additional

manual testing is required.

Towards Polyglot Architecture

● Continuous Delivery + Microservices gives teams the ability to choose the right stack for the job.

● For SEEK, CD is opening up the possibility migrating legacy apps from Windows to Linux.○ Who knows, in the future, running Windows containers will be a

viable alternative.● SEEKs approach to Continuous Delivery provides better logical

separation automation steps.● Time to first commit still a problem and this will only get worse with

the proliferation of different stacks and tools.

Machine Factories

Manual Development Environments

■ Are a bad experience for your developers.■ Are time consuming and therefore costly.■ Make it difficult for devs to move between projects.■ Are rarely accurately reconstructed.■ Encourages manual configuration rather than the more

reliable and repeatable practice of Infrastructure-as-code.

Can lead to environments that have all of the parts, but not quite in the right order.

“It works on my box”

■ Sign of environmental divergence.■ The Butterfly effect: Minor differences in environment

configuration can produce obscure bugs in Prod.■ This issue is compounded when deployments carried out by

another person (e.g. Ops).

Dev vs Production Parity

■ Environment Parity = Confidence.■ Overcompensate lack of parity with more, possibly manual tests.■ Environment Parity…

○ enables continuous delivery .○ requires diligence and automation for accurate replication.○ encourages infrastructure-as-code.

Wouldn’t it be great if we...

■ Could use declarative formats for setup automation, to minimize time and cost for new developers joining the project.

■ Had a clean contract with the underlying operating system, offering maximum portability between execution environments.

Wouldn’t it be great if we...

■ Built apps suitable for deployment on modern cloud platforms, obviating the need for servers and systems admin.

■ Could minimize divergence between development and production, enabling continuous deployment.

■ Could scale up without significant changes to tooling, architecture, or development practices.

The “12 Factor App”

■ These 5 principles are what loosely describe the 12 factor App, a guideline for modern SaaS products.

■ 12 Factor Apps, Factor #10: Dev/prod parity

■ “Keep development, staging, and production as similar as possible”.

■ Vagrant can help us solve the environment parity issue.

Vagrant, Docker Compose & FriendsOnward to immutable development environments

Scorched Earth Dev Environments

■ Dev automation typically come in 2 flavours:○ Use of CM Tools like Puppet, Chef (Boxen, Babushka)○ Virtual Machine Runners (Vagrant, Docker Compose)

■ CM Tools fraught with challenges:○ Not isolated○ Hard to automate across OS/Platforms○ Doesn’t force developers to automate

What is Vagrant?

“A tool to help create and configure lightweight, reproducible, and portable development environments.”

■ A replacement for manually configured boxes. ○ Enforces infrastructure-as-code.

■ A machine runner (typically a VM or a Container)■ Separates authoring environment from development runtime■ Isolated & repeatable

What is Vagrant?

The following is a logical view of how Vagrant interacts with your authoring environment:

■ Vagrant has spun up Virtualbox VM with the name ‘Web’, and is running it on a Mac OSX desktop

■ Code is synchronised from /c/proj/x -> /vagrant of the running environment

What is Vagrant?

■ The Web application spins up a server using the code in /vagrant, which binds to a private network on the VM on port 80

■ Port 8080 is exposed to the outside ‘world’ (in this case, a Mac OSX desktop) which forwards requests to port 80 in the running VM

■ The Developer is free to author in whichever IDE of her choosing

The TAO of Vagrant

■ vagrant up to get a fully working dev runtime.■ Development occurs in preferred IDE: Visual Studio,

Sublime, ViM, with your terminal, your shortcuts - etc.■ vagrant destroy && vagrant up will destroy the

machine and create a new, clean, isolated environment.■ Workflow repeatable and transferrable to other projects.

Developing on EcosystemsFrom monoliths to ecosystems

From Monolith to Microservices

■ Applications are usually more interesting than a single UI, they are usually connected to things.○ They form part of a functioning ecosystem.

■ Vagrant good for Monoliths and ecosystems.■ Explosion of microservices highlights this need.

From Monolith to Microservices

Commentary on Architecture:

■ Multiple components, in disparate technologies■ Distributed system■ Layered architecture■ = complexity

Increased complexity increases the number of things that can go wrong. We can combat this with parity and local simulation.

From Monolith to Microservices

■ Vagrant can replicate ecosystems locally, while still having the benefits of isolation, repeatability, immutability etc.

■ Running ecosystems locally gives us flexibility.■ Vagrant can simulate or test a number of scenarios, for

example to:○ Model distributed systems behaviour○ Perform disaster testing; outages, network partitions ...○ Replace components independently○ Test OS/DB upgrades○ Run entire production replica ecosystems, locally

Local Ecosystem Setup

The following is a Logical View of this fictional ecosystem with Web and API applications exposed to the ‘World’ and a database hidden ‘behind the firewall’:

Local Ecosystem Setup

■ Note that we are running multiple virtualisation technologies here as if they were all running locally

■ Vagrant will spin up 3 local VMs: ○ Web and API applications exposed to the ‘World’ and a

database box hidden ‘behind the firewall’:■ The 4 VMs are able to communicate through a private

network on their native ports (80, 80 and 27017 respectively).

Native to Docker and is fast, but some caveats:

■ Unless on linux, networking is not automatically exposed to the host machine.

■ You need to ensure that your apps are compiled/built within the Docker containers - otherwise you’ll still need to bootstrap your dev box anyway!

■ Hybrid VM/Docker setups not supported.

Docker Compose

Machine Image Pipelines with PackerWhere environments are born

Cattle in Production, Pets elsewhere

■ Dev == Prod - but what about our build/other servers?■ We don't want cattle in production and pets elsewhere.■ Quality starts before coding begins

○ Must think about environments we develop, build and deploy to.

■ How do we ensure other environments are configured identically?-> Immutable servers everywhere!

The Machine Factory

■ How do we go about creating and managing all of these machine images?

■ At SEEK, we needed a way to generate all of these machine images in a sensible, standardised and organised way.

■ Enter: The Machine Factory:○ “A process for generating machine images for use in

different contexts such as development, test, build, prod and so on.”

■ So what does this look like?

[Insert diagram here]

Windows Server 2012

AWS AMI

Machine Factory

■ Assembly-line style manufacturing:○ Input images (AMI, ISO, Docker Image etc.)○ Apply Build Steps with shared infrastructure code, using

CM tools, shell scripts etc.○ Output Images (AMI, ISO, Docker etc.)

■ Images put in stasis for their destined environment/context where they will be reanimated.

■ CM tools as a code smell:○ Your application might be too big. For this step,

lightweight options usually suffice.

Tuning the assembling line: Pipelines

■ Booting machines and provisioning from scratch is slow.■ 2 main strategies: Intermediate images & enrichment.■ Create intermediate images to shortcut future builds:

○ Install common infra such as log forwarders, statsd, AV, patches/updates etc.

○ Engineering wide (Platform), team wide (Base) or specific Application images.

Tuning the assembling line: Enrichment

■ Prod should have only the runtime dependencies○ Better Security.○ Better Performance.

■ Dev will have the most. This can be expressed as:

Prod ⊊ Build Server ⊊ Dev

■ Enrich machines from left to right in your pipeline, feeding previous image into next step.

Machine Factory @ SEEKWalking the walk

Applying CD to Talent Search

■ Talentsearch previously part of .NET monolith.■ Talent Search is a web-based tool that Recruiters use to

connect with candidates, and was previously part of the monolith Warner discussed earlier.

Carving out the relevant bits

■ Isolate scope, cut out vertical slices from the monolith.■ Enables us to Focus our automation efforts.■ Broadly this is achieved using 2 approaches:

○ Mocking out downstream APIs for local dev.○ Consumer Driven Contracts, using Pact.

■ These give us isolation and speed.

Automating Windows

■ Windows tooling support in past year is much improved.■ AWS & Vagrant all have first class support.■ Seek have contributed a number of plugins for Vagrant to

make Windows development easier,■ ...a number of useful chocolatey packages,■ ...and a large suite of DSC resources to assist configuring

Windows machines.■ And then there is Packer...

Automating Windows: Packer

“A tool for creating identical machine images for multiple platforms from a single source configuration”

■ Packer is the core of our Machine Factory.○ It will boot a machine from an image such as an ISO○ Configure it with scripts or CM tools. ○ Export it into multiple formats - such as a Vagrant box,

Docker image or an AMI.■ Uses a simple JSON DSL to easily comprehend and visualise

the steps applied in the manufacturing process.

Automating Windows: Packer

■ Packer uses SSH, which is very fragile on Windows.■ SEEK have helped create a port of Packer to communicate

natively to Windows over WinRM.■ The result is Packer Community, a set of plugins for Packer

that make Windows machine automation a breeze.■ Using Packer we have automated:

○ our local dev boxes in Vagrant,○ our build/test/prod servers in AWS,○ ...across Linux, Windows Server and Mac OSX.

#winning

■ Dev env < 10 mins for .NET API + UI■ Dev -> Build -> Prod from Factory

○ Vagrant: Vbox, Parallels, Mac + Windows

○ AWS■ Commit to Release in 7 mins.■ Proven ‘vertical slices’ strategy.

Bonus: Challenges & Lessons on WindowsTips on effective use of Vagrant, Packer and DSC.

Vagrant

■ Other authoring environment dependencies = anti-pattern. ○ Installing VM provider (e.g. Virtualbox) + Vagrant should

be the goal○ Consider scripting the install (DSC, BoxStarter,

Powershell scripts etc.) if you can’t avoid.■ Shared folders and IIS Applications don’t play nicely

○ Use rsync synced folder type instead■ When provisioners get too big/slow, move into a Packer

script - vagrant up should be fast and reliable

Vagrant

■ Always set $VAGRANT_HOME on multi-partitioned machines due to disk space restrictions (n+1 space requirements)

■ Distributing Images○ Use Versions + Atlas, if possible○ S3 also good option

■ GUI Testing (e.g. Protractor tests)○ Consider having a GUI fallback (at the cost of slightly

bigger images)○ X-Forwarding on *nix hosts not ideal

Vagrant

■ Cross-OS concerns do occasionally come into play:○ NPM for example has issues writing to a linux

networked file system where the underlying host is Windows.

■ Domain access can be problematic and increases complexity - avoid if possible.○ Use Vagrant Windows Domain plugin if you can’t.

■ Use multiple NICs with private_network if application requires multiple SSL certificates.

Vagrant

Box cleanup:

■ Clean up after yourself and remove any binaries/installers/cruft not required for day-to-day use

■ Defrag and compress virtual disk images if possible■ Matt Wrock has a number of good tips on how to cut it right

down.

Vagrant

Build .NET apps in Vagrant without Visual Studio:■ Install MS Build Tools and VS Agents■ Package reference assemblies from VS installation into

guest machine.

Install the remote debugger and port forward on 4018 to debug from Visual Studio

DSC

■ Expect pain/effort, it’s in its infancy and is immature. ■ Error messages can be very unhelpful/confusing, and you

should be prepared to get comfortable with Windows debugging in the event viewer and the CLI.

■ IMHO, DSC is still a better choice than the *nix open source alternatives such as Chef/Ansible (although support is improving).

DSC

■ Use the Vagrant and the DSC Plugin to iterate on a box; Vagrant allows you to test your DSC configurations in a clean environment and test any potential issues with conditions/ordering, without polluting your host environment

■ Prefer the feature rich SEEK-Jobs/DSC community resources over the Powershell team (experimental) xModules

■ Avoid the Script resource - It is rarely a good idea to use it. Write your own custom resource instead.

Packer■ Don't try to do it all in one go. Build complexity■ When building a new image, create scripts, run them

manually and then automate once you’re comfortable they’re working as expected.

■ Iteration times can be slow, particularly when booting and installing an OS○ Create intermediate boxes to build off to avoid that

overhead.■ Using Vagrant to test those scripts is often a good starting

point.

Packer

■ Sysprep Windows machines so licenses don’t expire.■ If you’re creating a Windows image, Use Packer Community

(only if you can’t use Linux! :)).■ Don’t forget to disable Windows Updates (at least for dev

machines).■ Windows boxes often need restarting, use the restart-

windows Packer Community provisioner to do so safely.

Packer

■ Build Packer images in pipelines - feed output of Packer runs into the start of another Packer run.

■ For example, use the Virtualbox ISO builder to boot and install a Windows machine with updates applied.

■ That box can now be fed into the Virtualbox OVF builder which runs the application specific provisioners (typically this is the more error prone step).

Thank you

@SEEK_Geek

CO PRESENTED BY:

@terrencebenade

@dius_au @warnergodfrey@matthewfellows

Resources

■ Vagrant (http://vagrantup.com)■ The TAO of Vagrant (http://mitchellh.com/the-tao-of-vagrant)■ Vagrant DSC Plugin (https://github.com/mefellows/vagrant-dsc/)■ Vagrant Windows Domain Plugin (https://github.com/SEEK-

Jobs/vagrant-windows-domain)■ DSC (http://blogs.technet.

com/b/privatecloud/archive/2013/08/30/introducing-powershell-desired-state-configuration-dsc.aspx)

■ Read the free DSC eBook (https://onedrive.live.com/?cid=7f868aa697b937fe&id=7F868AA697B937FE!107) for a small but comprehensive guide to DSC

Resources

■ SEEK-Jobs/DSC resource (https://github.com/SEEK-Jobs/DSC/)■ Setting up a server for use with DSC (see packer templates https:

//github.com/mefellows/packer-community-templates)■ Packer (http://packer.io)■ Packer Windows Plugins (aka Packer for Windows https://github.

com/packer-community/packer-windows-plugins)■ Sample Windows Packer Templates (https://github.

com/mefellows/packer-community-templates)■ Legacy Windows Packer Templates and provisioning scripts (Uses

OpenSSH, however there is plenty of inspiration to be found - https://github.com/joefitzgerald/packer-windows)