2017-10-24 all day devops - disposable development environments

DISPOSABLEDEVELOPMENT ENVIRONMENTS

1

THE DISPOSABLE DEVELOPMENT ENVIRONMENT

ABOUT ME

▸ Boyd Hemphill

▸ CTO - Victory CTO

▸ Community guy …

▸ Austin DevOps Meetup

▸ DevOps Days Austin

▸ Container Days Austin

▸ Docker Austin

▸ Been a Developer, DBA, and Operator.

2

QUICKLY EXPERIMENT WITH INFRASTRUCTURE AND BUILDS THE SAME WAY AS CODE.

Chuck Norris

THE DISPOSABLE DEVELOPMENT ENVIRONMENT 3


CONCEPTS

▸ Separate runtime from application code.

▸ Be able to take risks in the runtime and reap rewards.

▸ Or … at least get back to work in less than 5 minutes.

▸ On board a new developer within 30 minutes

▸ Take risks with the build

▸ Or … at least get back to the original build in less than 5 minutes

4


OUTCOMES

▸ Better developer productivity

▸ Better development for non-functional requirements

▸ Better security and compliance

▸ Better defect reproduction

5


REASON FROM SECURITY PERSPECTIVE

▸ Using security as a motivator …

▸ Separate the runtime from the code in development

▸ Docker

▸ Vagrant

▸ Pursue some security benefits of disposability

▸ Wave hands about other applications of the same thinking

▸ Profit

6

DOCKER DOCKERDOCKER

7


THINGS TO NOTICE

▸ The Docker Container is running the application

▸ The code is still on the host

▸ Developers still use local tools like their IDE.

▸ The `-v $HOME/code/victory-cto/disposable-development-environments/:/code` makes it happen

▸ It is not awesome that we are running Ubuntu:12.04

12

VAGRANTHASHICORP

13


THINGS TO NOTICE

▸ The Vagrant guest is running the application

▸ The code is still on the host

▸ Developers still use local tools like their IDE.

▸ The line in the vagrant file: `config.vm.synced_folder ".", “/opt/code”` makes it happen

▸ It is _still_ not awesome that we are running Ubuntu:12.04

18

THE CASE OF VAGRANT V. DOCKER WILL NOW BE HEARD

Supreme (Kangaroo) Court



CONTRASTS - VAGRANT PERSPECTIVE

▸ Vagrant is running a full machine just like you are in production:

▸ Getting in and trying things is easy

▸ You can potentially rerun the provisioner

▸ Idempotency is hard:

▸ To rerun the provisions you would need Chef, Ansible …

20


CONTRASTS - DOCKER PERSPECTIVE

▸ Docker is running a container. There is not a full OS.

▸ Getting in and trying things is a fairly esoteric

▸ You cannot change a running container

▸ It’s cheap to kill and start a new one though

▸ Immutability is hard:

▸ Each attempted runtime is a new image and container

▸ Different set of skills than most are used to

▸ Need to get good at optimizing Dockerfiles for build time.

21


KEY CONCEPTS

▸ Vagrant and Docker solve the same problem with respect to the developer environment.

▸ Use the one that best represents your production environment,

▸ not your aspirations to be a cool kid,

▸ not resume-driven-development

▸ The configuration of the runtime is defined in code

▸ Dockerfile

▸ Vagrantfile

▸ We can now hack on them like we can on code.

▸ Let’s hack …

22

ALL YOUR APPLICATION ARE BELONG TO THEM.

Every Cyber Security Bulletin



SECURITY REPORT

▸ The Quay container registry offers Claire. So let’s look at it for a baseline.

▸ There are 75 medium CVEs.

▸ Only 4 are fixable.

▸ Occam’s Razor tells us to upgrade the OS.

24


PANIC

▸ OMFG! Upgrade the OS!!!

▸ It’s raining razor blades!

▸ Our runtime is separate from the code and it is described by code.

▸ We have super powers.

25


UBUNTU 12.04 -> 16.04

▸ Note that to upgrade the OS, all that is done is to change line 2!

▸ The container is rebuilt and tested.

▸ The candidate image is pushed to the registry

26


OUR NEW SECURITY PROFILE

▸ We are down to 68 medium CVEs.

▸ Still, only 4 can be addressed.

▸ However the remaining 64 have a hope of being addressed because 16.04 is still under support.

27


LET’S PROFIT - SHARING

▸ The container (or VM with Vagrant) is using the Ubuntu 16.04

▸ We have run tests.

▸ We have pushed it to the registry and seen the change in our security posture

28


LET’S PROFIT - SHARING

▸ Other developers:

▸ `git stash`

▸ `git pull`

▸ `git merge ubuntu-16.04`

▸ `docker kill …`

▸ `docker run …`

▸ Could be from the registry

▸ Could be a locally built container

▸ `git stash apply`

▸ Git back to work.

▸ Total time: 5 minutes

29

SORRY 143,000,000 PEOPLE STRUTS WAS REALLY HARD TO UPDATE.

Equifax



LIBRARY MANAGEMENT

▸ Flask ~= Struts for our purpose here.

▸ And our Flask version is … ahem … experienced.

31


STEPS TO FIX - HAPPY PATH

▸ One team member upgrades flask.

▸ She runs tests

▸ She gets coffee

▸ Tests pass

▸ She commits and pushes.

▸ Build pipeline registers the container.

▸ She notifies the team

▸ Team recycles their containers to the new one.

32


STEPS TO FIX - SAD PATH

▸ One team member upgrades flask.

▸ She runs tests

▸ She gets coffee

▸ Tests fail

▸ She exercises the darker side of her vocabulary.

33


STEPS TO FIX - SAD PATH - 2

▸ This is reality. Upgrades to critical libraries often cause pain.

▸ Our intrepid developer notifies her team lead.

▸ She works to fix issues in the code.

▸ She gets a tiger team together for the same.

▸ Once tests are passing, she shares the new run time and necessary code changes in the _same commit_!

▸ For all other developers, it’s a 5 minute change, and some effort to integrate any work on their branches/features.

▸ This used to mean developers take hours/days to get up to date because of the bespoke nature of their environment.

34


STEPS TO FIX - TRAGEDY

▸ Sometimes you cannot take on the risk to do the sad path work.

▸ Before, this could mean losing the developer(s) for a week while she re-crafted her bespoke environment.

▸ Now it means:

▸ `git checkout new-branch`

▸ `docker run …` (or `vagrant up`)

▸ Tragic? Yes … goats were sad.

▸ Time consuming? Not nearly as much …

35


THE BIGGER PICTURE

▸ When a necessary upgrade occurs, it can be done quickly and painlessly.

▸ Quick and painless means it will happen more often.

▸ Happening more often means better security.

▸ Developers are contributing to production security, not fighting with their personal environments.

▸ Money/time is saved can be used for better security tooling.

36

BUT WAIT … THERE’S MORE

Ronco



ZOOMING OUT FARTHER

▸ This same thinking can be used for any sort of change.

▸ Disposable development environments is really nothing more than reaping the benefits of standardization and configuration as code.

▸ See Adam Jacobs, Jez Humble or any other Adjutant to the Regional DevOp

▸ Want to try some crazy code to solve a performance issue, but need to tweak the runtime for more memory?

▸ Want to constrain memory to see how your code performs in a simulated situation?

▸ Want to test the addition of firewall rules to each machine (AWS recommended best practice)?

▸ Want to reproduce network segmentation faults to ensure a graceful user experience?

39


INNOVATION COSTS LESS

▸ All of these things are innovations

▸ Disposability is the pinnacle of the, “Fail fast” mentality.

▸ Replace the biggest voice in the room with a functional model and data to make a decision.

40

YOU ARE LIMITED BY YOUR IMAGINATION AND LAPTOP MEMORY.

Boyd



COLOPHON

▸ A github repo with the all the code and instructions for your to play with: https://goo.gl/dvejTS

▸ The container image with a live security scan from a package perspective: https://goo.gl/qgHzoT

▸ The container image with a live security scan from a CVE perspective: https://goo.gl/B7WE1Z

▸ A blog post coming this week with much of the explanations and thinking written out: https://victorycto.com/blog

42

https://goo.gl/dvejTS

https://goo.gl/qgHzoT

https://goo.gl/B7WE1Z

https://victorycto.com/blog

2017-10-24 all day devops - disposable development environments

Technology