The Docker ecosystemand the future of

application deployment

1 / 110

Bom dia!I'm Jérôme.

2 / 110

Bom dia!I'm Jérôme.

First time in Brazil!

3 / 110

Bom dia!I'm Jérôme.

First time in Brazil!

Unfortunately, I don't speak Portuguese ☹

4 / 110

Bom dia!I'm Jérôme.

First time in Brazil!

Unfortunately, I don't speak Portuguese ☹

Therefore, I will continue in English. Thank you!

5 / 110

Who am I?French software engineer living in California

I have built and scaled the dotCloud PaaS

I know a few things about running containers (in production)

I ♥ Open Source

I think Docker will help us to build better apps!

6 / 110

OutlineEvolution vs Revolution

Containers: evolution or revolution?

How Docker can help us today

How Docker will help us tomorrow

7 / 110

Evolution vs

revolution

8 / 110

9 / 110

First transatlantic flightVickers Vimy modified bomber from World War I

Altitude: 12,000 ft

Speed: 115 mph

Cross the Atlantic in hours instead of days

10 / 110

Evolution

or

revolution?

11 / 110

RevolutionPaved the way for air travel

Air travel was initially very expensive

Ultimately became the main method for long-distance passenger travel

12 / 110

13 / 110

First transatlantic supersonic jetConcorde

Altitude: 60,000 ft

Speed: 1,350 mph

Cross the Atlantic even faster

14 / 110

Evolution

or

revolution?

15 / 110

EvolutionAwesome technological achievement (Paris/New York in 3.5 hours!)

We haven't done better since 50 years ago

But:

did not replace subsonic jets

did not turn the impossible into possible

16 / 110

17 / 110

Apollo missionsSaturn V

Altitude: the Moon (~1,300,000,000 ft)

Speed: 24,500 mph

Sends people to the Moon, and back

18 / 110

Evolution

or

revolution?

19 / 110

RevolutionExcept for those people who don't believe that we reallywent to the moon, of course...

20 / 110

We sent people to the Moon...

Now what?

21 / 110

22 / 110

Space programAltitude and speed: irrelevant

Benefits:

put satellites into orbit

land rovers and probes on Mars, Titan...

examine planets, comets, and more

launch probes outside of the solar system

Picture: Delta Rocket launching a GPS satellite

23 / 110

Evolution

or

revolution?

24 / 110

RevolutionNumerous scientific breakthroughs

And consumer applications:

GPS

satellite TV

internet access in planes (!)

etc.

25 / 110

How does this apply to us, developers?

26 / 110

27 / 110

Physical machinesOrder time: hours

Delivery time: days

Install time: hours

28 / 110

29 / 110

Virtual machinesOrder time: minutes

Delivery time: minutes

Install time: hours

30 / 110

31 / 110

Virtual machines with an APIOrder time: seconds

Delivery time: seconds

Install time: minutes

This is typically called "IaaS" or "Cloud" ☺

32 / 110

The cloud:

evolution or

revolution?

33 / 110

34 / 110

ProblemTraffic patterns have daily peaks

Low traffic during the night (and sometimes, low traffic during week-ends)

Must provision servers according to the peaks

Note:if you are dealing with worldwide traffic, its distribution changes during the day.

35 / 110

Solution

36 / 110

37 / 110

Solution: autoscalingAdd servers automatically, when load increases

Remove them automatically, when load decreases

Requires on-demand servers:

available in minutes (not hours or days)

elastic pricing model ("pay for what you use")

automation (API)

This was not possible before.

38 / 110

Revolution

39 / 110

ProblemOld model: developers write code, ops put it in production and run it

Applications are increasingly complex

It's increasingly hard for ops to fix problems

It's increasingly hard for devs to anticipate them

40 / 110

Solution

41 / 110

42 / 110

Solution: DevOpsDesign capacity and scaling upstream

Devs and Ops in the same team

Everybody is on call

No more manual operations (racking machines, deploying OS...)

Everything can be scripted, automated

This was not possible before.

43 / 110

Revolution

44 / 110

ProblemDeploying new code is risky

Consequence: we are afraid of deploying

Consequence: we deploy less often

Consequence: when we deploy, we deploy bigger changes

Consequence: deploying new code is even riskier!

45 / 110

Solution

46 / 110

47 / 110

Solution: blue/green deployment"Blue" servers are in production

Deploy new release to "green" servers

Progressively move traffic from blue to green

If anything goes wrong: go back to blue

Requires:

2x the number of servers (for a short period of time)

... or servers on demand (Cloud!)

This was not possible/realistic before.

48 / 110

Revolution

49 / 110

Containers :

evolution or

revolution?

50 / 110

51 / 110

KitematicOS X GUI to start containers

Order time: seconds

Delivery time: seconds

Install time: seconds

52 / 110

53 / 110

54 / 110

Docker CLI and APIOrder time: milliseconds

Delivery time: milliseconds

Install time: milliseconds

55 / 110

Evolution

or

revolution?

56 / 110

Containers seen by hosting industryContainers = lightweight virtual machines

Higher density (3x-10x better than VMs)

Faster boot times

Possibility of bare metal performance

$$$

57 / 110

Evolution

58 / 110

Come on, can we

do better?

59 / 110

60 / 110

61 / 110

62 / 110

Testing, continuous integrationTesting environments must be:

short-lived (tests can run in seconds)

disposable (destroyed after the tests)

quick to boot (so that tests can start rapidly)

Containers are the perfect fit!

63 / 110

When a test depends on a databasePlan A: use a common "test database"

undefined state

needs to be reloaded before each test

slow (especially on big test sets)

still vulnerable to concurrent access

64 / 110

When a test depends on a databasePlan B: spin multiple copies of the DB in VMs

expensive (spin up 100 VMs if you have 100 tests)

slow (spin up 1 VM, test, repeat)

complex to implement (even with the help of IAAS or Vagrant)

65 / 110

When a test depends on a databasePlan C: spin multiple databases in containers

cheap, low resource usage (even with big databases)

fast boot times (even with big databases)

easy to set up (thanks to CLI and API)

66 / 110

Testing / CI platforms were early adopters of Docker, before itwas stable. Why?

67 / 110

68 / 110

Testing is critical to release quality,

but it does not affect uptime.

(At least, not directly.)

69 / 110

I believe this isrevolution.

What do you think?

70 / 110

Docker can do more.

71 / 110

How Docker canhelp us today

72 / 110

73 / 110

Software deployment before DockerInstall packages A, B, C

Install libraries X, Y, Z

It doesn't work: wrong versions

Try again

Wasted time, frustration

74 / 110

Software deployment with Dockerdocker run joaodubas/orientdb

It works!

Avoid "dependency hell"

75 / 110

Software dependencies before DockerThis software depends on:

library A version X

library B version Y

library C version Z

other hidden, unspecified, unknown dependencies

76 / 110

Software dependencies with DockerThis container image depends on:

Intel 64 bits CPU instructions

Linux kernel ABI

Those things are very, very stable.

77 / 110

Software packaging before DockerPackaging is distribution-specific (different for Red Hat, Debian, OS X...)

Packaging can also be language-specific (Node.js npm, Ruby gems, Python eggs...)

Packaging is a useful, but uncommon skill (who here knows how to make a package?)

Packaging is often done by separate people (ops, maintainers...)

78 / 110

Software packaging with DockerDevelopers can build container images easily

Images are described by a Dockerfile

If you know how to write a shell script, then you know how to write a Dockerfile!

79 / 110

Sample DockerfileFROM debian:jessieMAINTAINER Jessica Frazelle <jess@docker.com>

RUN apt-get update && apt-get install -y \ build-essential \ ca-certificates \ git \ qmlscene \ qt5-qmake \ qt5-default \ qtdeclarative5-dev \ qml-module-qtquick-controls \ qml-module-qtgraphicaleffects \ qml-module-qtquick-dialogs \ qml-module-qtquick-localstorage \ qml-module-qtquick-window2 \ --no-install-recommends

RUN git clone --recursive https://github.com/Swordfish90/cool-retro-term.git /srcWORKDIR /srcRUN qmake \ && make

ENTRYPOINT [ "/src/cool-retro-term" ]

80 / 110

Another example:

tech support

81 / 110

Tech support before Docker

82 / 110

Tech support before Docker

83 / 110

Tech support before Docker"Works For Me"

"Can't reproduce"

"I would appreciate if you could test between 3 and 4am"

"So to trigger the bug you have to install X and Y thenconfigure A, B, and C, then download the extra file, put it inthis directory (which doesn't exist?!?) and then if yourestart three times in approximatively 5 minutes butsometimes it takes longer you will see that the images areshifted by a few pixels but if it doesn't work try to upgradeY to version Z and try all over again..."

84 / 110

Tech support with DockerGet a well-defined, reproducible environment

Define this environment in a Dockerfile

Build this Dockerfile into a container image

Run this container image anywhere

Get exactly the same behavior

85 / 110

Revisiting blue/green deployment

86 / 110

Blue/green deployment before DockerMust create server images (slow)

Server images are often specific to IAAS (cannot run them locally for testing)

Servers must be stateless (data files are wiped out at each deploy)

87 / 110

Blue/green deployment with DockerContainer image creation is fast

Container images can be deployed anywhere (local dev env, physical servers, VMs...)

Blue/green can be on the same machine, so data can be preserved across deploys

88 / 110

Onboarding new team members

89 / 110

Onboarding before Docker

90 / 110

Onboarding with DockerHire developer*

Give them a computer

Install Docker

git clone ...

docker-compose up ...

Your stack is up and running

*Actually the most difficult part.

91 / 110

Docker Composeweb: build: src/front links: - redis - postgres:db - api - zookeeper:zk

redis: image: redis

postgres: image: postgres

api: build: src/backend links: - redis - postgres:db - cassandra - zookeeper:zk

zookeeper: image: jplock/zookeeper

cassandra: image: spotify/cassandra

92 / 110

Cold, hard dataHow long does it take for a developer to join a new project?

Before Docker: 2 days

After Docker: 2 hours

(Source: Worldline)

93 / 110

Microservices

94 / 110

Microservices before DockerDifferent languages, frameworks, build systems

Service discovery challenge

Differences between dev and prod environments:

dev = many processes on one VM

prod = many processes on many VMs

different network layout

network layout is visible to the processes

95 / 110

Microservices with DockerEverybody ships containers

docker-compose does service discovery in dev, and offers reliable setup of the whole stack

Differences between dev and prod environments:

dev = many containers on one VM

prod = many containers on many VMs

different network layout

network layout is invisible to the processes

96 / 110

I vote for:

revolution

97 / 110

How Docker willhelp us tomorrow

98 / 110

Docker Machine

99 / 110

Docker MachineCreate Docker hosts with a single tool

docker-machine create -d virtualbox my-dev

docker-machine create -d digitalocean prod-001

docker-machine create -d amazonec2 prod-002

Work in progress!

100 / 110

Docker Swarm

101 / 110

Docker SwarmControl a cluster of Docker hosts, as if it were a single host

Talk to Swarm using the Docker API

Use the Docker CLI or any existing Docker client

Swarm talks to Docker hosts with the same API

No change to existing clients and hosts!

102 / 110

Scaling Swarm with MesosOn large-scale clusters (100s, 1000s of nodes), scheduling is a hard problem

Mesos solves the scheduling problem elegantly

Swarm and Mesos will integrate:

you talk to Swarm with the Docker API,

Swarm offloads the hard job to Mesos

103 / 110

Network extensionsBuild overlay networks to connect containers across multiple hosts and multiple clouds

Integrate Docker containers with existing networks (Open vSwitch, VLANs, VXLAN, etc.)

Some extensions:

flannel

pipework

socketplane

weave

104 / 110

Storage extensionsStateful containers (databases) are challenging

Moving a stateless container is easy (deploy image; start container; done)

Moving a stateful container is hard

See: Flocker by ClusterHQ

105 / 110

Docker on the DesktopWhat for?

better dependency management for proprietarybinaries (e.g. Chrome, Skype, Spotify...)

better control over resources (both quantitative and qualitative)

Check Jessica Frazelle's blog: https://blog.jessfraz.com/posts/docker-containers-on-the-desktop.html

106 / 110

CRIU (Checkpoint/Restore In Userspace)Save process state (not just filesystem state!)

Obvious use-case: live migration of containers

Less obvious, but easier use-cases:

slow start processes

JIT compilers

107 / 110

Recap

108 / 110

RecapBuild with Dockerfiles.

Test rapidly.

Assemble complex stacks with docker-compose.

Implement powerful software patterns easily: immutable infrastructure, microservice architectures.

Manage machines with docker-machine.

Orchestrate clusters with Swarm.

Leverage the numerous tools in the ecosystem.

Last but not least: all of this is Open Source.

109 / 110

Obrigado! Perguntas?

@jpetazzo @docker

110 / 110