lean environment management with docker containers (otech magazine - fall 2015)
TRANSCRIPT
OTech – Fall 2015
Author: Lucas Jellema (AMIS)
Lean Environment Management with Docker Containers
You really must have been living under that proverbial rock to not have heard anything about Docker
[Containers]. So much buzz has surrounded Docker, you could hardly fail to notice it. And the buzz is still
getting louder as well, with companies such as Oracle gradually, and initially a little grudgingly, becoming
part of the conversation.
This article discusses Docker with a specific target audience in mind. I am assuming that like me you are a
developer or platform administrator with an interest in trying out new stuff. You want to run beta
software, try out new tools and frameworks, do some R&D, prepare for demonstrations or training
sessions – and you want to do so quickly, smoothly, and without messing up your environment. Besides,
you have limited physical computer resources (such as CPU, Memory, Disk Space) and you need a lean
approach that optimizes the use of these resources.
You may want to create things you want to share with friends, colleagues or even the world at large –
such as a running environment with various components installed and configured to work together or a
running application prototype. And you would like to share your stuff without creating elaborate
instructions for installing and configuring and certainly without support conversations around ‘it works
on my machine’ ‘okay, big sigh, send me your config files and I will look into them’. You may want to run
some of your stuff on a cloud environment – without too much hassle of getting it installed, deployed
and configured on the cloud (“just lift, shift and run sounds nice, thank you very much!”). From all the
buzz you have a certain interest in learning more about ‘that Docker thing’. And finally: even though the
host operating system of your laptop could be Mac OS or Windows, you do not suffer from a severe
Linux allergy.
If you match the above description of the target audience, then you definitely should read on. This article
will then provide you with a number of things:
insight into what this Docker thing is and why it has become such a hype?
understanding of what Containers are and how they compare to Virtual Machines?
feel for how you can build, ship | share | distribute and run containers, also through cloud
infrastructure?
a way to more efficiently leverage the physical resources in your computer than through juggling
VMs
a structured and fast way to try out new software without messing up your local environment.
a set of tools to get started with Docker on a non-Linux laptop and pointers to going with Docker
on your own
a feel for the status of Docker and current developments such as the Open Container Initiative as
well as the relation between Oracle and Docker
Build, Ship and Run The tag line for Docker is ‘build, ship and run any app, anywhere’. The initial focus for Docker – and for us
in this article - was Run. Subsequently, we will discuss Ship and Build. This section will provide a high
level overview of build, ship and run. The next section will provide more details and concrete examples.
Run
Docker offers the ability to run isolated environments (called containers) in a Linux host that to all
intents and purposes feel like standalone Virtual Machines. These containers have their own file system,
users and groups, network configuration and IP address and process table. However, they do not contain
a full blown operating system and they do not require a hypervisor layer in between. The containers
share the Linux Kernel and the physical resources are allocated at runtime, not pre allocated in a VM
definition. As a result, even though containers walk and talk like VMs, they are far leaner than VMs: they
consume less runtime resources and start and stop much faster. They are also much smaller in terms of
disk space, because of the snapshot or layer based way of defining containers. Because containers run
within the same Linux (host) environment and are therefore ‘closer together’ than true VMs are, there
are some shortcuts available to have containers interact and share resources.
Docker can be seen as a great way to expose core Linux Kernel container namespace isolation features in
a way that made them accessible to the masses. Docker sits on top of what the Linux Kernel offers at
least since release 3.10 (2013). The Docker project was born in 2013 too. Docker is free and open source
under Apache 2 license. All code is available on GitHub (https://github.com/docker/docker).
Docker consists of several parts. The most relevant for our discussion are the Docker host – the engine
that runs Docker containers inside a Linux host – and the Docker client – the command line interface
through which we interact with the Docker host. The Docker host exposes an API that can be accessed
over HTTP. Through this APIs, containers can be started and stopped, built, published and monitored. A
plethora of tools has come into existence that leverage this API from pretty user interfaces that allow
easy Docker container administration.
The fact that containers can be so quickly started and stopped on demand, with relatively small physical
overhead, makes them a great vehicle for implementing dynamic clusters. On top of Docker Host and
this API, there are also various solutions for container orchestration that support running such dynamic
clusters of Docker Containers. Docker Swarm and Google’s Kubernetes are prominent examples of this,
along with native facilities offered by several container-enabled cloud vendors.
Ship
A container can be committed as an image. This means that the state of the container is persisted, to be
used locally as a starting point for new containers. This state consists of the image from which the
container was started originally, complemented with all installation, configurations and other
manipulation that has taken place during the lifetime of the container. The image also has a default
command that is executed when a container is run from the image. Usually this command starts
whatever engine is the heart of the micro service offered by the container, such as the database, the
web server or the application server.
The container image – the frozen state of the container - can be pushed to an image registry, such as
Docker Hub. Image registries can be personal, private (within an enterprise) or public – cloud based.
Anyone with the appropriate access privileges to an image registry can pull images from the registry and
use them locally run containers from.
The public Docker Hub registry contains 1000s of Docker container images. These range from very
specific, hardly reusable configurations to official distributions from software vendors and open source
products. Among the most popular images on Docker Hub are NGINX, Tomcat, Jenkins, MongoDB, Redis,
Ubuntu, Java, Wordpress, Node[JS], PostgreSQL, MySQL, Debian, CentOS, Ruby, Rails, and the Docker
Registry image. Oracle has also – besides MySQL - published images for Oracle Enterprise Linux – though
not for any other software.
Docker Hub – and various other cloud based registries – provide private image repositories where access
is limited to parties you have explicitly authorized. These private registries are typically not free. A
private, local – for example enterprise wide- image registry can be created by running the Docker
Registry container image, which results in a running container that provides the image registry micro
service.
In addition to exchanging images through a registry, local containers and images can also be delivered as
a tarball and distributed using traditional means. This tarball can be loaded into Docker environments to
produce a local image from which containers can be run.
Portability The portability provided by Docker is one of its most prominent features. Portable container images –
either through registries or in tarballs – can very easily be handed around. And because Linux servers
with Docker support are omnipresent, the Docker container image can almost anywhere be loaded and
run. Remember that the image contains not just the software specific to whatever service the container
provides but also the necessary setup of the underlying operating system and platform engine. The
container is a ready to run service specific IaaS + PaaS + SaaS package – that needs little additional
generic IaaS + PaaS to actually run.
Not surprisingly, this portability has led many software vendors to leveraging the Docker Container
image as an important or even their main means of distributing software. Instead of tiresome installation
procedures and complex configuration instructions, they make the container available that has it all. The
number of private enterprise data centers ready to manage Docker containers is rapidly growing and
there is a large set of cloud providers that are also Docker container enabled. The latter means that
when you package any workload you want to run on the cloud can very easily be shipped to the cloud as
a Docker container image. If you can make your cloud bound application run in a local container – fine
tuning all installation, setup, configuration and deployment – then it is a piece of cake to turn that
container in an image that you can then hand off to the cloud.
Some use cases for the shipping power provided by Docker include:
using images as the distribution mechanism for beta-software (software for which no installers
are available yet and whose test results should not depend on environment issues)
using images as the vehicle to hand out training environments (and containers as the means to
run these environments)
using container images as the artifact delivered from the Continuous Delivery process: instead of
application artifacts like WAR and EAR files, CD brings a container that is ready to run with
software and environment to the testing and production environment; especially when Docker
Containers are adopted as a production infrastructure, this delivery approach seems desirable
Implicit Image Interface Container images can have implicit interfaces. These interfaces describe the dependencies of an image –
that need to be satisfied through injection when you run a container based on the image. The Wordpress
image for example states a dependency on a MySQL container into which it can create its content store.
The MySQL image defines an interface through which we can specify a directory on the Docker host file
system to use for storing data files (instead of having them inside the container).
Docker makes it easy to inject dependencies when a container is run. It works with three kinds of
injection:
link – under a certain alias - to another container; this is used to inject the MySQL container into
the Wordpress container
volume – a mapping from a local directory on the Docker host to a mount point inside the
container, replacing a local directory in the container; the MySQL image uses a volume to
optionally store data files in what amounts to a directory in the host rather than in the container
parameter - used to inject a value to be assigned to a predefined environment variable inside the
container; the MySQL image for example allows us to set the name of a database to be created
on image startup
Note that there is no formalized format for specifying the image dependencies. The image description in
the Docker Hub typically provides hints for whatever can be configured and injected into containers run
from the image – but not in a structured, standardized way.
Layers and the Union File System Before discussing the container build process, we need to look a little bit into the internal structure of
images. Images capture the state of a container. They do so through gathering all the files that make up
the file system of the container. Now for the smart parts: when an image – say Tomcat - is created from
a container that was run from the Ubuntu image – Docker is aware of the fact that in the Tomcat image
is hidden away the base Ubuntu image. And when you create an image from a container that was based
on the Tomcat image, Docker will know that too. Because every image is uniquely identified – by a very
long number – Docker can leverage the same base image multiple times in any environment. That means
that only a single copy of a base image needs to be present in any Docker host. In the figure we have five
containers running that are based – directly or indirectly – on the Ubuntu image. This 188 MB sized
image exists only once on the local environment.
An image can add files to the contents of the base image, and also change or delete files from the base
image. Docker makes use of so called union file systems to combine the layers from various images into a
single image. Changes take precedence top to bottom: changes in Tomcat override files in Ubuntu. Union
file systems allow files and directories of separate file systems, known as branches, to be transparently
overlaid, forming a single coherent file system.
Another smart element in how Docker operates: when you run a container, based on an image
definition, a writable layer is created – which is initially empty. When a new file is created in that
container – or an existing file inherited from one of the underlying images is changed or removed – then
the change is recorded in that writable layer. For many containers, this layer will be fairly small in
physical size on disk.
The figure visualizes how container My Web App was started from the image Tomcat that is based on the
Ubuntu image. In the container, some changes were made to the server.xml file that is inherited from
the Tomcat image. This file was copied-on-write to the writable layer created for My Web App. When the
container is stopped – the writable layer is preserved. When the container is started again, it runs with
the same state it had when it was stopped.
At some point, a container can be committed – saved as an image. The image in this example consists
logically of the changed server.xml and a reference to the Tomcat image (that in turn contains a
reference to Ubuntu image).
One or many new containers can be started from the My Web App image. Each new running container
adds exactly nothing in terms of disk space usage – and only consumes memory for its isolated Tomcat
process. This explains why running many containers can be done quickly and with a relatively small
overhead.
Build
Building a Docker container is the least interesting of the threesome build, ship and run. Docker provides
a build mechanism that processes a recipe – the Dockerfile – and executes the commands in that recipe,
starting from the specified base image.
Most commands boil down to the execution of Shell-scripts to create, install or configure items on the
Linux file system. During the build process, any file available in the build context – the directory that
contains the Dockerfile including all subdirectories – is available, to copy, execute or otherwise leverage
from the commands.
After this standard Docker build process is complete and the container is running, any additional change
wrought in a container adds to the definition of the container – and can therefore be seen as part of an
extended build process. For the final image, it does not matter whether the Shell-script based standard
Docker build process modifies the state of the container or whether a subsequent operation takes over
for more complex configuration management activities.
In recent years, we have collectively moved away from Shell-scripting based environment provisioning
and configuration management. These scripts are hard to create, hard to maintain, hard to reuse, very
fragile, not easily created as idem potent (multi-time runnable) and typically not portable. For these and
other reasons, we have embraced tools such as Ansible, Salt, Chef and Puppet to help us perform
provisioning and configuration management in a better way: more declarative (focus on what to achieve
instead of what to do), easier to executed in automated workflows, more portable, easier to configure
through parameters and with a large degree of reuse. The Puppet Forge – one case of a repository of
reusable configuration management recipes - contains many ready to use modules that can perform
provisioning of for example a complete SOA Suite 12c environment.
Because of this, I recommend the workflow illustrated in the next figure for building Docker container
(images).
First, the foundation is laid through the Dockerfile – the container based on a designated image (step 1 )
with some groundwork in place, such as the installation of a configuration management tool of choice
(step 2, using Puppet in this case). Next, the manifests that describe the result to be achieved through
Puppet are copied into the container (step 3). When the classic Docker build is complete and the
container is running, the execution of the Puppet manifests takes place (step 4). In this step, the complex
installation and configuration activities take place that may include running installers and post-
installation steps, cloning projects from GitHub or other source code repositories, deploying custom
applications on top of standard platform components (for example your web application on top of
Tomcat). Finally, when Puppet has done its thing, we can decide to commit the container as a new image
(step 5). This image can be our starting point for running many containers.
Note that some steps in the Dockerfile contribute to the runtime attributes of the container image –
such as the exposed ports and the command executed during start up. These commands cannot be
replaced by configuration management tools.
Docker on Windows or Mac OS Docker exposes and leverages Linux Kernel features. It is therefore obvious that Docker does not run
natively on Windows or Mac OS (or indeed Solaris, AIX or any other operating systems besides a
relatively recent distribution of Linux). In order to work with Docker on a non-Linux machine, we need to
work with a Virtual Machine that does run Linux. This can be a very tiny, trimmed down VM – entirely
focused on running the Docker engine.
Several tools are available to help work with Docker on Mac OS or Windows. Boot2docker was
the most widely used tool; it was recently succeeded by the Docker Toolbox. A third option – one
that I prefer myself – is Vagrant. All three tools basically offer a similar way of working: through a
hypervisor – commonly Oracle VirtualBox – a Virtual Machine is created and run, based on Linux.
This dockerhost VM is Docker enabled (the latest version of Docker is installed and configured).
Subsequently, Docker commands can be handed to the tool of choice – boot2docker, Vagrant of
the toolbox – and are sent to the Docker API running inside the VM. Commands include
instructions to run (first time), build, start and stop containers, to commit or export containers,
to pull, push or save images and to execute commands in a running container. All provided in the
native non-Linux host environment – all taking effect in the Docker environment.
Microsoft has announced native container support in Windows Server 2016. Additionally, Docker and
Microsoft have stated that the Docker client will become available on top of those container features.
Note that this does not mean of course that Docker images based on Linux will be portable to the
Windows host running the Docker client. Oracle has announced the same thing for Solaris Zones (with
the large difference being the fact that Solaris Zones has been around since 2005 and support for
containers in Windows is still being invented).
Oracle and Docker Oracle has not been at forefront of Docker supporters. Only in the last few quarters did Oracle make any
noise around and overtures in the general direction of Docker. For example, there have been no
announcement around support for running Docker containers on the Oracle Public Cloud. Oracle
provides a few Docker images – for MySQL and for Oracle Enterprise Linux – but not for any other
software. The main reason given is that a copyright license agreement has to be explicitly accepted
before Oracle software can be downloaded and installed. Perhaps Docker container images can be
published similar to the prebuilt developer VMs on OTN
(http://www.oracle.com/technetwork/community/developer-vm/index.html ) – but for the moment the
only images available are complete, stand-alone VirtualBox VMs. Note that for some of the beta-
programs, Oracle is starting to use Docker container images as a way to distribute the beta-software. In
the absence of the finalized installer and in order to prevent beta-issues from environment problems,
this sounds like a good way of working.
Oracle has published the Dockerfile and instructions to build a Docker container image with WebLogic
12c. You have to download a few pieces of software – JDK 8, WebLogic installer – and then you can run
the provided Dockerfile to build the container. The final result is a WebLogic image that – if it weren’t for
the license situation – Oracle could have published in its entirety. WebLogic is also officially certified by
Oracle to run on Docker – as was announced in the Spring of 2015:
https://blogs.oracle.com/WebLogicServer/entry/oracle_weblogic_server_now_running.
Important steps were taken in the direction of Docker in early Summer 2015: the announcement of
Docker integration into Solaris Zones and the fact that Oracle joined the Open Container Initiative (OCI).
The latter is an industry wide collaboration to develop a standard for container definitions – as open
source, based on the Docker container definition – to ensure portability and prevent emergence of
competing container definitions. Again, Oracle joined OCI a bit later – lagging by just a few weeks - than
the initial launch of the initiative.
More clarity around Oracle’s relationship with Docker is expected to emerge during Oracle OpenWorld
2015, later this month.
Getting started with Docker Getting started with Docker is pretty easy. Especially when you are already on a Linux machine. The all it
takes is install Docker, run Docker and start your first container with a single command, such as:
docker run –d –p 80:80 nginx
which will start a new container, based on the NGINX image, run it in the background and expose the
webserver at port 80 of the local host. With
curl localhost:80
you can access the NGINX engine in the container and see its HTML response.
To get started with Docker on a Windows or Mac OS machine, one approach is to install VirtualBox
(preferably 5.0) and Vagrant. Then have Vagrant generate the VM with Linux and Docker and
subsequently create and run the container inside that VM.
I have created a small workshop to quickly get started with Docker – especially on a Windows or Mac OS
machine. In a few hours, you will see the essential bits and pieces that allow you to add Docker to your
personal tool belt. The workshop has you run containers with NGINX, MySQL and Wordpress, Java,
JDeveloper (yes, a container with a GUI) and Oracle XE database. You will find the workshop in GitHub:
https://github.com/lucasjellema/introduction-docker-workshop.
Conclusion Docker provides a widely accepted infrastructure for running containers – light weight, isolated compute
cells that are similar to VMs but far outperform them. The Docker container image format is portable
across all environments that can run Docker containers – of which there are many, from laptop to public
cloud. Creating and sharing such images is easy – and ready to run images are available for most popular
frameworks, technologies and products.
Docker provides a great mechanism for any developer or platform administrator who wants to try out
new software, quickly get started with a product, share an application or environment configuration with
colleagues, peers or for example students in a training session. Or with the entire world for that matter.
Through Docker, isolated, stand-alone environments can be created, linked together, started and
stopped and published – all while using minimal physical computer resources.
The build process that produces the container images can be automated. A combination of the out of the
box Docker container build with an advanced configuration management tool such as Chef or Puppet
seems to provide the optimal workflow.
Docker leverages Linux kernel features. To run Docker containers on a non-Linux host, an intermediate
Linux-layer is required: a virtual machine that runs Docker on top of a Linux distribution. Various tools
are available to create and manage this intermediate layer – such as the Docker Toolbox and Vagrant.
The industry is embracing containerization in general and Docker specifically- through the publication of
Docker images, support for containerization and for running containers on public cloud infrastructures.
Oracle has made some moves – and is expected to make some more in the near future.
Now is a very good time for you to get your hands dirty and try out Docker for yourself. In 30 minutes
you should be able to have your first containers up and running. What are you waiting for?