Download - Understanding and Hardening Linux Containers
-
8/18/2019 Understanding and Hardening Linux Containers
1/122
NCC Group Whitepaper
Understanding and Hardening
Linux ContainersApril 20, 2016 – Version 1.0
Prepared byAaron Grattafiori – Technical Director
AbstractOperating System virtualization is an attractive feature for efficiency, speed and mod-
ern application deployment, amid questionable security. Recent advancements of the Linux kernel have coalesced for simple yet powerful OS virtualization via Linux
Containers, as implemented by LXC, Docker, and CoreOS Rkt among others. Recent
container focused start-ups such as Docker have helped push containers into the
limelight. Linux containers offer native OS virtualization, segmentedby kernel names-
paces, limited through process cgroups and restricted through reduced root capa-
bilities, Mandatory Access Control and user namespaces. This paper discusses these
container features, as well as exploring various security mechanisms. Also included is
an examination of attack surfaces, threats, and related hardening features in order to
properly evaluate container security. Finally, this paper contrasts different container
defaults and enumerates strong security recommendations to counter deployment
weaknesses– helping support and explain methods for building high-security Linux
containers. Are Linux containers the future or merely a fad or fantasy? This paper
attempts to answer that question.
-
8/18/2019 Understanding and Hardening Linux Containers
2/122
Table of Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Virtualization Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Benefits of An OS-Virtualization System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Drawbacks of an OS-Virtualization system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Linux Containers Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1 A Brief History of OS Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Linux Containers: where are they now? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Prior Art: Linux Container Security, Auditing and Presentations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 TL;DR Linux Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3 Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.1 Namespaces Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2 Namespaces Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3 Mount Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.4 IPC Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.5 UTS Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.6 PID Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.7 Network Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.8 User Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4 Control Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.1 Cgroups Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 Working with Vanilla cgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3 Containers and cgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.4 Future of cgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5 Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.1 Capabilities Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.2 Additional Introductory Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2 | Understanding and Hardening Linux Containers NCC Group
-
8/18/2019 Understanding and Hardening Linux Containers
3/122
5.3 Understanding Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.4 Exploring Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.5 Capabilities and User Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.6 Capability Defaults In Modern Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.7 A World Without Root . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6 Configuration and Basic Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.1 LXC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.2 Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.3 CoreOS Rocket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7 Understanding Container Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.1 The Linux Kernel Itself . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.2 Exploring Container Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.3 LXC Specific Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
7.4 Docker Specific Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.5 CoreOS Rkt Specific Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
7.6 Indirect or Unexpected Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8 Recent Security Advancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.1 The User Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.2 Mandatory Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.3 Syscall Filtering with Seccomp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
9 LXC, Docker and CoreOS Rocket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
9.1 LXC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
9.2 LXC Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
9.3 LXC Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
9.4 Brief LXC Security Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
9.5 Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
9.6 Docker Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
9.7 Docker Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3 | Understanding and Hardening Linux Containers NCC Group
-
8/18/2019 Understanding and Hardening Linux Containers
4/122
9.8 Brief Docker Security Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
9.9 CoreOS Rocket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.10 CoreOS and Rkt Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.11 Rkt Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
9.12 Rkt Security Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
9.13 Container Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
10 Security Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
10.1 Generation Container Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
10.2 LXC Specific Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
10.3 Docker Specific Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
10.4 CoreOS Rkt Specific Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
10.5 Relevant Kernel Hardening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
11 The Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
11.1 Containers on the Desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
11.2 New Potential Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
11.3 Additional Lightweight Isolation and Sandbox Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
11.4 The Open Container Initiative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
11.5 Containers In Other Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
11.6 Container Specific Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
11.7 Unikernels and Microhypervisors and Hybrid Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
11.8 The Big Idea Of Microservices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
12 The End . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
12.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
12.2 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
12.3 About The Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4 | Understanding and Hardening Linux Containers NCC Group
-
8/18/2019 Understanding and Hardening Linux Containers
5/122
1 Introduction
``I am large, I contain multitudes'' – Walt Whitman
Linux containers have recently developed into a production-ready technology for OS level virtualization, 1
yet very little security research or best practices have been made public, and concerns for deployment2 and
security3, 4 abound. This whitepaper seeks to expand on what little security information exists on what con-
tainer technologies are capable of and can ultimately provide. This paper starts by examininghow containerscompare to hardware virtualization5 and what prior vulnerabilities have occurred in these systems. This
paper explores Linux container security underpinnings and discusses building and leveraging containers
to provide strong application isolation. The paper also explores future areas of potential vulnerability or
security research.
Much more than ``chroot on steroids'', Linux containers and the underlying features which power them offer
an entire ecosystem of software. Various pieces form the ability to build a operating system jail or application
sandbox, or to simply better package applications. This can all be done while offering extremely low re-
source overhead and many features typically restricted to hardware virtualization such as snapshots, pausing
virtual machines (VMs) or live VM migration. Linux kernel namespaces, capabilities and resource limits via
cgroups offer excellent tools for building defense in depth, a strongly encouraged security paradigm for all
types of applications. From web applications and network services to desktop applications and thick clients,many of the container methods or software discussed within this paper also support different versions of
the Linux kernel on almost any supported hardware. This can offer much needed security improvements
in embedded or Internet of Things (IoT) devices. Finally, Mandatory Access Controls (MAC) and system
call (syscall) filtering, given new life by container deployments, offer additional and formidable protections
against application or container to host compromise.
Containers are quickly growing in popularity due to the ongoing shift in application or entire datacenter de-
ployments from once traditional three-tier architectures on bare metal to large, powerful computers running
a number of virtual machines. Deployments are now shifting from service oriented architectures (SOA) to
Platform as a Service (PaaS) or ``microservices'', distilling services down. Microservices and PaaS are, as part
of their core design, highly available, fault tolerant, easilyscalable, service oriented and container driven. It is
hardly surprising, given this general movement from hardware to software, there is a surge in the popularity
of containers via LXC, Docker and CoreOS Rocket (rkt), which are helping push the use of Linux containers
out of data centers at Google and IBM and into servers and desktops everywhere. Over the past several
years, container focused companies such as Docker and IBM, along with a growing number of others, have
greatly raised the profile and ease-of-use for Linux containers.
This paper is intended for a wide technical audience, with sections benefiting everyone from security con-
sultants and researchers to enterprise blue teams, application developers, and devops teams. Readers may
be looking to either evaluate container implementations, security risks and challenges, or understand the
hardening features offered by modern container systems. Before proceeding, NCC Group warns against
taking statements, configuration options or other details within this paper as verbatim. Containers are a hot
topic and new developments are announced seemingly every week. In addition to the evolving containerofferings, OS virtualization is a quickly moving target. New potential threats will undoubtedly be uncovered,
hopefully kept in line by proactive security enhancements. Along with a general defense in depth design,
assumed protections should constantly be re-evaluated, explored, tested and improved upon.
1https://en.wikipedia.org/wiki/Operating-system-level_virtualization2http://www.csoonline.com/article/2984543/vulnerabilities/as-containers-take-off-so-do-security-concerns.html3http://www.infoworld.com/article/2923852/security/containers-have-arrived-and-no-one-knows-how-to-secure-them.html4http://www.eweek.com/security/security-is-a-key-concern-for-container-users.html5https://en.wikipedia.org/wiki/X86_virtualization
5 | Understanding and Hardening Linux Containers NCC Group
https://en.wikipedia.org/wiki/Operating-system-level_virtualizationhttp://www.csoonline.com/article/2984543/vulnerabilities/as-containers-take-off-so-do-security-concerns.htmlhttp://www.infoworld.com/article/2923852/security/containers-have-arrived-and-no-one-knows-how-to-secure-them.htmlhttp://www.eweek.com/security/security-is-a-key-concern-for-container-users.htmlhttps://en.wikipedia.org/wiki/X86_virtualizationhttps://en.wikipedia.org/wiki/X86_virtualizationhttp://www.eweek.com/security/security-is-a-key-concern-for-container-users.htmlhttp://www.infoworld.com/article/2923852/security/containers-have-arrived-and-no-one-knows-how-to-secure-them.htmlhttp://www.csoonline.com/article/2984543/vulnerabilities/as-containers-take-off-so-do-security-concerns.htmlhttps://en.wikipedia.org/wiki/Operating-system-level_virtualization
-
8/18/2019 Understanding and Hardening Linux Containers
6/122
Containers offer many overall advantages. From a security perspective, they create a method to reduce
attack surfaces and isolate applications to only the required components, interfaces, libraries and network
connections. In an age where complexity and the lines of code for applications such as Microsoft Office
in 2013 are greater than the entirety of Windows XP,6 application sandboxing should be the rule, not the
exception. Moving fully to memory safe native code languages such as Golang and Rust will likely take
many more years, and code fuzzing or qualified security review can only catch so many vulnerabilities.Inner-application sandboxing and outer-application containers offer a method to cut losses and draw a line
in the virtual sand. When performance differences are negligible, is there a realistic reason to not isolate
applications? Apart from time and effort, the security due diligence is encouraged.
Before continuing, NCC Group also cautions readers that OS-level virtualization will always be fundamen-
tally less secure than hardware-level virtualization, which itself is less secure than physical server isolation.
However, such security absolutes should offer little discouragement. Containers chiefly offer a method
to decrease attack surface and improve isolation with a relatively small added complexity and hardware
resources when compared to ``traditional'' virtualization. Any security solution should be well hardened and
vetted, follow the principal of least privilege, the principal of least access and defense in depth.7 Finally,
it should also be noted that several components used by Linux containers and discussed in this document
are relatively new. This includes the User Namespace and unprivileged containers, as well as seccomp-bpf
filtering. As some of these features deal with complex, high-risk areas of the kernel, additional vulnerabilities
or weaknesses are likely to occur within these components.
1.1 Motivation
The motivation for using containers varies depending on the ultimate and intended use case. This may
consist of decreasing power or storage costs related to hardware virtualization inefficiency, allowing de-
velopers to more easily ship and test code or increasing security on a workstation platform through ap-
plication sandboxing (such as ChromeOS). It appears through industry observations and repeated security
engagements by NCC Group, by far the mostpopular motivation is essentially applicationpackaging, testing
and deploying entire stacks on a Platform-as-a-Service (PaaS). Motivations for using containers may involve
providing internal clouds or production infrastructure on top of bare metal or virtual machines. Docker,
Google, Amazon AWS, Rackspace, Heroku, CloudFoundry, Stackato, CoreOS, Flockport, Rancher among
others are an example of the use, and rising popularity of, minimal-overhead Linux OS virtualization.
Modern Linux use stretches from servers, desktops and laptops to mobile phones, routers, IoT and a vast
number of other embedded devices. From micro form-factor devices to supercomputers, Linux is every-
where. Security and application isolation, going beyond simple Discretionary Access Controls (DAC) or
standard Kernel hardening should be available to all, regardless of the hardware architecture of their in-
tended platform. Containers or the technologies that power them can be used to help achieve improved
security with little to no performance overhead.
Future use of the individual features powering containers (such as namespaces, capabilities and cgroups), or
Linux containers as a whole, mayallow for increased security, application sandboxing and isolationanywhere
modern versions of the Linux kernel are supported. This allows for strong security and helps developers to
implement security ideals such as privilege separation, the principle of least access, isolated root capabil-
ities and resource limits. Many of these container security features have yet to be realized in Linux based
embedded devices in much need of security improvements. This ranges from consumer and commercial
grade routers, smart phones such as Google Android, as well as the general Internet of Things (IoT).
6http://www.informationisbeautiful.net/visualizations/million-lines-of-code/7https://en.wikipedia.org/wiki/Layered_security
6 | Understanding and Hardening Linux Containers NCC Group
http://www.informationisbeautiful.net/visualizations/million-lines-of-code/https://en.wikipedia.org/wiki/Layered_securityhttps://en.wikipedia.org/wiki/Layered_securityhttp://www.informationisbeautiful.net/visualizations/million-lines-of-code/
-
8/18/2019 Understanding and Hardening Linux Containers
7/122
Just as Mandatory Access Control can attempt to limit process capabilities, there exists little to no reason
why a modern web browser needs unfettered access to the users' computer. Slowly major operating sys-
tem vendors are implementing such limitations by default, along with other inner-application sandboxing,
requiring exploits to use privilege escalation or force attackers to use multi-vulnerability chaining. However,
these protections thus far are largely unimplemented outside of web browsers and document readers,8 in
addition to mainly being implemented by non-Linux operating systems.
Containers and their supporting features help support an overall model of defense in depth through layered
security9 for applications on a server or a desktop. In the age of Advanced or Persistent Threats and nation
state attackers, defense in depth as a principal is only possible method which may realistically prevent
successful attacks. It should be clear, containers alone do not offer a perfect solution, but they can and
should be used to quickly raise the bar and frustrate system compromises through application exploits,
primarily by adding isolation and reducing system attack surfaces.10
1.2 Virtualization Background
Before exploring Linux containers and their security, it is important to understand the fundamentals of what
the software or system is capable of providing, in addition to general security considerations. Largely bor-rowing terminology from virtualization, the term host in this paper will be used to indicate the primary
Operating System (OS) or device on which the container exists (where LXC is setup, where the Docker
daemon or engine is running, etc). The term guest or container will refer to the collection of processes
or application container itself, running within the host. Finally, the term escape will correspond to a guest
interacting with, or otherwise compromising, the host in a manner not intended. Escaping will often take
the form of violating the core security principals of isolation, such as the guest breaking out of the container.
1.2.1 Full-Virtualization
Since roughly 2006 most commodity x86, x86-64 and ARMv7 microprocessors11 from Intel, ARM and AMD
offer a hardware-assisted virtualization through special CPU instructions. This provides essentially what is
complete isolation between guestkernels and the host, and allows running manydifferent operating systems
within the same physical host.
VMware's ESX, the Xen HVM and KVM within Linux are examples of "Virtual Machine (VM)" technology or
``Hypervisors''. This hardware mode allows the host to support different guest operating systems (such as a
Microsoft Windows or FreeBSD guest on a Linux KVMhost). While speedis oftencomparable to ``bare metal''
execution, full virtualization is still the slowest of the three types discussed within this paper. Although this
form of virtualization requires a number of virtualized hardware devices, the security is quite robust. In some
cases, data is passed through directly to hardware devices, however the attack surface is typically quite small.
This security robustness is largely due to well vetted virtual hardware, often presenting a minimal hypervisor
attack surface when compared to other virtualization methods. When discussing the security and implemen-
tation of full virtualization, the attack surface may differ between so called ``Type one'' hypervisors on bare
metal (e.g. Citrix Xen, VMWare ESXi and KVM) vs ''Type two'' hypervisors, which are implemented on top of a normal kernel (e.g. VirtualBox, VMware Workstation, and QEMU).
8Provided, this is the first logical step, as web browsers and document readers offer significant attack surfaces, are often reachable
by remote attackers and facilitate an ease of exploitation through heap massaging and other factors.9https://en.wikipedia.org/wiki/Layered_security
10With the exception of the Linux kernel, although tools such as seccomp-bpf can help for most hardware platforms.11The IBM POWER, AS400, OS/2 and otherCPU architectures were designed specifically for hardwarevirtualization. These methods
and systems are out of scope for this paper, as they are often implemented within large organizations with specific requirements
(banks, universities, supercomputing labs and research centers).
7 | Understanding and Hardening Linux Containers NCC Group
https://en.wikipedia.org/wiki/Layered_securityhttps://en.wikipedia.org/wiki/Layered_security
-
8/18/2019 Understanding and Hardening Linux Containers
8/122
Security considerations: In terms of security guarantees, OpenBSD's often gruff leader, when speaking on
hardware virtualization via hypervisors takes a particularly pessimistic stance:
``x86 virtualization is about basically placing another nearly full kernel, full of new bugs, on top of a nasty
x86 architecture which barely has correct page protection. Then running your operating system on the
other side of this brand new pile of shit. — You are absolutely deluded, if not stupid, if you think thata worldwide collection of software engineers who can't write operating systems or applications without
security holes, can then turn around and suddenly write virtualization layers without security holes. You've
seen something on the shelf, and it has all sorts of pretty colours, and you've bought it. That's all x86
virtualization is.''
- An openbsd-misc email by Theo de Raddt
Theo's opinion aside, escaping from hardware virtual machines is considered quite difficult and rare,
although surely possible. Weakness have been discovered in several major platforms, typically in
the areas of virtualized device drivers. Most recently in QEMU (as used by Xen) a large number of
vulnerabilities have been discovered, causing regular cloud hosting providers to perform painful host
reboots.12, 13
For instance, the RTL8139 driver contained a heap overflow (CVE-2015-5165), and anemulated block device contained a use after free issue (CVE-2015-5166). Issues were even discovered
within the floppy controller and the PCNET NIC driver (CVE-2015-3209)14 . Vulnerabilities have been
discovered within the QEMU/Xen IDE subsystem (CVE-2015-5154) and within the hvm_msr_read_-
intercept function (CVE-2014-7188). Each of the above issues risked guest escape, Denial of Service
(DoS) or allowed readingdata fromthe hypervisor, depending on the configuration. Historically, within
VMware Workstation (which is a type two hypervisor 15), a guest could escape and execute arbitrary
code on the host (CVE-2009-1244). Finally, Microsoft's Hyper-V has also contained at least one known
escape (MS15-068).
While the paper is now fairly dated, An Empirical Study into the Security Exposure to Hosts of Hostile
Virtualized Environments by Tavis Ormandy offers a strong security overview. Additionally an Analysis
of Hypervisor Breakouts by Insinuator found, somewhat unsurprisingly, that increase in attack surfacethrough drivers, graphics shaders, DMA and other features which travel from guest to hypervisor or
guest to hardware risks additional vulnerabilities, especially in type two hypervisors.16 Despite these
examples, the risk of escape is much lower and the difficulty of host or guest-to-guest exploitation
much higher than other forms of virtualization (excluding various network attacks). Apart from physi-
cally separate hardware, this method offers the strongest guest isolation.17
Deployment considerations: Full virtualization is typically less energy and storage efficient than other virtual-
ization methods. Due to the special CPU instructions, this technology is also only supported on distinct
hardware (e.g. x86, x86_64, ARM), limiting the deployment scenarios. As such, this virtualization type
is also not subtable for low power devices and only recently supported on ARM.18 Finally, full virtualiza-
tion often follows the model of virtualizing the entire operating system; adding additional security to
individual applications within the system is left up to traditional hardening best practices. This includes
12http://vmblog.com/archive/2014/09/29/rackspace-joins-amazon-in-cloud-reboot-over-xen-hypervisor-bug.aspx13http://www.theregister.co.uk/2015/02/28/new_xen_vuln_causes_cloud_reboot/14This arguably overhyped vulnerability was also marketed as ``VENOM'' by an adversarial ``threat'' focused security company.15http://www.golinuxhub.com/2014/07/comparison-type-1-vs-type-2-hypervisor.html16This type one vs type two issue is also easily illustrated by the number of VMware Workstation escapes when compared to
ESX/ESXi.17Unrelated to x86 and x86-64 Hypervisors, the PS3 hack and VM escape was particularly impressive. See How the PS3 Hypervisor
was hacked by Nate Lawson for an excellent write-up.18http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0438i/CHDCHAED.html
8 | Understanding and Hardening Linux Containers NCC Group
https://marc.info/?l=openbsd-misc&m=119318909016582https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5165https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5166https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5166https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5166https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3209https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5154https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5154https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5154https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-7188https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1244https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1244https://support.microsoft.com/en-us/kb/3072000https://support.microsoft.com/en-us/kb/3072000http://taviso.decsystem.org/virtsec.pdfhttp://taviso.decsystem.org/virtsec.pdfhttp://www.insinuator.net/2013/05/analysis-of-hypervisor-breakouts/#more-2167http://www.insinuator.net/2013/05/analysis-of-hypervisor-breakouts/#more-2167http://vmblog.com/archive/2014/09/29/rackspace-joins-amazon-in-cloud-reboot-over-xen-hypervisor-bug.aspxhttp://www.theregister.co.uk/2015/02/28/new_xen_vuln_causes_cloud_reboot/http://www.golinuxhub.com/2014/07/comparison-type-1-vs-type-2-hypervisor.htmlhttp://rdist.root.org/2010/01/27/how-the-ps3-hypervisor-was-hacked/http://rdist.root.org/2010/01/27/how-the-ps3-hypervisor-was-hacked/http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0438i/CHDCHAED.htmlhttp://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0438i/CHDCHAED.htmlhttp://rdist.root.org/2010/01/27/how-the-ps3-hypervisor-was-hacked/http://rdist.root.org/2010/01/27/how-the-ps3-hypervisor-was-hacked/http://www.golinuxhub.com/2014/07/comparison-type-1-vs-type-2-hypervisor.htmlhttp://www.theregister.co.uk/2015/02/28/new_xen_vuln_causes_cloud_reboot/http://vmblog.com/archive/2014/09/29/rackspace-joins-amazon-in-cloud-reboot-over-xen-hypervisor-bug.aspxhttp://www.insinuator.net/2013/05/analysis-of-hypervisor-breakouts/#more-2167http://www.insinuator.net/2013/05/analysis-of-hypervisor-breakouts/#more-2167http://taviso.decsystem.org/virtsec.pdfhttp://taviso.decsystem.org/virtsec.pdfhttps://support.microsoft.com/en-us/kb/3072000https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1244https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-7188https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5154https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3209https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5166https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5165https://marc.info/?l=openbsd-misc&m=119318909016582
-
8/18/2019 Understanding and Hardening Linux Containers
9/122
using least or multi privilege models, using strong authentication, least privilege, controlling system
access and other security standards. Reboots of the host hardware for full virtualization, although
rarely required, are extremely painful (as each VM needs to boot up after the host). Reboots cannot
be performed gracefully unless live migration is supported or another high availability or fault tolerant
system is in place.
In use by: Amazon AWS via EC2 when selecting "private virtualization", powering the Google Compute
Engine (KVM), OpenStack deployments, VMware vCloud Air (ESX) hosting and a vast majority of large
number of private companies with internal server hosting, traditional or modern Platform as a Service
(PaaS), various internal datacenters or QA environments. Type two hypervisors, which use many of the
same hardware technologies, are common on development workstations, with IT personnel, security
researchers and modern power-user systems (this is especially the case given the wide CPU support
in modern workstations and laptops).
1.2.2 Paravirtualization
Paravirtualization (PV) requires a custom OS kernel, with patches for the specific para-virtualization APIs, also
referred to as hypercalls. This also allows for different virtual hardware, at the expense of a modified kernel.
Xen was one of the first to implement paravirtualization, and is the long standing and primary hypervisorplatform for many cloud providers or high-security desktop operating systems.19
Xen offers a powerful production and battle tested paravirtualized environment, as well as support for non-
paravirtualized operating systems via the hardware virtual machine (HVM) mode, similar to the full virtual-
ization method as previously discussed. A large amount of device emulation code within the Xen HVM is
based on code from QEMU project.
Security considerations: This custom patch-set may lag behind the kernel with regard to security issues and
may be incompatible with some hardening protections such as grsecurity. Many updates and patches
will require a reboot of the host, similar to OS virtualization which may add pressure to avoid updating.
Escaping from PV guests may be easier than full virtualization, and has occured several times, notably
and quite recently in Xen, such as XSA 148. Finally, PV guests may introduce weaknesses if configuredalongside PVHVMs, such as a mixed Xen environment.20
Deployment considerations: Hardware support and kernel support may be lacking, or may require addi-
tional updates or modification before a guest can paravirtualized. Security for PV may place hosts
at the mercy of vendor updates, such as the case with Xen. Reboots may also be extremely difficult
depending on the infrastructure.
In use by: Amazon AWS/EC2, Rackspace, and many other hosting providers.
1.2.3 OS-Virtualization
This method provides the highest performance,21 fastest ``start-up'' time, and is widely considered the most
efficient virtualization method. It uses a shared kernel across both the host and guest. Without virtualizationof hardware devices, speed and efficiency are greatly increased, although due to this shared kernel, the
guest is limited to an identical OS as the host (such as Linux on Linux). This method is typically referred to as
``Containers'' on Linux, ``Jails'' on FreeBSD or ``Zones'' on Solaris.
19https://www.qubes-os.org/20http://xenbits.xen.org/xsa/advisory-148.html21http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf
9 | Understanding and Hardening Linux Containers NCC Group
http://xenbits.xen.org/xsa/advisory-148.htmlhttp://xenbits.xen.org/xsa/advisory-148.htmlhttps://www.qubes-os.org/http://xenbits.xen.org/xsa/advisory-148.htmlhttp://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdfhttp://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdfhttp://xenbits.xen.org/xsa/advisory-148.htmlhttps://www.qubes-os.org/http://xenbits.xen.org/xsa/advisory-148.html
-
8/18/2019 Understanding and Hardening Linux Containers
10/122
Security considerations: The shared kernel, which permits the desired efficiency and speed, also gives way
to an increased risk of host compromise or guest escapes. This is due to the fundamentally shared
resources and significantly larger attack surface compared to paravirtualization or full virtualization.
With this shared kernel, syscalls, shared networking, direct device or disk access, information leaks and
the vast majority of kernel vulnerabilities are typically the source of OS virtualization security failures.
Deployment considerations: OS virtualization can be used anywhere a modern Linux kernel is supported.
This allows the capability to use containers equally from non-hardware virtualized big iron to small
embedded systems. Speed, boot time, storage efficiency and flexibility are key advantages. For
large amounts of I/O or network-bound processing, OS-virtualization may be comparable to hardware
virtualization, due to poorly optimized and default NAT or custom storage such as AUFS in Docker.22
In use by: Linux Containers (LXC), Docker, CoreOS Rkt, OpenVZ, Heroku Dynamos, RancherOS, FreeBSD
Jails, Solaris Zones, Illumos/SmartOS, the defunct OpenBSD Sysjail and a number of other systems
such as Mirage OS (a reimplementation of Solaris Zones).23
The following image24 helps illustrate OS virtualization, using a single kernel / ring0 vs hardware (full) virtu-
alization with multiple instances on top of a hypervisor:
1.3 Benefits of An OS-Virtualization System
Overall, the hardware vs software virtualization discussion can easily be easily understood in the context
of RAID (Redundant Array of Independent Disks) solutions, as both hardware and software RAID imple-
mentations exist and are in widespread use. The hardware version typically offers improved performance
and reliability but comes at a cost of yet another kernel driver, electronics cost, restricted disk types, lim-ited motherboard layout and other form factor requirements. On the other hand, a software RAID offers
increased portability, flexibility, source code customization, almost zero hardware requirements, improved
block device support and reasonable performance across a number of hardware platforms.
Another more specific analogy comes from Docker's Jérôme Petazzoni, who likens this virtualization dif-
22http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/File/rc25482.pdf 23http://media.ccc.de/browse/congress/2014/31c3_-_6443_-_en_-_saal_2_-_201412271245_-_trustworthy_secure_modular_
operating_system_engineering_-_hannes_-_david_kaloper.html24Image from Wikimedia Commons.
10 | Understanding and Hardening Linux Containers NCC Group
http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/File/rc25482.pdfhttp://media.ccc.de/browse/congress/2014/31c3_-_6443_-_en_-_saal_2_-_201412271245_-_trustworthy_secure_modular_operating_system_engineering_-_hannes_-_david_kaloper.htmlhttp://media.ccc.de/browse/congress/2014/31c3_-_6443_-_en_-_saal_2_-_201412271245_-_trustworthy_secure_modular_operating_system_engineering_-_hannes_-_david_kaloper.htmlhttp://media.ccc.de/browse/congress/2014/31c3_-_6443_-_en_-_saal_2_-_201412271245_-_trustworthy_secure_modular_operating_system_engineering_-_hannes_-_david_kaloper.htmlhttp://media.ccc.de/browse/congress/2014/31c3_-_6443_-_en_-_saal_2_-_201412271245_-_trustworthy_secure_modular_operating_system_engineering_-_hannes_-_david_kaloper.htmlhttp://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/File/rc25482.pdf
-
8/18/2019 Understanding and Hardening Linux Containers
11/122
ference to brick walls (hard, slow to setup, messy to add or move) vs room dividers (fragile, deployed in
seconds, moved and added quickly). It also safe to say, even with a cumbersome configuration manage-
ment, hardware Virtual Machines (VMs) and other hypervisors will diverge in version, patch freshness, and
configuration which can often result in security gaps or vulnerabilities due to difficult upgrades. On the
other hand, minimal OS virtualization can be easier to keep organized, may offer no additional cost, and
DevOps or system administrators typically better understand the ''threat landscape'' of these environments(as opposed to complex Hypervisors or other virtualization management solutions).
Speed: The shared kernel architecture offers almost non-existent startup speed, as starting containers is
little more than launching new processes. This can speed up testing or deployment, and reduces idle
OS time. These quick start-up times also allows for desktop application containers to be a realistic and
attainable goal with little end-user awareness.
Management: A number of tools already exist to monitor, explore, configure and troubleshoot containers.
While orchestration and service discovery is an ongoing development area (and potential security
pain-point for major container platforms or deployments), there are many emerging and successful
platforms.
Disk Footprint: Required storage is relativelysmall formost base images anddeltas canrequire an extremely
minimal additional footprint. Advanced filesystem options allow for only deltas to be saved and
multiple containers to share the vast majority of the root filesystem through Copy on Write (CoW)
filesystems and Union filesystems such as OverlayFS. OS virtualization can also reduce applications
to their required code, libraries, components and interfaces, a key differentiator from full hardware
virtual machines.
Implement anywhere: Containers and many (but not all) of the supporting kernel or security features work
anywhere Linux can run, compared to only x86, x86-64, and newer versions of ARM for HW virtualiza-
tion.
Ability to freeze and unfreeze containers: This can allow for quick power saving on a massive, data-centerwise scale or to save laptop battery power. By essentially sending a SIGSTOP to all of the processes
bound to a container (for a specific cgroup), these processes will halt or ``freeze'', providing easy parity
with hardware virtualization.
Containers can easily be moved easily: To any Linux host, from the cloud to local machines, spin-up will
work quickly compared to HW virtualization which may have incompatible image or file formats (re-
quiring moving a much larger image than an application container). Note that moving containers is
currently limited to non-running containers, although some recent developments by Docker and LXD
are attempting to solve this limitation.
1.4 Drawbacks of an OS-Virtualization system
Security: Discussions on the risks of OS-Virtualization system almost always center on the relatively immatureimplementation of Namespaces, the problem of root (capabilities) and the shared kernel, which is
a fundamental risk and unfortunately bountiful attack surface. In the words of noted hardware and
virtualization security researcher Joanna Rutkowska, of the Invisible Things Lab, when speaking on
process isolation of Operating Systems25:
25Joannais also a core contributorand authorof thehigh security Xen hypervisor desktop "CubesOS". Seethe article How CubesOS
is different for more information.
11 | Understanding and Hardening Linux Containers NCC Group
http://blog.invisiblethings.org/2012/09/12/how-is-qubes-os-different-from.htmlhttp://blog.invisiblethings.org/2012/09/12/how-is-qubes-os-different-from.htmlhttp://blog.invisiblethings.org/2012/09/12/how-is-qubes-os-different-from.htmlhttp://blog.invisiblethings.org/2012/09/12/how-is-qubes-os-different-from.html
-
8/18/2019 Understanding and Hardening Linux Containers
12/122
``Sure, the inter-process isolation provided by a monolithic kernel such as Windows or Linux could never
be compared to the inter-VM isolation offered even by the most lousy hypervisors. This is simply because
the sizes of the interfaces exposed to untrusted entities (processes in case of a monolithic kernel; VMs in
case of a hypervisor) are just incomparable.''
- Shattering the myths of Windows security by Joanna RutkowskaSoftware all the way down: The lack of CPU/hypervisor enforced isolation pushes security 100% into soft-
ware. While this makes the solution more flexible, it fundamentally increases the potential for attack,
especially if hardware must be exposed directly (such as the sound or video card interfaces) or the
kernel attack surface cannot be reduced (such as through system call filtering). This is essentially an
extension of ``all of the eggs in one software basket'', which can sometimes be a difficult concept to
accept.
Problems with legacy code: Several areas of the kernel still lack support or lack their own namespaces,
a key isolation mechanism for OS virtualization. For example in Linux, the procfs and sysfs pseudo-
filesystems are not namespace aware. This is easily illustrated by examining the required steps for
protecting from privileged containers (that is, containers without the User namespace), various infor-
mation leaks and other vulnerabilities. The problem of procfs is also readily apparent in sysstat related
programs26 which for instance will not report cgroup resource isolations, but instead reflect the ` f̀ree''
memory of the host OS.
Live Migration: The current inability to ``live migrate'', moving a running container from one host to an-
other, when compared to some hardware Virtual Machine (VM) platforms or OpenVZ. It should be
mentioned, developments are underway within Docker to support this27 and with LXD for LXC.28
A homogeneous environment: The inability to virtualize other Operating Systems, as the shared kernel must
remain the same for the userland processes, may be a blocker for some deployments. While a homo-
geneous environment has some security advantages such as less overall patching, more consistent
configuration management and less "edge cases", it risks mass exploitation or vulnerability.
Single point of failure: It goes without saying, a single kernel is a single point of failure, be that performance,
stability or security. If the host platform is disabled, crashes, needs to be rebooted or is compromised,
all of the guest containers are affected. In the case of a host or kernel compromise, the containers
must also be considered compromised. See Software compartmentalization vs. physical separation
for more information and examples by Joanna Rutkowska.
Complexity at scale: Orchestration frameworks (Rancher, MESOS/Aurora, Docker Swarm, LXD, OpenStack
Containers, Kubernetes/Borg, etc) are only recently catching up to the container craze, there are
too many competing models to list. Many have questionable or unaudited security or leave major
requirements out, such as secret management. While containers may be easy to get working within
a workstation or a few servers, scaling them up to production deployment is another challenge alto-
gether, even assuming your application stack can be properly ``containerized''.
For those interested in the comparison between these different virtualization approaches and want more
information, see all three parts of Hypervisors are Dead, Long Live the Hypervisor on osv.io for a good
overview of the problems and solutions.
26http://fabiokung.com/2014/03/13/memory-inside-linux-containers/27http://www.slideshare.net/Docker/live-migrating-a-container-pros-cons-and-gotchas28https://insights.ubuntu.com/2015/05/06/live-migration-in-lxd/
12 | Understanding and Hardening Linux Containers NCC Group
http://blog.invisiblethings.org/2014/01/15/shattering-myths-of-windows-security.htmlhttp://invisiblethingslab.com/resources/2014/Software_compartmentalization_vs_physical_separation.pdfhttp://osv.io/blog/blog/2014/06/19/containers-hypervisors-part-1/http://fabiokung.com/2014/03/13/memory-inside-linux-containers/http://www.slideshare.net/Docker/live-migrating-a-container-pros-cons-and-gotchashttps://insights.ubuntu.com/2015/05/06/live-migration-in-lxd/https://insights.ubuntu.com/2015/05/06/live-migration-in-lxd/http://www.slideshare.net/Docker/live-migrating-a-container-pros-cons-and-gotchashttp://fabiokung.com/2014/03/13/memory-inside-linux-containers/http://osv.io/blog/blog/2014/06/19/containers-hypervisors-part-1/http://invisiblethingslab.com/resources/2014/Software_compartmentalization_vs_physical_separation.pdfhttp://blog.invisiblethings.org/2014/01/15/shattering-myths-of-windows-security.html
-
8/18/2019 Understanding and Hardening Linux Containers
13/122
2 Linux Containers Overview
2.1 A Brief History of OS Containers
A review of past container methods starts with the infamous chroot written by Bill Joy. Included in version
7 Unix in 1979 and BSD in 1982 chroot, this can be thought of as the first "OS container". Due to the
weak implementation chroot security is quickly broken29 if an adversary or compromised process can gain
superuser or ``root'' access.
30
Skipping forward in time includes FreeBSD Jails,
31
Linux OpenVZ, User ModeLinux (UML), Solaris Zones (in 2005), and AIX Workload Partitions (in 2007 32). While these are also shared-
kernel virtualization systems, this paper is focused on modern, native Linux solutions and will offer minimal
comparisons going forward.
Linux VServers, were introduced around 2001 and represented a leap forward in usability and speed when
paravirtualization was objectively more popular, and hardware support for hypervisors was still weakly sup-
ported or prohibitively expensive. Armed with a basic Linux kernel patchset and some userland tools,
VServers allowed for most, if not all, of what we think of as containers today. This solution broke out different
running applications implemented within instances of Linux distributions into different ``security contexts''.
Despite all the advancements of VServers, kernel namespaces, a topic that is further discussed in Section 3
on page 20, were weakly supported. Many namespaces were still undergoing active development or yet to
be implemented entirely. A lack of cgroups also resulted in difficult performance isolation across differentservers, where existing tools were inadequate to easily manage process groups. In general, security was
nowhere near as complete (or as incomplete, depending on your current perspective of container security).
Jump forward to 2016, where Linux kernel technologies such as namespaces, cgroups, and capabilities
separately and in concert support LXC, Docker, CoreOS Rocket/rkt, Heroku, Joyent, SubgraphOS, RacherOS
and countless other container solutions and PaaS systems. One only needs to view the list of companies
supporting the Open Container Initiative to witness the seriousness of containers. The current push to move
to Microservices as a platform also uses containers as a primary driver and key component. Finally, new and
intriguing efforts to create a hybrid of hardware and OS virtualization, such as Intel's Clear Containers offers
one possible future combining the best of both virtualization strategies.
2.2 Linux Containers: where are they now?
2.2.1 Servers
At a basic level, container-related systems and sandboxes are used in everyday software, even chroot is
still widely used simply due to its simplicity. Many common Open Source daemons contain out of the box
support for a ``chrooted'' environment, such as Apache or Postfix. Some network daemons are almost always
chrooted. If privilegeseparationis enabled in OpenSSH, which is the default, an unprivilegedhelper process
will be chrooted into an empty directory to handle pre-authentication network traffic for each client. 33 The
newest versions of OpenSSH support a ``sandbox'' directive for UsePrivilegeSeparation, this offers ``ad-
ditional protections'' using three different methods for different platforms (systrace for OpenBSD, seatbelt
for OSX, seccomp for Linux and POSIX rlimits (as a fallback and for other platforms). While these solutions
are not ``containers'', many of the security features and goals are shared.
Moving to the containers we normally think of such as LXC, Docker, and CoreOS Rkt, many companies
are planning a massive increase in container use and deployment. These transitions are happening for a
number of reasons, including better economy for (PaaS) providers, enabling better development pipelines,
29This may partially be due to security not being a key design goal, but testing.30By creating a directory, ``chrooting'' into it, then using a directory traversal sequence (e.g. ../ ) to escape the ``outer'' chroot.31Which had their own serious vulnerabilities: CAN-2005-2218, CVE-2007-0166, CVE-2010-2022, and CVE-2014-3001.32http://www.ibm.com/developerworks/aix/library/au-workload/33http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config.5?query=sshd_config
13 | Understanding and Hardening Linux Containers NCC Group
http://www.ibm.com/developerworks/aix/library/au-workload/http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config.5?query=sshd_confighttp://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config.5?query=sshd_confighttp://www.ibm.com/developerworks/aix/library/au-workload/
-
8/18/2019 Understanding and Hardening Linux Containers
14/122
establishing better parity between dev/test environments, adding additional security layers, cutting energy
costs and other reasons. Cloud providers such as Rackspace and Amazon 34 now officially support support
Docker containers.
Google, a major technology leader, has stated ``everything at Google runs in a container'', and Google is
no small operation (reportedly starting and turning over two billion containers each week35
) as everythingfrom search to gmail is packaged and run inside unique containers.36 Although the exact details are sparse,
Google's containerization is likely powered at at least one layer by Linux KVM ,37 cgroups and historically a
variant of LXC called ``lmctfy'' (Let Me Contain Than For You)38 are also assumed to be key components.
It should be noted that recent efforts have been applied to Docker's libcontainer (now part of the Open
Container Initiative), according to the lmctyf README. Recently, Google has also released ``Kubernetes'' 39
for managing containers, based off their Borg platform, as well as cAdvisor for container resource usage
and statistics. To further support containers in their provided cloud environment, the Google App Engine
supports Docker images in managed VMs.40
Traditional hardware virtualization companies, such as VMware, are also quickly ramping up support for
containers. VMware Photon / VMware Cloud supports several Linux Container formats through Photon41
and allows managing containers alongside existing hypervisors and virtual machines 42 through ``ProjectBonneville''.43 Openstack has also added support for containers through their Nova Virt API.44, 45
In parallel to these big names, a number of companies are publicly developing and releasing container
tools, container focused operating systems, are offering hosting services and in some cases powering their
entire infrastructure via containers. This includes (in no particular order) Intel, Google, VMware, Docker,
CoreOS, Amazon, Rackspace Cloud, IBM, Engine Yard, Joyent, Cloud Foundry, Heroku, Sandstorm.io, Red-
Hat OpenShift, eBay, Cloud Foundry, HP (Stackato), StackEngine, OpenStack, DigitalOcean, ClusterHQ,
Spotify, and many more. Countless others which remain unnamed, unpublished or cannot be mentioned
here are deploying containers or using the features that power them in an ad-hoc fashion internally. Heroku,
one of the first major PaaS platforms, has built their business from a container model of minimally executing,
tightly controlled instances called ``Dynos''. According to the Heroku documentation:
``Dynos execute in complete isolation from one another, even when on the same physical
infrastructure. This provides protection from other application processes and system-level processes
consuming all available resources. The dyno manager uses a variety of technologies to enforce
this isolation, most notably LXC for subvirtualized resource and process table isolation, independent
filesystem namespaces, and the pivot_root syscall for filesystem isolation. These technologies
provide security and evenly allocate resources such as CPU and memory in Heroku's multi-tenant
environment.
- Heroku Dyno isolation and security
34Amazon EC2 offers their ECS system. See https://aws.amazon.com/containers/ for more information.35
https://speakerdeck.com/jbeda/containers-at-scale36http://googlecloudplatform.blogspot.com/2014/06/an-update-on-container-support-on-google-cloud-platform.html37Andrew Honig of the cloud security team at Google has stated in presentations ``KVM it is the killer feature''.38https://github.com/google/lmctfy39http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubernetes.html40https://cloud.google.com/compute/docs/containers41https://vmware.github.io/photon/42https://blogs.vmware.com/cto/vmware-containers-containers-without-compromise/43http://venturebeat.com/2015/06/22/vmware-previews-project-bonneville-a-docker-runtime-that-works-with-vsphere/44http://docs.openstack.org/liberty/config-reference/content/lxc.html45https://wiki.openstack.org/wiki/Docker
14 | Understanding and Hardening Linux Containers NCC Group
https://devcenter.heroku.com/articles/dynos#isolation-and-securityhttps://aws.amazon.com/containers/https://speakerdeck.com/jbeda/containers-at-scalehttp://googlecloudplatform.blogspot.com/2014/06/an-update-on-container-support-on-google-cloud-platform.htmlhttp://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdfhttp://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdfhttps://github.com/google/lmctfyhttp://blog.kubernetes.io/2015/04/borg-predecessor-to-kubernetes.htmlhttps://cloud.google.com/compute/docs/containershttps://vmware.github.io/photon/https://blogs.vmware.com/cto/vmware-containers-containers-without-compromise/http://venturebeat.com/2015/06/22/vmware-previews-project-bonneville-a-docker-runtime-that-works-with-vsphere/http://docs.openstack.org/liberty/config-reference/content/lxc.htmlhttps://wiki.openstack.org/wiki/Dockerhttps://wiki.openstack.org/wiki/Dockerhttp://docs.openstack.org/liberty/config-reference/content/lxc.htmlhttp://venturebeat.com/2015/06/22/vmware-previews-project-bonneville-a-docker-runtime-that-works-with-vsphere/https://blogs.vmware.com/cto/vmware-containers-containers-without-compromise/https://vmware.github.io/photon/https://cloud.google.com/compute/docs/containershttp://blog.kubernetes.io/2015/04/borg-predecessor-to-kubernetes.htmlhttps://github.com/google/lmctfyhttp://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdfhttp://googlecloudplatform.blogspot.com/2014/06/an-update-on-container-support-on-google-cloud-platform.htmlhttps://speakerdeck.com/jbeda/containers-at-scalehttps://aws.amazon.com/containers/https://devcenter.heroku.com/articles/dynos#isolation-and-security
-
8/18/2019 Understanding and Hardening Linux Containers
15/122
2.2.2 Clients
Although containers in data centers and servers are where almost all of the focus is (largely due to the ease
with which containers allow `shipping'' software), servers should not be the only focus. As many security
professionals have known for years, attackers have long since switched to targeting clients and workstations.
Advancing the state of Linux sandboxing through application containers, dropping privileges, reducing or
eliminating suid binaries and other isolation mechanisms employed by containers can help improve Linux
application security. Application containers can also easily limit CPU, Disk and Memory for everything from
resource hungry (and highly exploited) web browsers to anonymous liberation technology systems which
can be critical to secure from indirect information leaks.
Unsurprisingly, Google is also a major player for client-side container technologies, although they don't use
actual containers. Both the Chrome OS distribution46 and the Chromium/Google Chrome web browser
heavily utilize the protections mechanisms powering containers, such as kernel Namespaces (Network and
PID), Seccomp-bpf, SUID sandboxing, or in newer versions full user namespaces. 47 Unfortunately for secu-
rity, and somewhat paradoxically, Google Android, one of the most widely deployed Linux distributions (if
you can call it a distribution) is surprisingly missing many of the modern container features provided by the
kernel, apart from the mount Namespace and Mandatory Access Control via SELinux.48
Android still chieflyrelies on Discretionary Access Controls (DAC) aka UNIX permissions and other enforcement via a normal
UID and process isolation based security model.
The recently released high-security Linux distribution SubgraphOS, (currently Alpha) offers application con-
tainers/sandboxing via Oz49 for a number of security sensitive applications. This is the default system along
with a grsecurity patched kernel, Tor based routing, X11 isolation via Xpra, gated outbound connections
via custom firewalls and a host of other features. Additional solutions for Linux containers as well as simple
application sandboxes using a subset of container technologies (namespaces, seccomp-BPF, piviot_root,
capabilities, unshare, etc) are explored and further discussed within Section 11.1 on page 112.
2.3 Prior Art: Linux Container Security, Auditing and Presentations
While Linux Container systems (LXC, Docker, CoreOS Rocket, etc) have undergone fast deployment and de-velopment, security knowledge has lagged behind.50 The number of people focused on container security,
and within those who publish security research for containers seems disproportionately small, given their
advantages, ongoing deployment and demand for security knowledge among companies large and small.
That said, a number of presentations, articles, and papers touch on or explore the various security subjects,
although often at a high level or for a specific container platform.
Included below is a list of compiled resources, almost solely focused on container security. Some in which,
despite their age, served as inspiration for this paper and yet other articles and presentations offer a good
overview of Container security strengths, weaknesses and adoption. If you are looking to brush up before
continuing with this paper, I suggest many of the following resources. However, as with any technology
publication, readers should keep in mind the resources included below may not contain the most up-to-dateinformation, such as User namespace support in Docker v1.10, using seccomp within LXC or exploring new
security features.
46https://www.chromium.org/chromium-os/chromiumos-design-docs/system-hardening47https://chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md#User_namespaces_sandbox.md48This may be due to the somewhat unicast nature of Intents multicast of broadcasts for IPC.49https://github.com/subgraph/oz50http://www.infoworld.com/article/2923852/security/containers-have-arrived-and-no-one-knows-how-to-secure-them.html
15 | Understanding and Hardening Linux Containers NCC Group
https://www.chromium.org/chromium-os/chromiumos-design-docs/system-hardeninghttps://chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md#User_namespaces_sandbox.mdhttps://github.com/subgraph/ozhttp://www.infoworld.com/article/2923852/security/containers-have-arrived-and-no-one-knows-how-to-secure-them.htmlhttp://www.infoworld.com/article/2923852/security/containers-have-arrived-and-no-one-knows-how-to-secure-them.htmlhttps://github.com/subgraph/ozhttps://chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md#User_namespaces_sandbox.mdhttps://www.chromium.org/chromium-os/chromiumos-design-docs/system-hardening
-
8/18/2019 Understanding and Hardening Linux Containers
16/122
2.3.1 Multi-Container
Linux Containers: Future or Fantasy? by Aaron Grattafiori, the author of this paper, was presented at DEF
CON 23. This talk explores the background of containers, how the different elements are secured, some
common flaws in common container systems (LXC, Docker, Rkt), what they provide, general container rec-
ommendations and formed the basis for expanding on much of it within this whitepaper.
Linux Containers (LXC), Docker, and Security by Jérôme Petazzoni, a Docker staff member, offers a basic
overview of some of the challenges faced by containers and a look into some security mechanisms and
potential architectures. Jérôme also has a great overview of Implementing a ``separation of concerns'' with
Docker containers.
Thoughts on interoperable containers by Fabio Kung of Heroku explores the security aspects, among other
features of Linux containers, specifically Docker and interactions with developers.
Linux Container Security by Matthew Garrett brings up some good points on Hypervisors vs containers. A
related discussion on LWN offers additional commentary, including a discussion of Sandstorm.io strategy.
Doger.io by Jay Coles contains an excellent and conscience overview of Linux Containers with a number of
links to other good resources. This site can be a good starting point.
Seven problems of Linux Containers by Kirill Kolyshkin of Parallels, Inc. explores how OpenVZ solved and
explored container problems in the Kernel.
2.3.2 LXC Specific
LXC Security Analysis by Roman Fiedler, Austrian Institute of Technology, offers a recent evaluation of LXC
security which explores several vulnerabilities and underlying risks with LXC, some which apply to Linux
containers in general.
Hard Containers - LXC and GrSecurity by Diego Elio Pettenò explores running LXC with grsecurity patches
enabled.
Secure Linux containers cookbook by IBM offers an overview of security mechanisms and examples for LXC
with SELinux and SMACK Mandatory Access Controls (MAC).
Security of Linux containers in the cloud by Dobrica Pavlinušić of University of Zagreb offers an overview of
LXC security mechanisms and how these can be combined with MAC systems.
2.3.3 Docker Specific
Docker & Security by Florian Barth and Matthias Luft, is a recent presentation that explores the security basics
and past vulnerabilities of Docker. The slides also include some discussion on Microservices, devops vs
security and provides and overview of other Docker services.
Green font, Black background: Docker Security by Example by Diogo Mónica of Docker at Dockercon EU in
2015 explores Docker security in depth and by-example and with many embedded demos.
A Docker Image Walks in to a Notary by Diogo Mónica of Docker at ContainerCamp 2015 covers Docker
Notary and Docker image security.
Vulnerability Exploitation In Docker Container Environments by Anthony Bettini of Flawcheck, was presented
at Blackhat Europe 201551 explores the insecurity and common issues around Docker container images.
51https://www.blackhat.com/docs/eu-15/materials/eu-15-Bettini-Vulnerability-Exploitation-In-Docker-Container-Environments.
pdf
16 | Understanding and Hardening Linux Containers NCC Group
https://www.youtube.com/watch?v=iN6QbszB1R8http://www.slideshare.net/jpetazzo/docker-linux-containers-lxc-and-securityhttp://www.slideshare.net/jpetazzo/implementing-separation-of-concerns-with-docker-and-containershttp://www.slideshare.net/jpetazzo/implementing-separation-of-concerns-with-docker-and-containershttp://www.slideshare.net/jpetazzo/implementing-separation-of-concerns-with-docker-and-containershttp://fabiokung.com/2014/06/11/my-dockercon-2014-talk/https://mjg59.dreamwidth.org/33170.htmlhttps://lwn.net/Articles/617842/http://doger.io/http://www.slideshare.net/kolyshkin/seven-problems-of-linux-containershttp://www.openwall.com/lists/oss-security/2015/07/23/2https://blog.flameeyes.eu/2012/04/hard-containershttp://www.ibm.com/developerworks/linux/library/l-lxc-security/index.htmlhttp://www.slideshare.net/dpavlin/security-of-linux-containers-in-the-cloudhttps://www.ernw.de/download/ERNW_Stocard_Docker-Devops-Security_fbarth-mluft.pdfhttps://www.youtube.com/watch?v=blNIreAq6hchttps://www.youtube.com/watch?v=JvjdfQC8jxMhttps://www.youtube.com/watch?v=77-jaeUKH7chttps://www.blackhat.com/docs/eu-15/materials/eu-15-Bettini-Vulnerability-Exploitation-In-Docker-Container-Environments.pdfhttps://www.blackhat.com/docs/eu-15/materials/eu-15-Bettini-Vulnerability-Exploitation-In-Docker-Container-Environments.pdfhttps://www.blackhat.com/docs/eu-15/materials/eu-15-Bettini-Vulnerability-Exploitation-In-Docker-Container-Environments.pdfhttps://www.blackhat.com/docs/eu-15/materials/eu-15-Bettini-Vulnerability-Exploitation-In-Docker-Container-Environments.pdfhttps://www.youtube.com/watch?v=77-jaeUKH7chttps://www.youtube.com/watch?v=JvjdfQC8jxMhttps://www.youtube.com/watch?v=blNIreAq6hchttps://www.ernw.de/download/ERNW_Stocard_Docker-Devops-Security_fbarth-mluft.pdfhttp://www.slideshare.net/dpavlin/security-of-linux-containers-in-the-cloudhttp://www.ibm.com/developerworks/linux/library/l-lxc-security/index.htmlhttps://blog.flameeyes.eu/2012/04/hard-containershttp://www.openwall.com/lists/oss-security/2015/07/23/2http://www.slideshare.net/kolyshkin/seven-problems-of-linux-containershttp://doger.io/https://lwn.net/Articles/617842/https://mjg59.dreamwidth.org/33170.htmlhttp://fabiokung.com/2014/06/11/my-dockercon-2014-talk/http://www.slideshare.net/jpetazzo/implementing-separation-of-concerns-with-docker-and-containershttp://www.slideshare.net/jpetazzo/implementing-separation-of-concerns-with-docker-and-containershttp://www.slideshare.net/jpetazzo/docker-linux-containers-lxc-and-securityhttps://www.youtube.com/watch?v=iN6QbszB1R8
-
8/18/2019 Understanding and Hardening Linux Containers
17/122
Docker, Docker Give Me The News: I Got A Bad Case Of Securing You was presented at DEF CON 23
by David Mortman of Dell and explores and touches on the security costs and potential risks of deploying
Docker and other container systems.
Are Docker containers really secure? by Dan Walsh, of RHEL and SELinux fame, is semi-involved with Docker
development, and can often found strongly and consistently advocating for SELinux.52
Docker Security: Who can we trust, what should we verify? by Nils Magnus at goto; conference explores the
basics of Docker security, attack surfaces and offers some good basic recommendations.
Least-privilege Microservices by Diogo Mònica and Nathan McCauley, both Docker security leads, provides
a basic overview of how how Docker Containers can support least-privilege Microservices.
Introduction to Container Security by an unknown author is a short Docker whitepaper, released in May of
2015 that includes discusses the available security features for Docker and discusses some general risks and
best practices when using it.
Security Properties of Containers Managed by Docker is a non-free Gartner research paper by Joerg Fritsch.
This paper primarily asks the question, can your Docker containers actually contain. .53 Apparently Dockerfaired decently enough, according to The Register and InformationWeek although many of the security
complaints revolve around security maturity and administration, not fundamental risks of OS virtualization
and attack surface analysis.
Analysis of Docker Security by Thanh Bui of the Aalto University School of Science, this document from late
2014 covers a concise overview of Docker isolation mechanisms, security features of an unknown but older
Docker version.
Docker, DevOps, Security by Chris Swan of cohesiveFT explores how security, containers and DevOps meet.
CIS Docker 1.6 Benchmark v1.0.0 by Pravin Goyal of VMware, Inc. with contributions from various container
companies, including Docker itself. This includes a comprehensive, if extremely dry, best practices security
document. Some of the best practices can also be explored and tested via the ``Docker Bench'' tool,54
created by Docker's Diogo Mónica.
Docker and SELinux by Daniel Walsh is a basic overview of Docker security issues and recommendations,
although his opinions may not be widely shared. (Takes half of the talk to explain SELinux basics)
Docker Security by Nathan McCauley and Diogo Mónica of Docker is a gentle overview video relating to
security by the lead Docker security team members.
Docker Security: Are Your Containers Tightly Secured To The Ship? and Docker Security: Secure container
deployment on Linux by Michael Boelen, CISOfy, covers some basic risks, mechanisms and recommenda-
tions for Docker security.
Container security: Kernel internals by an unknown author offers a kernel viewpoint of Container security,
exploring from the hardware ``up''. This presentation has some good information and overall perspective.
Creating Containers by Michael Crosby of Docker is an excellent and in-depth exploration of Linux Contain-
ers, while there isn't a focus on security, this blog post (and the others on his website) are worth reviewing.
52https://lwn.net/Articles/515034/53http://blogs.gartner.com/joerg-fritsch/can-you-make-your-containers-contain/54https://github.com/docker/docker-bench-security
17 | Understanding and Hardening Linux Containers NCC Group
https://www.youtube.com/watch?v=7ouzigqFUWUhttps://opensource.com/business/14/7/docker-security-selinuxhttp://gotocon.com/dl/goto-berlin-2015/slides/NilsMagnus_InsightsInContainerSecurity.pdfhttp://www.slideshare.net/Docker/docker-security-for-slidehttps://d3oypxn00j2a10.cloudfront.net/assets/img/Docker%20Security/WP_Intro_to_container_security_03.20.2015.pdfhttps://www.gartner.com/doc/2956826/security-properties-containers-managed-dockerhttp://www.theregister.co.uk/2015/01/12/docker_security_immature_but_not_scary_says_gartner/http://www.informationweek.com/cloud/infrastructure-as-a-service/gartner-gives-thumbs-up-to-docker-security/d/d-id/1318612http://arxiv.org/pdf/1501.02967.pdfhttp://www.slideshare.net/CohesiveNetworks/cohesive-ft-dockermeetupchicago20140717securityhttps://benchmarks.cisecurity.org/downloads/show-single/index.cfm?file=docker16.100https://www.youtube.com/watch?v=zWGFqMuEHdwhttps://www.youtube.com/watch?v=8mUm0x1uy7chttp://www.slideshare.net/MichaelBoelen/docker-security-are-your-containers-tightly-secured-to-the-shiphttp://www.slideshare.net/MichaelBoelen/docker-security-secure-container-deployment-on-linuxhttp://www.slideshare.net/MichaelBoelen/docker-security-secure-container-deployment-on-linuxhttp://www.slideshare.net/smart_bit/docker-and-kernel-security?http://crosbymichael.com/creating-containers-part-1.htmlhttps://lwn.net/Articles/515034/http://blogs.gartner.com/joerg-fritsch/can-you-make-your-containers-contain/https://github.com/docker/docker-bench-securityhttps://github.com/docker/docker-bench-securityhttp://blogs.gartner.com/joerg-fritsch/can-you-make-your-containers-contain/https://lwn.net/Articles/515034/http://crosbymichael.com/creating-containers-part-1.htmlhttp://www.slideshare.net/smart_bit/docker-and-kernel-security?http://www.slideshare.net/MichaelBoelen/docker-security-secure-container-deployment-on-linuxhttp://www.slideshare.net/MichaelBoelen/docker-security-secure-container-deployment-on-linuxhttp://www.slideshare.net/MichaelBoelen/docker-security-are-your-containers-tightly-secured-to-the-shiphttps://www.youtube.com/watch?v=8mUm0x1uy7chttps://www.youtube.com/watch?v=zWGFqMuEHdwhttps://benchmarks.cisecurity.org/downloads/show-single/index.cfm?file=docker16.100http://www.slideshare.net/CohesiveNetworks/cohesive-ft-dockermeetupchicago20140717securityhttp://arxiv.org/pdf/1501.02967.pdfhttp://www.informationweek.com/cloud/infrastructure-as-a-service/gartner-gives-thumbs-up-to-docker-security/d/d-id/1318612http://www.theregister.co.uk/2015/01/12/docker_security_immature_but_not_scary_says_gartner/https://www.gartner.com/doc/2956826/security-properties-containers-managed-dockerhttps://d3oypxn00j2a10.cloudfront.net/assets/img/Docker%20Security/WP_Intro_to_container_security_03.20.2015.pdfhttp://www.slideshare.net/Docker/docker-security-for-slidehttp://gotocon.com/dl/goto-berlin-2015/slides/NilsMagnus_InsightsInContainerSecurity.pdfhttps://opensource.com/business/14/7/docker-security-selinuxhttps://www.youtube.com/watch?v=7ouzigqFUWU
-
8/18/2019 Understanding and Hardening Linux Containers
18/122
On the Security of Containers by Eric Windisch formerly of Docker offers a good overview behind why
lightweight application containers offer additional security and how Virtual Machines can augment where
trust is weak or the risk is high. ``Containers are not contradictory to other, existing best-practices''.
2.4 TL;DR Linux Containers
As we will explore within this paper, Linux containers (LXC, Docker, Rkt, etc) are typically comprised of five
major components:
1. Kernel namespaces are the major building block of Linux containers, which isolate the applications
within different ``userspaces'' such as network, processes, users themselves and the file system. Further
information, examples and exploration is included within Section 3 on page 20.
2. Control Groups also known as cgroups are essentially ulimit on steroids, which limit various host hard-
ware resources. This currently includes CPU count and usage, disk performance, memory, and other
process limits. Further information can be found within Section 4.1 on page 27.
3. Root Capabilities help enforce namespaces in so-called ``privileged'' containers by reducing the power
of root, in some cases to no powerat all. Furtherinformation, such as the background, implementation,use and defaults for major container platforms are included within Section 5.1 on page 30.
4. Pivot_root is a syscall to ``pivot'' into the new container environment, by changing the root file system.
Although the use and implementation of pivot_root(2) and related initialization steps of the con-
tainer's init is not explicitly discussedwithin this paper, ``pivoting'' correctly can be crucial for security.55
5. Manditory Access Controls (MAC) such as AppArmor and SELinux are not required for creating contain-
ers, but are oftena key element to their security. MAC helps enforce thesecurity controls implemented
by other container features, adding defense in depth and general platform security for any permission
level within the container, privileged user or otherwise. Further information can be found within 8.1.6
on page 69.
Container security threats, largely aimed at escaping confinement can originate from many sources. Attacks
could originate from a compromised container, a maliciously uploaded container, the applications within a
container or attacks against the local network from within the container. The various threats to containers is
included and explored in Section 7 on page 49.
To counter the identified threats, several recent Linux security advancements (not solely limited to contain-
ers), such as the User namespace and Seccomp are discussed in Section 8.1 on page 66 and Section 8.3
on page 74 respectively. Not to be left out, the historically under-configured, and I would argue under-
appreciated, Mandatory Access Controls (MAC) systems are also covered 8.1.6 on page 69. These features
offer a great reduction in attack surface and defense in depth within supported container platforms.
After understanding the new security features, the historical and current threats, potential risks (and available
security features) I have included a brief overview of each of the major container platforms explored in this
paper (LXC, Docker, Rkt). This may help understand and evaluate the motivations, project priorities, security
threats (past and present) and available security options. This overview, background and security analysis of
each platform starts in Section 9 on page 81. To further support this information, a table of secure defaults
and support options can be found in Section 9.13 on page 96.
The cursory exploration of the current security strengths and weaknesses in Section 9 referenced above, is
largely to set the stage for this paper's recommendations section. Hardening your container deployment
55File descriptors must be closed, ptrace(2) must be limited, and confused deputy attacks understood.
18 | Understanding and Hardening Linux Containers NCC Group
https://medium.com/@ewindisch/on-the-security-of-containers-2c60ffe25a9ehttps://en.wikipedia.org/wiki/Confused_deputy_problemhttps://en.wikipedia.org/wiki/Confused_deputy_problemhttps://medium.com/@ewindisch/on-the-security-of-containers-2c60ffe25a9e
-
8/18/2019 Understanding and Hardening Linux Containers
19/122
and configuration against many of the earlier identified threats is discussed in Section 10 on page 97. The
recommendations are grouped this section first as general Linux and container platform agnostic terms as
well as specific recommendations section for each of the three platforms covered within this paper: LXC
in Section