Couchbase Containers withBare Metal Performance

CTObryan@joyent.com

Bryan Cantrill

@bcantrill

Elastic infrastructure, circa Dot Com Boom

• In the late 1990s, the only way to meaningfully scale a database was up — and the physical infrastructure had to scale with it

• This was excruciatingly expensive — and became a non-starter in the post-apocalyptic nuclear winter of the early 2000s...

Elastic infrastructure, circa Dot Com Bust

• The rise of rack-and-stack commodity servers brought with it new distributed software architectures like memcached that were designed to scale across many machines

• The rise of these architectures afforded new operational possibilities: if the computer itself is a commodity, why buy it at all? Why not rent from someone who runs it cheaper and better?

• From the perspective of compute providers, economies of scale could only be realized if hardware is shared across tenants...

• Multi-tenancy demands virtualization, but where in the stack to virtualize?

Hardware-level virtualization?

• The historical answer to virtualization — since the 1960s — has been to virtualize the hardware:

• A virtual machine is presented upon which each tenant runs an operating system that they choose (and must manage)

• There are as many operating systems on a machine as tenants!

• Can run entire legacy stacks unmodified...

• ...but operating systems are heavy and don’t play well with others with respect to resources like DRAM, CPU, I/O devices, etc.

• Hardware-level virtualization limits tenancy and performance!

Platform-level virtualization?

• Virtualizing at the application platform layer addresses the tenancy challenges of hardware virtualization, and presents a much more nimble (& developer friendly!) abstraction...

• ...but at the cost of dictating abstraction to the developer

• This is the “Google App Engine” problem: developers are in a straightjacket where toy programs are easy — but sophisticated applications are impossible

• Virtualizing at the application platform layer poses many other challenges with respect to security, containment, etc.

OS-level virtualization!

• Virtualizing at the operating system hits a sweet spot:• A single operating system (i.e. a single kernel) allows for efficient use of

hardware resources, maximizing tenancy and performance

• Disjoint instances are securely compartmentalized by the operating system

• Gives tenants what appears to be a virtual machine (albeit a very fast one) on which to run higher-level software: PaaS ease with IaaS generality

• Applications run directly on the hardware!

• Model was pioneered by FreeBSD jails and taken to their logical extreme by Solaris zones — and then aped by Linux containers

OS-level virtualization at Joyent

• Joyent runs OS containers in the cloud via SmartOS — and we have run containers in multi-tenant production since ~2006

• Adding support for hardware-based virtualization circa 2011 strengthened our resolve with respect to OS-based virtualization

• We emphasized their operational characteristics — performance, elasticity, tenancy — and for many years, we were a lone voice...

Containers as PaaS foundation?

• Some saw the power of OS containers to facilitate up-stack platform-as-a-service abstractions

• For example, dotCloud — a platform-as-a-service provider — built their PaaS on OS containers

• Struggling as a PaaS, dotCloud pivoted — and open sourced their container-based orchestration layer...

...and Docker was born

Docker revolution

• Docker has used the rapid provisioning + shared underlying filesystem of containers to allow developers to think operationally

• Developers can encode deployment procedures via an image

• Images can be reliably and reproducibly deployed as a container

• Images can be quickly deployed — and re-deployed

• Docker will do to apt what apt did to tar

Broader container revolution

• The Docker model has pointed to the future of containers

• Docker’s challenges today are largely operational: network virtualization, persistence, security, etc.

• Security concerns are not due to Docker per se, but rather to the architectural limitations of the Linux “container” substrate

• For multi-tenancy, state-of-the-art for Docker containers is to run in hardware virtual machines (!!)

• Deploying OS containers in hardware virtual machines negates their economic advantage!

Container-native infrastructure?

• SmartOS has been container-native since its inception — and running in multi-tenant, internet-facing production for many years

• Can we achieve an ideal world that combines the development model of Docker with the container-native model of SmartOS?

• This would be the best of all worlds: agility of Docker coupled with production-proven security and on-the-metal performance of SmartOS containers

• But there are some obvious obstacles...

Docker + SmartOS: Linux binaries?

• First (obvious) problem: while it has been designed to be cross-platform, Docker is Linux-centric — and the encyclopedia of Docker images will likely forever remain Linux binaries

• SmartOS is Unix — but it isn’t Linux…

• Fortunately, Linux itself is really “just” the kernel — which only has one interface: the system call table

• We resurrected (and finished) a Sun technology for Linux system call emulation, LX-branded zones, the technical details of which are beyond the scope of this presentation...

LX-branded zones: tl;dr

LX-branded zones: tl;dr, cont.

Docker + SmartOS: Provisioning?

• With the binary problem being tackled, focus turned to the mechanics of integrating Docker with SmartOS provisioning

• Provisioning a SmartOS zone operates via the global zone that represents the control plane of the machine

• docker is a single binary that functions as both client and server — and with too much surface area to run in the global zone, especially for a public cloud

• docker has also embedded Go- and Linux-isms that we did not want in the global zone; we needed to find a different approach...

Docker Remote API

• While docker is a single binary that can run on the client or the server, it does not run in both at once…

• docker (the client) communicates with docker (the server) via the Docker Remote API

• The Docker Remote API is expressive, modern and robust (i.e. versioned), allowing for docker to communicate with Docker backends that aren’t docker

• The clear approach was therefore to implement a Docker Remote API endpoint for SmartDataCenter, our (open source!) orchestration software for SmartOS

Triton: Docker + SmartOS

• In March, we launched Triton, which combines SmartOS and SmartDataCenter with our Docker Remote API endpoint

• With Triton, the notion of a Docker host is virtualized: to the Docker client, the datacenter is a large Docker host

• All of the components to Triton are open source: you can download and install SmartDataCenter and run it yourself

• Triton is currently in early access at Joyent — and moving into general availability in Q2CY2015

Container landscape

• It is becoming broadly clear that containers are the future of application development and (especially) deployment

• But the upstack ramifications are entirely unclear — there are many rival frameworks for service discovery, deployment, etc.

• The rival frameworks are all open source:

• Unlikely to be winner-take-all

• Productive mutation is not just possible but highly likely

• Triton takes a deliberately modular approach: the container as general-purpose foundation, not prescriptive framework

Containers and Couchbase

• Couchbase is particularly appropriate for containers: its scale-out architecture demands elastic infrastructure — and its use cases demand on-the-metal performance

• But hardware-virtualized Docker hosts undermine the efficacy of containers — and force an allocation-oriented disposition instead of allowing a consumption-oriented one

• The Triton model represents Couchbase containers without compromise: like Couchbase itself, the infrastructure can grow as needed — while still delivering bare-metal performance!

Future of containers

• For nearly a decade, we have believed that OS-virtualized containers represent the future of computing — and with the rise of microservices + Docker, this is no longer controversial

• But to achieve the full promise of containers, they must run directly on-the-metal — multi-tenant security is a constraint!

• The virtual machine is a vestigial abstraction; we must reject container-based infrastructure that implicitly assumes it

• Triton represents our belief that containers needn’t compromise: multi-tenant security, operational elasticity and on-the-metal performance!

Thank you!

• @joshwilsdon, @trentmick, @cachafla, @orlandov, @fredfkuo and @notmatt and Todd Whiteman for their work on sdc-docker

• Jerry Jelinek, @pfmooney, @jmclulow, @rmustacc and @jperkin for their work on LX branded zones

• The countless engineers who have worked on or with SmartOS because they believed in OS-based virtualization before the rest of the world figured it out