mirage: extreme specialization of cloud appliances
DESCRIPTION
Public compute clouds provide a flexible platform to host applications as a set of appliances, e.g., web servers or databases. Each appliance usually contains an OS kernel and userspace processes, within which applications access resources via APIs such as POSIX. The flexible architecture of the cloud comes at a cost: the addition of another layer in the already complex software stack. This reduces performance and increases the size of the trusted computing base. Our new Mirage operating system proposes a radically different way of building these appliances. Mirage supports the progressive specialisation of functional language (OCaml) application source code, and gradually replaces traditional OS components with type-safe libraries. This ultimately results in “unikernels”: sealed, fixed-purpose images that run directly on the hypervisor without an intervening guest OS such as Lin ux. Developers no longer need to become sysadmins, expert in the configuration of all manner of system components, to use cloud resources. At the same time, they can develop their code using their usual tools, only making the final push to the cloud once they are satisfied their code works. As they explicitly link in components that would normally be provided by the host OS, the resulting unikernels are also highly compact: facilities that are not used are simply not included in the resulting unikernel. For example, the self-hosting Mirage web server image is less than a megabyte in size! We will describe the architecture of Mirage in the talk, show some code examples, and interesting benchmark results that compare the performance of our unikernels to traditional applications such as Apache, BIND and OpenSSH.TRANSCRIPT
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 1/36
Anil Madhavapeddy University of Cambridge Richard Mortier University of Nottingham
Dave Scott Citrix Systems R&D
MIRAGE: EXTREME SPECIALISATION OF
CLOUD APPLIANCES
@avsm@mort___
@mugofsouphttp://openmirage.org
http://decks.openmirage.org/oscon13/http://www.youtube.com/watch?v=2Mx8Bd5JYyo
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 2/36
THE CLOUD THREAT MODEL
InternetSAN/NASTenantsSDN
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 3/36
THE CLOUD THREAT MODEL
Type-safety in the
application layer defeats
several external threats.
Type-safety in the kernel
will make all external I/O
safe, but at what cost?
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 4/36
THE CLOUD THREAT MODEL
Most attacks never come fromwithin a guest, only from externaltraffic. So why all these privilegechecks?
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 5/36
KEY DESIGN INSIGHTS
The hypervisor gifts us a stablehardware interface. Cures thecurse of library operating systems!
Protocol-level compatibilitybetween special-purposecloud appliances, e.g. awebserver, a database VM.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 6/36
UNIKERNELS!
Virtual machines are UNIX
processes “done right” on the cloud.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 7/36
CONTRIBUTIONS
The unikernel approach to building single-purpose appliances
Library OS + high level programming interface
Single-address space layout
Evaluation of these techniques using a functional programming
language (OCaml)
Benefits of type-safety need not damage performance
Static typing + modules = high level manipulation
Language extensions for systems programming in OCaml
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 8/36
CURRENT VIRTUAL APPLIANCES
Compiler has to stop at userspace.Every level has a different API, calling convention, and privilegerequirements.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 9/36
SPECIALISED VIRTUAL APPLIANCES
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 10/36
SPECIALISED VIRTUAL APPLIANCES
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 11/36
SPECIALISED VIRTUAL APPLIANCES
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 12/36
PROGRESSIVE SPECIALISATION
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 13/36
MICROBENCHMARKS!
Unikernel appliances are:
Smaller, both in kLoC & image size
Boot faster, at packet round-trip time timescalesHigh performance
Type-safe
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 14/36
APPLIANCE IMAGE SIZE
Appliance StandardBuild
Dead CodeElimination
DNS 0.449 MB 0.184 MB
Web Server 0.674 MB 0.172 MB
Openflow learning switch 0.393 MB 0.164 MB
Openflow controller 0.392 MB 0.168 MB
All configuration and data compiled into the image by the toolchain.
Live migration is easy and fun :-)
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 15/36
BOOT TIME
Unikernels are compact enough to boot and respond to network traffic inreal-time.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 16/36
HOW LARGE IS LARGE?
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 17/36
EVENT DRIVEN CO-THREADS
Garbage collected heap management is more efficient in a singleaddress-space environment. Thread latency can be reduced byeliminating multiple levels of scheduling.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 18/36
THREAD SCALING SINGLE INSTANCE
Threads are heap allocated values, so benefit from the faster garbagecollection cycle in the Mirage Xen version, and the scheduler can beoverridden by application-specific needs.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 19/36
BLOCK STORAGE
Additionally, reading from a Mirage NAS-like storage VM achieves942MiB/s buffered, and 1.8GiB/s unbuffered.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 20/36
TECHNIQUES
Several implementation techniques give rise to these benefits:Simplified memory managementZero-copy IO buffer managementHypervisor security extension for VM sealing (W^X)
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 21/36
MEMORY MANAGEMENT SPECIALISED!
Compiled native source code and runtime
statically linked with random start offset.
IO memory is mapped into
a reserved area and can be
distinguished.
OCaml heap is contiguous,
with simpler write barriers
as a result.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 22/36
BUFFER MANAGEMENT ZERO-COPY IO
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 23/36
OPTIONAL VM SEALING
Single address-space and no dynamic loading
W^X address space
Address offsets are randomized at compile-time
Dropping page table privileges:
Added freeze hypercall called just before app starts
Subsequent page table updates are rejected by Xen
Exception for I/O mappings if they are non-exec and do not
modify any existing mappings
Very easy in unikernels due to focus on compile-time specialisationinstead of run-time complexity
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 24/36
MACROBENCHMARKS
We have implemented several larger appliances.
We discuss deens, our DNS server in detail here.
We also have:
a simple webserver,
an OpenFlow Switch, and
an OpenFlow Controller.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 25/36
DNS SERVER CODE
let main () = lwt zones = read key "zones" "zone.db" in Net.Manager.bind (fun mgr dev -> let src = `any_addr, 53 in Dns.Server.listen dev src zones )
Cooperative threads as
functions Statically evaluatedconfiguration
Functional callbacks
Libraries directly linkto network stack
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 26/36
DNS SERVER PERFORMANCE BASELINE
Baseline figures, running Bind (standard) and NSD (high performance).NSD achieves around 70 kreqs/s.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 27/36
DNS SERVER PERFORMANCE MIRAGE
Comparing against Mirage appliance, with and without simplememoisation. This algorithmic optimisation added just 6 lines of code.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 28/36
DNS SERVER PERFORMANCE C/MINIOS
A rudimentary C-based appliance linking NSD directly against MiniOS.
Poor user-space library performance vastly outweighs language effects.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 29/36
SCALING VIA PARALLEL INSTANCES
Request throughput for serving a simple static page using Apache onLinux vs. a Mirage appliance. Rather than pay the cost of interlocking forthread-level parallelism, we scale by running many instances of theMirage appliance.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 30/36
OPENFLOW CONTROLLER
OpenFlow controller is competitive with NOX (C++), but much higher
level. Applications can link directly against the switch to route their data.
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 31/36
SUMMARY
OCaml is the baseline language for all new code
Our C runtime is small, and getting smaller
Is fully event-driven and non-preemptive
Rewriting protocols wasn’t that hard
Not necessarily the best research strategy though
But an extremely useful learning experience
Tech transfer is vital
Unikernels fit perfectly on the cloud
Internet protocol building blocks
Seamless interop with legacy code through VMs
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 32/36
PURE OCAML CODE
Device driversnetfront, blkfrontxenstore
Networking/CommunicationIP/TCP/UDP/DHCP/DNS(SEC) (v4)HTTP, SSH, OpenFlow (controller/switch)vchan IPC
StorageNFS, FAT32
distributed k/v store9P :-)
GITHUB.COM/MIRAGE
http://arakoon.org
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 33/36
MIRAGE ONLINE
Website,
Code,
O'Reilly OCaml book,
OPAM package manager
Allows constraints to be applied to package installation
Very useful for managing assembly of the many small OCaml
modules that construct a Mirage appliance
http://www.openmirage.orghttp://github.com/mirage
http://realworldocaml.org
http://opam.ocamlpro.com
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 34/36
KEY RESEARCH DIRECTIONS
Interoperability — with billions of VMs out thereA unikernel per-language?
Interconnect strategies? Heap sharing?
Formal method integration easier or harder?
Coordination — planetary scale computersResources are highly elastic now.
How to coordinate a million microkernels?
“Warehouse Scale Computing”
Library Applications — where are they?Irminsule, a git-like functional distributed database
Beanstalk, a self-scaling web server
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 35/36
WE'RE HIRING! MUST ENJOY OPEN SOURCE!
OCamlLabs, Cambridge, UK
Real world functional programming with OCaml
Need compiler hackers, protocol heads, PL/type theory systems
Networks-as-a-Service, University of Nottingham, UK
(2 posts, 3 years, deadline Friday August 2nd)
Joint with University of Cambridge & Imperial College
Network virtualisation for millions of microkernel apps
Need Mirage hackers, network protocol experts, graph theorists
http://www.cl.cam.ac.uk/projects/ocamllabs
http://bit.ly/13sBjjC
7/31/13 Mirage: Extreme Specialisation of Cloud Appliances
file://localhost/Users/mort/research/projects/mirage/src/v2/mirage-decks/files/slides/oscon13/complete.html#/ 36/36
Thanks! Questions?
(...and please rate the talk!)
HTTP://TWITTER.COM/MORT___