uds 2012 xen
TRANSCRIPT
Xen in Ubuntu Raring
PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux
Agenda
Brief overview of Xen
What's new in 4.2
What's coming in 4.3
What is a great Ubuntu Xen experience?
Integration issuesQemu
Libvirt
Other improvements
Dom 0:In a typical Xen set-up Dom0 contains a smorgasboard of functionality:System boot
Device emulation & multiplexing
Administrative toolstack
Drivers (e.g. Storage & Network)
Etc.
LARGE TCB BUT, Smaller as in a Type 2 hypervisor
Driver/Stub/Service Domains: also known as Disaggregation
Xen Overview
PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux
Basic Xen Concepts
Xen HypervisorControl domain
(dom0)Host HWVMnVM1VM0Guest OSand Apps
XL, XM (deprecated)
MemoryCPUsI/O
Scheduler, MMU
One or more
driver, stub or
service domainsControl Domain aka Dom0Dom0 kernel with drivers
Xen Management Toolstack
Trusted Computing Base
Guest DomainsYour apps
E.g. your cloud management stack
Driver/Stub/Service Domain(s)A driver, device model or control service in a box
De-privileged and isolated
Lifetime: start, stop, kill
Dom0 KernelDom 0:In a typical Xen set-up Dom0 contains a smorgasboard of functionality:System boot
Device emulation & multiplexing
Administrative toolstack
Drivers (e.g. Storage & Network)
Etc.
LARGE TCB BUT, Smaller as in a Type 2 hypervisor
Driver/Stub/Service Domains: also known as Disaggregation
Xen HypervisorControl domain
(dom0)Host HWGuest VMnAppsMemoryCPUsI/O
Linux PV guests have limitations:limited set of virtual
hardware
AdvantagesFast
Works on any system
(even without virt extensions)
Driver DomainsSecurity
Isolation
Reliability and Robustness
HW DriversPV Back EndsPV Front Ends
Driver Domain
e.g. Disk
Network
HW DriverPV Back EndDom0 Kernel**) Can be MiniOS
PV Domains & Driver DomainsGuest OS
HVM
Xen HypervisorDom0Host HWGuest VMnDisadvantagesSlower than PV
due to Emulation
(mainly I/O devices)
AdvantagesInstall the same way as native Linux
Stub DomainsSecurity
Isolation
Reliability and Robustness
Device ModelHVM & Stub Domains
IO Emulation IO EventVMEXITStubdomnDevice ModelMini OSGuest VMn
IO Emulation IO EventVMEXITDevice Model emulated in QEMUModels for newer devices are much faster, but for now PV is even faster
Xen 4.2 and 4.3
PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux
Xen 4.2 Release
xl default toostacklibxl stable interface
xend deprecated
Scalability4095 host CPUs, 5TiB RAM
cpupools for more flexible partitioning
Scheduler, NUMA improvements
Security improvements
http://wiki.xen.org/wiki/Xen_4.2_Feature_List
Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)
Who writes Xen?
Xen 4.3 Release
Slated for 17 June 2013
ARM server port
PVH mode: PV with some HVM extensions
NUMA
Numerous I/O path improvements
Qemu upsteam
Roadmap: http://wiki.xen.org/wiki/Xen_Roadmap/4.3
Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)
Xen and Ubuntu
PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux
Integration: qemu
Debian developmentsQemu-dm: Xen qemu fork
4.2: qemu-upstream missing pass-through, migration
4.3: qemu-upstream will be default
Debian has already removed qemu-dm
SolutionsQemu-xen-dm package
Backport features
Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)
Integration: libvirt
Libvirt 0.10.2 has bindings for xend, 4.1 libxl
4.2 libxl bindings incompatible, xend deprecated
OptionsStick with 4.1 (bad)
Xen 4.2, but only xend bindings
Back-port 4.2 libxl support
Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)
What does a great Xen on Ubuntu look like?
As a Xen hostEasy to install, set up
Reliable, good performance
Switching between Xen and non-Xen modes
Good integration with libvirt, &c
As a Xen guestInstallation
Reliable, good performance
Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)
Potential improvements
Xen Host option in installer
Make configuring Grub2 easier
Switching between Xen / non-Xen
Getty for PV console (hvc0)
Xen-tools
Guest installation?
Keeping an eye on linux-xen perf tweaks to backport
Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)
Backup slides
Architecture Considerations
Type 1: Bare metal HypervisorA pure Hypervisor that runs directly on the hardware and hosts Guest OSs.
Type 2: OS HostedA Hypervisor that runs within a Host OS and
hosts Guest OSs inside of it, using the host OS services to provide
the virtual environment.Provides partition isolation +
reliability,
higher securityLow cost, no additional drivers
Ease of use & installationHost HWMemoryCPUsI/O
Host HWMemoryCPUsI/O
HypervisorSchedulerMMU
Device Drivers/ModelsVMnVM1VM0Guest OSand AppsHost OS
Device Drivers
Ring-0 VM Monitor
Kernel VMnVM1VM0Guest OSand Apps
User
AppsUser-level VMM
Device Models
At this point I want to make a quick detour into the different hypervisor architectures from a viewpoint of security.
Lets look at type 1 hypervisor:Basically a very simple architecture, where the Hypervisor replaces the kernel
The architecture is significantly simpler that a Type 2 hypervisor, because it does not need to provide rich process semantics, like user, filesystems, etc.
BUT: the trade-off is that all the device drivers need to be rewritten for each hardware platform
Type 2 is hosted- The hypervisor is just a driver that typically works with user-level monitor.HW access is intercepted by the ring 0- VM monitor passed to the User level Virtual Monitor, which passes requests to the kernel
Re-use of device drivers is traded off against security and a large trusted computing base (green)
Xen: Type 1 with a Twist
Control domain
(dom0)Host HWVMnVM1VM0Guest OSand AppsMemoryCPUsI/O
Thin hypervisor
Functionality moved to Dom0
Using Linux PVOPSTake full advantage of PV
PV on HVM
No additional device drivers (Linux 3.x dom0)
In other wordslow cost (drivers)
Ease of use & Installation
Isolation & Security
HypervisorSchedulerMMU
Drivers
Device Models
Linux, BSD, etc.
XSM
Dom 0:In a typical Xen set-up Dom0 contains a smorgasboard of functionality:System boot
Device emulation & multiplexing
Administrative toolstack
Drivers (e.g. Storage & Network)
Etc.
LARGE TCB BUT, Smaller as in a Type 2 hypervisor
10 Freescale i.MX53 Loco Quickstart boards
Running Debian "armhf" with a mainline 3.2.0 kernel
Speed up development of Xen for Cortex A15
(avoid cross compilation)
A bit of fun: our ARM Build Farm
Xen 4.2 Release
Security: Intel Supervisor Mode Execution Protection, XSM / Flask improvements
Scalability: increased VM density for VDI use-cases, up to 256 Host CPUs for 64 bit HV , Multiple PCI segment support, prefer oxenstored
Performance: PCI pass-through for Linux Guests, AMD SVM DecodeAssist support, Remus memory image compression
EFI support
Libvchan cross domain comms in Xen mainline
XL improvements, XEND is formally deprecated
Documentation improvements (e.g. man pages)
Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit the title text formatClick to edit Master title style
Click to edit the title text formatClick to edit Master title style
Click to edit the title text formatClick to edit Master title style
Click to edit the title text formatClick to edit Master title style
Click to edit the title text formatClick to edit Master title style
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit the title text formatClick to edit Master title style
Click to edit the title text formatClick to edit Master title style
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level
Seventh Outline LevelClick to edit Master text styles
Second level
Third level
Fourth level
Fifth level