course linux system administration · virtual machines typically aren’t give direct access to...

Linux system administration course

Spring 2019: 6th slideset: Virtualization, configuration management

1

Virtual machines

Spring 2019: 6th slideset: Virtualization, configuration management

2

Linux system administration, 6th slide set © Jani Jaakkola 2019

What is virtualization● Hardware virtualization or platform virtualization refers to the creation of a

virtual machine (VM) that acts like a real computer with an operating system. Software executed on these virtual machines is separated from the underlying hardware resources. [wikipedia]

○ Enables Linux hosts to run Windows OS in a virtual machine and vice versa● The software that implements the environment for running virtual machines is

called the hypervisor.● The physical machine where the hypervisor software is running is called the

host machine.● The virtual machines instances running on the host are called guests.


Virtualization● Virtualization is an old invention: IBM did it with mainframes 1964.● During last 20 years multi core CPUs made computers powerful enough that a

single physical PC could run multiple operating system instances.● In year 1999 VMWare figured out how to virtualize PC hardware so that

multiple operating system instances could share the same hardware and run mostly native code.

○ This involved a large performance penalty: Intel i386 architecture did not natively support this and many CPU instructions needed to be emulated, including the MMU.

● Later both AMD and Intel added virtualization features to CPUs to enable efficient virtualization.


Why virtualize?● Allows more efficient use of hardware resources.

○ Any single network service idle most of their lifetime. CPUs are better utilized when running multiple services.

● Allows running different operating system instances in the same hardware.○ Active Directory domain controller and Linux web server can run in the same host.○ Allows operating system upgrades without requiring new hardware.

● Hypervisors allow live migration of virtual machines.○ Hardware upgrades can be separated from software updates.○ Guest load can be automatically balanced between hosts.○ Hardware can be added on demand.

● Security: services are separated from each other in different VMs.● Creation of a VM is cheap.


Virtual machine cloning and snapshots● Virtual machines are just state and files on a running process.

○ They can be stopped, saved, and copied.● Saved virtual machine instance can be migrated to another host.

○ Hypervisors support live migration, where the migration can be performed with zero downtime. ● Making a snapshot of a virtual machine makes a copy-on-write clone of the

virtual machine.○ A snapshot enables returning VM to a known good state after a failed software upgrade or a

security breach. ○ Cloning a VM enables launching new VMs from existing templates immediately without any

prior OS installation. ● Virtual applicances are complete OS and software installations distributed as a

VM images


Storage virtualization● In computer science, storage virtualization is "the process of presenting a logical

view of the physical storage resources to"a host computer system, "treating all storage media (hard disk, optical disk, tape, etc.) in the enterprise as a single pool of storage." [wikipedia]

● Virtual machines typically aren’t give direct access to physical storage devices.○ Instead, the hypervisor provides way to create and manage virtual storage devices.○ Simplest virtual storage devices are just files.○ VM migration typically requires that source and target hosts have access to shared storage.


Network virtualization● The hypervisor needs to provide virtual networks for VMs for their virtual

network interfaces.● Virtual machines needs to able to communicate with each other and the outside

network.○ There needs to be a virtual network switch or a virtual bridge.○ Or the VM is given a direct access to a physical network interface.

■ Shared by the host or dedicated.● In Linux and QEMU/KVM these network switches are virtual networks

implemented inside the kernel. ○ Kernel firewall, NAT and routing are used instead of hypervisor implemented network stack.


Virtualization caveats● Virtual machine is always slower than a physical machine.

○ Mostly this does not matter.○ With modern virtualization HW extensions virtual CPUs are just as fast as native ones.○ Input/Output latency can be a problem.

■ Disk and network IO are emulated in the hypervisor. This adds latency and users CPU time.

■ Hypervisors can allow VMs direct access to disks and network interfaces.○ 3D acceleration is a problem: 3D hardware can’t be properly virtualized yet and needs either

slow emulation or VM must be given direct access to hardware (or a slice of it).● VMs can have unexpected latency when there is host load. ● Breaking host hw or crashing hypervisor takes down all its VMs with it.

○ Multiple host machines with shared disk drives can automatically restart crashed VMs.


Hypervisor types● Type-1, native or bare-metal hypervisors

○ These hypervisors run directly on the host's hardware to control the hardware and to manage guest operating systems. For this reason, they are sometimes called bare metal hypervisors. The first hypervisors, which IBM developed in the 1960s, were native hypervisors.

○ Xen, Oracle VM Server for x86, Microsoft Hyper-V, VMware ESX/ESXi.● Type-2 or hosted hypervisors

○ These hypervisors run on a conventional operating system (OS) just as other computer programs do. A guest operating system runs as a process on the host.

○ VMware Workstation, VMware Player, VirtualBox, Parallels Desktop for Mac and QEMU are examples of type-2 hypervisors.


Hypervisor configurations and paravirtualization● Full hardware emulating hypervisor:

○ The VM looks exactly like a physical machine, with emulated real hardware devices.○ Useful mostly for OS and debugging and security researchers.

● Paravirtualized hypervisor:○ The guest operating system is running special drivers to access hypervisor services.○ The drivers are simpler and faster than emulating actual existing hardware.

■ Includes disk drivers, network drivers, display drivers, …○ Memory balloon drivers allow host to reclaim unused guest memory or to dynamically adjust the

memory available to guest.○ Agents are user space software running inside the guest to enable sharing of clipboards or guest

OS status to the hypervisor.● Emulating hypervisor: allows guests to have different HW architecture.

○ ARM emulation allows ARM development on PC hardware.


Hardware support for virtualization● Both Intel and AMD added hardware support for privileged instruction

emulation in 2005.○ Intel VT-x (code-named Vanderpool) and AMD-V (code-named Pacifica).○ All new CPUs have these features.

● I/O MMU virtualization (AMD-Vi and Intel VT-d)○ Allows physical hardware devices with DMA and interrupts to be allocated directly to VMs.○ Also called PCI passthrough.○ Includes network cards, disk controllers and GPUs.

● These features require support from motherboard and BIOS.○ Typically they are disabled by default in BIOS to protect user from (theoretical) hyperjacking.


Hypervisor software.● Virtualization turned out to be a very lucrative business.● VMWare provides multiple different hypervisor products:

○ VMWare workstation: run virtual hosts in Windows workstation.○ VMWare Player: allow premade VM images to be run.○ VMWare ESX: type 1 bare bones hypervisor for large Enterprise setups.

● Citrix XenServer: main competitor for VMWare● Oracle VirtualBox: easy to use hypervisor for workstations.● Hyper-V: MS Windows native hypervisor.● QEMU/KVM: the open source hypervisor and emulator.

○ RedHat virtualization uses QEMU/KVM.


VMWare ESX● Mature Enterprise system, but expensive. ● Web based management interface, running inside a VM.● Supports dynamically adding and removing VM hosts, without downtime.

○ Needs shared storage for this.● Paravirtual Linux drivers:

○ RAM memory balloon driver: wmv_balloon

○ Network driver: wmv_vmxnet3

○ Storage driver: wmw_pvscsi

○ Display driver: wmwwgfx

● VMware Guest agent: open-vm-tools○ https://github.com/vmware/open-vm-tools

https://github.com/vmware/open-vm-tools


Qemu (Quick EMUlator)● QEMU is a generic and open source machine emulator and virtualizer.● Was originally an emulator: does not need any hardware support from host.

○ Supports x86, ARM, PowerPC, Sparc, ...○ Enables user process emulation: emulates a process environment instead of hardware.○ Emulation is slow.

● Maps guest CPU cores to host threads. ● Emulates a complete PC with a motherboard, ISA-, PCI- and USB-bus, block

devices, network devices and audio devices.○ Many different emulated controllers.

● Supports paravirtualization through virtio devices and Spice protocol.● Needs help from kernel (KVM) to run native code.


KVM: Kernel Based Virtual Machine● Enables kernel based hypervisor with Intel or AMD hw virtualisation support.

○ Kernel modules kvm_intel and kvm_amd● Implements virtio paravirtualization call interface.● Setups VMs address space and allows boot to VM virtual firmware.● KVM API is accessed by Qemu through /dev/kvm interface.● SeaBIOS is a firmware image for legacy BIOS boot.

○ Supports PXE boot.● OVMF is a UEFI firmware image for KVM virtual machines.

○ Supports secure boot and PXE boot.○ Uefi BIOS non volatile settings are saved in a separate file in the VM image.


Qemu: virtio devices● Virtio devices are high performance paravirtualized devices presented to VM

guests. ○ IO implemented through shared memory with the VM and hypervisor.○ Support online adding and removal of devices.

● Block device driver module: virtio_blk

● Network device driver: virtio_net

● Memory balloon driver: virtio_balloon

● VGA display driver module: qxl

● Drivers available for Windows 7, 8 and 10 from RedHat / Fedora○ Installation can be painful: Windows does support changing of system disk driver after

installation.


Qemu: block devices and qcow2 format● Qemu supports emulated hardware for IDE, SATA, USB and SCSI block

devices.○ Allows running unmodified Windows images, but with a performance penalty.

● Block devices can be used directly from /dev/● Files can be used as raw disc images.● Native image format for Qemu is qcow2

○ Supports copy on write, snapshots and shared backing store devices.○ Managed from command line with qemu-img○ Image information: qemu-img info

○ Convert image formats: qemu-img convert

○ Manage snapshots: qemu-img snapshot

○ Commit a snapshot: qemu-img commit


Qemu: network configuration● Qemu emulates some network hardware:

○ Intel e1000 and realtek rtl8149○ Use virtio network device driver if possible.

● Qemu does no network infrastructure emulation.○ DHCP servers and virtual bridges, network address translation need to be set separately if

required.● Qemu macvtap support enables raw network access through existing configured

network interface.○ Adds another mac address to existing NIC and transfers raw network packets from guest to host

and back.○ Does not work properly for WiFi.


Qemu: hardware passthrough● Qemu supports USB device passthrough using Linux /dev/usb interface.

○ USB devices is removed from host and passed directly to VM.● PCIe device passthrough need vfio module support from Qemu host.

○ Vfio module implements IOMMU support for m-apping DMA IO to guest.○ The PCIe device is bound to vfio driver in Linux host.○ Qemu uses /dev/vfio/ interface to the pcie device to guest.○ Enables direct GPU and NIC support in guest.


Qemu display and remote access.● Qemu emulates physical VGA and Cirrus VGA controllers.● Supports also VMWare VGA and Xen paravirtual display controllers.

○ Windows driver support might be better or easier to setup.● QXL is the native paravirtual display controller for Qemu.

○ Enables some hardware acceleration: video might play smoothly.● The video picture is transferred either through VNC or Spice protocols.

○ VNC is a old, slow, but widely supported remote video protocol.● The SPICE project aims to provide a complete open source solution for remote

access to virtual machines in a seamless way so you can play videos, record audio, share usb devices and share folders without complications.

○ Supports guest agents.


Libvirt: Qemu VM management● libvirt is an open-source API, daemon and management tool for managing

platform virtualization. [wikipedia]○ Supports KVM, Xen, VMware ESXi, QEMU and other virtualization technologies.

● The libvirtd daemon configures and runs QEMU hypervisor processes.○ API is provided to access and configure libvirt.○ Can be used over ssh.

● Implements also storage and network configuration management. ○ Adds a NAT virtual bridge virbr0 to systems in default configuration with DHCP.○ Default storage for disk images: /var/lib/libvirt/images

● Command line VM management tool virsh● GUI tool virt-manager

Containers

23


Containers● Emulating a complete machine is overkill if you just want to package a service

and its dependencies to a neat self contained package.● OS-level virtualization refers to an operating system paradigm in which the

kernel allows the existence of multiple isolated user-space instances called containers.

○ Completely removes virtualization overhead. Normal system calls are used to access kernel.● Simplest container mechanism is implemented with Unix syscall chroot():

○ The visible root directory is changed and inherited by child processes.○ Does not provide resource utilisation management. Special directories /dev, /sys and /proc need

to be handled somehow.


Container isolation: cgroups● cgroups (abbreviated from control groups) is a Linux kernel feature that limits,

accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes. [wikipedia]

○ Limits set with ulimit() syscall only apply to single processes.● Systemd also uses cgroups to separate system daemons and user sessions in to

separate cgroups.● Cgroups enable:

○ Resource limiting: Limit available CPU, Memory or I/O○ Resource prioritization: Lower IO latency or CPU latency for realtime processes.○ Resource accounting: Measuring of resource usage.○ Resource control: Freezing and checkpointing processes.


Container isolation: namespaces● Namespaces are a feature of the Linux kernel that partitions kernel resources

such that one set of processes sees one set of resources while another set of processes sees a different set of resources.

○ Includes pids, uids, hostname, file names, network resources.● Network namespaces have different routing tables, IP addresses, sockets and

firewalls.● User namespaces allow containers to have own user accounts with a user

account that looks like root.


Linux container implementations● LXC: Run isolated Linux systems using cgroups and namespaces.● Docker: Same as LXC, but more features.

○ On October 15, 2014, Microsoft announced integration of the Docker engine into the next Windows Server release, and native support for the Docker client role in Windows. On June 8, 2016, Microsoft announced that Docker now could be used natively on Windows 10 with Hyper-V Containers, to build, ship and run containers utilizing the Windows Server 2016 Technical Preview 5 Nano Server container OS image. [wikipedia]

● Kubernetes: running Docker containers in a networked cluster. ● Snappy packet manager:

○ Canonical project to package distributions software to snap packages, which are internally containers.

Exam Monday 6th May!

28


Exam topics● Different licenses and distributions● File system layout, user accounts● Software package management● Block devices, LVM, file systems, /etc/fstab● Kernel, drivers, kernel modules, initrd● Daemons: Systemd, udev, sshd, dbus-daemon, Display Manager, X server,

Network Manager● TLS and Certificates.● Web servers, database servers, File servers (NFS and Samba)● User accounts: LDAP, Kerberos, PAM, nsswitch, sssd.● Virtual machines and containers.


Topics NOT in the exam● Hardware implementation details.● Power management. ● Audio.● Video decoding.● Bluetooth.● Java servers.● Active directory.● Printing. ● Lustre● Installation management, Ansible.

Linux installation management

Automated installs, Ansible.

31


Automated Linux installations options● Distributions provide tools for fully automated installations:

○ RedHat: kick start files○ Debian and Ubuntu: preseed files○ Installs can be very slow and writing preseed or kick start files needs lots of debugging.

● Cloning: ○ Partition, format file systems and unpack existing Linux system image○ Installation is as fast as hardware enables. Possible to use all cores.

● Scripting:○ Run scripts on a distributions default installation, possibly with Ansible.○ Easier to implement, can use normal Linux installation tools.

● Packaged configuration:○ Own software package repository, where configuration is packaged.○ The repository is probably needed anyway.


Kickstart files● RedHat kickstart configuration files provide options for RedHat installer to

automatically install and configure RedHat (or Fedora) systems.● Kickstart installations can be performed using a local DVD, a local hard drive,

or via NFS, FTP, HTTP, or HTTPS.● Used for Ukko2 node installations.


Preseed files● Ubuntu and Debian configuration files for automatic installation.● Preseeding provides a way to set answers to questions asked during the

installation process, without having to manually enter the answers while the installation is running.

○ Used debconf configuration system.○ … only way to know the questions is by trying to install the packages.○ https://wiki.debian.org/DebianInstaller/Preseed

https://wiki.debian.org/DebianInstaller/Preseed


Cloning● Linux systems can be cloned: there are no license restrictions and hardware

serial numbers are allowed change at every boot.● During system cloning:

○ Ssh hosts keys and machine-id number need to regenerated.○ Hostname needs to be changed.○ If there is any state information or logs in /var it should be cleared. ○ Care needs to be taken that uid and gid numbers of file files stay the same. ○ Initrd needs to be recreated if clone has different hardware than the original.○ Grub needs to be installed.

● This is very useful for mass installs of virtual machines: you can have one VM template which can be instantly cloned with whole file systems.


Repository management● You probably need to maintain your own package repository.

○ This is likely to require lots of work.● You need a package repository if:

○ You create your own software packages or import 3rd party software packages to your own repository.

○ You manage configuration changes with package installs.● Repository mirroring can be implemented without maintaining a complete

repository.○ A simple http cache is good enough for mirroring and network load reduction. ○ External Linux repositories are often out of sync or broken. This can be mitigated with a

completer repository mirror.


Ansible● Ansible is an open-source software provisioning, configuration management,

and application deployment tool. [wikipedia]○ Provides tools for configuring Linux hosts remotely over ssh.○ Or for pulling configurations from a repository for applying locally.

● Agentless: it connects with ssh and runs Python scripts.● Serverless: you can run ansible scripts from any hosts that has ssh access. ● Configuration is implemented with Ansible modules, which actually change the

system settings.○ The changes can (and should) be idempotent: it does not matter if they are run multiple times.

● Ansible ping module: ansible -m ping -u root b221

● Ansible shell module: ansible -m shell -u root -a "uname -a" b221


Ansible modules● Ping: check host availability● Shell: run a command in a shell● File: create files on the host● Mount: edit /etc/fstab● User and group: manage local users and groups● Apt and Dnf: manage installed packages with apt and dnf


Ansible playbooks● Ansible playbooks provide lists tasks (plays) to be applied to a system. ● Playbooks are expressed in YAML format (see YAML Syntax) and have a

minimum of syntax, which intentionally tries to not be a programming language or script, but rather a model of a configuration or a process.

○ However: yaml syntax has a number of caveats. It only looks simple.

Playbook example---- hosts: webservers remote_user: root

tasks: - name: ensure apache is at the latest version yum: name: httpd state: latest - name: write the apache config file template: src: /srv/httpd.j2 dest: /etc/httpd.conf

- hosts: databases remote_user: root

tasks: - name: ensure postgresql is at the latest version yum: name: postgresql state: latest - name: ensure that postgresql is started service: name: postgresql state: started


Ansible pull● Pulls ansible playbooks from repositories instead of pushing them over remote

ssh connections.● Playbooks can be maintained in a git repository.

# Cron job to git clone/pull a repo and then run locallyPATH=/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin12 */2 * * * root /usr/sbin/ansible-pull \

-d /var/lib/ansible/local \-U https://version.helsinki.fi/it4scifi/cubbli18-ansible.git \>> /var/log/ansible-pull.log 2>&1

course linux system administration · virtual machines typically aren’t give direct access to...

Documents