Performance Monitoring in Linux KVM Cloud Environment

Anshuman Khandual Linux Technology Centre

IBM Systems and Technology Lab anshuman.khandual@in.ibm.com

Abstract— This paper provides detailed explanation about the capabilities, challenges and future direction for Linux perf based virtualization performance monitoring in KVM cloud environment. It also explores various ways of utilizing the the collected performance data to detect, analyse and correct guest virtual machine's performance problems. Keywords — kvm, linux, performance, resource utilization, virtualization

I. INTRODUCTION KVM has been the standard virtualization platform for many Linux based open cloud deployments, where many guest virtual machines execute on top of a host machine competing for various common resources both in hardware and software. Finding various virtualization performance bottlenecks not only improves individual virtual machine performance but also improves the over all resource utilisation of the host. Fast, accurate and flexible collection of guest-host performance data can significantly improve performance, efficiency, resource utilisation of the entire cloud deployment adding value to both customers and the cloud service providers. Resource optimised cloud deployments reduces the asset ownership and operating costs. Less energy consumption makes it more environmental friendly with reduced carbon footprints. Kernel based virtualization aka KVM enables multiple guest machines to be run on top of the host kernel through qemu emulator. Individual guest machine appears to be qemu user space processes from the host perspective. Entire guest kernel and guest user space remains inside the qemu address space isolated from the host.

II. LINUX PERF Perf subsystem has been evolved in mainline kernel to unify performance measurement across the system for processor PMU, software and trace point events. This uniform framework enables user to understand various system performance bottlenecks in a holistic manner. Also perf has been improved to accommodate performance measurement capabilities in a virtualized environment.

III. PERFORMANCE MONITORING IN KVM VIRTUALIZED ENVIRONMENT

Guest virtual machine's individual performance and its impact on the host machine (as a qemu user space process) can be measured from various directions with the help of perf tool. Performance data collected from all these methods help us monitor and detect a situation of performance degradation. This would help in profiling the system to find out the root cause of the problem.

Fig. 1 Various methods of performance monitoring in KVM virtualized environment

IV. PERF KVM – CAPABILITIES Perf tool has been improved to have capabilities to profile KVM virtualized environment. A new subcommand kvm has been added to that effect. Perf kvm understands how qemu process address space encapsulates the entire guest kernel and guest user space and how to resolve addresses inside that into guest kernel symbols.

Α. Accessing guest file system Perf kvm requires access to guest /proc/modules and /proc/kallsyms file sets to be able to resolve all the captured event instruction addresses into respective guest kernel symbols. Users can either transfer these files from guests to the host and then provide them explicitly while invoking perf kvm session or they can mount the guest root file system (typically with sshfs) in the host, so that perf kvm can extract required files from the designated mount point.

978-1-4673-4422-7/12/$31.00 ©2012 IEEE

Β. Profiling options There are two different methods available in perf kvm to profile either the host or guest virtual machines. One is perf kvm top and the other one being perf kvm record followed by perf kvm report. Any perf session is always initiated from the host machine which can subsequently profile either the host or a guest or both. The profiling methods mentioned above are sampled counter based which associates event captured instruction addresses with respective symbols and sorts the symbols according to their relative percentage across the workload. During the session, perf captures the event's sample data from the kernel and stores them in a file named perf.data.kvm (perf.data.guest or perf.data.host if they are profiled individually). Though in case of perf top, this file is created and analysed on the run and the results are refreshed periodically.

Fig. 2 Basic operating principles of perf top and perf record-report

C. perf kvm examples This section explains the perf kvm operating

principles with the help of some experimental results. The snapshots demonstrate perf kvm top capabilities. Similarly guest and host profile can be achieved through perf kvm record followed by perf kvm report.

Fig. 3 Layout of three guest virtual machines on the host kernel

Fig. 4 Perf kvm top output for all guest virtual machines and the host machine during guest idle time

Fig. 5 Perf kvm top output for all guest virtual machines and the host machine during guest workload execution

V. PERF KVM – CHALLENGES There are some challenges in profiling the guest virtual machine which executes inside qemu process address space from the host perspective. A. Resolving guest address space symbols Guest virtual machines run as a process on the host kernel. So both guest kernel and guest user space address space appears to be qemu process address space from the host's perspective. With the access to the guest file sets /proc/kallsyms and /proc/modules, the host can now resolve any address inside qemu to respective guest kernel symbols. But the guest user space still remains opaque to the host which does not know about guest's individual processes and their addresses spaces inside the guest. So its difficult for perf kvm which runs on the host kernel to resolve the guest user space symbols.

Fig. 6 Guest user space symbol disambiguation challenges in perf kvm

B. Guest access of processor PMU The host perf exclusively configures, initiates and terminates the PMU access for any process requesting PMU events. The direct control over the PMU cannot be granted to the guest virtual machine as they are not aware of other guests who might be requesting PMU events at the same point of time. Host is always required to arbiter access to the PMU. Because of these reasons, perf inside guest does not support hardware PMU events.

VI. PERF INSIDE GUEST Perf can also be used inside the guest. As explained

earlier hardware PMU events are not available inside the guest. Nevertheless perf inside the guest can still profile workloads for software and trace point events which are captured inside the guest kernel without the help of PMU. An event listing with perf list command would show all the available software and trace point events inside the guest virtual machine at any point of time. In this situation perf views the guest virtual machine as an architecture which lacks support for processor PMU and continues with the rest.

Fig. 7 kernel memory software events available inside the guest virtual machine

perf top, perf stat and perf record/report can be used for any of the available software or trace point events. Call graph option can also be used to profile the stack traces of the workload.

Fig. 8 Perf results for all kernel memory events for ls command inside the guest virtual machine

Fig. 9 Perf profile on kmem:kmalloc event for ls workload inside the guest virtual machine

VII. ANALYSING GUEST MEMORY PERFORMANCE In a real world cloud environment, it is very much important to detect the situation where guests are starving or competing with each other for some common host resources. After detecting that some kind of resource congestion is happening, a holistic approach can be applied to trail down the exact location and cause of the problem. Here are the steps which could be followed to detect and identify places resources congestion.

Fig. 10 Steps to debug performance degradation

There are number of types of resource congestion which can happen in a virtualized environment like memory, processor, i/o etc. The above mentioned steps can be used in any of the above types. This paper details out the steps for memory starvation type of performance degradation with a popular transactional micro benchmark named ebizzy which resembles an web application server (highly multi threaded, large in memory working set, and allocates and deallocates memory frequently). This workload enables us to configure it in various ways to create memory starvation situations in the guest which would have visible impact on the host as well.

A. Test configuration We have a host machine with 4 CPU and 8 GB of physical main memory. Created a guest virtual machine which has 1 CPU with 1 GB of physical memory. The guest has a workload represented by the following ebizzy command line.

ebizzy -t 1 -s 1024 -n 1000 -S 1000 This workload would run single threaded, would try to allocate 1024x1000 bytes of memory and execute for total 1000 seconds. This would result in number of page faults inside the guest.

B. Performance indicators Here are some of the events which would serve as indicators for memory starvation inside the guest. These events need to be monitored in order to detect memory starvation in the guest.

PMU events cache-misses L1-dcache-load-misses L1-dcache-store-misses L1-dcache-loads KVM MMU trace events kvmmmu:kvm_mmu_pagetable_walk kvmmmu:kvm_mmu_paging_element KVM trace events kvm:kvm_entry kvm:kvm_exit kvm:kvm_inj_virq kvm:kvm_apic_accept_irq kvm:kvm_emulate_insn kvm:kvm_userspace_exit kvm:kvm_mmio

C. Baseline normal operating environment The normal operating environment of the guest virtual machine should be base lined against the above mentioned indicators. These data points would act as the base against which we would monitor guest memory performance at various points in time.

D. Monitoring of these performance indicators Increased values of these counters may signify memory starvation in the guest. There are no hard and fast rules regarding delta of these numbers which should indicate performance degradation but may vary with various types of workload and system configuration.

Fig. 11 PMU event counts on the host for various guest machine workload conditions

Fig. 12 PMU event counts on the host for various guest machine workload conditions

As observed above, various cache miss event counts increase in the host with increase in memory demand inside the guest but settles down at some point of time. Trends of this nature can indicate increased memory starvation inside the guest virtual machine. But again interpreting PMU event count trends with respect to normal operating condition numbers would vary with various guest workloads and system configurations.

Fig. 13 KVM event counts for various guest machine workload conditions

As observed above, host kvm activity level first increases but then flattens out at certain level and remains there irrespective of the memory starvation inside the guest machine. After detecting memory starvation trends in the guest machine, the next step would be to profile the guest & the host to find out the cause of the problem. E. Profile the guest Perf kvm should profile the guest on cache-misses event to find out responsible guest symbols for memory starvation. This step is necessary to figure out whether the symbol responsible for the cache-misses events belongs to guest kernel or guest user space. From the profile data, it can concluded that the highest cache-misses generating symbol is inside the guest user space. Now with this data, we would try to profile inside the guest user space. As cache-misses is a hardware event, we cannot use it to profile applications inside the guest machine. Instead we would use normal perf top to find out which

symbols are running most of the time and from there conclude the cause of the problem.

Fig. 14 perf kvm profile for guest on cache-misses event

Fig. 15 Perf top inside the guest virtual machine

VIII. GUEST VCPU BINDING ON THE HOST Placement of guest virtual machine on the host machine is critical from guest performance stand point. This section focuses on how guest placement and its cpu binding on the host can be improved with the help of perf. Here we would consider an ebizzy workload, where two independent threads each running on a different host cpu operate on the same working set memory. This results in page faults inside the guest.

ebizzy -t 2 -s 1024 -n 10 -S 10000 Because both these threads share the same working set memory, they can benefit from non-competing and equidistant memory in the host. Perf can be used detecting this kind of scenarios. This situations can not be detected inside the guest virtual machine. Here we would concentrate on the cache performance for various scenarios. Lower count of various level cache misses improves overall throughput of the hardware thread.

A. Guest binding inside the processor chip Hardware threads on different cores seems to have more cache-misses than threads running on the same core almost all the time.

Fig. 16 Cache-misses event performance for guest for different cpu binding on the same processor chip

B. Guest binding inside the system node The same principle can be applied to guest binding in a system node which consists of multiple processor chips. If guests have threads which operate on same working set memory, its always good to have them near to each other to make memory performance optimal improving overall throughput. Perf can help detect these kind of situations where they are far apart resulting in increased number of cache-misses. This helps in placing the guest appropriately on the host system node.

IX. CONCLUSION There are efforts in Linux kernel community to improve perf capabilities for the guest-host KVM environment. One such effort is to virtualize the processor PMU called Guest PMU, so that hardware events can be directly used inside the guest virtual machine. With ever evolving perf, going forward it would be easier to find performance bottlenecks in virtualized KVM environment.

ACKNOWLEDGMENT The Author wishes to acknowledge and thank

Vaidyanathan Srinivasan, Prerna Saxena and other colleagues for their help and guidance.

LEGAL STATEMENT This work represents the views of the authors and

does not necessarily represent the view of IBM. Linux is a registered trademark of Linus Torvalds. Other company, product, logos and service names may be trademarks or service marks of others. This document is provided “AS IS”, with no express or implied warranties. Use the information in this document at your own risk. Results mentioned in the presentation is for reference purposes only, and are not to be relied on in any manner.

REFERENCES

[1] (2012) Ebizzy micro benchmark. [ONLINE]. Available: http://sourceforge.net/projects/ebizzy/

[2] (2012) Linux perf subsystem. [ONLINE]. Available: https://perf.wik.kernel.org/index.php/Main_Page

[3] (2012) Perf KVM. [ONLINE]. Available: http://www.linux-kvm.org/page/Perf_events

[4] (2012) Guest PMU development. [ONLINE]. Available: http://www.linux-kvm.org/page/Guest_PMU