Understanding “Host” and “Guest” Memory Usage and Related Memory Management Concepts

Kit ColbertSr. Staff Engineer, VMware

DisclaimerThis session may contain product features that are currently under development.

This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features discussed or presented have not been determined.“These features are representative of feature areas under development. Feature commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.”

Agenda

Motivation

Define host and guest memory usage and ask some questions

Memory management concepts

Answer our questions

Best practices

Summary

Q & A

Motivation

Host and Guest Memory Usage

Using Host and Guest Memory Usage

Useful for quickly analyzing a VM’s status– Coarse-grained information

– Important for prompting further investigation

Requires an understanding of memory management concepts– Many aspects of host/guest memory interaction are not obvious

Define host and guest memory and ask some questions

Host Memory Usage Defined

Host Memory Usage– The amount of physical host memory in megabytes allocated to a

guest• Includes virtualization overhead (e.g. hypervisor data)

Guest Memory Usage Defined

Guest Memory Usage– The amount of memory in megabytes actively used by a guest

operating system and it’s applications

– The percent of total guest memory actively used by a guest

Memory Terminologymemory size

total amount of memory presented to a guest

allocated memorymemory assigned to

applications

unallocated memorymemory not assigned

active memoryallocated memory recently

accessed or used by applications

inactive memoryallocated memory not recently accessed or

used

Guest memory usage measures this

Host memory usage measures this, sorta…

Some commonly asked questions

Why is host memory usage so high?

Why is host memory usage greater than guest memory usage?

Why is host/guest memory usage different than what I see inside the guest?

Memory Management Concepts

Discussion of Concepts

Focus on high-level concepts, not implementation details– Simpler and easier to understand

– Keep the fundamental ideas and drop the unimportant ones

Other presentations may present this information differently– May discuss more implementation than concept

– That’s ok – different viewpoints for same thing!

Virtual Memory

Creates uniform memory address space– Operating system maps application virtual

addresses to physical addresses

– Gives the operating system memory management abilities that are transparent to the application

“virtual” memory

“physical” memory

“machine” memory

guest

hypervisorHypervisor adds extra level of indirection

– Maps guest’s physical addresses to machine addresses

– Gives the hypervisor memory management abilities that are transparent to the guest

Virtual Memory

guest

hypervisor

“machine” memory

“physical” memory

“virtual” memory

“virtual” memory

“physical” memory

“machine” memory

guest

hypervisor

Application

Operating System

Hypervisor

App

OS

Hypervisor

VM

Application Memory Management

Starts with no memory

Allocates memory through syscall to operating system

Often frees memory voluntarily through syscall

Explicit memory allocation interface with operating system

Hyper visor

OS

App

Operating System Memory Management

Assumes it owns all physical memory

No memory allocation interface with hardware

– Does not explicitly allocate or free physical memory

Defines semantics of “allocated” and “free” memory

– Maintains “free” list and “allocated” lists of physical memory

– Memory is “free” or “allocated” depending on which list it resides

Hyper visor

OS

App

How “Allocated” and “Free” Lists Work

OS

AppAllocated List Free List

Memory becomes “allocated” or “free” based on which list (piece of memory) has a pointer to it

Hypervisor Memory Management

Very similar to operating system memory management

– Assumes it owns all machine memory

– No memory allocation interface with hardware

– Maintains lists of “free” and “allocated” memory

Hyper visor

OS

App

VM Memory Allocation

VM starts with no physical memory allocated to it

Physical memory allocated on demand

– Guest OS will not explicitly allocate

– Allocate on first VM access to memory (read or write)

Hyper visor

OS

App

VM Memory Reclamation

Guest physical memory not “freed” in typical sense

– Guest OS moves memory to its “free” list

– Data in “freed” memory may not have been modified

Hyper visor

OS

App

Hypervisor isn’t aware when guest frees memory

– Freed memory state unchanged

– No access to guest’s “free” list

– Unsure when to reclaim “freed” guest memory

Guest Free List

Guest OS (inside the VM)– Allocates and frees…

– And allocates and frees…

– And allocates and frees…

Hyper visor

AppGuestfree list

VM– Allocates…

– And allocates…

– And allocates…

Hypervisor can’t reclaim memory through guest frees!

Inside the VM

OS

VM

VM Memory Reclamation Cont’d

What to do about VM memory reclamation?

Why not just watch the free list?

Idea– Hypervisor could identify memory used by guest for it’s free list

– Watch that memory area for changes and free physical memory that backs guest memory added to free list

Problem– Free list may be hard to identify

• Depends on guest type and may change from release to release

– “Free” is not always free• “Free memory” may be used in buffer cache or elsewhere by guest

– Guest may not free memory when hypervisor wants it to

What can the hypervisor do to reclaim VM memory?

Nothing– Route taken by pretty much everyone except VMware

– No memory overcommitment! (Inefficient use of memory)

– May try to convince you memory overcommitment isn’t important

Something– VMware-innovated techniques

– Supports memory overcommitment• Efficient use of memory

• Strict resource control guarantees

Memory Overcommitment

Aggregate guest memory greater than host memory– Not enough physical memory to satisfy all VM memory needs

– Eventually VM may want physical memory when none is left

VM 1 VM 2 VM 3

Hyper visor

Why is memory overcommitment important?

Ensures physical memory is actively used as much as possible– Guest VMs may have lots of unused or inactive memory

– No reason to back one VM’s unused/inactive memory with physical memory if other VMs can use it

– Hypervisor can reclaim unused/inactive memory and redistribute to other VMs who will actively use it

Increases VM-to-host consolidation ratio– Each VM has a smaller physical memory footprint since only its

active memory is resident

– You can fit more VMs in the same amount of physical memory

VM Memory Reclamation Techniques

Simple idea: why maintain many copies of the same thing?

– If 4 Windows VMs are running, there are 4 copies of Windows code

– Only one copy is needed

Share memory between VMs when possible

– Background hypervisor thread identifies identical sets of memory

– Points all VMs at one set of memory, frees the others

– VMs are unaware of the change

VM 1 VM 2 VM 3

Hyper visor

VM 1 VM 2 VM 3

Hyper visor

Transparent Page Sharing (TPS)

Hypervisor wants to reclaim memoryGuest OS is not aware of this

– Thinks it owns all physical memory– Sits inside its own “box”, unaware it’s

running in a VM or that other VMs are running

Goal: make the guest aware so it frees up some of its memory

Hyper visor

OS

App

OS

App

VM 1

Solution: artificially create memory pressure inside the VM

– “Push” memory pressure from the hypervisor into the VM

– Use “balloon” driver inside the VM to create memory pressure

VM 2

memory pressure

Ballooning

Hyper visor

OS

GuestApp

Guestfree list

BalloonDriver

1. Balloon driver allocates memory

2. Balloon driver pins allocated memory

3. Guest may reclaim other memory

4. Balloon driver tells hypervisor what memory it allocated

5. Hypervisor frees machine memory backing memory allocated by balloon driver

6. Hypervisor now has more free physical memory

List of memory

Hypervisor: 8 pieces of memory are allocated

Guest: 3 pieces of memory are allocated

Hypervisor: 5 pieces of memory are allocated

Guest: 6 pieces of memory are allocated

Inflating Balloon

Hyper visor

OS

BalloonDriver

Why can the hypervisor reclaim ballooned memory?

1. Balloon driver doesn’t

2. OS doesn’t

3. Nothing in the VM relies on the memory’s value for correctness

What in the VM assumes the memory has a specific value?

Because of a contract with the hypervisor

Because the memory allocated to an app List of

memory

Hypervisor can safely reclaim ballooned memory because the VM does not rely on a particular value for that memory.

Inflating Balloon Cont’d

Hyper visor

OS

GuestApp

Guestfree list

BalloonDriver

Guest OS swapping, a possible side effect of ballooning Two possibilities for guest free memory:

2. VM doesn’t have much free memory

1. VM has lots of free memory

Guestfree list

1. No swapping necessary!

2. Needs to swap!

Guest OS chooses whether to swap or not!

Inflating Balloon Cont’d

Hypervisor can swap VM memory– Swaps out to a per-VM swap file

– Transparent to the VM

Hypervisor swapping is a last resort– Both TPS and ballooning preferred

• Low overhead for TPS

• Guest swapping due to ballooning performs better than hypervisor swapping

• But both take time

– Hypervisor swapping quickly reclaims memory, but is more expensive overall

Hyper visor

OS

App

Swapping

When to reclaim memory

Quick Review

Physical memory is not reclaimed on a guest that is “free”– VM may accrue lots of physical memory

Transparent page sharing is always running– May or may not help reduce a VM’s physical memory consumption

Ballooning or swapping is the only other way to reclaim memory– Both have performance overhead

– Want to use only when necessary

Question: when to invoke ballooning or swapping?

When To Reclaim Memory

Memory overcommitment– Aggregate guest memory is greater than host memory

Not memory overcommitted– Never reclaim memory from guest through ballooning or swapping

• No need to!• (Transparent page sharing is always running)

Memory overcommitted– Only reclaim once free physical memory drops below threshold

• Start ballooning when memory starts to fall toward threshold• Start swapping as memory nears/passes threshold

* Assumes no VM or resource pool memory limits are set!

Answer our questions

High Host Memory Usage

Why is host memory usage so high?– VM may have used lots of memory in the past

• On boot, Windows zeros all guest memory

• Starting several guest applications together, e.g., at boot time or when building a big project, can lead to high memory usage

• Guest workload spike

– Host memory usage represents “high water mark” of guest’s memory usage on non-memory overcommitted hosts

• No reason to reclaim memory from VM when not memory overcommitted

Host Memory Greater Than Guest Memory

Why can host memory be greater than guest memory?– At some point in the past, the guest consumed a larger amount of

memory than it is actively using now

– With little to no memory pressure on the physical host, there is no reason for hypervisor to reclaim memory from the guest

– The guest maintains a high host memory usage even though its guest memory usage is small

Expected behavior– Nothing to worry about!

Host/Guest Memory Usage Different Inside Guest

Why is host/guest memory usage different than what I see inside the guest OS?

– Guest memory• Guest has better visibility while estimating “active” memory

• ESX active memory estimate technique can take time to converge

– Host memory• Host memory usage doesn’t correspond to any memory metric within the

guest

• Host memory usage size is based on a VM’s relative priority on the physical host and memory usage by the guest

Again, this is expected!

Best Practices

Host Memory Usage

Large host memory usage OK– Expected behavior much of the time!

Host memory usage shouldn’t limit guest memory usage– Host memory usage should be large enough to accommodate guest

workload

– Don’t over commit host memory forcing the guest to continuously swap

Use shares to adjust relative priorities among VMs when memory is overcommitted

– Host memory usage is based on shares when overcommitted

Guest Memory Usage

VM’s memory size should be slightly larger than the average guest memory usage

– Accommodates workload spikes without guest swapping

– Hypervisor will handle excess host memory usage

– Larger VM memory size means more overhead memory

Summary

Summary

Host/guest memory usage is not as simple as it looks!– Represents complex set of data

– Requires understanding of memory management concepts

– Many aspects that are not obvious!

Memory Management Concepts– Guest doesn’t “free” memory in a typical sense

– Hypervisor isn’t aware of memory a guest has “freed”

– Hypervisor uses transparent page sharing, ballooning, and swapping to reclaim memory

– Ballooning and swapping used only when host is memory overcommitted!

Summary Cont’d

Host/guest memory can be very useful with proper knowledge– Quick way to determine the status of VM

– Important tool to prompt further investigation

– Requires good mental model of memory management concepts• Hopefully we’ve helped you with that today!

Q&AKit ColbertSr. Staff Engineer VMware

Thank you for coming.

Rate your session and watch for the highest scores!

Backup Slides

Measuring Active Guest Memory

Guest self-measurement won’t work– Each guest uses different method to estimate active memory

– Can’t compare activeness data from Windows guest with Linux guest

– Comparable estimate important for making allocation decisions

Statistical Sampling– Select a subset of memory at random

– Compute the percentage of pages accessed in a minute

– Average percentage gives an estimate of “active” guest memory

Ballooning and Swapping

Ballooning– Provide adequate swap space in guest OS

• Ballooning can lead to high guest memory swapping

• Ballooning not effective if a guest can’t swap

– Limit maximum balloon size if lots of guest memory pinned• Pinned memory cannot be swapped

• Use sched.mem.memctl.max to set maximum balloon size

Swapping– Ensure adequate swap space on VMFS datastore

• Swap size = VM memory size – VM memory reservation