Scheduling I/O in Virtual Machine Monitors
Diego Ongaro, Alan L. Cox, and Scott RixnerRice UniversityMarch 5, 2008
2Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Server Consolidation
Virtual machine monitors enable server consolidation
Consolidated domains have wildly different workloads
Do current solutions actually achieve fairness?
Streaming Media Server
Voice-over-IP Server
Application Server
Web Server
Database Server
. . .
Virtualized Host
Guest Domain 1
Guest Domain 2
Guest Domain 3
Guest Domain 4
. . .
consolidation
3Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Xen Performance
02468
101214161820
Pro
cess
or
Uti
liza
tio
n (
%)
G1 G2 G3 G4 G5 G6 G7
Computation Guests
020406080
100120140160180200
Ban
dw
idth
(M
bp
s)G1 G2 G3 G4 G5 G6 G7
Streaming Guests
Ping latency to G8: 259.2ms Ping latency to G8: 6.6ms
Ping additional idle guest (G8):
4Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Hypervisor – thin virtualization layer Driver Domain – hardware control Guest domains given restricted access
Xen Architecture
Hardware
DriverDomain
Xen ControlSoftware
Guest Domain
UserSoftware
GuestDomain
UserSoftware
GuestDomain
UserSoftware
Xen Hypervisor
5Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Receiving a Network Packet
time
Hardware
DriverDomain
Xen ControlSoftware
Guest Domain
UserSoftware
GuestDomain
UserSoftware
GuestDomain
UserSoftware
Xen Hypervisor
Har
dwar
eIn
terr
upt
Virt
ual
Inte
rrup
t
Driv
er D
omai
n
Targ
et D
omai
n
Sche
dule
r
Sche
dule
r
Virt
ual
Inte
rrup
t
Pack
et A
rriv
es
6Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Scheduling Delays Scheduler has a large effect on latency
As packets arrive, two opportunities to run any number of domains, delaying I/O
Delay proportional to the number of runnable domains
time
Har
dwar
e In
terr
upt
Virt
ual I
nter
rupt
Driv
er D
omai
n
Targ
et D
omai
n
Sche
dule
r
Sche
dule
r
Virt
ual I
nter
rupt
Pack
et A
rriv
es
Oth
er D
omai
ns
Oth
er D
omai
ns
7Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Scheduler Fairness Virtual machine monitors must fairly support
both computation and I/O domains Xen supports computation domains well, but has
mixed results with I/O domains
The scheduler plays a significant role in I/O performance Improvements for I/O are not obvious Some of Xen’s current scheduler optimizations for
I/O are beneficial, others are not
8Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Xen's Credit scheduler
Scheduler “optimizations” for I/O
Conclusions
Outline
9Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Credits are assigned to each domain Approximate the fraction of processor resources
each domain will receive Do not indicate when each domain will receive its
fraction
Scheduler increments/decrements credits Periodically deducts credits from running domain Adds credits when majority of credits in the
system have been consumed
Xen’s Credit Scheduler
10Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Scheduler Operation Domain states
Under – domain has credits remaining Over – domain is over its credit allowance
Domains are run in FIFO order by state Over domains only run if no under domains A domain may run for up to 30ms if it has enough credits
After running, return to the run queue by state Behind all other domains in the same state Regardless of runtime or remaining credits
This approach is biased against I/O
O2 O1
Run Queue
U4 U2U3Next domain
to runU1
11Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Scheduler “Optimizations” for I/O I/O Domain Requirements
Low latency High bandwidth Independent of other domains’ workloads
“Optimizations” Boosting idle domains
Included in Xen Ordering the run queue by credits
Proposed for Xen Tickling the scheduler
Included in Xen
12Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Boosting Idle Domains Initially, scheduler had no special features for
I/O domains
Additional scheduling state: boost Higher priority than under Used when an idle domain is sent a virtual interrupt
With boost
O2 O1 U1U4 U2U3
I1U5
O2 O1 U1U4 U2U3
I1B1
Next domainto run
Next domainto run
13Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Impact of Boost Boost improves latency for I/O domains
0
50
100
150
200
250
300
350
400
Time
La
ten
cy
(m
s)
No Boost Boost
14Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Caveats with Boost Ineffective for I/O domains under heavy loads
Must be idle to receive boost Must block quickly to sustain boost
Effect is negated with many I/O domains
O2 O1 B1B4 B2B3
Next domainto run
O2 O1 U1U4 U2U3
vs.
15Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Ordering the Run Queue I/O domains tend to quickly block
Within each state, sorts domains by credits remaining
Short-running I/O domains are re-inserted near the head of the run queue
16Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Impact of Ordering the Run Queue Ordering the run queue can reduce latency for
I/O domains under larger loads Surprisingly, often complements boost
Configuration Latency CPU Time
Boost,
Not Ordered258.2ms 1.2%
No Boost,
Ordered39.0ms 3.1%
Boost,
Ordered3.5ms 13.3%
17Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Tickling the Scheduler Xen tickles the scheduler after each virtual
interrupt
Tickling can reduce latency Often runs virtual interrupt recipient immediately
time
Har
dwar
e In
terr
upt
Virt
ual I
nter
rupt
Driv
er D
omai
n
Targ
et D
omai
n
Sche
dule
r
Sche
dule
r
Virt
ual I
nter
rupt
Pack
et A
rriv
es
Oth
er D
omai
ns
Oth
er D
omai
ns
18Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Tickling Too Soon? Tickling as virtual interrupts are sent may be
too soon Scheduler is uninformed about upcoming virtual
interrupts Gives arbitrary priority to first domain(s) to be sent
a virtual interrupt
Don’t tickle while sending virtual interrupts All recipient domains are then runnable Scheduler runs the domain with the highest priority
19Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Impact of Tickling
020406080
100120140160180200
Ban
dw
idth
(M
bp
s)G1 G2 G3 G4 G5 G6 G7
No Tickling
020406080
100120140160180200
Ban
dw
idth
(M
bp
s)
G1 G2 G3 G4 G5 G6 G7
Tickling
20Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Transcending Xen Scheduling mixed domains
Native vs. VMM scheduling
21Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Mixed I/O- and compute- domains don’t underutilize resources VMM scheduler can not schedule only the I/O process Low latency for a mixed domain can not be guaranteed
Mixed Domains
Hypervisor
GuestDomain
CPU process
GuestDomain
CPU process
I/O process
GuestDomain
CPU process
Driver Domain
Scheduler
22Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Native vs. VMM Scheduling
Hypervisor
Guest
CPU
Guest
CPU
I/O
Guest
CPU
Scheduler
Driver Domain
?
Native
Scheduler
CPU process
I/O processCPU process
CPU process
Native VMM
23Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Summary of Results Boost reduces latency with few I/O domains
Does little harm in other cases
Ordering the run queue favors short-running I/O domains Often complementary to boost
Tickling reduces latency in some cases Often causes unfairness in bandwidth and latency
I/O-intensive applications need their own domains
24Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro
Conclusions Scheduling has a large impact on I/O fairness
Can cause exorbitant latency Can cause bandwidth inequity
Xen’s Credit scheduler is not fair for I/O Does well with compute domains No notion of timeliness, which is needed for I/O
Further work needs to be done to solve the mixed domains problem