the future of virtio: riddles, myths and surprises · 2018. 1. 8. · the future of virtio:...
TRANSCRIPT
![Page 1: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/1.jpg)
1
The future of virtio: riddles, myths and surprises
Michael S. TsirkinJens Freimann
Fall 2017
![Page 2: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/2.jpg)
2
Virtio initialization
Virtio device
ID: 0x1 Device Features: 0x120000000
Guest OS: load driver for ID: 0x1
Driver: supported features: 0x100001000
Device Features: 0x120000000 Driver features:0x100000000
&
Driver features:0x100000000 status
OK / Fail
![Page 3: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/3.jpg)
3
Virtio initialization
Virtio device
ID: 0x1 Device Features: 0x120000000
Guest OS: load driver for ID: 0x1
Driver: supported features: 0x100001000
Device Features: 0x120000000 Driver features:0x100000000
&
Driver features:0x100000000 status
OK / Fail
![Page 4: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/4.jpg)
4
Myth #1: “changing virtio would break existing drivers”
● Really:feature negotiation can ensure compatibility
● Forward and backward● For devices and drivers
● Let’s see it in action ...
![Page 5: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/5.jpg)
5
Virtio input: add multitouch feature
● Feature bit: VIRTIO_INPUT_F_MULTITOUCH = 0● New (multi-touch aware) device: device features = 0x1● New driver: supported features = 0x1● Driver features: 0x1 & 0x1 = 0x1● Device and driver:
if (driver_features & (1 << VIRTIO_INPUT_F_MULTITOUCH)) enable multi-touch support
● Updated device & driver: multi-touch enabled!
![Page 6: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/6.jpg)
6
Compatibility: existing drivers
● Device features = 0x1● Driver supported = 0x0● Driver features = 0x0● 0x0 & (1 << VIRTIO_INPUT_F_MULTITOUCH) == 0● Device: option 1: disable multi-touch: compatible!● Device: option 2: set status = fail
Not worse than building a new device!Can suggest upgrading a driver.
![Page 7: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/7.jpg)
7
Compatibility: existing devices
● Device features: 0x0● Driver supported: 0x1● Driver features: 0x0● 0x0 & (1 << VIRTIO_INPUT_F_MULTITOUCH) == 0● Driver: option 1: disable multi-touch● Driver: option 2: set status = fail
Can suggest upgrading a device.
![Page 8: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/8.jpg)
8
Compatibility: virtio 0.9 versus 1.0
● virtio 1.0 – made default Jul 2016● Switched devices to a different register layout● Gated by a feature bit:
/* v1.0 compliant. */#define VIRTIO_F_VERSION_1 32
● No one noticed!
![Page 9: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/9.jpg)
9
Myth #2Changing virtio requires writing a
specification ● Absolutely the right thing to do● Does not have to be step 0!
● Virtio priorities:– Code compatibility– IPR compatibility– Interface compatibility
![Page 10: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/10.jpg)
10
Code compatibility:avoid conflicting with others
● New device: reserve an ID. Spec patch:
● Existing device: reserve a feature bit. E.g. :
diff --git a/content.tex b/content.tex@@ -3022,3 +3022,5 @@ Device ID & Virtio Device \\ \hline+23 & misc device \\+\hline \end{tabular}
@@ -4800,5 +4802,6 @@ guest memory statistics \item[VIRTIO_BALLOON_F_DEFLATE_ON_OOM (2) ] Deflate balloon on guest out of memory condition.+\item[VIRTIO_BALLOON_F_XXXX (3) ] Reserved for+ feature XXXX. \end{description}
![Page 11: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/11.jpg)
11
How to get it in the spec?
● git clone https://github.com/oasis-tcs/virtio-specEdit :)
● sh makeall.sh (needs xelatex, e.g. from texlive)● [email protected]● Patch: [email protected]● If no comments – email, ask for a vote ballot● Total time: up to 2 weeks
![Page 12: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/12.jpg)
12
IPR compatibility: allow others to implement compatible devices
● Open-source an implementation● Subscribe to [email protected]● Agree to IPR rules (non-assertion mode)● Send a copy of the patches (e.g. qemu, linux,
dpdk) to [email protected]● Virtio GPU at this point now.
![Page 13: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/13.jpg)
13
Interface compatibility
● Document assumptions for inter-operability● Virtio membership is not required● Membership is open - members vote on ballots● Hints:
– Document device and driver separately– Use MUST/SHOULD/MAY keywords– Ask for help!
● Virtio crypto and input at this point
![Page 14: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/14.jpg)
14
Myth #3virtio has lowest possible overhead
for host/guest communication
● “Efficient: Virtio devices consist of rings of descriptors for both input and output, which are neatly laid out to avoid cache effects from both driver and device writing to the same cache lines”.
● True - but is this really efficient?
![Page 15: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/15.jpg)
15
Virt queue: shared memory host/guest communication
buffer
Descriptor: addr: len: next
head
head + len
avail idx used idx
avail ring
used ring
![Page 16: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/16.jpg)
16
CPU caching
● Communication through shared memory requires cache synchronization (invalidate, miss, …).
● This impacts latency.
MEMORY
1
2
CACHE
CACHE
Coherence protocol
![Page 17: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/17.jpg)
17
Counting misses: no batching
● Access = cache miss →5 cache misses per request
![Page 18: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/18.jpg)
18
Counting misses: batching
● Virtio 1.0 queue layout: batching
● Batch=4 →5 misses per batch1.25 misses per request
1 Miss per batch
1 Miss per batch
1 Miss per up to 32 reqs
1 Miss per up to 16 reqs
1 Miss per up to 4 reqs
![Page 19: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/19.jpg)
19
Cache miss cost
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 160
20
40
60
80
100
120
140
160
180
200
Shared memory
Shared cache
batch
ns
![Page 20: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/20.jpg)
20
Reducing the overhead
● Information is spread across too many data structures
● Tighter packing will save cache misses.● How about packing everything in a single data
structure?
![Page 21: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/21.jpg)
21
Descriptor ring
● Driver writes out available descriptors in a ring● Device writes out used descriptors in the same
ring● Descriptor: addr, len, avail, used● To mark a descriptor available, flip the avail bit● To mark a descriptor as used, flip the used bit
![Page 22: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/22.jpg)
22
Descriptor states
Avail: 0 → 1
Driver
Used: 0 → 1
Device
Avail: 1 → 0
Driver
Used: 1 → 0
Device
Ringwrap around
Avail = used: ok for guest to produceAvail != used: ok for host to consume
![Page 23: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/23.jpg)
23
Host: pseudo code (in-order)
static int used = 1;
while(desc[idx].avail == used) ←miss?
relax();
process(&desc[idx]);
desc[idx].used = used; ←miss?
idx = idx + 1;
if (idx == size)
Idx = 0;
used = !!used;
![Page 24: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/24.jpg)
24
Out-of-order: descriptor id● Host: used 1
11/0
1/0
● Guest: available 9
0/0idid
id
Host: use
Avail/used
1/0
1/01/01/01/01/01/01/0
0/00/00/00/00/00/0
0/0
1/01/01/01/01/01/0
0/00/00/00/00/00/0
1/10/00/00/01/01/01/0
Guest: available
![Page 25: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/25.jpg)
25
CPU caching
● Both host and guest incur misses on access● No batching: 2 to 4 misses per descriptor● Batch=4:
2 to 4 misses per batch4 descriptors per cache line →0.5 to 1 misses per descriptor
● Better than virtio 1.0 even in the worst case
![Page 26: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/26.jpg)
26
In-order: descriptor id● Host: consumed 9
11/0
1/0
● Guest: produced 9
0/0idid
id
Host: consume
Guest: produce
Avail/used
1/0
1/01/01/01/01/01/01/0
0/00/00/00/00/00/0
0/0
1/01/01/01/01/01/0
0/00/00/00/00/00/0
0/00/00/00/01/01/01/0
One write per batch of descriptorsDriver ensures avail != used
![Page 27: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/27.jpg)
27
Request processing: comparison
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 160
20
40
60
80
100
120
140
160
180
200
Virtio 1.0
Virtio 1.0 shared cache
Virtio 1.1
Virtio 1.1 shared cache
batch
ns
![Page 28: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/28.jpg)
28
Mpps16
17
18
19
20
21
22
23
virtio 1.0virtio 1.1
Virtio queue is not optimalwe will fix it
64 byte packet throughput
![Page 29: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/29.jpg)
29
Riddle #1: event suppression
● Each queue has two event index structures● Which descriptor should trigger an interrupt● Can we put this in the descriptor itself?● Should we?● Just use polling?
![Page 30: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/30.jpg)
30
Riddle #2: why powers of 2?
2 VQs * 1K descriptors
fills a 32K cache
2VQs * 0.75K descriptors
free cache for data
![Page 31: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/31.jpg)
31
Powers of two: pseudo code
unsigned next_power_of_two(unsigned index, unsigned size){ return (index + 1) & (size - 1);}
unsigned next_non_power_of_two(unsigned index, unsigned size){ return ++index >= size ? 0 : index;}
![Page 32: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/32.jpg)
32
Surprise #1: hardware is special
● Let’s assume a pass-through device implementing virtio. Shouldn’t this just work?
● Maybe – but not optimally!● Hypervisor: processes descriptors one by one● Hardware: can process many in parallel● Needs to be told how many are available● Include number of available entries in a kick
![Page 33: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/33.jpg)
33
Surprise #2: writes are expensive
● PCI Express payload is full dword.● Flipping single bits across PCIE is expensive● In-order processing will help reduce number of
writes
![Page 34: The future of virtio: riddles, myths and surprises · 2018. 1. 8. · The future of virtio: riddles, myths and surprises Michael S. Tsirkin Jens Freimann Fall 2017. 2 Virtio initialization](https://reader033.vdocuments.net/reader033/viewer/2022051811/60208065bde5ab2c635eda5c/html5/thumbnails/34.jpg)
34
Summary
● Virtio 1.1 is shaping up to be a big release – Performance– Hardware offloads
● Join the fun– Still lots of open questions– Implementation and benchmarking of the new
features– Virtio BoF tomorrow