xpds14: efficient interdomain transmission of performance data - john else, citrix
DESCRIPTION
As users demand greater scalability from Citrix XenServer, the transmission of performance data from guests via xenstore is increasingly becoming a bottleneck. Future use of service domains is likely to make this problem worse. A simple, efficient way of transmitting time-varying datasets between userspace components in different domains is required. This talk will propose a lock-free mechanism to allow interdomain reporting of performance data without relying on continuous xenstore usage, and describe how it fits into the XAPI toolstack.TRANSCRIPT
![Page 2: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/2.jpg)
XAPI projectXenAPIXAPIXCP/XenServer toolstack
![Page 3: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/3.jpg)
XAPI projectHigh level OO API around XenHTTP interfaceCollects performance data from dom0/guests
![Page 4: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/4.jpg)
Performance dataNumericChanges regularly - requires samplingIf we sometimes lose a sample - that's OK
![Page 5: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/5.jpg)
Interdomain performance dataCurrently one metric per domU: free memoryTransmitted via xenstoreEven this has been enough to cause performance problems!
![Page 6: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/6.jpg)
Improvements so farData collection caused laggy API response from XAPI-> XAPI disaggregation: rrdd, networkd and xenopsd
![Page 7: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/7.jpg)
Improvements so farHigh xenstore load reduced bootstorm performance
-> Watch xenstore instead of polling
![Page 8: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/8.jpg)
What next?Take xenstore out of the data pathCreate a generic API for:
Driver domain monitoringApplication monitoring...
![Page 9: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/9.jpg)
rrdd plugins - dom0
![Page 10: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/10.jpg)
rrdd plugins - interdomain
![Page 11: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/11.jpg)
rrdd plugins - protocol v1header 10 bytesdata length (ASCII) 8 bytesmd5sum (ASCII) 32 bytesjson data inc. timestamp ? bytes
![Page 12: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/12.jpg)
rrdd plugins - protocol v1DATASOURCES <- header00000a79 <- data length6cc0472a94896d245dcea04a37c26474 <- data md5sum{ "timestamp": 1407715758, "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value": "0.55", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}
![Page 13: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/13.jpg)
rrdd plugins - protocol v1DATASOURCES <- header00000a79 <- data length6cc0472a94896d245dcea04a37c26474 <- data md5sum{ "timestamp": 1407715758, "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value": "0.55", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}
![Page 14: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/14.jpg)
rrdd plugins - protocol v2Separate data from metadataPack as much as possible as binary
![Page 15: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/15.jpg)
rrdd plugins - protocol v2header 10 bytesdata crc32 4 bytesmetadata crc32 4 bytesdatasource count (int32) 4 bytestimestamp (int64) 8 bytesdatasource values n * 8 bytesmetadata length (int32) 4 bytesjson metadata ? bytes
![Page 16: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/16.jpg)
rrdd plugins - protocol v2header 10 bytesdata crc32 4 bytesmetadata crc32 4 bytesdatasource count (int32) 4 bytestimestamp (int64) 8 bytesdatasource values n * 8 bytesmetadata length (int32) 4 bytesjson metadata ? bytes
![Page 17: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/17.jpg)
rrdd plugins - protocol v2DATASOURCES <- header.... <- data CRC.... <- metadata CRC.... <- datasource count........ <- timestamp........ <- datasource value.... <- metadata length{ "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}
![Page 18: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/18.jpg)
rrdd plugins - protocol v2DATASOURCES <- header.... <- data CRC.... <- metadata CRC.... <- datasource count........ <- timestamp........ <- datasource value.... <- metadata length{ "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}
![Page 19: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/19.jpg)
rrdd plugins - protocol v2V2 protocol gives about a 10x speedup in the "usual" case, i.e. set ofexported datasources doesn't change.
![Page 20: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/20.jpg)
Demo
![Page 21: XPDS14: Efficient Interdomain Transmission of Performance Data - John Else, Citrix](https://reader034.vdocuments.net/reader034/viewer/2022042814/555c4376d8b42a2c068b4f0d/html5/thumbnails/21.jpg)
Further infoProtocol:Frontend:Slides:IRC: freenode #xen-apiEmail: [email protected]
github.com/xapi-project/rrd-transportgithub.com/xapi-project/ocaml-rrdd-pluginjohnelse.github.io/xen-summit-2014