an smb3 engineer’s view of windows server 2012 hyper-v ......windows 2012: command run1 run2 run3...
TRANSCRIPT
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
An SMB3 Engineer’s View of Windows Server 2012 Hyper-V Workloads
Gerald Carter [email protected]
EMC/Isilon
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
One sunny day in San Jose….
FAST 2013 - “Virtual Machine Workloads: The Case for New NAS Benchmarks” V.Tarasov, Stony Brook University; D.
Hildebrand, IBM Research—Almaden; G. Kuenning, Harvey Mudd College; E. Zadok, Stony Brook University
https://www.usenix.org/conference/fast13/virtual-machine-workloads-case-new-nas-benchmarks
2
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Questions…
Do the existing characterizations of NFS data stores for hypervisors apply to Hyper-V and SMB3?
What do the I/O and jump distance patterns look like?
What SMB3 protocol features does Hyper-V use?
3
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Addendum to Abstract…
Original scope of abstract was reduced due to time constraints and hardware availability SMBDirect Failover Multi-Channel
Addition of tools for packet capture introspection
4
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Test Setup
5
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Test Bench Specifications
Hardware Intel Quad CPU Q9650 @ 3GHz, 16Gb RAM,
SATA II
Networking Cisco SG-200 1Gbps switch
Environment Windows Server 2012 Standard Hyper-V and
File Services
6
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
VM Configuration
7
Operating System RAM CPU Disk (Thin)
Ubuntu 12.04 x64 1 GB 1x vCPU (4x vCPU)
40 GB IDE
Windows XP SP3 x86
512 MB 1x vCPU
40 GB IDE
Windows Server 2012 Standard
2 GB 1x vCPU
60 GB IDE (60 GB SCSI)
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Deployment to SMB File Share
8
PS> Import-VM -path $SourceXml -copy -GenerateNewID -VirtualMachinePath $TargetDir -SnapshotFilePath $TargetDir -SmartPagingFilePath $TargetDir -VhdDestinationPath “$TargetDir\Virtual Hard Disks\” PS> Move-VMStorage <VMName> -DestinationStoragePath \\server\share\VMName
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Methodology & Tools
9
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Scenarios
Boot Linux/WinXP/Win2012 VM image Single host and multiple hosts Idle for 10 minutes
Build Linux Kernel Single host build and multiple host builds 1x vCPU and 4x vCPU
Random file operations Explorer and robocopy copies, small and
large files (4Kb – 1GB)
10
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Methodology
1. Cold boot Win2012 SMB3 file server 2. Start packet capture (dumpcap.exe) 3. Cold boot Win2012 Hyper-V host 4. Launch scenario VM(s) 5. Cleanly shutdown scenario VM(s) 6. Stop packet capture 7. Shutdown Win2012 Hyper-V host 8. Shutdown Win2012 SMB3 File Server
11
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Sharktools
Python extension library to allow using Wireshark’s packet dissector feature https://github.com/armenb/sharktools
Required small fixes for building against
Wireshark 1.10.1
Returns a dictionary of fields per frame { 'frame.number': 10,
'smb2.sesid': 2L, 'smb2.seq_num': 9589L, … } 12
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Import pyshark
13
#!/usr/bin/env python import sys import pyshark pcap = pyshark.read( ‘/home/coffeedude/sample.pcap’, [ ‘frame.number’, ‘smb2.seq_num’ ], ‘smb2 and smb2.cmd == 0x05’) for f in pcap: print f {'frame.number': 10, 'smb2.seq_num': 9589L} {'frame.number': 101, 'smb2.seq_num': 9691L}
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Dictionary Composition
14
SMB2 Request
SMB2 Response
SMB2 Request
SMB2 Response
SMB2 Request
SMB2 Response
SMB2 Request
SMB2 Response
OpExchange (Create)
File Handle f[name] = {
}
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Scenario – Single Client Boot
15
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Command Distribution
16
Linux WinXP Windows 2012 Command Run1 Run2 Run3 Run1 Run2 Run3 Run1 Run2 Run3 Negotiate 0.01% 0.01% 0.01% 0.01% 0.01% 0.01% 0.00% 0.00% 0.00%
SessionSetup 0.46% 0.49% 0.50% 0.45% 0.44% 0.46% 0.28% 0.27% 0.25% Logoff 0.44% 0.47% 0.48% 0.43% 0.42% 0.45% 0.27% 0.26% 0.25%
TreeConnect 0.47% 0.50% 0.51% 0.45% 0.45% 0.47% 0.29% 0.27% 0.26% TreeDisconnect 0.45% 0.48% 0.49% 0.44% 0.43% 0.46% 0.28% 0.27% 0.25%
Create 1.91% 2.04% 2.08% 1.67% 1.66% 1.75% 1.08% 1.03% 0.97% Close 0.51% 0.51% 0.57% 0.41% 0.46% 0.43% 0.27% 0.26% 0.25% Flush 0.36% 0.35% 0.34% 3.24% 3.34% 3.29% 1.39% 1.32% 1.29% Read 35.60% 32.61% 32.55% 38.09% 28.22% 29.31% 54.75% 54.94% 54.89%
Write 1.33% 1.60% 1.32% 14.20% 12.46% 13.43% 7.43% 7.56% 6.99% Lock 0.24% 0.26% 0.27% 0.12% 0.12% 0.13% 0.08% 0.07% 0.07% IoCtl 1.35% 1.44% 1.47% 1.14% 1.13% 1.19% 0.90% 1.00% 0.83%
Cancel 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Echo 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
QueryDirectory 0.00% 0.00% 0.00% 0.02% 0.02% 0.02% 0.00% 0.00% 0.00% ChangeNotify 0.44% 0.47% 0.48% 0.38% 0.38% 0.40% 0.25% 0.24% 0.22%
QueryInfo 56.23% 58.55% 58.73% 38.84% 50.36% 48.10% 32.65% 32.45% 33.39% SetInfo 0.20% 0.21% 0.22% 0.11% 0.11% 0.11% 0.07% 0.07% 0.07%
OplockBreak 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Command Occurrences
17
Linux WinXP Windows 2012 Command Run1 Run2 Run3 Run1 Run2 Run3 Run1 Run2 Run3 Negotiate 1 1 1 1 1 1 1 1 1
SessionSetup 60 60 60 58 58 58 58 58 58 Logoff 58 58 58 56 56 56 56 56 56
TreeConnect 61 61 61 59 59 59 59 59 60 TreeDisconnect 59 59 59 57 57 57 57 57 58
Create 251 251 251 218 219 219 220 221 222 Close 67 63 69 53 61 54 55 56 57 Flush 47 43 41 422 442 412 284 284 295 Read 4668 4005 3926 4964 3730 3669 11170 11815 12526
Write 174 196 159 1851 1647 1681 1515 1626 1596 Lock 32 32 32 16 16 16 16 16 16 IoCtl 177 177 177 149 149 149 183 214 190
Cancel 0 0 0 0 0 0 0 0 0 Echo 0 0 0 0 0 0 0 0 0
QueryDirectory 0 0 0 2 2 2 0 0 0 ChangeNotify 58 58 58 50 50 50 51 51 51
QueryInfo 7373 7190 7085 5061 6657 6021 6660 6978 7619 SetInfo 26 26 26 14 14 14 15 15 15
OplockBreak 0 0 0 0 0 0 0 0 0
Total Requests 13112 12280 12063 13031 13218 12518 20400 21507 22820
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
VHDX Handle – Linux
18
Start Length Tags Access Mask Share Flags Create Options
1379126225.583680 15.566940 ['DH2Q' 'QFid' 'RqLs'] 0x120089 0x7 0x400060
1379126225.765800 15.385081 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126225.767010 15.384112 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126225.768160 15.383215 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126225.769270 15.382348 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126225.770370 643.326370 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126225.771510 15.380348 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40
1379126225.771860 0.083857 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48
1379126225.852160 15.299951 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126225.855980 632.726568 ['DH2Q' 'RqLs'] 0x12019f 0x1 0x48
1379126284.873770 0.019414 0x100080 0x7 0x400060 … … … … … …
1379126859.659730 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x7 0x400060
1379126859.660920 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126859.662020 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126859.663040 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
1379126859.664030 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 … … … … … …
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
VHDX Handle – Windows XP SP3
19
Start Length Tags Access Mask Share Flags Create Options 1378998568.137740 12.084596 ['DH2Q' 'QFid' 'RqLs'] 0x120089 0x7 0x400060 1378998568.238960 11.983569 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.239860 11.98279 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.240740 11.982038 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.241500 11.981468 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.242230 9999999999 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.242950 11.980128 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378998568.243260 0.071878 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378998568.312350 11.910827 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998568.315360 598.671185 ['DH2Q' 'RqLs'] 0x12019f 0x1 0x48 1378998627.382870 0.008148 0x100080 0x7 0x400060 1378998627.391410 12.39557 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.392170 12.395022 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.392950 12.394439 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.393670 12.393823 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998627.394340 12.393332 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378998978.645760 0.0013 0x100080 0x7 0x400060 1378998978.647370 11.562656 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.648080 11.562144 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.648790 11.56155 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.649480 11.56098 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378998978.650150 11.560427 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378999166.986770 9999999999 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378999166.988020 9999999999 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
VHDX Handle – Windows 2012
20
Start Length Tags Access Mask Share Flags Create Options 1378994455.071740 8.265029 ['DH2Q' 'QFid' 'RqLs'] 0x120089 0x7 0x400060 1378994455.242910 38.686086 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.244040 38.685176 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.244800 38.684545 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.245520 38.683969 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.246210 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.246910 38.682697 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378994455.247200 0.066620 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378994455.311260 38.618467 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994455.314040 617.598727 ['DH2Q' 'RqLs'] 0x12019f 0x1 0x48 1378994514.122400 0.007406 0x100080 0x7 0x400060 1378994514.130140 12.115971 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.130890 12.115480 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.131590 12.114990 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.132260 12.114522 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994514.132920 12.113976 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378994832.641740 0.000866 0x100080 0x7 0x400060 1378994832.642900 12.051110 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.643690 12.050523 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.644390 12.049935 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.645060 12.049385 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378994832.645710 12.048850 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378995056.964460 18.050300 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378995056.965210 18.049824 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48 1378995056.965970 18.049272 ['DH2Q' 'RqLs'] 0x120089 0x3 0x40 1378995072.912980 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x1 0x48 1378995072.914410 9999999999.000000 ['DH2Q' 'RqLs'] 0x120089 0x3 0x48
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
0
20
40
60
80
100
120
140
VHDX – Write (Linux)
I/O Size - Linux (Ubuntu 12.04)
21
0
500
1000
1500
2000
2500
3000
VHDX – Read (Linux)
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
I/O Size - Windows XP SP3
22
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Read - WinXP VHDX
0
200
400
600
800
1000
1200
512
1024
1536
2048
2560
3072
3584
4096
4608
5120
5632
6656
7168
8192
9216
9728
1075
212
288
1638
420
480
2457
628
672
3686
440
960
4505
653
248
5734
465
536
Write - WinXP VHDX
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
I/O Size - Windows Server 2012
23
0
500
1000
1500
2000
2500
3000
3500
4000
4500
512
2560
4608
6656
8704
1075
212
800
1484
816
896
1894
420
992
2304
025
088
2713
629
184
3123
233
280
3532
837
888
3993
642
496
4556
847
616
5017
653
760
5632
058
368
6041
662
976
6502
467
072
6963
271
680
7475
277
312
8345
687
040
9011
293
184
9728
010
2400
1080
3211
1616
1146
8812
2368
1305
60
Read – Win2012 VDHX
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
I/O Size - Windows Server 2012
24
0
100
200
300
400
500
600
700
800
Write – Win2012 VHDX
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Definition: Jump Distance
25
Time Jump Distance Offset Read Size
1378952386.399140 0 0 4096
1378952386.411040 61440 65536 4096
1378952386.412700 61440 131072 4096
1378952386.413250 61440 196608 65536
1378952386.415470 1835008 2097152 65536
1378952386.425300 0 2162688 4096
1378952386.428070 978944 3145728 1048576
1378952386.538270 -4194304 0 512
1378952388.136860 37748224 37748736 32768
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump – Linux (Ubuntu 12.04)
26
-5E+09
-4E+09
-3E+09
-2E+09
-1E+09
0
1E+09
2E+09
3E+09
4E+09
5E+09
Read Jump Distances: 0 – 633 sec
-5E+09
-4E+09
-3E+09
-2E+09
-1E+09
0
1E+09
2E+09
3E+09
4E+09
5E+09
Read Jump Distances: 0 – 30 sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump – Linux (Ubuntu 12.04)
27
-5E+09
-4E+09
-3E+09
-2E+09
-1E+09
0
1E+09
2E+09
3E+09
4E+09
5E+09
Read Jump Distance: 22 – 30 sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump – Linux (Ubuntu 12.04)
28
-4E+09
-3E+09
-2E+09
-1E+09
0
1E+09
2E+09
3E+09
4E+09
Write Jump Distance: 0 – 633 sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump - Windows XP SP3
29
-2E+09
-1.5E+09
-1E+09
-5E+08
0
500000000
1E+09
1.5E+09
2E+09
Read Jump Distance: 0 – 90 sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump - Windows XP SP3
30
-2E+09
-1.5E+09
-1E+09
-5E+08
0
500000000
1E+09
1.5E+09
2E+09
Write Jump Distance: 0 – 140 sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump - Windows Server 2012
31
-1E+10
-8E+09
-6E+09
-4E+09
-2E+09
0
2E+09
4E+09
6E+09
8E+09
1E+10
Read Jump Distance: 0 – 240 sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump - Windows Server 2012
32
-1E+10
-8E+09
-6E+09
-4E+09
-2E+09
0
2E+09
4E+09
6E+09
8E+09
1E+10
Write Jump Distance: 0 – 440sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Boot – Single Host Summary
All reads / writes to the VHDX file are in multiples of 512 bytes Favorites are 4Kb, 32Kb, and 128Kb
Size and jump distance distribution changes with
guest OS
Multiple persistent handles (4 or more) opened per VM instance; only one opened for write access 33
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Scenario – Kernel Compile
Linux
34
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Kernel Build Script
35
#!/bin/bash scp -p nas1:~/tmp/linux-3.10.4.tar . tar xf linux-3.10.4.tar cd linux-3.10.4 make defconfig && make -j2 all cd .. /bin/rm -rf linux-3.10.4* sync shutdown -h now
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
36
1x vCPU (1080 sec) 4x vCPU (360 sec) Command Run1 Run2 Run1 Run2 Run3 Negotiate 0.00% 0.00% 0.00% 0.00% 0.00%
SessionSetup 0.22% 0.23% 0.19% 0.28% 0.16% Logoff 0.21% 0.22% 0.18% 0.27% 0.16%
TreeConnect 0.22% 0.23% 0.19% 0.28% 0.16% TreeDisconnect 0.22% 0.22% 0.19% 0.27% 0.16%
Create 0.80% 0.80% 0.66% 0.96% 0.58% Close 0.18% 0.17% 0.12% 0.17% 0.12% Flush 0.94% 0.63% 0.45% 0.26% 0.43% Read 34.88% 34.25% 37.87% 44.77% 43.79%
Write 36.53% 44.25% 46.89% 42.91% 48.96% Lock 0.04% 0.04% 0.03% 0.05% 0.04% IoCtl 0.55% 0.56% 0.45% 0.67% 0.40%
Cancel 0.00% 0.00% 0.00% 0.00% 0.00% Echo 0.00% 0.00% 0.00% 0.00% 0.00%
QueryDirectory 0.00% 0.00% 0.00% 0.00% 0.00% ChangeNotify 0.18% 0.19% 0.16% 0.23% 0.14%
QueryInfo 24.74% 18.17% 12.58% 8.84% 4.85% SetInfo 0.28% 0.03% 0.03% 0.04% 0.05%
OplockBreak 0.00% 0.00% 0.00% 0.00% 0.00%
Command Distribution
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
37
1x vCPU (1080 sec) 4x vCPU (360 sec) Command Run1 Run2 Run1 Run2 Run3 Negotiate 1 1 1 1 1
SessionSetup 60 60 60 60 60 Logoff 58 58 58 58 58
TreeConnect 61 61 61 61 61 TreeDisconnect 59 59 59 59 59
Create 218 213 207 205 215 Close 49 44 38 36 44 Flush 255 168 143 55 160 Read 9461 9070 11929 9609 16304
Write 9909 11718 14769 9210 18229 Lock 10 10 10 10 16 IoCtl 149 149 143 143 148
Cancel 0 0 0 0 0 Echo 0 0 0 0 0
QueryDirectory 0 0 0 0 0 ChangeNotify 50 50 50 50 53
QueryInfo 6711 4813 3962 1898 1805 SetInfo 75 9 9 9 17
OplockBreak 0 0 0 0 0
Total Requests 27126 26483 31499 21464 37230
Command Occurrence
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
I/O Size – Kernel Build (4x vCPU)
38
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
VHDX Read Sizes
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
I/O Size – Kernel Build (4x vCPU)
39
0
1000
2000
3000
4000
5000
6000
VHDX Write Sizes
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump – Kernel Build (4x vCPU)
40
-6E+09
-4E+09
-2E+09
0
2E+09
4E+09
6E+09
Read jump Distance: 0 – 360 sec
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump – Kernel Build (4x vCPU)
41
-6E+09
-4E+09
-2E+09
0
2E+09
4E+09
6E+09
Chart Title
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Scenario – Random Tree Copy
Windows Server 2012
42
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Scenario Details
Guest using SCSI virtual disk Tree copy of deep directory tree 5.8 Gb of allocated space 25,000 files 2,500 directories
Concurrent copies using explorer.exe robocopy.exe
43
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
44
Command Distribution
Command Occurrences Negotiate 0 0.00%
SessionSetup 1 0.00% Logoff 0 0.00%
TreeConnect 2 0.00% TreeDisconnect 1 0.00%
Create 55 0.07% Close 52 0.06% Flush 1089 1.30% Read 51900 61.95%
Write 26789 31.98% Lock 19 0.02% IoCtl 2653 3.17%
Cancel 0 0.00% Echo 0 0.00%
QueryDirectory 0 0.00% ChangeNotify 4 0.00%
QueryInfo 1131 1.35% SetInfo 78 0.09%
OplockBreak 0 0.00%
Total Requests 83774
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
I/O Size – Random Tree Copy
45
0
500
1000
1500
2000
2500
3000
512 1024 2048 4096 8192 12288 16384 20480 31232 32768 65536 122880 131072 917504 925696 1048576
SCSI VHDX Read Sizes
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
I/O Size – Random Tree Copy
46
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
4096
2048
036
864
5324
869
632
8601
610
2400
1187
8413
5168
1515
5216
7936
1843
2020
0704
2170
8823
3472
2498
5626
6240
2826
2429
9008
3153
9233
1776
3481
6036
4544
3809
2839
7312
4136
9643
0080
4464
6446
6944
4833
2850
7904
5242
8854
0672
5570
5657
3440
5980
1661
8496
6430
7266
7648
6922
2470
8608
7290
8874
5472
7659
5278
6432
8110
0883
1488
8519
6887
2448
9093
1293
3888
9707
5210
0352
010
1990
410
3628
8
SCSI VHDX Write Sizes
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Jump – Random Tree Copy
47
-3E+09
-2E+09
-1E+09
0
1E+09
2E+09
3E+09
Write Jump Distance
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Closing Thoughts
48
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Conclusions
SMB3 command distribution shows heavy read/write but with large QueryFileInfo calls to the machine’s XML config
Hyper-V commonly uses larger SMB2 I/O sizes than, but LargeMTU does not appear critical
Pattern shows multiple long lived file handles per guest instance
Read and Write requests are not interleaved on the same handle
49
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Future Work
(Re-)expand scope to cover SMBDirect, Multi-Channel, and Failover
Expand to include more guest workloads Apache, MS-SQL Server, etc…
Expand to include other generalized SMB3
workloads Home directories
50
2013 Storage Developer Conference. © Gerald Carter, EMC. All Rights Reserved.
Questions or Comments?
Gerald Carter [email protected]
EMC/Isilon
51