tunneling scsi over smb: shared vhdx files for guest clustering in windows server 2012 … ·...
TRANSCRIPT
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Tunneling SCSI over SMB:
Shared VHDX files
for Guest Clustering
in Windows Server 2012 R2
Jose Barreto, Program Manager, Microsoft
Matt Kurjanowicz, Software Development Engineer, Microsoft
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
The need for Shared VHDX
Virtualization is everywhere
High availability is a desired feature
There are two main methods to accomplish it
Host Clustering
Guest Clustering
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Two types of Clustering
Host Clustering Guest Clustering
4
Hyper-V 2Hyper-V 1
VM
Hyper-V 2Hyper-V 1
VM2VM1
Shared Storage
App App
Shared Storage
FailoverFailover App
Requires iSCSI to the Guest or Virtual FC.
Exposes shared storage infrastructure to VM.
Could we simplify it?
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
File Storage
Block Storage
Introducing: Shared Virtual Disks
VHDX
VHDX
Guest Clustering
Simply share virtual disks to provide
shared storage for Guest Clustering
Hyper-V Guest Failover Clustering
with commodity storage
Maintains separation between
infrastructure and tenants
Virtual SAS
VM presented with virtual SAS disks
Shared SAS disks, ready to cluster
Used for data disk only
Cluster Shared Volumes (CSV)
for block storage
Scale-Out File Server
for file storage
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Hyper-V Parent 2Hyper-V Parent 1
CSV
Shared VHDX Configurations
VM2
Config
VHDX2Boot
Disk
Shared SAS Storage
Disk2 Disk4Disk1 Disk3
Shared
Disk
VHDX3
P
Disk5
Storage Spaces
VHDX2
VM1
Config
VHDX1Boot
Disk
Shared
Disk
P
VHDX1
Space1
S
Regular VHDX
Shared VHDX
Hyper-V Parent 2
VM2
Config
VHDX
2
Boot
Disk
File Server 1
Share1
File Server 2
Share1
Shared SAS Storage
Disk2 Disk4Disk1 Disk3
Shared
Disk
VHDX3
P
Disk5
Storage Spaces
VHDX2
S
Hyper-V Parent 1
VM1
Config
VHDX
1
Boot
Disk
Shared
DiskP
VHDX1
Space1
CSV
SMB
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Deploying Shared VHDX
Deployment options
Hyper-V Manager GUI
PowerShell
VMM 2012 R2 (Service Template)
Inside the VM
Disks behave just like regular disks
Two things to watch: anti-affinity and heartbeating
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Shared Virtual Disks - GUI
In Hyper-V Manager, go to VM Settings
Add a new hard drive (make sure it’s a VHDX, not an old VHD)
Expand “Advanced Features” under “Hard Drive”
Check the box for “Enable virtual hard disk sharing”
Hit “Apply” only after you checked the box
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Shared Virtual Disks - PowerShell
1 ShareVirtualDisk
2 ShareVirtualDisk
SupportPersistentReservations
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Inside the VM – Just a regular disk
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Experiencing Shared VHDX
Recorded demo from TechEd 2013 Conference
https://skydrive.live.com/redir?resid=DFBE7FC34A56
43C6!12108&authkey=!AABY-a4Zro0Kyus
Step-by-step for Shared VHDX
http://blogs.technet.com/b/josebda/archive/2013/07/3
1/windows-server-2012-r2-storage-step-by-step-with-
storage-spaces-smb-scale-out-and-shared-vhdx-
physical.aspx
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Supported OS versions
For Hyper-V hosts and File Server
Must be running Windows Server 2012 R2
For Guests (Virtual Machines)
Windows Server 2012 or 2012 R2 are supported
Install the Hyper-V Integration Components
Nothing architecturally preventing the use of older
versions, but we do not have test coverage
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Supported virtual disk formats
Data disks, not OS disks
OS disks always require exclusive access
VHDX, not VHD
Metadata (persistent reservations) stored on the
VHDX file header
Fixed or Dynamic, not Differencing
Differing does not make sense for a data VHDX
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Backup
As with all other guest cluster options (iSCSI to
the guest and Virtual FC), backup from the guest
Host-based backups and snapshots not
supported
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Failover and Move
Supported Failover scenarios
Single guest (virtual machine) failover
Single Hyper-V host failover
Single File Server node failover
Also works when you
Move the CSV volume
Move the Virtual Machines
Storage Migration not supported
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Comparing Guest Cluster Options
Capability Shared VHDX Virtual FC iSCSI to the
Guest
Supported storage back-ends SAS, FC, iSCSI,
SMB, Spaces
FC SAN iSCSI SAN
How storage is exposed in the guest as Virtual SAS Virtual FC LUN iSCSI LUN
Flows through the Hyper-V virtual switch No No Yes
VM storage configured at the Hyper-V host Yes Yes No
Provides solution for low latency/low CPU use Yes (RDMA, FC) Yes (FC) No
Requires specific hardware on Hyper-V host No Yes No
Request switch reconfiguration on migration No Yes No
Exposes hoster storage architecture to guests No Yes Yes
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
File Server
Hyper-V
File Server
Hyper-V
Comparing two stacks
“Classic” Hyper-V over SMB Shared Virtual Disk
17
VM
Disk
SMB Client (RDR)
SMB Server (SRV)
Storage VSP (StorVSP)
VHD Stack (VHDMP)
File System (NTFS)
VM
Disk
SMB Client (RDR)
SMB Server (SRV)
Storage VSP (StorVSP)
VHD Stack (VHDMP)
File System (NTFS)
Parser (PVHDPARSER)
Shared VHDX (SVHDXFLT)
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Why RSVD?
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
General Protocol Characteristics
Built on top of SMB3
“Block Semantics” on top of “File Semantics”
Most SCSI commands simply encapsulated
and forwarded
Client driven
Failover through SMB Continuous Availability
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Session establishment sequence
Client Server
SMB3 Connection Establishment
IRP_MJ_CREATE
Get Starting Information
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Characteristics of a SVHDX Open
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Open as Block Device ECP
GUID: {9ECFCB9C-C104-43E6-980E-158DA1F6EC83}
Contents:
Client version
Initiator ID
Initiator Hostname
Flags describing the open
Flags describing where the open came from
Request ID
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Versioning
Client identifies version in ECP
Server identifies version in Get Starting
Information tunneled command
STATUS_SVHDX_VERSION_MISMATCH when
invalid tunneled operations are received
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
RSVD_SYNC_TUNNEL_REQUEST
0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9
20 1 2 3 4 5 6 7 8 9
30 1
OperationCode (ULONG)
StatusCode (NTSTATUS)
Request ID (ULONGLONG)
…
Payload (Variable Length)
64KB Max Tunnel Size
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Get Starting Information (0x02000001)
When: Immediately after CreateFile
Why: Versioning, Layout of the underlying disk
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Get Starting Information
0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9
20 1 2 3 4 5 6 7 8 9
30 1
Header
ServerVersion (ULONG)
SectorSize (ULONG)
PhysicalSectorSize (ULONG)
VirtualSize (ULONGLONG)
…
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
SCSI Command Request (0x02000002)
When: At client application request
Why: Issue a SCSI command to the virtual disk
Reads & Writes handled differently
Important SCSI commands for Clustering
PERSISTENT RESERVE IN
PERSISTENT RESERVE OUT
Not all SCSI commands have been tested
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
SCSI Command Request
0 1 2 3 4 5 6 7 8 9
1
0 1 2 3 4 5 6 7 8 9
2
0 1 2 3 4 5 6 7 8 9
3
0 1
Header
Length Reserved1
CdbLength SenseInfoExLength DataIn Reserved2
SrbFlags
DataTransferLength
CDBBuffer
….
…
…
Reserved3
DataBuffer (variable size)
…
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
SCSI Command Response
0 1 2 3 4 5 6 7 8 9
1
0 1 2 3 4 5 6 7 8 9
2
0 1 2 3 4 5 6 7 8 9
3
0 1
Header
Length SrbStatus ScsiStatus
CdbLength SenseInfoExLength DataIn Reserved
SrbFlags
DataTransferLength
SenseDataEx
….
…
…
…
DataBuffer (variable size)
…
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Check Connection (0x02000003)
When: Periodically, at client application request
Why:
Ensure that the connection is still alive
Ensure that the disk is currently accessible
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Check Connection
0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9
20 1 2 3 4 5 6 7 8 9
30 1
Header
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Get Disk Information (0x02000005)
When: At client application request
Why: To get extended SCSI status information
about the failure
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Get Disk Information
0 1 2 3 4 5 6 7 8 9
1
0 1 2 3 4 5 6 7 8 9
2
0 1 2 3 4 5 6 7 8 9
3
0 1
Header
DiskType
DiskFormat
BlockSize
LinkageID
…
…
…
IsMounted Reserved
FileSize
…
VirtualDiskId
…
…
…
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Validate Disk (0x02000006)
When: At client application request
Why: To ensure that the disk is fully valid for IO
operations
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Validate Disk
0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9
20 1 2 3 4 5 6 7 8 9
30 1
Header
IsValid Reserved
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Moving large amounts of data
SMB optimizes File Read & File Write over
FSCTL
FSCTL 64KB Maximum Payload Size
40
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Block addressing model
2 3
2 VirtualSectorSize3 VirtualSectorSize
1536B1024B
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
SCSI has a lot of error information
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Extended Error Handling
Client Server
IRP_MJ_READ / WRITE
Key = STATUS_SVHDX_ERROR_STORED
IRP_MJ_FILE_SYSTEM_CONTROL(RSVD OpCode == RSVD_TUNNEL_SRB_STATUS_OPERATION,
ErrorKey = Key & 0xFF)
SRB Status, SCSI Status, Sense Data
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Get SRB Status (0x02000004)
When: After a read or write has failed with
certain error codes
Why: To get extended SCSI status information
about the failure
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Get SRB Status
0 1 2 3 4 5 6 7 8 9
1
0 1 2 3 4 5 6 7 8 9
2
0 1 2 3 4 5 6 7 8 9
3
0 1
Header
StatusKey
SrbStatus ScsiStatus SenseInfoExLength Reserved
SenseDataEx
…
…
…
…
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
STATUS_SVHDX_ERROR_STORED
00 FF
The last byte is passed to Get SRB Status
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
File Size
Valid Data Length
End of File
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Sessions terminated through CloseFile
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
FSCTL_QUERY_SHARED_VIRTUAL_DISK_SUPPORT
0 1 2 3 4 5 6 7 8 910 1 2 3 4 5 6 7 8 9
20 1 2 3 4 5 6 7 8 9
30 1
SharedVirtualDiskSupport (0x01)
CurrentSharingStatus
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
STATUS_VHD_SHARED
•
•
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Resources
[MS-RSVD]: Remote Shared Virtual Disk Protocol
Specification
TechEd
Application Availability Strategies for the Private Cloud
TechNet
Deploy a Guest Cluster Using a Shared Virtual Hard Disk
Virtual Hard Disk Sharing Overview
SCSI
SCSI Primary Commands – 3 (SPC-3)
SCSI Block Commands – 2 (SBC-2)
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Q&A
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Compatibility
•
•
•
•
•
•
•
•
•
•
•
•
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Compatibility
•
•
•
•
•
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Blocked things
Unaligned IO
Cached IO
Oplocks
IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHR
ONIZATION
Direct IO
Copy Offload (for SVHDX files)
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Troubleshooting
Performance counters
Events
Filter Manager (FLTMC.EXE)
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Performance Monitor
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Event Viewer
Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Shared-VHDX/Operational
Get-WinEvent -LogName Microsoft-Windows-Hyper-V-Shared-VHDX/Reservation
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Filter Manager (FLTMC.EXE)
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Unsupported Bonus Scenario
Single host Shared Virtual Disk for test/dev/learning scenarios
Install Windows Server 2012 R2 on the host
Enable Failover Clustering feature
o Install-WindowsFeature Failover-Clustering
o No need to actually create a cluster for this scenario
Manually attach the Shared Virtual Disk filter
o FLTMC.EXE attach svhdxflt X:
o Use the step-by-step blog post to try it out… Not a supported scenario
Not really highly available
!
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Hyper-V Cluster
Hyper-V
Host 1
Component view - CSV
Storage
VSC
Boot VM1
Storage
VSPVHD Stack
Physical
NIC
Shared
Data
Shared VHDX
Parser
Shared VDHX Filter
CSV
Hyper-V
Host 2Storage
VSC
Boot VM2
Storage
VSP
CSV2
VHD Stack
Physical
NIC
Shared
Data
Shared VHDX
Parser
Shared VDHX Filter
CSV
Guest Cluster
CSV1
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Hyper-V ClusterHyper-V Cluster
Hyper-V
Host 1
Component view - SMB
Storage
VSC
Boot VM1
Storage
VSPVHD Stack
SMB Client
Shared
Data
Shared VHDX
Parser
Hyper-V
Host 2Storage
VSC
Boot VM2
Storage
VSPVHD Stack
SMB Client
Shared
Data
Shared VHDX
Parser
Guest Cluster
VHD Stack
SMB Server
Shared VDHX
Filter
CSV
SMB Server
FS2
VHD StackShared VDHX
Filter
CSV
FS1
2013 Storage Developer Conference. © Microsoft Corporation. All Rights Reserved.
Shared VHDx Open
•
•
•
•
•