san school book
TRANSCRIPT
-
7/29/2019 San School Book
1/71
STORAGE ARCHITECTURE/
GETTING STARTED:SAN SCHOOL 101Marc Farley
President of Building Storage, Inc
Author, Building Storage Networks, Inc.
-
7/29/2019 San School Book
2/71
Agenda
Lesson 1: Basics of SANs
Lesson 2: The I/O path
Lesson 3: Storage subsystems
Lesson 4: RAID, volume managementand virtualization
Lesson 5: SAN network technology
Lesson 6: File systems
-
7/29/2019 San School Book
3/71
Basics of storage networking
Lesson #1
-
7/29/2019 San School Book
4/71
Router
Concentrator
Bridge
Dish
Network Switch/hub
Computer
System
HBA or
NIC
VPN
Connecting
-
7/29/2019 San School Book
5/71
Networking or bus technology
Cables + connectors
System adapters + network device drivers
Network devices such as hubs, switches, routers
Virtual networking
Flow control
Network security
Connecting
-
7/29/2019 San School Book
6/71
Tape
Drives
Disk
Drives
RAID
Subsystem
Volume
Manager
Software
Mirroring
Software
(wiring, network transmission frame)
Storage Command and Transfer Protocol
Storage
Device
Drivers
HostSoftware
Storage
Protocol
Storage
Devices
Storing
-
7/29/2019 San School Book
7/71
Device (target) command and control Drives, subsystems, device emulation
Block storage address space
manipulation (partition management) Mirroring
RAID Striping
Virtualization
Concatentation
Storing
-
7/29/2019 San School Book
8/71
User/Application View
C:\directory\file
User/Application View
Database
Object
(Storage)
Logical Block Mapping
Filing
-
7/29/2019 San School Book
9/71
Namespace presents data to end users andapplications as files and directories (folders)
Manages use of storage address spaces
Metadata for identifying data
file name
owner dates
Filing
-
7/29/2019 San School Book
10/71
Connecting, storing and filing as a
complete storage system
Computer System
HBA or
NIC
Network Switch/hub
Disk Drive
Wiring StoringFiling
Storing function in
an HBA driver
Cable Cable
Connecting
-
7/29/2019 San School Book
11/71
NAS and SAN analysis
NAS is filing over a network
SAN is storing over a network
NAS and SAN are independent technologies
They can be implemented independently
They can co-exist in the same environment
They can both operate and provide services to the same
users/applications
-
7/29/2019 San School Book
12/71
Protocol analysis for NASand SAN
Storing
Wiring
FilingNAS
SAN
Network
Filing
Connecting
Storing
-
7/29/2019 San School Book
13/71
Storing
Wiring
Filing
NAS Head
ServerSystem
NAS
Client
Wiring
SAN
Storage
NAS Server
+SAN Initiator
NAS HeadSAN
Target
Connecting
Storing
Connecting
Filing
Integrated SAN/NAS environment
-
7/29/2019 San School Book
14/71
Storing
Wiring
Filing
NAS Head
ServerSystem
NAS
ClientSAN
Storage
SAN
Target
Storing
Connecting
Filing
NAS Head
Common wiring with NASand SAN
-
7/29/2019 San School Book
15/71
The I/O path
Lesson #2
-
7/29/2019 San School Book
16/71
Host hardware path components
Processor MemoryBus
SystemI/O Bus
Storage
Adapter(HBA)
Memory
-
7/29/2019 San School Book
17/71
Host software path components
ApplicationOperatingSystem
FilingSystem
VolumeManager
Device
Driver
Multi-Pathing
CacheManager
-
7/29/2019 San School Book
18/71
Network hardware path components
Cabling
Fiber opticCopper
Switches, hubs, routers, bridges, gatways
Port buffers, processors
Backplane, bus, crossbar, mesh, memory
-
7/29/2019 San School Book
19/71
RoutingFabricServices
Virtual
Networking
Access and
SecurityFlow
Control
Network software path components
-
7/29/2019 San School Book
20/71
Network
Ports
Access and
SecurityInternal Busor Network
Cache Resource
Manager
Subsystem path components
-
7/29/2019 San School Book
21/71
Disk drives Tape drives Solid state devicesTape Media
Device and media path components
-
7/29/2019 San School Book
22/71
The end to end I/O path picture
Disk
drives
Tape
drives
Network Systems
Access
and
Security
Internal Bus
or NetworkCache
Cabling RoutingFabric
Services
Virtual
NetworkingAccess and
Security
ProcessorMemory
Bus
SystemI/O Bus
Storage
Adapter
(HBA)
MemoryApp Operating
System
FilingSystem
VolumeManager
Device
DriverMulti-
Pathing
CacheManager
Subsystem
Network PoirtResource
Manager
Flow
Control
-
7/29/2019 San School Book
23/71
Storage subsystems
Lesson #3
-
7/29/2019 San School Book
24/71
StorageResources
Generic storage subsystem model
NetworkPorts
Cache Memory
Controller (logic+processors)
Access control
Resource manager
Internal Bus
or Network Power
-
7/29/2019 San School Book
25/71
Redundancy for high availability
Multiple hot swappable power supplies
Hot swappable cooling fans
Data redundancy via RAID
Multi-path support Network ports to storage resources
-
7/29/2019 San School Book
26/71
Physical and virtual storage
Physicalstorage
device
Physicalstorage
device
Physicalstorage
device
Physicalstorage
device
Subsystem Controller
Resource Manager
(RAID, mirroring,
etc.)
Exported
storage
Exported
storage
Exported
storage
Exported
storage
Exported
storage
Exported
storage
Exported
storage
Exported
storageHotSpare
Device
-
7/29/2019 San School Book
27/71
SCSI communications are independent of connectivity
SCSI initiators (HBAs) generate I/O activityThey communicate with targets
Targets have communications addresses
Targets can have many storage resources
Each resource is a single SCSI logical unit (LU) with a universal
unique ID (UUID) - sometimes referred to as a serial number An LU can be represented by multiple logical unit numbers (LUNs)
Provisioning associates LUNs with LUs & subsystem ports
A storage resource is not a LUN, its an LU
SCSI communications architecturesdetermine SAN operations
-
7/29/2019 San School Book
28/71
Provisioning storage
Physicalstorage
devices
Physicalstorage
devices
Physicalstorage
devices
SCSI LU
UUID B
SCSI LU
UUID A
SCSI LUUUID C
SCSI LUUUID D
Port S4
Port S3
Port S2
Port S1
LUN 0
LUN 1
LUN 1
LUN 2
LUN 2
LUN 3
LUN 3
LUN 0
Controller functions
-
7/29/2019 San School Book
29/71
-
7/29/2019 San School Book
30/71
Exported
Volume
Exported
Volume
Exported
Volume
Exported
Volume
Controller Cache Manager
Read Caches
1. Recently Used
2. Read Ahead
Write Caches
1. Write Through (to disk)
2. Write Back (from cache)
Caching
-
7/29/2019 San School Book
31/71
Tape Subsystem Controller
Tape SlotsRobot
Tape subsystems
TapeDrive
TapeDrive
TapeDrive
TapeDrive
-
7/29/2019 San School Book
32/71
ExportedStorageResource
Management station
browser-based
network mgmt software
Ethernet/TCP/IP
Out-of-band management port
Storage SubsystemIn-band
management
Now
withSMIS
NowwithSMIS
Subsystem management
-
7/29/2019 San School Book
33/71
Duplication
Parity
Difference
2n
n+1
-1
d(x) = f(x) f(x-1) f(x-1)
Data redundancy
-
7/29/2019 San School Book
34/71
I/O PathMirroringOperator
I/O PathAA
I/O PathBBTerminate I/O & regenerate new I/Os
Error recovery/notification
Host-based Within a subsystem
Duplication redundancy with mirroring
-
7/29/2019 San School Book
35/71
Host
Uni-directional
(writes only)
A BAAA
Duplication redundancy with remote copy
-
7/29/2019 San School Book
36/71
Subsystem Snapshot
Host
A BAAA C
Point-in-time snapshot
-
7/29/2019 San School Book
37/71
Lesson #4
RAID, volume management
and virtualization
-
7/29/2019 San School Book
38/71
-
7/29/2019 San School Book
39/71
Late 1980s R&D project at UC Berkeley David Patterson
Garth Gibson
(independent)
Redundant array of inexpensive disks Striping without redundancy was not defined (RAID 0)
Original goals were to reduce the cost and
increase the capacity of large disk storage
History of RAID
-
7/29/2019 San School Book
40/71
Capacity scaling Combine multiple address spaces as a single virtual
address
Performance through parallelism Spread I/Os over multiple disk spindles
Reliability/availability with redundancy Disk mirroring (striping to 2 disks)
Parity RAID (striping to more than 2 disks)
Benefits of RAID
-
7/29/2019 San School Book
41/71
RAID
Controller(resource
manager)
Storage
extent 1
Exported
RAID
disk
volume
(1 address)
Storage
extent 2
Storage
extent 3
Storage
extent 4
Storage
extent 5
Storage
extent 6
Storage
extent 7
Storage
extent 8
Storage
extent 9
Storage
extent10
Storage
extent11
Storage
extent12
Combined extents1 - 12
Capacity scaling
-
7/29/2019 San School Book
42/71
RAID controller (microsecond performance)
Diskdrive
Diskdrive
Diskdrive
Disk
drive
Disk
drive
Disk
drive
Disk drives (Millisecond performance)
from rotational latency and seek time
12 3 4 5
6
Performance
-
7/29/2019 San School Book
43/71
RAID arrays use XOR for calculating parity
Operand 1 Operand 2 XOR ResultFalse False FalseFalse True TrueTrue False TrueTrue True False
XOR is the inverse of itself
Apply XOR in the table above from right to left
Apply XOR to any two columns to get the third
Parity redundancy
-
7/29/2019 San School Book
44/71
Reduced mode operations
When a member ismissing, data that isaccessed must be
reconstructed with xor
An array that isreconstructing data is saidto be operating in reduced
mode
System performanceduring reduced modeoperations can besignificantly reduced
XOR {M1&M
2&M
3&P}
-
7/29/2019 San School Book
45/71
RAID Parity RebuildThe process of recreating data on a replacement memberis called a parity rebuild
Parity rebuilds are often scheduled for non-production
hours because performance disruptions can be so severe
XOR {M1&M
2&M
3&P}
Parity rebuild
-
7/29/2019 San School Book
46/71
Hybrid RAID: 0+1
Diskdrive
Diskdrive
Diskdrive
Diskdrive
1 2 3 4 5
Diskdrive
Diskdrive
Diskdrive
Diskdrive
Diskdrive
Diskdrive
Mirrored pairs of striped members
RAID 0+1, 10
RAID Controller
-
7/29/2019 San School Book
47/71
Volume management and virtualization
Storing level functions
Provide RAID-like functionality in host systems
and SAN network systems
Aggregation of storage resources for:
scalability
availability
cost / efficiency
manageability
-
7/29/2019 San School Book
48/71
RAID & partitionmanagement
Device driver layer
between the kerneland storage I/Odrivers
OS kernel File system
Volume Manager
HBA drivers
HBAs
Volume Manager
Volume management
-
7/29/2019 San School Book
49/71
SAN disk
resources
Volume
manager
Virtual
Storage
Server system
HBA drivers
SAN HBA
SCSI BusSCSI HBA
SCSI disk resource
SAN SwitchSAN cable
Volume managers can use allavailable connections and
resources and can span multipleSANs as well as SCSI and SAN
resources
-
7/29/2019 San School Book
50/71
RAID and partition management in SAN systems
Two architectures:
In-band virtualization (synchronous)
Out-of-band virtualization (asynchronous)
SAN storage virtualization
-
7/29/2019 San School Book
51/71
Disk
subsystems
SANvirtualizationsystem
Exported virtual storage
I/O Path
System(s),
switch or
router
In-band virtualization
-
7/29/2019 San School Book
52/71
Distributed volumemanagement
Virtualization agentsare managed from acentral system in theSAN
Virtualizationagents
Disk
subsystems
Virtualizationmanagementsystem
Out-of-band virtualization
-
7/29/2019 San School Book
53/71
Lesson #5
SAN networks
-
7/29/2019 San School Book
54/71
The first major SAN networking technology
Very low latency
High reliability Fiber optic cables Copper cables
Extended distance
1, 2 or 4 Gb transmission speeds
Strongly typed
Fibre channel
-
7/29/2019 San School Book
55/71
A Fibre Channel fabric presents a consistent interfaceand set of services across all switches in a network
Host and subsystems all 'see' the same resources
SAN
TargetStorage
Subsystem
SAN
TargetStorage
Subsystem
SAN
TargetStorage
Subsystem
Fibre channel
-
7/29/2019 San School Book
56/71
FC ports are defined by their network role
N-ports: end node ports connecting to fabrics
L-ports: end node ports connecting to loops NL-ports: end node ports connecting to fabrics or loops
F-ports: switch ports connecting to N ports
FL-ports: switch ports connecting to N ports or NL ports in
a loop E-ports: switch ports connecting to other switch ports
G ports: generic switch ports that can be F, FL or E ports
Fibre channel port definitions
-
7/29/2019 San School Book
57/71
Ethernet / TCP / IP SAN technologies
Leveraging the install base ofEthernet and TCP/IP networks
iSCSI native SAN over IP
FC/IP FC SAN extensions over IP
-
7/29/2019 San School Book
58/71
Native storage I/O over TCP/IP New industry standard
Locally over Gigabit Ethernet
Remotely over ATM, SONET, 10Gb Ethernet
iSCSI
TCP
IP
MAC
PHY
iSCSI
-
7/29/2019 San School Book
59/71
Storage NICs (HBAs)
SCSI drivers
Cables Copper and fiber
Network systems Switches/routers
Firewalls
iSCSI equipment
-
7/29/2019 San School Book
60/71
FC/IP
Extending FC SANs over TCP/IP networks
FCIP gateways operate as virtual E-port connections
FCIP creates a single fabric where all resources appear
to be local
FCIPGateway
FCIPGateway
FCIP
Gateway
FCIP
Gateway
TCP/IP
LAN, MANor WANE-port E-port
One fabricOne fabric
-
7/29/2019 San School Book
61/71
SAN switching & fabrics
High-end SAN switches have latencies of 1 - 3
sec
Transaction processing requires lowest latency Most other applications do not
Transaction processing requires non-blockingswitches
No internal delays preventing data transfers
-
7/29/2019 San School Book
62/71
Switches 8 48 ports
Redundant power supplies Single system supervisor
Directors 64+ ports
HA redundancy
Dual system supervisor
Live SW upgrades
Switches and directors
-
7/29/2019 San School Book
63/71
Star
Simplest
single hop
Dual star
Simple network
+ redundancy
Single hop
Independent or integrated
fabric(s)
SAN topologies
-
7/29/2019 San School Book
64/71
N-wide star Scalable
Single hop
Independent or
integrated fabric(s)
Core - edge Scalable
1 3 hops
integrated fabric
SAN topologies
-
7/29/2019 San School Book
65/71
Ring
Scalable
integrated fabric
1 to N2 hops
Ring + Star
Scalable
integrated fabric
1 to 3 hops
SAN topologies
-
7/29/2019 San School Book
66/71
Lesson #6
File systems
-
7/29/2019 San School Book
67/71
Name space
Access control
Metadata
Locking
Address space management
File system functions
-
7/29/2019 San School Book
68/71
StoringStoringFilingFiling
-
7/29/2019 San School Book
69/71
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 1819 20 21 . . . . 26 27
28 29 . . . . . 35 36
37 38 . . . . . 44 45
46 . . . . . . 53 54
55 . . . . . . 62 6364 . . . . . . 71 72
73 . . . . . . 80 81
82 83 84 85 86 87 88 89 90
Think of the storage address space as a sequenceof storage locations (a flat address space)
-
7/29/2019 San School Book
70/71
SuperblocksSuperblocks are known addresses used to findfile system roots (and mount the file system)
SB
SB
-
7/29/2019 San School Book
71/71
Filing and Scaling
StoringStoringStoringStoringFilingFiling
1 2 3 4 5
6 7 8 9 10
11 12 13 14 15
16 17 18 19 20
21 22 23 24 25
1 2 3 4 5 6
7 8 9 10 11 12
13 14 15 16 17 18
19 20 21 22 23 24
25 26 27 28 29 30
31 32 33 34 35 36
37 38 39 40 41 42
File systems must have a known anddependable address space The fine print in scalability - How does the filing function
know about the new storing address space?