brksan-1121 storage area networking core edge design best...

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public

Storage Area Networking

Core Edge Design

Best Practices

BRKSAN-1121

1

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKSAN-1121

Session BRKSAN-1121 Abstract

SAN Core-Edge Design Best Practices

This session gives non-storage-networking professionals the fundamentals to understand and implement storage

area networks (SANs). This curriculum is intended to prepare attendees for involvement in SAN projects and I/O

Consolidation of Ethernet & Fibre Channel networking. You will be exposed to the introduction of Storage

Networking terminology & Designs. Specific topics covered include Fibre Channel (FC), FCoE, FC services, FC

addressing, fabric routing, zoning, virtual SANs (VSANs). The session includes discussions on Designing Core-

Edge Fibre Channel Networks and the best practice recommendations around them. This is an introductory

session and attendees are encouraged to follow up with other SAN breakout sessions and labs to learn more

about specific advanced topics.

2


Who Am I?

Chad Hintz

Technical Solutions Architect-Data Center/Virtualization

CCIE #15729

Routing & Switching, Security, Storage

3


What Are Storage Area Networks?

4


Session Agenda

History of Storage Area Networks

Fibre Channel Basics

SAN Design Requirements

Introduction to Typical SAN Designs

Introduction to NPIV/NPV

General SAN Best Practices

Core-Edge Design Review

Recommended Core-Edge Designs for Scale and Availability

Next Generation Core-Edge Designs

5


Direct Attached Storage: DAS

DAS – Direct-Attach Storage

Dedicated High Speed Access

Can‘t share capacity

Difficult to share

data

Difficult to manage

7


Network Attached Storage: NAS

NAS

NAS – Network-Attached Storage

Centralized Storage Attached over LAN

File sharing

More efficient

capacity usage

Performance

limits

usefulness of

NAS – mainly

used for file

storage and

low-end

databases

LAN

8


Storage Area Network: SAN

SAN – Storage Area Network

Dedicated ‘Back End’ Network

Match DAS

performance

Capacity

deployed and

redeployed

Centralized

management

Diskless servers

– simplified

management,

reduced power

and cooling

NAS

SAN

LAN

9


SAN Protocols

Fibre Channel

‒ A gigabit-speed network technology primarily used for storage networking

Fibre Channel over Ethernet (FCoE)

‒ An encapsulation of FibreChannel frames over Ethernet networks. This allows

Fibre Channel to use 10 Gigabit Ethernet networks while preserving the Fibre

Channel protocol

iSCSI

‒ A TCP/IP-based protocol for establishing and managing connections between

IP-based storage devices, hosts and clients

10


Session Agenda










11


SAN Components

Servers with host bus adapters

Storage systems

‒ RAID

‒ JBOD

‒ Tape

Switches

SAN management software

13


Types of Fibre Channel Switches

Redundancy/Services

Cisco MDS 9000

MDS 9506, 9509, 9513

MDS 9200

MDS 91XX

Small/Medium Business

Enterprise and Service Provider

FC Bladeswitch

Edge Modular Director

Nexus 5500

14


Fibre Channel Port Types

‗N‘ port: Node ports used to connect devices to switched

fabric or point to point configurations.

‗F‘ port: Fabric ports residing on switches connecting ‗N‘

port devices

‗L‘ port: Loop ports are used in arbitrated loop

configurations to build networks without FC switches.

These ports often also have ‗N‘ port capabilities and are

called ‗NL‘ ports.

‗E‘ port: Expansion ports are essentially trunk ports used

to connect two Fibre Channel switches

‗GL‘ port: A generic port capable of operating as either an

‗E‘ or ‗F‘ port. Its also capable of acting in an ‗FL‘ port

capacity. Auto Discovery.

N N

N F

NL FL

E E

15


G_Port

G_Port

E_Port

F_Port

F_Port

E_Port

N_Port

N_Port

Fibre Channel Port Types

Fibre Channel Switch

NPV

Switch

Input

Port Output

Port

Fabric

X

F_Port NP_Port Fabric

Switch

TE_Port Fabric

Switch TE_Port

End

Node

End

Node

16


Start from the Beginning…

Start with the host and a target that need to

communicate

‒ Host has 2 HBAs (one per fabric) each with a

WWN

‒ Target has multiple ports to connect to fabric

Connect to a FC Switch

‒ Port Type Negotiation

‒ Speed Negotiation

FC Switch is part of the SAN ―fabric‖

Most commonly, dual fabrics are deployed

for redundancy

FC

HBA

Core

Initiator

Target

FABRIC A

Edge

17


My Port Is Up…Can I Talk Now?

FLOGIs/PLOGIs Step 1: Fabric Login (FLOGI)

determines the presence or absence of a Fabric

exchanges Service Parameters with the Fabric

switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID)

initializes the buffer-to-buffer credits

Step 2: Port Login (PLOGI)

required between nodes that want to communicate

similar to FLOGI – transports a PLOGI frame to the designation node port

In p2p topology (no fabric present), initializes buffer-to-buffer credits

N_Port

F_Port

FC

HBA

Core

Initiator

Target

E_Port

Edge

18


Buffer to Buffer Credits

B2B Credits used to ensure that FC transport is lossless

# of credits negotiated between ports when link is

brought up

# Credits decremented with each packet placed on the

wire

‒ Independent of packet size

‒ If # credits == 0, no more packet transmission

# of credits incremented with each

―transfer ready‖ received

B2B Credits need to be taken into

consideration as distance and/or bandwidth increases

Fibre Channel Flow Control

16

16 P

acke

t

15

R_

RD

Y

Host

Fibre Channel Switch

16

19


Fabric Name and Addressing: WWN

Every Fibre Channel port and node has a hard-coded address called

World Wide Name (WWN)

During FLOGI the switch identifies the WWN

in the service parameters of the accept frame

and assigns a Fibre Channel ID (FCID)

Switch Name Server maps WWNs to FCID

‒ WWNN uniquely identify devices

‒ WWPN uniquely identify each port in a device

20


Switch Topology

Model

Switch

Domain Area Device

Fabric Channel ID (FCID)

FCID assigned to every WWPN

corresponding to an N_Port

FCID made up of switch domain, area and

device

Domain ID is native to a single FC switch

limitation of domain IDs in a single fabric

Forwarding decisions made on domain ID

found in first 8 bits of FCID

Fibre Channel Addressing Scheme

FC Fabric

Fabric A

21


Fabric Shortest Path First (like OSPF)

Fibre Channel Forwarding

FSPF ―routes‖ traffic based on destination

domain ID found in the destination FCID

For FSPF a domain ID identifies a single

switch

‒ The number of Domains IDs are limited to 239/75

(theoretical limited/tested and qualified) within the

same fabric (VSAN)

FSPF performs hop-by-hop routing

FSPF uses total cost as the metric to

determine most efficient path

FSPF supports equal cost load balancing

across links

22


Directory Server/Name Server (Like DNS)

Repository of information regarding the components that make up the

Fibre Channel network

Located at address FF FF FC (some readings call this the name server)

Components can register their characteristics with the directory server

An N_Port can query the directory server for specific information

‒ Query can be the address identifier, WWN and volume names for all SCSI

targets

23


Login Complete…

Almost There Fabric Zoning Zones are the basic form of data path

security

Zone members can only ―see‖ and talk to other members of the zone

Devices can be members of more than one zone

Default zoning is ―deny‖

Zones belong to a zoneset

Zoneset must be ―active‖ to enforce zoning

Only one active zoneset per fabric or per VSAN

FC Target

fcid 0x10.00.01 [pwwn 10:00:00:00:c9:76:fd:31] [Initiator]

fcid 0x11.00.01 [pwwn 50:06:01:61:3c:e0:1a:f6] [target]

pwwn 50:06:01:61:3c:e0:1a:f6

FC Fabric

Initiator

pwwn

10:00:00:00:c9:76:fd:31

24


Zoning—Enforcement Zoning is used to control access in a SAN

Soft zoning

‒ Enforced by name server query responses

‒ Name server sends membership list to N_Port

‒ N-port accesses members only

Hard zoning

‒ Enforced by hardware (forwarding ASIC) at wire speed pWWN, fWWN, FC_ID, FC_Alias

Zone-1

Array

Zone-2

Array

Host

FC

MDS MDS

Host

FC

Soft Zone Hard Zone

Zone-1

Array

Zone-2

Array

Host

FC

MDS MDS

Host

FC

25


Hard Zone

26


Soft Zone

27


Enhanced vs. Basic Zoning

Basic Zoning Enhanced Zoning Enhanced Advantages

Administrators can make simultaneous configuration changes

All configuration changes are made within a single session. Switch locks entire fabric to implement change

One configuration session for entire fabric to ensure consistency within fabric

If a zone is a member of multiple zonesets , an instance is created per zoneset.

References to the zone are used by the zonesets as required once you define the zone.

Reduced payload size as the zone is referenced. The size is more pronounced with bigger database

Default zone policy is defined per switch.

Enforces and exchanges default zone setting throughout the fabric

Fabric-wide policy enforcement reduces troubleshooting time.

28


Enhanced vs. Basic Zoning

Basic Zoning Enhanced Zoning Enhanced Advantages

Managing switch provides combined status about activation. Will not identify a failure switch.

Retrieves the activation results and the nature of the problem from each remote switch.

Enhanced error reporting reduces troubleshooting process.

To distribute zoneset must re-activate the same zoneset.

Implements changes to the zoning database and distributes it without activation.

This avoids hardware changes for hard zoning in the switches.

During a merge MDS specific types can be misunderstood by non-cisco switches.

Provides a vendor ID along with a vendor-specific type value to uniquely identify a member type

Unique Vendor type

29


Virtual SANs (VSANs)

Virtual Fabric Separation

Analogous to VLANs in Ethernet

Virtual fabrics created from larger cost-effective redundant physical fabric

Reduces wasted ports of a SAN island approach

Fabric events are isolated per VSAN which gives further isolation for High Availability

Statistics can be gathered per VSAN

Each VSAN provides Separate Fabric Services

‒ FSPF, Zones/Zoneset, DNS, RSCN

A Virtual SAN (VSAN) Provides a Method to

Allocate Ports within a Physical Fabric and

Create Virtual Fabrics

Physical SAN

Islands Are

Virtualizedonto

Common SAN

Infrastructure

VSANs supported on MDS

and Nexus 5000 Product lines

30


SAN Islands – Before VSANs

SAN A

DomainID=1

Production SAN Tape SAN Test SAN

DomainID=8 DomainID=7

SAN B

DomainID=2

SAN D

DomainID=4

SAN F

Domain ID=6

SAN E

DomainID=5

SAN C

DomainID=3

31


SAN Islands – with Virtual SANs

Production SAN Tape SAN Test SAN

SAN B

DomainID=2

SAN D

DomainID=4

SAN F

Domain ID=6 SAN E

DomainID=5

SAN C

DomainID=3

SAN A

DomainID=1

32


VSANs and Zones—Complimentary

Hierarchical relationship—

‒ First assign physical ports to VSANs

‒ Then configure independent zones per VSAN

VSANs divide the physical infrastructure

Zones provide added security and allow sharing

of device ports

Zones can change frequently (e.g. backup)

Ports are added/removed non-disruptively to

VSANs

Virtual SANs and Fabric Zoning Are Very Complementary

VSAN 3

Physical Topology

VSAN 2

Disk1

Host2 Disk4

Host1

Disk2 Disk3

Disk6

Disk5

Host4

Host3

ZoneA

ZoneB

ZoneC

ZoneA

ZoneD

Relationship of VSANs to Zones

33


Over-Subscription

FAN-OUT Ratio

Over-subscription (or fan-out) ratio for sizing ports and links

Factors used

‒ Speed of Host HBA interfaces

‒ Speed of Array interfaces

‒ Type of server and application

Storage vendors provide guidance in the process

Ratios range between 4:1 - 20:1

FC

6 x 4G Array ports 3 x 8G ISL ports

Example:

10:1 O/S ratio 60 Servers with

4 Gb HBAs

240 G 24 G 24 G

34


Inter-Switch Link PortChanneling

Criteria for forming a PortChannel

‒ Same speed links

‒ Same modes (auto, E, etc.) and states

‒ Between same two switches

‒ Same VSAN membership

Treated as one logical ISL by upper layer protocols (FSPF)

Can use up to 16 links in a PortChannel

Can be formed from any ports on any modules—HA enabled

Exchange-based in-order load balancing

‒ Mode one: based on src/dstFC_IDs

‒ Mode two: based on src/dst FC_ID/OX_ID

Much faster recovery than FSPF-based balancing

Given logical interface name with aggregated bandwidth and derived routing metric

A PortChannel Is a Logical Bundling of Identical Links

E.g., 8-Gbps

PortChannel

(Four x 2

Gbps)

E.g., 4-Gbps

PortChannel

(Two x 2 Gbps)

35


PortChannel vs. Trunking

ISL = inter-switch link

PortChannel = E_Ports and ISLs

Trunk = ISLs that support VSANs

Trunking = TE_Ports and EISLs

36


Session Agenda






General SAN Design Best Practices




37


SAN Design

High Availability - Providing a Dual Fabric (current best practice)

Meeting oversubscription ratios established by disk vendors

Effective zoning

Providing Business Function/Operating System Fabric Segmentation and

Security

Fabric scalability (FLOGI and domain-id scaling)

Providing connectivity for virtualized servers

Providing connectivity for diverse server placement and form factors

Key Requirements

39


The Design Requirements

Fibre Channel SAN

‒ Transport and Services are on the same layer in the same devices

‒ Well defined end device relationships (initiators and targets)

‒ Does not tolerate packet drop – requires lossless transport

‒ Only north-south traffic, east-west traffic mostly irrelevant

Network designs optimized for Scale and Availability

‒ High availability of network services provided through dual fabric architecture

‒ Edge/Core vs Edge/Core/Edge

‒ Service deployment

Classical Fibre Channel

Client/Server

Relationships are

pre-defined

I(c)

I(c) T(s)

T2

I5

I4 I3 I2

I1

I0

T1 T0

Switch Switch

Switch

DNS FSPF

Zone RSCN DNS

FSPF Zone

RSCN

DNS

Zone

FSPF

RSCN

Fabric topology, services and

traffic flows are structured

40


Session Agenda










41

Typical SAN Designs


SAN Design – Single Tier Topology

Collapsed Core Design

Servers connect to the Core switches

Storage devices connect to one or more

core switches

Core switches provide storage services

Large amount of blades to support Initiator

(Host) and Target (Storage) ports

Single Management per Fabric

Normal for Small SAN environments

HA achieved in two physically separate, but

identical, redundant SAN fabrics

FC

Core Core

43


How Do We Avoid This?

44


SAN Design – Two Tier Topology

―Core-Edge‖ Topology- Most Common

Servers connect to the edge switches

Storage devices connect to one or more core

switches

Core switches provide storage services to

one or more edge switches, thus servicing

more servers in the fabric

ISLs have to be designed so that overall fan-

in ratio of servers to storage and overall end-

to-end oversubscription are maintained



FC

Core Core

Edge Edge

45


SAN Design – Three Tier Topology

―Edge-Core-Edge‖ Topology


Storage devices connect to one or more edge

switches

Core switches provide storage services to one or

more edge switches, thus servicing more servers

and storage in the fabric

ISLs have to be designed so that overall fan-in

ratio of servers to storage and overall end-to-end

oversubscription are maintained



FC

Core

Edge

Core

Edge

EdgeEdge

46


Session Agenda










47


What Is NPIV? and Why?

N-Port ID Virtualization (NPIV) provides a means to assign multiple FCIDs

to a single N_Port

‒ Limitation exists in FC where only a single FCID can be handed out per F-port.

Therefore and F-Port can only accept a single FLOGI

Allows multiple applications to share the same Fiber Channel adapter port

Usage applies to applications such as Virtualization

Application Server FC NPIV Core Switch

Email

Web

File Services

Email I/O N_Port_ID 1

Web I/O N_Port_ID 2

File Services I/O N_Port_ID 3

F_Port

F_Port

N_Port

49


What Is NPV? and Why? N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a ―switch‖ to act

like a server performing multiple logins through a single physical link

Physical servers connected to the NPV switch login to the upstream NPIV core switch

No local switching is done on an FC switch in NPV mode

FC edge switch in NPV mode does not take up a domain ID

‒ Helps to alleviate domain ID exhaustion in large fabrics

N-Port

Application Server

FC NPIV Core Switch

FC1/1

FC1/2

FC1/3

Server1 N_Port_ID 1

Server2 N_Port_ID 2

Server3 N_Port_ID 3

F_Port

F-Port

F-Port NP-Port

NPV Switch

50


NPV Auto Load Balancing

‒ Server loads are not tied to any

uplink

Benefit

‒ Optimal uplink bandwidth utilization

Uniform balancing of server loads on NP links

Bla

de

1

Bla

de

4

Blade Server Chassis

Bla

de

2

SAN

Balanced

load on

NP links

Bla

de

3

1

3 2

4

51


NPV Auto Load Balancing

Automatically moves the failed

servers to other available NP links

‒ Servers made to re-login

immediately after experiencing

―short‖ traffic disruption.

Benefit

‒ Downtime greatly reduced

Automatic failover of loads on NP links

Bla

de

1

Bla

de

4


Bla

de

2

SAN

Bla

de

3

1

2

3

4

Disrupted

servers re-

login on

other uplink

X

52


F-Port Port Channel

F-Port PortChannels

‒ Bundle multiple ports in to 1 logical

link

‒ Similar to ISL portchannels in FC

and EtherChannels in Ethernet

Benefits

‒ High-Availability- no disruption if

cable, port, or line cards fail

‒ Optimal bandwidth utilization &

higher aggregate bandwidth with

load balancing

Enhance NPV uplink Resiliency Storage

Bla

deS

yste

m

Blade 1

Blade 2

Blade N

F-Port Port

Channel

F-Port NP-Port

Core Director

SAN

Interface port-channel 1

no shut

Interface fc1/1

channel-group 1

Interface fc1/2

channel-group 1

53


NPV F-Port Port Channel

Link failures do not affect the server

connectivity

No application disruption

No traffic disruption

Bla

de

1

Bla

de

4


Bla

de

2

SAN

Bla

de

3

X

54


F-Port Trunking

F-Port Trunking

‒ Uplinks carry multiple VSANs

Benefits

‒ Extend VSAN benefits to Blade

servers

‒ Separate management domains

‒ Traffic Isolation and ability to host

differentiated services on blades

Extend VSAN Benefits to Blades

Storage

Bla

de S

yste

m

Blade N

Core Director

VSAN 1

VSAN 2

VSAN 3

F-Port Trunking

on

F-Port Channel

F-Port NP-Port

SAN

NPV

Interface fc1/1

trunk mode on

trunk allowed-vsan 1-

3

Interface port-channel

1

trunk mode on

trunk allowed-vsan 1-

3

Blade 2

Blade 1

55


Session Agenda










56


SAN Design





Effective zoning


Security


Key Requirements

58


Zones/ZoneSet

Create Device-Alias for End Devices

‒ Create a readable name for end devices tied to their PWWN

‒ As device moves between VSANs their Device Alias stays the same

Create 2 Member Zones

‒ Hardware zoning on MDS

Recommended to have more zones with 2 members in larger SANs

Single Management of Zones/Zoneset per Fabric

‒ Use Distribute full Zoneset command per VSAN to keep from isolation in

‒ Basic Zoning or use Enhanced Zoning.

If a device has different active members the ISL will become isolated

59


SAN Design

Key Requirements





Effective zoning


Security


60


Virtual SANs

Consolidate SAN Islands into OS or

Department VSANs

Reduction of SAN islands into a single

Fabric while keeping Isolation

Example is to have Test, Development and

Production in their own VSANs

Separate Tape or SAN extension VSANs

Security

Create Separate Administrative Roles per

VSANs

Use TACACS+ for authorization and

auditing of Switches

VSANs supported on MDS

and Nexus 7/5x00 Product

lines

61


SAN Design





Effective zoning


Security


Key Requirements

62


SAN Design

Use port-channeling/trunking to enhance bandwidth available between

devices

Factors used

‒ Speed of Host HBA interfaces

‒ Speed of Array interfaces

‒ Type of server and application

Keep ISL Oversubscription ratio lower than Array oversubscription ratio

Ratios range between 4:1 - 20:1

FAN-OUT Ratio

FC

6 x 4G Array ports 3 x 8G ISL ports

Example:

10:1 O/S ratio 60 Servers with

4 Gb HBAs

240 G 24 G 24 G

63


SAN Security Scope

Fabric security augments overall application security ‒ Host and disk security also required

Six key areas of focus 1. SAN management access—secure access to

management services

2. Fabric access—secure device access to fabric service

3. Target access —secure access to targets and LUNs

4. SAN protocols—secure switch-to-switch communication protocols

5. IP storage access—secure FCIP and iSCSI services

6. Data integrity and secrecy—encryption of data in transit and at rest

*Check Reference Slides for more details around SAN security

1.SAN

Management

Security

4. SAN Fabric

Protocol Security

2.Fabric

Access

Security

6.Data Integrity and Secrecy

3.Target

Access

Security

5.IP Storage

Security

(iSCSI/FCIP)

iSCSI

Cisco

MDS 9000

SAN

Target

Host

64


Session Agenda










65


SAN Design – Two Tier Topology

―Edge-Core‖ Topology- Most Common


Storage devices connect to one or more core

switches

Core switches provide storage services to

one or more edge switches, thus servicing

more servers in the fabric

ISLs have to be designed so that overall fan-

in ratio of servers to storage and overall end-

to-end oversubscription are maintained



FC

Core

Edge

Core

Edge

67


Blade Switch Explosion Issues

Scalability

‒ Each Blade Switch uses a single Domain ID

‒ Theoretical maximum number of Domain IDs is 239 per VSAN

‒ Supported number of domains is quite smaller (depends on OSM)

EMC: 40 domains

Cisco Tested: 75

HP: 40 domains

Other OSM Do Not Post

Manageability

‒ More switches to manage

‒ Shared management of blade switches between storage and server administrators

68


Server Consolidation with Top-of-Rack Fabric

Switches

2X ISL to Core at 10 G

32 Host Ports at 4 G

A B 96 Storage Ports at 2 G

28 ISL to Edge at 10 G

14 Racks

32 Dual Attached Servers per Rack

A

B

MDS

91XXs

Top of Rack

Top of Rack Design

Ports Deployed:

Storage Ports (4 G Dedicated):

Host Ports (4 G Shared):

Disk Oversubscription (Ports):

Number of FC switches in the

fabric

1200

192

896

9.3 : 1

30

69


Server Consolidation with Blade servers

2X ISL to Core at 4G

16 Host Ports at 4G

120 Storage Ports at 2 G


Five Racks

96 Dual Attached Blade Servers per Rack

A B

A B

Blade Servers

Blade Server

Design Using 2 x 4

G ISL per Blade

Switch;

Less cables/power Ports Deployed:

Storage Ports (4 G

Dedicated):


Disk Oversubscription

(Ports):

Number of FC switches in the

fabric

1608

240

480

8 : 1

62

70


Session Agenda










71

Recommended Core-Edge Designs for

Scale and Availability


SAN Design




‒ Meeting oversubscription ratios established by disk vendors

‒ Effective zoning

‒ Providing Business Function/Operating System Fabric Segmentation and

Security


Key Requirements

73


N-Port Virtualizer (NPV) Reduces Number of

FC Domain IDs

2 ISL to Core at 10 G

32 Host Ports at 4 G

14 Racks

32 Dual Attached Servers per Rack

Top of Rack

Top of Rack Design

Fabric Switches in

NPV mode

Ports Deployed:




Number of FC switches in the fabric

1200

192

896

9.3 : 1

2

A B 96 Storage Ports at 2 G


A

B

MDS 91xx

in NPV

mode

NPIV Core

74


NPV Blade Switch

2 ISL to Core at 4G

16 Host Ports at 4G

Blade Servers

Blade Server Design

Using 2 x 4 G ISL per

Blade Switch;

Less cables/power

Ports Deployed:




Number of fabric switches to manage

1608

240

480

8 : 1

2

120 Storage Ports at 2 G


Five Racks

96 Dual Attached Blade Servers per Rack

A B

A B

NPIV Core

MDS 91xx

in NPV

mode

75


F-Port Port Channel

F-Port PortChannels

‒ Bundle multiple ports in to 1 logical

link

‒ Similar to ISL portchannels in FC

and EtherChannels in Ethernet

Benefits

‒ High-Availability- no disruption if

cable, port, or line cards fail

‒ Optimal bandwidth utilization &

higher aggregate bandwidth with

load balancing

Enhance NPV Uplink Resiliency

Storage

Bla

deS

yste

m

Blade 1

Blade 2

Blade N

F-Port Port

Channel

F-Port N-Port

Core Director

SAN

Interface port-channel 1

no shut

Interface fc1/1

channel-group 1

Interface fc1/2

channel-group 1

76


Control and monitor VMs in the SAN using NPIV

NPIV gives Virtual Servers SAN identity

‒ Designed for virtual server environments

Allows SAN control of VMs

‒ Zoning and LUN Masking at VM level

Multiple applications on the same port can use different IDs

‒ Better utilization of the server connectivity

FC

LUN1(pwwnD1)

LUN2 (pwwnD2)

LUN3(pwwnD3)

Control and monitor VMs in the SAN

FC

HBA

N_Port Controller

vpwwn1 FCID=1.1.1

vpwwn2 FCID=1.1.2

vpwwn2 FCID=1.1.3

F_Port

Virtual Servers

Email

Web

Print

Zone_Email vpwwn1 pwwnD1 Zone_Web vpwwn1 pwwnD1 Zone_Print vpwwn1 pwwnD1

77


Nested NPIV

Two levels of NPIV usage

‒ From server to first level switch (NPV)

‒ From NPV to the core SAN

Virtual servers connected to the NPV devices

‒ Servers Supporting NPIV

‒ VmWare ESX in RDM mode

Connecting NPIV capable hosts to NPV

NP

F

P2 N

P

F

P1

NPV Edge Switch

NPV-Core Switch

F F

P3 = vP1 P4 = vP5

vP2 vP3

vP4

vP6 vP7

vP8

NP

IV

NP

IV

78


VM-Aware SANS

It is important to follow the guidelines from the virtual machine vendors for

assigning port WWNs to virtual machines. For example, VMware requires

the use of Raw Device Mode (RDM) instead of Virtual Machine File

System (VMFS) to get access to raw LUNs.

Using NPIV/Nested NPV with RDM we can give QOS, incident isolation

(VSANS) and visibility into a Virtualized environment

79


Summary of Recommendations

High Availability - Provide a Dual Fabric

Use of Port-Channels and F-Port Channels with NPIV to provide the

bandwidth to meet oversubscription ratios

Use NPIV/NPV to provide Domain ID scaling and ease of management

Use of host level NPIV and Nested NPV to provide visibility to Virtualized

servers

80


Session Agenda










81


Fibre Channel over Ethernet

What Enables It?

10Gbps Ethernet

Lossless Ethernet

‒ Matches the lossless behavior guaranteed in FC by B2B credits

Ethernet jumbo frames

‒ Max FC frame payload = 2112 bytes

Eth

ern

et

Hea

de

r

FC

oE

Hea

de

r

FC

He

ad

er

FC Payload

CR

C

EO

F

FC

S

Same as a physical FC frame

Control information: version, ordered sets

(SOF, EOF)

Normal ethernet frame, ethertype = FCoE

83


Unified Fabric

Why?

Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and

HCAs

Limited number of interfaces for Blade Servers

All

traffic

goes

over

10GE

IPC Traffic HCA

FC Traffic FC HBA

CNA

CNA

FC Traffic FC HBA

NIC LAN Traffic

NIC LAN Traffic

NIC Mgmt Traffic

NIC Backup Traffic

84


Today‘s Unified I/O Architecture

Ethernet FC

LAN SAN B SAN A

Today I/O Consolidation with FCoE

SAN B LAN SAN A

FCoE

Nexus

5000/UCS

85


Nexus 5K FCOE Switches Also Use NPV to

Achieve Both Server and IO Consolidation

A B

Attached Servers per Rack

A

B

Nexus 5K/2k in

NPV mode

LAN Core

32 Host CNA Ports at 10G

Core connectivity using FC modules on N5K

86


Cisco UCS Fabric Interconnects Also

Use NPV

A B

UCS 6X00 fabric

Interconnect in

NPV mode

LAN Core

87


UCS Core-Edge Design

F_Port Channeling and Trunking from MDS to

UCS

FC Port Channel behaves as one logical

uplink

FC Port Channel can carry all VSANs (Trunk)

UCS Fabric Interconnects remains in NPV end

host mode

Server vHBA pinned to an FC Port Channel

Server vHBA has access to bandwidth on any

link member of the FC Port Channel

Load balancing based on FC Exchange_ID

‒ Per Flow

Loss of Port Channel member link has no

effect on Server vHBA (hides the failure)

‒ Affected flows to remaining member links

‒ No FLOGI required

N-Port Virtualization Forwarding with MDS

F_ Port

Channel &

Trunk

SAN B SAN A

Server 1 VSAN 1

vFC 1 vFC 1

N_Proxy

F_Proxy

N_Port

6100-A 6100-B

F_Port

vFC 2 vFC 2

Server 2 VSAN 2

vHBA 1 vHBA 0 vHBA 1 vHBA 0

VSAN 1,3 VSAN 1,2

NPIV NPIV

88


Nexus 5X00 Core_Edge

Nexus 5000 access switches operating

in NPV mode

With NX-OS release 4.2(1) Nexus 5000 supports F-Port Trunking and Channeling on the links between an NPV device and upstream FC switch (NP port -> F port)

F_Port Trunking: Better multiplexing of traffic using shared links (multiple VSANs on a common link)

F_Port Channeling: Better resiliency between NPV edge and Director Core

‒ No host re-login needed per link failure

‒ No FSPF recalculation due to link failure

Simplifies FC topology (single uplink from NPV device to FC director)

F_Port Trunking and Channeling

Fabric ‘A’ Supporting

VSAN 2

F Port Trunking & Channeling

VLAN 10,30

VLAN 10,20

VSAN 3

Fabric ‘B’ Supporting

VSAN 3

VF

VN

TF

TNP

Server ‘1’

VSAN 2 Server ‘2’

VSAN 3

Nexus 5000

NPV

VLAN 30=VSAN 3

VLAN 20=VSAN 2

VSAN 2

VLAN 10

89


FCoE Multi-Tier

Multi-hop edge/core/edge topology

Core SAN switches supporting FCoE

N7K with DCB/FCoE line cards

MDS with FCoE line cards (Sup2A)

Edge FC switches supporting either

N5K - E-NPV with FCoE uplinks to the FCoE enabled core (VNP to VF)

N5K or N7K - FC Switch with FCoE ISL uplinks (VE to VE)

Scaling of the fabric (FLOGI, …) will most likely drive the selection of which mode to deploy

Larger Fabric Multi-Hop Topologies

Edge FCF Switch Mode

Servers, FCoE attached Storage

N7K or MDS FCoE enabled Fabric Switches

FC Attached Storage Servers

VE

VE

Edge Switch in E-NPV Mode

VF

VNP VE

VE

90


Fibre Channel Aware Device

What does an FCoE-NPV device do?

‖FCoE NPV bridge" improves over a "FIP snooping bridge" by intelligently proxying FIP functions between a CNA and an FCF

Active Fibre Channel forwarding and security element

FCoE-NPV load balance logins from the CNAs evenly across the available FCF uplink ports

FCoE NPV will take VSAN into account when mapping or ‗pinning‘ logins from a CNA to an FCF uplink

Emulates existing Fibre Channel Topology (same mgmt, security, HA)

Avoids Flooded Discovery and Configuration (FIP & RIP)

FCoE NPV

FCF

Fibre Channel Configuration and Control

Applied at the Edge Port

Proxy FCoE VLAN Discovery

Proxy FCoE FCF Discovery

FCoE NPV

VF

VNP

91


Session Agenda










92


Reference Sessions

BRKCOM-2002-UCS Supported Storage Architectures and Best Practices

with Storage

BRKDCT-1044-FCoE for the IP Network Engineer

BRKSAN-2282- Operational Models for FCOE Deployments- Best

Practices and Examples

BRKCOM-2001 UCS Deep Dive

93


Recommended Reading

NX-OS and Cisco Nexus Switching (ISBN:

1587058928), by David Jansen, Ron Fuller,

Kevin Corbin. Cisco Press 2010.

Storage Networking Fundamentals(ISBN-10:1-

58705-162-1; ISBN-13: 978-11-58705-162-3),

by Marc Farley. Cisco Press. 2007.

Storage Networking Protocol Fundamentals

(ISBN: 1-58705-160-5), by James Long, Cisco

Press. 2006.

Available Onsite at the Cisco Company Store

94


Complete Your Online

Session Evaluation Give us your feedback and you

could win fabulous prizes.

Winners announced daily.

Receive 20 Passport points for each

session evaluation you complete.

Complete your session evaluation

online now (open a browser through

our wireless network to access our

portal) or visit one of the Internet

stations throughout the Convention

Center.

Don‘t forget to activate your

Cisco Live Virtual account for access to

all session material, communities, and

on-demand and live activities throughout

the year. Activate your account at the

Cisco booth in the World of Solutions or visit

www.ciscolive.com.

95

http://www.ciscolive.com


Final Thoughts

Get hands-on experience with the Walk-in Labs located in World of

Solutions, booth 1042

Come see demos of many key solutions and products in the main Cisco

booth 2924

Visit www.ciscoLive365.com after the event for updated PDFs, on-

demand session videos, networking, and more!

Follow Cisco Live! using social media:

‒ Facebook: https://www.facebook.com/ciscoliveus

‒ Twitter: https://twitter.com/#!/CiscoLive

‒ LinkedIn Group: http://linkd.in/CiscoLI

96

http://www.ciscolive365.com/

https://www.facebook.com/ciscoliveus

https://twitter.com/

http://linkd.in/CiscoLI

Reference Slides


NPV Traffic Engineering

Allows user to select external

interface per server interface

Benefits

‒Allows customized Bandwidth

Management

‒Allows use of shortest path

‒Enables use of Persistent FCIDs

Bla

de

1

Bla

de

N


Storage

Bla

de

2

….

Traffic-map

SAN

Customize

d Traffic

Pattern

1

2

N

npv traffic-map server-interface fc1/2 external-interface

fc1/1

npv traffic-map server-interface fc1/3 external-interface

fc1/1

…….

npv traffic-map server-interface fc1/N external-interface

fc1/5

Fc1/3 Fc1/4 Fc1/24

Fc1/1 Fc1/5

99


Number of NPIV Logins: MDS 9200/9500

Type of Logins Verified Logins

Logins per Port 126 (1) / 256 (2)

Logins per Line Card 400

Logins per Switch 2,000

Logins per physical

fabric

10,000

These are the number of logins allowed on all Gen1, Gen2 and Gen3 line cards. The limits applied to on a per switch will also apply to all MDS 9200 and MDS 9500. MDS 9124/9134 and Blade switches will have different limits and will be shown later.

(1) SAN-OS 3.x, NX-OS 4.1(1) (2) NX-OS 4.1(2)

100


Number of NPIV Logins:

MDS 9124/9134 and Blade Switches

Switching Mode NPV Mode

Logins per Port 42 (1) / 89 (2) 114

Logins per Port-Group 168 114

Logins per MDS 9124 1008 684

Logins per MDS 9134 1680 1140

Logins per MDS 9124e 1008 684

Logins per IBM Blade Switch 840 570

The stated numbers are verified / supported number of logins.

(1) Using 2 member zoning (2) Using default zone-permit instead of zoning

101

Introduction to FCOE


Fibre Channel over Ethernet

What Enables It?

10Gbps Ethernet

Lossless Ethernet

‒ Matches the lossless behavior guaranteed in FC by B2B credits

Ethernet jumbo frames

‒ Max FC frame payload = 2112 bytes

Eth

ern

et

Hea

de

r

FC

oE

He

ad

er

FC

Hea

de

r

FC Payload

CR

C

EO

F

FC

S

Same as a physical FC

frame

Control information: version, ordered

sets (SOF, EOF)

Normal ethernet frame, ethertype =

FCoE

103


Unified Fabric

IEEE DCB Standard / Feature Status of the Standard

IEEE 802.1Qbb Priority-based Flow Control (PFC)

In Sponsor Ballot

IEEE 802.3bd Frame Format for PFC

In Sponsor Ballot

IEEE 802.1Qaz Enhanced Transmission Selection (ETS) and Data Center Bridging eXchange (DCBX)

Just completed WG recirculation ballot. New recirculation expected next week in order to go to Sponsor Ballot after the May interim

IEEE 802.1Qau Congestion Notification

Done!

IEEE 802.1Qbh Port Extender

In its first task group ballot

Developed by IEEE 802.1 Data

Center Bridging Task Group (DCB)

All technically stable

Final standards expected by mid

2010

CEE (Converged Enhanced Ethernet) is an informal

group of companies that submitted initial inputs to the

DCB WGs. 104


FC Traffic FC HBA

Unified Fabric Why?

Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and

HCAs

Limited number of interfaces for Blade Servers

All

traffic

goes

over

10GE

CNA

CNA

FC Traffic FC HBA

NIC LAN Traffic

NIC LAN Traffic

NIC Mgmt Traffic

NIC Backup Traffic

IPC Traffic HCA

105


What‘s the difference between DCE, CEE and

DCB ?

All three acronyms describe the same thing, meaning the architectural

collection of Ethernet extensions (based on open standards)

Cisco has co-authored many of the standards associated and is focused

on providing a standards-based solution for a Unified Fabric in the data

center

The IEEE has decided to use the term ―DCB‖ (Data Center Bridging) to

describe these extensions to the industry.

http://www.ieee802.org/1/pages/dcbridges.html

106

http://www.ieee802.org/1/pages/dcbridges.html


Priority Flow Control Fibre Channel over Ethernet Flow Control

Pa

cke

t

R_R

DY

Fibre Channel

Transmit Queues Ethernet Link

Receive Buffers

Eight

Virtual

Lanes

One One

Two Two

Three Three

Four Four

Five Five

Seven Seven

Eight Eight

Six Six

STOP PAUSE

B2B Credits

Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1p

When link is congested, CoS assigned to FCoE will be PAUSEd so traffic will not be dropped

Other traffic assigned to other CoS will continue to transmit and rely on upper layer protocols for retransmission

107


DCB ―Virtual Links‖ An Example

VL1

VL2

VL3

LAN/IP Gateway

VL1 – LAN Service – LAN/IP VL2 - No Drop Service - Storage

Ability to support QoS queues within the ―lanes‖

Campus Core/

Internet

Storage Area

Network

108


Fiber Channel over Ethernet Protocol Host Side – FIP and DCBX Configuration

1st portion of the

MAC is the FC-

MAP of the

Nexus 5000

FC-MAP

(0E-FC-xx)

FC-ID

7.8.9 FC-MAC

Address

FC-MAP

(0E-FC-xx)

FC-ID

10.00.01

2nd portion of the MAC

is the FC-ID

109


FCoE Building Blocks

The Acronyms Defined

FCF (FCoE Forwarder): A Fibre Channel switching element that is able to forward FCoE frames (Nexus 5000, Nexus 7000, MDS 9000)

FPMA : ‖Fabric Provided MAC Address‖ -- A unique MAC address that is assigned by an FCF to a single Enode

Enode: ―End Node‖ -- a Fiber Channel end node that is able to transmit FCoE frames using one or more ENodeMACs.

FCoE Pass-Through : any DCB device capable of passing FCoE frames to an FCF

FIP Snooping Bridge

FCoE N-Port Virtualizer

Single hop FCoE : running FCoE between the host and the first hop access level switch

Multi-hop FCoE : the extension of FCoE beyond a single hop into the Aggregation and Core layers of the Data Centre Network

110


FCoE Building Blocks

Fibre Channel Forwarder

FCF (Fibre Channel Forwarder) is the Fibre Channel switching element

inside an FCoE switch

‒ Fibre Channel logins (FLOGIs) happens at the FCF

‒ Consumes a Domain ID

FCoE encap/decap happens within the FCF

‒ Forwarding based on FC information

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Eth

port

Ethernet Bridge

FC

por

t FC

por

t FC

por

t FC

por

t

FCF

FCoE Switch FC Domain ID : 15

111


Fibre

Channel

Drivers

Ethernet

Drivers

Operating System

PCIe

Eth

ern

et

Fib

re C

ha

nn

el

10G

bE

10G

bE

Link

Ethernet Driver

bound to

Ethernet NIC PCI

address

FC Driver

bound to FC

HBA PCI

address

Replaces multiple adapters per

server, consolidating both Ethernet and FC on a single interface

Appears to the operation system as individual interfaces (NICs and HBAs)

First Generation CNAs from support PFC and CIN-DCBX

Second Generation CNAs support PFC, CEE-DCBX as well as FIP

Single chip implementation

Half Height/Length

Half power consumption

FCoE Building Blocks Converged Network Adapter

112


Translate to FCoE…

Same host to target communication

Host has 2 CNA‘s (one per fabric)

Target has multiple ports to connect to fabric

Connect to a capable switch

Port Type Negotiation (FC port type will be handled by FIP)

Speed Negotiation

DCBX Negotiation

Access switch is a Fibre Channel Forwarder (FCF)

Dual fabrics are still deployed for redundancy

FC

CNA

FC Fabric

ENode

Target

Ethernet Fabric

DCB capable Switch

acting as an FCF

Unified Wire

113


VE_Port

VF_Port

VF_Port

VE_Port

VN_Port

VN_Port

Fibre Channel Over Ethernet Port Types

Fibre Channel over Ethernet Switch

FCoE_

NPV

Switch

VF_Port VNP_Port FCF

Switch

End

Node

End

Node

FCoE Switch : FCF

114


My Port Is Up…Can I Talk Now? FIP and FCoE Login Process

VN_Port

VF_Port

FIP Discovery

E_ports or

VE_Port

Step 1: FIP Discovery Process

enables FCoE adapters to discover which VLAN to transmit & receive FCoE frames

enables FCoE adapters and FCoE switches to discovers other FCoE capable devices

verifies Lossless Ethernet is capable of FCoE transmit

Step 2: FIP Login Process Simular to existing Fibre Channel Login process - sends

FLOGI to upstream FCF

adds the negotiation of the MAC address to use Fabric Provided MAC Address (FPMA)

FCF assigns the host a Enode MAC address to be used for FCoE forwarding

FC

CNA

FC or FCoE Fabric

Target

ENode

**Multi-hope FCoE with VE_Ports not supported until Eaglehawk Release

115


Enode MAC Address Fibre Channel over Ethernet Addressing Scheme

Enode MAC assigned for each FCID

Enode MAC composed of a FC-MAP and FCID

FC-MAP is the upper 24 bits of the Enode‘s MAC

FCID is the lower 24 bits of the Enode‘s MAC

FCoE forwarding decisions still made based on FSPF and the FCID within the Enode MAC

FC Fabric

Domain ID

FC-MAP

(0E-FC-xx)

FC-ID

7.8.9

FC-

MAC

Address

FC-MAP

(0E-FC-xx) FC-ID

10.00.01

Fibre Channel

FCID

Addressing

116


Login Complete…Almost There Fabric Zoning

FC

FC/FCoE Fabric

Initiator

Target

FCoE fabric zoning done the same as FC fabric zoning

Zoning is enforced at the FCF

Zoning can be configured on the Nexus 5000 using the CLI or Fabric Manager

If Nexus 5000 is an ―FCoE Pass-Through‖ device, zoning will be configured on the upstream core switch and pushed to the Nexus 5000

fcid 0x10.00.01 [pwwn 10:00:00:00:c9:76:fd:31] [tnitiator]

fcid 0x11.00.01 [pwwn 50:06:01:61:3c:e0:1a:f6] [target] pwwn 10:00:00:00:c9:76:fd:31

pwwn 50:06:01:61:3c:e0:1a:f6

**Multi-hope FCoE with VE_Port not supported until Eaglehawk

FCF with Domain ID 10

117


Host connected over unified wire to first hop access switch

Access switch (Nexus 5000) is the FCF

Fibre Channel ports on the access switch can be in NPV or Switch mode for native FC traffic

DCBX is used to negotiate the enhanced Ethernet capabilities

FIP is use to negotiate the FCoE capabilities as well as the host login process

FCoE runs from host to access switch FCF – native Ethernet and native FC break off at the access layer

FC

CNA

FC Fabric

ENode

Target

Ethernet Fabric

DCB capable Switch

acting as an FCF

Unified Wire

Single Hop Design Today‘s Solution

118


Fabric A

CEE-DCBX

Generation 1 CNA

CIN-DCBX

Generation 2 CNA

Fabric B LAN Fabric

VN

VF Direct attach

VN_Port to

VF_Port

Generation 1 CNA

limited to direct attached CNAs at the access

Utilized Cisco, Intel, Nuova Data Center Bridging Exchange protocol (CIN-DCBX)

Generation 2 CNA

Utilizes Converged Enhanced Ethernet Data Center Bridging Exchange protocol (CEE-DCBX)

Utilizes FCoE Initialization Protocol (FIP) as defined by the T.11 FC-BB-5 specification

Supports both direct and multi-hop attachment (through a Nexus 4000 FIP Snooping Bridge)

Single Hop Design The CNA Point of View

Nexus 5000

FCF-A Nexus 5000

FCF-A

119


Unified Fabric with FCoE CNA: Converged Network Adapter

Standard drivers

Same management

Operating System sees:

‒ Dual port 10 Gigabit

Ethernet adapter

‒ Dual Port 4 GbpsFibre

ChannelHBAs

120


vfc2

Eth1/2 PC1

vfc1

vfc3

mac-address

The Virtual Fibre Channel Interface Binding the vfc Virtual Fibre Channel Interface (vfc) : Where is the

logical Fibre Channel wire is terminated (the FCF)

Today this corresponds to an ―F_Port‖

Three options for binding a vfc interface :

Physical Interface: Direct Attach CNAs and FCoE_NPV devices (future)

Single link port-channel: Direct Attach CNAs connected via a two port vPC

MAC-Address over an Ethernet Cloud: through a FIP-Snooping Device

Nexus 5000

FCF

Nexus 4000

FIP-Snooping

Nexus 5000

FCF

121


VLAN 10,30

VLAN 10,20

FCoE VLANs are treated differently than native Ethernet VLANs

No flooding, MAC learning, broadcasts, etc.

The FCoE VLAN must not be configured as a native VLAN

FIP uses native VLAN

FCoE VLANs should not be configured on Ethernet links that are not carrying FCoE traffic

Unified Wires must be configured as trunk ports and STP edge ports

! VLAN 20 is dedicated for VSAN 2 FCoE traffic

(config)# vlan 20

(config-vlan)# fcoevsan 2

VSAN 2

STP Edge Trunk

Fabric A Fabric B LAN Fabric

Nexus 5000

FCF

Nexus 5000

FCF

VSAN 3

Single Hop Design The FCoE VLAN

122


VLAN 10,30

VLAN 10,20

FCoE Fabric ‗A‘ will have a different VLAN topology than FCoE Fabric ‗B‘ which are different from the LAN Fabric

PVST+ allows unique topology per VLAN

MST requires that all switches in the same Region have the same mapping of VLANs to instances

MST does notrequire that all VLANs be defined in all switches

A separate instance must be used for FCoE VLANs

spanning-tree mst configuration

name FCoE-Fabric

revision 5

instance 5 vlan 1-19,40-3967,4048-4093

instance 10 vlan 20-29

instance 15 vlan 30-39


VSAN 3 VSAN 2

VLAN 10

Nexus 5000

FCF-A

Nexus 5000

FCF-B

Single Hop Design The FCoE VLAN and STP

123


VLANs and VSANs FCoE Considerations

VSANs use VLAN hardware table resources

FCoE requires a VLAN and a VSAN that you bind the VLAN to.

Hence for each FCoE VSAN you should count using 2 VLANs

Enabling FCoE burns two internal VSAN/VLAN resources

vFC binds to the Port-Channel, as long as there is one single port in the

port-channel attached to the switch

124


Optimal layer 2 LAN design often

leverages Multi-Chassis Etherchannel

(MCEC)

Nexus utilizes Virtual Port-Channel (vPC)

to enable MCEC either between switches

or to direct attached servers (using LACP

or static port-channels)

MCEC provides network based load

sharing and redundancy without

introducing layer 2 loops in the topology

MCEC maintains the separation of LAN

and SAN high availability topologies

‒ FC maintains separate SAN ‗A‘

and SAN ‗B‘ topologies

‒ LAN utilizes a single logical

topology

Direct Attach vPC

Topology

MCEC

vPC Peers

vPC Peer Link


Nexus 5000

FCF-A

Nexus 5000

FCF-B

Single Hop Design What is MCEC??

125


vPC enabled topologies with FCoE

must follow specific design and

forwarding rules…

A ‗vfc‘ interface can only be

associated with a single-port port-

channel

While the port-channel configurations are the same on N5K-1 and N5K-2, the FCoE VLANs are different

vPC configuration works with Gen-2 FIP enabled CNAs ONLY

FCoE VLANs are ‘not’ carried on the vPC peer-link

FCoE and FIP ethertypes are ‘not’ forwarded over the vPC peer link

Direct Attach vPC

Topology

VLAN 10,30

VLAN 10,20

STP Edge Trunk

VLAN 10 ONLY HERE!


Nexus 5000

FCF-A

Nexus 5000

FCF-B

Single Hop Design Unified Wires and MCEC

vPC contains only 2 X

10GE links – one to each

Nexus 5000

126

CLI Configuration Sample


Sample Working Topology

NPV Core

NPV Edge

128


Enabling the Required Features

NPV Edge Switch:

pod7-5020-51(config)# feature fcoe

FC license checked out successfully

fc_plugin extracted successfully

FC plugin loaded successfully

FCoE manager enabled successfully

FC enabled on all modules successfully

pod7-5020-51(config)# feature npv

Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current

configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y

NPV Core Switch:

pod3-9216i-70(config)# feature npiv

pod3-9216i-70(config)# feature fport-channel-trunk

Admin trunk mode has been set to off for

1- Interfaces with admin switchport mode F,FL,FX,SD,ST in admin down state

2- Interfaces with operational switchport mode F,FL,SD,ST.

129


Configure the VSANs

NPV Edge Switch:

pod7-5020-51(config)# vsan database

pod7-5020-51(config-vsan-db)# vsan 10

NPV Core Switch:

pod3-9216i-70(config)# vsan database

pod3-9216i-70(config-vsan-db)# vsan 10

pod3-9216i-70(config-vsan-db)# vsan 10 interface fc1/12

130


Configure Trunking F_Port Port Channel

NPV Core Switch:

pod3-9216i-70(config)# interface port-channel 1

pod3-9216i-70(config-if)# switchport mode f

pod3-9216i-70(config-if)# switchport trunk mode on

pod3-9216i-70(config-if)# channel mode active

pod3-9216i-70(config-if)# interface fc2/13, fc2/19

pod3-9216i-70(config-if)# switchport mode f

pod3-9216i-70(config-if)# switchport rate-mode dedicated

pod3-9216i-70(config-if)# switchport trunk mode on

pod3-9216i-70(config-if)# channel-group 100 force

fc2/13 fc2/19 added to port-channel 100 and disabled

please do the same operation on the switch at the other end of the port-channel, then do "no shutdown"

at both end to bring them up

pod3-9216i-70(config-if)# no shut

131


Configure Trunking F_Port Port Channel

NPV Edge Switch:

pod7-5020-51(config)# interface san-port-channel 1

pod7-5020-51(config-if)# switchport mode np

pod7-5020-51(config-if)# switchport trunk mode on

pod7-5020-51(config-if)# interface fc2/1-2

pod7-5020-51(config-if)# switchport mode np

pod7-5020-51(config-if)# switchport trunk mode on

pod7-5020-51(config-if)# channel-group 1

fc2/1 fc2/2 added to port-channel 1 and disabled

please do the same operation on the switch at the other end of the port-channel, then do "no shutdown"

at

both ends to bring it up

pod7-5020-51(config-if)# no shut

132


Configure FCoE on NPV Edge Switch

pod7-5020-51(config)# vlan 10

pod7-5020-51(config-vlan)# fcoevsan 10

pod7-5020-51(config-vlan)# interface ethernet 1/3

pod7-5020-51(config-if)# switchport mode trunk

pod7-5020-51(config-if)# switchport trunk allowed vlan 1,10

pod7-5020-51(config-if)# spanning-tree port type edge trunk

Warning: Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting hubs,

concentrators, switches, bridges, etc... to this interface when edge port type (portfast) is enabled, can cause

temporary bridging loops. Use with CAUTION

pod7-5020-51(config-if)# interface vfc3

pod7-5020-51(config-if)# bind interface ethernet 1/3

pod7-5020-51(config-if)# vsan database

pod7-5020-51(config-vsan-db)# vsan 10 interface vfc3

pod7-5020-51(config-vsan-db)# interface vfc3

pod7-5020-51(config-if)# no shut

133


Verify NPV Fabric Connectivity (Edge) pod7-5020-51# sh npv flogi-table

--------------------------------------------------------------------------------

SERVER EXTERNAL

INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE

--------------------------------------------------------------------------------

vfc3 10 0x0f0100 21:00:00:c0:dd:12:04:f3 20:00:00:c0:dd:12:04:f3 Spo1

Total number of flogi = 1.

Verify that the VFC interface is pinned to the SAN Port Channel

pod7-5020-51# sh npv traffic-usage

NPV Traffic Usage Information:

----------------------------------------

Server-If External-If

----------------------------------------

vfc3 san-port-channel 1

----------------------------------------

134


Verify on Core that N5K and FCoE Workstation

Are Logged into the Fabric.

pod3-9216i-70(config)# show flogi database

--------------------------------------------------------------------------------

INTERFACE VSAN FCID PORT NAME NODE NAME

--------------------------------------------------------------------------------

fc1/12 10 0x0f00dc 21:00:00:20:37:a9:cd:6e 20:00:00:20:37:a9:cd:6e

fc1/12 10 0x0f00e0 21:00:00:20:37:a9:89:7e 20:00:00:20:37:a9:89:7e

fc1/12 10 0x0f00e2 21:00:00:20:37:af:de:85 20:00:00:20:37:af:de:85

fc1/12 10 0x0f00e4 21:00:00:20:37:a9:d6:49 20:00:00:20:37:a9:d6:49

fc1/12 10 0x0f00e8 21:00:00:20:37:a9:d7:bf 20:00:00:20:37:a9:d7:bf

fc1/12 10 0x0f00ef 21:00:00:20:37:a9:94:89 20:00:00:20:37:a9:94:89

port-channel 1 1 0x670102 24:01:00:0d:ec:a3:da:40 20:01:00:0d:ec:a3:da:41

port-channel 1 10 0x0f0100 21:00:00:c0:dd:12:04:f3 20:00:00:c0:dd:12:04:f3

Total number of flogi = 8.

135


Configure Zoning

NPV Core Switch:

pod3-9216i-70(config)# zone name npv_vsan10 vsan 10

pod3-9216i-70(config-zone)# member pwwn 21:00:00:20:37:a9:cd:6e

pod3-9216i-70(config-zone)# member pwwn 21:00:00:20:37:a9:89:7e

pod3-9216i-70(config-zone)# member pwwn 21:00:00:20:37:af:de:85

pod3-9216i-70(config-zone)# member pwwn 21:00:00:c0:dd:12:04:f3

pod3-9216i-70(config-zone)# exit

pod3-9216i-70(config)# zoneset name npv_v10_zs vsan 10

pod3-9216i-70(config-zoneset)# member npv_vsan10

pod3-9216i-70(config-zoneset)# zoneset activate name npv_v10_zs vsan 10

Zoneset activation initiated. check zone status

pod3-9216i-70(config)# show zoneset active

zoneset name npv_v10_zs vsan 10

zone name npv_vsan10 vsan 10

* fcid 0x0f00dc [pwwn 21:00:00:20:37:a9:cd:6e]

* fcid 0x0f00e0 [pwwn 21:00:00:20:37:a9:89:7e]

* fcid 0x0f00e2 [pwwn 21:00:00:20:37:af:de:85]

* fcid 0x0f0100 [pwwn 21:00:00:c0:dd:12:04:f3]

136

SAN Security


SAN Management Potential Threats Three Main Areas of Vulnerability: 1. Intentional disruption of switch processing

‒ CPU hogging from unnecessary queries ‒ Denial-of-service attacks

Result: switch can‘t react to fabric events

2. Compromised fabric stability

‒ Altered/lost switch configurations ‒ Removal of other security services ‒ Disabled switches/ISLs/device ports

Result: loss of service, unplanned down time

3. Compromised data integrity and secrecy

‒ Altered target (and LUN) visibility ‒ Altered zoning configuration

Result: LUN corruption, data corruption, data theft, or loss

Out-of-Band Ethernet Management Connection

Accidental or Intentional Harmful

Management Activity

138


SAN Management Security Securing access to all management

facilities on MDS SAN

‒ Must secure console sessions

‒ Must secure GUI application access

‒ Must secure API access (SMI-S)

‒ Must also secure file transfer

to/from switch

Equally important to

enable audit mechanisms

‒ Integrated RADIUS for user accounting and

switch scope assignment

‒ Integrated syslog for

switch-event accounting

‒ Integrated SNMP traps for

access-denial accounting

‒ Network time protocol (NTP) support to

synchronize clocks, log entry time stamps

SAN Management Security Infrastructure

Management

Network

RADIUS Server for

User Authentication

switch> config t

switch(config)>

analyzer on

switch(config)>

exit

switch>

Cisco Fabric

Manager

Using SNMPv3

NX-OS CLI

Using SSH/SFTP

TACACS+ Server for

User Authentication

SNMP Polling Server

Using SNMPv3

NTP Server

for Time/Date

Synchronization

Integrated RFC 2625 IP-over-FC Provides

Redundant IP Connectivity for Security Services over In-Band

FC Link

Out-of-Band Ethernet

Management

Connection

139


Used for AAA (Authentication, Authorization, and Accounting) services

‒ Limit management access to a subset of switches

‒ MDS supports up to five HA server definitions

RADIUS—Remote Authentication Dial In User Service (IETF RFC-2865 standard)

‒ Initially used for dial-in networks—now greatly expanded to a variety of uses

System user account centralized authentication

Network-device user account AAA services

Dial-in/VPN service AAA services

iSCSI host authentication

TACACS+—Terminal Access Controller Access Control System (based on RFC-1492)

‒ Widely used and supported by Cisco

‒ Freely available from Cisco—similar to RADIUS

Flexible RADIUS and TACACS+ Services

Cisco MDS SAN

Authentication Calls and Accounting

Records Are Sent to Centralized RADIUS or TACACS+ Servers

RBAC Role Membership Info Is Authorized by RADIUS/TACACS+ Servers

RAD

LDAP Server

DB

Database Server

(Oracle, mySQL,

etc.)

Roles Are Populated into MDS Switches

Dial/VPN Servers for Remote

Access System Console Terminal Servers

Network Management

Stations

NMS

Datacenter Routers and

Switches

RADIUS and TACACS+ Deployments

RAD

Windows 2000 IAS Server (RADIUS)

Linux TACACS+Server

Microsoft Active

Directory

Redundant Server

140


Configuration Consistency Analysis

Important to keep consistent

configurations across all switches

‒ Especially important for security configurations:

RADIUS/TACACS+, remote syslog, NTP,

SNMP communities, authentication, and roles

Configurations can be

extracted from switches as a flat text file

‒ Allows for easy and regular archiving

Cisco Fabric Manager provides

fabric configuration analysis tool

‒ Checks all switch configurations against

policy switch or file

‒ Can take corrective action to fix configurations

‒ Also has zone-merge analysis tool to

validate zone-merge validity

Policy Reference

Switch

Define Analysis

Rules

Fabric Configuration Analysis Part of

Cisco Fabric Manager

Review Results and Take

Corrective Actions

Administrator Compares Policy Reference Config

to All Switches in Fabric

141


SAN Management Recommendations

Use RBAC to grant adequate privilege to SAN administrators

‒ Example: not every administrator needs capability to disable modules

‒ Reserve select functions to fewer super-admin RBAC role:

‒ VSAN definition, firmware upgrades, roles definition, RADIUS, and SSH configuration

Use RADIUS or TACACS+ for centralized user account administration

‒ Ensures consistent and timely removal of users if required

‒ Use RADIUS accounting feature for audit log of configuration events

Use all secure forms of management protocols—disable others

‒ SSH, SFTP, SCP, SNMPv3, SSL for SMI-S support

‒ Disable Telnet, FTP, TFTP, SNMPv1,v2

Enable NTP across all switches for consistent time stamping of events

Log and archive everything

‒ Enable centralized syslog

‒ Take regular copies of switches configurations (can use CiscoWorks RME)

‒ Turn on MDS call-home feature to alert of anomalies

142


Fabric and Target Access Potential Threats Three Main Areas of Vulnerability: Compromised application data

‒ Unauthorized access to targets and LUNs

‒ High potential for data corruption, loss, or theft

Result: unplanned down time, costly data loss

Compromised LUN integrity

‒ LUN corruption due to unintentional OS mount

‒ Accidental formatting of LUN—loss of data


Compromised application performance

‒ Unauthorized I/O potentially causing congestion

‒ Injected fabric events causing disruption; i.e., rogue HBA hammering fabric controller

Result: unplanned down time, poor I/O performance

Unauthorized Fabric Service

Unauthorized Target Access

143


Fabric Access Security: Port Modes Port-mode security—allow edge ports

to form F_Ports or FL_Ports only,

i.e., no ISL/EISL

‒ MDS supports an Fx_Port mode which allows F_Port or

FL_Port only

‒ Limit users who can change port mode via

roles-based access control assignments

VSAN-based security—only allow access to

devices within attached VSAN

‒ Strict isolation based on fabric service

partitioning and explicit frame tagging

‒ Independent name server table per VSAN

‒ Independent active zoneset per VSAN

‒ Part of ANSI T11 fabric expansion

study group

Management port access security

‒ Provides IP access control lists (ACLs) for management

traffic (SNMP, SSH, Telnet, etc.)

IP Access Lists (ACL) Based on Source and Destination IP Addresses, TCP/UDP Ports, and TCP Connection Flags

Any Port Type

Auto Mode

E_Port Mode

F_Port Mode

Fx_Port Mode

F, FL Only F Only

Fx_Port Mode

E_Port or Auto Mode

Management Network

Port Mode and VSAN-Based Security

VSAN 1

VSAN 2

Both

Disk Array Connected to

Multiple VSANs

Unique Services

per VSAN

One Active VSAN Only

EISLs Carrying Multiple VSANs

144


Fabric Access Security

MDS access security technology

‒ Grant selective access to fabric based on device identity

‒ Failure results in link-level login failure

‒ Prevents FC frame S_ID spoofing through hardware frame filtering

Supports switch-to-switch (fabric binding) and device-to-switch (port security)

‒ Auto-learning mode to ease initial configuration

Uses grouping of attributes to define binding configuration

‒ WWN or Port_ID – port identifier on switch (i.e. fc1/2)

‒ Multiple groups are created and activated as a group set to enforce desired policy

Default configuration

‒ Set port administrative default value to SHUT

‒ Do not put ports in VSAN 1

‒ Ports by default in VSAN 4094 (isolated)

145


Fabric Access Security: Fabric Binding

Used to allow

ISL establishment

Attributes to define binding

configuration:

‒ fWWN—fabric WWN

of switch port

‒ sWWN—switch WWN

‒ Port_ID—port identifier

on switch (i.e., fc1/2)

fWWN-1 Port_ID-1

sWWN-2

pWWN-1

nWWN-1

pWWN-2

sWWN-1

fWWN-2 Port_ID-2

fWWN-3 Port_ID-3

fWWN-4 Port_ID-4

fWWN-6 Port_ID-6

fWWN-5 Port_ID-5

pWWN-3

pWWN-4

nWWN-2

sw-2

Security Group—sw-1 sWWN-2

Bind sw-2 to sw-1 ISL

Security Group – sw-1 sWWN-2

Port_ID-5 or fWWN-5

Bind sw-2 to sw-1/port 5 ISL

sw-1

146


Fabric Access Security: Port Security

Used to allow device-to-switch login

Attributes to define binding configuration

‒ pWWN—port WWN of attaching device

‒ nWWN—node WWN of attaching device

‒ fWWN—fabric WWN of switch port

‒ Port_ID—port identifier on switch (i.e. fc1/2)

Bind Host to sw-1 (Any Port)

Bind Host to sw-1/port 2

Bind Host, disk to sw-1 (Any Port)

Security Group – sw-1 pWWN-1 or nWWN-1

Security Group – sw-1

pWWN-1 or nWWN-1 Port_ID-2 or fWWN-2


pWWN-1 or nWWN-1 pWWN-3 or nWWN-2


pWWN-1 Port_ID-2 or fWWN-2 Bind Host HBA-1 to sw-1/port 2

fWWN-1

Port_ID-1

sWWN-2

pWWN-1

nWWN-1

pWWN-2

sWWN-1

fWWN-2

Port_ID-2

fWWN-3

Port_ID-3 fWWN-4

Port_ID-4

fWWN-6

Port_ID-6

fWWN-5

Port_ID-5

pWWN-3

pWWN-4

nWWN-2

sw-2

sw-1

147


Fabric Access Security: Authentication

Device authentication provides

stronger means of ensuring

device identity

‒ WWNs can be spoofed by

simple means

ANSI T11 FC-SP security

protocols working group

‒ Cisco was the prime contributor

DH-CHAP provides

authentication mechanism

‒ Switch-to-switch authentication

‒ Device-to-switch authentication

(when adopting HBA supporting

DH-CHAP)

Fibre Channel Fabric Authentication

Management Network

RADIUS Server for User

Authentication

RAD

TACACS+ Server for User Authentication

RADIUS and TACACS+ Servers Can Be Used to Hold DH-CHAP User

Accounts and Passwords for Centralized Authentication

Out-of-Band Ethernet Management Connection

New Switch Wanting to

Join the Fabric

New Host Wanting to Join

the Fabric

Equipped with HBA

Supporting DH-CHAP (Emulex, Qlogic)

DH

-CH

AP

FCIP Network

New Switches Wanting to Join the Fabric over FCIP

148


Fabric Access Recommendations Use IP ACLs on management interfaces to block unused services

‒ Enable logging of denied attempts—block denial-of-service attacks

Hard-fix switch-port administrative modes to assigned port function

‒ Lock (E)ISL ports to only be (T)E_Ports—set to E_Port mode

‒ Lock access ports to only be F(L)_Ports—set to Fx_Port mode

Use VSANs to isolate departments

‒ Provides security and availability benefits

‒ RBAC management control per VSAN allows individual admin assignment

Use port security features everywhere

‒ Bind devices to switch as a minimum level of security

‒ Bind devices to a port as an optimal configuration

Consider binding to line card in case of port failure

‒ Bind switches together at ISL ports—bind to specific port, not just switch

Use FC-SP authentication for switch-to-switch fabric access

‒ Use device-to-switch when available 149


Fabric Protocols Potential Threats

Three Main Areas of Vulnerability:

Compromised fabric stability

‒ Injection of disruptive fabric events

‒ Creation of traffic black-hole

Result: unplanned down time, fabric instability

Compromised data security

‒ Injection of harmful zone reconfiguration data

‒ Open access to fabric targets


Compromised application performance

‒ Unauthorized I/O potentially causing congestion

‒ Numerous disruptive topology changes

Result: unplanned down time, poor I/O performance

Fabric Control Protocol Integrity

Rogue Switch

150


SAN Fabric Protocols Security

Very important to secure the fabric control protocols to ensure fabric stability ‒ Securing access to control protocol

configuration via Cisco RBAC is first step

‒ Enable port-security for switch binding

‒ Using FC-SP for switch-to-switch authentication is next critical step to block rogue ISLs

Plug-n-play fabric protocol configuration is convenient—however, static configuration is more secure ‒ Configure static principle switch

‒ Enable static domain IDs

‒ Enable static FCIDs optional but recommended

Great benefit for HP/UX and AIX environments

‒ Enable RCF-reject, especially on long-haul links

‒ Enable RSCN-suppression where necessary

Use VSANs to divide and manage individual fabric configuration and resiliency

Cisco MDS 9216

Dept ‗A‘

VSAN Trunk

Bundles

Cisco MDS 9500

Multilayer Director

Port Channeling for HA and

Performance

Fabric Protocols Security Dept ‗B‘

DWDM or CWDM Network

VSAN Trunks

over Optical

Enterprise Tape VSAN

1 1 2 2

3 3 4 4 3 4

5 5 6 6 1 7 2 8

Statically Assigned domain_IDs, One per Active VSAN

Minimizes Potential for Disruptive RCF

Statically Assigned Principle Switch

per VSAN

RCF-Reject Configured to Protect Against Remote Initiated Fabric Rebuild

151

brksan-1121 storage area networking core edge design best...

Documents