43935463 n5k n2k ethernet fcoe lab guide student v2 1
TRANSCRIPT
Nexus 5000/Nexus 2000 Lab Guide
Overview This lab is designed to help attendees understand how to deploy Nexus 5000 and Nexus 2000 in the datacenter. It covers the configuration and management perspective for Nexus 5000 and Nexus 2000. Attendees will learn how to design and deploy
Lab List This guide includes these activities:
Lab 1: Basic configuration of N5k
Lab 2: Deploy N5k/N2k using EtherChannel
Lab 3: Deploy N5k/N2k using static pinning
Lab 4: Configuration of FCoE
Nexus 5000 and Nexus 2000 Overview
The Cisco Nexus™ 5000 Series Switches comprise a family of line-rate, low-latency, lossless 10 Gigabit Ethernet, Cisco® Data Center Ethernet, and Fibre Channel over Ethernet (FCoE) switches for data center applications. The switch family is highly serviceable, with redundant, hot-pluggable power supplies and fan modules. Its software is based on data center-class Cisco NX-OS Software for high reliability and ease of management.
The Cisco Nexus 2148T provides 48 Gigabit Ethernet server ports and 4 10 Gigabit Ethernet uplink ports in a compact one rack-unit (1RU) form factor. The Cisco Nexus 2000 Series Fabric Extender integrates with the parent switch, allowing "zero-touch" provisioning as well automatic configuration and software upgrade when a rack is connected to the parent switch. This integration allows large numbers of servers to be supported with the same feature set of the parent Cisco Nexus 5000 Series switch, along with specific configuration parameters including security and quality of service (QoS). The FEX does not run Spanning Tree Protocol even when multiple links are connected to the parent switch, thus giving customers loop-free, active-active connectivity between the two.
Lab Topology and Access
EveryoneortwostudentswillshareonePOD.EachonePODincludesoneNexus5000andoneNexus
2000switch.TheN2kisconnectedtointerfacee1/19ande1/20ofNexus5000.
Lab Topology
IP address and console assignment
Network: 128.107.65.192/27 255.255.255.224 Gateway 128.107.65.193 Terminal Server: 128.107.65.194 Servers IP Address Subnet Console APC APC Port Server-1 128.107.65.211 128.107.65.192 128.107.65.221 22,23 Server-2 128.107.65.212 128.107.65.192 128.107.65.221 20,21 Server-3 128.107.65.213 128.107.65.192 128.107.65.221 18,19 Server-4 128.107.65.214 128.107.65.192 128.107.65.221 15,17 Server-5 128.107.65.215 128.107.65.192 128.107.65.221 7,9 Server-6 128.107.65.216 128.107.65.192 128.107.65.221 5,6 Server-7 128.107.65.217 128.107.65.192 128.107.65.221 3,4 Server-8 128.107.65.218 128.107.65.192 128.107.65.221 1,2 N5k IP Address Subnet Console APC APC Port TM1 128.107.65.201 128.107.65.192 2033 128.107.65.220 23 TM2 128.107.65.202 128.107.65.192 2034 128.107.65.220 22 TM3 128.107.65.203 128.107.65.192 2035 128.107.65.220 21 TM4 128.107.65.204 128.107.65.192 2036 128.107.65.220 20 TM5 128.107.65.205 128.107.65.192 2037 128.107.65.220 19 TM6 128.107.65.206 128.107.65.192 2038 128.107.65.220 18 TM7 128.107.65.207 128.107.65.192 2039 128.107.65.220 17 TM8 128.107.65.208 128.107.65.192 2040 128.107.65.220 15 Core IP Address Subnet Console APC APC Port 6506-1 N/A N/A 2042 128.107.65.221 8 Terminal Server 128.107.65.194 login --> labops 128.107.65.220 6
Password: Terminal server Login password: labops. Enable password: nuovanuova Nexus 5000: admin/nbv12345 Windows Server: administrator/nuovanuova APC: apc/apc
Lab 1 Basic configuration of Nexus 5000
Lab Objective InthisexerciseyouwillconfigurethebasicconnectivitytoNexus5000andgetfamiliarwiththemostcommonlyusedNX‐OSCLI
Lab Procedure Step 1. Access the switch
Telnet to 128.107.65.194, sign on with a password of labops
Issue the show host command:
Nuova_TermServer>show host
[snip]
Host Port Flags Age Type Address(es)
tm1 2033 (perm, OK) 21 IP 1.1.1.1
tm2 2034 (perm, OK) 21 IP 1.1.1.1
tm3 2035 (perm, OK) 21 IP 1.1.1.1
tm4 2036 (perm, OK) 21 IP 1.1.1.1
tm5 2037 (perm, OK) 21 IP 1.1.1.1
tm6 2038 (perm, OK) 21 IP 1.1.1.1
tm7 2039 (perm, OK) 21 IP 1.1.1.1
tm9 2044 (perm, OK) 21 IP 1.1.1.1
tm8 2040 (perm, OK) 21 IP 1.1.1.1
tm6500 2042 (perm, OK) 21 IP 1.1.1.1
mds 2047 (perm, OK) 21 IP 1.1.1.1
Console in to the 5020 for your pod, for example, POD 1 would be TM1
Nuova_TermServer>tm1
Translating "tm1"
Trying tm1 (128.107.65.194, 2033)...
% Connection refused by remote host
Deleting login session
If you see the above error message you will need to get enable access to the terminal server and clear the line as shown below. The enable password is nuovanuova. If you do not see the error message above you can proceed to step 2
Nuova_TermServer>ena
Password:
Nuova_TermServer#clear line 33
[confirm] <- Hit enter
[OK]
Nuova_TermServer#tm1
Translating "tm1"
Trying tm1 (128.107.65.194, 2033)... Open Hit enter here
switch# <- Should see this prompt
Step 2. Configuretheswitchnameforthe5020inyourpod,managementVRF,enableSSHserver,andverifyconnectivitythroughthemanagementport.
switch(config)# hostname TM1 POD specific
TM1(config)# int mgmt 0
TM1(config-if)# ip address 128.107.65.201/24 Replace with the address for the 5020 in
TM1(config-if)# vrf context management
TM1(config-vrf)# ip route 0.0.0.0/0 128.107.65.193
TM1(config-vrf)# sh run int mgmt0
version 4.0(1a)N2(1)
interface mgmt0
ip address 128.107.65.201/24
TM1(config-vrf)# exit
TM1(config)# ssh key rsa 1024
rsa keys already present, use force option to overwrite them
TM1(config)# no ssh server enable
XML interface to system may become unavailable since ssh is disabled
TM1(config)# no ssh key
TM1(config)# ssh key rsa 1024
generating rsa key(1024 bits).....
.
generated rsa key
TM1(config)# ssh server enable
TM1(config)# end
Executeshowcommanddirectlyfromconfiguration
mode
DisableSSHanddeleteSSHkeyifthiserrormessageisprinted
Ping the default gateway for the management network.
TM1# ping 128.107.65.193 vrf management
PING 128.107.65.193 (128.107.65.193): 56 data bytes
64 bytes from 128.107.65.193: icmp_seq=0 ttl=254 time=2.167 ms
64 bytes from 128.107.65.193: icmp_seq=1 ttl=254 time=2.133 ms
64 bytes from 128.107.65.193: icmp_seq=2 ttl=254 time=2.034 ms
64 bytes from 128.107.65.193: icmp_seq=3 ttl=254 time=1.748 ms
64 bytes from 128.107.65.193: icmp_seq=4 ttl=254 time=1.841 ms
--- 128.107.65.193 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 1.748/1.984/2.167 ms
TM1#
Step3.TelnetorSSHtotheNexus5000inyourPODandgetfamiliarwithNX‐OSCLICheckthesoftwareversionandimagefilelocation
TM1# sh version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.
Software
BIOS: version 1.2.0
loader: version N/A
kickstart: version 4.0(1a)N2(1)
system: version 4.0(1a)N2(1)
BIOS compile time: 06/19/08
kickstart image file is: bootflash:/n5000-uk9-kickstart.4.0.1a.N2.1.bin
kickstart compile time: 2/25/2009 0:00:00 [02/25/2009 08:29:12]
NX‐OSsoftwareversion
Imagefilelocation
system image file is: bootflash:/n5000-uk9.4.0.1a.N2.1.bin
system compile time: 2/25/2009 0:00:00 [02/25/2009 08:56:57]
Hardware
cisco Nexus5020 Chassis ("40x10GE/Supervisor")
Intel(R) Celeron(R) M CPU with 2074308 kB of memory.
Processor Board ID JAB123900G8
Device name: TM1
bootflash: 974848 kB
Kernel uptime is 2 day(s), 22 hour(s), 51 minute(s), 8 second(s)
Last reset at 340459 usecs after Tue Mar 10 23:55:39 2009
Reason: Reset by installer
System version: 4.0(0)N1(2)
Service:
plugin
Core Plugin, Ethernet Plugin
ChecktheexpansionmoduleofNexus5000
TM1# show module
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ---------------------- ------------
1 40 40x10GE/Supervisor N5K-C5020P-BFS active *
2 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok
3 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok
Mod Sw Hw World-Wide-Name(s) (WWN)
--- -------------- ------ --------------------------------------------------
1 4.0(1a)N2(1) 1.0 --
2 4.0(1a)N2(1) 1.0 20:41:00:0d:ec:a3:db:80 to 20:44:00:0d:ec:a3:db:80
Expansionmodules
3 4.0(1a)N2(1) 1.0 20:81:00:0d:ec:a3:db:80 to 20:84:00:0d:ec:a3:db:80
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 000d.eca3.db88 to 000d.eca3.dbaf JAB123900G8
2 000d.eca3.dbb0 to 000d.eca3.dbb7 JAB1250004E
3 000d.eca3.dbb8 to 000d.eca3.dbbf JAB12500048
TM1#
Lab 2 Deploy N5k/N2k with EtherChannel
Lab Objective InthisexerciseyouwilllearnhowtoaddNexus2000toNexus5000usingEtherChannelasconnectionbetweenNexus2000andNexus5000
Lab Procedure N2kismanagedbytheparentswitchN5k.AftertheN5kdiscoverstheconnectedN2kitfirstchecksthesoftwarecompatibilitybetweenN2kandN5k.TheconfigurationdataisthenpushedtoN2k.TheconfigurationismaintainedbyN5k.ThereisnolocalconfigurationdatastoredonN2k.N5kcollectsandmaintainthetrafficstatisticsaswell.
TherearetwowaystoconnectN2ktoN5k:
• Etherchannel:FirstistobundleallthelinksbetweenN2kandN5k(Upto4)asanEtherChannelandtheEtherChannelwillcarrythetrafficforallthe48N2kports.
• “Staticpinning”:EachofthelinksbetweenN5kandN2kcarriestrafficforagroupofN2kports.
InthislabexercisewewilllearnhowtoconnectN2ktoN5kusingEtherChannel
Step1AddN2krelatedconfigurationonN5kTM1(config)# fex 100
TM1(config-fex)# pinning max-links 1
TM1(config-fex)# description "fex 100 pod1"
TM1(config-fex)# int e1/19-20
TM1(config-if-range)# channel-group 100
TM1(config-if-range)# int port-channel 100
TM1(config-if)# switchport mode fex-fabric
TM1(config-if)# fex associate 100
TM1(config-if)# end
Step2CheckN2kstatusandverifythatN2kisupandonline.
DisplayallN2ksconnectedtothisN5k
TM1# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
EtherChannelmodeisON.NoLACPbetweenN5kandN2k
EtherChannelisconsidersas
onelogicallink
100 fex 100 TM1 Online N2K-C2148T-1GE JAF1244ATEQ
DisplaythedetailinformationaboutaN2k
TM1# sh fex 100 detail
FEX: 100 Description: fex 100 TM1 state: Online
FEX version: 4.0(1a)N2(1) [Switch version: 4.0(1a)N2(1)]
FEX Interim version: 4.0(1a)N2(1)
Switch Interim version: 4.0(1a)N2(1)
Extender Model: N2K-C2148T-1GE, Extender Serial: JAF1244ATEQ
Part No: 73-12009-02
Card Id: 70, Mac Addr: 00:0d:ec:b1:3e:82, Num Macs: 64
Module Sw Gen: 19 [Switch Sw Gen: 19]
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/1
Fabric interface state:
Po100 - Interface Up. State: Active
Eth1/1 - Interface Up. State: Active
Eth1/2 - Interface Up. State: Active
Fex Port State Fabric Port Primary Fabric
Eth100/1/1 Down Po100 Po100
Eth100/1/2 Down Po100 Po100
Eth100/1/3 Down Po100 Po100
Eth100/1/4 Down Po100 Po100
Eth100/1/5 Up Po100 Po100
Eth100/1/6 Down Po100 Po100
Eth100/1/7 Down Po100 Po100
Eth100/1/8 Down Po100 Po100
Eth100/1/9 Down Po100 Po100
Eth100/1/10 Down Po100 Po100
Eth100/1/11 Down Po100 Po100
Eth100/1/12 Down Po100 Po100
Eth100/1/13 Down Po100 Po100
Eth100/1/14 Down Po100 Po100
Eth100/1/15 Down Po100 Po100
Eth100/1/16 Down Po100 Po100
Eth100/1/17 Down Po100 Po100
Eth100/1/18 Down Po100 Po100
Eth100/1/19 Down Po100 Po100
Eth100/1/20 Down Po100 Po100
Eth100/1/21 Down Po100 Po100
Eth100/1/22 Down Po100 Po100
Eth100/1/23 Down Po100 Po100
Eth100/1/24 Down Po100 Po100
Eth100/1/25 Down Po100 Po100
Eth100/1/26 Down Po100 Po100
Eth100/1/27 Down Po100 Po100
Eth100/1/28 Down Po100 Po100
Eth100/1/29 Down Po100 Po100
Eth100/1/30 Down Po100 Po100
Eth100/1/31 Down Po100 Po100
Eth100/1/32 Down Po100 Po100
Eth100/1/33 Down Po100 Po100
Eth100/1/34 Down Po100 Po100
Eth100/1/35 Down Po100 Po100
Eth100/1/36 Down Po100 Po100
Eth100/1/37 Down Po100 Po100
Eth100/1/38 Down Po100 Po100
Eth100/1/39 Down Po100 Po100
Eth100/1/40 Down Po100 Po100
Eth100/1/41 Down Po100 Po100
Eth100/1/42 Down Po100 Po100
Eth100/1/43 Down Po100 Po100
Eth100/1/44 Down Po100 Po100
Eth100/1/45 Down Po100 Po100
Eth100/1/46 Down Po100 Po100
Eth100/1/47 Down Po100 Po100
Eth100/1/48 Down Po100 Po100
Logs:
[03/18/2009 00:01:47.921366] Module register received
[03/18/2009 00:01:47.922299] Registration response sent
[03/18/2009 00:01:47.950304] Module Online Sequence
[03/18/2009 00:01:49.252440] Module Online
TM1#
Step3ConfigureN2kinterfacesTM1(config)# vlan 10X Replace X with your POD number
TM1(config-vlan)# int ethernet 100/1/1
TM1(config-if)# switchport access vlan 10X Replace X with your POD number
Pleasenotethatthefexinterfacessupport802.1qtrunkingandcanbeconfiguredastrunkport.
Step4ConfigureinterfaceconnectedtoCatalyst6500TM1(config)# feature lacp
TM1(config)# interface port-channel 1
TM1(config-if-range)# exit
TM1(config)# interface ethernet 1/1-2
TM1(config-if-range)# channel-group 1 mode active
Ethernet1/1 Ethernet1/2 added to port-channel1
TM1(config-if-range)# interface port-channel 1
TM1(config-if)# switchport mode trunk
TM1(config-if)# switchport trunk allowed vlan 10X Again POD specific
TM1(config-if)# no shut
Verify the Port channel is up and both interfaces are up and active in the port channel
TM1# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-------------------------------------------------------------------------------
1 Po100(SU) Eth LACP Eth1/1(P) Eth1/2(P)
Step5Verifyserverconnectivity
FindouttheIPaddressoftheserverinyourPOD.RemoteDesktoptotheserverandassignIPaddressfortheNICportconnectedtoFEX.TheIPaddressfortheserveris100.100.X.10with“X”beingyourPODID.
FirststarttheNetworkconnectionwindowbyclickingthenetworkiconontherightbottomofthedesktop.
Thendoubleclickthenetworkconnectionnamed“Nexus2000”.
AssignIPaddresstothe1GENICportnamed“Nexus2000”
TheIPaddressshouldbe“100.100.X.10”“X”beingthePODID.
POD ServerNICaddress Catalyst6500address
1 100.100.1.10 100.100.1.1
2 100.100.2.10 100.100.2.1
3 100.100.3.10 100.100.3.1
4 100.100.4.10 100.100.4.1
5 100.100.5.10 100.100.5.1
6 100.100.6.10 100.100.6.1
7 100.100.7.10 100.100.7.1
8 100.100.8.10 100.100.8.1
PingtheCatalyst6500SVIipaddress:100.100.X.1
Step6SimulatethereplacementofN2kbyshutdown/noshutdownthefabriclinkbetweenN5kandN2k.InthisstepwewillsimulatethescenariowhereweneedtoreplaceanexistingN2kbydisconnectandreconnecttheN2ktoN5k.NotethattheconfigurationofN2karerestoredautomaticallyafterthenewN2kcomesupandonline.
ChecktheN2kinterfacesconfiguration
TM1(config-if-range)# sh run int ethernet 100/1/1
version 4.0(1a)N2(1)
interface Ethernet100/1/1
spanning-tree port type edge
switchport access vlan 101
Shutdownallthefabricinterfaces
TM1(config)# int ethernet 1/19-20
TM1(config-if-range)# shut
2009 Mar 24 22:33:28 TM1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/2 is down
2009 Mar 24 22:33:28 TM1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from Ethernet1/2 to Ethernet1/1
2009 Mar 24 22:33:28 TM1 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number JAF1241BLGA)
2009 Mar 24 22:33:28 TM1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet100/1/1 is down (module removed)
2009 Mar 24 22:33:28 TM1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet100/1/2 is down (module removed)
2009 Mar 24 22:33:28 TM1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet100/1/3 is down (module removed)
2009 Mar 24 22:33:28 TM1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet100/1/4 is down (module removed)
2009 Mar 24 22:33:28 TM1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet100/1/5 is down (module removed)
2009 Mar 24 22:33:31 TM1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: Ethernet1/1 is down
2009 Mar 24 22:33:31 TM1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: port-channel1 is down
2009 Mar 24 22:33:31 TM1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from Ethernet1/1 to none
CheckalltheN2kinterfacesarenolongexistonN5k.NX‐OSmaintainstheconfigurationsothatwhentheFEX100comesuponlinetheconfigurationwillberestored.
TM1(config-if-range)# sh run int ethernet 100/1/1
^
% invalid command detected at '^' marker.
Invalid range
Bringupthefabricinterfaces
TM1(config-if-range)# no shut
TM1(config-if-range)# 2009 Mar 24 22:33:59 TM1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/2 is up
2009 Mar 24 22:33:59 TM1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1: first operational port changed from none to Ethernet1/2
2009 Mar 24 22:34:00 TM1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel1: Ethernet1/1 is up
2009 Mar 24 22:34:09 TM1 %SATCTRL-FEX100-2-SOHMS_ENV_ERROR: FEX-100 Module 1: Check environment alarms.
2009 Mar 24 22:34:10 TM1 %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line (Serial Number JAF1241BLGA)
ChecktheN2kcomesonlineandalltheN2kinterfacesrelatedconfigurationarerestoredautomatically
TM1(config-if-range)# sh run int ethernet 100/1/1
version 4.0(1a)N2(1)
interface Ethernet100/1/1
spanning-tree port type edge
switchport access vlan 101
TM1(config-if-range)# sh int ethernet 100/1/1
Ethernet100/1/1 is up
Hardware is 1000 Ethernet, address is 000d.ecb1.2cc2 (bia 000d.ecb1.2cc2)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Step7Checkthetrafficmappingbetweenfabricinterfaceandserverfacinginterfaces.TM1# sh interface Po100 fex-intf Fabric FEX Interface Interfaces --------------------------------------------------- Po100 Eth100/1/1 Eth100/1/2 Eth100/1/3 Eth100/1/4 Eth100/1/5 Eth100/1/6 Eth100/1/7 Eth100/1/8 Eth100/1/9 Eth100/1/10 Eth100/1/11 Eth100/1/12 Eth100/1/13 Eth100/1/14 Eth100/1/15 Eth100/1/16 Eth100/1/17 Eth100/1/18 Eth100/1/19 Eth100/1/20 Eth100/1/21 Eth100/1/22 Eth100/1/23 Eth100/1/24 Eth100/1/25 Eth100/1/26 Eth100/1/27 Eth100/1/28 Eth100/1/29 Eth100/1/30 Eth100/1/31 Eth100/1/32 Eth100/1/33 Eth100/1/34 Eth100/1/35 Eth100/1/36 Eth100/1/37 Eth100/1/38 Eth100/1/39 Eth100/1/40 Eth100/1/41 Eth100/1/42 Eth100/1/43 Eth100/1/44 Eth100/1/45 Eth100/1/46 Eth100/1/47 Eth100/1/48
Lab 3 Deploy N2k with Static Pinning WhenN2kisconnectedtoN5kwithstaticpinning,thetrafficof48serverfacingportaredistributedamongtheuplinksinafixedfashion.The48portsaredividedintothesamenumberofgroupsasthenumberofuplinksbetweenN2kandN5k.Trafficforeachsetoftheseportsiscarriedbyoneuplink,unliketheN2k/N5kdeploymentscenariowithPortChannelwheretrafficdistributionisdeterminedbythehashalgorithmresult.
Step1Firstremovetheconfigurationofpreviouslab.
TM1(config)# no int Po100
TM1(config-fex)# int e1/19-20
TM1(config-if-range)# no switchport mode
TM1(config-if-range)# no fex associate 100
TM1(config-if-range)# no fex 100
TM1(config)#
AddN2krelatedconfigurationonN5k
TM1(config)# fex 100
TM1(config-fex)# pinning max-links 2
Change in Max-links will cause traffic disruption.
TM1(config-fex)# description "POD1 FEX 100"
TM1(config-fex)# int ethernet 1/19-20
TM1(config-if-range)# switchport mode fex-fabric
TM1(config-if-range)# fex associate 100
Step2CheckN2kstatusandverifythatN2kisupandonline.TM1# sh fex 100
FEX: 100 Description: POD1 FEX 100 state: Online
FEX version: 4.1(3)N1(1) [Switch version: 4.1(3)N1(1)]
Extender Model: N2K-C2148T-1GE, Extender Serial: JAF1307BHCE
Part No: 73-12009-05
pinning-mode: static Max-links: 2
Fabric port for control traffic: Eth1/19
Fabric interface state:
Eth1/19 - Interface Up. State: Active
Eth1/20 - Interface Up. State: Active
TM1#
Step3ConfigureN2kinterfaces
ThisnumberhastomatchwiththeactuallinksbetweenN2kandN5k.
TM1(config)# vlan 10X Replace X with your POD number
TM1(config-vlan)# int ethernet 100/1/1
TM1(config-if)# switchport access vlan 10X Replace X with your POD number
Step4Checkthetrafficmappingbetweenfabricinterfaceandserverfacinginterfaces.TM1# sh interface ethernet 1/19 fex-intf Fabric FEX Interface Interfaces --------------------------------------------------- Eth1/19 Eth100/1/24 Eth100/1/23 Eth100/1/22 Eth100/1/21 Eth100/1/20 Eth100/1/19 Eth100/1/18 Eth100/1/17 Eth100/1/16 Eth100/1/15 Eth100/1/14 Eth100/1/13 Eth100/1/12 Eth100/1/11 Eth100/1/10 Eth100/1/9 Eth100/1/8 Eth100/1/7 Eth100/1/6 Eth100/1/5 Eth100/1/4 Eth100/1/3 Eth100/1/2 Eth100/1/1 TM1# sh interface ethernet 1/20 fex-intf Fabric FEX Interface Interfaces --------------------------------------------------- Eth1/20 Eth100/1/48 Eth100/1/47 Eth100/1/46 Eth100/1/45 Eth100/1/44 Eth100/1/43 Eth100/1/42 Eth100/1/41 Eth100/1/40 Eth100/1/39 Eth100/1/38 Eth100/1/37 Eth100/1/36 Eth100/1/35 Eth100/1/34 Eth100/1/33 Eth100/1/32 Eth100/1/31 Eth100/1/30 Eth100/1/29 Eth100/1/28 Eth100/1/27 Eth100/1/26 Eth100/1/25
Step5 VerifyserverconnectivitybypingtheSVIinterfaceofCatalyst6500“100.100.X.1”.Step6 Bringdownonefabricinterfaceandverifytheserverinterfacespinnedtothisfabricinterfacesarebroughtdown
TM1(config)# int e1/19
TM1(config-if)# shut
TM1(config-if)# sh int ethernet 100/1/1
Ethernet100/1/1 is down (fabricIfDown)
Hardware: 1000 Ethernet, address: 000d.ecc9.ec42 (bia 000d.ecc9.ec42)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is access
auto-duplex, 1000 Mb/s
<snip> ….
TM1(config-if)#
Lab 4 Configuration of FCoE
Lab Objective
In this exercise you will configure basic connectivity to the switch, 802.1q trunking, LACP Port Channels, Fibre Channel Port Channels, and FCoE connectivity.
Note: The exercises in the lab were performed on POD 1. Please use the correct switches, VSANs, VLANs, IP addresses, and host associated with your assigned POD.
Lab Procedure
Continuing from the N5K/N2K lab, access the Nexus 5000 switch for your pod via the terminal server.
Step 1. In this step you will enable FCoE feature for the Nexus 5000 switch.
First turn on the FCoE feature. This will require a reboot of the switch to go into effect.
TM1# configure terminal
TM1(config)# feature fcoe
2008 Aug 13 22:56:01 switch %$ VDC-1 %$ %PFMA-2-FC_LICENSE_DESIRED: FCoE/FC feature will be enabled after the configuration is saved followed by a reboot
TM1(config)# copy running-config startup-config
[########################################] 100%
Packaging and storing to flash: \
TM1(config)# reload
Step 2. Creating a VSAN for the Fibre Channel Database
Once the switch comes back up, login into the switch and create the vsan needed for your POD. Please note that the convention used would be VSAN 20X, where X is your Pod #. Example would be for Pod1, would need to create VSAN 201, Pod 2 would be VSAN 202, etc.
TM1#configure terminal
TM1(config)# vsan database
TM1(config-vsan-db)# vsan 20X Again POD specific
TM1(config-vsan-db)# end
Step 3. Creating a SAN Port-Channel
TM1(config)# interface san-port-channel 10X Again POD specific
TM1(config-if)# interface fc2/3-4
TM1(config-if)# channel-group 10X force Again POD specific
fc2/3 fc2/4 added to san-port-channel 10X and disabled
please do the same operation on the switch at the other end of the channel, then do "no shutdown" at both ends to bring them up
Please note that the MDS9134 has already been initially configured to be in switching mode and the Port-Channels has been created on it as well. So the students nor instructor "DO NOT" need to do anything on the MDS switch.
TM1(config-if)# no shut
TM1(config)# interface san-port-channel 10X Again POD specific
TM1(config-if)# no shut
TM1(config-if)# end
Verify the SAN Port Channel is up and Trunking.
TM1# show interface san-port-channel 10X
san-port-channel 10X is trunking
Hardware is Fibre Channel
Port WWN is 24:0b:00:0d:ec:6d:90:40
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Speed is 8 Gbps
Trunk vsans (admin allowed and active) (1,20X)
Trunk vsans (up) (1,20X)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
[snip]
TM1# show san-port-channel summary
--------------------------------------------------------------
Group Port Type Protocol Status Member Ports
Channel
-----------------------------------------------------------------
101 san-port-channel 101 FC NONE U fc2/3 (P)
fc2/4 (P)
-----------------------------------------------------------------
Step 4. Configuring a VLAN to be FCoE capable. This also does the VLAN to VSAN mapping.
Note: The VLAN number will be following the convention of VLAN 20X, where X is your Pod #. Example would be VLAN 201 for Pod 1, VLAN 202 for Pod 2, etc.
TM1# configure terminal
TM1(config)# vlan 20X Again POD specific
TM1(config-vlan)# fcoe vsan 20X Again POD specific
TM1(config-vlan)# show vlan fcoe
VLAN VSAN Status
-------- -------- --------
201 201 Operational
Step 5. Create the virtual Fibre Channel interface, and move it to the VSAN assigned to your pod.
TM1(config-vlan)# interface vfc5 The "vfc" number is not relevant and can be any number the student chooses. But for consistency, please use #5 for all pods. All student pods should have interface "vfc5" when this is completed.
TM1(config-if)# bind interface Ethernet1/5
TM1(config-if)# no shut
Add Virtual Fibre Channel (vfc5) Interface to the VSAN database for your Pod.
TM1 (config-if)#vsan database
TM1 (config-vsan-db)#vsan 20X interface vfc5 Again POD specific
Step 6. Allow the FCoE VLAN to traverse interface ethernet 1/5 TM1(config-vsan-db)# interface ethernet 1/5
TM1(config-if)# switchport mode trunk
TM1(config-if)# switchport trunk allowed vlan 10X, 20X Again POD specific
TM1(config-if)# switchport trunk native vlan 10X Again POD specific
Step 7. Verify vfc interface is up Note: If the student does a "show interface vfc5", the vFC interface will be down awaiting the FCoE VLAN assignment, please be patient and the CNA port will come up.
vfc5 is down (STP not forwarding in FCoE Mapped VLAN)
Verify your CNA host is logged into the fabric and registered with the name server
TM1# show interface vfc5
vfc5 is up
Bound interface is Ethernet1/5
Hardware is GigabitEthernet
Port WWN is 20:04:00:0d:ec:b2:51:7f
Admin port mode is F
snmp link state traps are enabled
Port mode is F, FCID is 0x3c0001
Port vsan is 201
Beacon is turned unknown
5 minutes input rate 344 bits/sec, 43 bytes/sec, 0 frames/sec
5 minutes output rate 472 bits/sec, 59 bytes/sec, 0 frames/sec
983 frames input, 99240 bytes
0 discards, 0 errors
1089 frames output, 126152 bytes
0 discards, 0 errors
Step 8. Verifying zoning of the pod is active
TM1# show zoneset active vsan 20X
zoneset name pod1 vsan 201
zone name pod1-server vsan 201
* fcid 0x73001e [pwwn 21:00:00:04:cf:26:96:89] Disk
* fcid 0x3c0001 [pwwn 10:00:00:00:c9:85:05:48] CNA
Note: Below is the list of all the active zonesets for all the Pods.
zoneset name pod1 vsan 201
zone name pod1-server vsan 201
* fcid 0x73001e [pwwn 21:00:00:04:cf:26:96:89]
* fcid 0x3c0001 [pwwn 10:00:00:00:c9:85:05:48]
zoneset name pod2 vsan 202
zone name pod2-server vsan 202
* fcid 0xdb006e [pwwn 21:00:00:04:cf:67:74:ff]
* fcid 0x470001 [pwwn 10:00:00:00:c9:85:03:c8]
zoneset name pod3 vsan 203
zone name pod3-server vsan 203
* fcid 0xa10002 [pwwn 22:00:00:20:37:f7:da:31]
* fcid 0x2b0001 [pwwn 10:00:00:00:c9:85:08:b0]
zoneset name pod4 vsan 204
zone name pod4-server vsan 204
* fcid 0x4d0000 [pwwn 10:00:00:00:c9:85:00:d8]
* fcid 0xe10002 [pwwn 21:00:00:20:37:f7:da:31]
zoneset name pod5 vsan 205
zone name pod5-server vsan 205
* fcid 0x3300cd [pwwn 50:00:00:e0:10:5c:24:e1]
* fcid 0x910001 [pwwn 10:00:00:00:c9:85:08:98]
zoneset name pod6 vsan 206
zone name pod6-server vsan 206
* fcid 0x7000e2 [pwwn 22:00:00:04:cf:fe:4e:a0]
* fcid 0x280000 [pwwn 10:00:00:00:c9:85:01:f8]
zoneset name pod7 vsan 207
zone name pod7-server vsan 207
* fcid 0xa80073 [pwwn 22:00:00:04:cf:67:47:fe]
* fcid 0x820001 [pwwn 10:00:00:00:c9:85:02:6c]
zoneset name pod8 vsan 208
zone name pod8-server vsan 208
* fcid 0xd400e8 [pwwn 21:00:00:04:cf:fb:20:a9]
* fcid 0xdf0001 [pwwn 10:00:00:00:c9:85:02:71]
Step 9. Checking on the Server Pod sees the new SAN Disk. Use “Remote Desktop” to the server for your POD and verify the host can see storage
Step 10. Open Disk Management.
Step 11. Find the LUN allocated for your server based on the size.