FCIP Theory

Revision 0210

CFP380

1

FCIP Theory

Revision 0210

CFP380

2

FCIP Theory

Revision 0210

CFP380

3

Footnote 1: Distances that utilize native FC can span 500km; these solutions incorporate dark fiber, C/DWDM, and form a single fabric.

For additional FCIP details, reference RFC 3821 – Fibre Channel Over TCP/IP (FCIP).

Brocade does not recommend FCIP for use in every distance extension scenario: no technical solution can be all things to all people. FCIP has inherent performance, reliability, data integrity, and manageability limitations when compared to native FC solutions. Delay and packet loss may create bottlenecks in IP networks. FCIP can support very long distances, as long as the carrier network is extremely high performance and reliable. FCIP is typically deployed when long-haul applications are not business critical, and do not need especially high performance. FCIP may not be suitable for tape, since tape usage will often fail if packets are dropped. In addition to its performance limitations, FCIP troubleshooting and performance analysis requires evaluating all aspects of the IP LAN and WAN networks in addition to all FC nodes, switches, and routers, which can make it more complex to manage than other extension options.

FCIP Theory

Revision 0210

CFP380

4

FC traffic over an IP network

• Interconnection of islands of Fibre Channel storage area networks over IP-based networks

• Distance Extension over IP LAN/WAN/MAN

FCIP Theory

Revision 0210

CFP380

5

After FC frames destined for devices at the remote side are encapsulated into TCP packets, a standard IP header is added to each packet. The packet is then sent to the next hop (usually an Ethernet router).

FCIP Theory

Revision 0210

CFP380

6

FCIP Theory

Revision 0210

CFP380

7

FCIP Theory

Revision 0210

CFP380

8

FCIP Theory

Revision 0210

CFP380

9

Footnote 1: An Upgrade License is required to enable all 16 FC ports and 6 GbE ports.

Footnote 2: Two GbE ports can be configured for copper (via built-in RJ45) or optical (via SFP module) cable connectivity. The other four GbE ports offer optical SFP module only. SFP ports will also accept 1 Gbps copper SFPs, Avago part number ABCU-5700RZ-BR1.Footnote 3: Link speeds of 1 Gbps require a 4 Gbps Brocade-branded SFP.

FCIP Theory

Revision 0210

CFP380

10

Footnote 1: The top two GbE ports (GE0 and GE1) are configured for copper via built-in RJ45. In standard configuration, users have the option of using either the top two GbE ports, which are configured for copper SFPs, or the bottom two, left-most, GbE ports (GE0 and GE1) which are configured for optical SFPs. The remaining four GbE ports can use either optical or copper via SFP module up to 1 Gbps. It is possible to configure GE0 as copper and GE1 as optical and vice versa.

FCIP Theory

Revision 0210

CFP380

11

Footnote 1: Fabric OS identifies power supplies left to right as Power Supply #1 and Power Supply #2. The Brocade 7800 requires a minimum of one power supply (connected and powered on) for switch operation. Footnote 2: Fabric OS identifies the fan assemblies left to right as Fan #1 and Fan #2.

Weight: 9.3 kg (20.6 lb)1Dimensions:Width: 42.87 cm (16.88 in)

Height: 4.30 cm (1.70 in) or 1UDepth: 61.0 cm (24.02 in)

Power Consumption: 84 Watts nominal, 91 Watts maximum

Both weight and power consumption numbers assume Brocade 7800 with two power supply/fan FRUs and zero SFPs installed.

FCIP Theory

Revision 0210

CFP380

12

The base unit has 2 GbE ports available for use. They can be the two copper ports, or the two fibre ports. The default is copper.

The upgrade license is named the 7800 Upgrade license.

FCIP Theory

Revision 0210

CFP380

13

Please refer to the Release Notes for the most up-to-date information.

Enabling either of the 2 x10 GbE ports requires a 10 GbE license, which is a slot-based license. Footnote 1: The supported operational modes are:

• 10 x 1 GbE port

• 10 x 1 GbE ports and 1 x 10 GbE port• 2 x 10 GbE ports

FCIP Theory

Revision 0210

CFP380

14

Each Cavium processor has 4G of DDR memory.

FCIP Theory

Revision 0210

CFP380

15

Footnote 1: Typically, port groups for Condor2 ASICs are 8-port trunk groups. There are 40 ports on a Condor2 ASIC. As seen in the notes section of the previous slide, there is a 5-port trunk to each of the four Blaster FPGAs. By definition, a trunk must be created from the same octet. This means that each octet from a Blaster trunk octet has 3 ports remaining in the octet. Thus, Brocade engineers used the available ports to create 2 other 2-port trunks to use some of the remaining ports.

Weight: 3.2 kg (7.0 lb)1Dimensions:

Width: 3.60 cm (1.41 in)Height: 42.06 cm (16.56 in)

Depth: 29.89 cm (11.77 in)Power Consumption: 235 Watts nominal Approximate weight with 0 SFPs installed. The minimum power consumption of the Brocade FX8-24 Extension Blade is 235 watts with 0 optical SFPs installed running at 8 Gbps.

FCIP Theory

Revision 0210

CFP380

16

Footnote 1: Requires the 10Gbps license.

FCIP Theory

Revision 0210

CFP380

17

FCIP Theory

Revision 0210

CFP380

18

Footnote 1:

FCIP Tunnels: •B7800 - up to 8 VE_Ports

•FX8-24 - up to 20 VE_PortsFCIP Trunking:

•B7800 - up to 4 circuits per trunk•FX8-24 - up to 4 circuits per trunk for 1 GbE; up to 10 circuits per trunk for 10 GbE

Footnote 2: 10 GbE, Advanced Extension and FICON Accelerator licenses for FX8-24 supported in DCX and DCX-4S are slot based licenses

Footnote 3: Not supported in initial Fabric OS v6.3.0 release.

FCIP Theory

Revision 0210

CFP380

19

Certified (Support is through the SFP vendor)

1/2/4 Gbps ELWL - 50km

1/2/4 Gbps ELWL - 100km

1/2/4 Gbps CWDM - 1470nm - 80km

1/2/4 Gbps CWDM - 1490nm - 80km

1/2/4 Gbps CWDM - 1510nm - 80km

1/2/4 Gbps CWDM - 1530nm - 80km

1/2/4 Gbps CWDM - 1550nm - 80km

1/2/4 Gbps CWDM - 1570nm - 80km

1/2/4 Gbps CWDM - 1590nm - 80km

1/2/4 Gbps CWDM - 1610nm - 80km

1/2/4 Gbps DWDM 80km (25 wavelengths from 1530nm-1560nm)

2/4/8 Gbps CWDM 40km(8 wavelengths from 1290nm-1430nm)

For the most up-to-date support, refer to the following site:

http://www.brocade.com/products-solutions/products/tranceivers/index.page

FCIP Theory

Revision 0210

CFP380

20

FCIP Theory

Revision 0210

CFP380

21

FCIP Theory

Revision 0210

CFP380

22

C2 and GE2 = Condor2 and GoldenEye2 ASICs

FCIP Theory

Revision 0210

CFP380

23

FCIP Theory

Revision 0210

CFP380

24

Footnote 1: FC flow control mechanisms include R_RDYs and ACKs. FC communications also utilize long distance modes, BB credits, and VC channels.

FCIP Theory

Revision 0210

CFP380

25

Footnote 1: There are 8 VE_Ports for 6 physical ge ports on the 7800 16/6. An FX8-24 blade can support 20 VE_Ports, and therefore 20 FCIP tunnels. Each FCIP tunnel is associated with a specific VE_Port. On FX8-24 blades, and on the 7800 switch, VE_Ports do not have to be associated with a particular GbE port.VE_Ports 12 through 21 may use GbE ports ge0 through ge9, or they may use XGE port 1. VE_Ports 22 through 31 can only be used by XGE port 0. The total bandwidth cannot exceed 20 Gbps.

FCIP Theory

Revision 0210

CFP380

26

Footnote 1: As of Fabric OS v6.3.0, VEX_Ports are not supported on the FX8-24.

FCIP Theory

Revision 0210

CFP380

27

The 7500E and 7500 switches and the FR4-18i blade support only one connection per GbE port, so strictly speaking, the FCIP circuit concept does not apply

FCIP Theory

Revision 0210

CFP380

28

VE ports and GbE ports are no longer 1:1 associated as they were on the 7500 and FR4-18i.

VE and GbE ports can have a 1:1 association, but they are not limited by the design.

Footnote 1: Configuring a tunnel with more than one circuit requires an Advanced Extension license. Without a license present, a second circuit will not be allowed to be configured. The administrator will receive a message stating as such.

FCIP Theory

Revision 0210

CFP380

29

FCIP trunking requires multiple FCIP circuits, therefore FCIP trunking cannot be implemented on the 7500E, 7500 switch, and the FR4-18i blade.

Footnote 1: Configuring a tunnel with more than one circuit requires an Advanced Extension license. Without a license present, a second circuit will not be allowed to be configured. The administrator will receive a message stating as such.

FCIP Theory

Revision 0210

CFP380

30

Time out values:

• Default FICON is 1 sec• Default FC is 4 sec

• Depending on the solution being extended, the time out values may need to be changed from default.

FCIP Theory

Revision 0210

CFP380

31

Example tunnel/circuit creation:

A tunnel using VE_Port 16 is created with an initial circuit 0 with a maximum rate of 1Gbps, metric of 0.

• IPPM_LINK_UP.bandwidth = 1GbpsCircuit 1 is added to Tunnel 16 with a max rate of 500Mbs, metric 0.

• TUNNEL_UPDATE.bandwidth = 1.5 GbpsCircuit 2 is added to Tunnel 16 with a max rate of 1Gps, metric 1.

• Since this is a higher metric, it is considered a standby and no TUNNEL_UPDATE is generated.

Circuit 3 is added to Tunnel 16 with a max rate of 1Gps, metric 1.• Since this is a higher metric, it is considered a standby and no

TUNNEL_UPDATE is generated.

FCIP Theory

Revision 0210

CFP380

32

Example circuit failures/recovery (after initial creation state)

Circuit 1 fails.• Since this was a low metric circuit (active) we will try to re-establish, no

TUNNEL_UPDATE is sent to CP (and no updated will be sent once it re-establishes).

Circuit 0 fails, 1 is still failed.

• Since this is the last of the low metric circuits, the tunnel will failover to the next lowest circuits, which are circuits 2 and 3. Since the combined bandwidth of circuits 2 and 3 is different than circuits 0 and 1, a TUNNEL_UPDATE.bandwidth = 2.0 Gbps is generated to the CP.

Circuit 0 and 1 come back on line.• Since these are again the low metric circuits, we will resume traffic on these

circuits and 2 and 3 become the hot standbys. Since the bandwidth has changed again, a TUNNEL_UPDATE.bandwidth = 1.5 Gbps is generated to the CP. Note that the bandwidth was reduced.

Example circuit deletions (after initial creation state);

Circuit 1 is deleted.• Since this was a low metric circuit, all traffic will be directed to circuit 0. Again,

the bandwidth has changed so the tunnel will generate a TUNNEL_UPDATE.bandwidth = 1.0 Gbps to the CP.

Deleting circuits 2 or 3 at this point would have no affect since they are standby and the bandwidth is not included in the calculations.

Example circuit metric changes (after initial creation state);

Circuit 1’s metric is changed to 1.• Since this was a low metric circuit, all traffic will be directed to circuit 0. The

bandwidth has changed, so the tunnel will generate a TUNNEL_UPDATE.bandwidth = 1.0 Gbps to the CP. This would basically have the same affect on bandwidth as if deleting the circuit instead.

Circuit 3’s is changed to metric 0 (circuit 1 still has a metric of 1)• Since this was a high metric circuit changed to a low. The bandwidth has

changed, so FTNL will generate a TUNNEL_UPDATE.bandwidth 2.0 Gbps to the CP (circuit 0 and circuit 3 are now “active”, circuits 1 and 2 are now in “standby” mode).

FCIP Theory

Revision 0210

CFP380

33

Footnote 1: Each 10 GbE port is assigned to a different FCIP subsystem. Because of this, tunnels cannot be created traversing the two ports. All circuits that reside in a tunnel must belong to the same FCIP subsystem.

FCIP Theory

Revision 0210

CFP380

34

FCIP Theory

Revision 0210

CFP380

35

3 Modes of operation for the Ethernet ports on FX8-24:

1G Only Mode: 10 x 1 Gbps ports are available for use (GE0 through GE9)10G Only: 2 x 10 Gbps ports are available for use (XGE0 and XGE1)1

Dual Mode: 10 x 1 Gbps ports and 1 x 10 Gbps port available for use (GE0 through GE9, XGE0)

FCIP Theory

Revision 0210

CFP380

36

Footnote 1: IP address subnets are used so that the TCP supervisor can route packets to the destination subnets. The FCIP subsystem contains an IP routing table that directs packets with a tunnel destination to the correct circuits and tunnels.Footnote 2: The B7800 contains 6 x 1 Gbps connections between field-programmable gate arrays (FPGAs) and the Ethernet ports.Footnote 3: If multiple circuits are configured on the same physical port, they can contain addresses from the same subnet or different subnets.

Footnote 4: For the FX8-24, each FCIP complex has ten 1 Gbps connections between FPGAs and the Ethernet ports. This is the reason that the XGE ports require 10 circuits to achieve 10 Gbps bandwidth.

FCIP Theory

Revision 0210

CFP380

37

• Packet loss re-transmissions are compounded when errors are bursty. Selective Acknowledgement (SACK) is an extension to a protocol which allows the acknowledge reception of specific packets or messages.The SACK option RFC 2883 [18] allows the receiver to acknowledge multiple lost packets in a single ACK, enabling faster recovery. An FCIP Entity may negotiate use of TCP SACK and use it for faster recovery from lost packets and holes in TCP sequence number space.Footnote 1: SACK improves loss detection, retransmission techniques, and enables faster recovery.

FCIP Theory

Revision 0210

CFP380

38

Hardware compression is performed at FC ingress and de-compression is performed at FC egress.

Compression before encapsulation allows the TCP and FCIP headers to be visible on the network.

FCIP Theory

Revision 0210

CFP380

39

Footnote 1: A value of 1 enables hardware compression. The 7800 switch provides two additional levels of software compression. Settings 2 and 3 provide incrementally higher compression ratios that can be used to improve performance on slower links. A value of 0 disables compression.

FCIP Theory

Revision 0210

CFP380

40

FCIP Theory

Revision 0210

CFP380

41

FCIP Theory

Revision 0210

CFP380

42

FCIP Theory

Revision 0210

CFP380

43

FCIP Theory

Revision 0210

CFP380

44

FCIP Theory

Revision 0210

CFP380

45

Ports ge4 and ge9 are available to be used, but are not in the example above.

FCIP Theory

Revision 0210

CFP380

46

FCIP Theory

Revision 0210

CFP380

47

FCIP Theory

Revision 0210

CFP380

48

Best practice is to use a commit rate that uses 90% of the bandwidth allocated to the FCIP traffic. In this example, both devices are configured for 100% of the total bandwidth of the WAN gateway, which means that the router is oversubscribed 2:1. This configuration can lead to many dropped frames and errors.

FCIP Theory

Revision 0210

CFP380

49

The configuration above is a better solution than the previous slide, but still has shortcomings. While the router is not oversubscribed, the bandwidth available for the WAN is not fully utilized when there is either a failure of one of the devices, or a simple case of a device needing less bandwidth than what is configured. This can leave the link underutilized during certain times.

FCIP Theory

Revision 0210

CFP380

50

Footnote 1: Best practice dictates setting maximum commit rates to 90% of physical connection.

The best solution would to use Adaptive Rate Limiting (ARL) to better utilize the available WAN bandwidth. By configuring each device with a minimum commit rate of 50% of the allocated bandwidth, each device is then guaranteed a minimum amount of bandwidth at all times. If an outage occurs in one of the devices, or a device is not using all of its minimum committed rate bandwidth, the other active device can utilize the extra bandwidth up to its configured maximum commit rate.

FCIP Theory

Revision 0210

CFP380

51

FCIP Theory

Revision 0210

CFP380

52

FCIP Theory

Revision 0210

CFP380

53

FCIP Theory

Revision 0210

CFP380

54

Brocade’s original virtual channel model (pre-Condor2/GoldenEye2) divides an ISL into 8 virtual channels to insure that traffic of multiple priorities can travel across the link at the same time, without being disrupted, or disrupting other traffic.

���������������� �����������

VC # Assigned to

0 Class F

1 Class 2 Ack/Link Control

2 Data

3 Data

4 Data

5 Data

6 Class 3 Multicast

7 Broadcast/Multicast

All Data Traffic

FCIP Theory

Revision 0210

CFP380

55

The QoS feature only comes into play if there is contention on the link. If there is no congestion on the link QoS will not engage.

The order of operations during congestion is as follows and repeats as necessary:1. VC0 then, 2. 6 frames of High priority traffic then,

3. 3 frames of Medium priority traffic then, 4. 1 frame of Low priority traffic

Results of QoS on a congested link:

FCIP Theory

Revision 0210

CFP380

56

FCIP Theory

Revision 0210

CFP380

57

Footnote 1: TCP port used is 3225.

FCIP Theory

Revision 0210

CFP380

58

FCIP Theory

Revision 0210

CFP380

59

FCIP Theory

Revision 0210

CFP380

60

FCIP Theory

Revision 0210

CFP380

61

FCIP Theory

Revision 0210

CFP380

62

FCIP Theory

Revision 0210

CFP380

63

FCIP Theory

Revision 0210

CFP380

64

FCIP Theory

Revision 0210

CFP380

65

FCIP Theory

Revision 0210

CFP380

66

FCIP Theory

Revision 0210

CFP380

67

FCIP Theory

Revision 0210

CFP380

68

FCIP Theory

Revision 0210

CFP380

69

FCIP Theory

Revision 0210

CFP380

70

FCIP Theory

Revision 0210

CFP380

71

FCIP Theory

Revision 0210

CFP380

72

FCIP Theory

Revision 0210

CFP380

73

FCIP Theory

Revision 0210

CFP380

74