network design and configuration guide -...

FlexFrame® Orchestrator

Version 1.1A

Network Design and Configuration Guide

Edition March 2015 Document Version 1.1

Fujitsu Limited

© Copyright Fujitsu Technology Solutions 2014

FlexFrame® and PRIMERGY™ are trademarks or registered trademarks of Fujitsu Limited in

Japan and other countries.

SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in Germany

and in several other countries

Linux® is a registered trademark of Linus Torvalds

SUSE® Linux is a registered trademark of Novell, Inc., in the United States and other coun-

tries

Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates

Intel® and PXE® are registered trademarks of Intel Corporation in the United States and oth-

er countries

MaxDB® is a registered trademark of MySQL AB, Sweden

MySQL® is a registered trademark of MySQL AB, Sweden

NetApp® and the Network Appliance® logo are registered trademarks and Network Appli-

ance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and other countries.

VMware®, ESX®, ESXi, VMware vCenter, VMware vSphere are registered trademarks or

trademarks of VMware, Inc. in the United States and/or other jurisdictions.

Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation and Intel

Corporation

Windows® and Word® are registered trademarks of Microsoft Corporation

All other hardware and software names used are trademarks of their respective companies.

All rights, including rights of translation, reproduction by printing, copying or similar methods,

in part or in whole, are reserved.

Offenders will be liable for damages.

All rights, including rights created by patent grant or registration of a utility model or design,

are reserved.

Delivery subject to availability. Right of technical modification reserved.

Network Design and Configuration Guide

Contents

1 Introduction ..................................................................................................... 1 1.1 Purpose of this Document ................................................................................. 1 1.2 Notational Conventions ..................................................................................... 1 1.3 Document History .............................................................................................. 1 1.4 Related Documents ........................................................................................... 2

2 Concept and Design ........................................................................................ 3 2.1 Switch Groups ................................................................................................... 3 2.2 Network Speed .................................................................................................. 5 2.3 Jumbo Frames .................................................................................................. 5 2.4 Network Connection .......................................................................................... 6 2.5 Linux Bonding .................................................................................................... 7 2.6 ESX NIC Teaming ............................................................................................. 8 2.7 NetApps Interface Grouping .............................................................................. 9 2.8 BX Link State Propagation ................................................................................ 9 2.9 Virtual LAN ...................................................................................................... 10 2.10 Uplink Connection ........................................................................................... 13 2.11 Client LAN Connection .................................................................................... 13

3 FlexFrame Network Versions ....................................................................... 15 3.1 Small Version .................................................................................................. 15 3.2 Directly Connected Versions ........................................................................... 15 3.3 Enterprise Version ........................................................................................... 19

4 Configuration Commands ............................................................................ 21 4.1 Linux Bonding Interface ................................................................................... 21 4.2 Linux VLAN Interface ...................................................................................... 22 4.3 NetApp Filer Configuration .............................................................................. 22 4.4 Network Switch Configuration ......................................................................... 23

5 API - Network ................................................................................................. 28 5.1 XML Description .............................................................................................. 28 5.2 HW Quickguide ............................................................................................... 30 5.3 Perl Module ..................................................................................................... 31

6 Abbreviations ................................................................................................ 35

7 Glossary ......................................................................................................... 37

8 Index ............................................................................................................... 40

Network Design and Configuration Guide 1

1 Introduction

1.1 Purpose of this Document

This document describes the FlexFrame networking concept and design and possible

networking topologies.

The reader should be familiar with IP networking and needs basic knowledge about

Ethernet-based networking with switches and virtual LANs. Knowledge of network switch

configuration is recommended. The knowledge of the FlexFrame Installation Guide and

the FlexFrame Administration and Operation Guide would be an advantage.

1.2 Notational Conventions

The following conventions are used in this manual:

Additional information that should be observed.

Warning that must be observed.

fixed font Names of paths, files, commands, and system output.

<fixed font> Names of variables.

fixed font User input in command examples

(if applicable using <> with variables).

1.3 Document History

Document Version Changes Date

1.0 First Edition 2014-12-03

1.1 Related Documents modi-

fied

2015-03-25

Introduction Related Documents

2 Network Design and Configuration Guide

1.4 Related Documents

FlexFrame® – Administration and Operation

FlexFrame® – HW Characteristics Quickguides

FlexFrame® – Installation and Configuration of LVM 2.1 Standard Edition

FlexFrame® – Installation Guide for SAP Solutions

FlexFrame® – Installation of a FlexFrame Environment

FlexFrame® – Management Tool

FlexFrame® – FlexFrame Agents Installation and Administration

FlexFrame® – FlexFrame Messenger Installation and Administration

FlexFrame® – FlexFrame LogAgent Installation and Administration

FlexFrame® – Security Guide

FlexFrame® – Technical White Paper

FlexFrame® – Upgrading FlexFrame Orchestrator 1.0A to 1.1A

ServerView Documentation

SUSE Linux Enterprise Server Documentation


2 Concept and Design

The design goals for the FlexFrame network concept:

avoid single points of failure (SPOF)

avoid unnecessary complexity

are achieved by the following means:

use redundant network

use a lightweight design

use standard interfaces and configurations

keep interface configuration as identical as possible on all server systems

keep design flexible enough to meet the requirements from small to medium up

to enterprise sized installations

reduce count of needed interfaces per system

and results in the concept described below.

2.1 Switch Groups

Switch 2

Switch 1

SwitchGroup

LAN A

LAN B

Application Node

PORT a

Application Node

Blade Rack

PORT c PORT b

Control Node 2

PORT a

PORT c PORT b

LAN A

LAN B

Application Node

LAN A

LAN B

Control Node 1

LAN A

LAN B

NAS

LAN A

LAN B

The very simplified drawing above may visualize the general concept (only data connec-

tions are considered). The core of the design is the switch group which provides redun-

dant usable ports. If one member of the switch group fails the ports of the other members

Concept and Design


are still working. Every end system and the uplink are using this redundancy as described

later. In this way a failure of a single member of the switch group is tolerated and allows

continuous operation.

FlexFrame distinguishes the following switch group types:

CAT3750-STACK

Two up to nine switches of the Cisco Catalyst 3750G, 3750E or 3750X switch family

are building a Cisco Catalyst switch stack. The switches are connected via Cisco

StackWise cabling and behave like a single switch. For more than 4 switches with

10GbE ports the StackWise cabling may be a bottleneck.

ICX6430-STACK

Two up to four switches of the Brocade ICX 6430 switch family are building a Bro-

cade FastIron traditional switch stack. The switches stacking ports are connected via

standard ethernet cabling and the switch stack behave like a single switch.

ICX6450-STACK

Two up to eight switches of the Brocade ICX 6450 switch family are building a Bro-

cade FastIron traditional switch stack. The switches stacking ports are connected via

standard ethernet cabling and the switch stack behave like a single switch.

NEXUS5000-VPC

Exactly two switches (nexus50xx) of the Cisco Nexus 5000 Switch family are building

a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel

between the switches FlexFrame configures two ports on each switch. For the vPC

peer-keepalive link as a necessary alternative Layer 3 connectivity between the

switches FlexFrame uses the mgmt0 interface IP addresses as recommended from

Cisco.

NEXUS5500-VPC

Exactly two switches (nexus55xx) of the Cisco Nexus 5000 Switch family are building

a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel

between the switches FlexFrame configures two ports on each switch. For the vPC

peer-keepalive link as a necessary alternative Layer 3 connectivity between the

switches FlexFrame uses the mgmt0 interface IP addresses as recommended from

Cisco.

VCS-LC

Exactly two Brocade VDX 6740 switches are building a Brocade VCS Logical Chas-

sis. For the fabric inter switch link as a necessary special connection between the

switches FlexFrame configures two ports on each switch. In addition Out-of-Band

management is used as basis for the logical chassis management.

For further details about supported switches see the FlexFrame Support Matrix.

For further details about the switches see the Cisco manuals.

For further details about the purpose of the different switch group types see below.

Concept and Design


2.2 Network Speed

FlexFrame supports network connections for data communication with the following net-

work speeds:

1Gbit/sec (1GbE)

10Gbit/sec (10GbE)

Pay attention to the fact that 10GbE connected servers may also request a 10GbE con-

nected NAS system and 10GbE uplinks between.

For further details about supported end systems see the FlexFrame Support Matrix.

2.3 Jumbo Frames

IEEE 802.3 defines the maximal payload size of Ethernet frames to 1500 bytes. This re-

sults in using a maximal transfer unit size of 1500 bytes on network interfaces (MTU size

1500).

NetApp recommends the use of MTU size 9000 (jumbo frames) for filer communication.

MTU size 9000 is standard with cDOT.

FlexFrame allows the use of MTU size of 9000 in storage networks and uses MTU size

1500 in all other networks. Switches are always configured to be able to transmit frames

with a payload of 9000 bytes.

Concept and Design


2.4 Network Connection

Network connection is established by connecting ports of end systems to ports of network

devices which itself are connected to other network devices. The characteristics of con-

nected ports must be compatible and a proper connector must be used.

FlexFrame classifies the ports of network devices as follows:

classifier meaning supported from

TX 10/100/1000 ethernet port

RJ45 connector

Cisco Catalyst 3750 family

Brocade ICX 6430, ICX6450

FX 1GbE ethernet port

SFP connector

Cisco Catalyst 3750G

Cisco Nexus 5000 family

Brocade ICX 6430

Brocade VDX 6740

TGBx 10GbE ethernet port

SFP+ connector

Cisco Nexus 5000 family

Cisco Catalyst 3750E

Brocade VDX 6740

TX ports are requested from end systems for RJ45 ports for management or 1GbE data

connection and for uplinks using this uplink port media. Use Cat5 cable for connections

up to 100 Mbit/sec and cat5e cable for connections up to 1000 Mbit/sec.

FX ports are requested for uplinks using this uplink port media. Use LC-LC fiber optic ca-

ble to connect SFP connectors or 1GBASE-CU SFP cable (Twinax) to connect ports suit-

ed for SFP connectors.

TGBx ports are requested from end systems for 10GbE data connection and 10GbE up-

links. Use LC-LC fiber optic cable to connect SFP+ connectors or 10GBASE-CU SFP ca-

ble (Twinax) to connect ports suited for SFP+ connectors. Use SC-LC fibre optic cable to

connect a Cisco Catalyst 3750E 10GbE port with another SFP+ connector.

Consider the support matrix of devices with respect to the supported SFP/SFP+ connect-

ors.

Concept and Design


2.5 Linux Bonding

Linux provides a mechanism called bonding which can aggregate several physical inter-

faces to a virtual interface (bond). A bond interface can be used like a usual interface and

is operational as long as at least one physical interface assigned to the bond is opera-

tional. Several bonding configurations are available.

Switch 2

Switch 1

active eth0

eth1

rack server or

server blade

bond0

active

When running Linux on a physical server (rack server or server blade) used as Applica-

tion Node FlexFrame aggregates 2 physical interfaces using adaptive load balancing

(mode 6) and failure detection relies solely on the link state provided by the adapter (MII

monitoring). This is an active/active method and has no requirements on the link partners

which can be switch blades or switch group switches. The requirement on the adapter to

be able to change the MAC address is fulfilled from all supported servers.

On the Control Nodes bonding mode 1 (active-backup) is used. No requirements have to

be fulfilled neither from the link partners nor from the adapter. Failure detection relies

solely on the link state provided by the adapter (MII monitoring).

Concept and Design


2.6 ESX NIC Teaming

ESX provides a mechanism called NIC teaming which is used if multiple physical Ether-

net adapters are assigned to a single virtual switch (vSwitch). A vSwitch is operational as

long as at least one physical Ethernet adapter assigned to the vSwitch is operational.

Several NIC teaming policies are available and can be set on the port group level.

Switch 2

Switch 1

active vmnic0

vmnic1

rack server or

server blade

vSwitch active

FlexFrame assigns 2 physical Ethernet adapters in ESX default configuration to the

vSwitch and puts all these adapters in the active list for the FlexFrame generated port

groups. The adapters are used with load balancing based on the originating virtual Port

ID and failure detection relies solely on the link state provided by the adapter

This method has no requirements on the link partners which can be switch blades or

switch group switches.

Concept and Design


2.7 NetApps Interface Grouping

NetApps Data ONTAP provides a mechanism called interface grouping which can aggre-

gate several physical interfaces to a virtual interface. An interface group can be used like

a usual interface and is operational as long as at least one physical interface assigned to

the group is operational. Several modes are available.

Switch 2

Switch 1

SwitchGroup

NetApp Filer

ifgrp

e0a

e0b

active

active

FlexFrame aggregates at least 2 physical interfaces into a dynamic multimode interface

group which is compliant with IEEE 802.3ad. All physical interfaces are active. The corre-

sponding ports on the switch group switches must build a channel also complaint with

IEEE 802.3ad (LACP).

2.8 BX Link State Propagation

BX switch blades provide a mechanism which allows associating several downstream

ports with an upstream port or channel for link state propagation. If the link of the up-

stream port or channel goes down the link of all associated downstream ports are also

forced to go down. In this way a bonding driver on a server blade will get aware of the

upstream problem and switches the traffic of concerned interfaces.

Concept and Design


active

eth0

eth1

server blade

bond0 active

Switch 2

Switch 1

SwitchGroup BX chassis

Switchblade 1

Switchblade 2

FlexFrame associates all downstream ports of a switch blade and a single upstream

channel for link state propagation. The upstream channel is compliant with IEEE 802.3ad.

The corresponding ports on the switch group switches must also build a channel com-

plaint with IEEE 802.3ad (LACP). The channel consists of at least two ports with crosso-

ver cabling. More ports can be used for higher bandwidth.

2.9 Virtual LAN

Virtual LAN (VLAN) is a mechanism which allows having multiple separated networks on

the same physical network infrastructure. A widely used standard is tagged VLAN accord-

ing IEEE 802.1q. The protocol specifies an extension of the Ethernet header by a VLAN

identifier (tag). A packet belongs to the VLAN identified by the tag. Communication be-

tween VLANs is only possible by routing as is common for usual networks. For compati-

bility reason packets without a tag are also allowed. A switchport can be configured to

assign received packets without tag to a specific VLAN and remove the tag when a pack-

et of this VLAN should be send. The VLAN is then called native or untagged. End sys-

tems which access only one VLAN or cannot handle VLAN tags are connected this way.

End systems used within FlexFrame like NAS systems or servers running Linux are able

to handle VLAN traffic according IEEE 802.1q.

FlexFrame can handle different pools with independent networks. The traffic between

communication partner types (client – server, server – server and server – storage) is al-

so separated. Therefore VLANs are used as helpful mechanism to reduce the necessary

physical infrastructure and increase the flexibility with respect to the pooling concept.

According IEEE 802.1q the VLAN tag has a value range from 1 to 4094 but there are

switch specific restrictions.

Concept and Design


Cisco Catalyst 3750 Switch Family

1005 VLANs are supported. Normal usable VLANs are in the range from 1 to 1001. VLAN

numbers 1002 through 1005 are reserved for Token Ring and FDDI VLANs. VLAN num-

bers 1006 through 4094 are extended-range VLANs and usable if not already used inter-

nally for routed ports.

Cisco Nexus 5000 Switch Family

507 VLANs are supported. Normal usable VLANs are in the range from 1 to 1005. VLAN numbers 1006 through 4094 are extended-range VLANs and usable with the following restrictions. The state is always active and VLANs 3968 to 4047 and 4094 are reserved for internal use.

Brocade ICX 6400 Switch Family

32 VLANs are supported. Normal usable VLANs are in the range from 1 to 4094. VLAN

IDs 4087, 4090, and 4093 are reserved for Brocade internal use only. VLAN 4094 is re-

served for use by Single STP.

Brocade VDX 6700 Switch Family

507 VLANs are supported. Normal usable VLANs are in the range from 1 to 3959. VLAN IDs 3960 through 4090 are internally-reserved VLAN IDs. However, the reservedvlan command can modify this range. VLANs above 4090 are not configurable. VLAN 1002 is the default FCoE VLAN.

Concept and Design


Physical Network Layout & Virtual LANs per Pool

V4: Client VLAN

V3: Control VLAN

V2: Server VLAN

V1: Storage VLAN

V1

V1 V2 V3 V4

Bond 0

eth 2 eth 1eth 0

IPMI

Onboard

LAN Port 3

Onboard

LAN Port 1

V1 V2 V3 V4

Bond 0

eth 2 eth 1eth 0

IPMIV1 V2 V4

Bond 0

eth 2 eth 1eth 0

IPMI

V3

ifgrp a1a

Physical View:

Redundant Connections

Control Node 1 Control Node 2 Application Node 1

Application Node n

Logical View: VLANs

Onboard

LAN Port 2

Onboard

LAN Port 3

Onboard

LAN Port 1

Onboard

LAN Port 2eXa eXb

Onboard

LAN Port 3 LAN Port 1 LAN Port 2

Switch 2Switch 1

NAS

System

Control Nodes and Linux Application Nodes (RX300), NetApp NAS (7-mode)

Switchgroup

The picture shows the physical and logical view of networking in FlexFrame with respect

to a single pool. That is:

An Application Node is assigned to a pool and connected to the pools client, server and

storage VLAN. When running on ESX server for every connection an interface is created

on the vSwitch, when running native the interfaces for client and server LAN are created

as VLAN interface on top of the bond. The storage LAN is also used for PXE boot and

assigned native because no tags can be handled in this early phase. Therefore the stor-

age LAN is accessible via the bond directly.

An ESX server is connected to all VLANs necessary for the Application Nodes and to the

control LAN which is assigned native.

IPMI interfaces of rack servers and management blades of BX cabinets are connected

native to the control LAN.

A Control Node is connected to all VLANs, the control LAN is native.

A NAS system is connected to the control LAN and storage VLANs of pools the NAS sys-

tem supports, all VLANs are tagged.

Concept and Design


2.10 Uplink Connection

Switch 2

Switch 1

SwitchGroup

FlexFrame can assign uplinks to a switch group. An uplink consists of ports from different

switch group members building a channel and carries the traffic of all used VLANs. An

uplink is used to connect a switch group to another switch group or to the customer cor-

porate LAN.

Examples for supported topologies are given in chapter 3.

When connected to customer corporate LAN the connected customer ports must also

build a channel. The channel must be compliant with IEEE 802.3ad (LACP) and all used

VLANs must be allowed tagged.

2.11 Client LAN Connection

For accessing a FlexFrame infrastructure solution connection to the client LAN must be

established. In general this means the client LAN must be available in the context of a

router and appropriate routing must be configured. This is a manual task and the custom-

er decides which customer network should have access to a special client LAN.

To bring a client LAN in the context of a customer router different ways are available:

a) If uplinks of switch groups are connected to the customer corporate LAN e.g. as

described in chapter 3.3 the client LANs are already available and no further ar-

rangements are necessary.

Concept and Design


b) Special uplinks for client LAN connections are configured (e.g. see

ff_swgroup_adm.pl) and connected to the customer corporate LAN like normal

uplinks. The VLANs on these uplinks are restricted to client LAN VLANs.

c) Special ports for client LAN connections are configured. These CLAN ports are

thought being connected to a router and therefore configured as spanning tree

edge ports. The ports can be distributed over several switch groups and it is

possible to have particular ports for every client LAN. Only Ports of switches of a

CAT3750-STACK are allowed to be used as CLAN ports. See below an exam-

ple how CLAN ports configured for a single client LAN can be used.

The ports configured for client LAN connection are connected to Layer 3 ports of different

Layer 3 switches. An IP address is assigned to every Layer 3 port and the Layer 3

switches are configured to build a logical router e.g. via HSRP. The logical router sup-

ports a virtual IP address (IP3) which is used as standard gateway for the special client

LAN.

Switch 2

Switch 1

SwitchGroup 1

Switch 2

Switch 1

SwitchGroup 2

uplink channel

Virtual Gateway Adress IP3

Layer 3 port IP1

Layer 3 port IP2

outside FlexFrame


3 FlexFrame Network Versions

The FlexFrame network design is flexible enough to meet requirements from small to en-

terprise size networks. In the following some examples are given.

3.1 Small Version

This version is based on one switch group as there is no need for

splitting FlexFrame into different data centers

providing more switch ports as one switch group can offer.

Switch 2

Switch 1

CAT3750-STACK

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

client LAN

All devices are connected to a single CAT3750-STACK. The connection of the devices is

shown simplified only with regard to the redundant connections for data communication.

For client LAN connection two ports are configured on different members of the switch

group. No uplink needs to be configured.

3.2 Directly Connected Versions

Topologies according to the rules mentioned in 2.10 with more than one switch group and

without any uplink connection to the customer corporate LAN are called directly connect-

ed versions.

The following version may meet the requirements if

FlexFrame has to be split into two data centers or

one switch group might have not enough ports for all devices

FlexFrame Network Versions


Example1: Two CAT3750-STACKs

Switch 2

Switch 1

CAT3750- STACK

Switch 2

Switch 1

CAT3750- STACK

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

Uplink Channel

client LAN

All devices are connected to one of two CAT3750-STACKs. The connection of the devic-

es is shown simplified only with regard to the redundant connections for data communica-

tion. For client LAN connection one port is configured on every switch group.

The switch groups are connected via uplink channel a) directly without any other network Layer2 device between per copper or for

greater distance per fiber optic cable.

b) via network devices allowing a Layer2 connection e.g. DWDM for even greater

distance. In this case the network devices between must be transparent to the

switch groups, especially regarding link state. That means, if a port is shutdown

on one switch group the corresponding port on the other switch group must see

a link down event. Also if the connection between is broken both ports must see

a link down event.



Example 2: single NEXUS5000-VPC with one CAT3750-STACK

Switch 2

Switch 1

NEXUS5000-VPC

Switch 2

Switch 1

CAT3750- STACK

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

Uplink Channel

client LAN

The CAT3750-STACK is directly connected to the NEXUS5000-VPC without any other

network Layer2 device between. The service LAN ports of rack servers and the control

nodes and the management ports of the nexus switches are connected to the CAT3750-

STACK. The end systems connected to NEXUS5000-VPC are 10GbE-connected. For

client LAN connection two ports are configured on different members of the CAT3750-

STACK.



Example 3: two NEXUS5000-VPCs each with connection to one CAT3750-STACK

Switch 2

Switch 1

CAT3750-STACK

Switch 2

Switch 1

CAT3750-STACK

Switch 2

Switch 1

NEXUS5000-VPC

Switch 2

Switch 1

NEXUS5000-VPC LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

LAN A

LAN B Node

uplink channel

client LAN

The service LAN ports of rack servers and the control nodes are connected to a

CAT3750-STACK which is connected to the NEXUS5000-VPC. The end systems con-

nected to NEXUS5000-VPC are 10GbE-connected.

The NEXUS5000-VPC switch groups are connected via uplink channel

directly without any other network Layer2 device between per fiber optic cable.

via network devices allowing a Layer2 connection e.g. DWDM for greater dis-

tance. In this case the network devices between must be transparent to the

switch groups, especially regarding link state. That means, if a port is shutdown

on one NEXUS5000-VPC switch group the corresponding port on the other

NEXUS5000-VPC switch group must see a link down event. Also if the connec-

tion between is broken both ports must see a link down event.



3.3 Enterprise Version

The possibilities of directly connected versions are restricted. Greater configurations need

uplinks to customer corporate LAN e.g. the customer core switches.

Customer own Core

Switch

Customer own Core

Switch

Customer Corporate LAN

Node

Node

Node

Node

Node

Node

Node

CAT3750-STACK

Uplink Channel

NEXUS5000-VPC

NEXUS5000-VPC

Node

Uplink Channel Uplink Channel

CAT3750-STACK

CAT3750-STACK

Uplink Channel

Uplink Channel

Each NEXUS5000-VPC switch group is connected with its uplink ports to core switch

ports. CAT3750-STACK switch groups may be connected to core switches or to

NEXUS5000-VPC switch groups. The switch groups may be connected to different core

switches. Like the Uplink the corresponding ports on the core switch(es) must:

build a channel

carry all VLANs used within FlexFrame

transfer traffic without blocking

The core switches are not part of FlexFrame, neither of automated configuration nor sup-

port. Configuration changes at core switches have to be done manually. There are no

messages or instructions from FlexFrame to support this.


4 Configuration Commands

As described in Chapter 2 FlexFrame uses high availability configurations concerning

network on many devices. This chapter describes how this is done in detail.

4.1 Linux Bonding Interface

On the control nodes the bonding configuration is performed via configuration files

/etc/sysconfig/network/ifcfg-bond0:

STARTMODE=onboot

BOOTPROTO=static

IPADDR=192.168.20.1

NETMASK=255.255.255.0

NETWORK=192.168.20.0

BROADCAST=192.168.20.255

BONDING_MASTER=yes

BONDING_SLAVE_0=eth0

BONDING_SLAVE_1=eth1

BONDING_MODULE_OPTS="miimon=100 mode=6"

/etc/sysconfig/network/ifcfg-eth0

/etc/sysconfig/network/ifcfg-eth1 (both are the same):

STARTMODE=auto

On application nodes running native the bonding configuration is performed during boot

via commands. The network parameters are obtained through DHCP and include the IP

address for the storage LAN.

Configuration Commands


4.2 Linux VLAN Interface

VLAN configuration is performed via configuration files

/etc/sysconfig/network/ifcfg-vlan<VLAN ID>

e.g. for VLAN 10: /etc/sysconfig/network/ifcfg-vlan10:

STARTMODE='onboot'

BOOTPROTO='static'

IPADDR=192.168.200.11

NETMASK=255.255.255.0

NETWORK=192.168.200.0

BROADCAST=192.168.200.255

ETHERDEVICE='bond0'

WIRELESS='no'

4.3 NetApp Filer Configuration

The configuration has to be done manually e.g. via serial cable (see the corresponding

HW Characteristics Quickguide you find on the Service CD in doc/hwinfo).


4.4 Network Switch Configuration

The network switch configuration is performed via commands. The following tables shows

Cisco IOS and Cisco NX-OS configuration commands for different purposes.

Cisco Catalyst 3750 Family Configuration (IOS commands)

Basic configuration service password encryption

enable password secret

clock timezone CET 2

clock summer-time cet recurring last Sun Mar 2:00 last

Sun Oct 3:00

vtp mode transparent

udld aggressive

no ip http server

snmp-server community public RO

spanning-tree mode rapid-pvst

spanning-tree loopguard default

no spanning-tree optimize bpdu transmission

spanning-tree extend system-id

VLAN configuration for a system

with a pool pool1

vlan 10

name client-pool1

vlan 11

name storage-pool1

vlan 12

name server-pool1

vlan 13

name control

port configuration for a data port

of a rack server used as applica-

tion node

interface GigabitEthernet1/0/1

description Node rx300s6 dataNic 1

switchport trunk encapsulation dot1q

switchport trunk native vlan 11

switchport trunk allowed vlan 10-12

switchport mode trunk

no ip address

speed auto

no mdix auto

udld port

storm-control broadcast level 20

storm-control multicast level 10

no shutdown




spanning-tree portfast trunk


of a rack server used as esx-

server


description Node rx300s6



switchport trunk allowed vlan 10-12,20-22


no ip address

speed auto

no mdix auto

udld port



no shutdown


port configuration for a IPMI port

of a rack


description Node rx300s6 mgmtNic



switchport trunk allowed vlan 13


no ip address

speed auto

no mdix auto

udld port



no shutdown


channel configuration e.g. for an

uplink

interface PortChannel 1

description ISL Uplink




no shutdown






no ip address




speed auto

mdix auto

udld port



cdp enable

channel-group 1 mode active

no shutdown






no ip address

speed auto

mdix auto

udld port



cdp enable


no shutdown

Save configuration copy running-config startup-config

For details see the “Catalyst 3750 Switch Software Configuration Guide” at

http://www.cisco.com.

Cisco Nexus 5000 Family Configuration (NX-OS commands)

Basic configuration no feature telnet

no telnet server enable

cfs eth distribute

feature udld

feature lacp

feature vpc

vpc domain 2

peer-keepalive destination 172.11.6.19

vrf context management

ssh key rsa 2048

udld aggressive

spanning-tree mode rapid-pvst

http://www.cisco.com/




spanning-tree loopguard default

VLAN configuration for a system

with a pool pool1

vlan 10

name client-pool1

vlan 11

name storage-pool1

vlan 12

name server-pool1

vlan 13

name control


of a rack server used as applica-

tion node

interface Ethernet1/1

description Node rx300s6 dataNic 1


switchport trunk allowed vlan 10-12


spanning-tree port type edge trunk

udld aggressive



no shutdown


of a rack server used as esx-

server


description Node rx300s6




spanning-tree port type edge trunk

udld aggressive



no shutdown

vpc configuration e.g. for an up-

link (same vpc number on both

nexus switches of the switch

group, per convention equal to

port-channel number)

interface port-channel2


switchport trunk allowed vlan 1,10-13,20-22


vpc 2



no shutdown



switchport trunk allowed vlan 1,10-13,20-22


udld aggressive







no shutdown

Save configuration copy running-config startup-config

For details see the “Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration

Guide” at http://www.cisco.com.

http://www.cisco.com/

API - Network


5 API - Network

FlexFrame defines an API for network as described below. Providing an implementation

of this API will increase the amount of FlexFrame supported switch types by the switch

types captured from this implementation.

The implementation has to comprise the following units:

XML Description

HW Quickguide

Perl module (SwitchConfig_xxx.pm, xxx to be replaced)

Template Definitions (SwitchConfig_xxx.tpl, xxx to be replaced)

5.1 XML Description

The XML description defines the switch so far as necessary for usage by FlexFrame as

follows.

switchTypeDefinition ::=

<switch id="switchType" maxLinkAggregates="number" >

PortDefinition+

expansionModulSlotsDefinition?

groupTypeDefinitionReference+

switchMigrationDefinition?

</switch>

portDefinition ::=

<ports portFamily="portTypes" first="number" count="number">

<format type="formatType">format</format>

</ports>

expansionModulSlotsDefinition ::=

<expansionModuleSlots first="number" count="number">

expansionModulRefDefinition+

</expansionModuleSlots>

expansionModulRefDefinition ::=

<expansionModule>expansionModule</expansionModule>

referenceToSwitchgroupTypeDefinition ::=

<group id="groupType" />

switchMigrationDefinition ::=

<switchMigration>

migrateToDefinition+

API - Network


</switchMigration>

migrateToDefinition ::=

<migrateTo>switchType</migrateTo>

expansionModuleDefinition ::=

<switch id="expansionModuleType">

PortDefinition+

</switch>

switchgroupTypeDefinition ::=

<switchgroup id="groupType" maxSwitches="number"

groupMgmtIPs="number" memberMgmtIPs="number"

impl="impl" domain="domain" >

managementPortDefinition?

memberConnectDefinition*

</switchgroup>

managementPortDefinition ::=

< memberMgmtPort id="interfaceName" type="portType" peer="peerType" />

memberConnectDefinition ::=

<memberConnect id="number" peer="peerType" lag="number" >

portUseDefinition+

</memberConnect>

portUseDefinition ::=

<portUse id="number" port="portID" />

FlexFrame allows the following definitions

formatType

formatType comment

1 format with one %s replacement for port number

2 format with two %s replacements, first for slot, second for port

number

needs number of slot in context

portTypes

portType comment

TX 10/100/1000 Mb/s Ethernet Ports with autosensing, RJ-45 connector

FX 1000 Mb/s SFP based Ethernet Port

API - Network


TGBx 10Gb/s SFP+ based Ethernet Port

portTypes may be a colon separated list of types e.g. FX:TGBx if a port can be used with

SFP or SFP+.

peerType

peerType port should be used for connection to

AN Application Node

BB Black Box

CLAN clientLAN

CLANUPLINK clientLAN as channel

CN Control Node

ESX ESX Server

FILER NAS System

GW Gateway

KVM KVM Server

MGMT Management Interface

NONE nothing

SW Switch Group (used on Switch Blades)

SWB Switch Blade (used on Switch Group)

UPLINK Switch Group / Customer Switches

VPC_PEERLINK memberConnect for Nexus

STACKING memberConnect for ICX67xx

VCS-ISL memberConnect for VCS fabric

… memberConnect for …

A new peer type for memberConnect is introduced solely per definition in hard-

wareXML. Introduction of another new peerType must be possible solely per adding

some definitions to the SwitchConfig_xxx.tpl file.

For examples see the actual hardwareXML file in the config directory of the FlexFrame

Service DVD.

5.2 HW Quickguide

The HW Quickguide describes the manual actions to be performed when a switch will be

integrated into the FlexFrame landscape, especially how the switch will get its initial

configuration. For examples see the doc/hwinfo directory of the FlexFrame Service

DVD.

API - Network


5.3 Perl Module

The Perl module is the interface to the switch handling. The Perl module includes the

template definitions file which is acting as a configuration file e.g. can be extended from

FlexFrame development when adding a new peerType. The Perl module fulfills the fol-

lowing requirements:

no persistent data

no dependencies on Perl modules other than defined in the sample

logging using methods error, warn, info and debug of logger provided by

get_logger()

Package Name

FSC_FF::SwitchConfig_xxx (xxx according hardwareXML impl value)

Class methods to be implemented

new()

return a new instance of the module

Instance methods to be implemented

register(groupCTX, groupID, switchID, attributes)

register access attributes for a switch identified by groupCTX, groupID,

switchID

return empty string if ok, error string else

Note: A switchID is unique within a group, a groupID is a cabinetID (group-

CTX = ‘cab’) or a switchGroupID (groupCTX = ‘swg’)

Note: If there is a management IP address associated to the group the method is

called only once with switchID and stype empty.

attributes = {

'name' => <name of object (may be used for file naming)>

'gtype' => <group type according XML, e.g. NEXUS5000-VPC>

'stype' => <switch type according XML, e.g. NEXUS5010>

'username' => <user to be used for login>

'password' => <password for user>

'mgmtip' => <mgmt ip for login>

}

globals(groupCTX, groupID, switchID, attributes)

request for switch globals configuration for specified object


Note: Configurations including the globals request may be applied manually us-

ing the HW Quickguide.

Note: The method is called for every object with specified mgmtip.

attributes = {

# use cn1,cn2 as ntp server, snmp trap target and syslog target

API - Network


# enable snmp agent for snmp read

# create user for login as given with register

# assign mgmtip to mgmtinterface

# assign mgmtip as given with register if different

# name as given with register may be configured as prompt

'cn1' => <IP address of CN1>

'cn2' => <IP address of CN2>

'snmp' => <snmp read community>

'name' => <name of object (may be used for file naming and

prompt)>

'mgmtip' => <mgmt ip in cdr notation> e.g. 'xxx/24'

'mgmtvlan' => <vlan associated to mgmt subnet>

'domainID' => <number, e.g. domainID of NEXUS5000-VPC>

}

name(groupCTX, groupID, switchID, attributes)

request for commands to set a new name


Note: The method is called for every object with a name associated.

attributes = {

'name' => <new name (for prompt)>

}

password(groupCTX, groupID, switchID, attributes)

request for commands to set a new password for user username

returns empty string if ok, error string else

Note: The method is called for every group member regardless if necessary or

not.

attributes = {

'username' => <name of user>

'password' => <new password>

}

vlanadd(groupCTX, groupID, switchID, attributes)

request for vlan add configuration



not.

attributes = {

'vlan' => <vlanID to be added>

'name' => <vlan name to be associated, eg.'client>

}

vlanrem(groupCTX, groupID, switchID, attributes)

request for vlan remove configuration



not.

API - Network


attributes = {

'vlan' => <vlanID to be removed>

}

portuse(groupCTX, groupID, switchID, portID, attributes)

request for port use configuration


attributes = {

'type' => <portfamily>, eg.'TX',

'peer' => <type of peer system>, eg.'AN',see table portPeerType

below

'desc' => <description string>,

'vlan' => <vlan ids>, eg.'t11,t12,u13'

'lag' => <link aggregate id>, eg.'5' # if port is part of a lag

'lagadd' => '1' # if lag is new to config with subsequent apply

}

portunuse(groupCTX, groupID, switchID, portID, attributes)

request for port unuse configuration


attributes = {

'vlan' => <vlan ids>, eg.'t11,t12,u13'

'lag' => <link aggregate id>, eg.'5' # if port is part of a lag

'lagrem' => '1' # if lag is obsolete to config with subsequent ap-

ply

}

portvlanadd(groupCTX, groupID, switchID, portID, attributes)

request for port vlan add configuration for a used port


Note: The method is called for every port of a lag regardless if necessary or not.

attributes = {

'vlan' => <vlan ids to be added>, eg.'t11,t12,u13'

'all' => <all vlan ids to be assigned>, eg.'t11,t12,u13'

'lag' => <link aggregate id>, eg.'5' # if port part of lag

}

portvlanrem(groupCTX, groupID, switchID, portID, attributes)

request for port vlan remove configuration for a used port


Note: The method is called for every port of a lag regardless if necessary or not.

attributes = {

'vlan' => <vlan ids to be removed>, eg.'t11,t12,u13'

'all' => <all vlan ids to be assigned>, eg.'t11,t12,u13'

'lag' => <link aggregate id>, eg.'5' # if port part of lag

}

portdown(groupCTX, groupID, switchID, portID, attributes)

request for port down configuration for a used port

API - Network



attributes = {

}

portup(groupCTX, groupID, switchID, portID, attributes)

request for port up configuration for a used down port


attributes = {

}

backup(groupCTX, groupID, switchID, attributes)

request for human readable configuration for backup purposes

prefer selected configuration type else available type

tftp may be used to copy files

append timestamp to backup filename


attributes = {

'ip' => <controlLAN address of CN running tftpd –s /tftpboot>

'dir' => <directory on CN where to save configuration>

'select' => <startup | running>

}

apply(attributes)

apply previously collected requests to associated switches and forget them

save generated config commands and communication protocol to temporary di-

rectory

print messages to STDOUT/STDERR as necessary


if dryrun then suppress communication to real switches

if todo then append messages concerning manual apply according HW Quick-

guide to the referenced string instead of direct output to STDOUT

attributes = {

'dryrun' => <0|1>

'tmpdir' => <path to temporary directory>

'todo' => <reference to string>

}


6 Abbreviations

cDOT Clustered Data ONTAP

DART Data Access in Real Time

DHCP Dynamic Host Configuration Protocol

FF FlexFrame

FF4S FlexFrame for SAP

FFO FlexFrame Orchestrator

IP Internet Protocol

LAN Local Area Network

MAC Media Access Control

MII Media Independent Interface

NAS Network Attached Storage

NIC Network Interface Card

ONTAP Open Network Technology for Appliance Products

PXE Preboot Execution Environment

SPOF Single Point Of Failure

TFTP Trivial File Transfer Protocol

UDP User Datagram Protocol

VLAN Virtual Local Area Network

VPC Virtual Port Channel


7 Glossary

Application Node (AN)

A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This

definition includes Application Servers as well as Database Servers.

Blade

A special form factor for computer nodes.

Client LAN

Virtual network segment within FlexFrame, used for client-server traffic.

Computing Node

From the SAP ACI perspective: A host that is used for applications.

Control LAN

Virtual network segment within FlexFrame, used for system management traffic.

Control Node (CN)

A physical computer system, controlling and monitoring the entire FlexFrame land-

scape and running shared services in the rack (dhcp, tftp, ldap etc.).

Control Station

A Control Node in an SAP ACI environment.

Dynamic Host Configuration Protocol (DHCP)

DHCP is a protocol for assigning dynamic IP addresses to devices on a network.

Dynamic Host Configuration Protocol server

A DHCP server provides configuration parameters specific to the DHCP client host,

required by the host to participate on the Internet.

Ethernet

A Local Area Network which supports data transfer rates of 10 megabits per second.

Filer

Network attached storage for file systems of NetApp.

FlexFrame® (FF)

The name FlexFrame® is a generic term for both „FlexFrame

® for SAP

®“ and „Flex-

Frame® Orchestrator“.

FlexFrame® Orchestrator (FFO)

This is the advancement of the Fujitsu solution FlexFrame for SAP and means a new

approach to offer enhanced functionality and features step by step and become more

and more independent from certain hardware and software components.

FlexFrame® for SAP

® (FF4S)

FlexFrame® for SAP

® is a Fujitsu solution and means a radically new architecture for

SAP environments. It exploits the latest business-critical computing technology to de-

Glossary


liver major cost savings for SAP customers. FlexFrame for SAP is a joint project in

which the main partners are SAP, Network Appliance, Intel and Fujitsu.

FlexFrame internal LAN Switch

Network switches which are integral part of the FlexFrame hardware configuration

and which are automatically configured by the FlexFrame software.

Gigabit Ethernet

A Local Area Network which supports data transfer rates of 1 gigabit (1,000 mega-

bits) per second.

Host name

The name of a node (assigned to an interface) that is resolved to a unique IP ad-

dress. One node can have multiple host names (cf. node name).

In SAP environments host names are currently limited to 13 alphanumeric characters

including the hyphen (“ - “). The first character must be a letter. In the SAP environ-

ment host names are case-sensitive.

Internet Protocol Address

A unique number used by computers to refer to each other when sending information

through networks using the Internet Protocol.

Local Area Network (LAN)

A computer network that spans a relatively small area. Most LANs are confined to a

single building or group of buildings. However, one LAN can be connected to other

LANs over any distance via telephone lines and radio waves. A system of LANs con-

nected in this way is called a Wide Area Network (WAN).

Local host name

The name of the node (physical computer); it can be displayed and set using the command /bin/hostname.

Media Access Control address

An identifier for network devices, usually unique. The MAC address is stored physi-

cally on the device.

NAS system

Network Attached Storage of any vendor (in our context: NetApp Filer).

Network Attached Storage (NAS)

A data storage device that is connected via a network to one or multiple computers.

Network Interface Card (NIC)

A hardware device that allows computer communication via networks.

Node

A physical computer system controlled by an OS.

Glossary


Node name

The name of a physical node as returned by the command uname -n. Each node

name within a FlexFrame environment must be unique.

Open Network Technology for Appliance Products (ONTAP)

The operating system of Network Appliance Filers.

Physical host

Name of a physical computer system (node).

Preboot Execution Environment (PXE)

An environment that allows a computer to boot from a network resource without hav-

ing a local operating system installed.

Server

A physical host (hardware), same as node.

Service

A software program providing functions to clients.

Storage LAN

A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS

systems.

Trivial File Transfer Protocol (TFTP)

A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram

Protocol (UDP) and provides no security features. It is often used by servers to boot

diskless workstations, X-terminals, and routers.

TFTP server

A simple FTP implementation.

Virtual host

The name of the virtual host on which an application runs; it is assigned to a physical

node when an application is started.

Virtual Local Area Network (VLAN)

A VLAN is a logically segmented network mapped over physical hardware according

to the IEEE 802.1q standard.


8 Index

C

concept and design 5

F

FlexFrame network versions 19

enterprise version 23

medium version 19

small version 19

J

Jumbo Frames 7

L

link aggregation 5, 10, 11, 12, 16

Linux

bonding interface 25

VLAN interface 26

N

NetApp Filer configuration 26

network speed 7

network switch configuration 27

node configuration 25

R

related documents 3

V

virtual LAN 13