OFA-IWG Interop EventApril 2007

Rupert DanceLamprey Networks

Sonoma Workshop Presentation

2

Agenda

Event Overview Program Details Vendor Participation Device Categories Devices Tested Test Environment Mandatory versus Beta testing Cluster Topology Problems Noted Summary

3

Event Overview

IBTA IL Compliance Testing 4/11/07 4/13/078 vendors, 13 devices and 35 cables testedThis was the 11th IBTA Plugfest

OFA IWG Interop Event 4/16/07 4/20/076 vendors, 22 devices and 31 nodes testedThis is the second official OFA-IWG Interop event

Hosted by University of NH Interoperability Lab (UNH-IOL)

Overall results very successful Next event scheduled for October 2007

4

Program Details

IBTA integrators List This list includes all products that have successfully completed

the rigorous testing procedures at IBTA sponsored Plugfests. Current List of devices is available at

http://www.infinibandta.org/itinfo/IL Updated list will be available in May 2007

OFA-UNH-IOL Logo Program Enable vendors to claim and market interoperability by using the

Logo on their products or in their product marketing collateral. Details available at

http://www.iol.unh.edu/services/testing/ofa/logo.pdf List of Logo Grants will be available in May 2007

5

Vendor Participation

InfiniBandCisco, Data Direct, Flextronics, Mellanox,

QLogic, VoltaireiWARP

Chelsio and NetEffectTest Equipment Vendors

Anne, Finisar and Tektronix

6

Device Types and Categories

InfiniBand9 HCAs running OFED 1.2 RC18 IB Switches1 iSER Target1 SRP Target5 SMs

Host Based SMs: OpenSM, Voltaire Switch Based SMs: Cisco, QLogic, Voltaire

iWARPTwo R-NICs

7

InfiniBand Devices Tested Cisco

SFS 7000D - 24-port DDR Switch SFS 7008 - 96-port SDR Switch

Data Direct S2A 9550 - IB SRP connected storage system

Flextronics F-X430066 - 8-Port SDR 4-X InfiniBand switch F-X430044 - 24 Modular DDR 4-X InfiniBand switch F-X430081 - 144 Modular DDR 4-X InfiniBand switch

Mellanox MHEA28-1TC - Dual-port, 10Gb/s w/ media adapter support, PCIe x8, 128MB memory MHEA28-XT - Dual-port, 10Gb/s w/ media adapter support, PCIe x8, mem-free MHGA28-1TC - Dual-port, 20Gb/s w/ media adapter support, PCIe x8, 128MB memory MHGA28-XTC - Dual-port, 20Gb/s w/ media adapter support, PCIe x8, mem-free MHES14-XT - Single-port, 10Gb/s w/ media adapter support, PCIe x4, mem-free MHES18-XTC - Single-port, 10Gb/s w/ media adapter support, PCIe x8, mem-free MHGS18-XTC - Single-port, 20Gb/s w/ media adapter support, PCIe x8, mem-free

QLogic QHT-7140 - InfiniPath QHT 7000 Series QLE-7140 - InfiniPath QLE 7000 Series 9024-CU24-ST2-DDR - SilverStorm 24 port 1U DDR 4x Switch\

Voltaire ISR 2012 - 288 ports infiniband Grid Director ISR9024D-M - 24 ports infiniband Grid Swith SMHost-3.4.8-110 - SMHost (GVFM) - 4.4.8-110 IPStor - iSER target (IPStor) - 5187 GridStack 4.1 - GridStack-4.1.5_6 GridStack Window - Windows GridStack - 2.5.615.741

8

iWARP Devices Tested

ChelsioS310 - 10 Gig Unified Wire Ethernet NIC

(NIC, TOE, iSCSI, iWARP)NetEffect

NE020 - NES Dual-port 10GE RNIC

9

Test Environment

RHEL 5 and SLES 10 SP1 OFED 1.2 RC1 13 Intel provided severs Cables provided by the following vendors

Amphenol Fujikura W.L. Gore Zarlink

Test equipment providers Anue Systems - http://www.anuesystems.com/ Finisar - http://www.finisar.com/home.php Tektronix - http://www.tek.com/

10

Mandatory versus Beta Testing

Test Status for April 2007 Interop Event

Test Procedure InfiniBand Devices

iWARP Devices

Transport Independent

IB Link Initialize Mandatory Not Applicable  

IB Fabric Initialization Mandatory Not Applicable  

IB IPoIB Mandatory Not Applicable  

TI iSER     Beta

IB SRP Mandatory Not Applicable  

TI SDP Mandatory Beta  

IB SM Failover/Handover Beta Not Applicable  

TI MPI - OSU     Beta

TI MPI Intel     Beta

TI uDAPL     Beta

IWARP Connectivity Not Applicable Mandatory  

Fibre Channel Gateway (IB) Beta Not Applicable  

Ethernet Gateway (IB) Beta Not Applicable  

11

Recommended Test Status for October 2007 Interop Event

Test Procedure Review status Recommended Status

IB Link Initialize Well tested Mandatory

IB Fabric Initialization Well tested Mandatory

IB IPoIB Well tested Mandatory

TI iSER Some issues to be resolved Beta

IB SRP Well tested Mandatory

TI SDP Well tested Mandatory

IB SM Failover/Handover Some issues to be resolved Beta

TI MPI - OSU Needs more testing Beta

TI MPI Intel Needs more testing Beta

TI uDAPL Some issues to be resolved Beta

IWARP Connectivity Well tested Mandatory

Fibre Channel Gateway (IB) Needs devices to run tests Beta

Ethernet Gateway (IB) Needs devices to run tests Beta

12

Cluster Topology

2 1 -Flextronics

24 Port DDR4 3

4 3Voltaire 24 Port

DDR2 1

6 5 4 3 2 1 Flextronics

144 Port DDRDevices7 Mellanox HCAs 3 DDR, 4 SDR2 QLogic HCAs SDR1 Cisco 24 Port switch DDR - managed1 Cisco 96 port switch SDR - managed1 Flextronicx 144 port DDR - unmanaged1 Flextronics 24 port switch DDR- un-managed1 Qlogic 24 port switch - managed1 Voltaire 288 port switch DDR - managed1 Voltaire 24 port switch - managed1 Voltaire iSER Target8 Data Direct SDR SRP Targets

Device SM RotationCisco - switch 96Cisco - switch 24Voltaire - switch 24Qlogic - switch 24Voltaire - switch 288Mellanox HCAQLogic HCAVoltaire Linux SM

IPoIB AddressingIP Address 10.0.0.xxNetmask 255.255.0.0

1 2 3 4 DataDirect SRP

Target

Ethernet AddressingIP Address 132.177.126.xxx Netmask 255.255.255.0

Configure IPoIB Addressingifconfig ib0 10.0.0.x netmask 255.255.0.0

Legend: 132.177.126.xx - Ethernet Address10.0.0.xx IPoIB Address

2 1 -Cisco 96 Port

7008 SDR5 6

6 5 4 3 2 1 Voltaire 288 Port

Grid Director ISR 2012 -DDR

3 2 1 -Cisco 24 Port7000P - DDR

6 5

2 1 -Flextronics8 Port SDR3 4

3 4 5Qlogic

9024 - DDR1 2

126.18210.0.0.6QLogic

QLE7140Miranda

126.14310.0.0.1QLogic

QHT7140-1qlogic-htx

Mellanox Cheetah DDR

Atlas

126.20410.0.0.2

Mellanox Cheetah SDR

Telesto

126.19010.0.0.3

Mellanox Tiger SDR

Titan

126.18410.0.0.4

126.20310.0.0.5

Mellanox Mini DDR Calypso

126.20510.0.0.7Mellanox Mini SDRPhoebe

Mellanox Cub DDR

Janus

126.16610.0.0.8

Mellanox Cub -SDRHyperion

126.18710.0.0.9

2 DDR Link

2 DDR Link

1 DDR Link

1

1 11

11

DDR Link

DDR Link

DDR LinkDDR Link

DDR Link

DDR Link

DDR Link

SDR LinkSDR LinkSDR Link

SDR Link

DDR Link

XXXX Flextronics-144 Voltaire-2881…….. Qlogic24 - 1 ……. Voltaire24 - 12 Voltair24 - 1 Qlogic24 - 23…….. Flex8 - 3 ………. Flex24 - 34 Flex24 - 4 Flex8 - 45……… Cisco96 - 5 ……… Cisco24 - 56 Cisco24 - 6 Cisco96 - 6

126.14010.0.0.14

Voltaire SMWindows

SDR Link

SDR LinkSDR Link

SDR Link

1 . Voltaire iSER Target

SDR Link

QlogicQLE7140

126.19110.0.0.10

1

24

MellanoxCheetah

SDRWin OFED

126.21210.0.0.88

1

5

no 126net10.0.0.13

Voltaire SMLinux

2

3

126.14810.0.0.11

Voltaire Linux GridStack

w/ Mellanox HCA

1

6

no 126net10.0.0.12

2

5

126.21 126.9

126.22

126.3

13

Problems Noted

Needed more time for Compliance and Interop testing. Suggested format Week one

Monday through Wednesday IBTA and iWARP compliance testing

Thurs-Fri Interop cluster setup InfiniBand Link and Fabric Init

Week two iWARP Connectivity, IPoIB, SDP, SRP, iSER, SM Failover,

Gateway Tests, DAPL and MPI Vendor review and input on the Test Plan is

needed because it may affect their Logo Grant.

14

Problems Noted - Continued

A link problem was discovered during the Link Init Test Tests requiring one to many testing are very time consuming. Need

automated scripts IPoIB SRP SDP iSER

iSER – possible issues with the OFED Stack code uDAPL

Single switch testing ran reasonably well but there were some unpredictable failures.

Multiple switch tests experienced some problems We need 30-50 DDR cables dedicated to the cluster We need to coordinate the Interop Event with the OFED Releases

15

Summary

Best Interop Event to dateConsiderable progress made on the Interop PlanTest Scripts much improvedTest execution went smoothlyUNH-IOL staff performed very well

Mandatory testing was very successful Beta testing

Need device participation in some categories such as FC and Ethernet Gateway devices

Need more SRP targets Need more time to complete all of the testing