sun datacenter infiniband switch 36 administration guide ... · 2 sun datacenter infiniband switch...

Sun Datacenter InfiniBand Switch 36

Administration Guide for Firmware Version 2.0

Part No.: E26430-01November 2011, Revision A

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected byintellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate,broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering,disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to usin writing.If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, thefollowing notice is applicable:U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are"commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specificsupplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms setforth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in anyinherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or hardware in dangerousapplications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. OracleCorporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon aretrademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks ofSPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd.This software or hardware and documentation may provide access to or information on content, products, and services from third parties. OracleCorporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, andservices. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-partycontent, products, or services.

Copyright © 2011, Oracle et/ou ses affiliés. Tous droits réservés.Ce logiciel et la documentation qui l’accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à desrestrictions d’utilisation et de divulgation. Sauf disposition de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire,diffuser, modifier, breveter, transmettre, distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme et parquelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté àdes fins d’interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu’ellessoient exemptes d’erreurs et vous invite, le cas échéant, à lui en faire part par écrit.Si ce logiciel, ou la documentation qui l’accompagne, est concédé sous licence au Gouvernement des Etats-Unis, ou à toute entité qui délivre la licence dece logiciel ou l’utilise pour le compte du Gouvernement des Etats-Unis, la notice suivante s’applique :U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customersare "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specificsupplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms setforth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.Ce logiciel ou matériel a été développé pour un usage général dans le cadre d’applications de gestion des informations. Ce logiciel ou matériel n’est pasconçu ni n’est destiné à être utilisé dans des applications à risque, notamment dans des applications pouvant causer des dommages corporels. Si vousutilisez ce logiciel ou matériel dans le cadre d’applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, desauvegarde, de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité. Oracle Corporation et ses affiliésdéclinent toute responsabilité quant aux dommages causés par l’utilisation de ce logiciel ou matériel pour ce type d’applications.Oracle et Java sont des marques déposées d’Oracle Corporation et/ou de ses affiliés.Tout autre nom mentionné peut correspondre à des marquesappartenant à d’autres propriétaires qu’Oracle.AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques déposées d’Advanced Micro Devices. Intel et Intel Xeon sont desmarques ou des marques déposées d’Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des marques ou des marquesdéposées de SPARC International, Inc. UNIX est une marque déposée concédée sous licence par X/Open Company, Ltd.Ce logiciel ou matériel et la documentation qui l’accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits etdes services émanant de tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ouservices émanant de tiers. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûtsoccasionnés ou des dommages causés par l’accès à des contenus, produits ou services tiers, ou à leur utilisation.

Contents

Using This Documentation vii

Troubleshooting the Switch 1

Switch Hardware Problems 1

InfiniBand Fabric Problems 3

Identifying LEDs 6

Front Status LEDs 7

Rear Status LEDs 8

▼ Check Chassis Status LEDs 8

▼ Check Network Management Port Status LEDs 9

▼ Check Link Status LEDs 10

▼ Check Power Supply Status LEDs 10

▼ Check Fan Status LEDs 11

Understanding Routing Through the Switch 12

Switch Chip Port to QSFP Connectors and Link LED Routes 12

QSFP Connectors and Link LEDs to Switch Chip Port Routes 13

Signal Route Through the Switch 14

Switch GUIDs Overview 14

Understanding Administrative Commands 17

Configuration Overview 17

Hardware Command Overview 19

InfiniBand Command Overview 19

iii

ILOM Command Overview 20

Administering the Chassis 21

Monitoring the Chassis 21

▼ Display Switch General Health 22

▼ Display Power Supply Status 22

▼ Check Board-Level Voltages 23

▼ Display Internal Temperatures 24

▼ Display Fan Status 24

▼ Display Switch Environmental and Operational Data 25

▼ Display Chassis FRU ID 26

▼ Display Power Supply FRU ID 26

▼ Display Switch Firmware Versions 27

Controlling the Chassis 28

▼ Restart the Management Controller 28

▼ Change the Administrator Password 29

Administering the I4 Switch Chip 31

Monitoring the I4 Switch Chip 31

▼ Display the Switch Chip Port to QSFP Connector Mapping 32

▼ Locate a Switch Chip or Connector From the GUID 33

▼ Display Switch Chip Boot Status 34

▼ Display Link Status 34

▼ Display Switch Chip Port Status 35

Controlling the I4 Switch Chip 36

▼ Disable a Switch Chip Port 36

▼ Enable a Switch Chip Port 37

Administering the Subnet Manager 39

Monitoring the Subnet Manager 39

iv Sun Datacenter InfiniBand Switch 36 Administration Guide for Firmware Version 2.0 • November 2011

▼ Display Subnet Manager Status 40

▼ Display Recent Subnet Manager Activity 40

▼ Display Subnet Manager Priority, Controlled Handover State, Prefix,and Management Key 41

▼ Verify the Subnet Manager Configuration Integrity 41

▼ Display the Subnet Manager Log 42

Controlling the Subnet Manager 43

▼ Set the Subnet Manager Priority 44

▼ Set the Subnet Manager Prefix 45

▼ Enable Subnet Manager Controlled Handover 46

▼ Set the Subnet Manager Management Key 46

▼ Clear the Subnet Manager Management Key 48

▼ Enable the Subnet Manager 48

▼ Disable the Subnet Manager 49

Administering the InfiniBand Fabric 51

Monitoring the InfiniBand Fabric 51

▼ Identify All Switches in the Fabric 52

▼ Identify All HCAs in the Fabric 53

▼ Display the InfiniBand Fabric Topology (Simple) 54

▼ Display the InfiniBand Fabric Topology (Detailed) 54

▼ Display a Route Through the Fabric 55

▼ Display the Link Status of a Node 55

▼ Display Counters for a Node 57

▼ Display Low-Level Detailed Information About a Node 58

▼ Display Low-Level Detailed Information About a Port 58

▼ Display the InfiniBand Fabric Configuration 60

▼ Display the InfiniBand Fabric Partition Configuration 60

Controlling the InfiniBand Fabric 61

Contents v

▼ Perform Comprehensive Diagnostics for the Entire Fabric 62

▼ Determine Changes to the InfiniBand Fabric Topology 62

▼ Find 1x, SDR, or DDR Links in the Fabric 63

▼ Determine Which Links Are Experiencing Significant Errors 64

▼ Clear Data and Error Counters 65

▼ Reset a Port 65

▼ Set Port Speed 66

▼ Disable a Port 67

▼ Enable a Port 68

Configuring the Fabric Director Node List 69

Fabric Director and Fabric Elements 70

fdconfig Command 71

▼ Create a Fabric Configuration 72

▼ Add or Remove an Element From the Fabric Configuration 75

▼ Modify an Element of the Fabric Configuration 77

▼ Exchange Master Fabric Director Status 79

Partitioning the InfiniBand Fabric 80

smpartition Command 81

▼ Remove User Partitions for Firmware Downgrades 82

▼ Use the smnodes Command 83

▼ Determine Partitions and P_Keys 84

▼ Create a User Partition 85

▼ Add or Remove a Port From a Partition 87

▼ Modify a Partition or Port 89

▼ Delete a Partition 91

Index 93

vi Sun Datacenter InfiniBand Switch 36 Administration Guide for Firmware Version 2.0 • November 2011

Using This Documentation

This administration guide provides detailed procedures that describe administrationof the Sun Datacenter InifiniBand Switch 36 from Oracle. This document is writtenfor technicians, system administrators, and users who have advanced experienceadministrating InfiniBand fabric hardware.

■ “Product Notes” on page vii

■ “Related Documentation” on page viii

■ “Feedback” on page viii

■ “Support and Accessibility” on page viii

Product NotesFor late-breaking information an known issues about this product, refer to theproduct notes at:

http://www.oracle.com/pls/topic/lookup?ctx=E26698-01

vii


Related Documentation

FeedbackProvide feedback on this documentation at:

http://www.oracle.com/goto/docfeedback

Support and Accessibility

Documentation Links

All Oracle products http://www.oracle.com/documentation

Sun Datacenter InfiniBandSwitch 36


Oracle Integrated Lights OutManager (ILOM) 3.0


Description Links

Access electronic supportthrough My Oracle Support

http://support.oracle.com

For hearing impaired:http://www.oracle.com/accessibility/support.html

Learn about Oracle’scommitment to accessibility

http://www.oracle.com/us/corporate/accessibility/index.html

viii Sun Datacenter InfiniBand Switch 36 Administration Guide for Firmware Version 2.0 • November 2011

http://www.oracle.com/documentation



http://www.oracle.com/goto/docfeedback

http://support.oracle.com

http://www.oracle.com/accessibility/support.html

http://www.oracle.com/us/corporate/accessibility/index.html

Troubleshooting the Switch

These topics help you resolve some basic problems that might occur with yourswitch.

■ “Switch Hardware Problems” on page 1

■ “InfiniBand Fabric Problems” on page 3

■ “Identifying LEDs” on page 6

■ “Understanding Routing Through the Switch” on page 12

■ “Switch GUIDs Overview” on page 14

Related Information

■ “Understanding Administrative Commands” on page 17

■ “Administering the Chassis” on page 21

■ “Administering the I4 Switch Chip” on page 31

■ “Administering the InfiniBand Fabric” on page 51

■ “Administering the Subnet Manager” on page 39

Switch Hardware ProblemsThis table lists situations that might occur with switch hardware and corrective stepsthat you can take to resolve the problem.

Situation Corrective Steps

The Attention LED on apower supply is lit or thepower supply seemsdysfunctional.

1. Check the power supply status.See “Display Power Supply Status” on page 22.

2. Unplug the respective power cord, wait 15 minutes, then reattach the power cord.3. If the previous steps do not rectify the situation, replace the power supply.

See Switch Service, servicing power supplies.

1

There is no networkcommunication with themanagement controller.

1. Verify that the management controller is powered without faults.See “Check Chassis Status LEDs” on page 8 .

2. Verify that there is a LinkUp condition at the network management port.See “Check Network Management Port Status LEDs” on page 9.

3. Verfiy that the DHCP server is providing the IP address you are using to access themanagement controller.See DHCP server documentation.

4. If you are able to access the management controller through the USB connector,restart the management controller.See Switch Installation, accessing the management controller and “Restart theManagement Controller” on page 28.

5. If you are unable to access the management controller through the USB connector,power cycle the switch.See Switch Service, powering off power supplies and Switch Service, powering onpower supplies.

6. If the previous steps do not rectify the situation, replace the switch.See Switch Service, removing the switch from the rack.See Switch Installation, installaing the switch into the rack.

The Attention LED onthe switch chassis is lit orthe managementcontroller seemsdysfunctional.

1. If you are unable to access the management controller, power cycle the switch.See Switch Service, powering off the power supply and Switch Service, powering onthe power supply.

2. If you are able to access the management controller, restart the managementcontroller.See “Restart the Management Controller” on page 28.

3. Check overall switch health.See “Display Switch General Health” on page 22

4. Verify that the switch is within operating temperatures and voltages.See “Display Switch Environmental and Operational Data” on page 25.

5. If the previous steps do not rectify the situation, replace the switch.See Switch Service, removing the switch from the rack.See Switch Installation, installing the switch into the rack.

The Attention LED on afan is lit or the fan seemsdysfunctional.

1. Check the fan speed.See “Display Fan Status” on page 24.

2. If the previous step does not rectify the situation, replace the fan.See Switch Service, servicing fans.

3. If the previous step does not rectify the situation, install the fan into anotheravailable slot.See Switch Service, servicing fans.

4. If no other slots are available, replace the switch.See Switch Service, removing the switch from the rack.See Switch Installation, installing the switch into the rack.


2 Sun Datacenter InfiniBand Switch 36 Administration Guide for Firmware Version 2.0 • November 2011

Related Information





InfiniBand Fabric ProblemsThis table lists situations that might occur with the InfiniBand fabric and correctivesteps that can be taken to resolve the problem.

After installation, nolinks are operational.

1. Verify that there is at least one Subnet Manager active on the InfiniBand fabric.See “Display Subnet Manager Priority, Controlled Handover State, Prefix, andManagement Key” on page 41.

2. If no Subnet Manager is active, start the Subnet Manager within the switch.See Switch Installation, staring the Subnet Manager.

3. If the previous steps do not rectify the situation, restart the Subnet Manager.See “Disable the Subnet Manager” on page 49 and “Enable the Subnet Manager”on page 48.

After installation, not alllinks are operational.

1. Determine which links are nonoperational.See “Display Link Status” on page 34.

2. For links that are “Down”, disable and re-enable the respective ports.See “Disable a Switch Chip Port” on page 36 and “Enable a Switch Chip Port” onpage 37.

3. If the previous steps do not rectify the situation, disable the respective port.See “Disable a Switch Chip Port” on page 36.

There was a poweroutage during afirmware update.

1. If you are able to access the management controller, restart the managementcontroller.See “Restart the Management Controller” on page 28.

2. If you are unable to access the management controller, power cycle the switch.See Switch Service, removing the switch from the rack.See Switch Installation, installing the switch into the rack.

3. Reperform the firmware upgrade.See Switch Remote Administration, upgrading the switch firmware.




Performance of theInfiniBand fabricseems diminished.

1. Determine if there are errors or problems with the InfiniBand fabric.See:“Perform Comprehensive Diagnostics for the Entire Fabric” on page 62“Find 1x, SDR, or DDR Links in the Fabric” on page 63“Determine Which Links Are Experiencing Significant Errors” on page 64

2. Locate the affected nodes by the GUID provided in the output of the ibdiagnetcommand.See “Locate a Switch Chip or Connector From the GUID” on page 33.

3. If the problem is at a cable connection, swap the suspect cable with a known good cableor reconnect the cable to a known good remote port and repeat Step 1.See Switch Service, servicing data cables.

4. If the problem still remains at the cable connection, disable and re-enable the respectiveport and repeat Step 1.See “Disable a Port” on page 67 and “Enable a Port” on page 68.

Temporary solution:• If the problem still remains, disable the affected port.

See “Disable a Port” on page 67.Permanent solution:• If the problem still remains, replace the affected component or the switch.

See Switch Service, servicing data cables.See remote port’s documentation for replacement procedures.See Switch Service, removing the switch from the rack.See Switch Installation, installing the switch into the rack.

An InfiniBand LinkLED is blinking.

1. Disconnect and properly reconnect both ends of the respective InfiniBand cable.See Switch Service, servicing the data cables.

2. If the LED is still blinking, determine the significance of the errors through use of theibdiagnet command.See “Determine Which Links Are Experiencing Significant Errors” on page 64.

3. Determine which connectors map to the affected link by deconstructing the node’sGUID and port.See “Locate a Switch Chip or Connector From the GUID” on page 33.

4. If some of the links are running at 1x or SDR, use that situation elsewhere in this tableto rectify the problem.

5. Disable and re-enable the respective ports.See “Disable a Port” on page 67 and “Enable a Port” on page 68.

6. If the errors are still significant, swap the cable with a known good one or reconnect thecable to a known good remote port, and repeat from Step 2.

7. Depending upon what does or does not rectify the problem, replace that component.See Switch Service, servicing the data cables.See remote port’s documentation for replacement procedures.


Some InfiniBandlinks are running at1x or SDR.

For a temporary solution:1. Identify the suspect links using the ibdiagnet command.

See “Find 1x, SDR, or DDR Links in the Fabric” on page 63. Look for text like this:-W- link with SPD=2.5 found at direct path "1,19"

From: a Switch PortGUID=0x00066a00d80001dd Port=19

To: a Switch PortGUID=0x00066a00d80001dd Port=24

2. Determine which connectors map to the affected link by deconstructing the node’sGUID and port.See “Locate a Switch Chip or Connector From the GUID” on page 33.

3. Verify the cable connection at both ends.See Switch Service, servicing the data cables.

4. Disable and re-enable the respective ports.See “Disable a Port” on page 67 and “Enable a Port” on page 68.

5. If the previous steps do not rectify the problem, disable the port.See “Disable a Port” on page 67.

For a permanent solution:1. Perform the steps for a temporary solution, Steps 1 to Step 4.2. Swap the cable with a known good cable or reconnect the cable to a known good

remote port, and repeat from Step 1.3. Depending upon what does or does not rectify the problem, replace that component or

the switch.See Switch Service, servicing the data cables.See the remote port’s documentation for replacement procedures.See Switch Service, removin gthe switch from the rack.See Switch Installation, installing the switch into the rack.

There are errors onsome InfiniBandlinks.

1. Clear the error counters.See “Clear Data and Error Counters” on page 65.

2. Start a fabric stress test.3. Identify the suspect links using the ibdiagnet command.

See “Determine Which Links Are Experiencing Significant Errors” on page 64. Look fortext like this:-W- lid=0x0006 guid=0x0021283a8816c0a0 dev=48438 Port=34

Performance Monitor counter : Value

link_recovery_error_counter : 0x1

symbol_error_counter : 0x25 (Increase by 3 during ibdiagnet)

4. For links that are experiencing recovery errors or substantial symbol errors, refer toother parts of this table to help identify the cause and rectify the problem.



Related Information





Identifying LEDsStatus LEDs are used on many components of the switch chassis as a means ofindicating the component’s state.

■ “Front Status LEDs” on page 7

■ “Rear Status LEDs” on page 8

■ “Check Chassis Status LEDs” on page 8

■ “Check Network Management Port Status LEDs” on page 9

■ “Check Link Status LEDs” on page 10

■ “Check Power Supply Status LEDs” on page 10

■ “Check Fan Status LEDs” on page 11

Related Information





Output ofInfiniBandcommands providesonly GUID and port,not switch chip orQSFP connectors.

1. You can find the location of a node in the switch by deconstructing the node’s GUIDand port.See “Locate a Switch Chip or Connector From the GUID” on page 33.

2. Use the dcsport command to provide port-to-connector and connector-to-portmapping.See “Display the Switch Chip Port to QSFP Connector Mapping” on page 32.



Front Status LEDs

FIGURE: Front Status LEDs

Related Information

■ “Check Power Supply Status LEDs” on page 10

■ “Check Fan Status LEDs” on page 11

Figure Legend

1 Power supply AC LED

2 Power supply Attention LED

3 Power supply OK LED

4 Fan Attention LED


Rear Status LEDs

FIGURE: Rear Status LEDs

Related Information

■ “Check Chassis Status LEDs” on page 8

■ “Check Network Management Port Status LEDs” on page 9

■ “Check Link Status LEDs” on page 10

▼ Check Chassis Status LEDsThe chassis status LEDs are located on the left side of the rear panel. See “Rear StatusLEDs” on page 8.

1. Visually inspect the chassis status LEDs.

2. Compare what you see to this table.

Figure Legend

1 NET status LEDs

2 Link status LEDs

3 Chassis status LEDs


Related Information■ “Display Switch Environmental and Operational Data” on page 25

▼ Check Network Management Port Status LEDsThe network management port status LEDs are located on the network managementconnector of the rear panel. See “Rear Status LEDs” on page 8.

1. Visually inspect the NET status LEDs.

2. Compare what you see to this table.

▼ Check Link Status LEDsThe link status LEDs are located at the data cable connectors of the rear panel. See“Rear Status LEDs” on page 8.

Glyph Location Name Color State and Meaning

Top Locator White On – No function.Off – Disabled.Flashing – The switch is identifying itself.

Middle Attention Amber On – Normal fault detected.Off – No faults detected.Flashing – No function.

Bottom OK Green On – Switch is functional without fault.Off – Switch is off or initializing.Flashing – No function.

Name Location Color State and Meaning

Link speed Left Amber or green Amber on – 100BASE-T link.Green on – 1000BASE-T link.Off – No link or link down.Flashing – No function.

Activity Right Green On – No function.Off – No activity.Flashing – Packet activity.


1. Visually inspect the link status LEDs.

2. Compare what you see for a particular link to this table.

Related Information■ “Display Link Status” on page 34

■ “Display the Link Status of a Node” on page 55

■ “Display Counters for a Node” on page 57

▼ Check Power Supply Status LEDsThe power supply status LEDs are located on the power supply at the front of thechassis. See “Front Status LEDs” on page 7.

1. Visually inspect the power supply’s status LEDs.

2. Compare what you see on the power supply to this table.

Caution – If a power supply has shut down because of a thermal or overcurrentcondition, signified by the amber Attention LED lighting, remove the respectivepower cord from the chassis. Allow the power supply to completely cool for at least

Name Color State and Meaning

Link Green On – Link established.Off – No link or link down.Flashing – Symbol errors.

Glyph Location Name Color State and Meaning

Top OK Green On – 12 VDC is supplied.Off – No DC voltage is present.Flashing – No function.

Middle Attention Amber On – Fault detected, 12 VDC shut down.Off – No faults detected.Flashing – No function.

Bottom AC Green On – AC power present and good.Off – AC power not present.Flashing – No function.


15 minutes. A shorter cooling time might cause damage to the power supply whenthe power cord is reattached. If the Attention LED lights amber upon reattaching thepower cord, replace the power supply.

Related Information■ “Display Power Supply Status” on page 22

■ “Check Board-Level Voltages” on page 23

▼ Check Fan Status LEDsThe fan status LEDs are located in the lower right corner of the fans at the front ofthe switch chassis. See “Front Status LEDs” on page 7.

1. Visually inspect the fan status LEDs.

2. If the LED is lit, there is a problem with that fan.

Related Information■ “Display Fan Status” on page 24

Understanding Routing Through theSwitchThe tables in these topics describe the routing through the switch. The first tablemaps the switch chip port to a QSFP connector. The second table provides a reversemap. When command output provides a switch chip port, you can use these tables todetermine the route that link is following.

■ “Switch Chip Port to QSFP Connectors and Link LED Routes” on page 12

■ “QSFP Connectors and Link LEDs to Switch Chip Port Routes” on page 13

■ “Signal Route Through the Switch” on page 14

Related Information






Switch Chip Port to QSFP Connectors and LinkLED Routes

Related Information

■ “Display the Switch Chip Port to QSFP Connector Mapping” on page 32

■ “Display Link Status” on page 34

■ “Display Switch Chip Port Status” on page 35



QSFP Connectors and Link LEDs to Switch ChipPort Routes

Port Connector Port Connector Port Connector Port Connector

1 17A 10 13B 19 0B 28 4A

2 17B 11 12A 20 0A 29 5B

3 16A 12 12B 21 1B 30 5A

4 16B 13 9B 22 1A 31 8A

5 15A 14 9A 23 2B 32 8B

6 15B 15 10B 24 2A 33 7A

7 14A 16 10A 25 3B 34 7B

8 14B 17 11B 26 3A 35 6A

9 13A 18 11A 27 4B 36 6B

ConnectorGroup

ConnectorA

ConnectorB

ConnectorGroup

ConnectorA

ConnectorB

0 20 19 9 14 13

1 22 21 10 16 15

2 24 23 11 18 17


Related Information






Signal Route Through the SwitchBy combining the information from the tables in “Understanding Routing Throughthe Switch” on page 12, it is possible to determine a route through the switch. Thistopic describes a sample situation that might occur.

1. A route is initiated at connector 2A. The LED blinks.

2. Using “QSFP Connectors and Link LEDs to Switch Chip Port Routes” on page 13,it is determined that the link routes to the switch chip through port 24.

3. The Subnet Manager instructs the switch chip to use port 4 to forward the link.

4. Using “Switch Chip Port to QSFP Connectors and Link LED Routes” on page 12, itis determined that the link routes to connector 16B. The LED blinks.

Related Information




■ “Display a Route Through the Fabric” on page 55

3 26 25 12 11 12

4 28 27 13 9 10

5 30 29 14 7 8

6 35 36 15 5 6

7 33 34 16 3 4

8 31 32 17 1 2

ConnectorGroup

ConnectorA

ConnectorB

ConnectorGroup

ConnectorA

ConnectorB


Switch GUIDs OverviewGlobal unique identifiers (GUIDs) are unique 64-bit strings that identify nodes suchas switches and channel adapters. For the Sun Datacenter InfiniBand switches andthis switch, GUIDs are modified to identify the node’s role and location. This tabledescribes the GUID’s structure.

The five fields of the GUID are described as follows:

■ MAC – Bits 63 through 16 are the Machine Allocation Code (MAC) address. Astandard for network components, the MAC address is typically provided bymanufacturers in a 6-byte, colon delimited string. For example, 00:11:22:33:44:55.

■ Type – Bits 15 through 12 is the type of board on which the node resides.

■ Pos – Bits 11 through 8 identify the position of the board within the switch.

■ Device – Bits 7 through 4 identify which device on the board has that node.

■ Num – Bits 3 through 0 are numbers reserved for the programs which modify theGUID. In most occurrences, the value is 0x2.

This table provides values for Type, Pos, Device.

For example, given this output from the ibswitches command:

The GUID is 0x0021283a831da0a2 or 0021283A831D A 0 A 2. Using the informationprovided in this topic:

63 16 15 12 11 8 7 4 3 0

MAC (48 bits) Type Pos Device Num

Board Type Position Device

Fabric card 0xF 0x0 (Fabric card 0)–0x8 (Fabric card 8) 0xA (I4 chip 0)–0xB (I4 chip 1)

Line card 0x1 0x0 (Line card 0)–0x8 (Line card 8) 0xA (I4 chip 0)–0xD (I4 chip 3)

Gateway 0xC 0x0 0xA (I4 chip 0), 0x0 (BX chip 0), 0x4(BX chip 1)

36-Port 0xA 0x0 0xA (I4 chip 0)

72-Port 0xB 0x0 0xA (I4 chip 0)–0xF (I4 chip 5)

[16] -> switch port {0x0021283a831da0a2}[21] lid 214-214 "Sun DCS 36 QDR FCswitch 2.0"


■ The MAC address is 0x0021283A831D or 00:21:28:3A:83:1D.

■ The type is 0xA, or a 36-port board.

■ The position is 0x0.

■ The device is 0xA, or a solitary I4 chip.

■ The number is 2.

Related Information

■ Switch Reference, ibnetdiscover command

■ Switch Reference, ibnodes command

■ Switch Reference, ibswitches command

■ Switch Reference, ibhosts command

■ “Identify All Switches in the Fabric” on page 52






Understanding AdministrativeCommands

These topics provide an overview of administrative tasks and the command sets toperform those tasks. Administering the switch requires accessing the managementcontroller.

■ “Configuration Overview” on page 17

■ “Hardware Command Overview” on page 19

■ “InfiniBand Command Overview” on page 19

■ “ILOM Command Overview” on page 20

Related Information

■ “Troubleshooting the Switch” on page 1





Configuration OverviewAfter you have installed the switch, you can configure its functionality. This tablelists functions of the switch to configure, in the order of dependency. If you do notwant to configure a particular functionality, skip that step.

17

Related Information




Hardware Command OverviewThe management controller uses a simplified Linux OS and file system. From the #prompt on the management controller, you can type hardware commands to performsome administrative and management tasks. Alternatively, you can use the OracleILOM restricted Linux shells, which are the prefferred method of issuing hardwarecommands. See Switch Remote Administration, Linux restricted shells.

Step Functionality Description Links

1. Configure theSubnet Manager.

Configure the priority. “Set the Subnet Manager Priority” on page 44

Configure the prefix. “Set the Subnet Manager Prefix” on page 45

Enable controlled handover. “Enable Subnet Manager Controlled Handover” onpage 46

Configure the M_Key. “Set the Subnet Manager Management Key” onpage 46

2. Configure theInfiniBand fabric.

Create a fabric configuration. “Create a Fabric Configuration” on page 72

3. Partition theInfiniBand fabric.

Collect GUIDs. “Identify All Switches in the Fabric” on page 52“Identify All HCAs in the Fabric” on page 53

Create the smnodes file. “Use the smnodes Command” on page 83

Create the partitioninformation file.

“Determine Partitions and P_Keys” on page 84

Configure partitions andP_Keys.

“Create a User Partition” on page 85

Add GUIDs to the partitions. “Add or Remove a Port From a Partition” onpage 87

4. Save theconfiguration.

Back up the configuration forrestoration at a later time,should it become necessary.

Switch Remote Administration, backing up theconfiguration.


Hardware commands are user-friendly and can perform some testing upon theswitch chip, enabling greater control of the switch and its operation.

After you log in to the root account, the shell prompt (#) appears, and you can entershell commands. Enter the hardware commands in this format:

# command [arguments][arguments]...

Related Information

■ Switch Reference, understanding hardware commands




InfiniBand Command OverviewThe InfiniBand commands are a means of monitoring and controlling aspects of theInfiniBand fabric. These commands are also installed on and run from themanagement controller, which is also the host of the Subnet Manager. Use of thesecommands requires thorough knowledge of InfiniBand architecture and technology.

After you log in to the root account, the shell prompt (#) appears, and you can entershell commands. Alternatively, you can use the Oracle ILOM restricted Linux shells,which are the prefferred method of issuing InfiniBand commands. See Switch RemoteAdministration, Linux restricted shells.

Enter the InfiniBand commands in this format:

# command [option][option] ...

Related Information

■ Switch Reference, understanding InfiniBand commands




Understanding Administrative Commands 19

ILOM Command OverviewThe Oracle ILOM CLI, web, SNMP, and IPMI interfaces enable additionaladministration features and capabilities. The Oracle ILOM CLI, SNMP, and IPMIinterfaces use remote clients for command-line administration of many Oracle ILOMtargets. Alternatively, the Oracle ILOM web interface permits point-and-clickadministration of the Oracle ILOM components and services.

After you log in to the ilom-admin account, the Oracle ILOM prompt (->) appears,and you can enter Oracle ILOM commands. Enter the Oracle ILOM commands inthis format:

-> command [option][target][property=value]...

Information about ILOM support of the switch is available in the Oracle IntegratedLights Out Manager (ILOM) 3.0 Supplement for the Sun Datacenter InfiniBand Switch 36,available online at:


Overall information about Oracle ILOM 3.0 is available online at:


Related Information

■ Switch Remote Administration, understanding Oracle ILOM commands







Administering the Chassis

These topics describe the administration of the switch.

■ “Monitoring the Chassis” on page 21

■ “Controlling the Chassis” on page 28

Related Information






Monitoring the ChassisThese topics enable you to display and check the operation and status of the switch.

■ “Display Switch General Health” on page 22

■ “Display Power Supply Status” on page 22


■ “Display Internal Temperatures” on page 24

■ “Display Fan Status” on page 24

■ “Display Switch Environmental and Operational Data” on page 25

■ “Display Chassis FRU ID” on page 26

■ “Display Power Supply FRU ID” on page 26

■ “Display Switch Firmware Versions” on page 27

Related Information

■ “Monitoring the I4 Switch Chip” on page 31

21

■ “Monitoring the InfiniBand Fabric” on page 51

■ “Monitoring the Subnet Manager” on page 39


▼ Display Switch General HealthAn easy way to perform a quick check of the switch’s operation is with theshowunhealthy command.

● On the management controller, type.

Related Information■ Switch Reference, showunhealthy command


▼ Display Power Supply StatusThe checkpower command performs a simple pass-fail test on the power supplies.

1. On the management controller, type.

2. If you see the words Alert or Not Present in the command output, performthese steps for the affected power supply.

a. Remove the power cord and affected power supply from the switch chassis.

See Switch Service, powering off the power supply and Switch Service, removingthe power supply.

b. Wait one minute.

c. Reinstall the power supply.

See Switch Service, installing the power supply.

# showunhealthyOK - No unhealthy sensors#

# checkpowerPSU 0 present OKPSU 1 present Alert#


d. Reattach the power cord.

See Switch Service, powering on the power supply.

e. If the power supply’s Attention LED lights or the checkpower command stillreports Alert or Not Present for the power supply, replace the powersupply.

See Switch Service, servicing the power supplies.

Related Information■ Switch Reference, checkpower command



▼ Check Board-Level VoltagesWhen you want to know if the voltages used by the various components within theswitch are within nominal values, use the checkvoltages command. If a voltagedeviates more than 10% of its nominal value, there is a problem. This check alsoreports if a battery has failed.


Related Information■ Switch Reference, checkvoltages command

■ “Display Power Supply Status” on page 22


# checkvoltagesVoltage ECB OKMeasured 3.3V Main = 3.27 VMeasured 3.3V Standby = 3.35 VMeasured 12V = 11.97 VMeasured 5V = 5.02 VMeasured VBAT = 3.24 VMeasured 2.5V = 2.52 VMeasured 1.8V = 1.78 VMeasured I4 1.2V = 1.22 VAll voltages OK#


▼ Display Internal TemperaturesIf you are concerned with the internal temperature of the switch, you can use theshowtemps command to display nominal and measured temperatures. If there is agreat deviation, there might not be enough cooling.


Related Information■ Switch Reference, showtemps command

■ “Display Fan Status” on page 24


▼ Display Fan StatusThe output of the getfanspeed command can help you determine fan speedinconsistencies, which might indicate a future failure.


Related Information■ Switch Reference, getfanspeed command


# showtempsBack temperature 32Front temperature 32SP temperature 44Switch temperature 45, maxtemperature 46All temperatures OK#

# getfanspeedFan 0 not presentFan 1 running at rpm 11212Fan 2 running at rpm 11313Fan 3 running at rpm 11521Fan 4 not present#


▼ Display Switch Environmental and OperationalDataThe env_test command enables you to perform several investigative tasks, helpingyou determine the internal environment and operational status of the switch.


# env_testEnvironment test started:Starting Environment Daemon test:Environment daemon runningEnvironment Daemon test returned OKStarting Voltage test:Voltage ECB OKMeasured 3.3V Main = 3.27 VMeasured 3.3V Standby = 3.35 VMeasured 12V = 11.97 VMeasured 5V = 5.02 VMeasured VBAT = 3.24 VMeasured 2.5V = 2.52 VMeasured 1.8V = 1.78 VMeasured I4 1.2V = 1.22 VVoltage test returned OKStarting PSU test:PSU 0 present OKPSU 1 present OKPSU test returned OKStarting Temperature test:Back temperature 32Front temperature 32SP temperature 44Switch temperature 44, maxtemperature 46Temperature test returned OKStarting FAN test:Fan 0 not presentFan 1 running at rpm 12075Fan 2 running at rpm 11960Fan 3 running at rpm 12075Fan 4 not presentFAN test returned OKStarting Connector test:Connector test returned OKStarting Onboard ibdevice test:Switch OKAll Internal ibdevices OK


Related Information■ Switch Reference, env_test command

▼ Display Chassis FRU IDThe showfruinfo command displays switch chassis FRU ID information.


Related Information■ Switch Reference, showfruinfo command

■ “Display Power Supply FRU ID” on page 26

▼ Display Power Supply FRU IDThe showpsufru command displays power supply FRU ID information.

Onboard ibdevice test returned OKEnvironment test PASSED#

# showfruinfo UNIX_Timestamp32 : Tue Mar 23 17:02:15 2010 Sun_Fru_Description : ASSY,NM2-36P Vendor_ID_Code : 11 E1 Vendor_ID_Code_Source : 01 Vendor_Name_And_Site_Location : 4577 CELESTICA CORP. SAN JOSE CA US Sun_Part_Number : 5413495 Sun_Serial_Number : 0110SJC-1010NG0045 Serial_Number_Format : 4V3F1-2Y2W2X4S Initial_HW_Dash_Level : 01 Initial_HW_Rev_Level : 50 Sun_Fru_Shortname : NM2, 36 ports Sun_Hazard_Class_Code : YL Sun_SpecPartNo : 885-1507-01#



where slot is the power supply slot (0 or 1). For example:

Related Information■ Switch Reference, showpsufru command

■ “Display Chassis FRU ID” on page 26

▼ Display Switch Firmware VersionsIn time, newer or updated switch firmware might become available. The versioncommand displays the versions of the firmware within the switch chassis.


# showpsufru slot

# showpsufru 0 Sun_SpecPartNo : 885-1390-01 UNIX_Timestamp32 : Sun Jan 3 15:35:39 2010 Sun_Part_Number : 3002234 Sun_Serial_No : BF15WA Vendor_ID_Code : 03ad Initial_HW_Dash_Level : 02 PSU_Voltage 1 : 0x04b0 (1200) PSU_Current 1 : 0x186a (6250) PSU_Voltage 2 : 0x014a (330) PSU_Current 2 : 0x012c (300) Sun_Hazard_Class_Code : Y IPMI_Board_Manufacturer : Delta Energy Systems IPMI_Board_Product_Name : A236 IPMI_Board_Serial_Number: 1357ZHO-0952BF15WA IPMI_Board_Part_Number : 300-2234-01#

# versionSUN DCS 36p version: 2.0.5-1Build time: Nov 25 2011 12:20:22SP board info:Manufacturing Date: 2009.02.19Serial Number: "NCD2S0133"Hardware Revision: 0x0100Firmware Revision: 0x0102


Related Information■ Switch Reference, version command

■ Switch Remote Administration, version command

Controlling the ChassisThese topics describe how you can manage the switch and its components.

Note – To use the commands described in these topics, you must either be the rootuser of the management controller or access them through the /SYS/Fabric_MgmtLinux shell target of the Oracle ILOM CLI.

■ “Restart the Management Controller” on page 28

■ “Change the Administrator Password” on page 29

Related Information

■ “Controlling the I4 Switch Chip” on page 36

■ “Controlling the InfiniBand Fabric” on page 61

■ “Controlling the Subnet Manager” on page 43


▼ Restart the Management ControllerShould the management controller enter an indeterminate state, you can reboot it.The Subnet Manager and other services are also restarted. Additionally, the links tothe I4 switch chip are disabled and then re-enabled when the management controllerreboots.

Note – The reboot command severs any management console link to themanagement controller. You must reaccess the management controller to regainadministrative control.

BIOS version: NOW1R112BIOS date: 04/24/2009#



The management controller power cycles and reinitializes.

Related Information■ “Change the Administrator Password” on page 29

▼ Change the Administrator PasswordThe default password for the root user is changeme. You can use the passwdcommand to make the password unique and more secure.


where user is name of the user who’s password is to be changed. For example:

Related Information■ Switch Installation, accessing the management controller

■ “Restart the Management Controller” on page 28

# rebootBroadcast message from root (pts/0) (Tue Oct 18 23:24:50 2011):The system is going down for reboot NOW!# Connection to 123.45.67.89 closed by remote host.Connection to 123.45.67.89 closed.#

# passwd user

# passwd rootChanging password for user root.New UNIX password: new_passwordRetype new UNIX password: new_passwordpasswd: all authentication tokens updated successfully.#


Administering the I4 Switch Chip

These topics describe the administration of the I4 switch chip.



Related Information






Monitoring the I4 Switch ChipThese topics enable you to display and check the operation and status of the I4switch chip.

Note – To use all the commands described in these topics, you must either be theroot user of the management controller or access them through the/SYS/Fabric_Mgmt Linux shell target of the Oracle ILOM CLI.


■ “Locate a Switch Chip or Connector From the GUID” on page 33

■ “Display Switch Chip Boot Status” on page 34



31

Related Information





▼ Display the Switch Chip Port to QSFP ConnectorMappingYou can use the dcsport command to display the mapping of a switch chip port toits respective QSFP connector and to display the mapping of a QSFP connector to itsswitch chip port.


where port is the number of the port (1–36).

Alternatively, type.

where connector is the number of the QSFP connector (0A–17B).

For example:

Note – For a complete mapping of all ports to all connectors, use the dcsport-printconnnectors command.

Related Information■ Switch Reference, dcsport command


# dcsport -port port

# dcsport -connector connector

# dcsport -port 7DCS-36P Switch port 7 maps to connector 14A# dcsport -connector 14AConnector 14A maps to Switch port 7#


▼ Locate a Switch Chip or Connector From theGUIDThe output of some InfiniBand commands identify a node by its GUID. You candetermine the switch chip or InfiniBand cable connection associated with that GUIDusing this procedure.

1. In the output of the InfiniBand command, identify the GUID and port inquestion.

For example, given this output from the ibdiagnet command:

In this example, the GUID is 0x03ba01234567a0a0 and the port is 21.

2. Truncate all digits of the GUID except the four digits on the right.

For example, GUID 0x03ba01234567a0a0 is truncated to a0a0.

The a0 means that the node is inside of a Sun Datacenter InfiniBand Switch 36.The a means that the node is located in I4 switch chip A, the only switch chip. Theport is 21.

3. Use the ibswitches command to verify the information.

4. Use the tables in “Understanding Routing Through the Switch” on page 12 tocross-reference the routing between ports and to connections.

For example, using “Switch Chip Port to QSFP Connectors and Link LED Routes”on page 12, I4 switch chip, port 21 routes to connector 1B.

5. Alternatively, you can use the dcsport command to display routing betweenports and connections.

See “Display the Switch Chip Port to QSFP Connector Mapping” on page 32.

Related Information■ “Switch GUIDs Overview” on page 14

■ Switch Reference, dcsport command

-W- lid=0x0001 guid=0x03ba01234567a0a0 dev=48438 Port=21

# ibswitchesSwitch : 0x03ba01234567a0a0 ports 36 "SUN DCS 36P QDR mnm-36p-2 10.172.144.66"enhanced port 0 lid 1 lmc 0Switch : 0x002128312345c0a0 ports 36 "SUN IB QDR GW switch mnm-gw-110.172.144.69" enhanced port 0 lid 15 lmc 0#


▼ Display Switch Chip Boot StatusIf you believe there is a problem with the switch chip, the checkboot commandprovides a quick pass-fail check of the switch chip. Should the switch chip fail thetest, you can use the ibdevreset command in an attempt to reboot the switch chip.


Related Information■ Switch Reference, checkboot command


■ Switch Reference, ibdevreset command

▼ Display Link StatusIn some situations, you might need to know the status of each route through theswitch. Additionally, the listlinkup command displays where InfiniBand cablesare connected to the switch.


Related Information■ Switch Reference, listlinkup command

# checkbootSwitch OK#

# listlinkupConnector 0A Present <-> Switch Port 20 up (Enabled)Connector 1A Present <-> Switch Port 22 up (Enabled)Connector 2A Present <-> Switch Port 24 up (Enabled)...Connector 17A Not presentConnector 0B Present <-> Switch Port 19 up (Enabled)Connector 1B Present <-> Switch Port 21 up (Enabled)...Connector 17B Not present#



▼ Display Switch Chip Port StatusIf by using other procedures, it seems that a particular port of the switch chip isproblematic, you can use the getportstatus command to provide the port state,width, and speed.


where:

■ connector is the name of the connector (0A – 17A or 0B – 17B).

■ port is the number of the port (1–36).

For example:

Note – The parameters with the string Active indicate the current conditions.

Related Information■ Switch Reference, getportstatus command


# getportstatus connector | Switch port

# getportstatus Switch 7Port status for connector 14A Switch Port 7Adminstate:......................EnabledLinkWidthEnabled:................1X or 4XLinkWidthSupported:..............1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkState:.......................ActivePhysLinkState:...................LinkUpLinkSpeedActive:.................10.0 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps#


Controlling the I4 Switch ChipThese topics describe how you can manage the I4 switch chip.


■ “Disable a Switch Chip Port” on page 36

■ “Enable a Switch Chip Port” on page 37

Related Information





▼ Disable a Switch Chip PortOn occasion, you might need to turn off a port. For example, a cable might becomedamaged and cause symbol errors that affect the switch chip’s performance. Use thedisableswitchport command to disable a switch chip port and its respectivecable connection.


where:

■ reason is the reason for disabling the port, Blacklist or Partition.

# disableswitchport [--reason=reason] Switch port





For example:

Related Information■ Switch Reference, disableswitchport command

■ “Disable a Port” on page 67


▼ Enable a Switch Chip PortYou can enable a disabled switch chip port with the enableswitchport command.


where:

■ reason is the reason for disabling the port, Blacklist or Partition.


# disableswitchport [--reason=reason] connector

# disableswitchport Switch 14Disable connector 9A Switch port 14MKEY 8 0000000000000000Adminstate:......................DisabledLinkWidthEnabled:................1X or 4XLinkWidthSupported:..............1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkState:.......................DownPhysLinkState:...................DisabledLinkSpeedActive:.................2.5 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps#

# enableswitchport [--reason=reason] Switch port


Note – If the port was disabled with the --reason option, (as seen with thelistlinkup command) it can only be enabled with the same reason option.



For example:

Related Information■ Switch Reference, enableswitchport command

■ “Enable a Port” on page 68


# enableswitchport [--reason=reason] connector

# enableswitchport Switch 14Enable connector 9A Switch port 14MKEY 8 0000000000000000Adminstate:......................EnabledLinkWidthEnabled:................1X or 4XLinkWidthSupported:..............1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkState:.......................DownPhysLinkState:...................PollingLinkSpeedActive:.................2.5 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps#


Administering the Subnet Manager

These topics describe the administration of the Subnet Manager.



Related Information






Monitoring the Subnet ManagerThese topics describe how to monitor the Subnet Manager.


■ “Display Subnet Manager Status” on page 40

■ “Display Recent Subnet Manager Activity” on page 40

■ “Display Subnet Manager Priority, Controlled Handover State, Prefix, andManagement Key” on page 41

■ “Verify the Subnet Manager Configuration Integrity” on page 41

■ “Display the Subnet Manager Log” on page 42

39

Related Information





▼ Display Subnet Manager StatusIf you want to quickly determine your Subnet Manager’s priority and state, thegetmaster command can also provide the LID and GUID of the hosting HCA.


Related Information■ Switch Reference, getmaster command





▼ Display Recent Subnet Manager Activity● On the management controller, type.

# getmasterLocal SM enabled and running20111021 13:46:23 Master SubnetManager on sm lid 1 sm guid 0x3ba01234567a0a0 :SUN DCS 36P QDR o4nm2-36p-2 10.172.144.66#

# getmaster -lLocal SM enabled and runningLast ring buffer history listed:20111018 19:16:12 whereismaster started20111018 19:16:12 Master SubnetManager on sm lid 0 sm guid 0x212856cfe2c0a0 :20111018 19:16:29 Master SubnetManager on sm lid 6 sm guid 0x212856cfe2c0a0 :SUN IB QDR 36p switch mnm34-98#


Related Information■ Switch Reference, getmaster command





▼ Display Subnet Manager Priority, ControlledHandover State, Prefix, and Management KeyIf you want to know your Subnet Manager’s priority, controlled handover state,prefix, and management key, use the list option of the setsmpriority command.


Related Information■ Switch Reference, setsmpriority command






▼ Verify the Subnet Manager ConfigurationIntegrityThe smconfigtest command performs a check of the Subnet Manager’sconfiguration and verifies the integrity.

# setsmpriority listCurrent SM settings:smpriority 8controlled_handover TRUEsubnet_prefix 0xfe80000000000000M_Key None#


Note – For the spine option of the smconfigtest command, the Subnet Manageris expected to have a priority of 5 or 8. For the leaf option, the expected priority is5.

● Verify the Subnet Manager configuration.

where:

■ spine – for spine switches.

■ leaf – for leaf switches.

For example:

Note – Output is only displayed if errors are found.

Related Information■ Switch Reference, smconfigtest command





▼ Display the Subnet Manager Log1. On the management controller, type.

# smconfigtest [spine|leaf]

# smconfigtest leaf#

# showsmlogJun 30 14:44:11 008981 [B6694B90] 0x02 -> osm_ucast_mgr_process: minhop tablesconfigured on all switchesJun 30 14:44:11 011981 [B6694B90] 0x02 -> SUBNET UPJun 30 14:44:11 025981 [B6694B90] 0x02 -> Fabric has 1 switches - topology isnot fat-tree. Falling back to default routingJun 30 14:44:11 025981 [B6694B90] 0x01 -> ucast_mgr_route: ftree: cannot buildlid matrices.


2. Tap the space bar to display the next screen of the log.

3. Press the Q key to quit.

Related Information■ Switch Reference, showsmlog command





Controlling the Subnet ManagerYou can enable the Subnet Manager with the enablesm command. When the SubnetManager starts, it reads the configuration file for configuration information. You candisable the Subnet Manager with the disablesm command.

These topics describe how to control the Subnet Manager:

Note – To use the commands described in these topics, you must either be the rootuser of the management controller or access them through the /SYS/Fabric_MgmtLinux shell targets of the Oracle ILOM CLI.

■ “Set the Subnet Manager Priority” on page 44

■ “Set the Subnet Manager Prefix” on page 45

■ “Enable Subnet Manager Controlled Handover” on page 46

■ “Set the Subnet Manager Management Key” on page 46

■ “Clear the Subnet Manager Management Key” on page 48

■ “Enable the Subnet Manager” on page 48

■ “Disable the Subnet Manager” on page 49

Jun 30 14:44:11 025981 [B6694B90] 0x02 -> osm_ucast_mgr_process: minhop tablesconfigured on all switchesJun 30 14:44:11 027981 [B6694B90] 0x02 -> SUBNET UP...


Related Information





▼ Set the Subnet Manager PriorityBy default, the Subnet Manager within the management controller is set to 0 priority.If there is more than one Subnet Manager in your InfiniBand fabric, you must set thepriority of each Subnet Manager appropriately. The Subnet Manager with the highestpriority is the primary (or Master) Subnet Manager.

1. On the managment controller, disable the Subnet Manager.

See “Disable the Subnet Manager” on page 49.

2. Set the Subnet Manager priority.

where priority is 0 (lowest) to 13 (highest). For example:

3. Enable the Subnet Manager.

See “Enable the Subnet Manager” on page 48.

Related Information■ Switch Reference, setsmpriority command





# setsmpriority priority

# setsmpriority 5Current SM settings:smpriority 5controlled_handover FALSEsubnet_prefix 0xfe80000000000000M_Key None#





▼ Set the Subnet Manager PrefixThe setsubnetprefix command writes a prefix value to the subnet_prefixparameter of the configuration file.



2. Set the Subnet Manager prefix..



Related Information■ Switch Reference, setsubnetprefix command








# setsubnetprefix 0xabbababeCurrent SM settings:smpriority 5controlled_handover FALSEsubnet_prefix 0xabbababeM_Key None#


▼ Enable Subnet Manager Controlled HandoverIf your InfiniBand fabric has two or more Subnet Managers, you can force aconstrained fallback protocol should the master Subnet Manager fail. See SwitchReference, setcontrolledhandover command for more information.



2. Enable controlled handover.



Related Information■ Switch Reference, setcontrolledhandover command








▼ Set the Subnet Manager Management KeyFor added security, a management key, or M_Key is used by the Subnet ManagerAgent to initialize and configure network nodes. See Switch Reference, setsmmkeycommand for more information.

# setcontrolledhandover TRUECurrent SM settings:smpriority 5controlled_handover TRUEsubnet_prefix 0xabbababeM_Key None#


Note – All Subnet Managers in the InfiniBand fabric must be configured with thesame M_Key value.



2. Set the management key.

where m_key is a 12-digit hexadecimal number

For example:



4. Repeat Step 1 to Step 3 for all Subnet Managers of the InfiniBand fabric.

Related Information■ Switch Reference, setsmmkey command








# setsmmkey m_key

# setsmmkey 0xabbababe0001Current SM settings:smpriority 5controlled_handover TRUEsubnet_prefix 0xabbababeM_Key 0xabbababe0001#


▼ Clear the Subnet Manager Management KeyIt might be necessary to remove the restrictions of the Subnet Manager managementkey. This procedure describes how to do so.



2. Clear the management key.



4. Repeat Step 1 to Step 3 for all Subnet Managers requiring the change.

Related Information■ Switch Reference, setsmmkey command








▼ Enable the Subnet ManagerThe Subnet Manager within the management controller is not enabled by default.Use the enablesm command to enable and start the Subnet Manager.

# setsmmkey noneCurrent SM settings:smpriority 5controlled_handover TRUEsubnet_prefix 0xabbababeM_Key None#



Related Information■ Switch Reference, enablesm command







▼ Disable the Subnet ManagerIf your InfiniBand fabric has too many Subnet Managers, you can disable and stopthe Subnet Manager within the management controller by using the disablesmcommand.


Related Information■ Switch Reference, disablesm command







# enablesmStarting IB Subnet Manager. [ OK ]Starting partitiond daemon. [ OK ]#

# disablesmStopping partitiond daemon. [ OK ]Stopping IB Subnet Manager.. [ OK ]#


Administering the InfiniBand Fabric

These topics describe the administration of the InfiniBand fabric.



■ “Configuring the Fabric Director Node List” on page 69

■ “Partitioning the InfiniBand Fabric” on page 80

Related Information






Monitoring the InfiniBand FabricThese topics enable you to display and check the operation and status of theInfiniBand fabric and components.



■ “Identify All HCAs in the Fabric” on page 53

■ “Display the InfiniBand Fabric Topology (Simple)” on page 54

■ “Display the InfiniBand Fabric Topology (Detailed)” on page 54

■ “Display a Route Through the Fabric” on page 55

51



■ “Display Low-Level Detailed Information About a Node” on page 58

■ “Display Low-Level Detailed Information About a Port” on page 58

■ “Display the InfiniBand Fabric Configuration” on page 60

■ “Display the InfiniBand Fabric Partition Configuration” on page 60

Related Information







▼ Identify All Switches in the FabricIf you need to know identity information about the switches in the InfiniBand fabric,you can use the ibswitches command. This command displays the GUID, name,LID, and LMC for each switch. The output of the command is a mapping of GUID toLID for switches in the fabric.

Note – The output for your InfiniBand fabric will differ greatly from the output inthe example.


Related Information■ Switch Reference, ibswitches command



# ibswitchesSwitch : 0x03ba01234567a0a0 ports 36 "SUN DCS 36P QDR mnm-36p-2 10.172.144.66"enhanced port 0 lid 1 lmc 0Switch : 0x002128312345c0a0 ports 36 "SUN IB QDR GW switch mnm-gw-110.172.144.69" enhanced port 0 lid 15 lmc 0#


▼ Identify All HCAs in the FabricSimilar to the ibswitches command, the ibhosts command displays identityinformation about the HCAs in the InfiniBand fabric. The output contains the GUIDand name for each HCA.

Note – Unlike the ibswitches command, the ibhosts command does not displaythe LIDs.


Note – The output in the example is just a portion of the full output and varies foreach InfiniBand topology.

2. (Optional) Save the output of ibhosts command as a text file namedhostGUIDs.txt.

Related Information■ Switch Reference, ibhosts command

■ Switch Reference, ibswitches command


▼ Display the InfiniBand Fabric Topology (Simple)To understand the routing that happens within your InfiniBand fabric, theshowtopology command displays the node-to-node connectivity. The output of thecommand depends on the size of your fabric.

# ibhostsCa : 0x00212800013e9312 ports 2 "mnm56 HCA-1"Ca : 0x00212800013e93f6 ports 2 "mnm57 HCA-1"Ca : 0x002128312345c000 ports 2 "SUN IB QDR GW switch mnm-gw-1 10.172.144.69Bridge 0"Ca : 0x002128312345c040 ports 2 "SUN IB QDR GW switch mnm-gw-1 10.172.144.69Bridge 1"...#



Related Information■ Switch Reference, showtopology command

■ “Determine Changes to the InfiniBand Fabric Topology” on page 62

■ “Perform Comprehensive Diagnostics for the Entire Fabric” on page 62

▼ Display the InfiniBand Fabric Topology(Detailed)To understand the routing that happens within your InfiniBand fabric, theibnetdiscover command displays the node-to-node connectivity. The output ofthe command depends on the size of your fabric.

Note – You must be the root user to run the ibnetdiscover command.

Note – You can use the ibnetdiscover command to determine the LIDs of theHCAs.


Related Information■ Switch Reference, ibnetdiscover command


▼ Display a Route Through the FabricOften you need to know the route between two nodes. The ibtracert commandcan provide that information by displaying the GUIDs, ports, and LIDs of the nodesalong the route.

# showtopology

# ibnetdiscover



where:

■ slid is the LID of the source node.

■ dlid is the LID of the destination node.

For example:

Note – The output for your InfiniBand fabric will differ from that in the example.

For this example:

■ The route starts at switch with GUID 0x0021283a8389a0a0 and is using port0. The switch is LID 15 and in the description, the switch host’s name is SunDCS 36 QDR switch localhost.

■ The route enters at port 23 of the CA with GUID 0x0003ba000100e38a andexits at port 2. The CA is LID 14.

Related Information■ Switch Reference, ibtracert command

▼ Display the Link Status of a NodeIf you want to know the link status of a node in the InfiniBand fabric, theibportstate command can tell you the state, width, and speed of that node.


where:

■ lid is the LID of the node.

# ibtracert slid dlid

# ibtracert 15 14From switch {0x0021283a8389a0a0} portnum 0 lid 15-15 "Sun DCS 36 QDR switchlocalhost"[23] -> ca port {0x0003ba000100e38a}[2] lid 14-14 "mnm-43 HCA-1"To ca {0x0003ba000100e388} portnum 2 lid 14-14 "mnm-43 HCA-1"#

# ibportstate lid port


■ port is the port of the node.

For example:

In the output, the Active parameters are the current state of the port.

Related Information■ Switch Reference, ibportstate command



■ “Find 1x, SDR, or DDR Links in the Fabric” on page 63

■ “Set Port Speed” on page 66

▼ Display Counters for a NodeTo help ascertain the health of a node, the perfquery command displays theperformance, error, and data counters for that node.

# ibportstate 15 28PortInfo:# Port info: Lid 15 port 28LinkState:.......................ActivePhysLinkState:...................LinkUpLinkWidthSupported:..............1X or 4XLinkWidthEnabled:................1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedActive:.................10.0 GbpsPeer PortInfo:# Port info: Lid 15 DR path slid 1; dlid 65535; 0,28 port 28LinkState:.......................ActivePhysLinkState:...................LinkUpLinkWidthSupported:..............1X or 4XLinkWidthEnabled:................1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedActive:.................10.0 Gbps#



where:



Note – If a port value of 255 is specified for a switch node, the counters are the totalfor all switch ports.

For example:

Note – The output in the example is just a portion of the full output.

Related Information■ Switch Reference, perfquery command

■ “Clear Data and Error Counters” on page 65

▼ Display Low-Level Detailed Information Abouta NodeIf intensive troubleshooting is necessary to resolve a problem, the smpquerycommand can provide very detailed information about a node.

# perfquery lid port

# perfquery 15 28# Port counters: Lid 15 port 28PortSelect:......................28CounterSelect:...................0x1b01SymbolErrors:....................0...VL15Dropped:.....................0XmtData:.........................33791184RcvData:.........................33789168XmtPkts:.........................469322RcvPkts:.........................469294#



where lid is the LID of the node.

For example, to see detailed information about a switch with LID 15, type.

Related Information■ Switch Reference, smpquery command

■ “Display Low-Level Detailed Information About a Port” on page 58

▼ Display Low-Level Detailed Information Abouta PortThe smpquery command can provide very detailed information about a port.


where:


# smpquery switchinfo lid

# smpquery switchinfo 15# Switch info: Lid 15LinearFdbCap:....................49152RandomFdbCap:....................0McastFdbCap:.....................4096LinearFdbTop:....................15DefPort:.........................0DefMcastPrimPort:................255DefMcastNotPrimPort:.............255LifeTime:........................18StateChange:.....................0LidsPerPort:.....................0PartEnforceCap:..................32InboundPartEnf:..................1OutboundPartEnf:.................1FilterRawInbound:................1FilterRawOutbound:...............1EnhancedPort0:...................1#

# smpquery portinfo lid port



For example, to see detailed information about port 23 on the switch with LID 15,type.

Note – The output in the example is just a portion of the full output.

Related Information■ Switch Reference, smpquery command

■ “Display Low-Level Detailed Information About a Node” on page 58

# smpquery portinfo 15 28# Port info: Lid 15 port 28Mkey:............................0x0000000000000000GidPrefix:.......................0x0000000000000000Lid:.............................0x0000SMLid:...........................0x0000CapMask:.........................0x0DiagCode:........................0x0000MkeyLeasePeriod:.................0LocalPort:.......................28LinkWidthEnabled:................1X or 4XLinkWidthSupported:..............1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkState:.......................ActivePhysLinkState:...................LinkUpLinkDownDefState:................PollingProtectBits:.....................0LMC:.............................0...SubnetTimeout:...................0RespTimeVal:.....................0LocalPhysErr:....................8OverrunErr:......................8MaxCreditHint:...................85RoundTrip:.......................16777215#


▼ Display the InfiniBand Fabric ConfigurationIf you have configured the InfiniBand fabric for SNMP operations, you can displaythe elements configured with the fdconfig command.


In this example, the element named primary has IP address 123.45.67.89 and is aSun Datacenter InfiniBand Switch 36. It is hosting the master Fabric Directordaemon. The other element named secondary has an IP address of 123.45.67.90and is a Sun Network QDR InfiniBand Gateway Switch.

Related Information■ Switch Reference, fdconfig command


▼ Display the InfiniBand Fabric PartitionConfigurationIf you have partitioned the InfiniBand fabric, you can display the active partitionconfiguration or the pending modified partition configuration with thesmpartition command.

# fdconfig list-current-fabric-configName IP Addr Type Role(s)isMaster Fabricname-------------------------------- ---------------- -------------------------- -------- ----------primary 123.45.67.89 sw36 leafSwitchyes testsecondary 123.45.67.90 sw36gw gatewayno test#



where partition is the partition configuration name (active or modified ).

For example:

In this example, this is the 12th configuration file to be created. The partition keyis set to a default of 7fff. IP protocol over InfiniBand is configured. All hosts(CAS), all switches, and the management controller itself are granted fullmembership priviledges.

Related Information■ Switch Command Reference, smpartition command



Controlling the InfiniBand FabricYou can perform these tasks to manage the InfiniBand fabric and its components.



■ “Determine Changes to the InfiniBand Fabric Topology” on page 62


■ “Determine Which Links Are Experiencing Significant Errors” on page 64

■ “Clear Data and Error Counters” on page 65

# smpartition list partition

# smpartition list active# Sun DCS IB partition config file# This file is generated, do not edit#! version_number : 12Default=0x7fff, ipoib : ALL_CAS=full, ALL_SWITCHES=full, SELF=full;SUN_DCS=0x0001, ipoib : ALL_SWITCHES=full;#


■ “Reset a Port” on page 65

■ “Set Port Speed” on page 66



Related Information







▼ Perform Comprehensive Diagnostics for theEntire FabricIf you require a full testing of your InfiniBand fabric, the ibdiagnet command canperform many tests with verbose results. The command is a useful tool to determinethe general overall health of the InfiniBand fabric.


Related Information■ Switch Reference, ibdiagnet command


▼ Determine Changes to the InfiniBand FabricTopologyIf your fabric has a number of nodes that are suspect, the generatetopologycommand enables you to take a snapshot of your fabric. At a later time, use thematchtopology command to compare the topology file to the present conditions.

# ibdiagnet -v -r


1. Take a snapshot of the fabric topology.

where date is the date of the snapshot. For example:

2. After an event, compare the present topology to that saved in the topology file.

where date is the date of the snapshot. For example:

Related Information■ Switch Reference, generatetopology command

■ Switch Reference, matchtopology command

■ “Display the InfiniBand Fabric Topology (Simple)” on page 54

▼ Find 1x, SDR, or DDR Links in the FabricYou can use the ibdiagnet command to determine which links are at 1x bandwidth,2.5 Gbps, or 5 Gbps data rate.

# generatetopology date.topo

# generatetopology Sept8.topoIt will take some time to generate a topology file. Please wait!topo.conf exist! will move it to topo.conf.oldwill create new topo.confTopo.conf file is created. Will now start generating the topo fileWrote Topology file:Sept8.topo#

# matchtopology date.topo

# matchtopology Sept8.topoTopology matching will take some time. Please wait!-I----------------------------------------------------I- Topology matching results-I----------------------------------------------------I- The topology defined in ib_topology.topo perfectly matches thediscovered fabric.----------------------------------------------------------------#



In this instance of the ibdiagnet command, there is a check for all links to be at4x QDR (-lw 4x -ls 10), and if not, to report those links that are not 4x QDR.



■ “Determine Which Links Are Experiencing Significant Errors” on page 64

▼ Determine Which Links Are ExperiencingSignificant ErrorsYou can use the ibdiagnet command to determine which links are experiencingsymbol errors and recovery errors by injecting packets.


In this instance of the ibdiagnet command, 1000 test packets are injected intoeach link. The -P all=1 option returns all Performance Monitor counters thatincrement during the test, respective to the GUID and port of the InfiniBanddevice.

2. In the output of the ibdiagnet command, search for thesymbol_error_counter string.

That line contains the symbol error count in hexadecimal. The preceding linesidentify the node and port with the errors. Symbol errors are minor errors. If thereare relatively few minor errors during the diagnostic, they can be monitored.

Note – According to InfiniBand specification 10E-12 BER, the maximum allowablesymbol error rate is 120 errors per hour.

# ibdiagnet -lw 4x -ls 10 -pc -pm -skip all

# ibdiagnet -c 1000 -P all=1


3. Also in the output of the ibdiagnet command, search for thelink_error_recovery_counter string.

That line contains the recovery error count in hexadecimal. The preceding linesidentify the node and port with the errors. Recovery errors are major errors. Therespective links must be investigated for the cause of the rapid symbol errorpropagation.




▼ Clear Data and Error CountersWhen you are optimizing the InfiniBand fabric for performance, you might want toknow how the throughput increases or decreases according to changes you aremaking. Alternatively, if you are troubleshooting a port, the perfquery commandprovides counters of errors occurring at that port. To determine the throughput as afunction of time, or if the problem at the port has been resolved, the ibdiagnet-pc command enables you to reset the data counters and error counters to 0.



■ Switch Reference, perfquery command


▼ Reset a PortYou might need to reset a port to determine its functionality.


where:


# ibdiagnet -pc

# ibportstate lid port reset



For example:




▼ Set Port SpeedYou can manually set the speed of a single port to help determine symbol errorgeneration. The ibportstate command can set the speed to 2.5, 5.0, or 10.0Gbyte/sec.


where:


# ibportstate 1 28 resetInitial PortInfo:# Port info: Lid 1 port 28LinkState:.......................DownPhysLinkState:...................PollingLinkWidthSupported:..............1X or 4XLinkWidthEnabled:................1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedActive:.................2.5 Gbps

After PortInfo set:# Port info: Lid 1 port 28LinkState:.......................DownPhysLinkState:...................Disabled

After PortInfo set:# Port info: Lid 1 port 28LinkState:.......................DownPhysLinkState:...................Polling#

# ibportstate lid port speed speed



■ speed is the speed of the port: 1 for 2.5 Gbyte/sec, 2 for 5.0 Gbyte/sec, and 4 for10.0 Gbyte/sec.

Note – Adding speed values enables either speed. For example, speed 7 is 2.5, 5.0,and 10.0 Gbyte/sec.

For example:



▼ Disable a PortIf a port is found to be problematic, you can disable it.


where:


# ibportstate 15 28 speed 1Initial PortInfo:# Port info: Lid 15 port 28LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps

After PortInfo set:# Port info: Lid 15 port 28LinkSpeedEnabled:................2.5 Gbps# ibportstate 15 28 speed 7Initial PortInfo:# Port info: Lid 15 port 28LinkSpeedEnabled:................2.5 GbpsAfter PortInfo set:# Port info: Lid 15 port 28LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps#

# ibportstate lid port disable



For example:





▼ Enable a PortAfter disabling a port, you can enable the port with the ibportstate command.


where:


# ibportstate 15 28 disableInitial PortInfo:# Port info: Lid 15 port 28LinkState:.......................ActivePhysLinkState:...................LinkUpLinkWidthSupported:..............1X or 4XLinkWidthEnabled:................1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedActive:.................10.0 Gbps

After PortInfo set:# Port info: Lid 15 port 28LinkState:.......................DownPhysLinkState:...................Disabled#

# ibportstate lid port enable



For example:





Configuring the Fabric Director NodeListThese topics enable you to configure the Fabric Director node list.

■ “Fabric Director and Fabric Elements” on page 70

■ “fdconfig Command” on page 71

■ “Create a Fabric Configuration” on page 72

■ “Add or Remove an Element From the Fabric Configuration” on page 75

■ “Modify an Element of the Fabric Configuration” on page 77

■ “Exchange Master Fabric Director Status” on page 79

# ibportstate 15 28 enableInitial PortInfo:# Port info: Lid 15 port 28LinkState:.......................DownPhysLinkState:...................DisabledLinkWidthSupported:..............1X or 4XLinkWidthEnabled:................1X or 4XLinkWidthActive:.................4XLinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 GbpsLinkSpeedActive:.................2.5 Gbps

After PortInfo set:# Port info: Lid 15 port 28LinkState:.......................DownPhysLinkState:...................PortConfigurationTraining#


Related Information




Fabric Director and Fabric ElementsThe Fabric Director monitors the InfiniBand fabric by polling the nodes or elements ofthe fabric. Each fabric element has a Fabric Director instance, and the twocommunicate with eachother in a one-to-one relationship. The Fabric Director gathersinformation about the fabric element through the LDA and ENVD daemons runningon the fabric element. From the information gathered, the Fabric Director creates alocal SNMP fabricMIB.

One Fabric Director is assigned the role of master. The master Fabric Director gathersinformation about all of the fabric elements by communicating out-of-band(management network) with the Fabric Directors instances on those fabric elements.This master Fabric Director creates a model of the fabric which is also in the form ofan SNMP MIB, as defined by the SUN-FABRIC-MIB specification.

By default, Fabric Directors run in non-master mode. Only when a Fabric Directorreads the fabric node list and discovers that its respective fabric element is to host themaster Fabric Director, does it become the master Fabric Director.

Fabric Directors generate fabricMIB tables such as:

■ fabricElemPortTable

■ fabricElemConnectorTable

and also generate fabricMIB scalars such as:

■ fabricElemMgrIpAddress

■ fabricElemName

■ fabricElemNumPorts

■ fabricElemOperStatus

■ fabricElemNumConnectors

Master Fabric Directors additionally generate fabricMIB tables such as:

■ fabricMgmtElemTable

■ fabricMgmtExtElemTable

■ fabricMgmtLinkTable

and generate fabricMIB scalars such as:

■ fabricMgmtFabricDescr


■ fabricMgmtFabricType

■ fabricMgmtFabricTopology

■ fabricMgmtFabricOperStatus

■ fabricMgmtFabricName

Related Information

■ Switch Reference, SUN-FABRIC-MIB mib






fdconfig CommandThe fdconfig command is used to configure a list of fabric elements expected to bein the fabric, the fabric node list. The list is used by the Fabric Directors, andprovides these attributes for each fabric element:

■ Name – The name of the fabric element. Can be the host name if desired.

■ IP address – The out-of-band management network IP address of the managementcontroller hosting the fabric element and Fabric Director.

■ Fabric element type – The name of the type of fabric element.

■ sw36 – Sun Datacenter InfiniBand Switch 36

■ sw36gw – Sun Network QDR InfiniBand Gateway Switch

■ unknown – The element’s type is unknown.

■ Role type – The purpose of the fabric element:

■ gateway – The element acts as a gateway.

■ spineSwitch – The element acts as a spine switch.

■ leafSwitch – The element acts as a leaf switch.

■ unknown – The element’s role is unknown.

■ Master – Whether the Fabric Director instance on the fabric element is the master.

Note – At this time, the fdconfig command is only available to the root user.


Note – See Switch Reference, fdconfig command, for more information.

Related Information

■ Switch Reference, fdconfig command






▼ Create a Fabric ConfigurationBefore you can use the fabric MIBs, you must first create a fabric configuration. Youmust create identical configurations on all elements of the fabric.


1. On the management controller, initiate a fabric configuration seesion.

2. Define the fabric’s name.

where name is the identifier of the fabric. For example:

# fdconfig start-fabric-config empty#

# fdconfig define-fabric-name name

# fdconfig define-fabric-name testFabric name updated#


3. Define an element of the fabric.

where:

■ name is the identifier of the element (can be host name).

■ IP_address is the IP address of an element.

■ type is the identifer of the type of element:




■ role is the purpose of the element:





For example:

4. Verify the fabric configuration.

5. Consider your next step.

■ If you want to add elements, perform Step 3 to Step 4 for each new element.

# fdconfig define-element -name name -ip IP_address -type type -rolerole1 [-role role2]

# fdconfig define-element -name primary -ip 123.45.67.89 -type sw36 -roleleafSwitchElement added#

# fdconfig list-in-progress-fabric-configName IP Addr Type Role(s)isMaster Fabricname-------------------------------- ---------------- -------------------------- -------- ----------primary 123.45.67.89 sw36 leafSwitchyes test#


■ If you want to abort the configuration altogether, type.

■ Otherwise, go to Step 6.

6. End the configuration session and commit to the new configuration.

7. Perform Step 1 through Step 6 on the management controllers of all elements ofthe fabric.

8. Reinitiate the fabric configuration on the element which is to host the masterFabric Director.

9. Configure the element to host the master Fabric Director.

Note – There must be only one element hosting the master Fabric Director withinthe fabric.

where name is the identifier assigned to the element to host the master FabricDirector. For example:


# fdconfig abortIn progress config aborted#

# fdconfig complete-fabric-config#

# fdconfig start-fabric-config from-current#

# fdconfig set-master name

# fdconfig set-master primaryElement primary updated#




The configuration files and fabric MIBs are created.







▼ Add or Remove an Element From the FabricConfiguration


1. Initiate a fabric configuration seesion.

2. Add an element to the fabric configuration.

where:










# fdconfig define-element -name name -ip IP_address -type type -rolerole1 [-role role2]





For example:

3. Or, remove an element from the fabric configuration.

where name is the identifier of the element. For example.




# fdconfig define-element -name secondary -ip 123.45.67.90 -typesw36gw -role gatewayElement added#

# fdconfig remove-element -name name

# fdconfig remove-element -name secondaryElement deleted#

# fdconfig list-in-progress-fabric-configName IP Addr Type Role(s)isMaster Fabricname-------------------------------- ---------------- -------------------------- -------- ----------primary 123.45.67.89 sw36 leafSwitchyes testsecondary 123.45.67.90 sw36gw gatewayno test#



7. If you added a new element, create a fabric configuration on the managementcontroller of that element using the same configuration information.

See “Create a Fabric Configuration” on page 72.







▼ Modify an Element of the Fabric ConfigurationYou can modify the parameters of an element.

Note – To change the master Fabric Director status, see “Exchange Master FabricDirector Status” on page 79.



2. Modify an element of the fabric configuration.

where:







# fdconfig redefine-element -name name [-ip IP_address|-typetype|-role role1 [-role role2]]








For example:










# fdconfig redefine-element -name primary -role spineSwitchElement modified#

# fdconfig list-in-progress-fabric-configName IP Addr Type Role(s)isMaster Fabricname-------------------------------- ---------------- -------------------------- -------- ----------primary 123.45.67.89 sw36 spineSwitchyes testsecondary 123.45.67.90 sw36gw gatewayno test#



▼ Exchange Master Fabric Director StatusShould you need to move the master Fabric Director status from one element toanother, follow this procedure.

Note – There must be one element in the fabric hosting the master Fabric Director.



2. Remove master Fabric Director status from the first element.

where name is the identifier of the element. For example:

3. Assign master Fabric Director status to the new element.

where name is the identifier of the element. For example:



# fdconfig reset-master name

# fdconfig reset-master primaryElement primary updated#

# fdconfig set-master name

# fdconfig set-master secondaryElement secondary updated#

# fdconfig list-in-progress-fabric-configName IP Addr Type Role(s)isMaster Fabricname-------------------------------- ---------------- -------------------------- -------- ----------










Partitioning the InfiniBand FabricThese topics enable you to partition the InfiniBand fabric and modify the partition’sconfiguration.

■ “smpartition Command” on page 81

■ “Remove User Partitions for Firmware Downgrades” on page 82

■ “Use the smnodes Command” on page 83

■ “Determine Partitions and P_Keys” on page 84

■ “Create a User Partition” on page 85

■ “Add or Remove a Port From a Partition” on page 87

■ “Modify a Partition or Port” on page 89

■ “Delete a Partition” on page 91

primary 123.45.67.89 sw36 spineSwitchno testsecondary 123.45.67.90 sw36gw gatewayyes test#



Related Information




smpartition CommandYou can partition your InfiniBand fabric using the smpartition command and itssubcommands. The smpartition command is issued from the switch that has themaster Subnet Manager and acts upon the partition configuration. There are twoconfigurations, the active configuration is the one currently in use, and the modifiedconfiguration is the one which you can affect.

You begin a configuration session with the smpartition start command. Youthen make changes to the modified configuration with the smpartition create,smpartition add, smpartition remove, smpartition modify, andsmpartition delete commands. You finally make the modified configuration intothe active configuration with the smpartition commit command.

See Switch Reference, smpartition command, for more information.

Note – There is a hardware limitation of 127 user-defined and 1 default partition foreach port of the ConnectX2 chips (in HCAs) and for each InfiniBand port of theBridgeX chips (in gateways).

Related Information

■ Switch Reference, smpartition command









▼ Remove User Partitions for FirmwareDowngradesYou must perform these steps before downgrading the firmware to a version before2.0.

Note – Only perform this procedure if you are downgrading the firmware.

Note – By removing user partitions, you might lose connectivity that was availableonly through those partitions. Additionally, you might gain undesired connectivitybecause all hosts become full members of the default partition.

1. On the management controller of the master Subnet Manager, remove all userdefined partitions.

See “Delete a Partition” on page 91.

2. Restore the default partition to default settings.

See “Modify a Partition or Port” on page 89, however use this command line tomodify the partition.

3. Consider your next step:

■ If your InfiniBand fabric has just one Subnet Manager, downgrade thefirmware.

See Switch Remote Administration, upgrading the firmware.

■ If your InfiniBand fabric has multiple Subnet Managers, go to Step 4.

4. Remove all entries from the Subnet Manager node list.

where IP_address are the IP addresses of the Subnet Manager nodes. For example:

5. Repeat Step 4 on the management controllers of all Subnet Managers in theInfiniBand fabric.

# smpartition modify -pkey 0x7fff -port ALL_CAS -flag ipoib -m full

# smnodes delete IP_address [IP_address ...]

# smnodes delete 123.45.67.89 123.45.67.90#


6. Downgrade the firmware.

See Switch Remote Administration, upgrading the firmware.

Related Information■ Switch Reference, smpartition command

■ Switch Reference, smnodes command








▼ Use the smnodes CommandIf you are partitioning your InfiniBand fabric, a list of valid Subnet Manager nodesmust exist in the file system of every management controller running a SubnetManager. You create this file with the smodes command. The file contains a list of IPaddresses of all active management controllers running a Subnet Manager in yourfabric. The file should have an entry for every Sun Datacenter InfiniBand Switch 36and Sun Network QDR InfiniBand Gateway Switch that runs a Subnet Manager inyour InfiniBand fabric.

Note – If the Subnet Manager nodes of your InfiniBand fabric ever change (disabled,added, and so on), you must update all copies of the Subnet Manager nodes file.

1. Determine the IP addresses of all management controllers in your InfiniBandfabric.

2. On management controller, propagate the Subnet Manager nodes file with theIP addresses of all Subnet Manager nodes.

where IP_address is the IP address of each management controller hosting a SubnetManager. For example:

# smnodes add IP_address IP_address ...

# smnodes add 123.45.67.89 123.45.67.90#


3. Repeat Step 2 for all management controllers in the InfiniBand fabric.

4. Determine how you will partition your InfiniBand fabric.

See “Determine Partitions and P_Keys” on page 84.

Related Information■ Switch Reference, smodes command








▼ Determine Partitions and P_KeysThe switch supports several partitions in the InfiniBand fabric. You assign a P_Key toeach partition as a simplified means of identifying the partition to the SubnetManager. P_Keys are 15-bit integers and have a value of 0x1 to 0x7fff. A P_Keyvalue of 0x7fff represents the default partition.

An additional bit, the membership bit, can identify the membership of the partition:

■ Full – The membership bit is 1. Full membership permits communication to allmembers within a partition.

■ Limited – The membership bit is 0. Limited membership permits communicationonly with a full member.

Combined together, P_Key and the membership bit comprise a 16-bit integer, and themost significant bit is the membership bit. In a full membership, the membership bitis set high. When this happens, the P_Key value is effectively increased by 0x8000.Similarly, if you were to define a P_Key with a value greater than 0x8000, themembership bit is automatically set to 1, and consequently is given full membership.

1. Use the output of the ibswitches and ibhosts commands to identify yourswitch and HCA node GUIDs.

See:



2. Determine the partitions you will have, their names, and their respectiveP_Keys.


3. Collate the GUIDs, partition names, and P_Keys into a partition informationtext file.

4. (Optional) If you will also configure vLANs, assign a unique vLAN number tosimilar P_Keys in the text file.

Related Information■ Switch Reference, ibswitches command









▼ Create a User Partition

Note – After creating user partitions, consider modifying the default partition tolimited membership for CAs and disabling IPoIB. The reason being that nodes withfull membership in different user partitions can communicate with each otherthrough the default partition. See “Modify a Partition or Port” on page 89 for anexample.

1. If you have not already done so, create the Subnet Manager nodes file.

See “Use the smnodes Command” on page 83.

2. Initiate a partition configuration session on the management controller.

3. Create the user partition.

where:

# smpartition start#

# smpartition create -n partition_name -pkey p_key[-flag [ipoib, mtumtu, rate rate, sl sl, scope scope]][-m defmember]


■ partition_name is the identifier of the InfiniBand partition.

■ p_key is the partition key (1 – 7fff or default).

■ mtu is the value of the MTU.

■ rate is the throughput of a link (link width + link speed) in Gbps from 2.5, 5,10, 20, 30 to a maximum of 40.

■ sl is the service level.

■ scope is the multicast address scope value (0 – F).

Note – The mtu, rate, sl, and scope parameters are for the multicast groupcreated when ipoib (IP over InfiniBand) is configured for the partition.

■ defmember is the default membership type (full, limited, or both) for thepartition.

Note – If ports are added to the partition without specifying the membership type,the default membership type is applied to the port

For example:

4. Verify the partition configuration.

5. Consider your next step.

■ If you want to add GUIDs, see “Add or Remove a Port From a Partition” onpage 87.

■ If you want to modify the configuration, see “Modify a Partition or Port” onpage 89.

■ If you want to delete the configuration altogether, see “Delete a Partition” onpage 91.

# smpartition create -n testpartition -pkey 5 -m full#

# smpartition list modified# Sun DCS IB partition config file# This file is generated, do not edit#! version_number : 13Default=0x7fff, ipoib : ALL_CAS=full, ALL_SWITCHES=full, SELF=full;SUN_DCS=0x0001, ipoib : ALL_SWITCHES=full;testpartition = 0x0005,defmember=full:#


■ Otherwise, go to Step 6.

6. End the configuration session and commit the configuration to the activepartition configuration.









▼ Add or Remove a Port From a PartitionYou can add or remove one or more ports from the partition at one time.


2. Add ports to the partition.

where:


■ p_key is the partition key (1 – 7fff).

■ port is the GUID of the port, or the special parameter, to add:

■ ALL – All of the CAs, switches, and routers in the InfiniBand fabric.

■ ALL_CAS – All CAs in the InfiniBand fabric.

■ ALL_SWITCHES – All switches.

# smpartition commit#


# smpartition add -n partition_name|-pkey p_key -portport|ALL|ALL_CAS|ALL_SWITCHES|ALL_ROUTERS|SELF [-m member]


■ ALL_ROUTERS – All routers.

■ SELF – The Master Subnet Manager.

■ member is the membership type (full, limited, or both) for the port.

For example:

3. Or, remove ports from the partition.

where:



■ port is the GUID of the port, or the special parameter, to remove:






For example:


# smpartition add -n testpartition -port 00212800013e9313 00212800013e931400212800013e93f7#

# smpartition remove -n partition_name|-pkey p_key -portport|ALL|ALL_CAS|ALL_SWITCHES|ALL_ROUTERS|SELF

# smpartition remove -n testpartition -port 00212800013e9314#

# smpartition list modified# Sun DCS IB partition config file# This file is generated, do not edit#! version_number : 13Default=0x7fff, ipoib : ALL_CAS=full, ALL_SWITCHES=full, SELF=full;SUN_DCS=0x0001, ipoib : ALL_SWITCHES=full;testpartition = 0x0005,defmember=full:0x00212800013e9313,0x00212800013e93f7;#











▼ Modify a Partition or PortYou can modify the partition’s configuration or the port(s) membership with thesmpartition modify command.


2. Modify the partition or port(s).

where:



■ mtu is the value of the MTU.

■ rate is the throughput of a link (link width + link speed) in Gbps from 2.5, 5,10, 20, 30 to a maximum of 40.

■ sl is the service level.



# smpartition modify -n partition_name|-pkey p_key[-flag [ipoib, mtu mtu, rate rate,sl sl, scope scope]]|[-port port|ALL|ALL_CAS|ALL_SWITCHES|ALL_ROUTERS|SELF [-mmember]


■ scope is the multicast address scope value (0 – F).

Note – The mtu, rate, sl, and scope parameters are for the multicast groupcreated when ipoib (IP over InfiniBand) is configured for the partition.

■ port is the GUID of the port, or the special parameter, to modify:






■ member is the membership type (full, limited, or both) for the port.

For example, to configure the default partition for limited membership for CAsand no IP over InfiniBand support:





# smpartition modify -pkey 0x7fff -port ALL_CAS -flag -m limited#

# smpartition list modified# Sun DCS IB partition config file# This file is generated, do not edit#! version_number : 13Default=0x7fff, ipoib : ALL_CAS=full, ALL_SWITCHES=full, SELF=full;SUN_DCS=0x0001, ipoib : ALL_SWITCHES=full;testpartition = 0x0005,defmember=full,ipoib:0x00212800013e9313,0x00212800013e93f7;#









▼ Delete a PartitionWhen you delete a partition, you effectively commit a blank default configuration.


2. Delete the partition.

where:



For example:







# smpartition delete -n partition_name|-pkey p_key

# smpartition delete -n testpartition#



Index

Aadding

element to configuration, 75ports to partition configuration, 87

administeringchassis, 21I4 switch chip, 31InfiniBand fabric, 51Subnet Manager, 39

administrative command overview, 17

Cchanging

element master status, 79password for administrator, 29

chassisadministering, 21controlling, 28FRU ID, 26monitoring, 21status LEDs, 8

checkboot command, 34checking

boardtemperature, 24voltage, 23

chassis status, 25environmental data, 25fan

speed, 24status, 24

LEDschassis status, 8fan, 11link, 10network management, 9power supply, 10

link status, 34port status, 35power supply, 22switch chip boot status, 34

checkpower command, 22checkvoltages command, 23clear

data counters, 65error counters, 65management key, 48

commandcheckboot, 34checkpower, 22checkvoltages, 23dcsport, 32disablesm, 49disableswitchport, 36enablesm, 48enableswitchport, 37env_test, 25fdconfig, 71

abort, 72complete-fabric-config, 72define-element, 75define-fabric-name, 72list-current-fabric-config, 60redefine-element, 77remove-element, 75reset-master, 79set-master, 79start-fabric-config empty, 72

generatetopology, 62getfanspeed, 24getmaster, 40getportstatus, 35ibdiagnet, 62, 63, 64, 65ibhosts, 53ibnetdiscover, 54

93

ibportstate, 55, 65, 66, 67, 68ibswitches, 52ibtracert, 55listlinkup, 34matchtopology, 62perfquery, 57reboot, 28setsmmkey, 41, 46, 48setsmpriority, 41, 44, 46setsubnetprefix, 45showfruinfo, 26showpsufru, 26showsmlog, 42showtemps, 24showtopology, 54showunhealthy, 22smconfigtest, 41smnodes, 83smpartition, 81

add, 87create, 85delete, 91list, 60modify, 89remove, 87start, 85

smpquery, 58version, 27

configuration overview, 17configuring

Fabric Director node list, 69starting configuration session, 72

controllingchassis, 28I4 switch chip, 36InfiniBand fabric, 61Subnet Manager, 43

creatingfabric configuration, 72partition configuration, 85

Ddata counters

clearing, 65dcsport command, 32deleting

partition configuration, 91disablesm command, 49

disableswitchport command, 36disabling

port, 67Subnet Manager, 49switch chip port, 36

displayingfabric configuration, 60FRU ID

chassis, 26power supply, 26

nodecounters, 57information, 58link status, 55

partition configuration, 60port information, 58route, 55Subnet Manager

activity, 40configuration integrity, 41controlled handover state, 41log, 42management key, 41prefix, 41priority, 41status, 40

switchfirmware, 27

switch health, 22topology

detailed, 54simple, 54

Eelement

adding, 75changing master status, 79modifying, 77removing, 75

enablesm command, 48enableswitchport command, 37enabling

port, 68Subnet Manager, 48switch chip port, 37

env_test command, 25error counters

clearing, 65


displaying, 57example route through switch, 14

Ffabric

diagnostics, 62element, 70

Fabric Director, 70fan

checkingLEDs, 11speed, 24status, 24

LED, 7fdconfig

abort command, 72complete-fabric-config command, 72define-element command, 75define-fabric-name command, 72list-current-fabric-config

command, 60redefine-element command, 77remove-element command, 75reset-master command, 79set-master command, 79start-fabric-config empty command, 72

fdconfig command, 71firmware

downgrades and partitions, 82switch, 27

front status LEDs, 7FRU ID

chassis, 26power supply, 26

full membership, 84

Ggeneratetopology command, 62getfanspeed command, 24getmaster command, 40getportstatus command, 35GUID

mappingto CXP connector, 33to switch chip, 33

nomenclature, 14

Hhardware commands

overview, 19hardware problems, 1HCA identity, 53

II4 switch chip

administering, 31controlling, 36monitoring, 31

ibdiagnet command, 62, 63, 64, 65ibhosts command, 53ibnetdiscover command, 54ibportstate command, 55, 65, 66, 67, 68ibswitches command, 52ibtracert command, 55identifying

HCAs, 53LEDs, 6switches, 52

ILOMcommand

overview, 20InfiniBand

commands, 19creating configuration, 72displaying configuration, 60fabric

administering, 51controlling, 61monitoring, 51problems, 3

LLEDs

chassis status, 8fan, 7, 11front, 7identifying, 6link, 8, 10network management, 8, 9power supply, 7, 10rear, 8

limited membership, 84link

Index 95

LEDs, 10status, 34status of node, 55

listlinkup command, 34

Mmanagement controller

restarting, 28mapping

GUIDto CXP connector, 33to switch chip, 33

LED to switch chip port, 13QSFP to switch chip port, 13, 32switch chip port

to LED, 12to QSFP, 12, 32

matchtopology command, 62membership

full, 84limited, 84

modifyingelement, 77partition configuration, 89port membership, 89

monitoringchassis, 21I4 switch chip, 31InfiniBand fabric, 51Subnet Manager, 39

Nnetwork management

checking LEDs, 9node

counters, 57information, 58link status, 55

Ooverview

administrative commands, 17hardware commands, 19ILOM

commands, 20InfiniBand commands, 19

Pp_keys, 84partition

creating configuration, 85displaying configuration, 60

partitioning, 84adding ports, 87deleting configuration, 91firmware downgrades, 82InfiniBand fabric, 80modifying configuration, 89removing ports, 87starting configuration session, 85

password, 29perfquery command, 57poor performing links, 63port

disabling, 67enabling, 68information, 58resetting, 65speed, 66status, 35

power supplychecking

LEDs, 10status, 22

FRU ID, 26LEDs, 7

problemhardware, 1InfiniBand fabric, 3

QQSFP connector

mappingfrom GUID, 33to switch chip port, 32

Rrear status LEDs, 8reboot command, 28recovery errors, 64removing

element from configuration, 75ports from partition configuration, 87


resettingport, 65

restartingmanagement controller, 28

routeQSFP to switch chip port, 13, 32switch chip port to QSFP, 12, 32through switch, 12, 14through the fabric, 55

Ssetsmmkey command, 41, 46, 48setsmpriority command, 41, 44, 46setsubnetprefix command, 45setting

port speed, 66Subnet Manager

controlled handover, 46management key, 46prefix, 45priority, 44

showfruinfo command, 26showpsufru command, 26showsmlog command, 42showtemps command, 24showtopology command, 54showunhealthy command, 22smconfigtest command, 41smnodes command, 83smpartition

add command, 87create command, 85delete command, 91list command, 60modify command, 89remove command, 87start command, 85

smpartition command, 81smpquery command, 58status

chassis, 25fan, 24power supply, 22Subnet Manager, 40

Subnet Managerclearing

management key, 48configuration integrity, 41controlled handover state, 41controlling, 43disabling, 49displaying

activity, 40configuration integrity, 41log, 42management key, 41prefix, 41priority, 41status, 40

enabling, 48management key, 41, 46, 48monitoring, 39setting

controlled handover, 46management key, 46prefix, 45priority, 44

switchGUIDs, 14identity, 52troubleshooting, 1

switch chipchecking boot status, 34port

disabling, 36enabling, 37

symbol errors, 64

Ttopology

change, 62displaying

detailed, 54simple, 54

troubleshooting the switch, 1

V

version command, 27

Index 97

sun datacenter infiniband switch 36 administration guide ... · 2 sun datacenter infiniband switch...

Documents