new dell poweredge c servers data analytics...

Dell PowerEdge C Servers Data Analytics Solution

A Dell Deployment Guide

Dell │ Greenplum

Release A0

September 2010

Dell | Greenplum Database Solution | Deployment Guide

Page ii

© 2010 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without

the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the DELL logo, and the DELL badge, PowerConnect, and PowerVault are trademarks of Dell Inc.

Microsoft, Windows, Windows Server, and Active Directory are either trademarks or registered

trademarks of Microsoft Corporation in the United States and/or other countries. Other trademarks and

trade names may be used in this document to refer to either the entities claiming the marks and names

or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than

its own.

September 2010


Page 1

CONTENTS

1. BILL OF MATERIALS ............................................................................................................................................ 2

1.1 RACKING ................................................................................................................................................................. 2

1.2 POWER OPTIONS ...................................................................................................................................................... 2

1.3 POWER JUMPERS ...................................................................................................................................................... 5

1.4 KEYBOARD, MONITOR, MOUSE ................................................................................................................................... 5

1.5 SERVERS .................................................................................................................................................................. 6

1.6 NETWORKING .......................................................................................................................................................... 7

2. INSTALLATION .................................................................................................................................................... 9

2.1 RACKING ................................................................................................................................................................. 9

2.2 NETWORK ............................................................................................................................................................. 10

2.3 SERVERS ................................................................................................................................................................ 17

3. VALIDATION ..................................................................................................................................................... 38

4. LABELS ............................................................................................................................................................. 39

4.1 RACKS .................................................................................................................................................................. 39

4.2 SERVERS ................................................................................................................................................................ 39

4.3 SWITCHES .............................................................................................................................................................. 39

4.4 PATCH PANELS ....................................................................................................................................................... 39

4.5 NETWORK CABLES ................................................................................................................................................... 39

5. PHYSICAL INSTALLATION DETAILS .................................................................................................................... 41

5.1 OPTION CONCERNS ................................................................................................................................................. 41

5.2 RACKING ............................................................................................................................................................... 41

5.3 NETWORK DIAGRAMS .............................................................................................................................................. 52

APPENDIX A: BILL OF MATERIALS......................................................................................................................... 56

APPENDIX A1: MASTER NODE BILL OF MATERIALS ............................................................................................................. 56

APPENDIX A2: SEGMENT NODE BILL OF MATERIALS ............................................................................................................ 57

APPENDIX A3: ETL NODE BILL OF MATERIALS ................................................................................................................... 59

APPENDIX A4: 42U RACK .............................................................................................................................................. 61

APPENDIX A5: 24U RACK .............................................................................................................................................. 61

APPENDIX A6: POWERCONNECT 6248............................................................................................................................. 61


Page 2

1. Bill of Materials The bill of materials for a cluster is dependent on the options selected. The primary option is the

number of segment nodes. This determines the capacity, performance, and physical size of the

solution. However, other options such as rack power requirements, component location in the racks,

and the specific configuration of master nodes are also important considerations.

This section does not include all of the manufacturer’s part numbers necessary to complete a cluster.

Please contact your Dell Sales Representative for a detailed BOM, including manufacturer’s part

numbers, for any specific cluster. This section is intended to give an overview of the elements included

in the clusters.

There are several general considerations common to all clusters. Under these classifications, all of the

supported options in the standard configuration can be addressed.

1.1 Racking

1.1.1 Racks

Racks need to be standard, 19”, four post racks that are not less than 1070mm deep. The rack

diagrams in section 6 are based on 42U high racks. Shorter racks can be used but the rack count,

weight per rack, and power requirements per rack need to be adjusted based on the new rack height.

The Dell PowerEdge 4220 24U Rack Enclosure is the rack unit for all 42U-rack installations. This rack

enclosure features:

More power distribution options than previous generations of Dell racks

Excellent airflow (doors are 80% perforated)

Various cable routing options.

The physical dimensions are: Height 78.7" (1999mm); Width 23.82" (605mm); Depth 42.15" (1071mm)

The static load rating for this rack enclosure is 2500 lbs.

The Dell Part Number is 42GFDS

The rack builder needs to define the size and number of filler panels needed in the 42U rack

configuration.

1.2 Power Options

1.2.1 North American Power Distribution

Each rack requires internal power distribution units (PDUs). All PDUs are mounted vertically in the rear

corners of the rack and do not consume rack units.

There are two power standards available, single-phase and three-phase PDUs. This relates to the design

of the power circuits. Single-phase circuits are more common and less expensive to install. Three-phase

circuits are common in large data centers as they more efficiently transport large amounts of power.


Page 3

1.2.1.1 Single-Phase

The recommended PDUs for single-phase power on North American deliveries are:

APC 7841 (Dell Part Number: A1643442)

More information on this part can be found at

http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=AP7841.

North American Single-Phase circuits should conform to the following parameters:

Input Power per Circuit

Default Plug Number of Circuits

Required Number of Supported

Segment Nodes

30A / 208V L6-30P twistlock;

3-prong 4 16 (14 on master rack)

Each rack will require four of these circuits to support the maximum number of servers in the rack with

full redundant power. The L6-30P plug requires an L6-30R receptacle:

L6-30R Receptacle L6-30R Plug

1.2.1.2 Three-Phase

Three-phase power uses three single-phase circuits aligned in a specific way to provide highly available

and reliable power. This circuit design is common to larger data centers but is expensive to install if

not already present. Three-phase should only be used in cases where the destination is already familiar

with the costs and requirements involved in provisioning three-phase circuits.

The recommended PDUs for three-phase power on North American deliveries are:

APC AP7868 (Dell Part Number: A0470127)

More information on these units is available at





Segment Nodes

35A / 208V CS8365 2 16 (14 on master rack)

Each rack will require two of these PDUs to support the maximum number of servers in the rack with

fully redundant power. CS8365 plugs require CS8365 receptacles.

CS8365 Receptacle CS8365 Plug

http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=AP7841http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=AP7868


Page 4

1.2.2 International Power Distribution

1.2.2.1 Single-Phase

The recommended PDUs for single-phase power on International deliveries are:

APC 7853 (Dell Part Number: A1666540)






Segment Nodes

30A / 208V IEC60309 2P+E 2 16 (14 on master rack)

Each rack will require two of these PDUs to support the maximum number of servers in the rack with

fully redundant power. IEC60309 2P+E plugs require IEC60309 2P+E receptacles.

IEC60309 2P+E Receptacle IEC60309 2P+E Plug

1.2.2.2 Three-Phase

Three-phase power uses three single-phase circuits aligned in a specific way to provide highly available

and reliable power. This circuit design is common to larger data centers but is expensive to install if

not already present. Three-phase should only be used in cases where the destination is already familiar

with the costs and requirements involved in provisioning three-phase circuits.

The recommended PDUs for three-phase power on International deliveries are:

ServerTech CS-24V4-P32MA


http://www.servertech.com/products/smart-pdus/smart-pdu-cs-24v-c13-c19-.




Segment Nodes

32A / 400V IEC 60309 3P+N+E

5-prong, 2 16 (14 on master rack)

Each rack will require two of these circuits to support the maximum number of servers in the rack with

full redundant power. The IEC 60309 3P+N+E plug requires an accompanying receptacle:

3P+E+N Receptacle 3P+E+N Plug

http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=AP7853http://www.servertech.com/products/smart-pdus/smart-pdu-cs-24v-c13-c19-


Page 5

1.3 Power Jumpers The default PDU is a base unit including a pigtail (the cord extending from the PDU to the electrical

source) but no power jumpers (cords that extend from the PDU to the backs of the servers).

Two jumpers per device (server, switch, etc.) are required. Each must be long enough to run from the

PDU, through any cable management, to the device.

The PDUs present C14 receptacles and the servers present C13 plugs. Power jumpers need to be C14

plugs on one end (shrouded with three vertical prongs) and C14 receptacles on the other (three vertical

holes).

C14 Plug C13 Receptacle

1.4 Keyboard, Monitor, Mouse A keyboard, monitor, mouse drawer is included in the master rack of each cluster. This is connected to

the secondary master node and used as the cluster console in the data center. The Dell recommended

part is the Dell 17” T1700 KVM Drawer.

In addition to the rack console tray, a small adapter called a VGA port saver is recommended. This

adapter fits on the end of the tray’s video cable and is what is plugged into servers. This protects the

pins in the tray’s cable, which is hard wired. Damage to the pins in the adapter from unplugging and

plugging the video cable is resolved by replacing the adapter. Damage to the video cable’s pins can

only be resolved by replacing the console tray itself.


Page 6

1.5 Servers Specific part numbers are not listed for servers here. For specific parts lists, please contact Dell.

1.5.1 Master Nodes

Both master nodes must be identically configured. The Bill of Materials for the Master Node

configuration is listed in Appendix A1.

1.5.2 Segment Nodes

All segment nodes must be identically configured. The Bill of Materials for the Segment Node

configuration is listed in Appendix A2.

1.5.2.1 Segment Disk Size Options

The following disk options are recommended:

Disk Size and Type Usable Space in Segment Node

RAID-5 RAID-10

300GB 6g 10k SAS 1012GB 549GB

450GB 6g 10k SAS 1592GB 896GB

600GB 6g 10k SAS 2181GB 1244GB

1.5.3 ETL Nodes

The Bill of Materials for the ETL Node configuration is listed in Appendix A3.

1.5.3.1 ETL Node Disk Size Options

The following disk options are recommended:

Disk Size and Type Usable Space in Segment Node

RAID-5 RAID-10

300GB 6g 10k SAS 1012GB 549GB

450GB 6g 10k SAS 1592GB 896GB

600GB 6g 10k SAS 2181GB 1244GB


Page 7

1.6 Networking

1.6.1 Administration Network

The administration network requires one CAT6 cable for each server in the cluster. These should be

orange. This only accounts for the cables in the racks; cables that would span racks need to be

addressed separately.

All servers including master, segment, and ETL nodes must be counted to determine the right

administration network switch count. The table shows the administration network requirements by

server count.

Management Connection Count (includes all devices in the cluster

with network management)

Administration Switch Requirements

Dell

Up to 21 segment and 4 ETL nodes 1 x Dell 6248

22 to 49 segment and 5 to 8 ETL nodes

2 x Dell 6248

When more than one switch is used, the switches must be cross-connected to ensure console access

through the master nodes.

1.6.1.1 Cables

The number of cables required for a solution varies according to the options selected. In general, each

server and switch installed will use one cable for the administration network.

The types of cables required are as follows:

Connection Type Description Cable Type Color

Administration

Connection between server NET MGT ports (BMC) and the administration

switches. The connection may include a patch panel.

Category 6 Orange

1.6.2 Interconnect Networks

Interconnect Connection Count (includes all devices in the cluster

with network management)

Administration Switch Requirements

Dell

Up to 22 2 x Dell 6248

23 to 45 4 x Dell 6248

The 1Gb interconnect is formed by four, 1Gb networks. Each node in the cluster is connected to each

network through a separate, 1Gb network interface (four connections per node).

1.6.2.1.1 Patch Panels

The 1Gb interconnect networks use RJ-45 patch panels. An RJ-45 patch panel is a device containing 24

pairs of RJ-45 ports organized back to back. Each pair is used to tie two network cables together. The


Page 8

back side of each pair is used to connect to servers inside the panel’s rack and the front side to

connect to switches outside the panel’s rack. Any connection between server and switch in the same

rack are connected directly.

If a server and switch in a connection are mounted in different racks, the server should connect

through a patch panel mounted in its rack. The switch in the second rack connects directly to the patch

panel in the first. If any ETL option is selected, each rack without an ETL switch requires an additional

patch panel.

Please consult the racking diagrams for information on how many of these panels each rack requires.

1.6.2.1.2 Cables

Each node in the cluster requires four connections to the interconnect networks. Cases where the node

and the switch(es) to which it connects are in the same rack, one cable per connection is used. Cases

where the node and the switch(es) to which it connects are not in the same rack, two cables are used.

The types of cables required are as follows:

Connection Type Description Cable Type Color

Interconnect Connection between server PCIe NICs and

interconnect switches. The connection may include a patch panel.

Category 6 Yellow

1.6.3 ETL Networks

If the total number of nodes including master, segment, and ETL nodes is greater than 46, a separate

ETL network is required. Implementing this network means adding an additional PCIe network card to

each node and using that card to connect each node through an additional set of switches.

Please consult Dell’s support team for assistance.

1.6.4 Customer Network(s)

The customer network will need one connection for each master node (two total) for connectivity to

the database cluster. Additional connections from the customer network cannot be accommodated in

the C2100. Each of these connections is cabled via a gray cable from the server(s) in question to the in-

rack patch panels. Connections from the patch panel to the customer network may be any color.


Page 9

2. Installation This section of the document provides instructions for configuring the servers in the cluster.

2.1 Racking Each configuration requires a specific rack plan. All rack plans can be found in section seven of this

document.

There are single- and multi-rack configurations determined by the number of servers present in the

configuration. A single-rack configuration is one where all the planned equipment fits into one rack.

Multi-rack configurations require two or more racks to accommodate all the planned equipment.

In general, these principles need to be followed:

2.1.1 Rules Applying to all Clusters

The following racking rules must be followed by all configurations.

Prior to racking any hardware, a site survey must be performed to determine what power option is desired, if power cables will be top or bottom of the rack, and whether network switches and patch panels will be top or bottom of the rack.

No more than 16 Dell C2100s in any rack.

The stand-by master node is racked mid-rack in the master rack to make the optical drive more accessible and to keep it close to the KVM tray.

All computers, switches, arrays, and racks must be labeled on both the front and back.

All computers, switches, arrays, and racks must be labeled as described in the section on labels later in this document.

All installed devices must be connected to two or more power distribution units in the rack where the device is installed.

2.1.2 Rules Applying to Single-Rack Clusters

This section contains racking rules that apply to configurations that fit in a single rack. Single-rack configurations are those for which the equipment to be racked does not exceed the power, weight, and space limitations of a single rack.

Single-rack configurations may not deploy any dedicated ETL switches. ETL nodes in the single rack configuration connect through the interconnect switches.

All single-rack configurations are top-mounted switches. If the network cables are to run through the floor, they must be run through the rack.

2.1.3 Rules Applying to Multi-Rack Clusters

This section contains racking rules that apply to configurations requiring more than one rack.

No more than three switches in any rack. This is to control the thickness of incoming cable bundles.

A clear document both listing and diagramming where each cable connects by device port identification should be prepared and delivered for each cluster built.


Page 10

2.2 Network In addition to configuring the network management capabilities of each switch, there are specific

cabling requirements. Please review the panel and rack diagrams in section eight for more information.

2.2.1 General Considerations

All cables must be run according to established cabling standards. Tight bends or crimps should be eliminated.

All cables must be clearly labeled at each end. The label on each end of the cable must trace the path the cable follows between server and switch. This includes: The switch name and port The patch panel name and port, if applicable The server name and port

Please see the example in section on labels in this document.

If both the server and switch in a connection are mounted in the same rack, they are connected directly with no patch panel in between.

If a server and switch in a connection are mounted in different racks, the server may connect through a patch panel mounted in its rack.

Inter-rack connections using CAT6 or fiber cables extend between the server rack’s patch panel and the switch directly.

2.2.2 Customer LAN Connections

Customer LAN connections are gray, CAT6 cables.

All LAN connections are cabled from server to patch panel inside the rack. The customer must provide the cables that connect the patch panels to the customer network infrastructure unless otherwise arranged prior to shipping the configuration.

Master nodes have two connections to the customer LAN.

ETL nodes must have enough network bandwidth to support the load required by the customer.

This table shows the server port to use on each server type for the customer's network.

Host Type Physical RJ-45 Port Network Interface Name

Primary Master Node Built-in port 1 eth0

Secondary Master Node Built-in port 1 eth0

ETL Nodes Built-in ports 1 and 2 eth0 and eth1

NOTE: Linux does not always name NICs consistently. Do not rely on the names listed in this table;

manually verify the name the OS assigns to each port on each server.

Assign IP addresses according to the local requirements.


Page 11

2.2.3 Administration Network

Administration network cables are orange, CAT5e cables.

If there are 22 or fewer servers counting master nodes, segment nodes, and ETL nodes, one 48-port switch is racked in the master rack.

If there are 22 to 59 servers counting master nodes, segment nodes, and ETL nodes two, 48-port admin switches are used with one racked in the master rack and another racked in the second segment rack with the third and fourth interconnect switches.

All servers connect to the administration network through the iBMC ports.

Master servers connect to the administration network through the second on-board NIC to provide access to the administration network.

This table shows the server port to use on each server type for the administration network where # is the server number. For example, sdw34 is given 192.168.0.34 and etl3 is given 192.168.0.203.

Host Type Physical RJ-45 Port Network Interface Name IP Address

Primary Master Node Network management iBMC 192.168.0.254

Built-in Port 3 eth1 192.168.0.241

Secondary Master Node Network management iBMC 192.168.0.253

Built-in Port 3 eth1 192.168.0.242

Segment Nodes Network management iBMC 192.168.0.#

ETL Nodes Network management iBMC 192.168.0.2#



All administration network switches present should be cross connected and all NICs attached to these switches participate in the 192.168.0.0/24 network.

Use the following table to determine the correct IP address for each non-server device:

Device IP Address Port

First Interconnect Switch 192.168.0.211 Net Management if available; 1/48 if not

Second Interconnect Switch 192.168.0.212 Net Management if available; 1/48 if not

Third Interconnect Switch 192.168.0.213 Net Management if available; 1/48 if not

Fourth Interconnect Switch 192.168.0.214 Net Management if available; 1/48 if not

First ETL Switch 192.168.0.221 Net Management if available; 1/48 if not

Second ETL Switch 192.168.0.222 Net Management if available; 1/48 if not

First Administration Switch 192.168.0.231 Net Management if available; 1/48 if not

Second Administration Switch 192.168.0.232 Net Management if available; 1/48 if not

NOTE: Use of 1/48 or 1/24 port for network management requires setup in the switch. Please see

the switch’s vendor-supplied documentation prior to connecting to properly configure this port.

Without configuration, connecting this port will form extraneous physical routes affecting cluster

performance.

Use the following map to determine where to connect cables.


Page 12

First Administration Switch:

19

M20

21

201

202

203

204

221

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

UP

N/A

211

212

254

253

241

242

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

The number in each port indicates which IP address the connected NIC will have on the 192.18.0.0/24

network.

Second Administration Switch:

44

M45

46

47

48

49

205

206

207

208

222

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

UP

N/A

213

214

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

The number in each port indicates which IP address the connected NIC will have on the 192.18.0.0/24

network.

2.2.4 Interconnect Networks

Interconnect network connections are yellow, CAT6 cables.

The first two interconnect switches are always racked in the master rack.

If there are four interconnect switches, the second two switches are racked in the first segment rack (rack number two for the cluster).

Master and segment nodes connect to the interconnect through add-in NICs (not built-in NIC ports).

When using only two gigabit interconnect switches, the first and third ports on the interconnect NIC are cabled to the first switch and the second and fourth ports cabled to the second.

When using four gigabit switches, the first port from the interconnect NIC is cabled to the first switch, second to the second switch, third to the third switch, and fourth to the fourth switch.


Page 13

2.2.4.1 2 x Gigabit Switches

This table shows the server port to use on each server type for the standard interconnect network where # is the server number. For example, sdw34 is given 192.168.1.34 and etl3 is given 192.168.1.203 on the first interconnect network.

Host Type Physical RJ-45 Port Network Interface

Destination Switch IP Address

Primary Master

1st port on PCIe card eth4 1st Interconnect

Switch 192.168.1.254

2nd port on PCIe card eth5 2nd Interconnect

Switch 192.168.2.254

3rd port on PCIe card eth6 1st Interconnect

Switch 192.168.3.254

4th port on PCIe card eth7 2nd Interconnect

Switch 192.168.4.254

Secondary Master

1st port on PCIe card eth4 1st Interconnect Switch

192.168.1.253


Switch 192.168.2.253


Switch 192.168.3.253


Switch 192.168.4.253

Segment

1st port on PCIe card eth4 1st Interconnect Switch

192.168.1.#


Switch 192.168.2.#


Switch 192.168.3.#


Switch 192.168.4.#

ETL


Switch 192.168.1.2#


Switch 192.168.2.2#


Switch 192.168.3.2#


Switch 192.168.4.2#



The first interconnect switch should be attached to NICs that participate in the 192.168.1.0/24 and 192.168.3.0/24 networks. The second switch should service the 192.168.2.0/24 and 192.168.4.0/24 networks.

The following diagrams show which ports are used by which NICs.


Page 14

First Interconnect Switch

3.

254

F F3.

253

3.

1

3.

2

3.

3

3.

4

3.

5

3.

6

3.

7

3.

8

3.

9

3.

10

3.

11

3.

12

3.

13

3.

14

3.

15

3.

16

3.

17

3.

18

3.

19

3.

20

3.

21

AD

M

1.

254

1.

253

1.

1

1.

2

1.

3

1.

4

1.

5

1.

6

1.

7

1.

8

1.

9

1.

10

1.

11

1.

12

1.

13

1.

14

1.

15

1.

16

1.

17

1.

18

1.

19

1.

20

1.

21

N/A

F F

The number in each port indicates which IP address the connected NIC will have on its interconnect

network. For example, the port labeled 1.6 is connected to the NIC with the 192.168.1.6 address.

Second Interconnect Switch

4.

254

F F4.

253

4.

1

4.

2

4.

3

4.

4

4.

5

4.

6

4.

7

4.

8

4.

9

4.

10

4.

11

4.

12

4.

13

4.

14

4.

15

4.

16

4.

17

4.

18

4.

19

4.

20

4.

21

AD

M

2.

254

2.

253

2.

1

2.

2

2.

3

2.

4

2.

5

2.

6

2.

7

2.

8

2.

9

2.

10

2.

11

2.

12

2.

13

2.

14

2.

15

2.

16

2.

17

2.

18

2.

19

2.

20

2.

21

N/A

F F



NOTE: No ports are shown for ETL nodes. These should be put starting in the last port in the top

row of each switch working backwards. If there are more connections to make than there are

ports when ETL nodes are added, increase to four switches.


Page 15

2.2.4.2 4 x Gigabit Switches

This table shows the server port to use on each server type for the extended interconnect network where # is the server number. For example, sdw34 is given 192.168.1.34 and etl3 is given 192.168.1.203 on the first interconnect network.

Host Type Physical RJ-45 Port Network Interface

Destination Switch IP Address

Primary Master


Switch 192.168.1.254


Switch 192.168.2.254

3rd port on PCIe card eth6 3rd Interconnect

Switch 192.168.3.254

4th port on PCIe card eth7 4th Interconnect

Switch 192.168.4.254

Secondary Master


Switch 192.168.1.253


Switch 192.168.2.253


Switch 192.168.3.253


Switch 192.168.4.253

Segment


Switch 192.168.1.#


Switch 192.168.2.#


Switch 192.168.3.#


Switch 192.168.4.#

ETL


Switch 192.168.1.2#


Switch 192.168.2.2#


Switch 192.168.3.2#


Switch 192.168.4.2#



The first interconnect switch should service the 192.168.1.0/24 network. The second switch should service the 192.168.3.0/24 network. The third switch should service the 192.168.2.0/24 network. Finally, the fourth switch should service the 192.168.4.0/24 network.

The following diagrams show which ports are used by which NICs.


Page 16

First Interconnect Switch

1.

22

F F1.

23

1.

24

1.

25

1.

26

1.

27

1.

28

1.

29

1.

30

1.

31

1.

32

1.

33

1.

34

1.

35

1.

36

1.

37

1.

38

1.

39

1.

40

1.

41

1.

42

1.

43

1.

44

AD

M

1.

254

1.

253

1.

1

1.

2

1.

3

1.

4

1.

5

1.

6

1.

7

1.

8

1.

9

1.

10

1.

11

1.

12

1.

13

1.

14

1.

15

1.

16

1.

17

1.

18

1.

19

1.

20

1.

21

N/A

F F



Second Interconnect Switch

2.

22

F F2.

23

2.

24

2.

25

2.

26

2.

27

2.

28

2.

29

2.

30

2.

31

2.

32

2.

33

2.

34

2.

35

2.

36

2.

37

2.

38

2.

39

2.

40

2.

41

2.

42

2.

43

2.

44

AD

M

2.

254

2.

253

2.

1

2.

2

2.

3

2.

4

2.

5

2.

6

2.

7

2.

8

2.

9

2.

10

2.

11

2.

12

2.

13

2.

14

2.

15

2.

16

2.

17

2.

18

2.

19

2.

20

2.

21

N/A

F F



Third Interconnect Switch

3.

22

F F3.

23

3.

24

3.

25

3.

26

3.

27

3.

28

3.

29

3.

30

3.

31

3.

32

3.

33

3.

34

3.

35

3.

36

3.

37

3.

38

3.

39

3.

40

3.

41

3.

42

3.

43

3.

44

AD

M

3.

254

3.

253

3.

1

3.

2

3.

3

3.

4

3.

5

3.

6

3.

7

3.

8

3.

9

3.

10

3.

11

3.

12

3.

13

3.

14

3.

15

3.

16

3.

17

3.

18

3.

19

3.

20

3.

21

N/A

F F



Fourth Interconnect Switch

4.

22

F F4.

23

4.

24

4.

25

4.

26

4.

27

4.

28

4.

29

4.

30

4.

31

4.

32

4.

33

4.

34

4.

35

4.

36

4.

37

4.

38

4.

39

4.

40

4.

41

4.

42

4.

43

4.

44

AD

M

4.

254

4.

253

4.

1

4.

2

4.

3

4.

4

4.

5

4.

6

4.

7

4.

8

4.

9

4.

10

4.

11

4.

12

4.

13

4.

14

4.

15

4.

16

4.

17

4.

18

4.

19

4.

20

4.

21

N/A

F F



NOTE: No ports are shown for ETL nodes. These should be put starting in the last port in the top

row of each switch working backwards. If there are more connections to make than there are

ports when ETL nodes are added, implement a dedicated ETL network.


Page 17

2.3 Servers

2.3.1 iBMC Configuration

The iBMC is a network-enabled device installed in each server allowing for remote control of the

server. Console access, SNMP monitoring, and more are available through the iBMC. There are two ways

to configure the network properties of the iBMC. Choose the one that is most convenient.

2.3.1.1 Configuring iBMC IP Address in BIOS

The iBMC’s network configuration can be performed in the server’s BIOS setup utility. Do the following:

- Press F2 during the Power-On Self Test (POST).

- Go to the Server menu and select Set BMC LAN Configuration.

- Set the network as shown. Use a correct iBMC IP address and subnet mask

appropriate to the network in use. To change a value, use the arrow keys to make

the field to change current and then the space bar to open the field for editing.


Page 18

Press Return to save the change.

- When all changes are made, Press F10 so save the changes and exit the BIOS Setup

Utility.


Page 19

2.3.1.2 Port Redirection

Out-of-band connectivity to the iBMC is done through ipmitool. The iBMC does not support ssh or telnet

connections. To enable ipmitool access to the iBMC and the console, make the following changes in the

BIOS setup utility:

- During the POST, press F2 to access the BIOS Setup Utility.

- Once in the utility, go to the Server menu.

- Set the fields as shown and press F-10 to exit and save the changes.


Page 20

2.3.1.3 Connecting to the BMC – IPMITOOL Connection

Connections to the BMC can be formed with the ipmitool from another server. In this way, it is possible

to access the server console and perform tasks during the server POST without having to browse to the

BMC.

To make a console connection to the server’s BMC, do the following:

Log into another server where ipmitool is available. Some ipmitool implementations do not work for this purpose but ipmitool as delivered with Red Hat or SuSE do work. If this connection is via ssh, use the –e ^ switch to change the escape key sequence from “~.” to “^.”. The escape sequence for ipmitool is also “~.”. If the ssh sequence is not remapped, closing the ipmitool session will drop the ssh connection rather than the ipmitool session.

Issue the following command: ipmitool -I lanplus -H [ip address of the target BMC] -U root -P root sol activate The session will immediately start. If the OS is configured to present a login on ttyS1, there will be a login prompt. If the server is rebooted, the POST will appear in this session.

To leave, press ~. As the first characters in a line. This will terminate the ipmitool session. If the connection to the server running ipmitool is via ssh and the escape sequence was not remapped, this will drop the ssh session to the server and may leave the ipmitool session active with no connection to a terminal.

HTTP Connection

Connections to the BMC are formed by browsing to the BMC address. Use ssh to the master node with

the following ssh command:

ssh -L 8000:[ip address or hostname of BMC desired]:443 root@mdw

Once the ssh session is connected, open a browser and browse to the following URL:

https://localhost:8000/login.html

A separate ssh connection using a different port (8001, 8002, etc.) can be made if connection to more

than one iBMC at a time is desired. Be sure the local port used is not already in use.


Page 21

2.3.2 Hard Disk Configuration – MASTER Node

Master nodes have seven, hot swappable disks. Configure six of these into a single, RAID-5 stripe set

with the following properties:

128k stripe width

Adaptive Read-ahead

Disk Cache Disabled

Cached I/O

This needs to be configured in the RAID card’s option ROM. Do the following:

- During the POST, press CTRL-H to enter the option ROM on the RAID card when

prompted.

- When the option ROM starts, select the correct adapter (typically there is only one)

and press the Start button.


Page 22

- Select the Configuration Wizard link.

NOTE: The image shown includes more disks than would be normal in the master

node.

- Select New Configuration.

This stripe set will be reported to the OS as /dev/sda. During the OS install,

partition this disk according to the relevant install guide standards.


Page 23

- Answer Yes to the confirmation question.

- Select the Manual Configuration option.

- Select the six drives to include in the stripe set and click Add To Array.

- Click the Accept DG button to save the disk group.


Page 24

- Click the Next button to go to the next screen.

- Add the current disk group to the span.

- Click Next to go to the next screen, Choose RAID5 for RAID Level and set the stripe

width to 128 KB as shown.


Page 25

- Click Accept to save the virtual drive and answer Yes to the confirmation question.

- Click Next to go to the next screen.


Page 26

- Click Accept to save the configuration and answer Yes to the confirmation

questions.

- Set VD0 to be the bootable disk and click Go.


Page 27

- Click Home.

- Select the remaining unconfigured disk (the image shown includes more disks than

would be present in a master node) and open its properties. Select the Make

Dedicated HSP property to designate the disk a hot spare.

- Click the Go button to save the change.

- Click the Home button and exit the utility.


Page 28

2.3.3 Hard Disk Configuration – Segment and ETL Nodes (using RAID Controller BIOS)

Use this process to configure the data disks in the RAID card’s option ROM. This can be done prior to

installing and operating system:

- During the POST, press CTRL-H to enter the option ROM on the RAID card when

prompted.

- When the option ROM starts, select the correct adapter (typically there is only one)

and press the Start button.

- Select the Configuration Wizard link.


Page 29

- Select New Configuration.

This stripe set will be reported to the OS as /dev/sda. During the OS install,

partition this disk according to the relevant install guide standards.

- Answer Yes to the confirmation question.

- Select the Manual Configuration option.


Page 30

- Select the six drives to include in the first stripe set and click Add To Array.

- Click the Accept DG button to save the disk group.

- Select the next six drives to include in the second stripe set and click Add To

Array.


Page 31

- Click Accept DG and then Next.

- Add the first disk group to the SPAN and click Next.


Page 32

- Change the properties as shown, click Accept, and Yes on the confirmation screen.


Page 33

- Click the Back button to return to the SPAN screen. Add the second disk group to

the SPAN.



Page 34

Set the virtual disk properties as shown, click Accept, then Yes on the confirmation

screen.


Page 35


- Click Accept and Yes on the confirmation screens.

- Click Home and exit the utility.


Page 36

2.3.4 Hard Disk Configuration – Segment and ETL Nodes (using MegaCLI)

Segment and ETL nodes have 12 hot-swappable disks.

The disks should be configured into two disk groups of six disks each. One virtual disk encompassing the

entire disk group should be configured on each.

NOTE: Skip this procedure if you configured the hard disks using the RAID Controller BIOS

described in the previous section.

This process can be performed with the MegaCLI utility. This process requires a running operating

system or the use of the RAID card’s Pre-Boot CLI environment:

First, determine how the server identifies everything. It is important to determine the

enclosure number and the physical disk slot numbers. These are not consistent server

to server. Typically, internal disks do not have an enclosure number but this is not

consistent. Use the following commands to determine this information:

MegaCli64 -EncInfo -a0 | grep -e Enclosure -e “Device ID”

This command lists all the attached enclosures on the first card. The output

will be something like this on a C2100 without any external disks:

Enclosure 0:

Device ID : 252

In this case, the Enclosure number is either nothing or 252. The correct

value is determined later.

MegaCli64 -PDList -a0 | grep -e ^Slot -e “^Device Id”

This command lists the slot and device IDs for all the disks attached to the

first card. The output will be something like this on a C2100 without any

external disks:

Slot Number: 8

Device ID: 8

Slot Number: 9

Device ID: 9

Slot Number: 10

Device ID: 10

Slot Number: 11

Device ID: 11

Slot Number: 12

Device ID: 12

Slot Number: 13

Device ID: 13

Slot Number: 14

Device ID: 14

Slot Number: 15

Device ID: 15

Slot Number: 16


Page 37

Device ID: 16

Slot Number: 17

Device ID: 17

Slot Number: 18

Device ID: 18

Slot Number: 19

Device ID: 19

This indicates that the disk slots are numbered started at 8 and

ending at 19.

MegaCli64 -PDInfo -PhysDrv[:19] -a0 | grep Slot

MegaCli64 -PDInfo -PhysDrv[252:19] -a0 | grep Slot

These commands show the details for a specific physical drive (the example

shows 19 since 19 is in the list from the previous command). The specifier

for the drive is Enclosure:Slot. In the first step, the enclosure’s ID was

identified as 252. These commands determine if the ID shown works in

commands or if the correct specifier for the enclosure is no value.

Successful output from these commands is:

Slot Number: 19

The command that succeeds is an example of the correct disk specification

for the next commands.

Next, list out the disk layouts. Typically, the first six disks will be in the first group and

the second six disks in the next. Using the previous example and assuming the correct

enclosure specification is no value, the raid groups would be:

First group: [:8,:9,:10,:11,:12,:13]

Second group: [:14,:15,:16,:17,:18,:19]

Finally, execute the commands to create the virtual disks:

MegaCli64 -CfgLdAdd –R5[:8,:9,:10,:11,:12,:13] WB ADRA Cached

NoCachedBadBBU -strpsz128 -a0

MegaCli64 -CfgLdAdd –R5[:14,:15,:16,:17,:18,:19] WB ADRA Cached

NoCachedBadBBU -strpsz128 -a0

These commands create two virtual disks. Each will be made up of six drives, use RAID-

5, write-back when the battery is good, adaptive read ahead, and a stripe width of

128k. These should appear in the OS as /dev/sdb and /dev/sdc.


Page 38

2.3.5 Firmware

2.3.5.1 Server

Apply the most recent firmware available from Dell for the C2100. This material is available from Dell

Support.

2.3.5.2 Disk Controller

Make sure that the RAID card has at least firmware version 2.30.03-0775 which is distributed by LSI as

part of FW Package 12.3.0-0022. The latest firmware is available at this URL:

http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/6gb_s_value_line/sas9

260-8i/index.html.

3. Validation Most of the validation effort is performed after the OS is installed and a variety of OS-level tools are

available. A checklist is included in the relevant OS installation guide that should be separately printed

and signed for delivery and includes the issues raised in this section.

At this stage, the following items should be examined and verified:

All devices are installed in positions consistent with the racking diagrams in this document.

All cables are labeled according to the standards in this document.

All servers are labeled according to the standards in this document.

All switches and patch panels are labeled according to the standards in this document.

All racks are labeled according to the standards in this document.

All devices power on.

All hot-swappable devices are properly seated.

No devices show any warning or fault lights.

All network management ports are accessible via the administration LAN.

All cables are neatly dressed into the racks and have no sharp bends or crimps.

All rack doors and covers are installed and close properly.

All servers extend and retract without pinching or stretching cables.

http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/6gb_s_value_line/sas9260-8i/index.htmlhttp://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/6gb_s_value_line/sas9260-8i/index.html


Page 39

4. Labels

4.1 Racks Each rack in a reference configuration is labeled at the top of the rack and on both the front and back.

Racks are named Master Rack or Segment Rack # where # is a sequential number starting at 1. A rack

label would look like this:

MASTER RACK SEGMENT RACK 1

4.2 Servers Each server is labeled on both the front and back of the server. The label should be the base name of

the server’s network interfaces. In other words, if a segment node is known as sdw15-1, sdw15-2,

sdw15-3, and sdw15-4, the label on that server would be sdw15.

sdw15

4.3 Switches Switches are labeled according to their purpose. Interconnect switches are i-sw, administration

switches are a-sw, and ETL switches are e-sw. Each switch is assigned a number starting at 1. Switches

are labeled on the front of the switch only because the back is generally not visible when racked.

i-sw-1

4.4 Patch Panels Each rack may have as many as three patch panels. The panels are named panel and use a letter (A, B,

or C) to differentiate within a rack. Each panel is labeled on its face only.

Single-rack configurations have no patch panels. Multi-rack configurations without the ETL option

selected may have a panel-A or both panel-A and panel-B. When the ETL option is added, panel-C is

added. In racks with only panel-A, adding the ETL option still adds panel-C so that those racks have

panel-A and panel-C.

panel-A

4.5 Network Cables

4.5.1 CAT6

CAT6 cables are used for the 1Gb interconnect. When using this interconnect, each system has

between three and nine connections depending on which system and which options are configured. In

addition, the connections flow through patch panels. This means any one connection between server

and switch may have two cables and four connection points.

CAT6 cables are large enough around that labels should be wrapped around the cable without leaving a

tab hanging off the cable. Tabs tend to tear off and become trapped in device fans. Also, tabs cluster

up near the switches leaving a messy appearance. Be sure that the top and bottom of the label overlap


Page 40

a small amount to ensure that the label will remain affixed to the cable. PTouch labels at least ¾-inch

high are acceptable; ½-inch PTouch do not stay on cables and cannot be used.

Each network cable is labeled at each end with enough information to trace the connection both

directions. The cable label needs the following elements:

The server identifier

A server is identified by the rack name where it is installed, the node name of

the server, and the interface number the current connection uses. For

example, the fully qualified server identifier for a connection from the net1

interface (nge1) in sdw10, a server racked in rack 2 would be:

rack-2.sdw10-2

The patch panel identifier

The unique name of a patch panel includes the rack name where the panel is

installed, the panel name and, the port used for the connection. There are

between one and three panels in each rack and each panel has 24 ports.

Therefore, a fully qualified patch panel name for the 10th port in the second

panel in the second rack would be:

rack-2.panel-B.10

For connections that go straight from server to switch leave this identifier blank

on the label.

Switch identifier

A switch identifier includes the rack where the switch is installed, the switch

name, and the port number used in the switch for the current connection. The

switch name indicates what kind of switch is in use for the connection; “a” for

Administration, “I” for Interconnect, and “e” for ETL. For example, a fully

qualified switch identifier for the 36th port in the second interconnect switch

installed in the first rack would be:

rack-1.i-sw-2.36

So, a label for a connection from sdw15’s net0 port to the first interconnect switch’s number 37 port

would look as follows if sdw15 is installed in the third rack and uses the 15th port in the first panel in

the third rack:

rack-3.sdw15-1

rack-3.panel-1.15

rack-1.i-sw-1.37

In the example, there are two cables connecting sdw15-1 and i-sw-1.37. Theses cables meet at rack-

3.panel-A.15. This means there are four labels, one at each end of each cable and all identical to the

example.


Page 41

5. Physical Installation Details

5.1 Option Concerns

5.1.1 Additional Segment Nodes

Each additional segment node has the same connections and place in the diagrams as sdw2.

When the 21st segment node is added and the 1Gb interconnect is used, the configuration requires four

Gig-E switches, one for each Class C IP range defined to the interconnect LAN.

In addition, adding the 23rd host of any type requires the use of an additional 48-port 10/100 switch for

use as the administration LAN. If two administration switches are present, they have to be cross-

connected to permit full access to all connected consoles.

5.1.2 Additional ETL Nodes and ETL Network

ETL networks are not supported on single rack configurations. A single rack configuration can include

one to four ETL nodes each of which connect the ETL nodes directly to the interconnect network.

The standard configuration allows for up to two ETL nodes for every 10 segment nodes. The maximum

supported ETL nodes is eight. Each additional ETL node has the same connections and place in the

diagrams as etl1.

Configurations where the segment to ETL node ratio is greater than 2 to 1, the segment nodes are

connected to the ETL network only once each. The ETL nodes can send data at no faster rate than

500MB/s. When the ratio is more than 2 segment nodes for each ETL node, no single segment node

needs more than 1Gb/s bandwidth to receive its share of the ETL nodes’ output.

Care must be taken to account for ETL node network management ports on the administration LAN.

The standard administration LAN is a 48-port, 10/100 switch. Typically, there is room to add a few ETL

nodes to the configuration and use the same switch. However, if the number of ETL nodes plus the

number of all the hosts in the configuration (administration, master, secondary master, and segment)

totals 48 or more, additional switches are required.

5.1.3 Top- or Bottom-Mounted Options

The rack power cables, network switches, and patch panels can be mounted in the top or bottom of

the racks. This option does not affect the network drawings but does change the rack layouts.

5.1.4 Interconnect Network Type

The drawings show two interconnect switches. In some cases using the 1Gb interconnect, more than

two switches may be deployed. This does not change the number of logical networks used or how they

are deployed.

5.1.5 Segment Node Disk Type

This option addresses the individual disk size in Dell C2100s and has no bearing whatsoever on the

network drawings.

5.2 Racking There are four distinct rack layouts applicable to the reference configuration. Which ones are used in

any cluster depends on the number and types of hosts included in the configuration.


Page 42

5.2.1 Single-Rack Cluster

A single rack cluster is one with only a master rack. Only the top-mounted master rack configuration is

supported in a single-rack cluster.

NOTE: Always mount switches and patch panels rear-facing. Shown here front facing to account for

each device’s position.

5.2.1.1 Network Connections

The following tables show the number of connections each host consumes in the single rack cluster

along with the connection type (direct to switch or patch panel). To determine the total number of

connections present in the solution, multiply the number of each host type (master and segment) by


Page 43

the column totals in the table.

Please note that no connection information is given for dedicated ETL networks. These are custom

designed on a case-by-case basis. Please consult with the Dell support team for more information on

how these are implemented.

5.2.1.1.1 1Gb Interconnect

Master Node (Primary or Secondary) Segment Node

Networks Direct to Switch

Patch Panel Direct to Switch

Patch Panel

Customer LAN 0 1 0 0

Administration 2 0 1 0

Interconnect 4 0 4 0

5.2.1.1.2 RJ-45 Patch Panel Maps

1Gb Interconnect, 2 to 14 Segment Nodes

Use this map to determine how to use the ports in the rack’s patch panel. This panel applies to 1Gb

networks. Two panels are used to preserve room in the rack to expand to a multi-rack solution.

N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

mdw

to

LAN

smdw

to

LAN


The connections labeled with mdw relate to the primary master and those with smdw relate to the

secondary master.

5.2.2 Multiple Racks

Clusters with more than one rack or that are expected to grow to more than one rack are considered

multi-rack configurations.

5.2.2.1 Master Rack

The master rack in a multi-rack configuration is the same as a single-rack configuration. However, a

master rack may use 48 or 52 port 10Gb switches when using the 10Gb interconnect. Also, master racks

may have the switches mounted in the bottom.

NOTE: The bottom-mount option is not shown. When using bottom mount, move all switches and

panels to the bottom of the rack and move all the other devices up.


Page 44


Page 45

5.2.2.1.1 Network Connections









Page 46

5.2.2.1.1.1 1Gb Interconnect, 2 – 21 Segment Nodes

Master Node (Primary or Secondary) Segment Node

Networks Direct to Switch

Patch Panel Direct to Switch

Patch Panel




5.2.2.1.1.2 1Gb Interconnect, 22 - 44 Segment Nodes

Master Node (primary or secondary) Segment Node

Networks Direct to

Switch Patch Panel

Direct to

Switch Patch Panel




5.2.2.1.1.3 RJ-45 Patch Panel Port Maps

Use this map to determine how to use the ports in the rack’s patch panel.

1Gb Interconnect, 2 to 21 Segment Nodes:


mdw

to

LAN

smdw

to

LAN


The connections labeled with mdw relate to the primary master and those with

smdw relate to the secondary master.

1Gb Interconnect, 22 to 44 Segment Nodes:

3.

11

3.

12

3.

13

3.

14N/A N/A N/A N/A N/A N/A

3.

254

3.

253

3.

1

3.

2

3.

3

3.

4

3.

5

3.

6

3.

7

3.

8

3.

9

3.

10

mdw

to

LAN

smdw

to

LAN

4.

11

4.

12

4.

13

4.

14N/A N/A N/A N/A N/A N/A

4.

254

4.

253

4.

1

4.

2

4.

3

4.

4

4.

5

4.

6

4.

7

4.

8

4.

9

4.

10

The connections labeled with mdw relate to the primary master and those with

smdw relate to the secondary master.


Page 47



5.2.2.2 First Segment Rack

The first segment rack contains the second two interconnect switches when using four interconnect

switches. These two switches are not necessary until the 22nd segment node is added to the cluster.

The space for these switches is reserved in the second segment rack since this rack could potentially

hold the 22nd segment node.

When the 22nd segment node is added, the switches are installed into the first segment rack and the

racks re-cabled to redistribute the network connections.

In addition, the second administration switch will be placed into this rack when needed.



Page 48








5.2.2.2.1.1 1Gb Interconnect, 2 to 21 Segment Nodes

Networks

Segment Node

Direct to Switch Patch Panel

Customer LAN 0 0

Administration 0 1

Interconnect 0 4

5.2.2.2.1.2 1Gb Interconnect, 22 to 44 Segment Nodes

Networks

Segment Node


Customer LAN 0 0

Administration 1 0

Interconnect 2 2


1 Gb Interconnect, 2 to 21 Segment Nodes

3..

20

3..

21N/A N/A

0.

15

0.

16

0.

17

0.

18

0.

19

0.

20

0.

21

0.

22

1.

15

1.

16

1.

17

1.

18

1.

19

1.

20

1.

21

3..

15

3..

16

3..

17

3..

18

3..

19

4.

20

4.

21N/A N/A

0.

23

0.

24

0.

25

0.

26

0.

27

0.

28

0.

29

0.

30

2.

15

2.

16

2.

17

2.

18

2.

19

2.

20

2.

21

4.

15

4.

16

4.

17

4.

18

4.

19

1 Gb Interconnect, 22 to 44 Segment Nodes

1.

27

1.

28

1.

29

1.

30

0.

15

0.

16

0.

17

0.

18

0.

19

0.

20

0.

21

0.

22

1.

15

1.

16

1.

17

1.

18

1.

19

1.

20

1.

21

1.

22

1.

23

1.

24

1.

25

1.

26

2.

27

2.

28

2.

29

2.

30

0.

23

0.

24

0.

25

0.

26

0.

27

0.

28

0.

29

0.

30

2.

15

2.

16

2.

17

2.

18

2.

19

2.

20

2.

21

2.

22

2.

23

2.

24

2.

25

2.

26


Page 49

5.2.2.3 Second Segment Rack

The second segment rack contains only segment or ETL nodes; no switches or other servers.


Page 50

5.2.2.3.1.1 1Gb Interconnect, Any Number of Segment Nodes

Networks

Segment Node


Customer LAN 0 0

Administration 0 1

Interconnect 0 4


1 Gb Interconnect, Any Number of Segment Nodes:

1.

43

1.

44

1.

45

0.

31

0.

32

0.

33

0.

34

0.

35

0.

36

0.

37

0.

38N/A

1.

31

1.

32

1.

33

1.

34

1.

35

1.

36

1.

37

1.

38

1.

39

1.

40

1.

41

1.

42

2.

43

2.

44

2.

45

0.

39

0.

40

0.

41

0.

42

0.

43

0.

44

0.

45N/A N/A

2.

31

2.

32

2.

33

2.

34

2.

35

2.

36

2.

37

2.

38

2.

39

2.

40

2.

41

2.

42

3.

43

3.

44

3.

45N/A N/A N/A N/A N/A N/A N/A N/A N/A

3.

31

3.

32

3.

33

3.

34

3.

35

3.

36

3.

37

3.

38

3.

39

3.

40

3.

41

3.

42

4.

43

4.

44

4.

45N/A N/A N/A N/A N/A N/A N/A N/A N/A

4.

31

4.

32

4.

33

4.

34

4.

35

4.

36

4.

37

4.

38

4.

39

4.

40

4.

41

4.

42


Page 51

5.2.2.4 Third Segment Rack

This rack contains only segment nodes; no switches or other servers.


The following tables show the number of connections each host consumes in the single-rack cluster








Page 52

5.2.2.4.1.1 1Gb Interconnect, Any Number of Segment Nodes

Networks

Segment Node


Customer LAN 0 0

Administration 0 1

Interconnect 0 4


10 Gb Interconnect, Any Number of Segment Nodes:

N/A N/A N/A N/A N/A0.

46

0.

47

0.

48

0.

49N/A N/A N/AN/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

5.3 Network Diagrams The following diagrams show network topographies for the Greenplum Data Warehouse cluster. These

diagrams do not show all the potential options, only those affecting network complexity.

5.3.1 1 Gb Interconnect

The following diagrams show examples of clusters using 1Gb interconnect links.


Page 53

5.3.1.1 Minimum Cluster

The minimum configuration for the Greenplum Data Warehouse appliance consists of three network

switches, two dedicated master nodes (primary and secondary), and two segment nodes. Two of the

switches are Gigabit Ethernet switches and the third is a 10/100 switch. Data has to be loaded through

the master nodes in this configuration.

192.168.4.0/24

Site Network Infrastructure

Interconnect Switch #1

48-port, 10/100/1000 Switch

192.168.1.0/24 192.168.3.0/24


48-port, 10/100/1000 Switch

192.168.2.0/24

Administration Network

24-port, 10/100 Switch

192.168.0.0/24

MDW Primary Master

C2100

iBMC 192.168.0.254

eth0 Local IP

192.168.1.254 eth4 eth1 192.168.0.203

192.168.2.254 eth5

192.168.3.254 eth6

192.168.4.254 eth7

SMDW Secondary Master

C2100

iBMC 192.168.0.253

eth0 Local IP

192.168.1.253 eth4 eth1 192.168.0.202

192.168.2.253 eth5

192.168.3.253 eth6

192.168.4.253 eth7

SDW1 Segment

C2100

iBMC 192.168.0.1

192.168.1.1 eth4

192.168.2.1 eth5

192.168.3.1 eth6

192.168.4.1 eth7

SDW2 Segment

C2100

iBMC 192.168.0.2

192.168.1.2 eth4

192.168.2.2 eth5

192.168.3.2 eth6

192.168.4.2 eth7

192.168.4.0/24


Page 54

5.3.1.2 Minimum Cluster with ETL Access

This drawing shows the minimum configuration including ETL capacity. In this case, additional network

interfaces are put into the segment nodes and used to connect to the customer network. Data is loaded

from existing servers over these interfaces.



48-port, 10/100/1000 Switch

192.168.1.0/24 192.168.3.0/24


48-port, 10/100/1000 Switch

192.168.2.0/24 192.168.4.0/24



192.168.0.0/24

MDW Primary Master

C2100

iBMC 192.168.0.254

eth0 Local IP

192.168.1.254 eth4 eth1 192.168.0.203

192.168.2.254 eth5

192.168.3.254 eth6

192.168.4.254 eth7


C2100

iBMC 192.168.0.253

Eth0 Local IP

192.168.1.253 eth4 eth1 192.168.0.202

192.168.2.253 eth5

192.168.3.253 eth6

192.168.4.253 eth7

SDW1 Segment

C2100

iBMC 192.168.0.1

192.168.1.1 eth4

192.168.2.1 eth5

192.168.3.1 eth6

192.168.4.1 eth7

SDW2 Segment

C2100

iBMC 192.168.0.2

192.168.1.2 eth4

192.168.2.2 eth5

192.168.3.2 eth6

192.168.4.2 eth7

eth0 Local IP

eth1 Local IP

eth0 Local IP

eth1 Local IP


Page 55

5.3.1.3 Minimum Configuration With ETL Server

This drawing shows the minimum configuration including an ETL server. This example builds on the last

by bringing the ETL server onto the interconnect network. In the previous diagram, the cluster was

opened up to the customer LAN to provide access to source data. In this case, the server with source

data is connected to the interconnect network. This is preferred because it is more secure and less

complicated.



48-port, 10/100/1000 Switch

192.168.1.0/24 192.168.3.0/24


48-port, 10/100/1000 Switch

192.168.2.0/24 192.168.4.0/24



192.168.0.0/24

MDW Primary Master

C2100

iBMC 192.168.0.254

eth0 Local IP

192.168.1.254 eth4 eth1 192.168.0.203

192.168.2.254 eth5

192.168.3.254 eth6

192.168.4.254 eth7


C2100

iBMC 192.168.0.253

eth0 Local IP

192.168.1.253 eth4 eth1 192.168.0.202

192.168.2.253 eth5

192.168.3.253 eth6

192.168.4.253 eth7

SDW1 Segment

C2100

iBMC 192.168.0.1

192.168.1.1 eth4

192.168.2.1 eth5

192.168.3.1 eth6

192.168.4.1 eth7

SDW2 Segment

C2100

iBMC 192.168.0.2

192.168.1.2 eth4

192.168.2.2 eth5

192.168.3.2 eth6

192.168.4.2 eth7

Customer Data Sourcet This is an existing system on the customer network that is

essentially an ETL node

Lan Connection(s)

192.168.1.46

192.168.2.46

192.168.3.46

192.168.4.46

The ETL server need not be an existing customer server; it can be a new server included in the cluster

for this purpose. It is important to note that this configuration needs to carefully account for the

number of network connections consumed by ETL servers. If the total number of servers on the

interconnect, including ETL, exceeds 21, four interconnect switches are required. If the number

exceeds 46, ETL servers need to be put on a dedicated network that includes the segment nodes from

the cluster but is not the interconnect.


Page 56

Appendix A: Bill of Materials

Appendix A1: Master Node Bill of Materials

Base Unit: PowerEdge C2100 Expander Backplane support for 3.5" Hard Drives Redundant

Power Supplies (224-8350)

Processor: Intel Xeon X5670, 2.93Ghz, 12M Cache, 5.86 GT/s QPI, Turbo, HT (317-4017)

Processor: Thermal Heatsink,CPU,C2100 (317-3934)

Processor: Thermal Heatsink Front,C2100 (317-3935)


Memory: 48GB Memory (12x4GB), 1333MHz Dual Ranked RDIMMs for 2 Processors, Optimized (317-3394)

Operating System: No OS, No Utility Partition (420-3323)

NIC: Intel Gigabit ET Quad Port 1GbE, PCIe x4 (430-0771)

Documentation Diskette:

C2100 Documentation (330-8774)

Additional Storage Products:

Hard Drive Carrier,3.5,1-12PCS,C2100 (342-0981)








300GB 15K RPM Serial-Attach SCSI 6Gbps 3.5in Hot Plug Hard Drive (342-1542)







Feature Add-in LSI 9260-8i controllers for up to 12 HP Drives total (342-0993)

Feature LSI 9260-8i SAS/SATA Card (342-1529)

Feature C2100 Sliding Rail Kit (330-8520)

Service: Dell Hardware Limited Warranty Initial Year (909-1677)

Service: Dell Hardware Limited Warranty Extended Year (909-1668)

Service: Pro Support for IT: Next Business Day Onsite Service After Problem Diagnosis, Initial Year (926-4080)

Service: Pro Support for IT: Next Business Day Onsite Service After Problem Diagnosis, 2Year Extended (923-2322)

Service: ProSupport for IT: 7x24 HW / SW Tech Support and Assistance for Certified IT Staff, 3 Year (923-2362)

Service: Thank you choosing Dell ProSupport. For tech support, visit


Page 57

http://support.dell.com/ProSupport or call 1-800-9 (989-3439)

Installation: On-Site Installation Declined (900-9997)

Misc: Hard Drive Carrier,3.5,1-12PCS,C2100 (342-0981)

Misc: Hard Drive Carrier,3.5,1-12PCS,C2100 (342-0981)

Misc: Hard Drive Carrier, 3.5, 1-12PCS, C2100 (342-0981)

Misc: Hard Drive Carrier, 3.5,1-12PCS, C2100 (342-0981)

Misc: 300GB 15K RPM Serial-Attach SCSI 6Gbps 3.5in Hot Plug Hard Drive (342-1542)




Misc: Power Cord, C13 to C14, PDU Style, 12 Amps, 2 meter, Qty 1 (330-7353)

Misc: Power Cord, C13 to C14, PDU Style, 12 Amps, 2 meter, Qty 1 (330-7353)

Appendix A2: Segment Node Bill of Materials

Base Unit: PowerEdge C2100 Expander Backplane support for 3.5" Hard Drives Redundant

Power Supplies (224-8350)


Processor: Thermal Heatsink, CPU, C2100 (317-3934)

Processor: Thermal Heatsink Front, C2100 (317-3935)


Memory: 48GB Memory (12x4GB), 1333MHz Dual Ranked RDIMMs for 2 Processors, Optimized (317-3394)

Operating System: No OS, No Utility Partition (420-3323)

NIC: Intel Gigabit ET Quad Port 1GbE, PCIe x4 (430-0771)

Documentation Diskette:

C2100 Documentation (330-8774)


Hard Drive Carrier, 3.5,1-12PCS, C2100 (342-0981)














600GB 15K RPM Serial-Attach SCSI 6Gbps 3.5in

new dell poweredge c servers data analytics...

Documents