oracleoukc.oracle.com/static09/opn/bp09/upk_12/31889/wzt5160/data/...overall architecture and major...

<Insert Picture Here>

Oracle

<Insert Picture Here>

WZT-5160

SPARC T3 Based Servers Architecture and Features

Welcome to the SPARC T3 Based Servers Architecture and Features module. This content will cover the overall architecture and major features of Oracle’s SPARC T3-1, SPARC T3-2, and SPARC T3-4 rack-mount servers along with Oracle’s SPARC T3-1B server blade.

3

Objectives

• Identify the features of the servers

• Explain the architecture of the servers

• List the server’s FRUs and CRUs

Upon completion of this module, you should be able to:

1Identify the features of the servers

2Explain their architecture and

3List their FRUs and CRUs

4

Additional Resources

Press PLAY (4) to Continue

Oracle, Inc., SPARC T3-1, T3-2 and T3-4 Servers Administration Guide,

Revision A, June 2010, Part Number 821-2063-xx.

Oracle, Inc., SPARC T3-1 Servers Installation Guide, Revision A, Oct 2010,

Part Number 821-2062-xx.

Oracle, Inc., SPARC T3-2 Servers Installation Guide, Revision A, Sept 2010,


Oracle, Inc., SPARC T3-4 Servers Installation Guide, Revision A, Aug 2010,


Oracle, Inc., SPARC T3-1B Servers Installation Guide, Revision A, Aug 2010,


Oracle, Inc., SPARC T3-1 Servers Service Manual, Revision A, Oct 2010,


Oracle, Inc., SPARC T3-2 Servers Service Manual, Revision A, Aug 2010,


Oracle, Inc., SPARC T3-4 Servers Service Manual, Revision A, Aug 2010,


The following references are 1the technical documentation for the SPARC T3 based server platforms.

5

Additional Resources


Oracle, Inc., SPARC T3-1B Servers Service Manual, Revision A, Sept 2010,


Oracle, Inc., SPARC T3-1 Servers Product Notes, Revision A, Oct 2010,


Oracle, Inc., SPARC T3-2 Servers Product Notes, Revision A, Aug 2010,


Oracle, Inc., SPARC T3-4 Servers Product Notes , Revision A, Aug 2010,

Part Number 821-2112- xx.

Oracle, Inc., SPARC T3-1B Servers Product Notes , Revision A, Aug 2010,


Oracle, Inc., Oracle Integrated Lights Out Manager (ILOM) 3.0 Web Interface

Procedures Guide, Revision A, May 2010, Part Number, 820-6411-xx.

Oracle, Inc., Oracle Integrated Lights Out Manager (ILOM) 3.0 CLI

Procedures Guide, Revision A, May 2010, Part Number 821-6412-xx.

Oracle, Inc., Oracle Integrated Lights Out Manager (ILOM) 3.0 SNMP and

IPMI Procedures Guide, Revision A, May 2010, Part Number 820-6413-xx.

The following references are 1the rest of the technical documentation for the SPARC T3 based server

platforms.

6

Which are the Servers?

SPARC T3-1 (Serpa)

1 CPU socket

SPARC T3-2 (Solana)

2 CPU sockets

SPARC T3-4 (Seville)

4 CPU socketsSPARC T3-1B

(Jumilla)

1 CPU

socket

There are four servers that support the new SPARC T3 CPU chip. 1Oracle’s SPARC T3-1 Server, code

named Serpa, is a 2U rack mount server that supports one CPU socket. 2Oracle’s SPARC T3-2 Server,

code named Solana, is a 3U rack mount server that supports 2 CPU sockets. 3Oracle’s SPARC T3-4

Server, code named Seville, is a 5U rack mount server that supports up to 4 CPU sockets. 4Oracle’s

SPARC T3-1B Server, code named Jumilla is a server blade that supports one CPU socket and is supported on the Sun Blade 6000 Modular System.

7

Which is the CPU chip?

NOTE: The manner in which the DIMMs and BoBs are packaged is platform

dependentPress PLAY (4) to Continue

All four servers support the SPARC T3 CPU chip. 1Each CPU chip has 16 cores and 8 threads per core

for a total of 128 threads per chip that runs off of a 1.65 GHz clock. 2Each core has 16 KBytes of

instruction cache, 8KBytes of data cache, a floating point and graphics unit along with a cryptographic accelerator.

3All the cores share 6 MBytes of level 2 cache that is accessed through the crossbar circuitry.

4The chip’s memory interface consists of two memory controllers which support two memory channels

each. External to the chip, the memory channels interface to the DDR3 DIMMs through memory buffers referred to as Buffer on Board chips or BoBs. Each memory channel supports two BoBs and each BoB has two channels that support two DIMM slots each for a total of 16 DIMM slots. Populating the DIMM slot with 8 Gbyte DIMMs gives you a maximum of 128 Gbytes of memory per CPU chip. 5Note, the manner in which the DIMMs and BoBs are packaged is platform dependent so we will discuss

each packaging scheme when we discuss the individual platforms later in this presentation.6

8


Which is the CPU chip?

Two coherence units, that provide 3 links each, are also available to provide shared memory paths out to other CPUs that may be on the server.

1In terms of I/O, the CPU chip provides two 8 lane PCIe

generation 2 interconnects 2and two XAUI 10 Gigabit Ethernet interfaces. These interfaces will be

discussed in more detail later in this presentation.3

9

Which I/O is Supported?

CPU

and Memory

Subsystem

PCIe2 Switch 0

IDT 48H12G2

Partitionable

(48 lane)

PCIe2 Switch 1

IDT 64H16G2

Partitionable

(64 lane)

PCIe2 x8 PCIe2 x8


PCIe2 x8

PCIe2 x1

PCIe2 x4

PCIe2 x8

PCIe2 x4

PCIe2 x1

There are several industry standard I/O interconnects and buses supported by the SPARC T3 based servers.

1The main I/O protocol used on the servers is PCIe generation 2 interconnects which originate at

the CPUs and are expanded within the server 2using two PCIe2 switch chips.

Depending on the server, the two PCIe2 switch chips used could be 48 or 64 lane types, as displayed here,

3and can support up to 8 lane PCIe2 interconnects within the server architecture. The 4 and 8 lane

interconnects support mainly PCIe2 ports while other I/O devices are supported directly or indirectly through the PCIe2 interconnects.

4

10

Which I/O is Supported?


CPU

and Memory

Subsystem

PCIe2 Switch 0

IDT 48H12G2

Partitionable

(48 lane)

10-Gbit

Ethernet ports

XAUI

Ethernet

Adapter(s)

LSI SAS 2008

SAS/SATA

Controller

RAID 0 and 1

PCIe2 –to-PCI

Bridge

PCI-to-USB

Bridge

Intel

28576 EB

MAC/PHY

REM(on SPARC T3-1B)

PCIe2 Switch 1

IDT 64H16G2

Partitionable

(64 lane)

The PCIe2 interconnects also support other interconnects and buses, through the use of bridge chips and daughter cards,

1such as LSI controller chip , USB controller chips, 1Gbit Ethernet controller chip and the

RAID Expansion Module or REM board on the SPARC T3-1B blade server. The SPARC T3 CPUs also

2provides two XAUI interfaces that are used to provide

10-Gbit Ethernet ports through an Ethernet adapter slot. The number of ports and adapter slots are dependent on the server platform. Since the I/O infrastructure is different for each server platform we will defer that discussion to that section of this presentation.

3

11

Server Features Comparison


Features SPARC T3-1B SPARC T3-1 SPARC T3-2 SPARC T3-4

Size (RU) 1 slot blade 2RU 3RU 5RU

Cores/Threads 16 / 128 (1 CPU) 16 / 128 (1 CPU) 32 / 256 (2 CPUs) 64 / 512 (4 CPUs)

Memory (max) 128 GBytes 128 GBytes 256 GBytes 512 GBytes

PCIe2 Slots 2 EM slots

2 NEM ports

6 10 16

1 GbE / 10 GbE ports 2 / 2 4 / 2 4 / 4 4 / 8

HDD or SSD (max) 4 16 6 8

LDoms (recommended) 16 32 32 64

Service Processor ILOM 3.0 ILOM 3.0 ILOM 3.0 ILOM 3.0

Jumilla SerpaSolana

Seville

A summary of the server features as displayed in this table for a side by side comparison. A copy of this table is available in PDF format that you can download by selecting the Server Features Comparison entry under the attachment tab.

1

12

Specific Server Architecture and Features

Jumilla

Serpa

Solana

SPARC T3-1B

SPARC T3-1

SPARC T3-2

SPARC T3-4

Seville


Now you have a choice to make. Select one of the four arrow icons. This will branch you to that specific server’s architecture and features. Note, you will be given the option to cover all the other server’s

information later on.1

13

Front of the SPARC T3-1 Server


Front

3

2

1

0

7

6

5

4

12

11

9

8

15

14

1310

CD/DVD (2)USB

System Indicators

• Locator LED (white) / Button

• Service Action Required LED (amber)

• Power/OK LED (green)

• Power Button

Fault Indicators

• Power Supply Fault LED (amber)

• Temperature Threshold LED (amber)

• Fan Fault LED (amber)

• Disk slots label

0

1

2

3

4

5

6 78-disk

config

CD/DVD (2)USB

The SPARC T3-1 server’s front panel has the following features. 1On the top left are the system

indicators that consist of the white locator LED and button assembly used as a beacon to locate the server within a data center, an amber service action required LED that is on when a hardware failure has occurred within the server, a green power OK LED that is used to determine the power state of the server and the power button that is used to turn on the power to the server’s host. 2There are 16 disk drive slots available on the SPARC T3-1 server that can support either 8 or 16 SAS or

SATA hard disk drives or solid state disk drives, known as HDDs or SSDs. The numbers represent their slot number as well as their SAS or SATA IDs.

3On the right of the faceplate above disk slots 10 and 13,

you have the CD/DVD drive along with USB ports 2 and 3. 4On the far right are the fault indicators and

the disk drive labels. The fault indicators consist of the power supply Fault LED, the amber temperature threshold LED and amber fan fault LED.

5

14

Rear of the SPARC T3-1 Server


Rear

Power Supplies

• 1+1 redundancy

• hot swap

• 1100 watts @ 110VAC or

1200 watts @ 220 VAC

• Gold+ efficiency rating

NET MGT

SER MGT

System Indicators (duplicate)

(2) USB

Video port

0

31

4

2

5

10-GbE

NET0 – NET3

0 1 2 3

The SPARC T3-1 server’s rear panel has the following features. 1On the left are two 1+1 redundant hot

swappable power supplies that run on 110 or 220 VAC which has a gold+ efficiency rating. At 110 VAC it generates up to 1100 watts of power and at 220 VAC it generates up to 1200 watts of power. 2There are six slots that support PCIe2 adapters with PCIe2 slots 0 and 3 able to support XAUI adapters.

Below PCIe2 slot 0, there is 3a serial and network management port that are connected to the service

processor and a duplicate set of system indicators. The same indicators we went over on the front faceplate. To the right of the network management port

4are four 1-Gbit Ethernet ports, numbered 0 to 3

left to right, that are used by server host. To the right of these ports 5are USB ports 0 and 1 and a video

port. These ports can be used to connect a console keyboard, mouse and a monitor.6

15

Inside the SPARC T3-1 Server


PCIe2

riser

PCIe2

riser

PCIe2

riser

PCIe2

Adapter

Slot 0 and 3

PCIe2

Adapter

Slot 1 and 4

PCIe2

Adapter

Slot 2 and 5

power supply

fan

assemblies

memory

memory

memory

memory

CPU

disk

backplane

and slotsSAS/SATA cable harness

SAS/SATA cable harness

The key components of the SPARC T3-1 server includes 1the CPU, its memory and

2the PCIe2 adapter

cards within the PCIe2 risers. The other components are 3the fan assemblies,

4the power supply

subsystem, 5the disk backplane and slots along with the SAS/SATA cabling harness. To display a list of

FRUs and CRUs click on the SPARC T3-1 FRUs and CRUs entry within the attachment tab. For a closer look at the inside of the server select SPARC T3-1 3-D entry from the attachment tab. Using the control panel that comes with it, you can see the server in any of 360 degree orientations , zoom in and out on components, and remove components to view other components within the enclosure.

6

16

SPARC T3-1 CPU and Memory


BoB

BoB

BoB

BoB

SPARC T3

DIMM DIMM DIMM DIMM

DIMM DIMM DIMM DIMM

DIMM DIMM DIMM DIMM

DIMM DIMM DIMMDIMM

A

B

C

D

• Densities: 2, 4 or 8-Gbyte registered DIMMs

• Speed: 1066 MT/sec

The SPARC T3-1 server has four memory channels each supporting a Buffer on Board or Bob. 1Each

Bob has two channels that supports two DIMM slots. 2Each DIMM slot supports either a 2, 4 or 8-Gbyte

registered DIMM 3that operate at 1066 Megatransfers per second. Note, the population and replacement

rules will be covered in the Installation and Replacement module of this course.4

17

SPARC T3-1 I/O Subsystem


Memory

SubsystemSPARC

T3

x8B3

x8

B5

x8B1

B3 PCIe2 port 3 (x8)

PCIe2 port 1 (x16)

PCIe2 port 5 (x8)

PCIe2 port 0 (x16)

PCIe2 port 4 (x8)

PCIe2 port 2(x16)B1

B5

x8A2

x8

A4

x8A0

A0

A4

A2

PCIe2 x8

PCIe2 x8PCIe2

Switch 0

48 ports

PCIe2

Switch 1

64 ports(2) XAUI

XAUI

XAUI

x4

Intel

82576EB

MAC/PHY

1

x4

Intel

82576EB

MAC/PHY

0

Quad 1-GbE

USB

Hub

PCI / USB

Bridge

NEC

uPD720101

USB

Hub

Front side

USBs

Rear side

USBs

Internal

USB

PCIe2 / PCI

Bridge

PLX

PEX8112

x1

C

C

x1

Service

Processor

SERMGT NETMGT

MAX3241 BCM5221 video

disk1

disk3

disk5

disk7

disk9

disk11

disk13

disk15

disk0

disk2

disk4

disk6

disk8

disk10

disk12

disk14

SAS/SATA

Expander

LSI SAS2008

SAS/SATA

Controller 0

RAID 0,1,0/1

LSI SAS2008

SAS/SATA

Controller 0

RAID 0,1,0/1

A

B

disk0

disk1

disk2

disk3

disk4

disk5

disk6

disk7

SAS/SATA

Expander

A

B

x8 x8

DVD

4

4

SATA

The I/O subsystem of the SPARC T3-1 servers originates directly from the SPARC T3 CPU through 1two

PCIe2 8 lane interconnects that interface to a 48 port and an 64 port PCIe2 switch. 2Each PCIe2 switch

supports 3 PCIe2 slots. All six slots support 8 lane PCIe2 adapters. Slots 0,1 and 2 can also support 16 lane PCIe2 adapters, physically but not electrically.

3Also, slots 0 and 3 are connected to the XAUI

interfaces of the CPU chip and can support 10-Gigabit Ethernet ports. 4Other PCIe2 8 lane interconnects support the internal SAS or SATA disk drives through two LSI

SAS/SATA controllers. Depending on disk backplane and the cable harness used, either 8 or 16 disks can be supported through a primary and a secondary path. One of the controllers also supports the CD/DVD drive. 5The 64 port PCIe2 switch provides two 4 lane PCIe2 interconnects that supports two Ethernet controllers

that hosts four 1-Gbit Ethernet ports located on the rear of the server. 6The 64 port PCIe2 switch also

supports the server’s five USB ports and 7the service processor’s interface to the CPU through two

separate one lane PCIe2 interconnects.8

18



Fan baffle

System Indicators and buttons:

• Locator LED/Button assembly (white)


• OK/Power LED (green)

• Power button

• Fault indicators: SP, Fan, CPU, Memory,

Power supplies and temperature

USB 2 & 3

video portCD/DVD drive

Disk Slots for

(6) SAS/SATA disks

012345

The SPARC T3-2 server’s front panel has the following features. 1On the top left are the system

indicators that consist of the white locator LED and button assembly used as a beacon to locate the server within a data center, an amber service action required LED that is on when a hardware failure has occurred within the server, a green power OK LED that is used to determine the power state of the server and the power button that is used to turn on the power to the server’s host. For hardware failure isolation there is amber fault LED for the service processor, fan module, CPU, memory, power supplies and temperature threshold. 2There are 6 disk drive slots available on the SPARC T3-2 server that can support SAS or SATA hard

disk drives or solid state disk drives, better known as HDDs and SSDs. The numbers represent their slot number as well as their SAS or SATA IDs.

3On the bottom right of the faceplate you have the CD/DVD

drive while on 4the bottom left are USB ports 2 and 3 and the video port.

5

19



Rear

Power Supplies

• N+1 redundancy

• hot swap

• 2000 watts @ 220 VAC

System Indicators (duplicate)

0 31 2 4 5 86 7 9

Network Module (NM) slot

for a Quad 10-GbE adapter

SER MGT

video

NET MGT

Quad 1-GbE ports

NET0 – NET3

(2) USB

0 1

2 3

The SPARC T3-2 server’s rear panel has the following features. 1On the left are two N+1 redundant hot

swappable power supplies that run on 220 VAC which can generate up to 2000 watts of power. 2These 10 slots support PCIe2 adapters. Every slot supports 8 lane adapters except for slots 4 and 5

which support 4 lane PCIe adapters. 3A dedicated slot, referred to as the network module slot or NM, is

able to support a Quad 10-Gigabit Ethernet adapter. Between the power supplies and PCIe2 slot 0 are 4a

duplicate set of indicators we went over on the front faceplate. 5A serial management port, a network management port and a video port, that are connected to the

service processor, are located between the NM slot and PCIe2 slot 5. Below the management ports are 6four host based 1-Gigabit Ethernet ports and USB ports 0 and 1. The video port and two USB ports can

be used to connect a monitor, keyboard and mouse for the system console when needed.7

20

Inside of the SPARC T3-2 Server


Fan

Modules

Power Supply

PCIe2 adapter

slots

and

NM slot

Disk

Drive

slots

Memory Riser Board 1

CPU1

CPU0

Filler

Filler

Filler

Filler




The key components of the SPARC T3-2 server includes 1two CPUs and up to two memory riser boards

per CPU. Notice that the open board slots need a filler to avoid disruption of the cooling paths. 2The

PCIe2 adapter slots and the network module slot are located towards the rear of the motherboard. 3The

disk drive slots and power supplies are located on the right side of the server chassis. To the right of the memory riser boards are

4the fan assemblies that provide the cooling for all of the server’s internal

components. To display a list of FRUs and CRUs click on the SPARC T3-2 FRUs and CRUs entry within the attachment tab. For a closer look at the inside of the server select SPARC T3-2 3-D entry from the attachment tab. Using the control panel that comes with it, you can see the server in any 360 degree orientation , zoom in and out on components, and remove components to view other components within the enclosure.

5

21

SPARC T3-2 CPU and Memory




CPU0 CPU1

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB





A B

D

CD

C

A B

SPARC

T3

SPARC

T3

The SPARC T3-2 server has four memory channels per CPU with each channel supporting a Buffer on Board or BoB.

1Each BoB has two channels that supports two DIMM slots.

2The memory subsystem for

each CPU is packaged in two memory riser cards with each supporting two BoBs and eight DIMM slots. 3Each DIMM slot supports either a 2, 4 or 8-Gbyte registered DIMM

4that operate at 1066 Megatransfers

per second. Note, the population and replacement rules will be covered in the Installation and Replacement module.

5

22

SPARC T3-2 I/O Subsystem


USB

Hub

PCI / USB

Bridge

NEC

uPD720101

USB

Hub

Front side

USBs

Rear side

USBs

Internal

USB

PCIe / PCI

Bridge

PLX

PEX8112

x1x4

Intel

82576EB

MAC/PHY

1

x4

Intel

82576EB

MAC/PHY

0

Quad 1-GbE

SPARC

T3

SPARC

T3

Memory

Subsystem

Memory

Subsystem

CPU0

CPU1

disk0

disk1

disk2

disk3

disk4

disk5

SAS/SATA

Expander

LSI SAS2008

SAS/SATA

Controller 0

RAID 0,1,0/1

x8

4

4

DVD

SATA

PCIe x8

PCIe x8

E

PCIe x8

PCIe

Switch 1

64 ports

D

E

PCIe x8

D

PCIe

Switch 0

64 ports

C

C

Service

Processor

SERMGT NETMGT


x1

(2) XAUI

(2) XAUI

NM Quad XAUI

(4) XAUI

A1x8

x8

A3

x8A0

x8

A3A4

x8

A3

PCIe port 3 (x8)PCIe port 1 (x8)

PCIe port 5 (x4)

PCIe port 0 (x8)

PCIe port 4 (x4)

PCIe port 2(x8)

A1

B5

A0

A4

A2

B7

PCIe port 7 (x8)

PCIe port 9 (x8)PCIe port 8 (x8)

PCIe port 6(x8)

B9B8

B6

x8B8

x8

B7x8

B5

x8B9

x8

B6

The I/O subsystem of the SPARC T3-2 server originates directly from the SPARC T3 CPUs through 1four

PCIe2 8 lane interconnects that interface to the two 64 port PCIe2 switches to create I/O paths from each CPU to both switches.

2Each PCIe2 switch supports 5 PCIe2 slots. Eight of the ten slots support 8 lane

PCIe2 adapters while slot 4 and 5 which support 4 lane PCIe2 adapters. 3Also, the XAUI interfaces of the

CPU chips are connected to a network module slot that supports an 10-Gigabit Ethernet adapter with up to 4 ports. 4Other PCIe2 8 lane interconnects support eight internal SAS or SATA disk drives through a LSI

SAS/SATA controller using a primary and a secondary path to each device. The LSI controller also supports the SATA CD/DVD drive. 5One of the 64 port PCIe2 switches provides two 4 lane PCIe2 interconnects that supports two Ethernet

controllers that hosts four 1-Gbit Ethernet ports located on the rear of the server. 6The same 64 port

PCIe2 switch also supports the server’s five USB ports and 7the service processor’s interface to the CPU

through two separate one lane PCIe2 interconnects.8

23



PM1 – CPU2 & 3

PM0 – CPU 0 & 1

A. System Indicators and buttons:




• Power button

• Fault indicators: temperature , Fan, and EMs

A

C

C. Ports

• video

• NET MGT

• (2) USB

B

B. Storage

• (8) SAS/SATA disks

01

23

45

67

D. Power Supplies:

• 2+2 redundancy

• 200 -240 VAC/unit

• 2000 watts/unit

D0 1 2 3

The unique feature of the SPARC T3-4 server it that all its components are accessible from either the front or rear of the unit. The SPARC T3-4 server’s front panel features are displayed.

1The CPU and memory is packaged on two

processor modules that contain 2 CPU sockets and 32 DIMM sockets each. 2On the left are the system

indicators that consist of the white locator LED and button assembly used as a beacon to locate the server within a data center, an amber service action required LED that is on when a hardware failure has occurred within the server, a green power OK LED that is used to determine the power state of the server and the power button that is used to turn on the power to the server’s host. For hardware failure isolation there is amber fault LED for the temperature threshold, fan module and Express Modules. Note, each system module has their own set of indicators for further isolation. 3There are 8 disk drive slots available on the SPARC T3-4 server’s main module that support SAS or

SATA hard disk drives or solid state disk drives, known as HDDs and SSDs. The numbers represent their slot number as well as their SAS or SATA IDs.

4To the left of the main module are the video port, network

management port and the two USB ports. 5Finally, on the bottom of the server chassis are the 4 power supply slots. When populated, the server

supports a 2+2 power supply redundancy with individual supplies that run at 200 to 240 VAC and can generate up to 2000 watts of power.

6

24



PM0

CPU0

PM1

CPU2

PM0

CPU1

PM1

CPU3

The SPARC T3-4 server’s rear panel has the following features. The 16 slots that support PCIe2 adapters are referred to as Express Modules or EMs.

1Assuming both processor modules are installed,

the EM slots are directly accessible by the individual CPUs as you see here.2

25



0 1 2 3 4

0

1

2

3

PM0

CPU0

PM0

CPU1

If processor module 1 is not present and a PM filler is in its place, the two remaining CPUs from PM0 directly access the EM slots as you see here. So, for proper operation of the server PM0 needs to be present in a two or four CPU configuration. 1On the top are the fan assembles that allows for front-to-rear air flow that cools the internal components

of the server. 2On the left of the chassis are the four AC power connectors that provides power to the four

power supplies in the front chassis.3

26



System Indicators10-GbE ports 4-7

10-GbE ports 0-3 SER MGT

NET MGT

1-GbE NET 0-3

3

1

2

0

To the right of the AC connectors is the rear I/O module, 1referred to as RIO, which contains the system

indicators and ports. 2The system indicators are a duplicate of the ones on the front chassis.

3The ROI

also supports 10-Gigabit Ethernet ports, that originate from the CPUs, through QSFP connections. 4It also

supports 1-Gigabit Ethernet ports, that originate on the onboard Kawela controller chips and are controlled by the service processor.

5The other two RJ-45 connectors support the serial and network

management interfaces to the service processor or SP. To display a list of FRUs and CRUs click on the

SPARC T3-4 FRUs and CRUs entry within the attachment tab. For a closer look at the inside of the server select SPARC T3-4 3-D entry from the attachment tab. Using the control panel that comes with it, you can see the server in any 360 degree orientation , zoom in and out on components, and remove components to view other components within the enclosure.

6

27

Inside of the SPARC T3-4 PM


CPU0

CPU1

CPU0 memory

CPU0 memory

CPU0 memory

CPU0 memory

CPU1 memory

CPU1 memory

CPU1 memory

CPU1 memory

The key components of the SPARC T3-4 processor module includes two CPUs and their corresponding memory, as you can see here.

1

28

Inside of the SPARC T3-4 Main Module


SP REM0

REM1

disks

disks

The key components of the SPARC T3-4 main module includes two RAID Expansion Modules, 8 disks and the service processor, as you can see here.

1

29

SPARC T3-4 Processor Module 0


CPU0 CPU1

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

A B

D

CD

C

A B

SPARC

T3

SPARC

T3

The SPARC T3-4 server has two identical processor modules. PM0 contains two CPU chips, CPU0 and CPU1, with four memory channels per CPU with each channel supporting

1a Buffer on Board or BoB.

Each BoB has two channels that supports two DIMM sockets for a total of 32 DIMM sockets within PM0.2

30

SPARC T3-4 Processor Module 1


CPU2 CPU3

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

DIMM DIMM

DIMM DIMM

BoB

A B

D

CD

C

A B

SPARC

T3SPARC

T3



PM1 contains CPU2 and CPU3 along with 32 DIMM sockets. 1Each DIMM socket supports either a 2, 4

or 8-Gbyte registered DIMM 2with speeds of up to 1066 Megatransfers per second. Note, the population

and replacement rules will be covered in the Installation and Replacement module.3

31

SPARC T3-4 CPU and I/O

memory memory

PM0

CPU0 CPU1

memory memory

PM1

CPU2 CPU3

PCIe2

Switch 0

PCIe2

Switch 2

PCIe2

Switch 1

PCIe2

Switch 3


The connection scheme between the CPU and the I/O was designed to get the most redundancy in the system. Notice that each PM has access to all four PCIe2 switches and thus have access to the complete

I/O subsystem. Each CPU also has a coherency link to each of the other CPUs, as you see here.1

32

SPARC T3-4 CPU and I/O

memory memory

PM0

CPU0 CPU1

PCIe2

Switch 0

PCIe2

Switch 2

PCIe2

Switch 1

PCIe2

Switch 3

PM1 Filler

Module


So if PM1 is removed and replaced with a filler card, PM0 will still have access to the complete I/O subsystem. The filler module serves to maintain air flow to the other components in the server but also

provides a pass-thru connection on the coherency connection between CPU0 and CPU1 for PM0.1

33

SPARC T3-4 CPU Main I/O Board


(5)

PCIe2 x8

to slots

PCIe2

SW2

0

PCIe2

SW2

1

PCIe2

SW0

2

PCIe2

SW2

3

PCIe2

SW2

4

PCIe2

SW0

5

PCIe2

SW2

6

PCIe2

SW0

7

PCIe2

SW1

8

PCIe2

SW3

9

PCIe2

SW1

10

PCIe2

SW3

11

PCIe2

SW1

12

PCIe2

SW1

13

PCIe2

SW3

14

PCIe2

SW1

15

(3)

PCIe2 x8

to slots

(5)

PCIe2

8 lane

(3)

PCIe2

8 lane

USB

Hub

PCI / USB

Bridge

NEC

uPD720101

USB

Hub

Front side

USBs

(FIO)

Rear side

USBs

(RIO)Internal

USB

PCIe / PCI

Bridge

PLX

PEX8112

PCIe2 x1

Service

Processor

SERMGT NETMGT


PCIe2 x1

Eight 10-GbE ports

CPU0

XAUI

CPU1

XAUI

CPU2

XAUI

CPU3

XAUI

10-Gbit

Network

Board

HDD 0 -3

REM1

BP

HDD 4-7

PCIe2 x8

BP

REM0

PCIe2 x8 PCIe2

Switch 2

PCIe2

Switch 1

PCIe2

Switch 3

PCIe2

Switch 0

Intel

82576EB

MAC/PHY

1

Intel

82576EB

MAC/PHY

0

Quad 1-GbE

PCIe2 x4

PCIe2 x4

Kawela Kawela

The four PCIe2 64 port switches support 1the 16 PCIe2 8 lane slots that are referred to as Express

Modules. 2PCIe2 switch 0 and 3 each connects to a RAID Expansion Module. Each REM supports a four

slot backplane that can house hard disk drives or solid state drives, referred to as HDDs and SSDs. 3PCIe2 Switch 0 also supports five USB connectors through a one lane PCIe2 connection. Two in the

front I/O assembly, two in the rear I/O board and one on the main I/O module. 4The four 1-Gigabit

Ethernet ports on the RIO is supported through two Kawela Ethernet controllers that is connected through two four lane PCIe2 interconnects,

5while a one lane PCIe2 interconnect supports the service processor.

6There are also eight 10-Gigabit Ethernet ports, that are supported directly by the XAUI interfaces on

each of the CPUs, that are resident on the 10-Gigabit Ethernet Network Board.7

34

Front of the SPARC T3-1B


01

23

(2)USBvideoSerial

management

port

P/N 530-3936

Option #4622A

System Indicators and buttons:


• Ready-to-Remove LED (blue)



• Power button

• Non-maskable Interrupt button (NMI)

The SPARC T3-1B server blade is supported on the Sun Blade 6000 chassis. As with all the Sun Blade 6000 server blades, it contains some standard features.

1The system indicators consist of the white

locator LED and button assembly used as a beacon to locate the server within a data center, a blue ready-to-remove LED that is on when the server blade is ready to be removed from its slot, an amber service action required LED that is on when a hardware failure has occurred within the server blade, a green power OK LED that is used to determine the power state of the server and the power button that is used to turn on the power to the server’s host. Also, a non-maskable interrupt button that can be used to send a non-maskable interrupt to the CPU. 2The server blade has four hard disk drive or solid state drive slots, referred to as HDDs and SSDs. All

the server blade’s ports come from 3the Universal Connector Port that requires an octopus cable, referred

to as a dongle cable, to access its serial management port, its two USB ports and the video port.4

35

Inside of the SPARC-1B


REM

FEM

SPCPU

memory

memory

memory

memory Disk

Slots 0 &1

Disk

Slots 2 & 3

The key components of the SPARC-1B server blade includes 1a single CPU and up to 16 DIMM slots.

2

The RAID Expansion Module or REM supports

3the four disk slots that are at the front of the

server blade. Another daughter board is 4the Fabric Expansion Module or FEM that supports the host’s

network ports through a Network Expansion Module or NEM. 5This server blade also has a service

processor, known as the SP, that can be easily replaced as we will see later in this course. To display a list of FRUs and CRUs click on the SPARC-1B FRUs and CRUs entry within the attachment tab. For a closer look at the inside of the server select SPARC T3-1B 3-D entry from the attachment tab. Using the control panel that comes with it, you can see the server in any 360 degree orientation , zoom in and out on components, and remove components to view other components within the enclosure.

6

36

SPARC T3-1B CPU and Memory


BoB

BoB

BoB

BoB

SPARC T3

DIMM DIMM DIMM DIMM

DIMM DIMM DIMM DIMM

DIMM DIMM DIMM DIMM

DIMM DIMM DIMMDIMM

A

B

C

D



The SPARC T3-1B server blade has four memory channels each supporting a Buffer on Board or Bob. 1Each Bob has two channels that supports two DIMM slots.

2Each DIMM slot supports either a 2, 4 or 8-

Gbyte registered DIMM 3that operate at 1066 Megatransfers per second. Note, the population and

replacement rules will be covered in the Installation and Replacement module of this course.4

37

SPARC T3-1B I/O Subsystems


Memory

SubsystemSPARC

T3

PCIe2 x8

PCIe2 x8PCIe2

Switch 0

48 ports

PCIe2

Switch 1

48 ports

C

C

x1

Service

Processor

SERMGT NETMGT


x8B1

x8B0

x8A1

x8A0

A0 A1 B0 B1

EM0 NEM0 EM1 NEM1

disk0

disk1

disk2

disk3

REM

x8

NEM0 NEM1

SAS2SAS2

x4

1-GbE

Intel

82576EB

MAC/PHY

0

1-GbE

NEM0 NEM1

(2) XAUI

XAUI XAUI

NEM0 NEM1

PCI / USB

Bridge

NEC

uPD720101

UCP

USBs

Internal

USB

PCIe2 / PCI

Bridge

PLX

PEX8112

x1

USB

Hub

The I/O subsystem of the SPARC T3-1B server blade originates directly from the SPARC T3 CPU through

1two PCIe2 8 lane interconnects that interface to two 48 port PCIe2 switches. Each PCIe2 switch

provides 2two 8 lane interconnects that are routed to the NEMs and EMs, as you can see here.

3The CPU

also provides two XAUI interfaces that are routed to NEM slots to produce 10-Gigabit Ethernet ports. 4The

onboard Ethernet controller generates 1-Gigabit Ethernet interfaces that is also routed to the NEM slots PCIe2 switch 0 also supports

5the RAID Expansion Module or REM and the onboard LSI disk controller

through individual 8 lane interconnects. The REM supports the four onboard disk slots that can house either hard disk drives or solid state drives, referred to as HDDs and SSDs. 6PCIe2 switch 1 supports three USB ports through a one lane interconnect. Two USBs through the UCP

and the other internally on the server motherboard. 7The service processor is also connected through a

one lane interconnect.8

38

Specific Server Architecture and Features


Return

You have just completed the specific architecture and features of one of the servers. If you want to cover another of the servers click on the return icon. If not, click on the play button at the bottom of the screen.

1

39

Software Stack


NOTE: Solaris 10 Update 8 coupled with MU9 was added and is

now the minimum supported OS version.

The SPARC T3-1, 2, 4 and -1B servers run Solaris 10 and Open Solaris. These servers will release with a minimum of Solaris Update 9 running the Sun4v architecture. As you can see, the software stack includes 1the management software OBP and ILOM along with hypervisor which covers the hardware platform and

presents a virtual platform to the Solaris OS. ILOM runs as a application of an embedded Linux OS. 2Note, the IPMI software is embedded within ILOM but you need the IPMITool user interface software to

be able to access its CLI. It comes under Solaris 10 within /usr/sfw/bin directory. Also note, that Fault Management Architecture Event Transport Module is also embedded with ILOM and assists in error data collection for the OS level FMA. 3POST provides hardware host platform testing that verifies that the host system is capable of booting

and running the Solaris OS. It runs on initial power on of the server as well as after a system error. More information about this software stack will be covered throughout this course.

4Note, Solaris 10 Update 8

coupled with Maintenance Update 9 was added and is now the minimum supported OS version.5

PROPERTIES

On passing, 'Finish' button: Goes to Next Slide

On failing, 'Finish' button: Goes to Next Slide

Allow user to leave quiz: At any time

User may view slides after quiz: At any time

User may attempt quiz: Unlimited times

41

Summary

• Identify the features of the servers

• Explain the architecture of the servers

• List the server’s FRUs and CRUs

Now that you completed this module, you should be able to 1identify and explain the features and architecture of

the servers. You should also be able to 2list its FRUs and CRUs. This completes this module.

42

Oracle is the Information Company

43

Oracle

oracleoukc.oracle.com/static09/opn/bp09/upk_12/31889/wzt5160/data/...overall architecture and major...

Documents