IS5200

4036E-1 Test Rack 1 4036E-2 Test Rack 2

IS5300

SX6036-4

SX6036-3SX6036-2SX6036-1

A iDataplexB iDataplexA2 A3C2 D iDataplex C3 C iDataplex Test Rack 2

10G G8124-3 10G G8124-4

1G G

8000

1G G

8000

1G G

8052

1G G

8000

10G

G81

24-2

10G

G81

24-1

40Gb/s Infiniband

1Gb/s Management,Provisioning &

GPFS Cntl

UOACampus

TSM

KAREN

Test Rack 1

9.6T Metadata24 x SSD 400G

144T48 x 7.2K 3T21.6T NeVE

24 x 10K 900G120T

60 x 7.2K 2T36T

60 x 15K 600G

Quad connected8G FC

T1DS1 T2DS1

120T60 x 7.2K 2T

T2DS2 NeVE + T2DS4T1DS2 SSDDS1T2DS3

96T24 x 7.2K 1T

GPFS NSD 1 GPFS NSD 2 GPFS NSD 3 GPFS NSD 4 GPFS Meta NSD 5 GPFS Meta NSD 6

1G Firewall x 2

4T Metadata10 x SSD 400G

3.5T24 x 15K 146G

7.2T24 x 15K 300G

Stage 1A Rack iDataplex 80 M3 Nodes, with 96G RAM 960 Westmere Cores 4096 Fermi GPU CoresIS5200 infiniband switchsimple 1G management network280T Storage in 4 SANSFour GPFS NSD ServersFour NeVE Servers3 Service nodes

Jan 2012

Stage 2AB Rack iDataplex 8 M3 Nodes with 96G RAM 68 M4 Nodes with 128G RAM 96 Westmere Cores 1088 Sandybridge Cores 8192 Fermi GPU CoresAdded leaf cards to IS5200Expanded 1G management networkAdded 7.2T to Metadata SANAdded 22T NeVE SANDual connected Fibre Channel

Stage 2BC Rack iDataplex 60 M4 nodes, each with 128G Ram 960 Sandybridge Cores 4096 Fermi GPU CoresAdded 4x36 Port SX6036 IB switchesExpanded 1G management networkAdded 96T SANRestructured GPFS filesystem

Stage 3D Rack iDataplex 77 M4 nodes, with 128G Ram 1232 Sandybridge Cores 24,960 Kepler GPU Cores 240 Intel Phi Cores3 x 512G 20 Core Westmere nodesAdditional xcat nodeAdded IS5300 IB switchRedesigned infiniband networkAdded 10G Core to 1G Mng NetworkAdded 144T to NeVE SANAdded 9.6T SSD SAN for MetadataAdded 4T SSD to existing Metadata SANQuad connected Fibre ChannelAdded dedicated Metadata GPFS NSDs

Aug 2012

Jan 2013

May 2013

Moved DatacentreMoved From Ak City to Tamaki

Nov 2012

PAN Cluster NeSI @ The University of Auckland Centre for eResearch

Infiniband100ns switch delay300ns max delayMPI latency measured at 6µs

Core Count1116 Westmere3280 Sandybridge16384 Fermi24960 Kepler240 Phi

Dual 10Gb/sManagement CoreInfiniband