ncar computing update

Copyright © 2003 University Corporation for Atmospheric Research Sponsored by the National Science Foundation

NCAR Computing Update

Tom EngelScientific Computing Division

National Center for Atmospheric ResearchComputing in the Atmospheric Sciences Workshop

11 September 2003


NCAR

• Managed by UCAR• Established 1959• 66 Member & 20 Academic

Affiliate Institutions


Member Institutions

University Corporation for Atmospheric Research

Board of Trustees

President Richard Anthes

Finance & Administration

Katy Schmoll, VP

Corporate Affairs

Jack Fellows, VP

UCAR Office ofPrograms

Jack Fellows, Director

NCARTim Killeen, Director

Larry Winter, Deputy Director

Education& Outreach

Roberta Johnson

ConstellationObservingSystem for

MeteorologyIonosphere

Climate(COSMIC)

Cooperative ProgramFor Operational

MeteorologyEducation and

Training(COMET)

GPS Science & Technology Program (GST)

UnidataVisiting Scientists

Programs(VSP)

Environmental& Societal

Impacts Group(ESIG)

High Altitude

Observatory(HAO)

Mesoscale &Microscale

Meteorological Division(MMM)

ScientificComputing

Division(SCD)

ResearchApplicationsPrograms

(RAP)

Joint Officefor Science

Support(JOSS)

TimothySpangler

BillKuo

MaryMarlino

Robert Harriss

MichaelKnölker

Bob Gall

BrantFoote

Al Kellie

RandolphWare

MohanRamamurthy

MegAustin

KarynSawyer

AtmosphericChemistry Division

(ACD)

AtmosphericTechnology Division

(ATD)

Advanced StudyProgram(ASP)

Climate & GlobalDynamics Division

(CGD)

MauriceBlackmon

AlCooper

DavidCarlson

DanielMcKenna

Nat’l ScienceDigital Library

(NSDL)

DavidFulker

6/03

Digital Library

for Earth System

Education

(DLESE)

Global Learning and

Observation to Benefit the

Environment(GLOBE)

Jack Fellows


Scientific Computing Division (SCD)

• enable the best atmospheric research in the world by providing and advancing high-performance computing technologies

• offer computing, research datasets, data storage, networking, and data analysis tools to advance NCAR's scientific research agenda

Purpose

Facility


SCD’s Computational Facilities

Two Computational Facilities One Infrastructure

FY03 ComputationalResource Usage

Community42%

Climate Simulation Laboratory

58%

• Climate Simulation Laboratory• Community Facility


Climate36%

Oceanography11%

Astrophysics12%

Basic Fluid Dynamics2%

Weather Prediction22%

Upper Atmosphere8%

Other6%

Cloud Physics3%

FY2003† Community Facility Computational Resource Usage by

Discipline

† To date: Oct’02-Aug’03


Oceanography5% Other

1%

Climate94%

FY2003† CSL Facility Computational Resource Usage by Discipline



Climate70%

Cloud Physics1%

Other3%

Upper Atmosphere3%

Weather Prediction9%

Basic Fluid Dynamics1%

Astrophysics5%

Oceanography8%

FY2003† Total Computational Resource Usage by Discipline



Advanced Research Computing System

• At last CAS ... Al Kellie announced the award of the ARCS Contract to IBM:– Fall ’01: 1.1 TFLOP upgrade to POWER3 system

(blackforest) ... was underway during last CAS– Fall ’02: 4.8 TFLOP† POWER4 system (bluesky)– Fall ’03: Federation switch upgrade– Fall ’04: 8.75 TFLOP† system (bluesky upgrade)

• A few things have changed ...– Fall ’02

• POWER4 capability short of contract commitment• additional initiative funding from NCAR

– bluesky: 38-frame, p690 (Regatta) system, 6.3 TF† capability performance commitments


NCAR Computational Facility


Peak Computing Capacity at NCAR

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03

Pea

k T

FL

OP

s at

NC

AR

blackforestWH-I

blackforestupgrade

bluesky

Cray C90/16

HP SPP2000

SGI Origin2000

blackforestWH-II

ARCSPhase 1

ARCSPhase 2


Bluesky diurnal workload distribution

bluesky diurnal workload distribution (Jan-Aug '03)(1024 total batch CPUs)

0

100

200

300

400

500

600

700

800

900

1000

0 2 4 6 8 10 12 14 16 18 20 22

time of day

Ave

rag

e C

PU

s al

loca

ted

CSL

COM


Production Workload Distribution - bluesky

Cumulative Workload Distribution - bluesky (Feb'03-Aug'03)

0%

5%

10%

15%

20%

25%

30%

1 p

e

2-3

pe

4 p

e

5-7

pe

8 p

e

9-15

pe

16 p

e

17-2

3 p

e

24 p

e

25-3

1 p

e

32 p

e

33-6

3 p

e

64 p

e

65-1

27 p

e

128

pe

129-

255

pe

256

pe

257-

511

pe

512

pe

513-

1023

pe

1024

pe

>10

24 p

e

% o

f to

tal j

ob

s/C

PU

tim

e

% of alljobs

% of totalreservedCPUhours


Production Workload Distribution - blackforest

Cumulative Workload Distribution - blackforest (Feb'03-Aug'03)

0%

5%

10%

15%

20%

25%

30%

35%

40%

1 p

e

2-3

pe

4 p

e

5-7

pe

8 p

e

9-15

pe

16 p

e

17-2

3 p

e

24 p

e

25-3

1 p

e

32 p

e

33-6

3 p

e

64 p

e

65-1

27 p

e

128

pe

129-

255

pe

256

pe

257-

511

pe

512

pe

513-

1023

pe

1024

pe

>10

24 p

e

% o

f to

tal j

ob

s/C

PU

tim

e

% of alljobs

% of totalreservedCPUhours


August ’03 queue-wait times - bluesky

8-way LPARs

CSL Community

Average Job

Wait Time (min)Standard Deviation # Jobs

Average Job Wait Time (min)

Standard Deviation # Jobs

Premium 0:24 0:59 422 0:06 0:22 519

Regular 0:35 1:23 1366 0:13 1:31 2412

Economy 1:49 4:24 296 0:34 1:30 1059

Stand-by 2:14 8:19 116 0:09 1:06 1932

32-way LPARs

CSL Community

Average Job




Premium 0:26 1:29 25 0:27 0:45 48

Regular 0:29 1:09 152 0:32 0:48 116

Economy 0:54 2:02 131 0:41 1:23 164

Stand-by 1:00 2:19 287 0:07 0:23 46


August ’03 queue-wait times - blackforest

Winterhawk-2 Nodes

CSL Community

Average Job




Premium 0:13 0:30 509 0:00 0:01 5892

Regular 0:11 0:29 2258 0:02 0:18 7290

Economy 0:37 1:31 1644 0:13 0:41 1397

Stand-by 0:59 2:50 130 0:22 1:18 912

Nighthawk-2 Nodes

CSL Community

Average Job




_nh 0:00 0:02 453 0:00 0:00 114


NCAR MSS Net Growth

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5


Net

Gro

wth

Rat

e (T

B/d

ay)

UniqueTotal

Dual copyinstituted

CoSinstituted

ARCSPhase 1

ARCSPhase 2


NCAR MSS Growth vs. Sustained Computing

0

100

200

300

400


un

iqu

e b

ytes

/ m

illi

on

flo

atin

g p

oin

t o

per

atio

ns

Unique

Total

Dual copyinstituted

CoSinstituted

ARCSPhase 1

ARCSPhase 2


HPM Statistics

HPM Statistics gathered between 07/28/03 and 09/03/03

bluesky8-way

bluesky32-way

blackforestWH-2

blackforestNH-2

Average % Efficiency of User Application Code 3.9 4.3 5.4 9.2

Average MFLOPs per CPU 143.7 141.9 54.9 63.2

Average Daily Peak MFLOPS per CPU 600.2 337.7 238.5 160.3

Average Loads per TLB miss 1491.0 1063.4 1476.1 63.2

Average Data TLB miss rate (/sec) 204475.5 117412.6 - -

Average Instruction TLB miss rate (/sec) 1931.9 2110.8 - -

Average L1 cache hit rate (%) 88.7 89.8 - -

Average L2 cache miss rate (%) 6.1 5.0 - -

Average L3 cache miss rate (%) 32.9 31.1 - -

Average L2 load bandwidth (MB/sec) 1020.9 869.2 - -

Average L3 load bandwidth (MB/sec) 38.8 30.8 - -

Average memory load bandwidth (MB/sec) 116.4 92.8 - -


bluesky workload computation rate


ARCS Status – Fall ‘03

We thought we had our future planned ...

Federation

IPCC


Commitment IPCC ScenariosExtra

extra


“bluesky upgrade”

• IBM p690 augmentation to bluesky (Sep ’03)– +448 1.3 GHz POWER4 processors (in 14 node x 32

processor configuration) – 2.3 TFLOPs peak

– +.896 TB memory, +10.5 TB disk

• Maintenance, Federation Switch, 2-year ARCS extension renegotiated– Federation now 2H04 option

reduces risk during critical IPCC runs

– Federation ECIP participation 3+4Q03

– 10 TFLOP POWER5 option


Peak TFLOPs at NCAR

Peak TFLOPs at NCAR

0

2

4

6

8

10

12

Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03 Jan-04

IBM p690/32 Regatta-H (bluesky)

SGI Origin3800/128(chinook)

IBM p690 (16)Regatta (bluedawn)

Compaq ES40/32(prospect)

IBM POWER3(blackforest)

IBM POWER3(babyblue)

SGI Origin2000/128(ute)

HP SPP-2000/64(sioux)

CRI Cray C90/16(antero)

Cray J90 series

Cray T3D

CRI Cray YMP/8(shavano)

blackforestWH-I

ARCS Phase 1blackforest upgrade

ARCS Phase 2bluesky

Cray C90/16

HP SPP2000

SGI Origin2000

blackforestWH-II

ARCS Phase 3 (IPCC)bluesky

SGI Origin3800


Sustained GFLOPs at NCAR

Sustained GFLOPs at NCAR

0

100

200

300

400

500

Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03 Jan-04

IBM p690/32 Regatta-H (bluesky)

SGI Origin3800/128(chinook)

IBM p690 (16)Regatta (bluedawn)

Compaq ES40/32(prospect)

IBM POWER3(blackforest)

IBM POWER3(babyblue)

SGI Origin2000/128(ute)

HP SPP-2000/64(sioux)

CRI Cray C90/16(antero)

Cray J90 series

Cray T3D

CRI Cray YMP/8(shavano)

blackforestWH-I

ARCS Phase 1blackforest upgrade

ARCS Phase 2bluesky

Cray C90/16

HP SPP2000

SGI Origin2000

blackforestWH-II

ARCS Phase 3 (IPCC)bluesky

SGI Origin3800


Thank You

Questions?

ncar computing update

Documents

basic fluid dynamics8881

fluid dynamics2

weather prediction22

weather prediction103753

best atmospheric research

research datasets

atmospheric researchcomputing

cloud physics3