Adrianne MiddletonNational Center for Atmospheric ResearchBoulder, Colorado

CAM T340- Jim Hack

Running the Community Climate Simulation Model (CCSM) at NERSC

0 500-1000Years (constant-1870-conditions control run)

TS

(G

loba

lly a

vera

ged

sur

face

te

mpe

ratu

re)1870 Control Run

380b

400c

420d

440e

360a0

500Years

TS

(G

loba

lly a

vera

ged

sur

face

te

mpe

ratu

re)

1870

2000

A B C D E

18701870

1870

1870

20002000

2000 2000

5 Member 1870-2000 Historical Ensemble

CCSM IPCC Graph

Climate of the last Millennium

Caspar AmmannNCAR/CGDModel Vs Observations

CCSM 3.0 T85 on bassi

Atmosphere128 pes

Ocean40 pes

Coupler8 pes

Sea Ice16 pes

Land16 pes

•Small number of processors (208), but long run time (16 days)•Want 5 model years/day•Queue is generally 3 days long•Model runs roughly 1 model year/3 hours or max 4 years/run slot

HPC dimensions of Climate Prediction

Data Assimilation

New Science

Spatial Resolution

Ensemble size

Timescale

Better Science(parameterization → explicit model)

(new processes/interactions not previously included)

(simulate finer details, regions & transients)

(quantify statistical properties of simulation) (decadal prediction/ initial value forecasts)

(Length of simulations)

Lawrence Buja (NCAR) / Tim Palmer (ECMWF)

Spatial Resolution (x*y*z)

Ensemble size

Timescale (Years*timestep)

TodayTerascale

5

50

500

Climate Model

70

10 2010Petascale

1.4°160km

0.2°22km

AMR

1000

400

1Km

Regular 10000

Earth System Model

100yr*20min

1000yr*

3min1000yr * ?

Code Rewrite

Cost Multiplier

Data Assimilation

ESM+multiscale GCRMNew Science Better Science

HPC dimensions of Climate Prediction

?

Lawrence Buja (NCAR)

10

1010

10

10 10

10

2015

Exascale

T42 2.8°

FV 2.0°

T85 1.4°

FV 1.0°

T170 0.7°

FV 0.5°T340 .36° FV 0.25°

0

50

100

150

200

250

300

Horizontal Grid Size (Km)

310km

220km

160km

110km

78km

55km

39km

28km

GlobalGeneral

Circulation

Continentallarge-scale

flow

Regional

local

Paleoclimate

BGC/Carbon CycleSpin-ups

MJO convergence

Resolve Hurricanes

IPCC AR42004 4TF

IPCC AR31998

IPCC AR52010 500TF

CCSM GrandChallenge2010 1PF

Community Climate System Model (CCSM)

Current Configuration• Hub and spoke design with 5 executables• Exchange boundary information through coupler• Each code quite large: 60-200k lines per code• Need 5 simulated years/day --> Must run at “low” resolution• Standard configuration run at scaling sweetspot of O(200) processors

Petascale Configuration• Single executable• 5 years wall-clock day• Targeting 10K - 120K processors per simulation

– CAM @ 0.25° (30 km, L66)– POP @ 0.1° Demonstrated 8.5 years/day on 28K Bluegene– Sea-Ice @ 0.1° Demonstrated 42 years/day on 32K Bluegene– Land @ 0.1°– Cpl

Data Processing

• Ample disk space (We use up to 600 GB at a time)• Ample memory (We use 2-4 GB)• Long serial interactive jobs are fine• Software we use includes the netCDF Operators (NCO),

NCAR Graphics Command Language (NCL), Ferret 5.81, and more.

“It's dependable, reliable, robust, and fits my requirements quite well.” -- Gary Strand

Thanks! Any Questions?