climate applications support at nersc

Helen He !!NERSC Climate PIs Telecon!March 16, 2016

Climate Applications Support at NERSC

Climate Projects

-2-

NERSC Workload Diversity

6,000users850projects

-3-

Applica4onsonHopperandEdisonbyhoursusedJanuary2014–December2014

Top5ScienceCategoriesbyalloca4on(2014)

MaterialsScience 20%

FusionEnergy 18%

Chemistry 12%

ClimateResearch 11%

LaDceQCD 11%

CESM,WRF,NCAR-LESamongshown

70 Climate Projects at NERSC (AY2016)

•  Awardsarepublishedat:–  hKps://www.nersc.gov/users/accounts/awarded-projects/2016-allocaTon-awards/

•  Seethedetailedlistofprojectsin“ExtraSlides”(orderedbyalloca4onhours).

•  30projectsuseCESM/ACMEorCESMcomponents.254ac4veusers

•  17projectsuseWRF.85users(36ac4ve).

-4-

AY16 Climate Allocations by PI’s Institutions

-5-

LBNLNCARPNNLU.ColoradoBoulderORNLLLNLUIUCUCIrvineScrippsInstUSEPAAMADU.WiscMadisonNOAAANLBrookhavenLabNorthCarolinaStateUmassAmherstUCDavisColoradoStateU.WashingtonVeturnPartnersHebrewUnivU.SussexUKUCBerkeleyU.MichiganU.MarylandNASAGISSCADeptofWaterResourceIGES-COLAUCMercedTerafluxCorpYaleUnivUCSanDiego

Climate Projects at NERSC (AY2016)-1

Repo ProjectTitle PI Organiza4on Codes

acme AcceleratedClimateforEnergy BillCollins LBNL ACME

k_2 Cloud-SystemSimulaTonswithaMulTscaleNonhydrostaTcGlobalAtmosphericModel

BillCollins LBNL ARAM

mp9 ClimateChangeSimulaTonswithCESM:ModerateandHighResoluTonStudies

GeraldMeehl NCAR CESM

m958 OceanAtmosphereReanalysesforClimateApplicaTons(OARCA)1831-2015

GilCompo UColorado,Boulder

ForecastModel,EnsembleFilter,…

m1704 MulTscaleMethodsforAccurate,Efficient,andScale-AwareModelsoftheEarthSystem

BillCollins LBNL CESM,ChomboAMRDycore

m1867 WaterCycleandClimateExtremesModeling(WACCEM) RubyLeung PNNL MPAS,WRF-CLM

m1199 High-LaTtudeApplicaTonandTesTng(HiLAT)ofGlobalandRegioanlClimateModels

PhilipRasch PNNL CESM

m2467 QuanTfyingandReducingBiogeochemicalandAerosolFeedbackUncertainTes

ForrestHoffman

ORNL ACME,CESM

m1041 PredicTngIceSheetandClimateEvoluTonatExtremeScales EsmondNg LBNL FELIX,CISM,BISICLES,POP2x

mp193 ProgramforClimateModelDiagnosisandIntercomparison His-YenMa LLNL CESM,CAM5,ACME,SAM

-6-

Thelast“Codes”columnisobtainedfromanswerstoERCAPrequestQuesTon20.

CESM Support

-7-

CESM Support •  Proac4velyworkwithNCARtoprepareCESMportand

configura4onchangesforupcomingNERSCsystems,andProgrammingEnvironmentupdatesonexis4ngsystems.

•  Sendto240+ac4veCESMusersmul4ple4mesayearonrecommendedconfigura4onstorunonNERSCsystems(Hopper,Edison,Cori)

•  Inves4gatebuildandrunissuesforindividualusers.HelpuserstoadapttonewNERSCsystemenvironment.

•  Helptoimproveworkflowforbestthroughput.Bundlejobs,dependencyjobs.Op4mizejobsizesandwall4merequests.

-8-

CESM on Cori P1 and Edison (1) •  TheCESM/1.2.2versionportedbyNCAR/NERSCtoCoriP1

(newmachine)andEdisonwithSLURMbatchscheduler.•  Sourcecodedistribu4onandscripts:/project/projectdirs/

ccsm1/collec4ons/cesm1_2_2/.Usesimilarscriptsasbeforetocreatenewcasesbyusingthe"-machcorip1”or“-machedison”op4on.

•  Func4onalportonCoriP1only,nottunedforop4malpe-layoutsorperformance.Basedonpopulardemandandresources,NCAR/NERSCcanhelptotunepe-layoutsforafewspecificmodelconfigura4ons.

•  Youcantunethesefngsin"env_mach_pes.xml"yourselfbyadjus4ngtheNTASKS_xxxandNTHRDS_xxxvaluesandthenredothecesm_setupstepsaierwards.

-9-

CESM on Cori P1 and Edison (2) •  OlderCESMversionsarenotportedtoCoriP1orEdison.

UsingEdisonasanexample,youcanget“mkbatch.edison”,“env_mach_specific.edison”fromthecesm1_2_2Machinesdirectoryalongwiththeedisonentryin"config_machines.xml"andcopythemintoyourversion.

•  UserswithDevelopersaccesscanusecesm1_5_beta04ornewer.Morecasestherewithpe-layouttuned.

•  Severalproblemsrelatedtomodelruncrasheswithspecificmodelconfigura4onshavebeenworkedoutwithusersandNCAR.CESMsourcecodesinNERSC/projecthavebeenmodified.

•  Workwithusersoncompila4onandrun4meissueswithspecificmodelconfigura4onsoncasebycasebasis.

-10-

CESM Survey, May 2014 •  SurveyconductedtohelpNERSCandNCARunderstandCESM

usagemodelforsystemprocurement.•  BiggestprojectsareatNCARandLBNL;othersincludePNNL,LLNL.

Widerangeofsciencegoals,includingIPCCproduc4onruns,extremeweathereventspredic4on,modeldevelopment,abruptclimatechangesimula4on,andmore

•  Jobprofileasafunc4onofmodelcomponentvaries:CAM~50-85%.

•  Threadparallelismalsovaries,using1,2,3,4,and6threadsperMPIperrunforCAMandPOP

•  Parallelismtypicallydrivenbymodelconfigura4on,componentefficiencies,queue4me,alloca4onsize,andotherconsidera4ons

•  Keypoint:duetothelarge#ofprojectsanddiversesciencegoals,itisimpossibletodis4nguishanyonesimulatorthatisdominantatNERSC

-11-

Typical CESM User Jobs, May 2014 (1/6) •  Repomp9,useraaa,IPCCAR5

–  Typical104-nodejobonHopper:•  aprun-n828-N8-d3./cesm.exe•  2603MB/task•  #cplntasks=128nthreads=1rootpe=0ninst=1•  #camntasks=768nthreads=3rootpe=0ninst=1•  #clmntasks=128nthreads=3rootpe=0ninst=1•  #cicentasks=640nthreads=3rootpe=128ninst=1•  #pop2ntasks=60nthreads=3rootpe=768ninst=1•  #sglcntasks=768nthreads=3rootpe=0ninst=1•  #swavntasks=768nthreads=3rootpe=0ninst=1•  #rtmntasks=128nthreads=3rootpe=0ninst=1

-12-

TOT Run Time: 10413.674seconds 28.531seconds/mday 8.3myears/wdayLND Run Time: 422.124seconds 1.157seconds/mday 204.68myears/wdayICE Run Time: 756.239seconds 2.072seconds/mday 114.25myears/wdayATM Run Time: 8673.825seconds 23.764seconds/mday 9.96myears/wdayOCN Run Time: 2268.196seconds 6.214seconds/mday 38.09myears/wdayGLC Run Time: 0seconds 0seconds/mday 0myears/wdayCPL Run Time: 1379.231seconds 3.779seconds/mday 62.64myears/wday

TorunonEdisonwithSLURMnow:#SBATCH-N104…setenvOMP_NUM_THREADS3srun-n828-c3./cesm.exe

TaskdistribuTonreflectsopTmalloadbalanceamongmodelcomponents.

NERSC Exascale Science Application Program (NESAP)

-13-

NESAP Climate Projects

•  ThegoalforNESAPistoprepareDOEOfficeofScienceusercommunityforCoriPhase2,theIntelXeonPhiKnightsLanding(KNL)manycorearchitecture.

•  SelectedClimateNESAPProjectsare:–  ACME(PI:HansJohansen,LBNL)–  CESM(PI:JohnDennis,NCAR)–  MPAS(PI:ToddRingler,LANL)–  WRF(PI:JohnMichalakes,NOAA)

•  Wehavepostdocposi4onsavailabletoworkonNESAPprojects.Pleasehelptospreadtheword.–  PosiTondescripTonat:hKp://cs.lbl.gov/careers-and-fellowshipsChoose“NERSCOnly”,RequisiTonIDis81356.

-14-

CESM NESAP Case Study

-15-

CESMNESAPTeamMembersNCAR:JohnDennis(PI),ChrisKerr,SeanSantosCray:MarcusWagnerIntel:NadezhdaPlotnikova,MartynCordenNERSCLiaison:HelenHe

MG2 Kernel

•  MG2isakernelforCESMthatrepresentsversion2oftheMorrison-Geslemanmicro-physicspackage.Typicallyconsumesabout10%ofCESMrun4me.

•  Kerneliscorebound–  Notbandwidthlimitedatall–  ShowsveryliKlevectorizaTon

•  Someloopboundsareshort(e.g.10)•  DependentsequenceofinstrucTons

–  Heavyuseofmathinstrinsicsthatdonotvectorize•  Kernelhaslongcomplexloopswithinterleavedcondi4onalsandelementalfunc4oncalls.

-16-

MG2 Vectorization Prototype •  Usecompilerreporttocheckandmakesurekey

func4onsarevectorized(andallfunc4onsonthecallstackarevectorizedtoo)

•  Add!$OMPDECLARESIMDand!DIR$ATTRIBUTEFORCEINLINEwhenneeded.

-17-

MG2 Optimization Steps

-18-

Op4miza4onStepsVersion1•  Simplifyexpressionstominimize#operaTons•  UseinternalGAMMAfuncTonVersion2•  Remove“elemental”aKribute,moveloopinside.•  InlinesubrouTnes.Divide,fuse,exchangeloops.•  Replaceassumedshapedarrayswithloops•  ReplacedivisionwithinversionofmulTplicaTon•  RemoveiniTalizaTonofloopstobeoverwriKenlater•  Usemoreaggressivecompilerflags•  Useprofile-guidedopTmizaTon(PGO)Version3(Intelcompileronly)•  Use!$OMPSIMDALIGNEDtoforcevectorizaTon



Version1•  Simplifyexpressionstominimize#opera4ons•  UseinternalGAMMAfunc4onVersion2•  Remove“elemental”asribute,moveloopinside.•  Inlinesubrou4nes.Divide,fuse,exchangeloops.•  Replaceassumedshapedarrayswithloops•  Replacedivisionwithinversionofmul4plica4on•  Removeini4aliza4onofloopstobeoverwrisen

later•  Usemoreaggressivecompilerflags.Trydifferent

compilers.•  Useprofile-guidedop4miza4on(PGO)Version3(Intelcompileronly)•  Use!$OMPSIMDALIGNEDtoforcevectoriza4on

0

200

400

600

800

1000

1200

Original Ver1 Ver2 Ver3

Tim

e pe

r ite

ratio

n (u

sec)

SandyBridge/Intel

IvyBridge/Intel

IvyBridge/CCE

MG2 Summary •  Direc4vesandflagscanbehelpful,howevernota

replacementforprogrammers’workoncodemodifica4ons.•  Breakuploopsandpushloopsintofunc4onswhere

vectoriza4oncanbedealtwithdirectlyandcanexposelogictocompiler.

•  IncrementalimprovementsnotnecessaryaBIGwinfromanyonething.Accumula4veresultsmaser.

•  Performanceandportabilityisamajorgoal:use!$OMPSIMDprovestobebeneficialbutveryhardtouseregardingtheneedofprovidingthealignedlist.

•  Requestedop4onalalignmentdeclara4oninFortranlanguagestandard.

-19-

WRF Support

-20-

WRF Support

•  UsersurveyconductedinOct2015.14responses(~40%fromac4veusers).Mostusersuseversion3.5.1andup,somehasowncustomiza4ons,100to500coreswithpureMPI.2usersuseMPI/OpenMP.

•  HelpedindividualuserstobuilddifferentWRFversionswithdifferentcompilersonHopperandEdison.

•  WRFisaTier3NESAPapplica4on.•  WRFteamhasconfigura4onforKNC;WillreleaseaKNL

versiontouseforCori.•  TheKNLversionwillbeop4malwithhybridMPI/OpenMP.•  Encourageuserstoexplorehybrid(dm+sm)onEdisonand

Corinow.

-21-

What NERSC Can Help

-22-

We can offer help on •  Por4ngandop4mizingapplica4ons•  Workflowop4miza4onandqueueanalysis,jobbundleadviceetc.forbeserjobturnaround

•  Computereserva4ons–  hKps://www.nersc.gov/users/queues/reservaTon-form/

•  Queueboostandusespecialqueue•  Networktuningforlargedatatransfer•  Sciencegateway(portals)forclimatedata•  LargedatastorageatNERSCHPSSarchivesystem

-23-

20th Century Reanalysis Project

-24-

•  PI:GilCompo,UniversityofColorado,Boulder

•  NERSChelpedon: –  ProvideopTonsfordatatransfer

betweenNERSCandNCAR.–  Tuninganddebuggingthe

applicaTon–  AddingmulT-levelparallelismto

bundleseveralassociatedparalleljobstogethertoobtainhighthroughputandsimplifydatahandling

–  Providespecialqueueaccessandincreasemaxjoblimitonmachines.

GlobalTroposphericCircula4onMaps

Global Ensemble Reforecast Project

•  PI:TomHamill,NOAA.•  NERSChelpedon:

–  DatatransferfromNERSCtoNOAA–  PorTngfromIBMtoCraysystems–  OpTmizinganMPIreducTonfuncTon(runTmereducedfrom24min

toafewseconds)–  SuggesTngpost-processingworkflowopTmizaTons–  Detailsat

hKps://www.nersc.gov/news-publicaTons/nersc-news/nersc-center-news/2012/end-to-end-network-tuning-sends-data-screaming-from-nersc-to-noaa/

–  WebGatewayforGlobalEnsembleReforecastData•  hKp://portal.nersc.gov/project/refcst/v2/•  ThisportalallowsuserstodownloadselecteddaysofthefullReforecastmodeloutput

-25-

Do You Have a “realtime” Use Case? •  Exampleusercasesare:

–  RealTmedataprocessingwithexperimentsorevents.–  RealTmedataanalysisfromusersofsciencegatewayportals.Owen

shortjobsthatneedanimmediateresponse–  Sowwaredevelopment/testofworkflowsystemswithrealTmequeue–  VisualizaTonjobs-thiswouldreplacethefuncTonalitytradiTonally

offeredbyavisnodeandallowsforinteracTvevisualizaTontasks

•  SLURMonCorinowsupportsa“real4me”par44on.–  Userscanrequestasmallnumberofon-demandnodesiftheirjobs

havespecialneedsthatcannotbeaccommodatedthroughtheregularbatchsystem.

–  RequestFormat:hKps://www.nersc.gov/users/queues/realTme-queue-request-form/

–  RequiresDOEapprovaltouse“realTme”onCori

-26-

We Welcome Science Highlights

•  Samplesciencehighlightscanbefoundat:–  hKps://www.nersc.gov/news-publicaTons/publicaTons-reports/science-highlights-presentaTons/

–  Acoupleofexamplesalsoshownin“ExtraSlides”

•  Anewwebformtosubmityourresearchtousat:–  hKps://www.nersc.gov/science/share-your-research/

-27-

Tools and Libraries

-28-

Available Tools and Libraries for Climate Applications

•  nco•  cdo•  ncview•  ncl(ncar)•  nccmp•  netcdf4-python•  cray-netcdf,cray-parallel-netcdf•  cray-hdf5,cray-parallel-hdf5•  esmf

-29-

Summary

•  Facilitateinforma4onexchangeamongNERSCclimateusers.

•  Keepthecommunica4onchannelsopenbetweenusersandNERSC/DOE.

•  Pleaseletusknow:–  yourscienceandcomputaTonalaccomplishment–  yourneedforrunningclimateapplicaTonsandperformingclimatedataanalysisatNERSC

-30-

Extra Slides

-31-

Climate Projects at NERSC (AY2016)-2 Repo ProjectTitle PI Organiza4on Codes

m1517 CalibratedandSystemaTcCharacterizaTonAKribuTonandDetecTonofExtremes

TravisO’Brien LBNL CESM

m1355 AnalysisofGlobalCoupled0.1-degPOP/CICEintheCESMFramework

JulieMcclean ScrippsInst CESM

m1795 ProjecTonsofIceSheetandOceanEvoluTon StephenPrice LANL POPCICLES,CISM,BICICLES

m2071

DeterminingimpactsofsmallicecrystalsonbulkscaKeringproperTesoficecloudstoimproverepresentaTonsoficecloudsonsatelliteretrievalalgorithmsandnumericalmodels.

JunshikUm U.IllinoisU-C AmsterdamdiscretedipoleapproximaTon

m2222 ExpandingthecomputaTonalfronTerofmulT-scaleatmosphericsimulaTontoadvanceunderstandingoflowcloud/climatefeedbacks

MichaelPrTchard

UCIrvine SPCESM

sobl SouthernOceanUptakeintheMPAS-OceanModel EdwardPaKon

NCAR NCAR-LES

m2098

CalculaTonsofsingle-scaKeringproperTesofsmallicecrystalstoimprovein-situmeasurements,satelliteretrievalalgorithms,andnumericalmodels

GregMcfarquhar

U.IllinoisU-C ADDA

m1657 EnvironmentaleffectsinthelifecycleofconvecTveclouds MikhailOvchinnikov

PNNL WRF,SAM

m2410 DevelopmentoftheLESARMSymbioTcSimulaTonandObservaTon(LASSO)Workflow

WilliamGustafson

PNNL SAM,WRF

-32-


m1204 CenteratLBNLforIntegraTveModelingoftheEarthSystem(CLIMES)

BillCollins LBNL CESM

m1231

EvaluaTonoftheInteracTonsAmongTroposphericAerosolLoading,RadiaTveBalance,Clouds,andPrecipitaTon

JonathanPleim

USEPAAMAD WRF-CMAQ,WRF

m2320 Consequencesofcloudsuper-parameterizaTonforland-atmospherecouplingphysics

MichaelPritchard

UCIrvine SPCAM,UPCAM

m1642 EvoluToninCloudPopulaTonStaTsTcsoftheMJO SamsonHagos

PNNL ARWRF

m2402 Next-GeneraTonIntensity-DuraTon-FrequencyCurvesConsideringSpaTotemporalNon-StaTonarityinClimate,IntensePrecipitaTonEvents,andSnowmelt

RubyLeung PNNL WRF

m1196 InteracTonsofClouds,ConvecTon,andClimate DavidRomps LBNL DAM

m2067

EvaluaTonofthelarge-scaleandregionalclimaTcresponseacrossNorthAfricatonaturalvariabilityinoceanicmodesandterrestrialvegetaTonamongtheCMIP5models

MichaelNotaro

U.Wisc.Madison

CESM

m1660 RegionalModelingofLand-Ocean-AtmosphereInteracTons LarryBeng PNNL WRF,WRF-Chem,CLM

m1602

InvesTgaTngtheRoleofBiogeochemicalProcessesintheNorthernHighLaTtudesUsinganEarthSystemModel

AtulJain U.IllinoisU-C IntegratedScienceAssessmentModel

-33-


m2190 NextGeneraTonGlobalPredicTonSystem(NGGPS)Benchmarking JohnMichalales

NOAA NGGPSbenchmarksuite

m2136 EvaluaTonandimprovementofConvecTveParameterizaTonsinACMEmodel

WuyinLin BNL ACME

m1929 AMulTscaleReduced-OrderMethodforIntegratedEarthSystemModeling

GeorgePau LBNL CESM-PFLOTRAN-ROM,PROMEs

m1637 ReducingtheUncertaintyinSimulaTngAerosolChemistryoverMulTpleSpaTalScales

JeromeFast PNNL WRF

m1372 HighresoluTonmodeldevelopmenttoquanTfytheimpactoficebergsonthestabilityoftheAtlanTcMeridionalOverturningCirculaTon

AlanCondron UMassAmherst

MITgcm

m1439 EarthSystemModeling YangZhang NorthCarolinaState

WRF/CAM5,WRF/CMAQ,WRF/Chem,WRF/Chem-ROMS

m2250 MajorimprovementsonthelongwaveradiaTveinteracTonsbetweensurfaceandcloudsinthePolarRegionsinatmosphericglobalcirculaTonmodel(GCM)

DanielFeldman

LBNL CESM,LBLRTM-DISORT

m1540 DynamicDownscalingofClimateProjecTonsforDoDInstallaTons RaoKotamarthi

ANL NRCM(WRF),RegCM4

m2249 IntegraTngClimateChangeintoAirQualityModeling AjithKaduwela

UCDavis CMAQ,WRF

-34-


m328 Globalcloudmodeling DavidRandall ColoradoState

CESM,GCRM,SAM,Super-CAM

m1822 UsingAMIEdatatostudycloudprocesseswithintheMadden-JulianOscillaTon

RobertHouze U.Washington

WRF

m2451 IntegratedPredicTveSystemsforSolarEnergywithModeling,PostProcessingandMachineLearning

CameronWhiteman

VertumPartners

WRF

m2478 LinkedRainfallandRunoffIntensity-DuraTon-FrequencyinfaceofClimateChangeandUncertainty

EugeneYan ANL WRF

m1481 ImprovementofRepresentaTonoftheCloudAerosolInteracToninLarge-ScaleModels

AlexanderKhain

HebrewUnivJerusalem

WRF

m726 SimulaTngthetropicalcloud-climateinteracTonsandMJOduringtheAMIEandDYNAMOfieldexperimentsusingtheNCARWRFmodel

YiWang U.SussexUK AMIE-WRF,CESM-UM

m2076 RoleofForestEcosystemsinMiTgaTngClimateChangeImpactsthroughLand-SurfaceEnergyandWaterProcesses

GuangshanChen

U.Wisc.Madison

RegCM4-CNDV

m1865 Predictabilityofthecarbon-climatesystemonseasonaltodecadalTmescales

InezFung UCBerkeley CCSM

mp79 Parallelmodelingofclimate,chemistry,andphysicsaffecTngtheglobalatmosphere

DonaldWuebbles

U.IllinoisU-C CESM,CAM-Chem,WACCM

-35-


mp231 Three-dimensionalGlobalAtmosphericAerosolandChemistryModeling

JoycePenner U.Michigan IMPACT,GCEcloudmodel

m2352 MarineboundarylayercloudssimulatedbySPCAM5-CLUBB StevenGhan PNNL SPCAM5

m1590 RegionalAerosolEmissionandItsDirectandCloud-RelatedForcingEsTmates

TamiBond U.IllinoisU-C

CESM

m1006 DimenTonReducTonoftheCloud-Aerosol-RadiaTon(CAR)EnsembleModelingSystem

Xin-ZhongLiang

U.Maryland CWRF/CAR

m411 InteracTonofAtmosphericChemistryandAerosolswithClimate PhilipCameron-Smith

LLNL CESM,IMPACT

m843 IntegraTngobservaTonsandsimulaTonsofshallowwarmcloudstoadvanceunderstandingofdrizzleformaTonandevoluTon

AnnFridlind NASAGISS DHARMA

m640 InfluencesoftheBoundaryLayerFlowonVegetaTon-AirExchangesofEnergy,WaterandGreenhouseGases

XuhuiLee NCAR NCAR-LES

m1576 ReducingUncertaintyofClimateSimulaTonsUsingtheSuper-ParameterizaTon

CrisTanaStan IGES-COLA SP-CCSM

m2319 UsingcarbonylsulfidetoquanTfyregionalterrestrialbiologicalcarbonfluxes

JohnCampbell

UCMerced STEM

-36-


m997 UsingtheCaliforniaCentralValleyGroundwater-SurfaceWaterSimulaTonModeltostudyimpactsofextendeddroughtongroundwaterlevels,surfacewaterflows,andagriculturalmanagementpracTces.

CharlesBrush CADeptofWaterResource

IWFM,PEST

refcst AmulT-decadalreforecastdatasettoimproveweatherforecastsforrenewableenergyapplicaTons

ThomasHamill

NOAA GFS

m2492 Paleo-megadroughtsandAbruptClimateChangesintheSpeleothemRecords

InezFung UCBerkeley CAM

m2420 NextGeneraTonEcosystemExperiments--Tropics CharlesKoven LBNL CLM,CESM,ACME

m184 High-SpaTalResoluTonModelsofAtmosphericChemistry,AerosolsforRegionalScaleClimateSimulaTons

RaoKotamarthi

ANL CESM,CLM,CMAQ,GEOS-Chem

m1178 DevelopmentofFrameworksforRobustRegionalClimateModeling

RubyLeung PNNL WRF-CLM-ROMS,MPAS-A/MPAS-O,HOMME-POP-ROMS

m1357 CAM5ParameterSensiTvityandUncertaintyQuanTficaTonforCSSEF

YunQian PNNL CAM5.1

m612 ConTnuousDynamicGridAdaptaToninRegionalandGlobalAtmosphericModels

JosephPrusa TerafluxCorp CEU,EULAG

m1269 AmulT-decadalreforecastdatasettoimproveweatherforecastsforrenewableenergyapplicaTons

NigelQuinn LBNL HydroGeosphere

-37-


m2095 InfluenceoftheBoundaryLayerFlowonVegetaTon-AirExchangeofEnergy,WaterandCarbonDioxide

JianpingHuang

YaleUniv NCAR-LES

m1861 TowardsparameterizaTonofroot-rockhydrologicinteracTonsintheEarthSystemModel

InezFung UCBerkeley CLM4/CLM4RR

m1374

MulTscaleModelingofAerosolIndirectEffectsonDecadalTimescales

StevenGhan PNNL CESM,SPCAM

m543 RiverTransportandHydrologyinCCSM MarciaBransteKer

ORNL CESM

m238 TheoreTcalCalculaTonofWaterVaporConTnuumAbsorpTon AndrewLacis NASAGISS atm

m1060 NorthPacificMesoscaleCoupledAir-OceanSimulaTonsComparedwithObservaTons

IvanaCerovecki

UCSD POP2,WRF

-38-

Typical CESM User Jobs, May 2014 (2/6)

•  Repom1040,ccc,IPCCruns•  304jobsonEdison,6to104nodes,upto6hrs•  Typical52-nodejob:

–  aprun-j2-n828-N8-d3./cesm.exe–  1214MB/task–  #cplntasks=128nthreads=1rootpe=0ninst=1–  #camntasks=768nthreads=3rootpe=0ninst=1–  #clmntasks=128nthreads=3rootpe=0ninst=1–  #cicentasks=640nthreads=3rootpe=0ninst=1–  #pop2ntasks=60nthreads=3rootpe=768ninst=1–  #sglcntasks=1nthreads=1rootpe=0ninst=1–  #swavntasks=128nthreads=1rootpe=0ninst=1–  #rtmntasks=128nthreads=3rootpe=0ninst=1

•  Typical21-nodejob:

–  aprun-n504-N24-d1./cesm.exe–  #cplntasks=480nthreads=1rootpe=0ninst=1–  #datmntasks=24nthreads=1rootpe=480ninst=1–  #clmntasks=480nthreads=1rootpe=0ninst=1–  #sicentasks=480nthreads=1rootpe=0ninst=1–  #socnntasks=480nthreads=1rootpe=0ninst=1–  #sglcntasks=480nthreads=1rootpe=0ninst=1–  #swavntasks=480nthreads=1rootpe=0ninst=1–  #rtmntasks=480nthreads=1rootpe=0ninst=1

-39-

TorunonEdisonwithSLURM:#SBATCH-N52…setenvOMP_NUM_THREADS3srun-n828-c3./cesm.exe

TorunonEdisonwithSLURM:#SBATCH-N21…setenvOMP_NUM_THREADS1srun-n504./cesm.exe


–  116jobsonHopper,86to104nodes,upto36hrs–  Typical86-nodejob:

•  aprun-n1680-N24-d1./ccsm.exe:-n192-N12-d2./ccsm.exe•  579MB/task•  #totalnumberoftasks=1872•  #maximumthreadspertask=2•  #cplntasks=1680nthreads=1rootpe=0•  #camntasks=1664nthreads=1rootpe=0•  #clmntasks=400nthreads=1rootpe=1280•  #cicentasks=1280nthreads=1rootpe=0•  #pop2ntasks=192nthreads=2rootpe=1680•  #sglcntasks=1nthreads=1rootpe=0Timingprofileforonerun:

-40-

TOT Run Time: 104327.625seconds 11.433seconds/mday 20.7myears/wdayLND Run Time: 5020.151seconds 0.55seconds/mday 430.27myears/wdayICE Run Time: 10427.512seconds 1.143seconds/mday 207.14myears/wdayATM Run Time: 55907.355seconds 6.127seconds/mday 38.64myears/wdayOCN Run Time: 80272.484seconds 8.797seconds/mday 26.91myears/wdayGLC Run Time: 0seconds 0seconds/mday 0myears/wdayCPL Run Time: 0seconds 0seconds/mday 0myears/wday

MPMDusingdifferentnum_threadsfordifferentexecutablesisnotyetsupportedinSLURM.


–  9jobsonEdison,16to84nodes,upto30hrs–  Typical84-nodejob:

•  aprun-j2-S6--ccnuma_node-n1008-N12-d4./ccsm.exe•  1188MB/task•  #cplntasks=960nthreads=4rootpe=0ninst=1•  #camntasks=960nthreads=4rootpe=0ninst=1•  #clmntasks=48nthreads=4rootpe=0ninst=1•  #cicentasks=912nthreads=4rootpe=48ninst=1•  #pop2ntasks=48nthreads=4rootpe=960ninst=1•  #sglcntasks=1nthreads=4rootpe=0ninst=1•  #swavntasks=1nthreads=4rootpe=0ninst=1•  #rtmntasks=48nthreads=4rootpe=0ninst=1

-41-


Typical CESM User Jobs, May 2014 (5/6) •  Reposcascade,mp193,usermmm•  26jobsonHopperforcascade,18jobsformp193,1to854nodes,upto48hrs

–  Typical854-nodejob: •  aprun-n20480./tstorms.exefilelist16

–  Typical320-nodejob•  aprun-n1280-N4-d6./ccsm.exe•  708MB/task•  #cplntasks=1280nthreads=6rootpe=0•  #camntasks=1280nthreads=6rootpe=0•  #clmntasks=1280nthreads=6rootpe=0•  #cicentasks=1280nthreads=6rootpe=0•  #docnntasks=1280nthreads=6rootpe=0•  #sglcntasks=1280nthreads=6rootpe=0

–  160jobsonEdison,84nodes•  aprun-j2-n1008-N12-d4-S6--ccnuma_node./cesm.exe•  1315MB/task

-42-

TOT Run Time: 16555.161seconds 22.678seconds/mday 10.44myears/wdayLND Run Time: 2476.78seconds 3.393seconds/mday 69.77myears/wdayROF Run Time: 15.698seconds 0.022seconds/mday 11007.77myears/wdayICE Run Time: 1492.282seconds 2.044seconds/mday 115.8myears/wdayATM Run Time: 11063.755seconds 15.156seconds/mday 15.62myears/wdayOCN Run Time: 10997.98seconds 15.066seconds/mday 15.71myears/wdayGLC Run Time: 0seconds 0seconds/mday 0myears/wdayWAV Run Time: 0seconds 0seconds/mday 0myears/wdayCPL Run Time: 1806.178seconds 2.474seconds/mday 95.67myears/wdayCPL COMM Time: 5557.865seconds 7.614seconds/mday 31.09myears/wday

SampleEdisonTimingProfile


Typical CESM User Jobs, May 2014 (6/6)

•  Repom1199,userjjj•  1203ccsm/cesmjobs,1826atmjobs,1to400nodes•  Typical400-nodejob:

–  aprun-n4800-N12-d2./cesm.exe–  668MB/task–  WallTme:6to8hrs–  #cplntasks=4800nthreads=2rootpe=0ninst=1–  #camntasks=4800nthreads=2rootpe=0ninst=1–  #clmntasks=4800nthreads=2rootpe=0ninst=1–  #cicentasks=4800nthreads=2rootpe=0ninst=1–  #docnntasks=4800nthreads=2rootpe=0ninst=1–  #sglcntasks=4800nthreads=2rootpe=0ninst=1–  #swavntasks=1024nthreads=1rootpe=0ninst=1–  #rtmntasks=1024nthreads=1rootpe=0ninst=1

-43-


Scientific Achievement Successinsimula4onstoassessimpactofcomplexterrainonwindturbineefficiencyisnowleadingtoassessmentofclimatechangeandlighteningeffectsonwindpowerproduc4onandrevenue.

Significance and Impact Electricu4li4esarekeenlyinterestedinevalua4ngtheeffectsofclimatechangeandextremeweatheronwindenergy.

Research Details –  UsestheWRFcodetosolvea4Dmodeloftheatmosphere,includingallbasicdynamicsandenergyequaTons,whileallowingfortargetedparameterizaTonsandinteracTonwiththelandandwaterbodies.

–  NERSCresourcesallowevaluaTonofthesensiTvityofthemodelresultstodifferentlandsurfacetypes,atmosphericdatasetsandotherforcingsbothinternalandexternaltothemodel.

Supercomputers and the Future of Wind Energy

CappsandWhiteman,15thAnnualWRFUsers'Workshop

Thisworkhasbeenusing4-Dnext-genera4onmesoscalenumericalweathersimula4onsthatmodelcomplexatmosphericdataresolvedtolocalwindfarmscales.ThiswillallowforopTmalturbinesiteselecTonandevaluaTngthetrueclimatechangeeffectstowindenergy. WorkwasperformedatLBNLusingNERSC

SBIR -44-

Impact on Climate Simulations •  M.E.MaltrudandE.C.Hunke,LosAlamosNa7onalLaboratory;J.L.

McClean,NavalPostgraduateSchool

• High-Resolu4onGlobalCoupledOcean/SeaIceModelingusingPOP• NERSCprovides

§  Environmentforhighlyparallelruns

§  Serviceandstability§  Specialqueuesupport

-45-

Thank you.

-46-

climate applications support at nersc

Documents