Exploring Use-cases for Non-Volatile Memories in support of HPC ResilienceOnkarPatil,SaurabhHukerikar,FrankMueller,ChristianEngelmann

Dept.ofComputerScience,NorthCarolinaStateUniversity,ComputerScienceandMathematicsDivision,OakRidgeNationalLaboratory

MOTIVATION• Exaflop Computersà largenumberofcompute+memorydevices+differentformsofinterconnects+coolingandpowerequipmentà CloseProximity• Manufacturingprocessesusedtomakethesedevicesarenotfoolproof

• Lowerdurabilityandreliabilityofthedevices.• Frequencyofdevicefailuresanddatacorruptions↑à effectivenessandutility↓

• FutureApplicationsneedtobemoreresilientwhilethey,• Maintainabalancebetweenperformanceandpowerconsumption• Minimizetrade-offs

• Non-volatilememory(NVM)technologiesà enablememorydevicesthatcanmaintainstateofcomputationintheprimarymemoryarchitecture• Morepotentialinusingthesememorydevicesasspecializedhardware• DataRetentionà criticalinimprovingresilienceofanapplicationagainstcrashes• PersistentmemoryregionstoimproveHPCresiliencyà keyaspectofthisproject

PROBLEM STATEMENT

• Designstrategy• Enablecheckpointing atthedatastructurelevel• Somedatastructuresaremorecriticalthanothersatdifferentstagesoftheapplicationintermsoffailurerecovery• Reducethespaceandtimeoverheadconsiderablyincomparisontotraditionalcheckpointing methods• EasytouseAPIwithminimalcodechanges

NVM-basedMainMemory

APPROACHAPPLICATION

STATICDATASTRUCTURES

DYNAMICDATASTRUCTURES

DRAM NVM

APPLICATIONSTATICDATASTRUCTURES

DYNAMICDATASTRUCTURES

DRAM NVM

APPLICATIONSTATICDATASTRUCTURES

DYNAMICDATASTRUCTURES

DRAM NVM

Application-directedCheckpointing DataVersioningvoidmain(){int *a,size;pmem_cpy_init();…a=(int *)pmem_only_alloc(size);…pmem_cpy_free(a);…}

voidmain(){int *a,size;pmem_cpy_init();…a=(int *)pmem_cpy_alloc(size);…pmem_cpy_update(a);…pmem_cpy_free(a);…}

voidmain(){int *a,size;pmem_cpy_init();…a=(int *)pmem_ver_alloc(size);…pmem_cpy_update(a);…pmem_cpy_free(a);…}

• ExperimentationSetup• 16-nodeclusterwithDualsocket,Quad-CoreAMDOpteron,128GBDRAMmemory,IntelSSDfrom100GBto256GB• DGEMMbenchmarkoftheHPCCbenchmarksuite• Testedfor4,8and16-nodeconfigurationsforamatrixsizesof1000,2000and3000elements

RESULTS

0.001

0.01

0.1

1

10

4 8 16

GFLO

PS

Nodes

GFLOPSinnodescalingforStarDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER

0.001

0.01

0.1

1

10

4 8 16

GFLO

PS

Nodes

GFLOPSinnodescalingforSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER

• DRAMonlyallocationandNVM-basedmainmemoryperformbetterthanApplication-directedCheckpointing andDataVersioningpartlyduetoaninefficientlookupalgorithm

• TheperformanceisconsistentforbothSinglematrixmultiplicationandmultiplematrixmultiplicationoperations

• Allmodesperformsimilarandconsistentlywhenscaledbynodesizeorproblemsize

0.01

0.1

1

10

100

4 8 16

Time(sec)

Nodes

ExecutiontimesinnodescalingforStarDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER

0.01

0.1

1

10

100

4 8 16

Time(sec)

Nodes

ExecutiontimesinnodescalingforSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER

0.0001

0.001

0.01

0.1

1

10

1000 2000 3000

GFLO

PS

No.ofelements

GFLOPSforproblemsizescalinginStarDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER

0.0001

0.001

0.01

0.1

1

10

1000 2000 3000

GFLO

PS

No.ofelements

GFLOPSforproblemsizescalinginSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER

• Theexecutiontimeincreasesexponentiallywhenusingtwotypesofmemoryinsteadofone

0.01

0.1

1

10

100

1000

10000

1000 2000 3000

Time(sec)

No.ofelements

ExecutiontimeforproblemsizescalinginStarDGEMMDRAMPMEM_ONLYPMEM_CPYPMEM_VER

0.01

0.1

1

10

100

1000

10000

1000 2000 3000

Time(sec)

No.ofelements

ExecutiontimeforproblemsizescalinginSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER

• Developthememoryusagemodestomakethemmoreefficientandmaintaincompletesystemstate• Minimaloverhead• Supportmorecomplexapplications

• Developlightweightrecoverymechanismstoworkwiththecheckpointingschemes• Reducedowntimeandrollbacktime

• Combinethemwithintelligentpoliciesthatcanhelpbuildresilientstaticanddynamicruntimesystem

FUTURE WORK

• Non-volatilememorydevicescanbeusedasspecializedhardwareforimprovingtheresilienceofthesystemandwedemonstratedthreepotentialmemoryusagemodelsthatshowconsistentperformanceforcomputeintensiveworkloads

• REFERENCES• Hukerikar,Saurabh,andChristianEngelmann."ResilienceDesignPatterns-AStructuredApproachtoResilienceatExtreme

Scale." arXiv preprintarXiv:1611.02717 (2016).• Mittal,Sparsh,andJeffreyS.Vetter."Asurveyofsoftwaretechniquesforusingnon-volatilememoriesforstorageandmain

memorysystems." IEEETransactionsonParallelandDistributedSystems 27.5(2016):1537-1550.• Hsu,TerryChing-Hsiang,etal."NVthreads:PracticalPersistenceforMulti-threadedApplications." ProceedingsoftheTwelfth

EuropeanConferenceonComputerSystems.ACM,2017.• Liu,Qingrui,etal."Compiler-directedlightweightcheckpointing forfine-grainedguaranteedsofterrorrecovery." High

PerformanceComputing,Networking,StorageandAnalysis,SC16:InternationalConferencefor.IEEE,2016.• Yang,Shuo,etal."Algorithm-DirectedCrashConsistenceinNon-VolatileMemoryforHPC." arXiv preprint

arXiv:1705.05541 (2017).• Wong,Daniel,G.S.Lloyd,andM.B.Gokhale. Amemory-mappedapproachtocheckpointing.No.LLNL-TR-635611.Lawrence

LivermoreNationalLaboratory(LLNL),Livermore,CA,2013.• Rezaei,Arash. FaultResilienceforNextGenerationHPCSystems.NorthCarolinaStateUniversity,2016.• ThisworkwassponsoredbytheU.S.DepartmentofEnergy'sOfficeofAdvancedScientificComputingResearch.This

manuscripthasbeenco-authoredbyUT-Battelle,LLCunderContractNo.DE-AC05-00OR22725withtheU.S.DepartmentofEnergy.TheUnitedStatesGovernmentretainsandthepublisher,byacceptingthearticleforpublication,acknowledgesthattheUnitedStatesGovernmentretainsanon-exclusive,paid-up,irrevocable,world-widelicensetopublishorreproducethepublishedformofthismanuscript,orallowotherstodoso,forUnitedStatesGovernmentpurposes.TheDepartmentofEnergywillprovidepublicaccesstotheseresultsoffederallysponsoredresearchinaccordancewiththeDOEPublicAccessPlan(http://energy.gov/downloads/doe-public-access-plan).

CONCLUSION