HPCchallengesforaerothermalpredic5onsin

aeronau5calengines

4thSciDACCScADSSummerWorkshop

B.Franzelli,PhDstudent

CERFACS,France

CERFACS•  EuropeanCenterforResearchandAdvancedTraininginScien5ficCompu5ng

•  Researchcivilsociety•  7shareholders:CNES,EDF,Météo‐France,ONERA,EADS,SAFRAN,TOTAL

•  110people,90researchersandengineerscomingfrommorethan10countries

•  Mainresearchfields:–  Aerodynamics–  Climateandenvironment–  Codecoupling–  Combus5on

− Dataassimila5on− Electromagne5sm− Parallelalgorithms

B.Franzelli,CERFACS,France2

Combus5oninaeronau5calengines

3

Compressor

Air‐intake

Turbine

MAINPHYSICALPHENOMENA:1. Fuelinjec5on(two‐phaseflow)2. ChemicalKine5cs3. Thermaltransfers:

‐Wallheattransfer‐Radia5veheattransfer

Burntgases

Combus:onchamber

NoiseSpecificconsump5on Noise

SpecificPowerLongevity

GOALS:• Predic5onofpollutants(CO,Nox,Soot)• Temperatureprofileatcombustorexit(anincreaseof20Kattheexitdividestheshelflifeoftheengineby2)

B.Franzelli,CERFACS,France

MULTIPH

YSICS

Thefullthermalproblem

Theobjec5veistodescribeatwo‐phasereac5veflowinacomplexgeometrytakingintoaccount:• Combus5on&sootproduc5on• Conduc5on• Radia5on• Convec5on

B.Franzelli,CERFACS,France

Thefullthermalproblemisresolvedcouplingthreecodesdedicatedtoeachseparatephenomenonviaaparalleldynamiccoupler.

STRATEGY=MULTIPHYSICSCOUPLING

4

Outline

•  Descrip5onofthecodesdevelopped•  Examplesofpar55oningproblems

– Efficientpar55oningalgorithmsfortwo‐phaseflowsimula5onsusingaLagrangianapproach

– Storageofhugechemicaldatabasesfortabula5onmethods

– Paralleliza5onoftheradia5vecode

•  Tomorrowchallenge:thefullthermalproblem

B.Franzelli,CERFACS,France5

Outline

•  Descrip5onofthecodesdevelopped•  Examplesofpar55oningproblems

– Efficientpar55oningalgorithmsfortwo‐phaseflowsimula5onsusingaLagrangianapproach

– Storageofhugechemicaldatabasesfortabula5onmethods

– Paralleliza5onoftheradia5vecode

•  Tomorrowchallenge:thefullthermalproblem

B.Franzelli,CERFACS,France

AVBPcodemaincharacteris5cs

6

•  Startedin1993

•  Compressiblereac5veNSequa5ons

•  Explicitin5me

•  Unstructured/hybridmeshes

•  Movingmeshes

•  Centredschemes

–  FiniteVolume/FiniteElement(2nd/3rdorder)

+controlledlocalar5ficialviscosity

•  MassivelyParallel

•  MachineIndependent M.Boileau(2007)

‐CrayXT3(Rochester,US)‐ AMDbi‐coreOpteron2.4Ghz‐ 700processors

‐Consump5on:112000hCPU‐Execu5on5me:160h‐Physical5me:50ms

AVBPcodemaincharacteris5cs

6

•  Startedin1993

•  Compressiblereac5veNSequa5ons

•  Explicitin5me

•  Unstructured/hybridmeshes

•  Movingmeshes

•  Centredschemes

–  FiniteVolume/FiniteElement(2nd/3rdorder)

+controlledlocalar5ficialviscosity

•  MassivelyParallel

•  MachineIndependent

AVTPcodemaincharacteris5cs

•  HeatEqua5onsolverforanon‐homogenousnon‐isotropicsolidmedium

•  DatastructureandnumericalmethodsinheritedbyAVBP

•  Unstructured/hybridmeshes

J.Amaya(2010)

B.Franzelli,CERFACS,France7

PRISSMAcodemaincharacteris5cs

•  Radia5veTransfertEqua5on(RTE)solver

•  BasedontheDiscreteOrdinatesMethod(DOM)

•  Unstructured/hybridmeshes

•  Differentglobalandnarrow‐bandmodelsforthespectralintegra5on

•  Radia5onproper5esofcombus5ongases(CO,CO2,H2O)andsoot J.Amaya(2010)

B.Franzelli,CERFACS,France8

Outline

•  Descrip5onofthecodesdevelopped•  Examplesofpar55oningproblems

1.Efficientpar55oningalgorithmsfortwo‐phaseflowsimula5onsusingaLagrangianapproach

2.Storageofhugechemicaldatabasesfortabula5onmethods

3.Paralleliza5onoftheradia5vecode

•  Tomorrowchallenge:thefullthermalproblem

B.Franzelli,CERFACS,France

1.Two‐phaseflowandHPC

Developmentof2solversinAVBP:

•  Gasphase:Eulersolver•  Liquidphase:Lagrangesolver

✗ Therearemillionsoffueldropletstobetrackeddownstreamfromtheinjector

✗ Needforefficienttwo‐constraintpar55oningalgorithms

9M.Garcia(2009)B.Franzelli,CERFACS,France

Imbalancedsimula:on

Balancedsimula:on

0

5000

10000

15000

20000

25000

Nprocs

1.Two‐phaseflowandHPC

No particles

0

100000

200000

300000

Nprocs

N° particles

RIB(one‐constraint)

METIS(two‐constraint)

Fortwo‐phaseflowswherepar5clesarenon‐uniformlylocated,two‐constraintpar55oningalgorithmsarerequiredto

avoidloadimbalancing.

B.Franzelli,CERFACS,France

M.Garcia(2009)

1.Two‐phaseflowandHPC

B.Franzelli,CERFACS,France

Duringthecalcula5onofanunsteadycase,par5clesdonotremainatthesameplace:• Howtomodifythepar55oningwith5meinordertopreserveacorrectpar5clesbalance??• Whichcriteriashouldbeusedtodeterminewhenandhowtomodifythepar55oning???

OPENQUESTIONS:

Imbalancedsimula:on

Balancedsimula:on

Fortwo‐phaseflowswherepar5clesarenon‐uniformlylocated,two‐constraintpar55oningalgorithmsarerequiredto

avoidloadimbalancing.

M.Garcia(2009)

2.Chemicalkine5csandHPC•  Inordertocorrectlydescribeintermediatespecies,pollutantsandsootinaturbulent

reac5veflow,anaccuratedescrip5onofthechemicalkine5csisrequired.

•  Tabula:onmethodsarewidelyworkedon:adatabaseofrelevantchemicalterms(massfrac5onsandtemperature)basedonsimula5onsofsimplecombus5onproblemsisusedtorebuildthechemicalsourcetermsinacomplex3Dcalcula5on.

B.Franzelli,CERFACS,France

Anaprioripar55oningofthedatabaseisimpossible:inanunsteadycomputa5on,eachgridnodemustbeabletoaccessanypartofthedatabaseateach5mestep.

10

• Readthewholedatabaseoneachprocessor:alargeamountofmemoryperprocessoristhenrequired,whichiscontrarytothetendencyofthenewgenera5onmachines.• Reducethedatabasesize:someinforma5oncouldbeneglectedorsingleprecisionfloa5ng‐pointformatcouldbeused,whichwoulddecreasetheaccuracyofthechemicaldescrip5on.• Readdynamicallyandpar:allythedatabaseoneachprocessor:averyquickI/Ocomunica5onisthenrequired.

POTENTIALSOLUTIONS:

3.Radia5onandHPCRadia:vesourcetermdependsonthephoton

frequencyandtheangulardirec:on.

FREQUENCYINTEGRATION:quan55esdependonlocalproper5es=>goodsubdomainparallelism.

ANGULARDIRECTION:includesdatafromALLthepointsofthedomain(longdistanceitera5ons).Sequen5alalgorithm=>problemforsubdomainparallelism.

J.Amaya(2010)

B.Franzelli,CERFACS,France11

Outline

•  Descrip5onofthecodesdeveloped•  Examplesofpar55oningproblems

1.Efficientpar55oningalgorithmsfortwo‐phaseflowsimula5onsusingaLagrangianapproach

2.Storageofhugechemicaldatabasesfortabula5onmethods

3.Paralleliza5onoftheradia5vecode

•  Tomorrowchallenge:thefullthermalproblem

B.Franzelli,CERFACS,France

Thefullthermalproblem

12

Thefullthermalproblemisresolvedusingspecializedcodesforeachtransfermode.

Acoupler(PALM)isusedtoexchangedatabetweenthecodes.

Thefullthermalproblem

B.Franzelli,CERFACS,France

CONDUCTIONAVTP

COMBUSTIONAVBP

RADIATIONPRISSMA

CHARACTERISTICTIME[s]

102

10‐3

10‐9

10‐11

CPUTIME1x

10x

100000x

Differentcomputa5onalresourcesandres5tu5on5mesofcalcula5onsmustbemanagedbythecoupler:difficulttoachievetheop5mumonamassivelyparallelmachine.

Thefullthermalproblemischaracterizedby:• differentphysics(characteris5c5me)• differentmeshes(CPU5me)

13

Thefullthermaldescrip5onandHPC

• Mul:codeproblem:howcouldthecouplermanagethehugeamountofinforma5onrequiredandcalculatedbythethreecodesinamassivelyparallelway(memoryandsynchronisa5onproblems)?

• Mul:machineproblem:howtomanagethecomunica5onof3codesrunningondifferentmachines?

• Dynamicaldistribu:onofprocessors:howcouldprocessorsbedynamicallydistributedbetweenthedifferentcodes(i.e.unsteadycasesormovingmeshes).

B.Franzelli,CERFACS,France14

SpecialThanksto:

J.Amaya,M.Garcia,D.Poitou,E.Riber,O.Vermorel,thePALMTeam

ThisresearchprojecthasbeensupportedbyaMarieCurieEarlyStageResearchTrainingFellowshipoftheEuropeanCommunity’sSixthFrameworkProgrammeundercontractnumberMEST‐CT‐2005‐020426.