AnnualTechnicalReport

DE-NA0002375

UniversityofUtah

TheUQ-PredicBveMulBdisciplinarySimulaBonCenterforHighEfficiencyElectricPowerGeneraBonwithCarbonCapture

CarbonCaptureMulBdisciplinarySimulaBonCenter

PhilipJ.Smith,PrincipalInvesBgator

philip.smith@utah.edu 801-585-3129

March1,2015throughFebruary29,2016

May31,2016

AnnualTechnicalReport

DE-NA0002375

March1,2015throughFebruary29,2016

CarbonCaptureMulBdisciplinarySimulaBonCenter

UniversityofUtah–InsBtuteforCleanandSecureEnergy

TheCarbonCaptureMulBdisciplinarySimulaBonCenter(CCMSC)existstodemonstrateposiBvesocietal impact of extreme compuBng by acceleraBng deployment of low-cost, low-carbonenergy soluBon for power generaBon: Advanced Ultra Super CriBcal (AUSC) Oxy-coalTechnology. The overall strategy includes collaboraBon with industrial partners andinterdisciplinary focus on development of technology. Three teams contribute to theoverarchingpredicBvedesign:thecomputerscienceteam,thephysicsteamandthevalidaBon/UQteam.ThecenterispartneredwithGeneralElectricPower.

OutreachandEduca-on

TheV/UQfacultyoftheUniversityofUtahandtheUniversityofCalifornia-Berkeleydesignedacoursethatwastaughtinfall2015.Thecoursewasdesignedforfirst-yeargraduatestudentsatthe threeCCMSCuniversiBes:BrighamYoungUniversity,UniversityofCalifornia-BerkeleyandUniversity ofUtah. Enrollmentwas 4 fromUCB, 3 fromBYU and 5 fromUU, plus five non-registeredobservers. InstructorswerePhilipSmithandSeanSmith(UU)andAndrewPackardandMichaelFrenklach(UCB).ThecoursewaspresentedattheUniversityofUtahorUniversityofCalifornia-Berkeleyand telecast toall threeuniversiBes. Thecourse,enBtled “Modeling/ValidaBon UQ“, covered (1) IntroducBon and MoBvaBon for V/UQ, (2) Semi-DefiniteProgramming, (3) Surrogate Modeling, (4) Experimental Uncertainty, (5) DimensionalityReducBon, (6)KennedyO’HagenAnalysis, (7)MCMCSampling, (8)Bounds-to-BoundsAnalysisand(9)PracBcalWorkflow. Lessonslearnedinthisfirstcoursewillbeaddressedindesignsforthefall2016course.MembersoftheNNSAcommunitywillbeinvitedtoparBcipateinthe2016course.

Several task invesBgators (Tom Fletcher, MarBn Berzins and Eric Eddings) akended andpresentedprogressattheAdvancedScienBficCompuBngInvesBgators’MeeBngthatwasheldinLasVegasinFebruary2016. OtherkeymeeBngsatwhichpresentaBonsweremadeincludetheStewardshipScienceAcademicPrograms (SSAP)Symposium inBethesda,SupercompuBng2015(SC15) inAusBn,AmericanPhysicalSociety(APS)MeeBng inSanAntonioandWorkshoponExascaleSolwareTechnologies(WEST)meeBnginAlbuquerque.

The UCB team collaborated with Professor Stas Shvartsman at Princeton University onapplicaBonoftheB2BDCmethodologytosystembiologyandhostedavisiBnggraduatestudent

from the Shvartsman Lab,HenryManngly. Mr.Manngly expressed interest in applying theB2BDCframeworktohisresearchduringsummer2015.

Cameron Christensen, Peer-Timo Bremer, DuongHoang, Valerio Pascucci, and Steve PetruzzaPresented tutorial, “BigDataand I/O forScienBficSimulaBons.” LawrenceLivermoreNaBonalLaboratory,California,March1-2,2016.

CameronChristensen, Peer-TimoBremer, RoohKhurram, SidharthKumar, BilelHadri,DoangHoang, Georgios Markomanolis, Valerio Pascucci, Steve Petruzza, and Madhu Srinivasan.Presented tutorial, “Big ScienBficDataMade SimpleAHands-on Tutorial inDataGeneraBon,Processing,andDeliveryforHighPerformanceCompuBngandHighResoluBonImaging.”KAUST,SaudiArabia,June28-29,2015.

Valerio Pascucci, Presented “VisualizaBon” to the Topology and Feature Analysis SummerSchool, STFC Hartree Centre SciTech DaresburyWarringtonWA4 4AD United Kingdom, June14-19,2015.

Four graduate students completed internships during year-two; namely, Teri Draper (ScokSkeenat SandiaNaBonal Laboratory – Livermore),OscarDiaz-Ibarra (Chris Shaddix at SandiaNaBonalLaboratory–Livermore),MarkKim (PeterLindstromatLawrenceLivermoreNaBonalLaboratory)andTroyHolland(JoelKressandT-1groupatLosAlamosNaBonalLaboratory).

John Holmen has scheduled his internship forMay through August 2016 at Sandia NaBonalLaboratorywithJonathanHuandRayTuminaro.AllenSanderson(professionalstaffmember)scheduledhis researchexperiencewithEricBruggeratLLNL forMay. JoshuaMcConnellhasscheduledhisstudentinternshipforfall2016withStefanDominoatSNLinAlbuquerque.

Duringyear-twotheseveralstudentsgraduatedandmovedintopost-graduateposiBons.MarkKimandSidharthKumar(Ph.D.2015)arenowworkingaspostdoctoralfellowsattheUniversityof Utah andAaditya Landge (M.S. 2015) is working at Twiker. Alex Abboud (Ph.D. 2015) isworking at Idaho NaBonal Laboratory. Troy Holland has accepted a posiBon at Los AlamosNaBonalLabtobegininJune2016.

ComputerScience:I/O,EDSL,Visualiza-on

RunBmeSystemandInfrastructure

Aswestartedyeartwo, large-scaleproducBonrunsonmorethan128KcoresonVulcanwerenotabletorunbecauseofexcessivememoryuse.Typically,runsconsumemorethan1.0GBpercore,exceeding themachine’scapacity. Theexcessivememoryusagewasdueto theHyprelibrary that is used for the Pressure solve and the RadiaBon calculaBon using the DiscreteOrdinatesmethod.Thisyear,weopBmizedtheuseofHypreandOpenMPthreadsworking inconjuncBonwithUintah’sMPIOnlyScheduler to significantly reduce thememoryuse forboththePressuresolveandtheRadiaBoncalculaBon.Theorderofmagnitudedecreaseinmemoryusagepermitsus to run jobsup to512KcoresonMirawhileusing justhalf thememorypernode.

Both the CS infrastructure team and the Physics applicaBon/VUQ teams worked on theintegraBonofKokkoswithArches,theproducBoncodeforthiscenter.Comparisonshavebeenpromising andwork is ongoing to assess the viability and performance characterisBcs of thisintegratedapproachforperformanceportabilityforcurrentandfuturearchitectures.

This year, Titan resources have been used to do tesBng on an infrastructure improvementdevelopmentbranch,aimedatreducingoverheadwithintheUintahinfrastructure.Specifically,it was focused on the decrease in weak scalability caused by the increase in local MPIcommunicaBon Bme (the sum of MPI receives and sends on a mulB-core node). We havedetermined the growth in this local communicaBon to be due to contenBon for sharedresources, specifically mutex-protected vectors of MPI communicaBon records and otherrelated criBcal secBons that unnecessarily serialized secBons of code. Thesemutex-protectedcommunicaBon lists have now been replacedwith thread-scalable lock-free lists, significantlyreducingthislocalcommunicaBonBme.WearesBlltesBngthesechanges.

Theprincipal result tonote is thenearorderofmagnitude reducBon in local communicaBonBme. AddiBonally, this work has fixed a criBcal memory leak which was caused by a racecondiBoninthepreviousMPIcommunicaBonrecordsthatmanagedoutstandingMPIrequests,a mutex protected vector that caused mulBple threads to simultaneously try and process areceivedmessage.Many threadswould allocatememory for amessage but only one threadwould de-allocate its memory. This memory leak was most pronounced with RMCRT, as itproducessignificantlymoreMPImessagesthantypicallyexpected.ThismemoryleakwasfixedbytheaddiBonoftheaforemenBoned,thread-scalabledatastructuresusedtoreducethelocalcommunicaBoncosts.

I/O

DuringtheyearweorganizedademonstraBonofthetechnologydevelopedandpresentedatthe SuperCompuBng conference. This allowedus to complete anumberof tasks including afirst complete prototype of the data-streaming infrastructure including direct I/O from thesimulaBon,accessviaViSUSserverandremotevisualizaBonViSUSclient. ThedemonstraBonhasbeensuccessfulandreceivedalotofakenBonduringtheSuperCompuBngconferenceandwenowhaveaVisusserverinstalledontheRemusmachineinUniversityofUtah’sCenterforHighPerformanceCompuBngthatholdsanumberofUintahdatasetsaccessibleremotely.

During this period we focused on scaling studies for the I/O infrastructure. This has led toscaling studies thatat themomenthavehadanemphasison theBlueGeneQarchitecture. InparBcular, we have accomplished a run on the fullMiramachine at the Argonne LeadershipCompuBngFacility.Theplotbelowshowsthescalingof thePIDX librarycomparedtothe IORstandardbenchmarkbothwithsinglesharedfileandwithsinglefileperprocess.ThisisaveryencouragingresultsinceindicatesthattheuBlizaBonoftheadvancedIDXfileformatdoesnotcomewith the downside of a slower performance. In fact, inmany cases, especially at largescale,thePIDxdumpsarefasterthanIOR.

This acBvity of pushing the limits of the I/O performance has been complemented by thedeploymentofthePIDXI/OfortheUnitahcodeusedforthesciencerunsinthisproject.TOthisendweareworkingatimprovingtheusabilityofthelibraryforscienceusers,withspecificfocuson theCPHC center at theUniversityofUtahand lateronatDOE faciliBes. InparBcular, theALCFwehavealso testedamore sophisBcateddataaccessmechanism that shouldbebekersuitedforusagewithinarestrictedenvironment.

Finally, we have started to generate documentaBon, examples, and other material fordistribuBonwiththecode.TofacilitateadopBonofthetechnology.Thishasresulted inafirsttutorial for the users of the KAUST high performance compuBng facility that we used as atesBnggroundforatutorialthatweplantoprovidetoDOEscienBstswithinthescopeofthePSAAP project. Some material from the tutorial is being distributed at the site: hkps://sites.google.com/site/bigdatahpc/

VisualizaBon

We conBnue to integrate Uintah and VisIt for in-situ analysis and visualizaBon. The in-situparallel implementaBon has been completed andwe have completed some very preliminaryscaling tests (~400 processes) on our local cluster (ash.chpc.edu). In addiBon to exploringsimulaBondatausersarenowabletoexploreprocessdatai.e.perprocessmemoryusage,taskBmes,mpicommunicaBonBmes,etc.(topimage).SuchexploraBoniscriBcaltodebugginglargescaleproblems.ThemainsimulaBon interfacehasbeenexpandedtoallowuserstoaddtaskssuchasre-gridingorsavingunscheduledcheckpints(rightsideofthesecondimage).Userscanstop, resume, and abort Uintah simulaBons. Uintah-VisIt SimulaBon Dashboard provides asnapshot and an interface to global state variables - some of which can bemodified by theapplicaBonscienBstallowingforcomputaBonalsteeringandvisualdebugging.AddiBonalstatevariablescanbeaddedbythecomponentdeveloper–i.e.thecomputerscienBst’shelpisnotneeded. TheIn-SituDashboardalsoprovidesrealBmeaccesstorunBme(on-the-fly)analysis

suchasMin/Maxvalues.Normally,thisinformaBoniswrikentodiskasseriesoffileswhichtheusersmustnavigate.

During thepast yearwehave conBnued support ofVisIt on variousplauorms.Most recentlyVisIt2.14hasbeenreleasedwithnewfuncBonalityleveragedfromtheDOESDAVprogramformorecomprehensivetoolsforflowfieldanalysis.WeconBnuetoworktointegrateUintahandVisItforin-situanalysisandvisualizaBon.ThecurrentbokleneckiswithanMPIcommunicaBonissuethatweconBnuetoinvesBgate.

We conBnue towork on the closest point sparse octree. The closest pointmethod is a PDEsolver, where normal three dimensional stencils can be used to solve parBal differenBalequaBons on two-dimensional embedded surfaces. By implemenBng a sparse octree for theclosestpointembedding,wecanscaleuptoagridsizeof16,384^3.PreviousimplementaBonsusedatwo-levelgridtosavespace,butnonehavescaledtothishighofaresoluBon.Further,totestourimplementaBon,weperformunsteadyflowlineintegralconvoluBonorUFLICusingthesparseoctreeandtheclosestpointmethod.

Duringthepastyear,wehavepublishedapaperonvolumerenderingandimagecomposiBngonGPU accelerated supercomputers to the IEEE TransacBons on VisualizaBon and ComputerGraphics journal. In thiswork,we used CUDA andGPUDirect RDMA for image composiBng.GPU Direct RDMA allows us to transfer data between GPUs in only one copy operaBoncompared to at least five thatwould be required ifwe did not useGPUDirect RDMA . Thisresultsinamuchfasterinter-nodecommunicaBonthatisessenBalforfastimagecomposiBng.Also,sinceGPUDirectRDMAonlyworksfromdevicememoryandnottexturememory,wehaveintroduced a new rendering pipeline that allows us to render offscreen to anOpenGL bufferobjectinsteadofrenderingtoatexture.ThisallowsustotransferdatafromtheOpenGLcontexttotheCUDAcontextwithoutanycopyoperaBon.WedidstrongscalingstudiesonthePizDaintsupercomputer for up to 4096 GPUs (Piz Daint has a total of 5272 GPUs) for image sizes:2048x2048,4096x4096and8192x8192 pixelsandwegotatleastcomparableperformancetoCPUsfor2048x2048imagesandmuchbekerperformanceontheGPUfor8192x8192images.WealsopresentedpartofthisworkattheGPUTechnologyConferenceinApril2016.

Algorithmsforsort-lastparallelvolumerenderingonlargedistributedmemorymachinesusuallydivideadatasetequallyacrossallnodesforrendering.Dependingonthefeaturesthatauserwants to see inadataset,all thenodeswill rarelyfinish renderingat thesameBme.ExisBngcomposiBngalgorithmsdonotolentakethis intoconsideraBon,whichcanleadtosignificantdelayswhennodesthatarecomposiBngwaitforothernodesthataresBllrendering.Wehavedeveloped an image composiBng algorithm that uses spaBal and temporal awareness todynamicallyscheduletheexchangeofregionsinanimageandprogressivelycompositeimagesastheybecomeavailable. RunningontheEdisonsupercomputeratNERSC,weshowedthatascheduler-basedalgorithmwithawarenessofthespaBalcontribuBonfromeachrenderingnodecan outperform tradiBonal image composiBng algorithms. Thisworkwill be published at theEurographicsSymposiumonParallelGraphicsandVisualizaBon2016.

EmbeddedDomainSpecificLanguage(EDSL)andFramework

Thisyear, theEDSL teamworkedcloselywith the framework teamtoachieve large-scaleCFDcalculaBonsextendedtofullTitan(over18,000GPUs).Thisinvolvedsignificantdevelopmentintheframeworktoimproveefficiencyandprovideappropriatesupportforthelargenumberof

tasks involved in full CFD calculaBons. We have demonstrated scalability of basic CFDalgorithmstoover18,000GPUsfor:

• Low-machalgorithmswithaglobalpoissonsolveforpressure.ThisinvolvesCPUlinearsolveswithGPUkernelsforthebalanceofthecalculaBon.

• FullycompressiblealgorithmdeployedenBrelyonGPU.

ThisisdirectlyfacilitatedbytheDSLNebo,whichalsohasbeenextendedtosupportboundarycondiBons,convecBveflux-limitertreatment,aswellasparBcle-cell interpolaBonoperatorsonGPU. This proof of concept is the first step toward scalable reacBng flow calculaBons usingGPUswithinUintah.

Year-one’sreviewidenBfiedlinearsolversasanareawherewehaverisksincewerelyheavilyonthemand therearenotpresently linear solversavailable fornewarchitectures suchasTitan.WehavebegunexploringalternaBvealgorithmicapproachesforLESofturbulentreacBngflowthat relyoncompressiblealgorithmswhichdonot requireglobal linearsolves. Coupling thiswith a dual-Bme integraBon technique that provides point-implicit treatment together withprecondiBoning to accelerate convergence in dual-Bme,we anBcipate thatwe candeploy analgorithmthatiscompeBBvewithlow-machsolverswhilebeingplauorm-portable.

We have added a fully compressible algorithm to Wasatch in anBcipaBon of the dual-Bmeapproachthatavoidstheneedforapressuresolver. This includedacompleterefactorofthemomentum transport abstracBon in Wasatch, development of new expressions forcompressibleflow,addiBonofnewboundary condiBons (nonreflecBngcondiBonsarenotyetimplemented). Wehavedemonstrated scalabilityout to>18,000GPUsonTitan for the corecompressible CFD algorithm. This also involved extension of all Wasatch flow-relatedexpressionstofuncBononacollocatedgridratherthanthestaggeredgridpreviouslyusedformomentum.

Thecompressibleandlow-MachalgorithmsalsohavebeenevaluatedforscalabilityonMiraupto524,188cores. For the low-Machalgorithm,acceptablescalingbehaviorwasobserved forpatchsizesof32x32x32andlarger,andapproximately80%oftheBmeforeachBmestepwasspent in the linear solve. For the compressible algorithm, acceptable scaling behavior wasobservedforpatchsizesof32x64x64or larger. Poorscalingresultsatsmallpatchsizes,wereduetoanMPIreducBonoftheBmestepperformedbyUintah

NebosupportforboundarycondiBons(applyingoperatorsover“mask”subsetsofpoints)washardenedanddeployedinWasatchtoenablesupportforboundarycondiBonsonGPUs. ThiswasanimportantsteptodemonstraBngthatthefullCFDalgorithmcouldbedeployedonGPU.

In support of parBcle moment methods as well as point-implicit algorithms, the EDSL nowsupportsdense linear algebra appliedpointwiseover a grid. This includes regression tesBngandperformance tesBng to ensureefficient serial,mulB threaded,GPUperformance. Thesefeatures includedtheEigendecomposiBonand linearsolveof listsofmatricesandvectorsontheGPU.

TheEDSLsupportsbackendsforserialCPU,mulBcoreCPUandGPU. Wehavedevelopedandmaintained thesebackends,buthavebeenobservingKokkosdevelopment toconsider itasaportablebackendlayerfortheDSL. WebegananexploratoryeffortatKokkosintegraBonfora

subsetof theEDSL features,andwill conBnuethiseffort in thecomingyears. Resultsat thispointaremixed,withKokkosperformingbekerinsomeinstancesandthenaBveEDSLbackendperformingbekerinothers.

Physics:LES,Mul-phaseFlow,Par-cleCombus-on,Radia-on

LargeEddySimulaBonIntegraBon

Ingeneral,theeffortsoftheLESIntegraBonteamoverthecourseoftheyearincluded:• coordinaBonwiththephysicsteam,enablingtheintegraBonofcoalandrelatedphysicsintotheLESalgorithm;

• coordinaBonwiththeV/UQteam,aidingtrouble-shooBngandprofilinglargeproducBonruns;• developmentofalgorithmicstabilityfortheintendedapplicaBon;• producBoncodetransiBoninadopBnganexascaleprogrammingparadigm;• providingproofofreliabilityforthemainLESalgorithm.

ThisyeareffortwasspentonprovidingsomeaddiBonalrobustnesstotheverificaBonoftheLESalgorithm.Aspartofthiseffort,thebaselineCFDtransportofscalars(e.g.,mass,momentum,energy)wasevaluatedfororderofaccuracyandconservaBon.InparBcular,theconservaBonofenergyandmass,whencoupledwiththeparBcleandwallmodels,wasofparBcularconcern.Testproblemsweredevelopedandrun.Itwasshownthatthecodeconservedmassandenergyin the full coal problemwith the conjugate wall heat transfer. In addiBon, the constant andvariabledensityalgorithmwastestedusingthemethodofmanufacturedsoluBons[1-3].Itwasshownthatthealgorithmformallyverifieduptothirdorderforconstantdensityproblemsbutremainsfirstorderforthehigher-orderRunge-Kukamethod.IthasbeenshownthatthebenefitofthehigherorderRKmethodswasareducBonofoverallnumericalerrorofthescheme,whichcanaddstabilitytothelargerproducBonruns.ThereducBoninoverallaccuracyforthehigher-orderRK-schemeshasbeenlinkedtothepressureprojecBon.Thereis,however,someworkleltobedonetoaccommodatethecoalphysicsinthehigher-orderRKschemes.

Nightlyregressiontestcoveragewasevaluatedthisyear.Tothisend,atoolwasdevelopedtoanalyzethenightlyregressionsuitecoverageoftheArchescode.TheintentofthetoolistoaiddevelopersinidenBfyingdeficienciesinthenightlytesBngforexisBngopBonsandnewopBonsas development is occurring. As a result, nightly regression coverage has increased over thecourseoftheyear.

Work conBnued on what has been called the Arches Task Interface, which provides a lightweight abstracBon layer tominimizeUintah boilerplate for the Arches code and enable easyaddiBon of new physics while offering some flexibility in how algorithmic components areassembled.Theinterfacewasimprovedfromaperformancestandpointandfromadeveloper’susagestandpoint.Thisworkisongoing.

The Arches team in collaboraBon with the Computer Science Team has made significantprogressonadopBngKokkosintotheArchescode.Thiseffortssupplantsthepreviousefforttoadoptthecenter’sDSLschemeintoArches.Thereasoningistwofold:1)TheadopBonofKokkosinto Arches presents less work and 2) Some of the algorithmic/modeling schemes anddeveloperpreferenceswerenotwellsuitedforahighlyabstractedDSL.Giventhelonghistoryof the Arches code, the transiBon to Kokkos consBtutes in the easiest terms a simple

replacement of the TradiBonalUintah iterator loops in the formof a C++ functor or Lambdaconstructs. As a result, large porBons of the coal physics have been reformulated into C++functorsandcanberunthroughtheKokkosframeworksupport.WhilethisworkrequiressomeverificaBon,weanBcipateanimpactontheproducBonrunssoon.

Awall-deposiBonmodel for the coal ash transformaBon coupledwith thewall heat transferapproach in Arches was also developed. The parBcle deposiBon model uBlizes the parBcleviscositycriteriatodeterminetheparBcledeposiBonprobabilityandalsoconsiderstheparBcletemperature-historyandthewalltemperatureeffects.ThedeposiBonmodelcoupledwithwallheattransfermodelontheARCHEScodecanpredicttheformaBonofpowderydepositsontheboiler heaBng surfaces. Themodel predicts three regimes for the ash deposits stage on theboilersurface,whichstronglyimpactthepredicBonof temperaturefieldandheatfluxofthewall.

Otherpointsofinterest/accomplishmentsinclude:• ReworkingofthediscreteconvecBveschemestoincreasecodeperformance• Exploring Bme-splinng methods for DQMOM, including Strang and Lie splinng to enablehigherordermethodsforthecoalcasesspecifically

• InstrumentedArcheswithrunBmestaBsBcmodulestoenablethegatheringofQOIstaBsBcs• InconjuncBonwiththeParBclePhysicsteam,CQMOMwiththesetofrequiredcoalphysicswasdemonstratedinaproof-of-concept.

VariableDensityAlgorithmDevelopment

Wehavedevelopedanew,consistentvariabledensity, low-MachpressureprojecBonmethod.ThemethodissuitableforexplicitBmeintegratorsandisshowntoberobustoverawiderangeofdensityraBosupto1000. Thishasbeenthoroughly testedagainstexisBngandnewMMSandanalyBcalsoluBons. Thisworkwassignificantly improvedthroughaweek-longworkshophostedatSandiathatinvolvedresearchersfromtheIllinoisandStanfordPSAAPcentersaswell.

MulBphaseFlow

During the reporBng period there have been several major shils in responsibility for thissubsecBon. Thecoreresponsibility,developmentandimplementaBonofcunngedgeEulerianquadrature-momentmethods,hasremainedthesame.However,duetothe learningthathasoccurred based on the year on V/UQ effort there, and due to personnel shils the followingchangeshaveresulted.First,basedonthe lessons learned inyearone,theprimaryvalidaBoncaseforwhichwewereresponsible—theoxy-firedcombustor(OFC)—wasdroppedfromthehierarchy.AtthesameBme,anewresponsibilitywasadded—modelingofwalldeposiBonandashtransformaBons.Also,throughouttheyeartherehavebeenseveralpersonnelchangesthathavedictatedtheprogressincertainaspectsoftheproject.TheindividualprimarilyresponsibleforthedevelopmentofthenewmulBphase-flowalgorithm,AlexanderAbboud,graduatedwitha Ph.D. in chemical engineering and moved-on to Idaho NaBonal Laboratory. As a resulttechnicalprogresshashalted,andemphasishasshiledtorecruiBngareplacement.Aleronestrongcandidatefellthrough,asecondhasbeenidenBfiedandwearehopefulthatthehirewilloccurinthenextfewmonths.

OurcenterhasbeenfortunatetohavearetainedtheservicesofProfessorYuxin“MarBn”Wuasapart-Bmeemployeeyear-roundandafull-Bmeemployeeduringhisleave-of-absencefromhis

home university. The Department of Thermal Engineering, TsinghuaUniversity in China is amost respected insBtuBon. Duringhis on-site visit,MarBnhas assumed responsibility for allsimulaBonsofAlstom’sboilersimulaBonfacility(BSF)aswellastheassociatedV/UQ.DuetotheBmingofMarBn’svisitandoverlapwithV/UQ,theseresultswillbereportedinothersecBonsofthisannualreport.

ForthecoremulBphase-floweffort,thestreamcrossingproofofconceptwasextendedtothreedimensionswithfourcrossingstreams.Apseudo-secondorderkineBcconvecBonscheme,thattakesadvantageofboththemulB-momentandmulB-environmentalaspects

oftheapproach,wasimplemented.ThisgreatlyreducesthenumericaldissipaBonincurredwiththemethod.AproofofconcepthasbeencreateddemonstraBngthemethodsabilitytohandlewallreflecBonsinaphysicallyrealisBcmanner.ThisimplementaBonisgeneralenoughto

handleavariableresBtuBoncoefficientfortheimpactwiththewall.TheArcheshasfoundthatstair-stepped representaBonof complexgeometry sufficientlyaccurate for theapplicaBonsofinterestwhenisappliestothefluidmechanicsandscalartransport.However,whenitcomestowall reflecBons stair steps results in a disBnctly different reflecBon behavior compared to anoriginal sloped surface. As such, this newmethod was extended to handle a reflecBon thatmorecloselyrepresentstheunderlyingsurface—nominortask.Evenmore,thismethodwas

extendedbeyond its core kineBc capability to simultaneously include scalarproperBesof thedispersedphase.Thisextensionalloweda smallproof-of-concept simulaBonof the fullmulB-physicsproblemfordevolaBlizingandcharoxidizingcoalparBcles.WorkneedstoconBnuetoscaleupthisproofofconcepttotheproducBonboilers.

TheashdeposiBonfromcoalcombusBonisofvitalimportancefortheboilersoperators.Firstlyitaffectsheat transferon theboiler,whichwoulddeteriorate the thermalefficiencyofboilersurfaceandeventually cause theproblem for theboileroperaBons. Secondly, sheddingof amassivedepositwoulddamagetheboilerashhopper.Physically,lotsofphysicalprocessesareinvolvedintheashdeposiBonprocess.Specifically,thedepositmechanismdependsonthecoalcomposiBon, parBcle sizedistribuBonof originalmineralmaker, thermal condiBons andflowdynamics in the gas phase. Moreover, slagging in the molten surface also act the erodingeffectsonthedeposiBon.ThepresentresearchaimsatusingthedeposiBonprobabilitymodelcoupledwiththewallheattransfermodelontheUintahcodetopredicttheashdeposiBoninthepulverizedcoalboiler.TheparBcledeposiBonmodeluBlizestheparBcleviscositycriteriatodetermine the parBcle deposiBon probability and also considers the parBcle temperature-historyandthewalltemperatureeffects.ThedeposiBonmodelcoupledwithwallheattransfermodelontheARCHEScodecanpredicttheformaBonofpowderydepositsontheboilerheaBngsurfaces.Thispartresearchsuccessfullypredictsthreeregimesfortheashdepositsstageontheboiler surface,whichstrongly impact thepredicBonof temperaturefieldandheatfluxof thewall.

LagrangianParBcleTransport

A new and simpler algorithm for parBcle-in-cell interpolaBon method that reduces thecomputaBonal cost by 60% is developed, verified and implemented. Previously, weimplementedtrilinearinterpolaBonforparBcle-cellinterpolants,whichmaintainssecondorderaccuracyforarangeofparBclesizes. However,forthegridsizesandparBclesizesinourcoalcombusBonsimulaBons,this isoverkillandwecansafelytreattheparBcleas interacBngwith

only one cell at a Bme. This reduces the cost of parBcle-cell interpolaBon by as much as60-80%.

Wehaveaddedsupport inUintahfor injecBngparBclesatrunBme,which is requiredforcoalsimulaBons and has been leveraged by other efforts as well. The compuBng faciliBes atArgonne NaBonal Labs (Mira) were used to invesBgate the scalability of Lagrangian parBcletransportwithinWasatch.ThenumberofCPUcoresusedinthescalingstudyrangedfrom1to524,288. Patchsizesconsideredrangedfrom16x8x864x64x32to gridcells.Acceptableweakscalingwasachievedforpatchsizesgreaterthanorequalto32x32x16forcaseswith1parBclepercell.Excellentweakscalingwasobservedforallpatchsizesconsideredfor10parBclepercellcases. Goodstrongscalingwasseenfordomainsizesequal toor less than256x128x128gridcells for1parBclepercell caseswhileoutstandingstrongscalingwasachieved forall10parBcle per cell cases. ErraBc scaling behavior was observed when both parBcle and fluidtransportwere considered. ExecuBon Bmings indicate that this is due to a spike in the BmerequiredfortheHyprematrixsolveincertaincases.Theexactcauseforthisisnotyetknown.Ingeneral,bothstrongandweakscalingimprovedastheparBclespercellwasincreased.

CoalDevolaBlizaBon

SixsimpledevolaBlizaBonmodelformswereimplementedandopBmizedinMatlabbytheBYUteamforSufcocoalandcompareddevolaBlizaBonmodelswithCPDmodel.EvaluaBveelementsincluded:• HeaBngrates:5×103K/s,1×104K/s,1×105K/sand1×106K/s.• Developed modified two-step model with distributed acBvaBon energy; accurate and lesscoefficientsthanRFmodel.

• OpBmizaBonintheworkstoverifymulBplecoaltypesforallmodelforms.

Literaturereviewofelementalanalysisofpyrolysisproducts(especiallycharandtar)byBYUisin progress with the goal to formulate a set of correlaBons to predict product elementalcomposiBonfromparentcoalproperBes.LabexperimentsinprogressatBYUtogatherdataforverificaBon and validaBon of primary pyrolysis, secondary pyrolysis, and char conversionprocessesinoxy-fuelcondiBonsforcoalsofinterestinCCMSCsimulaBons.

CoalCharOxidaBonandGasificaBon

AsaresultoftheV/UQeffortsinyear1,weshiledemphasisfromdevolaBlizaBonmodelingtocharoxidaBon/gasificaBonmodeling.ThisinvolveddetailedexaminaBonofcharmodelsrangingfrom very detailed to relaBvely simple. These were paired with simple and complexdevolaBlizaBonandgas-phasechemistrymodelstoassesstheeffectsofvariousmodelpairings.Modelswereassessedusingsingle-parBcledatafromSandiaNaBonalLaboratories.Specifically,the nth-order Langmuir Hinshelwood (simplemodel) and the Char Conversion KineBcs (CCK)(detailed)modelswereconsidered.TheChemicalPercolaBonandDevolaBlizaBon(CPD)andatwo-stepmodelwereconsideredforcoaldevolaBlizaBon.AgasphasechemistrymodelbasedontheGRI3.0mechanism(detailedkineBcs,orDK)wasused inaddiBonto infinitely fastgasphasechemistry.

In general, the high-fidelitymodel combinaBons weremore capable of accurately predicBngparBcletemperatureandmassloss. Useoftheinfinitelyfastgas-phasechemistrymodelolenresulted in significant underesBmaBon of the parBcle temperature, while use of detailed

kineBcs resulted in overesBmaBon at elevated oxygen concentraBons. Choice of gas phasechemistry had likle effect on the calculatedmass loss of the coal parBcle. The CCKmodelproduced parBcle mass histories that matched experimental observaBons, however itoveresBmatedparBcletemperaturesatelevatedoxygenconcentraBons.Conversely,thesimplerLanmuir-HinshelwoodmodelpredictedparBcletemperaturesfairlyaccurately,butfellshortonparBcle mass predicBons. Use of CPD for devolaBlizaBon gave beker results for masspredicBonscomparedtothetwo-stepdevolaBlizaBonmodel,whichoveresBmatedthevolaBlemass loss. ParBcle temperature calculaBonswere not sensiBve to the chosen devolaBlizaBonmodel.

AnewmodelforthedistribuBonoftheliberatedheatduringcharheterogeneousreacBonsisdeveloped. This model determines the fracBon (ɑ) of the liberated heat by heterogeneousreacBons (mostly char oxidaBon) released into the gas phase. The convecBve heat transfercoefficient, coal heat conducBvity, capacity and coal size are incorporated in thismodel. ThenewmodelisevaluatedinconjuncBonwiththreecharoxidaBonmodels:firstorder,Langmuir–Hinshelwood and CCK of oxygen composiBons and coal types. The new model is moresuccessful than previous model in capturing parBcle temperature profiles observedexperimentallyoverawiderangeofoxygencomposiBonsandcoaltypes.

TheBYU team rewrote thebackboneof theChar ConversionKineBcs (CCK) code to facilitateBmeandtoleranceconvergence. ComponentswereaddedtotheCCKcode(nowtheCCK\oxycode)toallowittoexecuteintheextremesofoxyfuelcondiBonsinsecondsinsteadofhoursordays.

We completed a global sensiBvity analysis of the comprehensive char conversion code by 3methods(veryconsistentfindingsforthemostimportantsub-modelsinneedofakenBon).WedeterminedthatbesidestheactualchemicalkineBcratecoefficients,thecharannealingmodelcoefficientswerethemostsensiBveparameters. BYUresearchersconstructedandcalibratedanew model for char-annealing (overwhelmingly the most sensiBve sub-model in thecomprehensivecharconversionmodel). Literaturedatawerecollected,potenBalformsofthenewmodelevaluated,andthenewmodeliscurrentlyinthefinalstagesoftesBng.

InternTroyHollandmadeapresentaBontoanIndustrialAdvisoryBoardatLosAlamostoadvisesimulaBon sciencegroup leaders in theelectricitygeneraBon industryon the capabiliBesandusefulnessofboththePSAAPcenterandtheCarbonCaptureSequestraBonIniBaBve(CCSI).

SootinCoalFlames

ArchessimulaBonsoftheoxy-firedcombustor(OFC)withthebasesootmodelwerecomparedwithpreviousRANSsimulaBons.Similar sootandtarconcentraBonswere foundbetweenthetwocases,aswellasthequalitaBveflamestructure. ValidaBontesBngofthesootmodelwasperformed by comparison to instantaneous line-of-sight soot measurements in the OFCperformed by Dale Tree at BYU. IniBal comparisons proved promising. New soot oxidaBonmodel was derived with parameters tuned to data gathered from eight different publishedexperimentsintheliterature.NewoxidaBonmodelincludesoxidaBonbymorespeciesthanO2only. NewsootgasificaBonmodelwasderivedwithparameterstunedtodatagatheredfromsix different published experiments in the literature. New soot gasificaBon model includesgasificaBonbybothCO2andH2O. Sootandtarcodedintopressuresolversusedbyinputfilesin the Arches solware. Parameters for both oxidaBon and gasificaBonmodels tuned using

Bayesian methods, allowing for beker quanBficaBon of uncertainty. Probability densityfuncBons derived for eachmodel parameter. Use of soot gasificaBonmodel eliminates thesmallamountsofsootobservedinfuel-leanregionsinbasesootmodel.

RadiaBon

WehavebeenexploringimprovementsinboththeDiscreteOrdinates(DO)radiaBonmodelandin theReverseMonteCaroRayTracing (RMCRT)model.The improvementsmadeduringyeartwo of this contract include both improved accuracy and improved efficiency. Theimprovements were olen made in the form of including mulBple opBons to opBmize theopBonsforthespecificscenario.

We began by developing a high-resoluBon 3D integral radiaBon benchmark for a domaincontaining hot and cold clouds to help quanBfy error in RMCRT formulB-grid development.NewradiaBonsolveropBonswereaddedforusewithdiscrete-ordinates.Thisincludedusingamoreefficient implementaBonof the interfacetothehypre libraryandthesheddingofsomelegacyFORTRANcode.GeometrycoarseningformulB-levelRMCRTwascoded.ThismulB-levelRMCRTalgorithmisintendedtoreduceerroraswellasimprovethespeedofthesolverelaBvetothesingle-levelapproach.

We have explored the viability of using Uintah infrastructure to handle a sweeping discreteordinates algorithm. Using the infrastructure/framework as is, we found sweeps to bemoreefficient up to a scale of approximately 16,000 processors. At this point the DO radiaBonsoluBon was more computaBonally efficient when Hypre was used for non-liner solves asopposedtothesweepingalgorithm.

DuringthisyearwehavecoupledtheRMCRTradiaBonmodeltotheenBremulB-physicsmodelwith only a few minor adjustments lel outstanding. We have improved RMCRT mulB-levelapproach and have shown that it now converges on the analyBcal soluBon for mulBplebenchmark problems. We found that RMCRT was making unnecessary approximaBons insampling the solid angle. Several alternaBveswereexploredand themosteffecBveapproachwasselected,greatlyreducingtheerrorsmadebyRMCRTincompuBngthefluxes.

Valida-on,Verifica-onandUncertaintyQuan-fica-on

VectorConsistency

The University of California-Berkeley team developed a new vector consistency measure fordatasetsinvolvinglinearandquadraBcsurrogatemodels.ThisconsistencymeasurecanbeusedtoidenBfyasmallnumberoflikelyincorrectasserBonsinaninconsistentdataset. Theytestedthe recently developed vector consistency measure on the GRI-Mech dataset for methanecombusBon,whichconsistsof77experimentsin102variablesandisinconsistent. ThevectorapproachidenBfies2experimentsascontribuBngtotheinconsistencyandsuggestsrelaxaBonsto resolve the dataset. These results confirm prior results found by uBlizing the scalarconsistencymeasure.

UCBdevelopedvariantsandfurthermodificaBonsofthevectorconsistencymeasureinordertobekeranalyzetheproperBesofadataset. Forinstance,introducingweighBngallowstheusertoincludescienBficopinionsregardingthereliabilityandaccuracyofcertainexperimentswhenresolving an inconsistent dataset. EvaluaBng the vector consistency measure for mulBple

weighBngconfiguraBonsprovidesdifferent relaxaBons, i.e.differentpaths toconsistency,andleads to interesBng tradeoff problems. This feature enables a much richer framework forconsistencyanalysis. Forexample, theweightedapproachreveals that theGRI-MechdatasetcanbemadeconsistentbyadjusBngonly1experiment.Amongnewfeatures,vectorconsistencynowallowsrelaxaBonstovariablebounds,accounBngfor inconsistencyduetoimproperpriorinformaBon.

They tested the vector consistencymeasure on the DLR dataset for syngas combusBon. Thiseffortwas conducted in collaboraBonwith combusBon scienBsts at DLR, Stukgart, Germany.The DLR dataset contains 83 experiments in 26 variables, with associated uncertainBes. Ourconsistency analysis demonstrates that this dataset is massively inconsistent — 53 of theexperimentalboundsneed tobeadjusted to reachconsistency. The results showed that thevector consistencymeasure can provide amore effecBve and insighuulway to solve datasetinconsistencyascomparedtoourprevioustools.

TheyconductedrandomtrialsofalinearprogrammingimplementaBonofvectorconsistency.Inthese trials, vector consistency was tested on hundreds of thousands of random systems oflinear inequaliBes with known errors. Vector consistency performs significantly beker thanseveralcomparablestrategies,butfailsinmanycasestoexactlyrecovertheerror.Theseresultsindicate the importanceof domain knowledge (i.e. vector consistencywithproperly assignedweights)inresolvinginconsistencyandhaveopenedseveralfurtheravenuesforinvesBgaBon.

Bound-to-BoundDataCollaboraBon

The UCB team is creaBng new, reworked version of the Bound-to-Bound-Data-CollaboraBon(B2BDC) toolbox. The capabiliBes being implemented include B2BDC calculaBons forconsistencyandpredicBonwithbothquadraBcandraBonal-quadraBcsurrogatemodels. Theyhave included an automated tool to render an inconsistent dataset into a consistent one byexpanding the dataset unit bounds detected from sensiBvity calculaBon. This methodology,however, is insufficient to address massively inconsistent datasets, such as the iniBal DLRexample, which can bemore effecBvely analyzed by vector consistency, to be available in afuturerelease.

UCB has developed heurisBc rules to minimize computaBon Bme in using "redundant"constraints to improve outer-bound results of SDP relaxaBons to nonconvex quadraBcprogrammingprograms. ThismethodhasbeentestedontheGRI-Mechdatasetforgas-phasemethanecombusBonandachieveda50%decreaseintheopBmalitygap.ThelatestreleaseofB2BDCimplementsthisstrategy.TheyhavetestedthenewfeaturesoftheB2BDCmethodologyin a collaboraBve study of H2/O2 combusBon system with several research groups incombusBon.

CCMSCCoalDatabase

TheUCBteamhasdevelopedanopen-sourcecoaldatabaseinthePrIMecyberinfrastructure;incollaboraBonwiththeUUandBYUteams,archiving1100datarecordsforcoaldevolaBlizaBon,charoxidaBonandnitrogenrelease.Experimentsspanacross269typesofsolidfuelsincludingfossil, chars,biomassandblends. Usinga cloud-based infrastructure,which supports crowd-sourcing, Sandia (Livermore) has contributed addiBonal data records including coal oxidaBonexperimentsfromtheirCombusBonResearchFacility.

TheyhavecreatedtutorialstosupportusersinconverBngexperimentaldataintothedatabaseformat.ThesetutorialswereusedbystudentsatBYUandUUwhenaddingalargesetofcharoxidaBonexperimentsfromSandia(Livermore). UCBcreatedfront-enduserapplicaBontotheCCMSCCoaldatabasetosupportusersbyallowingasimplewayoffindingrelevantcoaldata.ThisapplicaBonsupportseasybrowsing,datacomparison,plonng,and searchqueriesbasedon text andquanBtaBvedata. TheCCMSCCoalDatabaseApplicaBon is available through theprimekineBcs.orgwebsite.TheyaddedsearchrefinementcapabiliBestothefront-enddatabaseapplicaBon. This gives users the ability to filter the database by mulBple search queries,supporBngeasieraccesstorelevantexperimentaldatathatwasnotpreviouslyavailable.

ReducedOrderModelTools

UCB has created a tool to discriminate between models in QOI space using support-vectormachines. AforwardaddiBvealgorithmwasusedtofindsetsofQOIswheretwomodelsareoverlapping. These tools were applied towards the CPD and DAEM models of coaldevolaBlizaBondevelopedbyCCMSC.

ExperimentalCampaign

ThemajorityofourexperimentalefforthasconBnuedtofocusonthe1.5MWpulverizedcoaltestfacility(L1500) inordertoprovidetheL1500simulaBonteamwiththebestpossibledatasetfortheV/UQtask.EffortstocharacterizethethermalproperBesofthedepositscoaBngthevarious surfaces inside the L1500 conBnue. Limited data sets of total emissivity and thermaldiffusivityweretakenpriortothecampaigninYear1.MoreextensivesetsofdataweretakeninYear2alertheendoftheYear1campaign.Fourhundreddepositsampleswerecollectedina1’x1’gridfromtheburnerdowntotheheatexchangersecBon(adistanceof45feet).AnFTIRwasusedtomeasurespectralreflecBvityatroomtemperature,fromwhichthetotalemissivitywascalculated.TofurtherheightenourunderstandingofthecharacterisBcsofthedeposits,weaimtomeasuretheemissivityofthedepositsatarangeoftemperaturescomparabletothoseofthefurnaceinYear3.Thethermaldiffusivitymeasurementstakenina1’x1’gridforthefirstfourfeetofthereactorwererepeatedinYear2alerthecampaigntoprovideabeforeandalercomparisoninthedepositproperBes.RepeBBonswereincreasedinordertoaidunderstandingofvariance.

The infraredheatfluxmeasurements takenwith thehigh-speed, infraredcamera fromYear1were analyzed. The infrared heat flux was converted to total heat flux measurements tocompare with radiometer measurements. The trends of the two measurement techniquesmatchedasthecondiBonsvaried;however,themagnitudesofthecamerameasurementswereabout half of the radiometer measurements. This has led to an in-depth analysis of theassumpBonsmade in calibraBng both the camera and the radiometers and the assumpBonsmadeinbothinstrumentmodels.Workontheseinstrumentmodelsisongoing.

ExtensiveplanningandeffortshavegoneintopreparingfortheYear2,L1500campaign.MajormodificaBonstothefurnaceweremade,includingthereplacementofthefirstfourcoolingcoilsin SecBons 1 and 2 with large, 6”x36” rectangular ports. These ports aremodular, with theability to house quartz glass windows for greatly enhanced opBcal measurements (previousmeasurements weremade through ports with 2.5” diameter ports) or to house soot-blown,coolingpanelsthatareflushwiththereactorinterior.ThesepanelswillprovideabackgroundofknowntemperatureforanyopBcalmeasurements.Mucheffortwasmadetocalculatetheheat

losstothesewindows/panelsinordertodesigntheportsatasizewheretheheatlosswouldbesimilartothatoftheformercoolingcoils.DuetovariousunavoidabledelaysresulBngfromthefurnacemodificaBons,thecampaignforYear2willactuallytakeplaceinYear3,inJune2016.

Commentsfromthepeerreviewteamhaveledustodevelophighlydetailedinstrumentmodelsin an effort to beker quanBfy and characterize the uncertainty in our measurements. Thispursuithasledtoamuchmorefundamentalunderstandingofwhatexactlywearemeasuring,the inherent assumpBons associated with these measurements, and the validity of thoseassumpBons. We are currently in the process of creaBng models for the radiometers, thecoolingcoils,thenewlyinstalledcoolingpanels,andtheinfraredcamera.Thesemodelsincludemethodology forstudying indetail theproperBesandcalibraBonof thecomponentsof thesediagnosBc techniques: the thermocouples that measure the inlet and outlet watertemperatureson the L1500, themassflowmeters thatmeasure theflow rateof the coolingwaterintheL1500,thesurfacethermocouplesusedtomeasureheatfluxinthecoolingpanelsand ceiling of the L1500, and the blackbody source used to calibrate the radiometers andinfraredcamera.

BoilerSimulatorFacility

GiventheposiBonoftheAlstomBoilerSimulatorFacility(BSF)withintheV/UQhierarchy,theBSFplaysacrucial role invalidaBng the integratedphysicsof thiscenter.During the lastyearsignificant progress has beenmade in addingmore physics-basedmodels, to achieve bekerconsistency with our quanBBes of interest. The peer review process from year onerecommended:“…thatsomefocusbeplacedonhowtotransferproperBesofthedeposiBontothewallheaBngmodel.”TherehasbeenasignificantfocuswithinthephysicsandV/UQgroupsduringthelastyearinaddingappropriatemodelstorepresentthedeposiBonofash,andaddthecontribuBonoftheashlayertotheenergybalanceatthewall.InaddiBon,V/UQeffortsininvesBgaBng sensiBve parameters have highlighted the importance of the char oxidaBonreacBons.Accordingly,wehave implementedacharoxidaBonmodel,whichhastheability torepresent some of the physics, which were missing in the previous char oxidizaBon model.GiventheextentofworkinexploringsensiBveparametersintheBSF,andtheaddiBonofnewphysics,year2V/UQconsistencyanalysisprovedsuccessful.TheBSFshowedconsistencywith22 O2measurements, 96 temperaturemeasurements, and 25 heat fluxmeasurements. Theconsistentregionwasfoundwithashslaggingtemperaturebetween1490-1580K(experimentalvalue1513K),refractorythermalconducBvitybetween3.7and4.4W/m/K(experimentalvalue3.6 W/m/K), and char oxidaBon acBvaBon energy between 20,000 and 26000 cal/mol(experimentalvalue22,000cal/mol).Year1VUQprovidedaneffecBvethermalconducBvityforthewall,whichwas not applicable to the predicBon cases. Year 2 V/UQ showed consistencywith data given a physics based deposiBonmodel (and char oxidaBonmodel), and providedinsight into the input parameter uncertainty for ash slagging temperature, char oxidaBonacBvaBonenergy,andforthespecificrefractoryusedintheBSF.Therangeofconsistencyintheashslaggingtemperature,andacBvaBonenergycanbeuseddirectlyinthe8-cornerpredicBoncase.

Responsetoapplicableyear2reviewcomments:• ConsiderconBnuousrecordingduringcalculaBon(e.g.velocityfield)toobtaincharacterisBcsof turbulentfield.Obtainmeanprofilesacross themixing layer togetabekersenseof theflowfield.DothisasamakerofcoursetocollectvaluableinformaBon.Thiscanprovidebasis

forcomparisonbetweenyouruseofLESandRANSmodels thatyour industrialpartnersareusing.

• Itwas not clear howdata requirements forQOI evaluaBon comparewith requirements forgeneralgraphicsupport/visualizaBon,e.g.visdumpevery100-200Bmesteps.

During year 2 the physics team implemented a variable staBsBcs module in the producBoncode,whichhastheabilitytocollectvariablestaBsBcs,whichareintegratedinBme.Thisallowsus to analyze Bme-averages over anywindow in Bmewhile requiring very likle I/O.We areinterestedinanalyzingthecharacterisBcsoftheturbulentfieldasthereviewersrecommendinproviding a beker basis for comparison between LES and RANS. Using the variable staBsBcsmodule we are able to easily evaluate the QOI of our system, in compuBng Bme-averagedvalues, which include averaging staBsBcs for every Bme-step. The I/O frequency of 100-200Bme-stepsisusedforgeneraBngvolumerenderedmovies(notforBme-averagingQOI).

GEPower8-cornerUnitSimulaBons

GEPower’s8-cornerunitwas introduced to thecenterearly inyear2.The8-cornerunit is a1000MWultra-supercriBcalairfiredPCboiler.TheunitisparBcularlyinteresBngtothecenterbecauseofthepotenBalimpactthecenter’ssimulaBonscanhaveinworkingwithourindustrialpartner.Thedetailsofthegeometryforthe8-cornerunit,andrunningcondiBonsweresenttothecenterneartheendofyear2.PreparaBonworkforthesimulaBonhasincludedthecreaBonofabase-casecontainingalloftheperBnentgeometrydetails,andrunningcondiBons.InordertorunthesimulaBonatarealisBcscalealotofworkhasbeendoneingenngtheproducBoncodetoscaleintermsofradiaBonsolves,I/O,andincreasingthespeedofthecalculaBon.

Responsetoapplicableyear2reviewcomments:• PrioriBze the 8-corner simulaBons versus the design boiler given the availability ofexperimentaldata.

• IdenBfyriskmiBgaBonifneededcyclesarenotavailable.• Consider using a higher-order CFD method or document the adequacy of your low-ordermethod.

• DevelopaplanfordealingwiththelossofINCITEcycles.• You described two large runs at different resoluBons but performance of the runs - withrespecttoscalability-wasnotclear.• YouappeartobeaddressingperformanceissuesoneataBme,e.g.I/O,linearsolver.• Ourexperienceisbokleneckswillshilasyouscaleup.

There may have been a miscommunicaBon with regards to the 8-corner unit havingexperimentaldata.GEPowerhasprovideduswithvarious steam-sidemeasurements,butnomeasurementsweremadeonthecombusBonside.Assuch,the8-cornerunitsimulaBonsarepredicBonswithoutdata.

InordertomiBgateriskassociatedwithobtainingcycles,andwiththeimpendingcompleBonoftheINCITEaward,weareprofilingourcodeatscaletodeterminewhereaddiBonalsavingscanbemadetospeedupourcomputaBons.

Withregardstohigher-orderCFDmethods,wearecurrentlyreadingaboutthenLESmethodtoseeifthereareanysignificantbenefitscomparedwithourcurrentapproach.

Althoughbothof thepredicBon cases (8-cornerunit andoxy-coal boiler) are large, therearesignificantdifferencebetweenthegeometriesandsomeofthemodelsusedbetweenthecases.Assuch,thescalingperformancevariesbetweenthecases.InparBcular,the8-cornerunithashundredsofinternalintrusionsobjectsrepresenBngplatensandpipes,whereastheoxy-designboilerdesign istargeBngheatremovalonlythroughthewalls.These internal intrusionsaffectthefluidmechanicsandthermalboundariesofthesystemandasresultsthelinearsolveBmesforboth thepressure solve and theDO radiaBon solves appear to takemuch longer.Wearecurrentlyprofiling theproducBoncode todetermine if there is anything that canbedone todecreasethecomputaBonBmeassociatedwiththelinearsolves.

Awardstokeypersons• ValerioPascuccireceivedtheDisBnguishedMentorAwardfromTheGraduateSchooloftheUniversityofUtah,May6,2016.

• Valerio Pascucci received Best Paper Award 2016 from the IEEE Pacific VisualizaBonSymposium.

• Valerio Pascucci received the Integrated Partnership Award from DOE Federal LaboratoryConsoriumforExcellenceinTechnologyTransfer,April29,2015.

StudentsandPost-doctoralFellows• MichaelBrown,MScandidate,UniversityofUtah• OscarDiaz-Ibarra,Ph.D.candidate,UniversityofUtah• TeriDraper,Ph.D.candidate,UniversityofUtah• PavolKlacansky,Ph.D.candidate,UniversityofUtah• LaurenKolcynski,Ph.D.candidate,UniversityofUtah• JoshuaMcConnell,Ph.D.candidate,UniversityofUtah• JohnCamiloParraAlvarez,Ph.D.candidate,UniversityofUtah• StevePetruzza,Ph.D.candidate,UniversityofUtah• DuongThaiHoang,Ph.D.candidate,UniversityofUtah• WilliamUsher,Ph.D.candidate,UniversityofUtah• MinMinZhou,Ph.D.candidate,UniversityofUtah• TroyHolland,Ph.D.candidate,BrighamYoungUniversity• AlexJosephson,Ph.D.candidate,BrighamYoungUniversity• AndrewRichards,Ph.D.candidate,BrighamYoungUniversity• ArunHegde,Ph.D.candidate,UniversityofCalifornia-Berkeley• WenyuLi,Ph.D.candidate,UniversityofCalifornia-Berkeley• JimOreluk,Ph.D.candidate,UniversityofCalifornia-Berkeley• BabakGoshayeshi,Post-doctoralFellow,UniversityofUtah

Collaborators(unpaid)• JohnMarion-DirectorTechnologyandR&D,GEPower

• BobKunkel-ChiefEngineer,GEPower• ArmandLevaser-PrincipleInvesBgatorBSF,GEPower• TracyMidgley-PrincipalConsulBngEngineer,GEPower• PaulChapman-CFDTechnologyManager,GEPower• DavidSloan-CFDspecialist,GEPower• Yen-MingChen-CFDspecialist,GEPower• ZhenyuanLiu(UCBerkeley),• JeromeSacks(NISS)• RuiPaulo(UniversityofLisbon,Portugal)• GonzaloGarcia-Donato(UniversityofCasBlla-LaMancha,Spain)• CraigNeedham(NCStateUniversity)• SathyaBaskaranandManiSarathy(KAUST,SaudiArabia)• MichaelBurke(ColumbiaUniversity)• RichardWest(NortheasternUniversity)• PhillipWestmoreland(NCStateUniversity)• NadjaSlavinskaya,JanStarckeandUweRiedel(DLR,Stukgart,Germany)• StanislavShvartsmanandHenryManngly(PrincetonUniversity)

Publica-onsandPresenta-ons

PublicaBons• Philip, S., B. Summa, J. Tierny, P. Bremer, and V. Pascucci. “Distributed seams for gigapixelpanoramas.” VisualizaBon and Computer Graphics, IEEE TransacBons on, 21(3):350-362,March2015.

• Liu,S.,B.Wang,J.J.Thiagarajan,P.-T.Bremer,andV.Pascucci.“High-dimensionalvisualizaBon:Visual exploraBon of high-dimensional data through subspace analysis and dynamicprojecBons.”ComputerGraphicsForum,34(3):271-280,June2015.

• Edwards, J., E. Daniel, V. Pascucci, and C. Bajaj. “ApproximaBng the generalized Voronoidiagramofcloselyspacedobjects.”ComputerGraphicsForum,34(2):299-309,May2015.

• Bremer,P.-T., D.Maljovec,A.Saha,B.Wang,J.Ganey,B.Spears,andV.Pascucci.“ND2AV:N-dimensional data analysis and visualizaBon analysis for the naBonal igniBon campaign.”CompuBngandVisualizaBoninScience,17(1):1{18,2015.

• Summa,B.,A.A.Gooch,G.Scorzelli,andV.Pascucci.“Paintandclick:UnifiedinteracBonsforimageboundaries.”ComputerGraphicsForum,34(2):385-393,May2015.

• BhaBa,H.,B.Wang,G.Norgard,V.Pascucci,andP.-T.Bremer.“Local,smooth,andconsistentJacobisetsimplificaBon.”ComputaBonalGeometry:TheoryandApplicaBons,48(4):311-332,2015.

• Earl, C., M. Might, A. Baguseky, J.C. Sutherland. (2016) “Nebo: An efficient, parallel, andportable domain-specific language for numerically solving parBal differenBal equaBons.”JournalofSystemsandSolware.(Inpress)doi:10.1016/j.jss.2016.01.023

• Saad, T., and J.C. Sutherland, "Wasatch: an Architecture-Proof MulBphysics DevelopmentEnvironmentusingaDomainSpecificLanguageandGraphTheory,"JournalofComputaBonalScience(2016).Inpress.doi:10.1016/j.jocs.2016.04.010

• Gyulassy,A.Knoll,K.C.Lau,B.Wang,P.-T.Bremer,M.E.Papka,L.A.CurBss,andV.Pascucci.“IntersBBaland interlayer iondiffusiongeometryextracBon ingraphiBcnanospherebakerymaterials.”IEEETrans.Vis.Comput.Graph,22(1):916-925,2016.

• BigDataandHighPerformanceCompuBng.LucioGrandinen,GerhardJoubert,marcelKunze,ValerioPascucci,editors.IOSpress.October,2015.

• McConnell,J.,&Sutherland,J.C.(2015).“TheEffectofModelFidelityonPredicBonofCharBurnout for Single-ParBcle Coal CombusBon.” InWestern States SecBon of the CombusBonInsBtute(pp1-15).Provo,UT.

• Cyr,E.C.,E.Phipps,M.A.Heroux,J.Brown,E.T.Coon,R.C.Kirby…R.Trok.(2015)“Algorithmsand AbstracBons for Assembly in PDE Codes: Workshop Report.” SAND2015-1379,Albuquerque,NM.

• Skraba,P.,P.Rosen,B.Wang,G.Chen,H.BhaBa,andV.Pascucci.“CriBcalpointcancellaBonin3dvectorfields:Robustnessanddiscussion.”IEEETrans.Vis.Comput.Graph,May,2016.

• Richards, A. and T.H. Fletcher. “A Comparison of Global KineBc Models for CoalDevolaBlizaBon.”(Inreview)

• Holland,T.,T.H.Fletcher,etal. “BayesianUncertaintyQuanBficaBonandCalibraBonofCharThermalAnnealing.”(InpreparaBon)

• Holland, T., T.H. Fletcher, et al. “Global SensiBvity Analysis for a Comprehensive CharConversionModelinOxy-fuelCondiBons.”(InpreparaBon)

• Josephson,A.,D.O.Lignell,A.Brown,andT.H.Fletcher.“RevisionstoModelingSootDerivedfromPulverizedCoal.”(InpreparaBon)

• Frenklach,M., A.Packard, J.Sacks,R.Paulo,andG.Garcia-Donato.“Comparisonof staBsBcand determinisBc frameworks of uncertainty quanBficaBon.” SIAM/ASA J. UncertaintyQuanBficaBon(Inpress).

• Oreluk,J.,C.D.Needham,S.Baskaran,S.M.Sarathy,M.P.Burke,R.H.West,M.FrenklachandP.R.Westmoreland.“DynamicChemicalModel forH2/O2CombusBonDevelopedThroughaCommunityWorkflow.”(InpreparaBon)

• Slavinskaya, N.A., J.H. Starcke, M. Auyelkhankyzy, U. Riedel, W. Li, J. Oreluk, A. Hedge, A.PackardandM. Frenklach. “Developmentof anUQ-PredicBveChemicalReacBonModel forSyngasCombusBon.”(InpreparaBonforsubmissiontoEnergy&Fuels)

PresentaBons• Holland,T.andT.H.Fletcher.“CoalParBcleCombusBon.”PosterpresentedattheStewardshipScienceAcademicPrograms(SSAP)Symposium,March11-12,2015,SantaFe,NM.

• Richards,A.andT.H.Fletcher.“Amodifiedtwo-stepmodelofdevolaBlizaBon.”Presentedatthe 15th InternaBonal Conference on Numerical CombusBon, April 19-22, 2015, Avignon,France.

• Lignell,D.O.,B.J.Isaac,A.Josephson,T.H.FletcherandJ.Thornock."LargeeddysimulaBonofsoot formaBon inanoxy-coalcombustor."Presentedat15th InternaBonalConferenceonNumericalCombusBon,April19-22,2015,Avignon,France.

• Josephson,A.J.,B.J.Isaac,D.O.LignellandT.H.Fletcher."LargeeddysimulaBonofanoxy-coalcombustor."Presentedatthe9thMeeBngoftheU.S. JointSecBonsoftheCombusBonInsBtute,May17-20,2015,CincinnaB,OH.

• Richards, A. and T. H. Fletcher. “A Comparison of Global KineBc Models for CoalDevolaBlizaBon.” Presented at the Western States SecBon of the CombusBon InsBtute,October5-6,2015,BrighamYoungUniversity,Provo,UT.

• Holland, T. and T.H. Fletcher. “Comprehensive Char ConversionGlobal SensiBvity Analysis.”Presented at the Western States SecBon of the CombusBon InsBtute, October 5-6, 2015,BrighamYoungUniversity,Provo,UT.

• Fry,A.,J.SpinB,I.Preciado,O.Diaz-IbarraandE.G.Eddings.“Pilot-scaleinvesBgaBonofheatflux, radiaBon and CO distribuBon from an oxy-coal flame.” Paper presented at the 2015American InsBtuteofChemical Engineering (AIChE)AnnualMeeBng,November9-13, 2015,SaltLakeCity,UT.

• Fry,A.,J.SpinB,I.Preciado,O.Diaz-IbarraandE.G.Eddings.“Pilot-ScaleInvesBgaBonofHeatFluxandRadiaBon fromanOxy-coalFlame.”PresentedatClearwaterCoalConference,May31-June4,2015,Clearwater,FL.

• Diaz-Ibarra,O., J. SpinB,A.Fry, J.N.Thornock,B. Issac,D.Harris,M.Hradisky,S.Smith,E.G.EddingsandP.J.Smith.“AvalidaBon/uncertaintyquanBficaBonanalysisofa1.5MWoxy-coalfired furnace.”Paperpresentedat theAmericanFlameResearchCommikee2015 IndustrialCombusBonSymposium,September9-11,2015,SaltLakeCity,UT.

• Fry,A., J.SpinB, I.Preciado,O.Diaz-IbarraandE.G.Eddings.“Pilot-scale invesBgaBonofheatfluxandradiaBonfromanoxycoalflame.”PaperpresentedattheAmericanFlameResearchCommikee2015IndustrialCombusBonSymposium,September9-11,2015,SaltLakeCity,UT.

• Fry, A., J. SpinB, I. Preciado, O. Diaz and E.G. Eddings. “Heat Flux, RadiaBon and CODistribuBonsfroma1.5MWOxy-CoalFlame."PresentaBonatthe2015AmericanInsBtuteofChemicalEngineering(AIChE)AnnualMeeBng,November9-13,2015,SaltLakeCity,UT.

• Draper,T.,E.G.EddingsandT.Ring.“ThermalCharacterizaBonofAshDeposits ina1.5MWReactor."PresentaBonatthe2015AmericanInsBtuteofChemicalEngineering(AIChE)AnnualMeeBng,November9-13,2015,SaltLakeCity,UT.

• Fry,A.,J.SpinB,I.Preciado,O.DiazandE.G.Eddings.“PredicBngHeatTransferCharacterisBcsofa1.5MWthOxy-CoalFlame."Paperpresentedat5thMeeBngoftheIEAGHGInternaBonalOxyfuelCombusBonResearchNetworkMeeBng,October27-30,2015,Wuhan,China.

• Diaz-Ibarra,O., J. SpinB,A.Fry,B.Schroeder, J.Thornock,B. Issac,D.Harris,M.Hradisky,S.Smith,E.Eddings,andP.Smith.“AValidaBon/UncertaintyQuanBficaBonAnalysisofa1.5MWOxy-CoalFiredFurnace."PaperpresentedattheAmericanFlameResearchCommikee(AFRC)2015IndustrialCombusBonSymposium,September9-11,2015,SaltLakeCity,UT.

• Fry,A., J. SpinB, I.Preciado,O.DiazandE.G.Eddings. “Pilot-scale InvesBgaBonofHeatFluxand RadiaBon from an Oxy-coal Flame." Paper presented at the American Flame ResearchCommikee(AFRC)2015 IndustrialCombusBonSymposium,September9-11,2015,SaltLakeCity,UT.

• Fry,A.,J.SpinB,O.Diaz,I.PreciadoandE.G.Eddings.“PredicBngHeatTransferCharacterisBcsof a 1.5 MWth oxy-coal Flame." Paper presented at The 40th InternaBonal TechnicalConferenceonCleanCoal&FuelSystems,May31-June4,2015,Clearwater,FL.

• Draper,T.,P.Toth,T.RingandE.Eddings.“OpBcalheatfluxandtemperaturemeasurementsona100kWoxy-fuelcombustor."Paperpresentedatthe7thEuropeanCombusBonMeeBng(ECM2015),March30-April2,2015,Budapest,Hungary.

• Isaac,B.,J.Thornock,S.T.SmithandP.Smith.“SimulaBonandValidaBonof15MwthOxy-CoalPowerBoiler.” Paper presented at 2015American InsBtute of Chemical Engineering (AIChE)AnnualMeeBng,November9-13,2015,SaltLakeCity,UT.

• Sutherland,J.C.,J.McConnell,andB.Goshayeshi.“AnAssessmentandComparisonofVariousCoal CombusBon Models.” Paper presented at First InternaBonal Oxyflame Workshop,February10-11,2016,Montabaur,Germany.

• Saad, T., C. Earl, A. Baguseky, M. Might, and J.C. Sutherland. (2015) “Uintah/Wasatch:Addressing MulBphsyics Complexity in a High-Performance CompuBng Environment.”Presented in SIAMComputaBonal ScienceandEngineeringConference,March14-18, 2015,SaltLakeCity,UT.

• Saad T., A. Baguseky and J.C. Sutherland. “Wasatch: A CPU/GPU-Ready MulBphysics CodeUsing a Domain Specific Language.” Poster presented in SIAM ComputaBonal Science andEngineeringConference,March14-18,2015,SaltLakeCity,UT.

• Saad, T., and J.S. Sutherland. “Nebo: an EmbeddedDomain-Specific Language for Plauorm-AgnosBc PDE Solvers.” Presented at the 24th InternaBonal Conference on ParallelArchitecturesandCompilaBonTechniques,October18-21,2015,SanFrancisco,CA.

• Saad,T.andJ.C.Sutherland. “Wasatch:AddressingMulBphysicsandHardwareComplexityina High-Performance CompuBng Environment, in Workshop on Solware DevelopmentEnvironments for High-Performance CompuBng.” Presented at the 24th InternaBonalConference on Parallel Architectures and CompilaBon Techniques, October 18-21, SanFrancisco,CA.

• Frenklach, M. and A. Packard. “Data-Model System Analysis and PredicBon underUncertainty.” Princeton University, Department of Chemical and Biological Engineering,October7,2015,Princeton,NJ.

• Frenklach, M. and A. Packard, “AcBve Subspace IdenBficaBon and UBlizaBon,” West CoastROMWorkshop,SandiaNaBonalLaboratories,November19,2015,Livermore,CA.

• BhaBa,H.,A.G.Gyulassy,V.Pascucci,M.Bremer,M.T.Ong,V.Lordi,E.W.Draeger,andJ.E.P.andqPeer-TimoBremer.“InteracBveexploraBonofatomictrajectoriesthroughrelaBve-angledistribuBon and associated uncertainBes.” In Proceedings of IEEE Pacific VisualizaBonSymposium.IEEEComputerSociety,2016.

• Landge,A.,S.Kumar,V.Pascucci,I.Rodero,M.Parashar,andP.-T.Bremer.“EvaluaBonofin-situanalysisstrategiesatscale forpowerefficiencyandscalability.” InProceedingsof16th IEEE/ACM InternaBonal Symposium on Cluster, Cloud and Grid CompuBng (CCGRID). IEEEComputerSociety,2016.

• Maljovec,D.,B.Wang,P.Rosen,A.Alfonsi,G.Pastore,C.RabiB,andV.Pascucci.“Topology-inspired parBBon-based sensiBvity analysis and visualizaBon of nuclear simulaBons.” InProceedingsofIEEEPacificVisualizaBonSymposium.IEEEComputerSociety,2016.

• Pascucci,V.InvitedtalkattheSalishanConference,“ExtremeDataManagementAnalysisandVisualizaBonforExascaleSupercomputersApril28,2016.

• Pascucci,V.PlenarytalkatXSEDE15:"ExtremeDataManagement,AnalysisandVisualizaBon:ExploiBngLargeDataforScienceDiscovery.”July28,2015,St.Louis,MO.

• Valerio Pascucci, Plenary talk at XSEDE 15: "Extreme Data Management, Analysis andVisualizaBon:ExploiBngLargeDataforScienceDiscovery.”July28,2015,St.Louis,MO.

• Livnat, Y. Invited talk at The Salishan Conference on HIGH-SPEED COMPUTING: “TowardsInteracBveAnalysisandExploraBonoftheHPCPerformanceLandscape.”April30,2015.

• Landge,A. PosteratTheSalishanConferenceonHIGH-SPEEDCOMPUTING:“ScalabilityandPowerEfficiencyofIn-SituAnalysisWorkflows”April30,“2015.