center for hybrid rocket exascale simulation technology
TRANSCRIPT
![Page 1: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/1.jpg)
Center for Hybrid Rocket Exascale Simulation Technology
Team: Varun Chandola, James Chen, Paul DesJardin, Matt Jones, Matt Knepley, AbaniPatra, and Mark SwihartTST: John Hewson (SNL), Bob Anderson (LLNL), Marianne Francois (LANL), Fady Najjar (LLNL), Sriram Swaminarayan (LANL), Greg Weirs (SNL) NNSA Lab Collaborators: Greg Burton (LLNL) Matthew McNenly (LLNL), Roger Pawlowski (SNL), Tom Smith (SNL), Jim Stewart (SNL), Cosmin Safta (SNL), Brian Williams (LANL)
![Page 2: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/2.jpg)
Team Members MAE CBE
CSE CCR
Paul DesJardin James Chen
Varun ChandolaMatt Knepley Matt Jones Abani Patra
Mark Swihart
Tufts
Letitia Thomas
SEAS
![Page 3: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/3.jpg)
Team MembersProfessional Staff
• 3 post doctoral scholars• 6 Ph.D. students, 3 UG
students
Software Architect (identified)
Rubik Asatryan(chemistry)
Marianne Sullivan(Administration Support)
Louis Llanos
Kenny Budzinski
Jake Henry Sophia Matla Nick McNallyGabe Surina
Mae Sementilli Venoos Amiri
Darsh Nathawani
![Page 4: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/4.jpg)
Outline
• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Technical Challenges
![Page 5: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/5.jpg)
Outline
• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Technical Challenges
![Page 6: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/6.jpg)
NASA Peregrine Motor
Advantages• Safe – no TNT equivalent• Cost effective for handling, storing, and
environmentally friendly
• Throttled like a liquid but has energy density of a solid
Disadvantages• Historically low regression rate and
therefore lower specific impulse
• Difficulty with multiple port designs
• Scaling less understood because of turbulent boundary layer
• O/F shift with constant ox mass flux
Cantwell, B., "Aircraft and Rocket Propulsion", Stanford University, 2019, Ch. 11 Hybrid Rockets
Application: Hybrid Rocket Motors
![Page 7: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/7.jpg)
• Response of ablating solid fuels – Phase transformations and mechanical deformations– Liquid layer formation, growth and atomization– Mixed modes of burning, individual droplet vs. group combustion– Mixed modes of heat transfer from reacting slurry to surface
Cantwell, B., "Aircraft and Rocket Propulsion", Stanford University, 2019, Ch. 11 Hybrid Rockets
Hybrid Rocket: High Regression Rate
![Page 8: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/8.jpg)
• Turbulent reacting multiphase
interfaces
– Shear-induced liquid layer
instability growth / atomization
– Secondary atomization
processes and droplet burning
– Gas phase combustion
• Coupled thermal and mechanical
response of fuel grain
– Heat transfer of the melting fuel
– Dynamically changing interface
Gas
Solid
Liquid
Liquid
&
Gas
Turbulent Oxidizer
Simulation
Slab Burner Exp. (20k fps)
![Page 9: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/9.jpg)
Wakes
BL’sGrid
Channels
Cut off limit ~1/D
Resolved scales
Modeled scales
Wave number
Ener
gy s
pect
ra
Curran, Henry. “Developing detailed chemical kinetic mechanisms for fuel combustion.” Proceedings of the Combustion Institute 37 (2019): 57-81
Turbulent Length Scales Chemical Reactions
DOF Scaling Challenges
![Page 10: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/10.jpg)
Ti
me
Sca
le, s
econ
ds
Engineering Scale of Interest
Convective Transport Nozzle
Expan
sion
10-10
102
10-8
10-6
10-4
10-2
100
Length Scale, meters
10-12
10-9
10-6
10-3
100
103
Fuel Decomposition & Liquid Formation
Wave Instability Growth & Shear Atomization
Paraffin wax, ~ C35H72
Multiphase C
ombustion
Scaling Challenges
![Page 11: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/11.jpg)
Ti
me
Scal
e, s
econ
ds
10-10 102
10-8 10-6
10-4 10-2
100
Length Scale, meters
10-12
10-9
10-6
10-3
100
103
For every decade in length scale: - 3 orders of magnitude increase
in nodes (for uniform mesh) - 1-2 order of magnitude increase
in time steps
106 cells 103 steps (megaflop)
109 cells 104 steps (petaflop)
1012 cells 105 steps (teraflop)
1015 cells 106 steps (exaflop)
Subgrid Scale (SGS) Modeling
Exascale SGS Modeling Development
Scaling Challenges
![Page 12: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/12.jpg)
ML Driven Model ReductionTurbulence Chemistry
• Identifying optimal flow regime (SGS Model) for LES
• Using Gaussian Process Classification• Non-linear models• Able to propagate input
uncertainty to output• Trained from DNS-LES-regime
outputs
• Mapping high-dimensional data to low-dimensional non-linear manifold
• Flamelet Generated Manifolds (FGM)
Z
Temperature(K)
0 0.2 0.4 0.6 0.8 1300
600
900
1200
1500
1800
2100
2400 Zst = 0.149Λ = Cst
x (m)Temperature(K)
0 0.003 0.006 0.009 0.012 0.015300
600
900
1200
1500
1800
2100
2400
2700
3000 t = 0st = 7.69e-4st = 5.09e-3st = 0.015st = 0.029st = 0.052st = 0.086st = 0.242s
Manifold prior for LES
![Page 13: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/13.jpg)
Outline
• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Challenges
![Page 14: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/14.jpg)
Data Development and ExperimentsSource / Activity Scale (m) Measurements V&V Outcomes
fuel characterization 10-9 - 10-4 wax viscosity, vapor pressure, specific heat, conductivity, etc.
burning droplet models
slab motor / DNS 10-4 - 10-2 regression rates, solid/gas temp., particle dynamics
chemical kinetics, atomization, droplet dynamics
sounding rocket / LES 10-2 - 10-1 specific impulse, coef. of thrust, characteristic velocity
LES near-wall subgridscale (SGS) models
NASA Perigrine / VLES 10-1 - 101 specific impulse, coef. of thrust, characteristic velocity
VLES boundary layer subgrid scale (SGS) models
Data Gathering/Experiments guided by QOI UQ outcomes
![Page 15: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/15.jpg)
Physical Property Data Sources
Melting point, vapor pressure, viscosity, thermal conductivity, heat capacity, enthalpies and entropies of
formation, fusion, and vaporization
Literature
Quantitative Structure-Property Relations
Computation
Experiment
![Page 16: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/16.jpg)
Slab Burner ExperimentGlow Plug Ignition (60 fps) Wollaston Prism Interferometry (8k fps)
Two-color Pyrometry
![Page 17: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/17.jpg)
Slab Exp: Regression Rates
Wax raw image during burn Binary image of wax sample
• RAW image converted to binary• Height of wax sample from each image• Difference in heights divide by time
interval• Obtain experimental global and local
regression rates
Change in height over time for and oxidizer mass flux of 14.959 k'/)*+
2% Error
Global Regression Rate
Local Regression Rate
![Page 18: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/18.jpg)
Slab Exp: Two-Color PyrometrySoot Formation Model
T = 2800K
• Ratio of detector signal at two narrowband wavelengths
! =#$( 1'( −
1'*)
,n.('*, !).('(, !) + ,123('()23('*) + ,1
'*'(
4+ ,1 53,(53,*
• Radiometric self-calibration using low and high temperature black body sources
≈ 7879
Ray-Tracing
Camera Response Function
3-D Reconstructed Temperature Profile
*Aphale, Siddhant S., and Paul E. DesJardin. "Development of a non-intrusive radiative heat flux measurement … based two-color pyrometry." Combustion and Flame 210 (2019): 262-278.
Radiative Heat Flux
![Page 19: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/19.jpg)
Slab Exp: Thermocouples & Heat Flux GaugesEmbedded thermocouples
Response time: 3msHukseflux Schmidt-Boelter gauge
Response time: 250msSacrificial micro heat flux gauge
(Thin skin calorimeter)
10 m
m
10 mm
Radiative heat flux comparison
!. # $$
Thermocouple and micro heat flux gauge location
Thermocouple locations
Micro heat flux gauge
Front
Middle
Back
![Page 20: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/20.jpg)
Slab Exp: Particle TrackingSlab Burner – Phantom Camera (11k fps)
Droplet - Particle Tracking (13k fps)
Droplet Velocity
Droplet Diameter
![Page 21: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/21.jpg)
Outline
• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Challenges
![Page 22: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/22.jpg)
Exascale Challenge
The Main Challenge is to run the entire analysis pipline at
scale.
![Page 23: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/23.jpg)
Exascale Challenge
The Main Challenge is to run the entire analysis pipline at
scale.
![Page 24: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/24.jpg)
Exascale Challenge
The Main Challenge is to run the entire analysis pipline at
scale.
![Page 25: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/25.jpg)
Exascale PlanA library compatibility layer will enable low-level optimizations:• Support Vendor Standards (CUDA, SYCL, HIP)• Leverage Vendor Libraries (cuBLAS/cuSparse, ACML, MKL, Kokkos)• Node-aware Communication (batch messages, use PDE structure)• GPU-aware Communication (custom message pack/unpack)
Composable abstractions will enable high-level optimizations:• Develop library of ML primitives (Isomap, MSVM, k-means, DNN surrogates)• Computation-Aware Partitioning (for heterogeneous machines)• Robust solvers for multiphysics (FAS, nonlinear elimination, bootstrap)• Optimization and Eigensolves (one-shot multigrid, bootstrap, integrated design)
![Page 26: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/26.jpg)
Outline
• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Challenges
![Page 27: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/27.jpg)
To build composable libraries,
based on PETSc,
for the entire analysis pipeline.
Software Plan
![Page 28: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/28.jpg)
CCR DeploymentCCR scalable development pipeline enables:• rapid prototyping on desktop,• deployment on CCR resources,• scaling up to national facilities.
CCR provides early access to:• XMS/XDMoD metrics service could be used for arch-aware
partitioning• “Ookami” ARM64-SVE testbed
![Page 29: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/29.jpg)
Why Libraries?
Libraries Hide
Hardware Details
![Page 30: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/30.jpg)
Why Libraries?
Libraries Hide
Hardware Details
![Page 31: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/31.jpg)
Why Libraries?
Libraries Hide
Hardware Details
![Page 32: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/32.jpg)
Why Libraries?
Libraries Hide
Implementation Complexity
![Page 33: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/33.jpg)
Why Libraries?
Libraries Hide
Implementation Complexity
![Page 34: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/34.jpg)
Why Libraries?
Libraries Hide
Implementation Complexity
![Page 35: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/35.jpg)
Why Libraries?
Libraries Disseminate
Project Advances
Validated Ablation and Droplet Burning Models
Scalable Isomap and Physics-Based Encoders
Scalable Nonlinear Solvers
![Page 36: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/36.jpg)
Why PETSc?• Dependability & Maintainability
• Portability & Robustness
• Performance & Scalability
• Optimality & Robustness
• Flexibility & Extensibility
![Page 37: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/37.jpg)
Why PETSc?• Dependability & Maintainability
• Nearly 30 years of continuous development• Portability & Robustness
• Performance & Scalability
• Optimality & Robustness
• Flexibility & Extensibility
![Page 38: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/38.jpg)
Why PETSc?• Dependability & Maintainability
• Portability & Robustness• Tested at every supercomputer installation and 10,000+ users
• Performance & Scalability
• Optimality & Robustness
• Flexibility & Extensibility
![Page 39: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/39.jpg)
Why PETSc?• Dependability & Maintainability
• Portability & Robustness
• Performance & Scalability• Many Gordon Bell Winners
• Optimality & Robustness
• Flexibility & Extensibility
![Page 40: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/40.jpg)
Why PETSc?• Dependability & Maintainability
• Portability & Robustness
• Performance & Scalability
• Optimality & Robustness• State-of-the-Art Linear and Nonlinear Solvers, Eigensolvers,
Optimization Solvers, Timesteppers• Flexibility & Extensibility
![Page 41: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/41.jpg)
Why PETSc?
• Dependability & Maintainability
• Portability & Robustness
• Performance & Scalability
• Optimality & Robustness
• Flexibility & Extensibility• More than 8000 citations and hundreds of application packages• Aerodynamics, Arterial Flow, Corrosion, Combustion, Data Mining,
Earthquake Mechanics, . . .
![Page 42: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/42.jpg)
Outline
• Center Vision Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Challenges
![Page 43: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/43.jpg)
Center Overview
Data Development
- Experiments- Exascale DNS
- Small, Intermediate and
System level milestones
Field Modeling
- DNS, LES, VLES- a posteriori ML
based SGS assessment
Computational Framework- PETSc Exascale
simulation framework
Prediction Assessment- ML based UQ
- a priori ML based SGS assessment
- V & V
Center for Exascale Simulation of Hybrid Rocket Motors
Experimental Design
Multiscale modeling & simulation
Large scale system-level testing
Small scale slab experiments
Uncertainty Quantification
DNSv.s.UQ
model
ML
DynamicSGS
model
LES
Slab motorexperiment
Validation
Exascale Sim.
![Page 44: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/44.jpg)
Roadmap
Years 1-2 (Small spatial/temporal scale – DNS – Slab motor)
Years 2-4 (Intermediate spatial/temporal scale – LES – Sounding rocket)
![Page 45: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/45.jpg)
Roadmap
Years 4-5 (Full spatial/temporal scale – VLES – Peregrine rocket motor)
Peregrine sounding rocket test at NASA Ames
![Page 46: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/46.jpg)
Outline
• Center Vision and Predictive Science • V&V/UQ Plan• Exascale / CS Plan • Software Plan• Integration Plan• Key Challenges
![Page 47: Center for Hybrid Rocket Exascale Simulation Technology](https://reader031.vdocuments.net/reader031/viewer/2022012512/618b1155cbe8ec5b60723f6f/html5/thumbnails/47.jpg)
Key Challenges• COVID-19 impediments which impacts all aspects of the
center beyond “normal challenges”• Technical Challenges
– Resolution requirements for resolving liquid fuel instability and atomization processes
– Soot kinetics with wax/GOX systems, what level of detail is adequate to sufficiently describe conjugate heat/mass transfer processes?
– Scaling of mixed Eulerian/Lagrangian formulations on exascaleplatforms with mixed hardware requirements
– Classification of burning regimes, fidelity and robustness of ML inspired FGM
– Propagation of uncertainty from 1D models into multi-dimensional contexts
– V&V using optical measurements, e.g., two-color pyrometry, Schlieren and other spectroscopy measurements