petascale direct numerical simulations of turbulent ... overview project title petascale direct...

Petascale Direct Numerical Simulations ofTurbulent Channel Flow

MyoungKyu [email protected]

Department of Mechanical EngineeringUniversity of Texas at Austin

ESP Meeting May, 2013

M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 1 / 30

Contents

Project Overview

Performance Optimization

Early Result

Conclusion


Project Overview

Project Title◮ Petascale Direct Numerical Simulations of Turbulent Channel Flow

Goal◮ Expanding our understand of wall-bounded turbulence

Personnel◮ P.I. : Robert Moser◮ Primary Developer : M.K.Lee◮ Software Engineering Support : Nicholas Malaya◮ Catalyst : Ramesh Balakrishnan


Turbulent Flow

We are living with turbulent flows.◮ Atmosphere◮ Transportation (Automobile, Airplane)◮ Blood flow in the heart

Directly related with energy loss by drag of fluid motion

Deterministic Chaos◮ Governing equation : Navier-Stokes equation◮ Analytic solution is unknown◮ Highly nonlinear / Unpredictable◮ Requires statistical approach

"The most important unsolved problem of classical physics" byRichard Feynman


Overlap Region

Connection between near-wall region and outer region

Not understood as well as near-wall region


History

1 10 100 1000y+

0

1

2

3

4

5

6

7y+

(dU

+/d

y+)

Re_tau = 180Re_tau = 590Re_tau = 950Re_tau = 2003

1 10 100 1000y+

0

5

10

15

20

25

30

U+

Re_tau = 180Re_tau = 590Re_tau = 950Re_tau = 2003

Reτ = 180 : Kim, Moin & Moser, 1987

Reτ = 590 : Moser, Kim & Mansour, 1999

Reτ = 940 : Del Álamo, Jiménez, Zandonade & Moser, 2004

Reτ = 2003 : Hoya & Jiménez, 2006


Computation of Overlap Region

High Reynolds(Re) number is required to obtain logarithmic layer.

Even with existing channel flow DNS data at the high Re,Reτ ≈ 2000, the region of log layer is small.

To expand our understanding of log layer, DNS data atReτ > 4000 ∼ 5000 is required.

Cost scales with Re4

Simulation at high Re is limited by computational power.

BG/Q enables us to simulate such a high Re flow


Simulation Geometry

Flow between two infinite parallel plates

Reτ ≈ 5000

Periodic boundary condition in streamwise / spanwise directions

No slip, no mass flux boundary conditions at wall


Simulation Geometry

Nx = 10,240 (Uniform) - DFT

Ny = 1,536 (Stretched, denser near wall) - B-SplineNz = 7,680 (Uniform) - DFT

242 Billion D.O.F.


Incompressible Navier-Stokes Equation

∂ui

∂t= −

∂P∂xi

+ Hi + ν∂2ui

∂xj∂xj,

∂uj

∂xj= 0

where

H1 = −

(

∂u2

∂x+

∂uv∂y

+∂uw∂z

)

H2 = −

(

∂uv∂x

+∂v2

∂y+

∂vw∂z

)

H3 = −

(

∂uw∂x

+∂vw∂y

+∂w2

∂z

)


Formulation (Kim, Moin & Moser, 1987)

Applying Laplacian and Curl operator

∂ωy

∂t= hg + ν∇2ωy

∂φ

∂t= hv + ν∇2φ

where

φ = ∇2v , At wall ωy = 0, v =

∂v∂y

= 0

hg =∂H1

∂z−

∂H3

∂x, hv =

∂2H2

∂x2 +∂2H2

∂z2 −∂2H1

∂x∂y−

∂2H3

∂y∂z

Two equations in the same form

No pressure term!!


Formulation (Spalart, Moser, & Rogers, 1991)

Time discretization : Low-Storage RK3 method

u′ = un +∆t[

L(

α1un + β1u′)

+ γ1Nn]

u” = u′ +∆t[

L(

α2u′ + β2u”)

+ γ2N ′ + ζ2Nn]

un+1 = u” + ∆t[

L (α3u” + β3un+1) + γ3N” + ζ3N ′]

Need only two additional fields

L : Linear operator (Laplacian) → Implicit + Explicit

N : Nonlinear Terms (hg ,hv ) → Explicit


Simulation Process

Periodic boundary condtion in X-dir and Z-dir◮ Fourier Transform◮ ∂/∂x → ikx , ∂/∂z → ikz◮ Ordinary differential equations in time

hg, hv : Quadratic nonlinear terms◮ Computed in realspace◮ Dealiasing : require 1/2 additional grid points


Simulation Process - Pencil Decomposition


For one time step

* 8 Racks, 1 MPI task/node


Solving Navier-Stokes Equation

B-Spline for y dir derivatives

Nx = 18432, Ny = 1536, Nz = 12288

131k 262k 393k 524k 786k

Number of Cores1

10

Ela

psed

tim

e (s

ec) Measured

Ideal100%

101%

102%

99.9%

96.5%


Discrete Fourier Transform

FFTW 3.3

Nx = 18432, Ny = 1536, Nz = 12288

131k 262k 393k 524k 786k

Number of Cores1

10

Ela

psed

tim

e (s

ec) Measured

Ideal100%

91.1%

93.6%

87.7%

89.6%


Data Transpose

FFTW 3.3 - MPI

Nx = 18432, Ny = 1536, Nz = 12288

131k 262k 393k 524k 786k

Number of Cores2

4

8

16

32

Ela

psed

tim

e (s

ec) Measured

Ideal100%

98.6%

100%98.6%

99.6%


Comparison with P3DFFT

parallel 3D FFT : Data transpose + FFT

ESP-FFTP3DFFT 2.5.1

◮ Most popular parallel 3D FFT library◮ Pencil decomposition

Differences (P3DFFT / ESP-FFT)◮ Communication Pattern : Fixed / Various◮ Odd-Ball wavenumber : Stored / Removed◮ (Buffer) Memory requirement : 3× / 1×◮ Threading capability : None / OpenMP (Data rearrangement, FFT)

* For benchmark, data is Fourier transformed in only x and z dir.


Comparison with P3DFFT (Small size)

Nx = 2048, Ny = 1024, Nz = 1024

128 256 512 1024 2048 4096 8192

Number of Cores0.01

0.1

1

10

100E

laps

ed ti

me

(sec

)P3DFFT (Measured)P3DFFT (Ideal)ESP-FFT (Measured)ESP-FFT (Ideal)

100%98.0%

97.7%98.6%

99.5%100%

101%

100%96.8%

115%116%

117%121%

123%

2.14 2.12 2.51 2.52 2.52 2.59 2.61


Comparison with P3DFFT (Large size)

Nx = 18432, Ny = 12288, Nz = 12288

65k 131k 262k 393k 524k 786k

Number of Cores1

10

Ela

psed

tim

e (s

ec)

P3DFFT (Measured)P3DFFT (Ideal)ESP-FFT (Measured)ESP-FFT (Ideal)

100%81.4%

89.6%

90.5%

100%

94.1%

89.5%

86.9%94.3%

81.5%

1.45 1.73 1.71 1.46


One full timestep

Nx = 18432, Ny = 1536, Nz = 12288

65k 131k 262k 393k 524k 786k

Number of Cores

10

100E

laps

ed ti

me

(sec

)MPI onlyHybrid (MPI + OpenMP)

1.19 1.13 1.02 1.14 1.00


Mean Velocity Profile

0 1000 2000 3000 4000 5000Y+

0

5

10

15

20

25

30

U+

1 10 100 1000Y+

0

5

10

15

20

25

30

U+


Variance & covariance of velocity components

0 1000 2000 3000 4000 5000Y+

-2

0

2

4

6

8

10

<uu

><uu><vv><ww><uv>


Energy Spectra

1 10 100 1000kx

1e-08

1e-06

0.0001

0.01

1

E_u

u

E_uuE_vvE_ww

1 10 100 1000kx

1e-12

1e-08

0.0001

1

E_u

u

E_uuE_vvE_ww

1 10 100 1000kz

1e-08

0.0001

1

E_u

u

E_uuE_vvE_ww

1 10 100 1000kz

1e-12

1e-08

0.0001

1

E_u

u

E_uuE_vvE_ww

* Left : Y+ = 10.5 / Right : Y+ = 4800


Energy Spectra (wavenumber multiplied)

0.1 1 10 100 1000kx

0

0.1

0.2

0.3

0.4

0.5

0.6

kE_u

u

kE_uukE_vvkE_ww

0.1 1 10 100 1000kx

0

0.025

0.05

kE_u

u

kE_uukE_vvkE_ww

0.1 1 10 100 1000kz

0

0.5

1

1.5

2

2.5

3

3.5

kE_u

u

kE_uukE_vvkE_ww

0.1 1 10 100 1000kz

0

0.05

0.1

0.15

0.2

0.25

kE_u

u

kE_uukE_vvkE_ww

* Left : Y+ = 10.5 / Right : Y+ = 4800


Streamwise Velocity


Spanwise Vorticity


Summary and future work

Summary◮ High Re DNS is required to expand our understanding of wall

bounded turbulent flow◮ Excellent scalability is achieved◮ Preliminary results are demonstrated

Future Work◮ Gathering more data until statistically converged◮ Investigating the multi-scale dynamics that governs the interaction

of the inner and outer turbulence◮ Upon completion of simulation, simulation data will be publicly

available at⋆ http://turbulence.ices.utexas.edu


THANK YOU!!

Questions?

[email protected]


petascale direct numerical simulations of turbulent ... overview project title petascale direct...

Documents