multi-dimensional parallel discontinuous galerkin โฆ...sample 2d mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข...
TRANSCRIPT
![Page 1: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/1.jpg)
Multi-dimensional Parallel Discontinuous Galerkin Method
Mentors: Dr. Ohannes Karakashian, Dr. Kwai Wong, Michael Wise
Zhe Zhu
The Chinese University of Hong Kong
7/29/2016
![Page 2: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/2.jpg)
Agenda
1. Background
2. How DG works
3. Solve 2D equation
4. Solve 3D Equation
5. Parallelization
6. Future work
![Page 3: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/3.jpg)
Background
Goal:
Implement DG-FEM to solve Poissonโs equation in parallel on HPC platform
โฮ๐ข = ๐ ๐๐ ฮฉ๐ข = ๐๐ ๐๐ ฮ๐ท๐๐ข
๐๐= ๐๐ ๐๐ ฮ๐
Ax=b Approximate solution of u
Problem Setting:
![Page 4: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/4.jpg)
How DG works
Element term
Interior edge term
Boundary edge term
Penalty term
Jump term
Diagonal local block
Local block Local block
Global matrix A
Solve Ax=b
๐โ(๐ข, ๐ฃ)Given
equation
Weak form
LHS: Ax RHS: b
๐โ(๐ข, ๐ฃ) F(v)
Divided into three parts
Computed in parallel
Combine all local blocks
Use Trilinos to finish parallel solving
Test function v
![Page 5: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/5.jpg)
How DG works
Given equation
Weak form
LHS: Ax RHS: b
๐โ(๐ข, ๐ฃ) F(v)
Test function v
โฮ๐ข = ๐ ๐๐ ฮฉ๐ข = ๐๐ ๐๐ ฮ๐ท
๐๐ข
๐๐= ๐๐ ๐๐ ฮ๐
![Page 6: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/6.jpg)
How DG works
Given equation
Weak form
LHS: Ax RHS: b
๐โ(๐ข, ๐ฃ) F(v)
Test function v
โฮ๐ข = ๐ ๐๐ ฮฉ๐ข = ๐๐ ๐๐ ฮ๐ท
๐๐ข
๐๐= ๐๐ ๐๐ ฮ๐
Bilinear Function for Stiffness Matrix:
:element term : jump term :penalty term
Solving Linear System:
![Page 7: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/7.jpg)
How DG works
Element term
Interior edge term
Boundary edge term
Penalty term
Jump term
Diagonal local block
Local block Local block
Global matrix A
Solve Ax=b
๐โ(๐ข, ๐ฃ)Given
equation
Weak form
LHS: Ax RHS: b
๐โ(๐ข, ๐ฃ) F(v)
Divided into three parts
Computed in parallel
Combine all local blocks
Use Trilinos to finish parallel solving
Test function v
![Page 8: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/8.jpg)
How DG works
Element term
Interior edge term
Boundary edge term
Penalty term
Jump term
Diagonal local block
Local block Local block
Global matrix A
Solve Ax=b
๐โ(๐ข, ๐ฃ)
Divided into three parts
Computed in parallel
Combine all local blocks
Use Trilinos to finish parallel solving
Bilinear Function for Stiffness Matrix:
:element term : jump term :penalty term
Solving Linear System:
![Page 9: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/9.jpg)
Solve 2D Equation
Sample 2D Mesh
โฮ๐ข = ๐ ๐๐ ฮฉ๐ข = ๐๐ ๐๐ ฮ๐ท๐๐ข
๐๐= ๐๐ ๐๐ ฮ๐
โข We want to solve ๐ข through numerical method
โข We want to get an approximation of ๐ข on the domain.
ฮฉฮ๐ท
Where ๐, ๐๐, ๐๐ is known
![Page 10: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/10.jpg)
Solve 2D Equation
ฮฉฮ๐ท
Choose nodal points
โข By calculating the value of u on these nodal points, we can get the approximation of u on ฮฉโข In general, the more nodal points you choose, the higher accuracy of the approximation you get
![Page 11: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/11.jpg)
1. Find a set of independent basic functions { ๐ฃ๐}
2. Try to use the linear combination to get the approximate
value on nodal points which can fit into the exact value.
3. Represent u as a linear combination of those basic
functions
Solve 2D Equation
๐ข =
๐
๐ข๐ ๐ฃ๐
How to get the approximation of u on these nodal points ?
![Page 12: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/12.jpg)
The two problem we need to solve:
1. How to choose the basic functions?
2. How to represent u as a linear combination of basic functions?
Solve 2D Equation
![Page 13: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/13.jpg)
Solve 2D Equation
Master cell
lagrange interpolation
๐๐ด = x
๐๐ต = y
๐๐ถ = 1-x-y
![Page 14: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/14.jpg)
Solve 2D Equation
๐๐ด = x
๐๐ต = y
๐๐ถ = 1-x-y
๐๐ด0
๐๐ต0
๐๐0
๐๐ด1
๐๐ต1
๐๐1
โฆ
๐๐ด7
๐๐ต7
๐๐7
Divide the mesh into several elements
![Page 15: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/15.jpg)
Solve 2D Equation
Use map to do integration in the master cell
a
b c
(x,y)
( ๐ฅ, ๐ฆ)
![Page 16: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/16.jpg)
Solve 2D Equation
Bilinear Function for Stiffness Matrix:
:element term : jump term :penalty term
Solving Linear System:
![Page 17: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/17.jpg)
Solve 2D Equation
๐0(๐ผ0)
๐1(๐ผ0)
๐0(๐ผ1)
๐2(๐ผ0)
๐2(๐ผ1)
๐ข0
(๐ผ0)
๐ข1
(๐ผ0)
๐ข2
(๐ผ0)
๐ข0
(๐ผ1)
๐ข1
(๐ผ1)
๐ข2
(๐ผ0)
= โฏ
๐1(๐ผ1)
๐0(๐ผ0) ๐1
(๐ผ0) ๐2(๐ผ0) ๐0
(๐ผ1)๐1
(๐ผ1)๐2
(๐ผ1)
โฆ
![Page 18: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/18.jpg)
Solve 2D Equation
Triangles
Interior Edges
Boundary Edges
++
--
+-
-+
8*
8*
8*
![Page 19: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/19.jpg)
Solve 2D Equation
![Page 20: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/20.jpg)
Solve 2D Equation
For triangles:
Element-element matrix:
![Page 21: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/21.jpg)
Solve 2D Equation
![Page 22: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/22.jpg)
Solve 2D Equation
![Page 23: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/23.jpg)
We use Gaussian quadrature to do the integration
โข In this example, the function is linear, so we only need one quadrature point to get the answer.
โข When calculate penalty term, we will need at least two quadrature point.
โข The match of the more than one pair of quadrature points is a problem.
Solve 2D Equation
![Page 24: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/24.jpg)
Solve 2D Equation
Get the right number for each blocks and combine those blocks into the global matrix
For edge (++ case)
![Page 25: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/25.jpg)
Solve 2D Equation
The RHS: Global matrixThe LHS: vector
![Page 26: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/26.jpg)
Solve 3D Equation
Nodal point *8
Tetrahedron *6
Inter face *6
Boundary face *12
![Page 27: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/27.jpg)
Solve 3D Equation
lagrange interpolation
๐๐ด = x
๐๐ต = y
๐๐ถ = z
๐๐ท = 1-x-y-z
![Page 28: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/28.jpg)
Solve 3D Equation
2D 3D
Element term Integration on surface Integration of 3 dimension
Number of quadrature points 0 0
Jump term Integration on line Integration on face
Number of quadrature points 1 1
Penalty term Integration on line Integration on face
Number of quadrature points 2 3
When our basic function is linear, the number of quadrature points needed for each term:
To calculate the penalty term of 3D case, we need to figure out how the three pairs of nodal points are matched.
![Page 29: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/29.jpg)
Solve 3D Equation
1. Find the way of mapping between the two cell and the
master cell respectively
2. Find out the relation between the two cell
3. Find the way of matching
4. Get the right answerIndex match
0 0, 1, 2
1 0, 2, 1
2 1, 0, 2
3 1, 2, 0
4 2, 0, 1
5 2, 1, 0
![Page 30: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/30.jpg)
Solve 3D Equation
Index match
0 0, 1, 2
1 0, 2, 1
2 1, 0, 2
3 1, 2, 0
4 2, 0, 1
5 2, 1, 0
Cell1: [1, 2, 3, 4] Cell2: [3, 1, 5, 4]
1. [1, 2, 3, 4] and [3, 1, 5, 4] are all map to [A, B, C, D]
2. The common face is [1, 3, 4] and [3, 1, 4]
3. The match way is [1, 0, 2]
4. Face [1, 3, 4] maps to [A, C, D] of the master cell
5. Face [3, 1, 4] maps to [A, B, D] of the master cell
6. Get the value of given functions at quadrature points
on [A, C, D] and [A, B, D]
7. Multiply those value according to the match way
![Page 31: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/31.jpg)
Parallelization
1. Divide the global matrix into the combination of several blocks
2. Calculate the right number for each blocks in parallel
3. Combine those blocks into the global matrix
4. Use conjugate gradient method to solve the equation in parallel
![Page 32: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/32.jpg)
Parallelization
After Global Matrix Construction: Solve linear system
โข C programming: dgesv on LAPACK
โข LU factorization: dense matrix solver
โฆ
Cell Matrices
MPI Assembly
Global Matrix
LAPACK dgesvdense matrix solver
Solution
![Page 33: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/33.jpg)
Parallelization
Parallelize the linear system solving process
โข AztecOO on Trilinos
โข Each processor has access to global rows
โฆ
Cell Matrices
MPI Assembly
Global Rows
AztecOO on Trilinos
Solution
![Page 34: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/34.jpg)
Parallelization
![Page 35: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/35.jpg)
Future works
โ Extend the partial differential equation to some time-dependent
equations
โ Finish the parallel code, which can be scaled on existing
supercomputers.
![Page 36: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/36.jpg)
Acknowledgement
This project is sponsored by
Oak Ridge National Laboratory
Joint Institute for Computational Sciences
University of Tennessee, Knoxville
The Chinese University of Hong Kong
Most sincere gratitude to my mentors
Dr. Ohannes Karakashian, Dr. Kwai Wong and Michael Wise
![Page 37: Multi-dimensional Parallel Discontinuous Galerkin โฆ...Sample 2D Mesh โฮ = ๐ ฮฉ = ฮ ๐๐ข ๐๐ = ๐ ฮ๐ โข We want to solve through numerical method ... Dr. Ohannes](https://reader033.vdocuments.net/reader033/viewer/2022050305/5f6d68ea7f730729676cee32/html5/thumbnails/37.jpg)
Q & A