parallelization of an extended finite element two-phase flow solver · 2011. 7. 5. · chair for...
TRANSCRIPT
![Page 1: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/1.jpg)
CHAIR FOR
COMPUTATIONAL
ANALYSIS OF
TECHNICAL
SYSTEMS
Parallelization of an Extended Finite Element
Two-Phase Flow Solver
H. Sauerland and T.-P. Fries
July 20, 20109. WCCM
Sydney, Australia
![Page 2: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/2.jpg)
History of our XFEM flow solver
Navier-Stokes solver for quadrilateral/hexahedral meshes withsemi-implicit time integration.
←−
increasingNo.
DOF
serial direct solverC, UMFPACK1
⇓
parallel direct solverMPI, MUMPS2
⇓
parallel iterative solverMPI, PETSc3
1http://www.cise.ufl.edu/research/sparse/umfpack/
2http://graal.ens-lyon.fr/MUMPS/
3http://www.mcs.anl.gov/petsc/
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 2
![Page 3: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/3.jpg)
Outline
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 3
![Page 4: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/4.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 4
![Page 5: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/5.jpg)
Challenges in two-phase flows
Moving interface:
Topological changes of the phases can occur→ Level-set method.
Discontinuous field variables:
Density differences
→ kink in the velocity/pressure field,Viscosity differencesSurface tension → jump in the pressure field→ XFEM.
up
ρ2, µ2
ρ1, µ1
u
p
ρ2, µ2
ρ1, µ1
γ
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 5
![Page 6: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/6.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 6
![Page 7: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/7.jpg)
Introduction to the XFEM
Interface is implicitly described using the level-set method.
Discontinuities inside elements accounted for with the XFEM.
Here: Pressure approximation space is locally enriched:
ph(x) =∑
i∈I
Ni (x)pi
︸ ︷︷ ︸
strd. FE approx.
+∑
i∈I⋆
Ni (x) · [ψ(x, t)− ψ(xi , t)] ai
︸ ︷︷ ︸
enrichment
.
Sign-enrichment:
ψ(x, t) = sign (φ(x, t)) =
−1 : φ < 0,
0 : φ = 0,
1 : φ > 0.
I ⋆
φ(x, t) = 0
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 7
![Page 8: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/8.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 8
![Page 9: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/9.jpg)
Iterative solvers with the XFEM
Occurrences of enrichment basis functions with very smallsupport seriously increase the condition number of the systemmatrix → slow/no convergence with iterative solvers.
Spectral condition number of the system matrix K:
κ(K) =
∣∣∣∣
λmax(K)
λmin(K)
∣∣∣∣, λ: Eigenvalues
Case κ(K)
Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444
Possibilities: Blocking degrees of freedom,’special’ (additional) preconditioning.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 9
![Page 10: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/10.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 10
![Page 11: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/11.jpg)
Blocking degrees of freedom
Reusken (CVS, 2008) introduced a modified XFEM space,excluding the basis functions with very small support.
Here: Identify basis functions with very small support simplyusing the area ratio of the two phases occupying the element:
Aphase 1
Aphase 2< Cblock ≪ 1, where Aphase 1 < Aphase 2.
Spectral condition number of the system matrix K:
Case κ(K)
Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444Cblock = 0.01 ∼ 22849 Cblock = 0.01
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 11
![Page 12: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/12.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 12
![Page 13: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/13.jpg)
Preconditioning: Diagonal scaling
Bechet et al. (IJNME, 2005) developed a preconditioner forthe XFEM based on a local Cholesky decomposition.
Here: Simple diagonal scaling of the linear system:
D−1/2KD−1/2(D1/2x
)= D−1/2b, with D = diag(K)
Applied in addition to the general purpose preconditioners.
Spectral condition number of the system matrix:
Case κ(K)
Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444Cblock = 0.01 ∼ 22849Diagonal scaling ∼ 27
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 13
![Page 14: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/14.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 14
![Page 15: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/15.jpg)
Convergence study: Blocking DOFs, diagonal scaling
Steady state test case with surface tension:
ρ1/ρ2 = 1.0,
µ1/µ2 = 1.0,
γ = 1 kg/s2.
Exact solution given by:
u(x) = 0, p2 − p1 = γ ·1
r.
Ω1
Ω2
2r4r
4r
Solver: GMRES, LU preconditioner.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 15
![Page 16: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/16.jpg)
Convergence study: Blocking DOFs, diagonal scaling
No convergence to a steady state without blocking andwithout diagonal scaling.
Steady state test case with surface tension:
10−2
10−1
10−4
10−3
10−2
10−1
100
Element size h
Pre
ssur
e L2
−no
rm
dia. scal.c=0.01c=0.1c=0.5c=1
1/15 1/20 1/40 1/80 1/1600
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Element size h
Rat
io o
f num
ber
of b
lock
ed a
nd e
nric
hed
node
s
c=0.01c=0.1c=0.5c=1
Conclusion: Diagonal scaling is very powerful in this case.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 16
![Page 17: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/17.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 17
![Page 18: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/18.jpg)
Study on iterative solvers and preconditioners
PETSc allows to change solver/preconditioner on execution.
Considered Krylov-Subspace methods:
GMRES(m), l-GMRES, f-GMRES,BiCGstab, i-BiCGstab, BiCGstab-l.
GMRES: satisfies an optimality condition; Comp. costs risewith the iteration count.
BiCGstab: Comp. costs independent of the iteration count;does not satisfy an optimality condition.
Considered preconditioners:
ASM, KSP, LU, ICC, ILU, SOR, SSOR.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 18
![Page 19: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/19.jpg)
Study on iterative solvers and preconditioners
Steady state test case with surface tension:
ρ1/ρ2 = 1.0,
µ1/µ2 = 1.0,
γ = 1 kg/s2,
80× 80 elements.
Exact solution given by:
u(x) = 0, p2 − p1 = γ ·1
r.
Ω1
Ω2
2r4r
4r
Convergence criteria:Residual of the solution vector: ε ≤ 10−10.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 19
![Page 20: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/20.jpg)
Study on iterative solvers and preconditioners
GMRES
PC ε× 10−10 # t [s]lu 0.00060 528 7.92
f-GMRES
PC ε× 10−10 # t [s]lu 0.00057 522 7.84
l-GMRES
PC ε× 10−10 # t [s]asm 0.00190 1473 7.13ilu 0.00058 1524 6.93lu 0.00058 453 7.78
i-BiCGstab
PC ε× 10−10 # t [s]asm 3083. 3410 22.03ilu 8455. 3410 19.72
BiCGstab-l
PC ε× 10−10 # t [s]asm 3992. 2936 20.32lu 12. 1000 24.29ssor 7604. 4894 19.61
BiCGstab
PC ε× 10−10 # t [s]asm 1918. 3059 21.25ilu 4542. 3085 19.15ssor 875. 7678 27.29
GMRES methods converge very fast.
BiCGstab methods do not reach the residual bound.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 20
![Page 21: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/21.jpg)
Study on iterative solvers and preconditioners
Tank sloshing test case:
ρ1/ρ2 = 1000.0,
µ1/µ2 = 0.01,
∆t = 0.05 s,
80× 120 elements,400 time-steps.
Ω2
Ω1
1.0
1.5
g
Convergence criteria: ε ≤ 10−4.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 21
![Page 22: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/22.jpg)
Study on iterative solvers and preconditioners
Tank sloshing test case:
Solver PC t [min]
GMRES lu 36f-GMRES lu 34
l-GMRESasm 37ilu 33lu 36
i-BiCGstab asm 25BiCGstab-l lu 19BiCGstab ssor 30
0 2 4 6 8 10 12 14 16 18 200.9
0.95
1
1.05
1.1
1.15
Time
Hei
ght
BiCGstab methods perform better here, as the convergencecriteria is not that strong as in the steady case.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 22
![Page 23: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/23.jpg)
Study on iterative solvers and preconditioners
Rising bubble test case:
ρ1/ρ2 = 1000.0,
µ1/µ2 = 0.01,
γ = 0.001 kg/s2 ,
r = 0.1m,
∆t = 0.025 s,
80× 160 Elements,1000 time-steps.
Ω1
Ω2
4r
8r
g
2r
Convergence criteria: ε ≤ 10−4.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 23
![Page 24: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/24.jpg)
Study on iterative solvers and preconditioners
Rising bubble test case:
Solver PC t [min] Cblock
l-GMRES asm 159 0.0BiCGstab-l lu - 0.0BiCGstab-l lu 118 0.001
GMRESBiCGs
Diagonal scaling and blocking with small Cblock makes sensehere.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 24
![Page 25: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/25.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 25
![Page 26: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/26.jpg)
Parallelization in the XFEM context
Basic approach is the same as for standard FEM:
Partitioning(ParMETiS)
Assembly onpartition-level
Assembly in parallelPETSc matrix
Parallel solver
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 26
![Page 27: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/27.jpg)
Parallelization in the XFEM context
XFEM: Enriched elements are computationally more expensive(more integration points, more shape function evaluations andthe additional DOFs)
Tank sloshing for 2 PEs
PE0: 75 Elements (12 enriched)PE1: 75 Elements (0 enriched)→ Imbalance
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 27
![Page 28: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/28.jpg)
Parallelization in the XFEM context
Solution: Assign weights to the elements before the partitionphase (larger ones for enriched).
ParMETiS uses the weights as an additional constraint.
Tank sloshing for 2 PEs (enriched elements weighted with 10):
PE0: 48 Elements (9 enriched)PE1: 102 Elements (3 enriched)→ Balanced?
Reasonable values for the weights have to be determined.
E.g. for the rising bubble test case a static load balancing willnot be sufficient (also cf. adaptive meshes).
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 28
![Page 29: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/29.jpg)
Dynamic load balancing
Dynamic load balancing:
Adapt the partitioning duringexecution to maintain a good loadbalance.
Specify criteria when to perform a load balancing step (ideallytaking into account migration/rebuilding costs).
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 29
![Page 30: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/30.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 30
![Page 31: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/31.jpg)
Scale test
Olmsted dam on the Ohio river (Illinois, USA):
http://www.lrl.usace.army.mil/olmsted_specproj/
Computational domain:
195m
1.16m3/s →
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 31
![Page 32: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/32.jpg)
Scale test
140000 DOFs(performed on an Intel Xeon cluster with 1840 cores in total).
Interface motion and pressure:
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 32
![Page 33: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/33.jpg)
Scale test
Parallel performance:
Speedup:
S(n) =T (1)
T (n).
Efficiency:
E (n) =T (1)
n · T (n),
n: Number of processors,
n ts/h S(n) E(n)
1 1.5 - -2 2.8 1.87 0.9354 5.1 3.38 0.8458 8.6 5.69 0.71116 15.3 10.10 0.63132 16.6 10.90 0.340
T (i): Execution time with i processors.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 33
![Page 34: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/34.jpg)
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 34
![Page 35: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/35.jpg)
Conclusion
Usage of standard iterative solvers and preconditioners withthe XFEM is difficult.
Diagonal scaling of the system matrix significantly improvesthe condition number.
Additional blocking of DOFs makes sense in extreme cases.
Dynamic load balancing is required for the XFEM in order toachieve optimal parallel performance.
Current implementation already scales quite good.
Outlook: Complete dynamic load balancing.
Parallelize our adaptive mesh refinement.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 35
![Page 36: Parallelization of an Extended Finite Element Two-Phase Flow Solver · 2011. 7. 5. · CHAIR FOR COMPUTATIONAL ANALYSIS OF TECHNICAL SYSTEMS Parallelization of an Extended Finite](https://reader036.vdocuments.net/reader036/viewer/2022071219/6057f286fb2abd20a7076d6e/html5/thumbnails/36.jpg)
Thank you for your attention.
www.xfem.rwth-aachen.de
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 36