1 high performance multiscale simulation for crack...
TRANSCRIPT
1
High Performance Multiscale Simulation for Crack Propagation
HPSEC 2006 - 18th August
Guillaume Anciaux, Olivier Coulaud and Jean Roman
ScAlApplix Project
2
Outline
1. Introduction
• Motivations
• State of the art
2. Our Approach
• Presentation of the method
• Coupling Algorithm
• Parallel algorithms and implementation
3. Results
• 1D & 2D case : wave propagations
• 2D case : crack
4.Conclusion
4
Introduction : context
Collaboration with the CEA DIF-DPTA - G. Zerah
Goals : produce a tool for
• CEA1. Better understanding of miscroscale phenomena2. Reduce computing time of molecular dynamics simulations
• ScAlApplix1. Analysis of coupling algorithms2. HPC parallel scaling processes3. Development of a framework for multiscale computation4. Generics : use of legacy codes
Impact on laser optics
5
Introduction : atomistic approach
simulation with molecular dynamics tool:
an empirical potential (one, two or three body interactions)
fine description of the studied system.
All structural phenomena are captured
MV
Teams :• A. Nakano (Louisiana State University)• H. Gao (Max Planck Institute for Metals Research)• …
Md2x tdt2
=− VM x t
6
Introduction : continuum approach
Elastodynamics :
Lagrangian formulation
Constitutive law for the material
Discretization : P1 finite elements Variational problem – energy minimization Allow to manipulate huge objects Well known technology
TTTbTy =+= divρ
)y(∇= TT
xΩ
),( txy
ω
7
Difficulties and limitations of the methods
MD limitations1.Space scale :
A 100 nm3 of silicon crystal contains near billion atoms
2.Time scale :
Time step = femto-second(10-15 s.)3.Boundary conditions (periodic)4.Huge data volumes ~ 1 Terabyte per
steps
FE limitations1.Discontinuities (Crack)
Constitutive law not valid anymore near the fracture : the model needs to be extended (XFEM, …)
2.Needs a fine mesh to capture pertinent information.
[ [ ] ] 0y ≠
8
Introduction : Multiscale approach (1)
Idea : using advantages of both models
• Continuum model– Reduce the size of the domain.– Take into account complex boundary conditions.
• discrete model– Near the discontinuities.
How to couple this two models ?
9
Introduction : Multiscale approach (2)
Multi-scale approaches :
• Junction– QC-method (Tadmor and al. 1996)
Static simulations and T= 0 – Macroscopic, Atomistic and Ab initio Dynamics (MAAD) (Abraham
and al. 1998)
• Bridging : duplication of the data– T. Belytschko (Bridging Method)– Bridging Scale Method (Liu)
10
Introduction : Bridging approach
Numerical difficulties : Avoiding that h = inter-atomic distance Different time/length scales
mecanical wave reflexions
Algorithmic difficulties1. Need for smart data handling at the interface2. Efficient computation of the FE shape functions
in the overlapping zone3. Decomposition of domain4. Load balancing
Problem 3 and 4 are tied together
12
Discrete/continuum couplingThe Bridging Method introduced by T. Belytschko & S. Xiao
Idea :
• Imposing equivalent displacements on atom sites
• Introduction of Lagrangian multipliers
• Weighting the computations of the multipliers to keep predominances of each models
α
13
Discrete/continuum couplingThe constraint :
with :
Then the correction of the velocities :
gi y X i−di= y X i− x i=0 A=rhs∀ i ,
rhsi=∑ Jy J J X i− xi
Ai , j=∑ J
X iX j JM J
−1
imi
i , j
Ai=∑ jAi , jA is condensed on its diagonal
y I= y It
IM I∑ j
j I X j x I= xI−t
1−imi
i
14
Needs to identify atoms in a given element
Solution to algorithmic difficulties
Double loop O(Natoms x Nelements)
Introduction of a grid– Place atoms and elements in the grid.– Map atoms to elements.
Complexity O(Natoms x Nbox-elements)
15
Pre-computations of the shape functions for all atom sites
Initialization of the bridging zone (2)
•The shape values are stored in an appropriate data structure
•Allowing, through atom/element mapping, accesses at constant time for any given atom site
16
Mapping the codes to processors
Strategy : distinct processor sets for each model
Molecular dynamics set Continuum mechanics set
Coupling interaction
MD weighting FE weighting
17
Diagram for the coupling model of parallel codes (SPMD)
Initialization
Bridging Zone initialization
Force computation
Computation of the Lagrange multipliers
Force computation
Position update Position update
T = Tmax?
Velocity update Velocity update
Initialization
no no
yesParallel Molecular dynamics Parallel Continuum elasticity
T ia T i
b
T ic
T s1bT s
1a
T s2a T s
2b
T c
T s3bT s
3a
18Details on the computation of the Lagrange Multipliers
Computation of RHS contribution
Summing the contributions
Solving the constraint system Solving the constraint system
Correcting the velocities Correcting the velocities
Computation of RHS contribution
Parallel Molecular dynamics Parallel Continuum elasticity
− xi ∑ Jy JX i
∑ J y JX i− xi=rhsi
Tc1a Tc
1b
Tc2
Tc3 Tc
3 i=rhsi/ Aii=rhsi/ Ai
Tc4a Tc
4b∑=
∆+=m
jjIj
IIM
t
1I
newI )x(yy ϕλ
α
iii m
t λα )1(
dd inewi −
∆−=
19
Constraint system data redistribution
To illustrate the talk we consider the following distribution over processors :
Atomic zoneBridging zoneContinuum Zone
20
Constraint system data redistributionCase where the mapping of models is made onto two distinct sets of processors : both set of processor own a parallel vector of the Lagrangian unknowns.
21Dynamical effects : atoms migrations management (1)
Coherency in the redistribution scheme is maintained by protocol involving processors of the bridging zone
22Dynamical effects : atoms migrations management (2)Coherency in the redistribution scheme is maintained by protocol involving communication between the two set.
Send new owner
Send new position
Standard MD atom migration
24
Wave Reflexions on th 1D Model
Wave reflexions a caused by :• Number of degree of freedom reducing
• Overlap zone impedance which depends on:– Waves frequencies– Waves phase
400 atoms 40 elements
25
Reproduction of the absorbsion results
unad
apte
d im
peda
nce
adapted impedance
Initial condition
Imepdence depends on :
overlapping size element size settings of the projection
26
2D Model : wave propagation
Some numbers :• Atoms : 53 743 (Lennard Jones)
• Finite elements : 2 749 nodes, 5 249 elements
• Overlap zone– 29 868 atoms (~55%)– 523 nodes (19%), 891 elements (~17%)
Simulation :• 7 + 1 processors
• 20 000 time steps
• 358 ms / time steps– 62 % atoms– 34 % elasticity– 4% coupling
580
An
g
150 Ang
27
2D example : crack propagation
Box of 600nmx800nm
Numbers :• 91 556 atoms (Lennard Jones)
• 1 129 nodes, 2 082 elements
Overlapping zone : • 45 732 atoms
• 912 nodes et 1 582 elements
Crack : ellipse 50Ang x 1Ang
28
57,06% 30,29%
4,68%0,37%
7,45%0,15%
Computational time repartition for a 2D sequential simulation
Atom Part
Elast Part
BuildRHS
Correct Surface Effect
Correcting
Solving constraint
optimal : 20MD/16FE
29
4 8 16 20 24 320
0,5
1
1,5
2
2,5
3
3,5
4
Sim ulat ion t im es of the different tasks on 36 processors
Ts(c)
Ts(a)
Tc
num ber of processors assigned to m olecular dynam ics
tim
e i
n s
eco
nd
s fo
r 1
00
sim
ula
tio
n t
ime
ste
ps
31
32
24
20
16
8
4
0 0,25 0,5 0,75 1 1,25 1,5 1,75 2 2,25 2,5 2,75 3
Overhead t im e due to coupling on 36 processors during 100 t im esteps
Tc(4a,4b)
Tc(3)
Tc(2)
Tc(1a,1b)
nu
mb
er
of
pro
cess
ors
ass
ign
ed
to
mo
lecu
lar
dy
na
mic
s
t im e in seconds
32
3D case of real size : under construction
2662400 atoms
36450 elements
332800 atoms in the overlapping zone
8100 elements in the overlapping zone
34
Conclusion
First version of the simulator Better understanding of the multi-scales problems Wave reflexion management
Results in 2D Waves Crack
Going to 3D simulations
Enhancing the parallelism Molecular dynamics codes : limitation for the 3D cases
Domain decomposition in boxes Take into account cost functions to map tasks to processors