chacon plenary

8/2/2019 Chacon Plenary

1/38

S c a l a b l e i m p l i c i t a l g o r i t h m s f o r s t i h y p e r b o l i c P D E s y s t e m s

L . C h a c n

O a k R i d g e N a t i o n a l L a b o r a t o r y

J . N . S h a d i d , E . C y r , P . T . L i n , R . T u m i n a r o , R . P a w l o w s k i

S a n d i a N a t i o n a l L a b o r a t o r i e s

D O E A p p l i e d M a t h e m a t i c s P r o g r a m M e e t i n g

O c t o b e r 1 7 - 1 9 , 2 0 1 1

W a s h i n g t o n , D C

L u i s C h a c n , c h a c o n l @ o r n l . g o v


2/38

O u t l i n e

M o t i v a t i o n : t h e t y r a n n y o f s c a l e s

B l o c k - f a c t o r i z a t i o n p r e c o n d i t i o n i n g o f h y p e r b o l i c P D E s

C o m p r e s s i b l e r e s i s t i v e M H D

C o m p r e s s i b l e e x t e n d e d M H D

I n c o m p r e s s i b l e N a v i e r - S t o k e s a n d M H D ( i n n i t e s o u n d - s p e e d l i m i t )



3/38

T h e t y r a n n y o f s c a l e s

( 2 0 0 6 N S F S B E S r e p o r t )

F i g u r e 1 : T i m e s c a l e s i n f u s i o n p l a s m a s ( F S P r e p o r t )



4/38

A l g o r i t h m i c c h a l l e n g e s i n t e m p o r a l s c a l e - b r i d g i n g

P D E s y s t e m s o f i n t e r e s t t y p i c a l l y h a v e m i x e d c h a r a c t e r , w i t h h y p e r b o l i c a n d p a r a b o l i c c o m p o n e n t s .

J H y p e r b o l i c s t i n e s s ( l i n e a r a n d d i s p e r s i v e w a v e s ) : (J) t f ast ttCF L 1

J P a r a b o l i c s t i n e s s ( d i u s i o n ) : (J) t Dx2

1

I n s o m e a p p l i c a t i o n s , f a s t h y p e r b o l i c m o d e s c a r r y a l o t o f e n e r g y ( e . g . , s h o c k s , f a s t a d v e c t i o n o f

s o l u t i o n s t r u c t u r e s ) , a n d t h e m o d e l e r m u s t f o l l o w t h e m .

I n o t h e r s , h o w e v e r , f a s t t i m e s c a l e s a r e p a r a s i t i c

, a n d c a r r y v e r y l i t t l e e n e r g y .

J T h e s e a r e t h e o n e s t h a t a r e u s u a l l y t a r g e t e d f o r s c a l e - b r i d g i n g .

B r i d g i n g t h e t i m e - s c a l e d i s p a r i t y r e q u i r e s a c o m b i n a t i o n o f a p p r o a c h e s :

J A n a l y t i c a l e l i m i n a t i o n ( e . g . , r e d u c e d m o d e l s ) .

J W e l l - p o s e d n u m e r i c a l d i s c r e t i z a t i o n ( e . g . , a s y m p t o t i c p r e s e r v i n g m e t h o d s )

J S o m e l e v e l o f i m p l i c i t n e s s i n t h e t e m p o r a l f o r m u l a t i o n ( f o r s t a b i l i t y ; a c c u r a c y r e q u i r e s c a r e ) .

K e y a l g o r i t h m i c r e q u i r e m e n t : S C A L A B I L I T Y

CPU O

N

np



5/38

A l g o r i t h m i c s c a l a b i l i t y v s . p a r a l l e l s c a l a b i l i t y

" T h e t y r a n n y o f s c a l e s w i l l n o t b e s i m p l y d e f e a t e d b y b u i l d i n g b i g g e r a n d f a s t e r c o m p u t e r s "

( N S F S B E S 2 0 0 6 r e p o r t , p . 3 0 )

O p t i m a l a l g o r i t h m : CPU N/np .

CPU N1+

n1p

; N =

L

d 0, algorithmic scalability 0, parallel scalability

M u c h e m p h a s i s h a s b e e n p l a c e d o n p a r a l l e l s c a l a b i l i t y ( ) .

H o w e v e r , p a r a l l e l ( w e a k ) s c a l a b i l i t y i s l i m i t e d b y t h e l a c k o f a l g o r i t h m i c s c a l a b i l i t y :

J N np CPU n+p r e q u i r e s = = 0!

E x p l i c i t I m p l i c i t ( d i r e c t ) I m p l i c i t ( K r y l o v i t e r a t i v e ) I m p l i c i t ( m u l t i l e v e l )

= 1/d = 2 2/d > 1 (varies) 0



6/38

H o w d o m u l t i l e v e l ( m u l t i g r i d ) m e t h o d s w o r k ?

M G e m p l o y s a d i v i d e - a n d - c o n q u e r a p p r o a c h t o a t t a c k e r r o r c o m p o n e n t s i n t h e s o l u t i o n .

J O s c i l l a t o r y c o m p o n e n t s o f t h e e r r o r a r e E A S Y t o d e a l w i t h ( i f a S M O O T H E R e x i s t s )

J S m o o t h c o m p o n e n t s a r e D I F F I C U L T .

I d e a : c o a r s e n g r i d t o m a k e " s m o o t h " c o m p o n e n t s a p p e a r o s c i l l a t o r y , a n d p r o c e e d r e c u r s i v e l y

S M O O T H E R i s m a k e o r b r e a k o f M G !

S m o o t h e r s a r e h a r d t o n d f o r h y p e r b o l i c s y s t e m s , b u t f a i r l y e a s y f o r p a r a b o l i c o n e s :

C a n o n e m a k e h y p e r b o l i c P D E s M G - f r i e n d l y ?



7/38

I m p l i c i t d i s c r e t i z a t i o n o f h y p e r b o l i c P D E s : a c a s e s t u d y

tu =

1

xv , tv =

1

xu ; =

k

i s a m e a s u r e o f h y p e r b o l i c s t i n e s s . D i s c r e t i z e i m p l i c i t l y i n t i m e :

un+1 = un +1

xv

n+1 , vn+1 = vn +1

xu

n+1.

V e r y i l l c o n d i t i o n e d a s

0! H o w e v e r , i f o n e c o m b i n e s e q u a t i o n s :

I

t

22x

un+1 = un +

t

xv

n

E q u a t i o n i s n o w w e l l - p o s e d w h e n 0 ( i . e . , i t i s a s y m p t o t i c - p r e s e r v i n g ) !

J L i m i t s y s t e m i s e l l i p t i c / p a r a b o l i c ( M G - f r i e n d l y ! )

JT e m p o r a l l y u n r e s o l v e d

h y p e r b o l i c t i m e s c a l e s h a v e b e e n p a r a b o l i z e d .

N o f u r t h e r m a n i p u l a t i o n o f P D E t h a n i m p l i c i t d i e r e n c i n g ( n o t e r m s a d d e d t o P D E ) !

T h i s f a c t c a n b e e x p l o i t e d t o d e v i s e o p t i m a l s o l u t i o n a l g o r i t h m s ( b l o c k f a c t o r i z a t i o n ) !



8/38

B l o c k - f a c t o r i z a t i o n o f h y p e r b o l i c P D E s

un+1 = un +

t xvn+1 , vn+1 = vn +

t xun+1

C o u p l i n g s t r u c t u r e : I t x

t x I

un+1

vn+1

=

un

vn

2

2 b l o c k c a n b e f o r m a l l y i n v e r t e d v i a b l o c k f a c t o r i z a t i o n :

D1

1U

1

L D2

=

I 1UD

12

0 I

D1

12

UD12 L 0

0 D2

I 0

1

D12 L I

O n l y i n v e r s e o f D1 UD12 L ( S c h u r c o m p l e m e n t ) i s r e q u i r e d !

D11

2UD12 L = I

t

22x



9/38

N o n l i n e a r h y p e r b o l i c P D E s :

J F N K a n d b l o c k f a c t o r i z a t i o n p r e c o n d i t i o n i n g

O b j e c t i v e : s o l v e n o n l i n e a r s y s t e m G(xn+1) =0 e c i e n t l y ( s c a l a b l y ) .

C o n v e r g e n o n l i n e a r c o u p l i n g s u s i n g N e w t o n - R a p h s o n m e t h o d : G

x

k

xk = G(xk) .

J a c o b i a n - f r e e i m p l e m e n t a t i o n :Gx

k

y = Jky = lim0

G(xk +

y)

G(xk)

K r y l o v m e t h o d o f c h o i c e : G M R E S ( n o n s y m m e t r i c s y s t e m s ) .

R i g h t p r e c o n d i t i o n i n g : s o l v e e q u i v a l e n t J a c o b i a n s y s t e m f o r y = Pkx :

JkP

1

k Pkx

y=

Gk

A p p r o x i m a t i o n s i n p r e c o n d i t i o n e r d o n o t a e c t a c c u r a c y o f c o n v e r g e d s o l u t i o n ; o n l y e c i e n c y !

B l o c k - f a c t o r i z a t i o n + M G w i l l b e o u r p r e c o n d i t i o n i n g s t r a t e g y .



10/38

I m p l i c i t r e s i s t i v e M H D s o l v e r

L . C h a c o n , P h y s . P l a s m a s

( 2 0 0 8 )



11/38

R e s i s t i v e M H D m o d e l e q u a t i o n s

t+ (v) = 0,

B

t+ E = 0,

(v)

t + vv BB v +I (p + B2

2 ) = 0,T

t+v T + ( 1)T v = 0,

P l a s m a i s a s s u m e d p o l y t r o p i c p n .

R e s i s t i v e O h m ' s l a w :

E = v B + B



12/38

R e s i s t i v e M H D J a c o b i a n b l o c k s t r u c t u r e

T h e l i n e a r i z e d r e s i s t i v e M H D m o d e l h a s t h e f o l l o w i n g c o u p l i n g s :

= L(, v)

T = LT(T, v)

B = LB(B, v)

v = Lv(v, B, , T)

T h e r e f o r e , t h e J a c o b i a n o f t h e r e s i s t i v e M H D m o d e l h a s t h e f o l l o w i n g c o u p l i n g s t r u c t u r e :

Jx =

D 0 0 Uv

0 DT 0 UvT

0 0 DB UvB

Lv LTv LBv Dv

T

B

v

D i a g o n a l b l o c k s c o n t a i n a d v e c t i o n - d i u s i o n c o n t r i b u t i o n s , a n d a r e e a s y t o i n v e r t u s i n g M G

t e c h n i q u e s . O d i a g o n a l b l o c k s L a n d U c o n t a i n a l l h y p e r b o l i c c o u p l i n g s .



13/38

B l o c k f a c t o r i z a t i o n o f r e s i s t i v e M H D

W e c o n s i d e r t h e b l o c k s t r u c t u r e :

Jx =

M U

L Dv

y

v

; y =

T

B

; M =

D 0 0

0 DT 0

0 0 DB

M i s e a s y t o i n v e r t ( a d v e c t i o n - d i u s i o n , n o t v e r y s t i , M G - f r i e n d l y ) .

S c h u r c o m p l e m e n t a n a l y s i s o f 2 x 2 b l o c k J y i e l d s :

M U

L Dv

1=

I 0

LM1 I

M1 0

0 P1Schur

I M1U

0 I

,

PSchur = Dv LM1U .

E X A C T J a c o b i a n i n v e r s e o n l y r e q u i r e s M1 a n d P1Schur .



14/38

P h y s i c s - b a s e d p r e c o n d i t i o n e r ( P B P )

3 - s t e p E X A C T i n v e r s i o n a l g o r i t h m :

Predictor : y = M1Gy

Velocity update : v = P1Schur[Gv Ly], PSchur = Dv LM

1U

Corrector : y =

y

M

1

Uv

M G t r e a t m e n t o f PSchur i s i m p r a c t i c a l d u e t o M1

.

W e c o n s i d e r h e r e t h e s m a l l - f l o w l i m i t : v vA M1 t I ( c h e a p )

W e h a v e e x t e n d e d t h e f o r m u l a t i o n t o a r b i t r a r y - o w s , v vA b a s e d o n c o m m u t a t i o n i d e a s 1 ( m o r e

e x p e n s i v e , b u t m o r e r o b u s t

2) .

1E l m a n , S I S C 2 7 , 1 6 5 1 ( 2 0 0 6 )

2L . C h a c n , J . P h y s i c s : C o n f . S e r i e s , 1 2 5 , 0 1 2 0 4 1 ( 2 0 0 8 )



15/38

P B P : S m a l l - o w l i m i t

S m a l l o w a p p r o x i m a t i o n : M1 t I i n s t e p s 2 & 3 o f S c h u r a l g o r i t h m :

y = M1 Gy

v P1SI [Gv Ly] ; PSI = Dv tLU

y y tUv

w h e r e :

PSI = nI/t + (v0 I + I v0

n2I)

+ t2W(B0, p0)

W(B0, p0) = B0 [I B0]j0 [I B0][I p0 + p0 I]

O p e r a t o r W(B0, p0) i s i d e a l M H D e n e r g y o p e r a t o r , w h i c h h a s r e a l e i g e n v a l u e s !

PSI i s p a r a b o l i c , a n d h e n c e b l o c k d i a g o n a l l y d o m i n a n t b y c o n s t r u c t i o n !

W e e m p l o y m u l t i g r i d m e t h o d s ( M G ) t o a p p r o x i m a t e l y i n v e r t PSI a n d M : 1 V ( 4 , 4 ) c y c l e



16/38

P B P : 2 D s e r i a l p e r f o r m a n c e ( t e a r i n g m o d e )

G r i d c o n v e r g e n c e s t u d y ( t = 1.0 A )

NG M R E S /

t CPUex p/CPU

t/

tCF L3 2 x 3 2 1 4 2 . 4 3 1 5 9

6 4 x 6 4 1 1 . 8 5 . 8 3 2 2

1 2 8 x 1 2 8 1 1 . 2 1 3 . 3 6 6 7

2 5 6 x 2 5 6 1 1 . 4 2 8 . 5 1 4 2 9

CPU O(N) OPTIMAL SCALING!

t c o n v e r g e n c e s t u d y ( 1 2 8 x 1 2 8 )

t G M R E S / t CPUex p/CPU t/tCF L

0 . 5 8.0 8 . 0 3 8 0

0 . 7 5

9.51 0 . 0 5 7 0

1 . 0 11.2 1 2 . 7 7 6 0

1 . 5 14.6 1 4 . 6 1 1 4 0

CPU O(t0.6) FAVORABLE SCALING!



17/38

P B P : 3 D s e r i a l p e r f o r m a n c e ( i s l a n d c o a l e s c e n c e )

1 0 t i m e s t e p s , t = 0.1, V ( 3 , 3 ) c y c l e s , m g t o l = 1 e - 2

G r i d G M R E S / t C P U

163 5 . 5 8 1

323 7 . 9 1 1 7 6

643 7 . 0 1 1 1 3 5

http://home/lch/Documents/artwork/pixie3d/movies/3dic-movie


18/38

P B P : 3 D p a r a l l e l p e r f o r m a n c e ( i s l a n d c o a l e s c e n c e )

( W e a k s c a l i n g , 163 p o i n t s p e r p r o c e s s o r , C r a y X T 4 ) t = 0.1 tCF L

K e y t o p a r a l l e l p e r f o r m a n c e :

J M a t r i x - l i g h t m u l t i g r i d , w h e r e o n l y d i a g o n a l s a r e s t o r e d ; r e s i d u a l s a r e c a l c u l a t e d m a t r i x - f r e e .

J O p e r a t o r c o a r s e n i n g v i a r e d i s c r e t i z a t i o n : a v o i d s f o r m i n g / c o m m u n i c a t i n g a m a t r i x .

C u r r e n t l i m i t a t i o n s : w e d o n o t f e a t u r e a c o a r s e - s o l v e b e y o n d t h e p r o c e s s o r s k e l e t o n g r i d .

J T h i s e v e n t u a l l y d e g r a d e s a l g o r i t h m i c s c a l a b i l i t y ( o n l y s h o w s a t > 1 0 0 0 - p r o c e s s o r l e v e l ) .



19/38

I m p l i c i t e x t e n d e d M H D s o l v e r



20/38

E x t e n d e d ( t w o - u i d , H a l l ) M H D m o d e l e q u a t i o n s

t+ (v) = 0,

Bt

+ E = 0,

(v)

t+

vv BB +

+

I (p +

B2

2)

= 0,

Tet

+v Te + ( 1)Te v = ( 1)

Q q

(1 + ),

=

i +

e ;

e = eve ; ve = v di

j

; v = v

di1 +

j

; =

TiTe

OhmsLaw :

E = v B + j + di (

j Bpe e ) electron EOM

E =

v

B +

j + di[t

v +

v

v +

1

(pi +

i )] ion EOM

N o t e t h a t E O M i - E O M e = E O M . A d m i t s a n e n e r g y p r i n c i p l e .

T h i s m o d e l s u p p o r t s f a s t d i s p e r s i v e w a v e s k2 .



21/38

E x t e n d e d M H D J a c o b i a n b l o c k s t r u c t u r e : e l e c t r o n E O M

( s t a n d a r d c h o i c e )

E = v B + j + di

(j Bpe e )

L i n e a r i z e d i n d u c t i o n e q u a t i o n B = E h a s t h e f o l l o w i n g c o u p l i n g s :

B = LB(B, v, , T)

J a c o b i a n c o u p l i n g s t r u c t u r e :

Jx =

D 0 0 Uv

LTB DT UBT UvT

LB LTB DB UvB

Lv LTv LBv Dv

T

B

v

W e h a v e a d d e d o - d i a g o n a l c o u p l i n g s t o b l o c k M .

S t i e s t b l o c k i s DB b r e a k s a p p r o x i m a t i o n s i n b l o c k - f a c t o r i z a t i o n a p p r o a c h . U N S U I T A B L E !



22/38

E x t e n d e d M H D J a c o b i a n b l o c k s t r u c t u r e : i o n E O M

E v B + j + di[tv +v v +1

(pi +

i [v])]

H a l l c o u p l i n g i s m a i n l y v i a tv .

J a c o b i a n c o u p l i n g s t r u c t u r e b e c o m e s :

Jx

D 0 0 Uv

0 DT 0 UvT

0 0 DB URvB + U

HvB

Lv LTv LBv Dv

T

B

v

W e c a n t h e r e f o r e r e u s e A L L r e s i s t i v e M H D P C f r a m e w o r k !



23/38

E x t e n d e d M H D p r e c o n d i t i o n e r

U s e s a m e b l o c k f a c t o r i z a t i o n a p p r o a c h .

M b l o c k c o n t a i n s i o n t i m e s c a l e s o n l y M1 t I i s a v e r y g o o d a p p r o x i m a t i o n

A d d i t i o n a l b l o c k UHvB :

PSIv = nv/t + (v0 v + v v0 +

)

+ t2W(B0, p0)v

W(B0, p0) = B0 [I B0 dit

I ]j0 [I B0][I p0 + p0 I]

A d d i t i o n a l t e r m b r i n g s i n d i s p e r s i v e w a v e s k2 !

W e c a n s h o w a n a l y t i c a l l y t h a t a d d i t i o n a l t e r m ( y e l l o w ) i s a m e n a b l e t o s i m p l e d a m p e d J a c o b i

s m o o t h i n g !

W e c a n u s e c l a s s i c a l M G !



24/38

O n t h e i s s u e o f d i s s i p a t i o n i n e x t e n d e d M H D

D i s p e r s i v e w a v e s

k2

r e q u i r e h i g h e r o r d e r d i s s i p a t i o n .

R e s i s t i v i t y i s u n a b l e t o p r o v i d e a d i s s i p a t i o n s c a l e .

D i s s i p a t i o n s c a l e d e n e d b y e l e c t r o n v i s c o s i t y , e :

e e

2( v) e4v

V i s c o s i t y c o e c i e n t c a n b e d e t e r m i n e d t o p r o v i d e a d e q u a t e d i s s i p a t i o n o f d i s p e r s i v e w a v e s

vAdik2 : ek

4 e > CdivAk,max

k3max

I n t h e p r e c o n d i t i o n e r , w e d e a l w i t h e b y c o n s i d e r i n g 2 s e c o n d - o r d e r s y s t e m s , a n d s o l v i n g

t h e m c o u p l e d w i t h i n M G .



25/38

E x t e n d e d M H D p e r f o r m a n c e r e s u l t s

( 2 D t e a r i n g m o d e )

di = 0.05, e = 2.5 106

1 0 0 t i m e s t e p s , t = 1.0 , 1 V ( 4 , 4 ) M G c y c l e

G r i d G M R E S / t CPUex p/CPU t/tex p t/tCF L

3 2 x 3 2 2 2 . 3 0 . 7 4 1 3 5 1 1 0

6 4 x 6 4 1 5 . 4 1 0 . 9 1 5 8 2 3 8 4

1 2 8 x 1 2 8 1 0 . 6 2 1 4 2 3 8 0 9 1 4 3 6

2 5 6 x 2 5 6 1 3 . 1 3 0 9 7 3 7 0 3 7 0 5 6 6 0



26/38

2 D n o n l i n e a r v e r i c a t i o n : G E M c h a l l e n g e

I o n H a l l v s . e l e c t r o n H a l l

90

100

110

120

130

140

150

160

0 5 10 15 20 25 30 35 40 45

Kinetic energy

Ion Halle Hall

0

1

2

3

4

5

6

7

8

9

0 5 10 15 20 25 30 35 40 45

Magnetic energy

Ion Halle Hall

6570

75

80

85

90

95

100

105

110

115

120

0 5 10 15 20 25 30 35 40 45

Thermal energy

Ion Halle Hall



27/38

I n c o m p r e s s i b l e N a v i e r - S t o k e s s o l v e r

C y r , S h a d i d , T u m i n a r o , J C P 2 0 1 1

E l m a n , H o w l e , S h a d i d , S h u t t l e w o r t h , T u m i n a r o , J C P 2 0 0 8



28/38

Block preconditioning:CFD example

Consider discretized Navier-Stokes equations

Properties of block factorization1. Important coupling in Schur-complement

2. Better targets for AMG leveraging scalability

Properties of approximate Schur-complement1. Nearly replicates physical coupling

2. Invertible operators good for AMG

Fully Coupled JacobianFully Coupled Jacobian PreconditionerPreconditioner

Required operators:

Multigrid

PCD, LSC,

SIMPLEC

Block FactorizationBlock Factorization

Coupling in Schur-complement


29/38

Discrete N-S Exact LDU Factorization Approx. LDU

Brief Overview of Block Preconditioning Methods for Navier-Stokes:

(A Taxonomy based on Approximate Block Factorizations, JCP 2008)

Now use AMG type methods on sub-problems.

Momentum transient convection-diffusion:

Pressure Poisson type:

Precond. Type References

Pres. Proj;

1st TermNeumann Series

Chorin(1967);Temam (1969);

Perot (1993): Quateroni et.al. (2000) as solvers

SIMPLEC Patankar et. al. (1980) assolvers; Pernice and Tocci

(2001) as smoothers/MG

PressureConvection /Diffusion

Kay, Loghin, Wathan,Silvester, Elman (1999 -

2006); Elman, Howle, S.,Shuttleworth, Tuminaro(2003,2008)


30/38

Fully coupled AlgebraicAggC: Aggressive Coarsening Multigrid

DD: Additive Schwarz Domain Decomposition

Block PreconditionersPCD & LSC: Commuting Schur complement

SIMPLEC: Physics-based Schur complement

CFD Weak Scaling: Steady Backward Facing Step

* Paper accepted: E. C. Cyr, J. N. Shadid, R. S. Tuminaro, Stabilization and Scalable Block Preconditioning for

the Navier-Stokes Equations, Accepted by J. Comp. Phys., 2011.

Take home: Block preconditioners competitive with fully

coupled multigrid for CFD

Take home: Block preconditioners competitive with fully

coupled multigrid for CFD

W k S li f NK S l ith F ll


31/38

Weak Scaling of NK Solver with Fully-

coupled AMG and Approx. Block

Factorization Preconditioners

Quad-core Nehalemswith Infini-band

SNL Red Sky

Transient Kelvin-Helmholtz instability

(Re = 5 x 103 shear layer, constant CFL = 2.5)


32/38

I n c o m p r e s s i b l e M H D s o l v e r



33/38

Incompressible MHD: 2D Vector Potential

Formulation

Magnetohydrodynamics (MHD) equations couple fluid flow to

Maxwells equations

u

t +u u

2u

+p +

1

0BB

+

1

20 B

2I

= f

u = 0

Az

t+ u Az

02Az = E

0

z

where B = A, A = (0, 0, Az)

Discretized using a stabilized finite element formulation


34/38

F B

TZ

B C 0Y 0 D

= I

BF1 I

Y F1

Y F1B

TS1 I

F B

TZ

S

BF1Z

P

where

S = CBF1BT

P = D

Y F1(I+ BTS1BF1)Z

Cis zero for mixed interpolation FE and staggered FV methods, nonzero for stabilized FE Indefinite system hard to solve with incomplete factorizations without pivotingBlock factorization of 3x3 system leads to nested Schur complementsUse an operator splitting approximation to factor

Reduces to 2 2x2 systems for Navier-Stokes and magnetics-velocity blocks;C need not be non-zero or invertible (C-1doesnt need to exist!)

Block LU Factorization


35/38

Fully coupled AlgebraicAggC: Aggressive Coarsening Multigrid

DD: Additive Schwarz Domain

Decomposition

Block PreconditionersSplit: New Operator split preconditioner

SIMPLEC: Extreme diagonal approximations

Take home: AggC and Split preconditioner scale algorithmically

1. SIMPLE preconditioner performance suffers with increased CFL

2. Run times are for unoptimized code3. AggC not applicable to mixed discretizations, block factorization is

Take home: AggC and Split preconditioner scale algorithmically

1. SIMPLE preconditioner performance suffers with increased CFL

2. Run times are for unoptimized code3. AggC not applicable to mixed discretizations, block factorization is

Transient Hydro-Magnetic

Kelvin-Helmholtz Problem(Re = 700, S = 700)


36/38

Take home: Split preconditioner scales algorithmicallyTake home: Split preconditioner scales algorithmically

Fully coupled Algebraic

AggC: Aggressive Coarsening Multigrid

DD: Additive Schwarz Domain Decomposition

Block Preconditioners

Split: New Operator split preconditioner

SIMPLEC: Extreme diagonal approximations

Driven Magnetic Reconnection: Magnetic Island

Coalescence Half domain symmetry on [0,1]x[-1,1]

with S = 10e+4

I iti l W k S li P f f AMG V l L d hi Cl M hi


37/38

Initial Weak Scaling Performance of AMG V-cycle on Leadership Class Machines

Cray XE6 and BG/P Weak Scaling

(Transport-reaction: Drift-diffusion simulations)

Steady-state drift-diffusion BJT

TFQMR time per iteration

Cray XE6 2.4GHz 8-core Magny-Cours

(Paul Lin)

Sub-domain smoothers: Impact of data

locality of smoother?

BG/P: ILU(2); overlap = 1

BG/P: ILU(0); overlap = 0

[Better scaling and faster time to

solution than ILU(2),ov=1]

Cray XE6: ILU(2); overlap = 1> 2200x


38/38

S u m m a r y a n d C o n c l u s i o n s

S t i h y p e r b o l i c P D E s d e s c r i b e m a n y a p p l i c a t i o n s o f i n t e r e s t t o D O E .

I n a p p l i c a t i o n s w h e r e f a s t t i m e s c a l e s a r e p a r a s i t i c , a n i m p l i c i t t r e a t m e n t i s p o s s i b l e t o b r i d g e

t i m e - s c a l e d i s p a r i t y .

A f u l l y i m p l i c i t s o l u t i o n m a y o n l y r e a l i z e i t s e c i e n c y p o t e n t i a l i f a s u i t a b l e s c a l a b l e a l g o r i t h m i c

r o u t e i s a v a i l a b l e .

H e r e , w e h a v e i d e n t i e d s t i - w a v e b l o c k - p r e c o n d i t i o n i n g ( a k a p h y s i c s - b a s e d p r e c o n d i t i o n i n g ) i n

t h e c o n t e x t o f J F N K m e t h o d s a s a s u i t a b l e a l g o r i t h m i c p a t h w a y .

J A n i m p o r t a n t p r o p e r t y i s t h a t i t r e n d e r s t h e n u m e r i c a l s y s t e m s u i t a b l e f o r m u l t i l e v e l p r e c o n d i -

t i o n i n g .

W e h a v e d e m o n s t r a t e d t h e e e c t i v e n e s s o f t h e a p p r o a c h i n i n c o m p r e s s i b l e N a v i e r - S t o k e s , i n c o m -

p r e s s i b l e M H D , a n d c o m p r e s s i b l e r e s i s t i v e M H D a n d e x t e n d e d M H D .

J I n a l l t h e s e a p p l i c a t i o n s , t h e a p p r o a c h i s r o b u s t a n d s c a l a b l e , b o t h a l g o r i t h m i c a l l y a n d i n

p a r a l l e l .
