preliminary investigations on resilient parallel numerical linear algebra … · 2018. 5. 25. ·...

SIAM EX14 WorkshopJuly 7, Chicago - IL

Preliminary Investigations on ResilientParallel Numerical Linear Algebra Solvers

Luc Giraud

joint work withE. Agullo, P. Salas, E. F. Yetkin, M. Zounonfunded by ANR RESCUE and G8-ECS

HiePACS Inria ProjectJoint Inria-CERFACS labINRIA Bordeaux Sud-Ouest

Context

L. Giraud - Resilient numerical linear algebra solvers 2/ 25

Resilience: Ability to compute a correct output in presence of faults

I Context: Numerical linear algebraI Goal: Keep converging in presence of faultI Method: Recover-restart strategy without Checkpoint

I HPC systems are not fault-freeI A faulty components (node, core, memory) loses

all its dataI Simulations at exascale have to be resilient

Outline

Faults in HPC Systems

Sparse linear systems

Interpolation methods

Numerical experiments

Resilience in eigensolvers

Concluding remarks and perspectives



Outline









Framework

Forecast for extreme scale systemsI Mean Time Between Failure (MTBF): less than one hourI Checkpoint time might be larger than MTBF

ObjectivesI Explore fault-tolerant schemes with less/no overheadI Numerical algorithms to deal with overhead issue

Faults in this presentationI Detected corrupted memory space (node crashes, damaged

memory pages, uncorrected bit-flip, . . . )



Outline










x bA

=

Ax = bWe attempt to design fault tolerant solversfor sparse linear system

Two classes of iterative methodsI Stationary methods (Jacobi, Gauss-Seidel, . . . )I Krylov subspace methods (CG, GMRES, Bi-CGStab, . . . )

I Krylov methods have attractive potential for Extreme-scale


Outline










Block row distributionx bA

P

P

P

P

1

2

3

4

=

We distinguish two categories of data:I Static dataI Dynamic data



��

��

x bA

P

P

P

P

1

2

3

4

Static data Dynamic data

=




��

��

x bA

P

P

P

P

1

2

3

4

Static data Dynamic data

=


Let’s assume that P1 fails



��

��

��

��

x bA

P

P

P

P

1

2

3

4

Static data Dynamic data Lost data

=


Let’s assume that P1 fails



��

��

��

��

x bA

P

P

P

P

1

2

3

4


=


Let’s assume that P1 failsI Failed processor is replacedI Static data are restored



��

��

x bA

P

P

P

P

1

2

3

4


0

=



Reset: Set (x1) to initial value



��

��

��

��

x bA

P

P

P

P

1

2

3

4

Static data Dynamic data Lost data Interpolatedv data

=



Our algorithms aim at recovering x1and restart


Overview of our fault tolerant algorithm


P

P

P

P

1

2

3

4

Time

I Sequential simulationsI Simulation of parallel

environment




P

P

P

P

1

2

3

4

Time

Fault


environment

I Generation of fault traceI Realistic probability distribution




P

P

P

P

1

2

3

4

Time

Fault Successful iteration


environment





P

P

P

P

1

2

3

4

Time

Fault Successful iteration Failed iteration


environment





P

P

P

P

1

2

3

4

Time

Fault Successful iteration Failed iteration Interpolation


environment





P

P

P

P

1

2

3

4

Time

Fault Successful iteration Failed iteration Interpolation

Restart


environment




Fault in linear system(A11 A12A21 A22

)(x1x2

)=

(b1b2

)

Linear Interpolation (LI) [Langou, Chen, Bosilca, Dongarra, SISC, 2007]

Solve A11x1 = b1 − A12x2

Least Squares Interpolation (LSI)

(A11A21

)x1 +

(A21A22

)x2 =

(b1b2

)x1 = argmin

x

∥∥∥∥(b1b2

)−(

A11A21

)x −

(A12A22

)x2

∥∥∥∥2





)(?x2

)=

(b1b2

)How to recover x1?




(A11A21

)x1 +

(A21A22

)x2 =

(b1b2

)x1 = argmin

x

∥∥∥∥(b1b2

)−(

A11A21

)x −

(A12A22

)x2

∥∥∥∥2





)(?x2

)=

(b1b2

)How to recover x1?



Least Squares Interpolation (LSI)(A11A21

)x1 +

(A21A22

)x2 =

(b1b2

)x1 = argmin

x

∥∥∥∥(b1b2

)−

(A11A21

)x −

(A12A22

)x2

∥∥∥∥2



Main properties - basic linear algebra

PropositionThe initial guess generated by LI after a fault does ensure that theA-norm of the forward error associated with the iterates computedby restarted CG or PCG is monotonically decreasing

PropositionThe initial guess generated by LSI after a fault does ensure themonotonic decrease of the residual norm of minimal residualKrylov subspace methods such as GMRES and MinRES after arestarting due to a failure



Main properties - basic linear algebra

PropositionThe initial guess generated by LI after a fault does ensure that theA-norm of the forward error associated with the iterates computedby restarted CG or PCG is monotonically decreasing[LI might not be defined for non-SPD matrices as diagonal blocksmight be singular]

PropositionThe initial guess generated by LSI after a fault does ensure themonotonic decrease of the residual norm of minimal residualKrylov subspace methods such as GMRES and MinRES after arestarting due to a failure



Outline









Impact of fault ratePreconditioned GMRES (Kim1 - 2 % data lost)

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 140 280 420 560 700 840 980 1120 1260 1400

||(b

-Ax)|

|/||b||

Iteration

Reset

LI

LSI

SC

REF

Figure: 4 faults




1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 140 280 420 560 700 840 980 1120 1260 1400

||(b

-Ax)|

|/||b||

Iteration

Reset

LI

LSI

SC

REF

Figure: 8 faults




1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 140 280 420 560 700 840 980 1120 1260 1400

||(b

-Ax)|

|/||b||

Iteration

Reset

LI

LSI

SC

REF

Figure: 17 faults




1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 140 280 420 560 700 840 980 1120 1260 1400

||(b

-Ax)|

|/||b||

Iteration

Reset

LI

LSI

SC

REF

Figure: 40 faults



Impact of lost data volumePreconditioned GMRES(100) (Averous/epb3 - 10 faults)

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 98 196 294 392 490 588 686 784 882 980

||(b

-Ax)|

|/||b||

Iteration

Reset

LI

LSI

SC

REF

Figure: 3 % data lost




1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 98 196 294 392 490 588 686 784 882 980

||(b

-Ax)|

|/||b||

Iteration

Reset

LI

LSI

SC

REF

Figure: 0.8 % data lost




1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 98 196 294 392 490 588 686 784 882 980

||(b

-Ax)|

|/||b||

Iteration

Reset

LI

LSI

SC

REF




Penalty of restart strategy

I Recover-restart strategyI When restarting, we lose the Krylov subspace built before the

faultI Consequence: delay of convergence due to restartI Restarting mechanism is naturally implemented in GMRES to

reduce the computational resource consumptionI CG does not need to be restarted



Penality of restart strategy on PCG

1e-13

1e-12

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 83 166 249 332 415 498 581 664 747 830

A-n

orm

(err

or)

Iterations

Reset

LI

LSI

SC

Figure: PCG on a 7-point stencil 3D Poisson equation with 70 faults -5 % data lost



Penality of restart strategy on PCG

1e-13

1e-12

1e-11

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

0 83 166 249 332 415 498 581 664 747 830

A-n

orm

(err

or)

Iterations

Reset

LI

LSI

SC

REF

Figure: PCG on a 7-point stencil 3D Poisson equation with 70 faults -5 % data lost



Outline









Recovery-restart for eigensolvers

Fault in eigenproblem(A11 A12A21 A22

)(x1x2

)= λ

(x1x2

)

Linear Interpolation (LI)Solve the linear system

(A11 − λI1

)x1 = −A12x2


(A11A21

)x1 +

(A21A22

)x2 = λ

(x1x2

)x1 = argmin

x

∥∥∥∥(A11 − λI1A21

)x +

(A12

A22 − λI2

)x2

∥∥∥∥2



Recovery-restart for eigensolvers

Fault in eigenproblem(A11 A12A21 A22

)(?x2

)= λ

(?x2

)How to recover x1?

Linear Interpolation (LI)Solve the linear system

(A11 − λI1

)x1 = −A12x2

Least Squares Interpolation (LSI)(A11A21

)x1 +

(A21A22

)x2 = λ

(x1x2

)x1 = argmin

x

∥∥∥∥(A11 − λI1A21

)x +

(A12

A22 − λI2

)x2

∥∥∥∥2




xA

=

x If Ax = λx with x 6= 0, where A ∈ Cn×n,x ∈ Cn, and λ ∈ C , then,

I λ : eigenvalueI x : eigenvectorI (λ, x) : eigenpair

Two classes of methodsI Fixed Point Methods (Power Method, Subspace iteration)I Subpace Methods (Jacobi-Davidson, Arnoldi, IRA/Krylov

Schur)


Thermo-acoustic test example

(a few smallest eigenvalues)



Jacobi-Davidson method

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

1

1e+01

0 24 48 72 96 120 144 168 192 216 240

||(A

x -

lam

bda*x

)||/||la

mbda||

Iteration

0 1 2 2 3 4

LSI

REF

Figure: Jacobi-Davidson method with 5 faults - 1 % lost data.Convergence history using LSI and Checkpoint of current iterate



Outline









Concluding remarks

SummaryI We have designed techniques to interpolate meaningfull lost

data based on simple linear algebra toolsI Our techniques preserve some of the key monotonicy of Krylov

solvers but lack of robustness of LI for non-SPD problemsI The restarting effect remains reasonable within the GMRES

contextI No fault, no overheadI These techniques can be adpated to multiple faultsI What about silent soft-error - CGPOP preliminary

experiments ?


Merci for your attentionQuestions ?

https://team.inria.fr/hiepacs/

preliminary investigations on resilient parallel numerical linear algebra … · 2018. 5. 25. ·...

Documents