solving sdd systems in nearly-linear...

Introduction New Algorithm Final Remarks

Solving SDD Systems in Nearly-Linear Time

Pegah Pournajafi

École Normale Supérieure de LyonCR02 Combinatorial Scientific Computing

Lyon, Jan. 14, 2020

Pegah Pournajafi Solving SDD Jan. 14, 2020 1 / 44


Table of Contents

1 Introduction

2 New Algorithm

3 Final Remarks



Ax = b

DefinitionAn n × n matrix A is symmetric diagonally dominant (SDD) if

AT = A,and Aii ≥

∑j 6=i |Ai j | for every i ∈ [n].

Definition

A function f (n) is called nearly linear, denoted by O(n) iff (n) = O(n poly log(n)).



SDD

Appears in applicationModeling electrical networksSimulating random walksImage Smoothing

Related to Laplacian matrices of graphs→ Graph AlgorithmsThe Jacobi and Gauss–Seidel methods for solving a linearsystem. . .



Previous Algorithms

Algorithms For SDD and Laplacian matrices:Iterative methods for PSD matrixes, e.g. Conjugate Gradient,Chebychev Method, . . . . mostly O(mn)Vaidya; 1990 : O(m1.75).Spielman, Teng; 2004 : O(mpoly log(n)).

Kelner, Miller, Peng; 2012 : O(m log n).

Kelner, Orecchia, Sidford, Zhu; 2013: O(m log2 n log 1ε ).



Advantages of This Algorithm

Why this new algorithm is good then?A new approachEasy to analyzeSimple and intuitiveAdaptableSize of words is log(n) not poly log(n)



Table of Contents

1 Introduction

2 New Algorithm

3 Final Remarks



The Problem

SDD Linear System

Given an n × n SDD matrix A, a non-zero vector b ∈ Rn, and apositive number ε we are going to solve the system

Ax = b

to find an approximation of the solution x s.t.

‖x − A†b‖A ≤ ε‖A†b‖A,

where ‖z‖A = ‖zT Az‖.

The following algorithm, solves a SDD linear system in nearly lineartime.



Graph Theory Background

Definition

Suppose G = (V ,E ,W ) is a weighted graph on n vertices and medges, and suppose W : E → (0,+∞) is its weight function. For anarbitrary orientation of edges of G, we can define its adjacency matrixU ∈ Rn×m, where the rows of U are indexed by the vertices and thecolumns of U are indexed by edges. If we denote

√W (e) by ρe, then

Uv ,e

−ρe if e = (v ,u) for some u ∈ V+ρe if e = (u, v) for some u ∈ V0 otherwise

Remark

We denote the Laplacian of G with L. Notice that L = UT U, no matterwhich orientation have been chosen.



The Algorithm: A Big Picture

SDD algorithm

Reduce the problem to Ly = d , where L is the Laplacian of aconnected graph.Represent the problem as Ly = UUT y = d .Construct a basis N for the null space of U.

Let K =

[UN

], and solve the system Kf =

[d0

](approximately).

Find y s.t. UT y = f (answer to Ly = d).Find x based on the reduction.Return x .



What to prove

So we need to prove that all the followings can be done in at mostnearly linear time:

1 Ax = b can be reduce to Ly = d .2 Kf = g can be solved.3 N can be found.4 y exists and can be found.



The Reduction

Given Ax = b , where A is SDD.Reduce it to Ly = d , where L is the Laplacian matrix of aconnected graph.



The Reduction

Reduction:Given SDD A and Ax = b

A = Ap + An + D1 + D2.

L =

(D1 + D2/2 + An −D2/2− Ap

−D2/2− Ap D1 + D2/2 + An

)Consider L

(x1

x2

)=

(b−b

)as Ly = d .

If(

x1

x2

)is a solution for Ly = d , then x1−x2

2 is a solution for Ax = b

If(

x1

x2

)is an ε-approximate solution for Ly = d , then x1−x2

2 is an

ε-approximate solution for Ax = b

O(nnz(A)) = O(m) time reduction.

L is 2n × 2n

Take connected components if necessary



Solving Kf = g: The Kaczmarz Method

We want to solve Kf = g.

K =

k1

k2...

km

and g =

g1g2...

gm

.

f ∗ is a solution of Kf = g ⇔ f ∗ ∈⋂m

i=1 〈ki , f 〉 = gi .



The Kaczmarz Method

Kaczmarz Algorithm for solving Kf = g.

input: K ∈ Rm×m, g ∈ Rm

output: an approximation xT of the solution1 Pick some arbitrary f02 Pick some i ∈ {1, . . . ,m}3 update ft so it satisfies the i-th equation→ ft+1

"orthogonal projection"4 Repeat 2 and 3 for T times.5 Output fT



The Kaczmarz Method

Question 1: Does it converge?

Question 2: What is the convergence rate?

Question 3: How to choose the hyperplanes?



The Kaczmarz Method

Question 1: Does it converge?Yes, as long as every hyperplane is considered an infinite number oftimes.

Question 2: What is the convergence rate?




The Kaczmarz Method


Question 2: What is the convergence rate?Could be slow. Bounds on the rate of convergence for general A existfor any deterministic hyperplane selection rule.




The Kaczmarz Method



Question 3: How to choose the hyperplanes?Randomly.



The Kaczmarz Method




Question 4: From which probability distribution?



The Kaczmarz Method




Question 4: From which probability distribution?Probability of choosing the i-th hyperplane is proportional to ‖ki‖2.



The Kaczmarz Method

Randomized Kaczmarz Algorithm for solving Kf = g.

input: K ∈ Rm×m, g ∈ Rm

output: an approximation xT of the solution1 Pick some arbitrary f0

2 Pick some i ∈ {1, . . . ,m} with probability distribution P(i) = ‖ki‖2

‖K‖2

3 update ft so it satisfies the i-th equation→ ft+1"orthogonal projection"

4 Repeat 2 and 3 for T times.5 Output fT



The Kaczmarz Method

What is the convergence rate of the randomized Kaczmarz?

DefinitionMoore–Penrose pseudo-inverseGiven matrix A, The inverse of the linear operator A, restricted to theimage of it, is well defined and is denoted by A+. This is called theMoore-Penrose pseudo-inverse of A.

Definition

The squared-Frobenius norm of a matrix A, denoted by ‖A‖2F , is

defined to be the sum all the entries of A.



The Kaczmarz Method

Definition

Given an m × n matrix A, let ‖A+‖2 be defined as follows

‖A+‖2 , supz∈Rn, Az 6=0

‖z‖2

‖Az‖2 .

The average condition number is defined to be

κ(A) , ‖A‖F‖A+‖.



The Kaczmarz Method

TheoremThe Randomized Kaczmarz method for solving Kf = g, starting withf0, satisfies

‖fτ − f ∗‖2 ≤ ε2‖f0 − f ∗‖2

for τ = O(κ2(K ) log 1ε ) with probability at least 0.9.



The Kaczmarz Method on this specific problem

Solving Kf = g in the algorithm:

K =

[UN

], where N is a basis for the null space of U.

Start with f0 s.t.

Kf0 =

[UN

]f0 =

[dz0

].

Run Kaczmarz.Claim: The following hold for each step:

Kft =[UN

]ft =

[d

zt → 0

].



The Kaczmarz Method on this specific problem

TheoremIf ft is the result of the t-th iteration of the Randomized Kaczmarzalgorithm on this specific problem, then

Kft =[UN

]ft =

[d

zt → 0

]where zt → 0 as t →∞.

"We converge on zt without changing Uft = d ."proof. On the board.



Each Iteration

On each iterationChosen Ki among N part,Time: O(poly log n) - not good.ft+1 = ft + αKi where

α =gi −

∑j Kij(f0)j −

∑j Kij(ft)j∑

j K 2ij

.

Some data structurerepresenting ft only on the U partevaluate

∑j Kij(ft)j is an easy way

Update ft+1 = ft + αKi

Output ffinal

Time: O(log n)



Number of Iterations

Kaczmarz algorithm never visits the U rows of K . It works onsum subspace of K .The number of iterations ≤ O(κ2(N))

In this case we can compute ‖N+‖2 as follows:

‖N+‖2 = supz∈Rm, Nz 6=0, Uz=0

‖z‖2

‖NZ‖2



Construction of N

In general, it is not easy to find the null space (Gaussianelimination)Here, we have an easy combinatorial way to construct N.This N has the property that κ(N)2 = O(m log(n)) which makesthe number of iterations of Kaczmarz good enough.



Construction of N

G: the graph corresponding to L = UUT

T : an spanning tree of GU =

[UT UN

], where UT represents T

For any e = (i , j) /∈ T , there is a unique path between i and jusing edges of TDenote this unique path by Pe

Denote the unique cycle Pe ∪ {e} by Ce



Construction of N

TheoremThere exists some WN , s.t. UN = UT WN .

proof. WN ∈ R(n−1)×(m−n+1).

(WN)e,e′ =

− ρe′ρe

if e ∈ Pe′ and e, e′ have the same direction on Ce′

+ρe′ρe

if e ∈ Pe′ and e, e′ do not have the same direction on Ce′

0 if e /∈ Pe′

Check UN = UT WN .



Construction of N

Theorem

N =[−W T

N I]∈ R(m−n+1)×m is a basis for the null space of U.

proof.

UNT =[UT UN

] [−WNI

]= −UT WN + UN = −UN + UN = 0.



Example

on the board.Electrical flow.



Choose a good N

Definition

Let T be a spanning tree of the graph G. Stretch of the edge e′ /∈ T ,is defined as

str(e′) =∑

e∈Pe′

(ρe′

ρe)2.

Also, we definestr(T ) =

∑e/∈T

str(e).



Choose a good N

Remember that the number of iterations is O(κ(N)2).We want to find a T s.t. κ(N)2 = ‖N‖2

F‖N+‖2 is small.



Choose a good T

Theorem

‖N+‖2 = supz∈Rm, Nz 6=0, Uz=0

‖z‖2

‖NZ‖2 ≤ 1.

proof. N =[−W T

N I]. z =

[zTzN

].

WNzN = zT . (cycle argument UZ = 0)

‖NZ‖2 = ‖ −W TN zT + zn‖2

= ‖W TN zT‖2 + 2zT

T WNzN + ‖zN‖2

= ‖W TN zT‖2 + ‖zT‖2 + ‖zN‖2

≥ ‖zT‖2 + ‖zN‖2 = ‖z‖2.



Choose a good T

Theorem

‖N‖2F = str(T ) + (m − n + 1).

proof.

‖N‖2F = ‖WN‖2

F + ‖I‖2F

=∑

e′ /∈T ,e∈Pe′

(ρe′

ρe)2 + (m − n + 1)

= str(T ) + (m − n + 1).



Choose a good T

Corollary

κ2(N) ≤ str(T ) + (m − n + 1).

So we have to find a low stretch tree.

Theorem (Abraham, Neiman; 2012)

For a graph G on n vertices and m edges, one can find a spanningtree T with str(T ) = O(m log log log n) in O(m log log log n) time.

They use Petal Decomposition in their algorithm.



y exists

Theorem

Let f be a solution for Kf =[UN

]f =

[d0

], then there is a y s.t.

UT y = f .

This y can be found easily due to the structure of U.

proof. on the board.



The Algorithm: In more details

SDD algorithm

Reduce the problem to Ly = d .Represent the problem as Ly = UUT y = d .Find a spanning tree of G with str(T ) = O(m log log log n).Using T, construct a basis N for the null space of U.

Let K =

[UN

], and consider the system Kf =

[d0

].

Solve the above system, using Kaczmarz method, withO(m log log log n) iterations.Find y s.t. UT y = f (answer to Ly = d).Find x based on the reduction.Return x .



Run Time of the Algorithm

SDD algorithm

Reduce the problem to Ly = d . O(m)

Represent the problem as Ly = UUT y = d . no costFind a spanning tree of G with str(T ) = O(m log log log n).

O(m log log log n)Using T, construct a basis N for the null space of U. O(m)

Let K =

[UN

], and consider the system Kf =

[d0

]. no cost

Solve the above system, using Kaczmarz method, withO(m log n log(n/ε)) iterations. O(m log n log(n/ε))×O(log(n))

Find y s.t. UT y = f (answer to Ly = d). < O(m)

Find x based on the reduction. no costReturn x .



Run Time of the SDD algorithm

The running time of the SDD algorithm is O(m log2(n) log(n/ε)).



Table of Contents

1 Introduction

2 New Algorithm

3 Final Remarks



Remarks

In the original paper, they have used electrical flows to explainand prove their theorem. - Similar to the example.This algebraic point of view of the algorithm is proposed by"Sivan Toledo, Haim Avron, and Haim Kaplan".Approach:

Representationexample: Cholesky factorizationIterationexample: Conjugate Gradient Method, Kaczmarz MethodCombination of the twoexample: this algorithm



Further Questions

Can the algorithm be parallelized?The constant in the running time is big. Could it become smaller?Can this method (combining good representation and gooditerative method) be used in other graph problems?



References

A Simple, Combinatorial Algorithm for Solving SDD Systems inNearly-Linear Time; Jonathan A. Kelner and Lorenzo Orecchiaand Aaron Sidford and Zeyuan Allen Zhu; 2013A talk by Sivan Toledo; available here.Lx = b Laplacian Solvers and Their Algorithmic Application;Nisheeth K. Vishnoi



Thank You.


solving sdd systems in nearly-linear...

Documents