ma 580; numerical analysis i
TRANSCRIPT
Part IXa: Residual Correction Methods
MA 580; Numerical Analysis I
C. T. KelleyNC State University
Version of November 28, 2016
NCSU, Fall 2016Part VIe: Residual Correction
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 1 / 52
Part IXa: Residual Correction Methods
Residual Correction
Suppose you have a linear solver: S where
S(b) ≈ A−1b.
Residual correction methods try to overcome problems with thesolver by using it iteratively.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 2 / 52
Part IXa: Residual Correction Methods
Algorithmic Sketch
x = x0r = b− Axwhile ‖r‖ > τ does = S(r)x← x + esr = b− Ax
end while
When S(b) = A−1b you converge in one iteration because
es = A−1r = A−1(b− Ax) = x∗ − x = e.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 3 / 52
Part IXa: Residual Correction Methods
Classic Iterative Improvement
Suppose S is a single precision solve?
S(b) = U−1s L−1s b
where Ls and Us are the single precision LU factors of A.
AS=single(A);
[LS, US]=lu(AS);
will do the job.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 4 / 52
Part IXa: Residual Correction Methods
The rules and the facts
Rule: computed residuals in full precision
Fact: relative error from solve will be single precision at best
Fact: cost of LU should be reduced by 1/8
Fact + Rule: you have to pay attention to thelanguage-dependent conversions from single to double andback.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 5 / 52
Part IXa: Residual Correction Methods
Precision Management
To compute S(b) you need get prepared
Convert A to single to get As
Compute the single precision LU factorization LsUs = A
You use the preparation by
Compute xs = U−1s L−1s b in single.
Convert xs to double to get x.
Compute r = b− Ax in double
Compute es = U−1s L−1s r in singleYou will probably not have to convert r to single to do this
Convert es to double to get ed
Compute x← x + ed in double.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 6 / 52
Part IXa: Residual Correction Methods
Some Matlab
Preparation:
AS=single(A);
[LS,US]=lu(AS);
Computing the correction:
xs=US\(LS\ b); xd=double(xs);
rd=b-A*xd;
es=US\(LS\rd); ed=double(es);
xd=xd+ed;
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 7 / 52
Part IXa: Residual Correction Methods
The iteration
AS=single(A);
[LS,US]=lu(AS);
xs=US\(LS\b);
xd=double(xs); rd=b-A*xd;
while norm(rd,inf) > 1.d-13
es=US\(LS\rd);
ed=double(es);
xd=xd+ed;
rd=b-A*xd;
end
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 8 / 52
Part IXa: Residual Correction Methods
Some Results: Method of Synthetic Solutions
Begin with
n=100;
A=rand(n,n); xe=rand(n,1); b=A*xe;
κ2(A) ≈ 1900.relative error relative residual
1.77e-05 1.52e-071.61e-11 2.25e-132.02e-14 1.02e-16
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 9 / 52
Part IXa: Residual Correction Methods
Analysis via Back-of-the-Envelope:I
Suppose δs is the relative error the single precision computations.
So x = U−1s L−1s b = x∗ − e,
e = O(δs),
and ed = U−1s L−1s r = e + ‖e‖O(δs) = e + O(δ2s ).
Which tells me that the correction is
x = xd + ed = x + e + O(δ2s ) = x∗ + O(δ2s )
So if δs = 10−k , you get k figures with every iteration.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 10 / 52
Part IXa: Residual Correction Methods
Analysis via Back-of-the-Envelope:II
What’s δs?
εs ≈ 10−8, single precision roundoff?
κ(A)εs ≈ 1900× 10−8 ≈ 10−5?
Looks like δs = κ(A)εs .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 11 / 52
Part IXa: Residual Correction Methods
A different perspective
Is this a stationary iterative method? Is the fixed point map
K(x) = x + S(x) ≡ x + I ds U−1s L−1s I sd (b− Ax)
So K is linear if the maps
I ds conversion from single to doubleNo problem: puts the same number is a bigger bucket
I sd conversion from double to singleThis rounding, just as linear as floating point addition
Bottom line: it’s close enough to linear.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 12 / 52
Part IXa: Residual Correction Methods
What’s the iteration matrix?
The map is
K(x) = x + I ds U−1s L−1s I sd (b− Ax)
= (I− I ds U−1s L−1s I sdA)x + I ds U
−1s L−1s I sdb ≡Msx + xs
So it’s preconditioned Richardson iteration and the convergencerate is
ρ(Ms) = ρ(I− I ds U
−1s L−1s I sdA
).
How might we estimate that?
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 13 / 52
Part IXa: Residual Correction Methods
Estimating ‖Ms‖
How about evaluating ‖Msx− x‖ for a random x?We already did that and got
relative error relative residual
1.77e-05 1.52e-071.61e-11 2.25e-132.02e-14 1.02e-16
So things look consistent, and I ds U−1s L−1s I sd is a good approximate
inverse of A.Warning! Matlab does not support sparse single precision arrays.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 14 / 52
Part IXa: Residual Correction Methods
There’s more!
Suppose you have two solvers S1 and S2 which are good atdifferent things.Suppose
x+ = S1(xc ,b) is a few iterations of an iterative solver
for Ax = b,with xc as the incoming iterate, andx+ as the new iterate.
S2(b) is a solver (iterative or direct) whichreturns a “converged” result for Ae = r.
Given the transfer maps, these solvers could have differentphysics, precisions, grids, . . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 15 / 52
Part IXa: Residual Correction Methods
Two-solver correction
You can let one correct the other via . . .
x = x0while Not happy dox1/2 = S1(x,b)r = b− Ax1/2e = S2(r)x = x1/2 + e
end while
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 16 / 52
Part IXa: Residual Correction Methods
Examples
Iterative refinement: S1 = I; S2 = U−1s L−1s .
MCSA: S1 is one step of Richardson: S2 is Monte Carlo solve
Two-grid: S1 is a couple steps of Jacobi, Gauss-Seidel, . . .S2 is a solve on a coarser grid.
Multigrid: Like two-grid, but S2 is a recursive call to MG.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 17 / 52
Part IXa: Residual Correction Methods
Two grid for Laplacian in 1-D
Recall our sad story with Jacobi for this problem.
We had a simple example.
Jacobi did ok for the first few iterations
and then went very, very slowly.
It seemed to kill the high frequency error fast,
and get stuck on the low frequency stuff.
Let’s look again.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 18 / 52
Part IXa: Residual Correction Methods
Jacobi Example
Let’s solve−u′′ = 0, u(0) = u(1) = 0.
with h = 1/101 and N = 100. The solution is u = 0. We will useas an initial iterate
u0 = x(1− x) +1
10sin(49πx)
We will take 100 Jacobi iterations.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 19 / 52
Part IXa: Residual Correction Methods
Matrix Representation
Au = f
where A is tridiagonal and symmetric
Ah =1
h2
2 −1 0 . . . 0, 0−1 2 −1 , 0 . . . 0
0 −1 2 −1, . . . 0...
. . .. . .
. . .. . .
...0 . . . , , 0, −1 2 −10 . . . , . . . , , 0 −1 2
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 20 / 52
Part IXa: Residual Correction Methods
Eigenvalues and Eigenvectors
Theorem: A is symmetric positive definite. The eigenvalues are
λn = h−22 (1− cos(πnh)) .
The eigenvectors un = (ξn1 , . . . , ξnN)T are given by
ξni =√
2/h sin(niπh)
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 21 / 52
Part IXa: Residual Correction Methods
Initial Error as Function of x
0 0.2 0.4 0.6 0.8 1−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
x
erro
r
Initial Error as Function of $x$
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 22 / 52
Part IXa: Residual Correction Methods
Final Error as Function of x
0 0.2 0.4 0.6 0.8 10
0.05
0.1
0.15
0.2
0.25
x
erro
r
Final Error as Function of $x$
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 23 / 52
Part IXa: Residual Correction Methods
Final Error as Function of x
0 20 40 60 80 1000.24
0.245
0.25
0.255
Iterations
l∞ e
rror
Convergence History
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 24 / 52
Part IXa: Residual Correction Methods
How’s this for a great idea?
Start at mesh size h.
Do a few Jacobi’s (one?)We’ll call Jacobi (or whatever) the smoother.
Transfer residual to mesh 2h.
Do an exact solve of D2h2 e2h = r2h
Send e2h to h mesh and correct x← x + eh
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 25 / 52
Part IXa: Residual Correction Methods
And the details?
Exactly how many Jacobi’s do I need here?
Is Jacobi the best tool for the job?Gauss-Seidel? SOR? Something else?
Can I do the obvious thing for intergrid transfer?
Is this really likely to work?!?
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 26 / 52
Part IXa: Residual Correction Methods
Two-grid algorithm: notation
Ωh: vectors corresponding to the h-mesh Rnd where there aren points in each direction andd = 1, 2, 3.
Ah: discrete differential operator on Ωh.
I 2hh : fine-to-coarse intergrid transfer.
I h2h: coarse-to-fine intergrid transfer.
TG (vh, fh): a single two-grid iteration for Ahvh = fh.
Sν(vh, fh): ν iterations of the smoother.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 27 / 52
Part IXa: Residual Correction Methods
Two-Grid Method
vh ← TG (vh, fh)
vh ← Sν1(vh, fh)rh = fh − Ahvh
r2h = I 2hh rh
Solve(?) A2he2h = r2h
eh = I h2he2h
vh = vh + eh
vh ← Sν2(vh, fh)
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 28 / 52
Part IXa: Residual Correction Methods
Testing . . .
Let’s make some questionable decisions.
One Jacobi presmooth (ν1 = 1, ν2 = 0)
I h2h: piecewise linear interpolation.
I 2hh : restriction by injection.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 29 / 52
Part IXa: Residual Correction Methods
Jacobi in Matlab
function us=jacobi(u,b)
n=length(u); h=1/(n+1); h2=h*h; us=zeros(n
,1);
us(2:n-1)=h2*b(2:n-1) + u(3:n) + us(1:n-2);
us(1) = h2*b(1) + u(2); us(n) = h2*b(n) + u(
n-1);
us=us*.5;
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 30 / 52
Part IXa: Residual Correction Methods
Restriction by Injection
function uc=inject(uf)
% INJECT
% fine to coarse intergrid transfer by
injection
% function uc=inject(uf)
%
nf=length(uf);
uc=uf (2:2:nf -1);
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 31 / 52
Part IXa: Residual Correction Methods
Piecewise Linear Interpolation
function uf = ctof(uc)
% CTOF
% function uf = ctof(uc)
% Coarse to fine intergrid transfer
%
nc=length(uc); nf=2*(nc+1) - 1; uf=zeros(nf ,1);
% Use zero boundary conditions
uf(1) =.5*uc(1); uf(nf)=.5*uc(nc);
% Vectorize the rest.
uf(3:2:nf -1) =.5*(uc(1:nc -1)+uc(2:nc));
uf(2:2: nf)=uc;
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 32 / 52
Part IXa: Residual Correction Methods
Restriction by Injection, two-grid, Jacobi
That would be great, but it seems to get stuck.
iterations0 10 20 30 40 50
|| r
||/||
r 0 ||
10-3
10-2
10-1
100
81 = 1
81 = 4
81=8
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 33 / 52
Part IXa: Residual Correction Methods
Gauss-Seidel is better. Why?
iterations0 5 10 15 20 25 30 35 40 45
|| r
||/||
r 0 ||
10-14
10-12
10-10
10-8
10-6
10-4
10-2
100
81 = 1
81 = 4
81=8
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 34 / 52
Part IXa: Residual Correction Methods
What happened?
We had this idea that Jacobi did great on high-frequency andpoorly on low, so
I assumed that Gauss-Seidel did the same.
I was half right.
It’s time to do an experiment.
Take one Jacobi/Gauss-Seidel iteration forAu = 0 with u0 = sin(iπx) for 1 ≤ i ≤ nand see what the reduction rate is.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 35 / 52
Part IXa: Residual Correction Methods
Results for Gauss-Seidel
wave number0 20 40 60 80 100 120 140
redu
ctio
n
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 36 / 52
Part IXa: Residual Correction Methods
Results for Jacobi
wave number0 20 40 60 80 100 120 140
redu
ctio
n
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 37 / 52
Part IXa: Residual Correction Methods
I’d like to save Jacobi because
it parallelizes and vectorizes well.One fix: damped Jacobi
unew = (1− ω)uold + ωujacobi
For our problem it’s
unewi = (1− ω)uoldi + ω(h2bi + uoldi−1 + uoldi+1)/2.
So what’s ω?
ω = 1 is Jacobi.
ω = 0 is do nothing.
The winner is ω = 2/3 because . . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 38 / 52
Part IXa: Residual Correction Methods
Analysis of Damped Jacobi
Damped Jacobi is
xn+1 = (1− ω)xn + ωD−1(L + U)xn + D−1b
with iteration matrix MhDJ = (1− ω)I + ωMJAC , where
MhJAC = −D−1(L + U).
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 39 / 52
Part IXa: Residual Correction Methods
Eigen-analysis
We showed that the eigenvalues of MJAC were
µn = 1− (h2/2)λn
with the same eigenvectors as Ah. Similarly
µn = (1− ω) + ω(1− (h2/2)λn) = 1− (h2/2)ωλn
= 1− ω(1− cos(πnh))
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 40 / 52
Part IXa: Residual Correction Methods
Optimal ω
Now suppose n ≥ N/2 (high frequency), then
µn = 1− ω(1− cos(πnh)) = 1− ω + ω cos(πnh)
because cos(πnh) ≤ cos(π/2) = 0, so
1− 2ω ≤ µn ≤ 1− ω.
Minimize |µn| to see that the optimal value of ω is 2/3. So
|µn| ≤ 1/3 for N/2 ≤ n ≤ N − 1
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 41 / 52
Part IXa: Residual Correction Methods
Results for Damped Jacobi
wave number0 20 40 60 80 100 120 140
redu
ctio
n
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 42 / 52
Part IXa: Residual Correction Methods
Comparison
wave number0 20 40 60 80 100 120 140
redu
ctio
n
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Damped JacobiGauss-SeidelJacobi
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 43 / 52
Part IXa: Residual Correction Methods
Damped Jacobi’s the winner. Let’s solve something.
iterations0 5 10 15 20 25 30 35 40
|| r
||/||
r 0 ||
10-15
10-10
10-5
100
81 = 1
81 = 4
81=8
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 44 / 52
Part IXa: Residual Correction Methods
Can we do better?
One problem is that restriction by injection ignores half of the finegrid points.A better idea is restriction by full weighting
u2hi = (uh2i−1 + 2uh2i + uh2i+1)/4
function vcoarse=ftoc(vfine)
% FTOC fine to coarse intergrid transfer by full
weighting
nf=length(vfine);
nc =.5*( nf+1) -1;
vtmp=zeros(nf+2,1);
vcoarse =.5* vfine (2:2:nf -1);
vcoarse=vcoarse +.25*( vfine (1:2:nf -2)+vfine (3:2:nf));
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 45 / 52
Part IXa: Residual Correction Methods
Restriction by full weighting, two-grid, Damped Jacobi
iterations0 5 10 15 20 25
|| r
||/||
r 0 ||
10-14
10-12
10-10
10-8
10-6
10-4
10-2
100
81 = 1
81 = 4
81=8
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 46 / 52
Part IXa: Residual Correction Methods
Something more to like. Matrix representations
I h2h =1
2
1 0 . . . 0 0 02 0 . . . 0 0 01 1 . . . 0 0 00 2 . . . 0 0 00 1 1 . . . 0 00 0 2 . . . 0 00 0 1 1 . . . 0...
......
......
...0 0 0 . . . 1 10 0 0 . . . 0 20 0 0 . . . 0 1
, I 2hh =1
2(I h2h)T (full weighting)
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 47 / 52
Part IXa: Residual Correction Methods
Two-Grid is a stationary iterative method
Go through the loops and
MTG = Mν2S (I− I h2h(A2h)−1I 2hh Ah)Mν1
S
where MS is the iteration matrix for the smoother.One standard configuration is
Damped Jacobi smoother
ν1 = ν2 = 1
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 48 / 52
Part IXa: Residual Correction Methods
Estimate the spectral radius
Look at ratio of relative residuals.
The ratios should converge to the spectral radius
unless something strange happens.
Do this for various values of h to see if the spectral radiusdepends on h or not.
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 49 / 52
Part IXa: Residual Correction Methods
Feel-good Prediction
Each application of damped Jacobi reduces high-frequencyerror by 1/3
So, if the coarse grid correction eliminates the low-frequencyerrors completely . . .
thenρ(MTG ) = 1/9.
Really?
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 50 / 52
Part IXa: Residual Correction Methods
We have a winner! Ratio = 1/9
iteration0 2 4 6 8 10 12 14
resi
dual
rat
io
0
0.1
0.2
0.3
0.4
0.5
0.6h=1/16h=1/32h=1/64h=1/128
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 51 / 52
Part IXa: Residual Correction Methods
Where are we?
Good:
TG is a stationary iterative method.Solver work moved to coarse grid.Convergence rate independent of h.
Bad:
Coarse mesh solver is still a matrix factorization.
Fix: replace the coarse mesh solver with another two-griditeration and get . . .
c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 52 / 52