cg bicg method

An introduction to CG and BICG Method

Wei-Len Lee

Department of Applied Mathematics, National University of Kaohsiung,Kaohsiung 811, Taiwan. E-mail: [email protected]

15 May 2007

Abstract

1.Conjugate Gradient Method (CG)

The Conjugate Gradient method is an e®ective method for symmetric positivede¯nite systems.It is the oldest and best known of the nonstationary methodsdisscussed here.The method proceed by generating vector sequences of iterates(i.e.,successive approximations to the solution),residuals corresponding to the it-erates,and search directions used in updating the iteratesand residuals. Althoughthe length of these sequences can become large, only a small number of vectorsneeds to be kept in memory .In every iteration of the method ,two inner productsare performed in order to compute update scalars that are de¯ned to make thesequences satisfy certain orthogonality conditions. On a symmetric positive de¯-nite linear system these conditions imply that this distance to the true solution isminimized in some norm. 2.BiConjugate Gradient (BiCG) The Conjugate Gradi-ent methid is not suitable for nonsymmetric systems because the residual vectorscannot be made orthogonal with short recurres (for proof of this see Voevodin [213]or Faber and Manteu®el [96]). The GMRES method retains orthogonality of theresiduals by using long recurrences, at the cost of a large storage demand. TheBiConjugate Gradient method takes another approach, replacing the orthogonal se-quence of residual by two mutually orthogonal seguences, at the price of no longerproviding a minimization.

1. Conjugate Gradient Method (CG)

The iteration x() are updated in each iteration by a multiple () of th seaarchdirection vector ()

() = (¡1) + ()

Correspondingly the residuals () = ¡ () are updated as

Preprint submitted to Elsevier Science 15 May 2007

PDF 檔案使用 "pdfFactory Pro" 試用版本建立 www.ahasoft.com.tw/FinePrint/

mailto:[email protected]

http://www.ahasoft.com.tw/FinePrint/

() = (¡1)¡() where () = () (1)

The choice = = (¡1) (¡1)()() minimizes ()¡1() over all

possible choices for in eqution (1)

The search directions are updated using the residuals

() = ()+¡1(¡1) (2)

where the choice = ()()(¡1) (¡1) ensures that () and (¡1) |

or equivalently, () and (¡1) | are orthogonal.In fact,one can show thatthis choice of makes () and () orthogonal to all previous () and ()

respectively.

1.1.theory

The conjugate gradient method constructs the ith iterate () as an elementof (0) + f(0) ¡1(0)g so that (() ¡ b)(() ¡ b) is minimized,

where b is the exact solution of = this minimum is guaranteed to existin general only if is symmetric positive de¯nite.

The above minimization of the error is equivalent to the residuals () = ¡ () .Since for symmetric an orthogonal basis for Krylov subspacef(0) ¡1(0)g can be constructed with only three-term recurrences,such a recurrence also su±ces for generating the residuals. In the ConjugateGradient method two coupled two-term recurrences are used; one that updatesresiduals using a search direction vector, and one updating the search directionwith a newly computed resiual.This makes the Conjugate Gradient methodquite attractive computationally.

There is a close relationship between the Conjugate Gradient method andthe Lanczos method for determining eigensystems,since both are based on thecontruction of an orthogonal basis for the Krylov subspace, and a similarlitytransformation of the coe±cient matrix to tridiagonal form. The coe±cientcomputed during the CG iteration then arise from the LU factrization of thistridiagonal matrix. Form the CG iteration one can reconstruct the Lanczosprocess, and vice versa;see Paige and Saunders [168] and Golub and Van Loan[109,x10.2.6]. This relationship can be exploited to obtain about the eigensys-tem of the matrix

1.2.Convergence

2



For the Conjugate Gradient method, the error can be bounded in terms of thespectral condition number 2 of the matrix (Recall that if max and min arethe largest and smalllest eigenvalues of a symmetric positive de¯nite matrix, then the spectral condition number of is 2() = max()min()).

If b is the exact solution of the linear system = , with symmetricpoistive de¯nite matrix , then for CG, it can be shown that

jj() ¡ bjj · 2()jj(0) ¡ bjj

where = (p2 ¡ 1)(

p2 + 1) (see Golub and Van Loan [109,x10.2.8], and

Kaniel [131],and jjjj2 ´ ()). From this relation we see that the numberof iterations to reach a relative reduction of in the error is proportional top2 .

1.3.Algorithm

step1 initial guess (0) = 0

step2 (0) = ¡ (0)

step3 (1) = (0)

step4 For = 1

= (¡1) (¡1)() ()

step5 () = (¡1) + ()

step6 () = ¡ () = (¡1) ¡ ()

step7

= () ()(¡1) (¡1)

step8 (+1) = () + ()

step9 check convergence;continue if necessary

step10 End for

3



2.BiConjugate Gradient (BiCG)

2.1.theory

The update relations for residuals in the Conjugate Gradient method are aug-mented in the BiConjugate Gradient method by relations that are similar butbased on instead of . Thus we update two sequences of residuals

() = (¡1) ¡ () e() = e(¡1) ¡ e()

and two sequences of search directions

() = (¡1) + ¡1(¡1) e() = e(¡1) + ¡1 e(¡1)

The choices

=(¡1) e(¡1)

()e() =()e()

(¡1)e(¡1)

ensure the bi-orthogonality relations

() e() = () e() = 0

2.3.Convergence

For theoretical results are known about the convergence of BiCG.For sym-metric positive de¯nite systems the method delivers the same results as CG,but at twice the cost per iteration. For nonsymmetric matrices it has beenshown that in phases of the process where there is signi¯cant reduction of thenorm of the residual, the method is more or less comparable to full GMRES(in terms of numbers of iterations)(see Freund and Nachtigal[102]). In practicethis is often con¯rmed, but it is also observed that the convergence behaviormay be quite irregular, and the method may even break down. The breakdownsituation due to the possible event that (¡1) e(¡1) ¼ 0can be circum-vented by so-call look-ahead strategies(see Parlett,Taylor and Liu[172]).Thisleads to complicated codes and is beyond the scope of this book. The otherbreakdown situation, () e() ¼ 0occurs when the LU-decompositionfails, and can be repaired by using another decomposition. This is done in theversion of QMR.

2.4.Algorithm

step1 initial guess (0) = 0

4



step2 e(0) = (0)

step3 (1) = (0)

step4 e(1) = (0)

step5 For = 2 3

¡1 =e(¡1)(¡1)e(¡2)(¡2)

step6 () = (¡1) + (¡1)(¡1)

step7 e() = e(¡1) + (¡1)e(¡1)

step8

= e(¡1) (¡1)

() e()

step9 () = (¡1) + ()

step10 () = (¡1) ¡ ()

step11 e() = e(¡1) ¡ e()

step12 check convergence;continue if necessary

step13 End for

References

[1] Richard L.Burden, J. DouglasFaires, Numerical analysis, 8th ed., CA:Brooks/Cole,2005.

[2]Wikipedia: The Free Encyclopedia, http://en,wikipedia.org/wiki/Electrostatics

[3]Zhaojun Bai ...[etal],Templates for the solution of Linear Systems:BuildingBlocks for Iterative Methods.

5


http://en,wikipedia.org/wiki/Electro


cg bicg method

Documents