Undergraduate Texts in Mathematics

Editors

S. AxlerF.W. Gehring

K.A. Ribet

SpringerNew YorkBerlinHeidelbergBarcelonaHong KongLondonMilanParisSingaporeTokyo

Undergraduate Texts in Mathematics

Anglin: Mathematics: A Concise Historyand Philosophy.Readings in Mathematics.

Anglin/Lambek: The Heritage ofThales.Readings in Mathematics.

Apostol: Introduction to AnalyticNumber Theory. Second edition.

Armstrong: Basic Topology.Armstrong: Groups and Symmetry.Axler: Linear Algebra Done Right.

Second edition.Beardon: Limits: A New Approach to

Real Analysis.Bak/Newman: Complex Analysis.

Second edition.BanchoffyWermer: Linear Algebra

Through Geometry. Second edition.Berberian: A First Course in Real

Analysis.Bix: Conies and Cubics: AConcretem Introduction to AlgebraicCurves.

Bremaud: An Introduction toProbabilistic Modeling.

Bressoud: Factorization and PrimalityTesting.

Bressoud: Second Year Calculus.Readings in Mathematics.

Brickman: Mathematical Introductionto Linear Programming and GameTheory.

Browder: Mathematical Analysis:An Introduction.

Buskes/van Rooij: Topological Spaces:From Distance to Neighborhood.

Cederberg: A Course in ModernGeometries.

Childs: A Concrete Introduction toHigher Algebra. Second edition.

Chung: Elementary Probability Theorywith Stochastic Processes. Thirdedition.

Cox/Little/O'Shea: Ideals, Varieties,and Algorithms. Second edition.

Croom: Basic Concepts of AlgebraicTopology.

Curtis: Linear Algebra: An IntroductoryApproach. Fourth edition.

Devlin: The Joy of Sets: Fundamentalsof Contemporary Set Theory.Second edition.

Dixmier: General Topology.Driver: Why Math?Ebbi nghaus/FIum/Thomas:

Mathematical Logic. Second edition.Edgar: Measure, Topology, and Fractal

Geometry.Elaydi: Introduction to Difference

Equations.Exner: An Accompaniment to Higher

Mathematics.Fine/Rosenberger: The Fundamental

Theory of Algebra.Fischer: Intermediate Real Analysis.Flanigan/Kazdan: Calculus Two: Linear

and Nonlinear Functions. Secondedition.

Fleming: Functions of Several Variables.Second edition.

Foulds: Combinatorial Optimization forUndergraduates.

Foulds: Optimization Techniques: AnIntroduction.

Franklin: Methods of MathematicalEconomics.

Frazier: An Introduction to WaveletsThrough Linear Algebra.

Gordon: Discrete Probability.Hairer/Wanner: Analysis by Its History.

Readings in Mathematics.Halmos: Finite-Dimensional Vector

Spaces. Second edition.Halmos: Naive Set Theory.Ha'mmerlin/Hoffmann: Numerical

Mathematics.Readings in Mathematics.

Hijab: Introduction to Calculus andClassical Analysis.

Hilton/Holton/Pedersen: MathematicalReflections: In a Room with ManyMirrors.

Iooss/Joseph: Elementary Stabilityand Bifurcation Theory. Secondedition.

Isaac: The Pleasures of Probability.Readings in Mathematics.

(continued after index)

Michael W. Frazier

An Introductionto Wavelets ThroughLinear Algebra

With 46 Illustrations

Springer

Michael W. FrazierMichigan State UniversityDepartment of MathematicsEast Lansing, MI 48824USA

Editorial Board

S. AxlerMathematics DepartmentSan Francisco State

UniversitySan Francisco, CA 94132USA

F.W. GehringMathematics DepartmentEast HallUniversity of MichiganAnn Arbor, MI 48109USA

K.A. RibetDepartment of

MathematicsUniversity of California

at BerkeleyBerkeley, CA 94720-3840USA

Mathematics Subject Classification (1991): 42-01 46CXX 65F50

Library of Congress Cataloging-in-Publication DataFrazier, Michael, 1956-

An introduction to wavelets through linear algebra / Michael W. Frazierp. cm. —(Undergraduate texts in mathematics)

Includes bibliographical references and index.ISBN 0-387-98639-1 (hardcover)1. Wavelets (Mathematics) 2. Algebras, Linear. I. Title.

II. Series.QA403.3.F73 1999515'.2433-dc21 98-43866

• 1999 Springer-Verlag New York, Inc.All rights reserved. This work may not be translated or copied in whole or in part withoutthe written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue,New York, NY 10010, USA), except for brief excerpts in connection with reviews or scholarlyanalysis. Use in connection with any form of information storage and retrieval, electronicadaptation, computer software, or by similar or dissimilar methodology now known orhereafter developed is forbidden.The use of general descriptive names, trade names, trademarks, etc., in this publication,even if the former are not especially identified, is not to be taken as a sign that such names,as understood by the Trade Marks and Merchandise Marks Act, may accordingly be usedfreely by anyone.

ISBN 0-387-98639-1 Springer-Verlag New York Berlin Heidelberg SPIN 10557627

Preface

Mathematics majors at Michigan State University take a “Capstone”course near the end of their undergraduate careers. The contentof this course varies with each offering. Its purpose is to bringtogether different topics from the undergraduate curriculum andintroduce students to a developing area in mathematics. This textwas originally written for a Capstone course.

Basic wavelet theory is a natural topic for such a course. By name,wavelets date back only to the 1980s. On the boundary betweenmathematics and engineering, wavelet theory shows students thatmathematics research is still thriving, with important applicationsin areas such as image compression and the numerical solutionof differential equations. The author believes that the essentials ofwavelet theory are sufficiently elementary to be taught successfullyto advanced undergraduates.

This text is intended for undergraduates, so only a basicbackground in linear algebra and analysis is assumed. We do notrequire familiarity with complex numbers and the roots of unity.These are introduced in the first two sections of chapter 1. In theremainder of chapter 1 we review linear algebra. Students should befamiliar with the basic definitions in sections 1.3 and 1.4. From ourviewpoint, linear transformations are the primary object of study;

v

Prefacevi

a matrix arises as a realization of a linear transformation. Manystudents may have been exposed to the material on change of basisin section 1.4, but may benefit from seeing it again. In section 1.5,we ask how to pick a basis to simplify the matrix representation ofa given linear transformation as much as possible. We then focus onthe simplest case, when the linear transformation is diagonalizable.In section 1.6, we discuss inner products and orthonormal bases. Weend with a statement of the spectral theorem for matrices, whoseproof is outlined in the exercises. This is beyond the experience ofmost undergraduates.

Chapter 1 is intended as reference material. Depending onbackground, many readers and instructors will be able to skip orquickly review much of this material. The treatment in chapter 1 isrelatively thorough, however, to make the text as self-contained aspossible, provide a logically ordered context for the subject matter,and motivate later developments.

The author believes that students should be introduced to Fourieranalysis in the finite dimensional context, where everything can beexplained in terms of linear algebra. The key ideas can be exhibitedin this setting without the distraction of technicalities relating toconvergence. We start by introducing the Discrete Fourier Transform(DFT) in section 2.1. The DFT of a vector consists of its componentswith respect to a certain orthogonal basis of complex exponentials.The key point, that all translation-invariant linear transformationsare diagonalized by this basis, is proved in section 2.2. We turn tocomputational issues in section 2.3, where we see that the DFT canbe computed rapidly via the Fast Fourier Transform (FFT).

It is not so well known that the basics of wavelet theory canalso be introduced in the finite dimensional context. This is donein chapter 3. The material here is not entirely standard; it is anadaptation of wavelet theory to the finite dimensional setting. It hasthe advantage that it requires only linear algebra as background. Insection 3.1, we search for orthonormal bases with both space andfrequency localization, which can be computed rapidly. We are ledto consider the even integer translates of two vectors, the mother andfather wavelets in this context. The filter bank arrangement for thecomputation of wavelets arises naturally here. By iterating this filterbank structure, we arrive in section 3.2 at a multilevel wavelet basis.

Preface vii

Examples and applications are discussed in section 3.3. Daubechies’swavelets are presented in this context, and elementary compressionexamples are considered. A student familiar with MatLab, Maple, orMathematica should be able to carry out similar examples if desired.

In section 4.1 we change to the infinite dimensional but discretesetting �2(Z), the square summable sequences on the integers.General properties of complete orthonormal sets in inner productspaces are discussed in section 4.2. This is first point where analysisenters our picture in a serious way. Square integrable functionson the interval [−π, π) and their Fourier series are developed insection 4.3. Here we have to cheat a little bit: we note that weare using the Lebesgue integral but we don’t define it, and weask students to accept certain of its properties. We arrive again atthe key principle that the Fourier system diagonalizes translation-invariant linear operators. The relevant version of the Fouriertransform in this setting is the map taking a sequence in �2(Z)to a function in L2([−π, π)) whose Fourier coefficients make upthe original sequence. Its properties are presented in section 4.4.Given this preparation, the construction of first stage wavelets onthe integers (section 4.5) and the iteration step yielding a multilevelbasis (section 4.6) are carried out in close analogy to the methodsin chapter 3. The computation of wavelets in the context of �2(Z)is discussed in section 4.7, which includes the construction ofDaubechies’s wavelets on Z. The generators u and v of a waveletsystem for �2(Z) reappear in chapter 5 as the scaling sequence andits companion.

The usual version of wavelet theory on the real line is presentedin chapter 5. The preliminaries regarding square integrable func-tions and the Fourier transform are discussed in sections 5.1 and 5.2.The facts regarding Fourier inversion in L2(R) are proved in detail,although many instructors may prefer to assume these results. TheFourier inversion formula is analogous to an orthonormal basis rep-resentation, using an integral rather than a sum. Again we see thatthe Fourier system diagonalizes translation-invariant operators. Mal-lat’s theorem that a multiresolution analysis yields an orthonormalwavelet basis is proved in section 5.3. The aformentioned relationbetween the scaling sequence and wavelets on �2(Z) allows us tomake direct use of the results of chapter 4. The conditions under

Prefaceviii

which wavelets on �2(Z) can be used to generate a multiresolutionanalysis, and hence wavelets on R, are considered in section 5.4.In section 5.5, we construct Daubechies’s wavelets of compact sup-port, and show how the wavelet transform is implemented usingfilter banks.

We briefly consider the application of these results to numericaldifferential equations in chapter 6. We begin in section 6.1 witha discussion of the condition number of a matrix. In section6.2, we present a simple example of the numerical solution of aconstant coefficient ordinary differential equation on [0, 1] usingfinite differences. We see that although the resulting matrix issparse, which is convenient, it has a condition number that growsquadratically with the size of the matrix. By comparison, in section6.3, we see that for a wavelet-Galerkin discretization of a uniformlyelliptic, possibly variable-coefficient, differential equation, thematrix of the associated linear system can be preconditioned to besparse and to have bounded condition number. The boundednessof the condition number comes from a norm equivalence propertyof wavelets that we state without proof. The sparseness of theassociated matrix comes from the localization of the wavelet system.A large proportion of the time, the orthogonality of wavelet basismembers comes from their supports not overlapping (using waveletsof compact support, say). This is a much more robust property,for example with respect to multiplying by a variable coefficientfunction, than the delicate cancellation underlying the orthogonalityof the Fourier system. Thus, although the wavelet system may notexactly diagonalize any natural operator, it nearly diagonalizes (inthe sense of the matrix being sparse) a much larger class of operatorsthan the Fourier basis.

Basic wavelet theory includes aspects of linear algebra, realand complex analysis, numerical analysis, and engineering. Inthis respect it mimics modern mathematics, which is becomingincreasingly interdisciplinary.

This text is relatively elementary at the start, but the levelof difficulty increases steadily. It can be used in different waysdepending on the preparation level of the class. If a long time isrequired for chapter 1, then the more difficult proofs in the laterchapters may have to be only briefly outlined. For a more advanced

Preface ix

group, most or all of chapter 1 could be skipped, which would leavetime for a relatively thorough treatment of the remainder. A shortercourse for a more sophisticated audience could start in chapter4 because the main material in chapters 4 and 5 is technically,although not conceptually, independent of the content of chapters2 and 3. An individual with a solid background in Fourier analysiscould learn the basics of wavelet theory from sections 4.5, 4.7, 5.3,5.4, 5.5, and 6.3 with only occasional references to the remainder ofthe text.

This volume is intended as an introduction to wavelet theorythat is as elementary as possible. It is not designed to be a thoroughreference. We refer the reader interested in additional informationto the Bibliography at the end of the text.

Michigan State University M. Frazier

April 1999

Acknowledgments

This text owes a great deal to a number of my colleagues andstudents. The discrete presentation in Chapters 3 and 4 wasdeveloped in joint work (Frazier and Kumar, 1993) with Arun Kumar,in our early attempt to understand wavelets. This was furtherclarified in consulting work done with Jay Epperson at Daniel H.Wagner Associates in California. Many of the graphs in this textare similar to examples done by Douglas McCulloch during thisconsulting project. Additional insight was gained in subsequent workwith Rodolfo Torres.

My colleagues at Michigan State University provided assistancewith this text in various ways. Patti Lamm read a preliminary versionin its entirety and made more than a hundred useful suggestions,including some that led to a complete overhaul of section 6.2. Shealso provided computer assistance with the figures in the Prologue.Sheldon Axler supplied technical assistance and made suggestionsthat improved the style and presentation throughout the manuscript.T.-Y. Li made a number of helpful suggestions, including providingme with Exercise 1.6.20. Byron Drachman helped with the index.

I have had the opportunity to test preliminary versions ofthis text in the classroom on several occasions. It was used atMichigan State University in a course for undergraduates in spring

xi

Acknowledgmentsxii

1996 and in a beginning graduate course in summer 1996. Theadministration of the Mathematics Department, especially Jon Hall,Bill Sledd, and Wei-Eihn Kuan, went out of their way to provide theseopportunities. The students in these classes made many suggestionsand corrections, which have improved the text. Gihan Mandour,Jian-Yu Lin, Rudolf Blazek, and Richard Andrusiak made largenumbers of corrections.

This text was also the basis for three short courses on wavelets.One of these was presented at the University of Puerto Rico atMayaguez in the spring of 1997. I thank Nayda Santiago for helpingarrange the visit, and Shawn Hunt, Domingo Rodriguez, and RamonVasquez for inviting me and for their warm hospitality. Anothershort course was given at the University of Missouri at Columbiain fall 1997. I thank Elias Saab and Nakhle Asmar for makingthis possible. The third short course took place at the Instituto deMatematicas de la UNAM in Cuernavaca, Mexico in summer 1998.I thank Professors Salvador Perez-Esteva and Carlos Villegas Blasfor their efforts in arranging this trip, and for their congenialitythroughout. The text in preliminary form has also been used incourses given by Cristina Pereyra at the University of New Mexicoand by Suzanne Tourville at Carnegie-Mellon University. Cristina,Suzanne, and their students provided valuable feedback and anumber of corrections, as did Kees Onneweer.

My doctoral students Kunchuan Wang and Mike Nixon mademany helpful suggestions and found a number of corrections in themanuscript. My other doctoral student, Shangqian Zhang, taught methe mathematics in Section 6.3. I also thank him and his son SimonZhang for Figure 35.

The fingerprint examples in Figures 1–3 in the Prologue wereprovided by Chris Brislawn of the Los Alamos National Laboratories.I thank him for permission to reproduce these images. Figures36e and f were prepared using a program (Summus 4U2C 3.0)provided to me by Bjorn Jawerth and Summus Technologies, Inc,for which I am grateful. Figures 36b, c, and d were created using thecommercially available software WinJPEG v.2.84. The manuscriptand some of the figures were prepared using LaTEX. The other figureswere done using MatLab. Steve Plemmons, the computer managerin the mathematics department at Michigan State University, aided

Acknowledgments xiii

in many ways, particularly with regard to the images in Figure36. I thank Ina Lindemann, my editor at Springer-Verlag, for herassistance, encouragement, and especially her patience.

I take this opportunity to thank the mathematicians whose aidwas critical in helping me reach the point where it became possiblefor me to write this text. The patience and encouragement of mythesis advisor John Garnett was essential at the start. My earlycollaboration with Bjorn Jawerth played a decisive role in my career.My postdoctoral advisor Guido Weiss encouraged and helped me inmany important ways over the years.

This text was revised and corrected during a sabbatical leaveprovided by Michigan State University. This leave was spent atthe University of Missouri at Columbia. I thank the University ofMissouri for their hospitality and for providing me with valuableresources and technical assistance.

At a time when academic tenure is under attack, it is worthcommenting that this text and many others like it would not havebeen written without the tenure system.

Contents

Preface v

Acknowledgments xi

Prologue: Compression of the FBI Fingerprint Files 1

1 Background: Complex Numbers and Linear Algebra 71.1 Real Numbers and Complex Numbers . . . . . . . . 71.2 Complex Series, Euler’s Formula, and the Roots of

Unity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.3 Vector Spaces and Bases . . . . . . . . . . . . . . . . . 291.4 Linear Transformations, Matrices, and Change of

Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401.5 Diagonalization of Linear Transformations and

Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 561.6 Inner Products, Orthonormal Bases, and Unitary

Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 79

2 The Discrete Fourier Transform 1012.1 Basic Properties of the Discrete Fourier Transform . 1012.2 Translation-Invariant Linear Transformations . . . . 128

xv

Contentsxvi

2.3 The Fast Fourier Transform . . . . . . . . . . . . . . . 151

3 Wavelets on ZN 1653.1 Construction of Wavelets on ZN : The First Stage . . 1653.2 Construction of Wavelets on ZN : The Iteration Step . 1963.3 Examples and Applications . . . . . . . . . . . . . . . 225

4 Wavelets on Z 2654.1 �2(Z) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2654.2 Complete Orthonormal Sets in Hilbert Spaces . . . . 2714.3 L2([−π, π)) and Fourier Series . . . . . . . . . . . . . 2794.4 The Fourier Transform and Convolution on �2(Z) . . 2984.5 First-Stage Wavelets on Z . . . . . . . . . . . . . . . . 3094.6 The Iteration Step for Wavelets on Z . . . . . . . . . 3214.7 Implementation and Examples . . . . . . . . . . . . 330

5 Wavelets on R 3495.1 L2(R) and Approximate Identities . . . . . . . . . . . 3495.2 The Fourier Transform on R . . . . . . . . . . . . . . 3625.3 Multiresolution Analysis and Wavelets . . . . . . . . 3805.4 Construction of Multiresolution Analyses . . . . . . 3985.5 Wavelets with Compact Support and Their

Computation . . . . . . . . . . . . . . . . . . . . . . . 429

6 Wavelets and Differential Equations 4516.1 The Condition Number of a Matrix . . . . . . . . . . 4516.2 Finite Difference Methods for Differential

Equations . . . . . . . . . . . . . . . . . . . . . . . . . 4596.3 Wavelet-Galerkin Methods for Differential

Equations . . . . . . . . . . . . . . . . . . . . . . . . . 470

Bibliography 484

Index 491