S E C O N D E D I T I O N

Introduction to Data Compression

Khalid Sayood University of Nebraska-Lincoln

: < e

M O R G A N K A U F M A N N P U B L I S H E R S

A N I M P R I N T O F A C A D E M I C P R E S S

A Harcourt Science and Technology Company

Amsterdam Boston Heidelberg London New York Oxford Paris San Diego San Francisco Singapore Sydney Tokyo

Contents

Preface xv

1 Introduction 1 1.1 Compression Techniques 3

1.1.1 Lossless Compression 4 1.1.2 Lossy Compression 5 1.1.3 Measures of Performance 5

1.2 Modeling and Coding 6 1.3 Summary 10 1.4 Projects and Problems 10

2 Mathematical Preliminaries for Lossless Compression 13 2.1 Overview 13 2.2 A Brief Introduction to Information Theory 13

2.2.1 Derivation of Average Information * 18 2.3 Models 22

2.3.1 Physical Models 23 2.3.2 Probability Models 23 2.3.3 Markov Models 23 2.3.4 Composite Source Model 26

2.4 Coding 27 2.4.1 Uniquely Decodable Codes 27 2.4.2 Prefix Codes 31 2.4.3 The Kraft-McMillan Inequality • 31

2.5 Summary 35 2.6 Projects and Problems 35

3 Huffman Coding 39 3.1 Overview 39 3.2 The Huffman Coding Algorithm 39

3.2.1 Minimum Variance Huffman Codes 43 3.2.2 Optimality of Huffman Codes • 45 3.2.3 Length of Huffman Codes -k 46 3.2.4 Extended Huffman Codes • 49

3.3 Nonbinary Huffman Codes * 52

viii C O N T E N T S

3.4 Adaptive Huffman Coding 55 3.4.1 Update Procedure 56 3.4.2 Encoding Procedure 59 3.4.3 Decoding Procedure 60

3.5 Golomb Codes 61 3.6 Rice Codes 64

3.6.1 CCSDS Recommendation for Lossless Compression 64 3.7 Tunstall Codes 66 3.8 Applications of Huffman Coding 68

3.8.1 Lossless Image Compression 69 3.8.2 Text Compression 71 3.8.3 Audio Compression 71

3.9 Summary 73 3.10 Projects and Problems 74

4 Arithmetic Coding 77 4.1 Overview 77 4.2 Introduction 77 4.3 Coding a Sequence 79

4.3.1 Generating a Tag 80 4.3.2 Deciphering the Tag 86

4.4 Generating a Binary Code 88 4.4.1 Uniqueness and Efficiency of the Arithmetic Code 88 4.4.2 Algorithm Implementation 91 4.4.3 Integer Implementation 97

4.5 Comparison of Huffman and Arithmetic Coding 104 4.6 Applications 106

4.6.1 Bi-level Image Compression—The JBIG Standard 106 4.6.2 JBIG2 I l l 4.6.3 Image Compression 113

4.7 Summary 114 4.8 Projects and Problems 115

5 Dictionary Techniques 117 5.1 Overview 117 5.2 Introduction 117 5.3 Static Dictionary 118

5.3.1 Digram Coding 119 5.4 Adaptive Dictionary 120

5.4.1 The LZ77 Approach 120 5.4.2 The LZ78 Approach 125

5.5 Applications 133 5.5.1 File Compression—UNIX c o m p r e s s 133 5.5.2 Image Compression—The Graphics Interchange Format (GIF) . . 133 5.5.3 Compression over Modems—V.42bis 134

C O N T E N T S ix

5.6 Summary 136 5.7 Projects and Problems 136

6 Predictive Coding 139 6.1 Overview 139 6.2 Introduction 139 6.3 Prediction with Partial Match (ppm) 141

6.3.1 The Basic Algorithm 141 6.3.2 The Escape Symbol 145 6.3.3 Length of Context 148 6.3.4 The Exclusion Principle 148

6.4 The Burrows-Wheeler Transform 149 6.4.1 Move-to-Front Coding 154

6.5 CALIC 155 6.6 JPEG-LS 160

6.6.1 "Current" Standard 160 6.6.2 "New" Standard 161

6.7 Multiresolution Approaches 163 6.7.1 Progressive Image Transmission 164

6.8 Facsimile Encoding 169 6.8.1 Run-Length Coding 169 6.8.2 CCITT Group 3 and 4—Recommendations T.4

and T.6 170 6.8.3 Comparison of MH, MR, MMR, and JBIG 173

6.9 Dynamic Markov Compression 174 6.10 Summary 176 6.11 Projects and Problems 177

7 Mathematical Preliminaries for Lossy Coding 181 7.1 Overview 181 7.2 Introduction 181 7.3 Distortion Criteria 183

7.3.1 The Human Visual System 185 7.3.2 Auditory Perception 186

7.4 Information Theory Revisited * 187 7.4.1 Conditional Entropy 188 7.4.2 Average Mutual Information 190 7.4.3 Differential Entropy 191

7.5 Rate Distortion Theory • 194 7.6 Models 201

7.6.1 Probability Models 201 7.6.2 Linear System Models . . . 203 7.6.3 Physical Models 209

7.7 Summary 210 7.8 Projects and Problems 210

x C O N T E N T S

8 Scalar Quantization 213 8.1 Overview 213 8.2 Introduction 213 8.3 The Quantization Problem 214 8.4 Uniform Quantizer 219 8.5 Adaptive Quantization 229

8.5.1 Forward Adaptive Quantization 230 8.5.2 Backward Adaptive Quantization 232

8.6 Nonuniform Quantization 238 8.6.1 pdf-Optimized Quantization 239 8.6.2 Companded Quantization 243

8.7 Entropy-Coded Quantization 249 8.7.1 Entropy Coding of Lloyd-Мах Quantizer Outputs 250 8.7.2 Entropy-Constrained Quantization * 250 8.7.3 High-Rate Optimum Quantization -Ar 251

8.8 Summary 254 8.9 Projects and Problems 255

9 Vector Quantization 257 9.1 Overview 257 9.2 Introduction 257 9.3 Advantages of Vector Quantization over Scalar Quantization 260 9.4 The Linde-Buzo-Gray Algorithm 266

9.4.1 Initializing the LBG Algorithm 269 9.4.2 The Empty Cell Problem 278 9.4.3 Use of LBG for Image Compression 278

9.5 Tree-Structured Vector Quantizers 283 9.5.1 Design of Tree-Structured Vector Quantizers 286 9.5.2 Pruned Tree-Structured Vector Quantizers 287

9.6 Structured Vector Quantizers 287 9.6.1 Pyramid Vector Quantization 289 9.6.2 Polar and Spherical Vector Quantizers 290 9.6.3 Lattice Vector Quantizers 291

9.7 Variations on the Theme 295 9.7.1 Gain-Shape Vector Quantization 295 9.7.2 Mean-Removed Vector Quantization 295 9.7.3 Classified Vector Quantization 296 9.7.4 Multistage Vector Quantization 297 9.7.5 Adaptive Vector Quantization 299

9.8 Trellis-Coded Quantization 300 9.9 Summary 305 9.10 Projects and Problems 306

10 Differential Encoding 309 10.1 Overview 309 10.2 Introduction 309

C O N T E N T S xi

10.3 The Basic Algorithm 312 10.4 Prediction in DPCM 316 10.5 Adaptive DPCM 321

10.5.1 Adaptive Quantization in DPCM 322 10.5.2 Adaptive Prediction in DPCM 322

10.6 Delta Modulation 326 10.6.1 Constant Factor Adaptive Delta Modulation (CFDM) 327 10.6.2 Continuously Variable Slope Delta Modulation 328

10.7 Speech Coding 329 10.7.1 G.726 331

10.8 Image Coding 333 10.9 Summary 334 10.10 Projects and Problems 335

11 Mathematical Preliminaries for Transforms, Subbands, and Wavelets 337 11.1 Overview 337 11.2 Introduction 337 11.3 Vector Spaces 338

11.3.1 Dot or Inner Product 339 11.3.2 Vector Space 339 11.3.3 Subspace 341 11.3.4 Basis 342 11.3.5 Inner Product—Formal Definition 343 11.3.6 Orthogonal and Orthonormal Sets 343

11.4 Fourier Series 344 11.5 Fourier Transform 346

11.5.1 Parseval's Theorem 348 11.5.2 Modulation Property 348 11.5.3 Convolution Theorem 349

11.6 Linear Systems 349 11.6.1 Time Invariance 350 11.6.2 Transfer Function 350 11.6.3 Impulse Response 351 11.6.4 Filter 352

11.7 Sampling 354 11.7.1 Ideal Sampling—Frequency Domain View 354 11.7.2 Ideal Sampling—Time Domain View 356

11.8 Discrete Fourier Transform 357 11.9 Z-Transform 360

11.9.1 Tabular Method 363 11.9.2 Partial Fraction Expansion 364 11.9.3 Long Division 367 11.9.4 Z-Transform Properties 368 11.9.5 Discrete Convolution 369

xii C O N T E N T S

11.10 Summary 370 11.11 Projects and Problems 371

12 Transform Coding 373 12.1 Overview 373 12.2 Introduction 373 12.3 The Transform 378 12.4 Transforms of Interest 382

12.4.1 Karhunen-Loeve Transform 383 12.4.2 Discrete Cosine Transform 384 12.4.3 Discrete Sine Transform 385 12.4.4 Discrete Walsh-Hadamard Transform 386

12.5 Quantization and Coding of Transform Coefficients 389 12.6 Application to Image Compression—JPEG 392

12.6.1 The Transform 392 12.6.2 Quantization 393 12.6.3 Coding 394

12.7 Application to Audio Compression 399 12.8 Summary 401 12.9 Projects and Problems 402

13 Subband Coding 405 13.1 Overview 405 13.2 Introduction 405 13.3 Filters 410

13.3.1 Some Filters Used in Subband Coding 414 13.4 The Basic Subband Coding Algorithm 416

13.4.1 Analysis 416 13.4.2 Quantization and Coding 419 13.4.3 Synthesis 419

13.5 Design of Filter Banks * 420 13.5.1 Downsampling * 421 13.5.2 Upsampling • 424

13.6 Perfect Reconstruction Using Two-Channel Filter Banks ~k 425 13.6.1 Two-Channel PR Quadrature Mirror Filters • 428 13.6.2 Power Symmetric FIR Filters • 430

13.7 M-Band QMF Filter Banks * 432 13.8 The Polyphase Decomposition -k 435 13.9 Bit Allocation 440 13.10 Application to Speech Coding—G.722 442 13.11 Application to Audio Coding—MPEG Audio 443 13.12 Application to Image Compression 444

13.12.1 Decomposing an Image 446 13.12.2 Coding the Subbands 447

C O N T E N T S xiii

13.13 Summary 451 13.14 Projects and Problems 452

14 Wavelet-Based Compression 455 14.1 Overview 455 14.2 Introduction 455 14.3 Wavelets 458 14.4 Multiresolution Analysis and the Scaling Function 462 14.5 Implementation Using Filters 468

14.5.1 Scaling and Wavelet Coefficients 471 14.5.2 Families of Wavelets 473

14.6 Image Compression 474 14.7 Embedded Zerotree Coder 479 14.8 Set Partitioning in Hierarchical Trees 486 14.9 JPEG 2000 493 14.10 Summary 493 14.11 Projects and Problems 494

15 Analysis/Synthesis Schemes 497 15.1 Overview 497 15.2 Introduction 497 15.3 Speech Compression 499

15.3.1 The Channel Vocoder 499 15.3.2 The Linear Predictive Coder (Government Standard LPC-10) . . . 502 15.3.3 Code Excited Linear Predicton (CELP) 509 15.3.4 Sinusoidal Coders 512 15.3.5 Mixed Excitation Linear Prediction (MELP) 515

15.4 Image Compression 517 15.4.1 Fractal Compression 517

15.5 Summary 526 15.6 Projects and Problems 527

16 Video Compression 529 16.1 Overview 529 16.2 Introduction 529 16.3 Motion Compensation 531 16.4 Video Signal Representation 534 16.5 Algorithms for Videoconferencing and Videophones 539

16.5.1 ITU-T Recommendation H.261 540 16.5.2 Model-Based Coding 546

16.6 Asymmetric Applications 548 16.6.1 TheMPEG-1 Video Standard 548 16.6.2 The MPEG-2 Video Standard 552 16.6.3 MPEG-4 554 16.6.4 MPEG-7 556

xiv C O N T E N T S

16.7 Packet Video 556 16.7.1 ATM Networks 556 16.7.2 Compression Issues in ATM Networks 557 16.7.3 Compression Algorithms for Packet Video 558

16.8 Summary 559 16.9 Projects and Problems 560

A Probability and Random Processes 561 A.l Probability 561

A.1.1 Frequency of Occurrence 561 A. 1.2 A Measure of Belief 562 A.1.3 The Axiomatic Approach 564

A.2 Random Variables 566 A.3 Distribution Functions 567 A.4 Expectation 569

A.4.1 Mean 571 A.4.2 Second Moment 571 A.4.3 Variance 571

A.5 Types of Distribution 571 A.5.1 Uniform Distribution 571 A.5.2 Gaussian Distribution 572 A.5.3 Laplacian Distribution 572 A.5.4 Gamma Distribution 572

A.6 Stochastic Process 573 A.7 Projects and Problems 575

В A Brief Review of Matrix Concepts 577 B.l A Matrix 577 B.2 Matrix Operations 578

С The Root Lattices 583

Bibliography 585

Index 601