Introduction to

The Design & Analysis of Algorithms ~~ I_ND EDIIIQN ~*

Introduction to~

The Design & Analysis of Algorithms IND EDITION "~

~

Anany LevitinVillanova University

Boston San Francisco New York London Toronto Sydney Tokyo Singapore Madrid Mexico City Munich Paris Cape Town Hong Kong Montreal

Publisher Acquisitions Editor Project Editor Production Supervisor Marketing Manager Project Management & Composition Technical Art Copyeditor Cover Design Cover Illustration Senior Pre press Supervisor

Greg Tobin Matt Goldstein Katherine Harutunian Marilyn Lloyd Michelle Brown Windfall Software, using ZzTX George Nichols Richard Camp Joyce Cosentino Wells Jennifer M. Kohnke Caroline Fell

Access the latest information about Addison-Wesley titles from our World Wide Web site: http://www.aw-bc.com/cornputing Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and AddisonWesley was aware of a trademark claim, the designations have been printed in initial caps or all caps.If you purchased this book within the United States or Canada you should be aware that

it has been wrongfully imported without the approval of the Publisher or the Author. Copyright 2007 by Pearson Education, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America.

ISBN 0-321-36413-9 3 4 5 6 7 8 9 10-DOH-09 08 07

Brief Contents

Preface1 Introduction

xvii

2 Fundamentals of the Analysis of Algorithm Efficiency3 Brute Force

4 Divide-and-Conquer 5 Decrease-and-Conquer 6 Transform-and-Conquer 1 Space and Time Tradeoffs 8 Dynamic Programming 9 Greedy Technique 1 0 Iterative Improvement11 limitations of Algorithm Power

12 Coping with the Limitations of Algorithm PowerEpilogueAPPENDIX A

1 41 97 123 157 197 249 279 307 335 379 415 465 469 473 487 497 541v

Useful Formulas for the Analysis of AlgorithmsAPPENDIX B

Short Tutorial on Recurrence Relations Bibliography Hints to Exercises Index

r..

I;

I!

Contents

Preface

xvii

1 Introduction1.1 What is an Algorithm?Exercises 1.1

13

89 9 11 11 12 12 12 13 14 15 17 19 19 20 21 21 22 22 23 23

1.2 Fundamentals of Algorithmic Problem SolvingUnderstanding the Problem Ascertaining the Capabilities of a Computational Device Choosing between Exact and Approximate Problem Solving Deciding on Appropriate Data Structures Algorithm Design Techniques Methods of Specifying an Algorithm Proving an Algorithm's Correctness Analyzing an Algorithm Coding an Algorithm Exercises 1.2

1.3 Important Problem TypesSorting Searching String Processing Graph Problems Combinatorial Problems Geometric Problems Numerical Problems Exercises 1.3

vii

---~-----

I;!'jli

viii

Contents

rll

!:!

1.4 Fundamental Data StructuresLinear Data Structures Graphs Trees Sets and Dictionaries Exercises 1.4 Summary

II: .,il

,.'II I!1:1

26 26 28 32 36 38 39

f:i

I'II:1\,

2

fundamentals of the Analysis of Algorithm EfficiencyMeasuring an Input's Size Units for Measuring Running Time Orders of Growth Worst-Case. Best-Case, and Average-Case Efficiencies Recapitulation of the Analysis Framework Exercises 2.1

41 42 43 44 45 47 50 50 52 52 53 54 55 56 57 58 59 61 67 69 76 78 79 80 83

I'-,,!i!i

if!li1

2.1 Analysis Framework

III!I

'

'

2.2 Asymptotic Notations and Basic Efficiency ClassesInformal Introduction a-notation 0-notation e-notation Useful Property Involving the Asymptotic Notations Using Limits for Comparing Orders of Growth Basic Efficiency Classes Exercises 2.2

2.3 Mathematical Analysis of Non recursive Algorithms Exercises 2.3 2.4 Mathematical Analysis of Recursive Algorithms Exercises 2.4 2.5 Example: Fibonacci NumbersExplicit Formula for the nth Fibonacci Number Algorithms for Computing Fibonacci Numbers Exercises 2.5

Contents 2.6 Empirical Analysis of Algorithms Exercises 2.6 2.7 Algorithm Visualization Summary

ix 84 90 91 95

3

Brute Force

97 98 98 100 102 103 103 104 105 107 107 108 112 114 114 115 116 119 120

3.1 Selection Sort and Bubble Sort Selection Sort Bubble Sort Exercises 3.1 3.2 Sequential Search and Brute-Force String Matching Sequential Search Brute-Force String Matching Exercises 3.2 3.3 Closest-Pair and Convex-Hull Problems by Brute Force Closest-Pair Problem Convex-Hull Problem Exercises 3.3 3.4 Exhaustive Search Traveling Salesman Problem Knapsack Problem Assignment Problem Exercises 3.4 Summary

4

Divide-and-ConquerExercises 4.1

123125 128 129 134 135 138

4.1 Mergesort

4.2 Quicksort Exercises 4.2 4.3 Binary Search Exercises 4.3

r)(

111

Contents 4.4 Binary Tree Traversals and Related Properties Exercises 4.4 4.5 Multiplication of Large Integers and Strassen's Matrix Multiplication Multiplication of Large Integers Strassen's Matrix Multiplication Exercises 4.5 139 142 144 144 146 148 149 149 150 154 155

I'I:]

i::!

,,

i'' ,.,

l:i

IIII

4.6 Closest-Pair and Convex-Hull Problems by Divide-and-Conquer Closest-Pair Problem Convex-Hull Problem Exercises 4.6 Summary

!I I.,

II:,f ,,iIii

I!

iii

5

Decrease-and-Conquer

157 160 163 164 165 167 170 172 176

:l i 'I; l!i'I',1

5.1 Insertion Sort Exercises 5.1 5.2 Depth-First Search and Breadth-First Search Depth-First Search Breadth-First Search Exercises 5.2 5.3 Topological Sorting Exercises 5.3 5.4 Algorithms for Generating Combinatorial Objects Generating Permutations Generating Subsets Exercises 5.4 5.5 Decrease-by-a-Constant-Factor Algorithms Fake-Coin Problem Multiplication a Ia Russe Josephus Problem Exercises 5.5 5.6 Variable-Size-Decrease Algorithms Computing a Median and the Selection Problem

:!I !'I

IIII!I il!i ,,

j,

,,lr

i!f.!'I lr

177 178 180 181183 183 184 185 187 188 188

Contents

xi 190 191 192 194 195

Interpolation Search Searching and Insertion in a Binary Search Tree The Game of Nim Exercises 5.6 Summary

66.1 6.2

Transform-and-ConquerPresorting Exercises 6.1 Gaussian Elimination LU Decomposition and Other Applications Computing a Matrix Inverse Computing a Determinant Exercises 6.2

197 198 201 203 208 209 210 212 214 214 218 222 223 223 227 229 230 231 233 236 237 238 239 240 241 244 245 247

6.3

Balanced Search Trees AVL Trees 2-3 Trees Exercises 6.3

6.4

Heaps and Heapsort Notion of the Heap Heapsort Exercises 6.4

6.5

Horner's Rule and Binary Exponentiation Horner's Rule Binary Exponentiation Exercises 6.5

6.6

Problem Reduction Computing the Least Common Multiple Counting Paths in a Graph Reduction of Optimization Problems Linear Programming Reduction to Graph Problerns Exercises 6.6 Summary

8

Dynamic Programming

279280 282 284 284 288 292 293 298 299301

8.1 Computing a Binomial Coefficient Exercises 8.1

8.2 Warshall's and Floyd's AlgorithmsWarshall"s Algorithm Floyd"s Algorithm for the All-Pairs Shortest-Paths Problem

Exercises 8.2

8.3 Optimal Binary Search Trees Exercises 8.3 8.4 The Knapsack Problem and Memory FunctionsMemory Functions

Exercises 8.4 Summary

303 304

Contents

xiii

Greedy Technique9.1 Prim's Algorithm Exercises 9.1 9.2 Kruskal's Algorithm Disjoint Subsets and Union-Find Algorithms Exercises 9.2 9.3 Dijkstra's Algorithm Exercises 9.3 9.4 Huffman Trees Exercises 9.4 Summary

307 308 313 315 317 321 323 327 328 332 333

10 Iterative Improvement10.1 The Simplex Method Geometric Interpretation of Linear Programming An Outline of the Simplex Method Further Notes on the Simplex Method Exercises 10.1 10.2 The Maximum-Flow Problem Exercises 10.2 10.3 Maximum Matching in Bipartite Graphs Exercises 10.3 10.4 The Stable Marriage Problem Exercises 10.4 Summary

335336 337 341 347 349 351 361 363 369 371 375 376

11

Limitations of Algorithm Power

379380 381 382

11.1 Lower-Bound Arguments Trivial Lower Bounds Information-Theoretic Arguments

~~

xiv

Contents

Adversary Arguments Problem Reduction Exercises 11.1 11.2 Decision Trees Decision Trees for Sorting Algorithms Decision Trees for Searching a Sorted Array Exercises 11.2 11.3 P, NP, and NP-complete Problems P and NP Problems NP-Complete Problems Exercises 11.3 11.4 Challenges of Numerical Algorithms Exercises 11.4 Summary

382 383 385 386 387 390 392 393 394 398 401 404 411 413

12 Coping with the Limitations of Algorithm Power12.1 Backtracking n-Oueens Problem Hamiltonian Circuit Problem Subset-Sum Problem General Remarks Exercises 12.1 12.2 Branch-and-Bound Assignment Problem Knapsack Problem Traveling Salesman Problem Exercises 12.2 12.3 Approximation Algorithms for NP-hard Problems Approximation Algorithms for the Traveling Salesman Problem Approximation Algorithms for the Knapsack Problem Exercises 12.3 12.4 Algorithms for Solving Nonlinear Equations Bisection Method Method of False Position

415416 417 418 419 421 422 424 425 428 430 432 434 436 446 451 452 454 457

Contents

XV

Newton's Method Exercises 12.4 Summary

458461 462

EpilogueAPPENDIX A

465

Useful formulas for the Analysis of AlgorithmsProperties of Logarithms Combinatorics Important Summation Formulas Sum Manipulation Rules Approximation of a Sum by a Definite Integral Floor and Ceiling Formulas Miscellaneous

469469 469 470 470 471 471 471

APPENDIX 18

Short Tutorial on Recurrence RelationsSequences and Recurrence Relations Methods for Solving Recurrence Relations Common Recurrence Types in Algorithm Analysis

473473 474 479

Bibliography Hints to Exercises Index

487 497 541

r

I

Preface

The most valuable acquisitions in a scientific or technical education are the general-purpose mental tools which remain serviceable for a life-time.-George Forsythe, "What to do till the computer scientist comes," 1968

lgorithms play the central role in both the science and practice of computing. Recognition of this fact has Jed to the appearance of a considerable number of textbooks on the subject. By and large, they follow one of two alternatives in presenting algorithms. One classifies algorithms according to a problem type. Such a book would have separate chapters on algorithms for sorting, searching, graphs, and so on. The advantage of this approach is that it allows an immediate comparison of, say, the efficiency of different algorithms for the same problem. The drawback of this approach is that it emphasizes problem types at the expense of algorithm design techniques. The second alternative organizes the presentation around algorithm design techniques. In this organization, algorithms from different areas of computing are grouped together if they have the same design approach. I share the belief of many (e.g., [BaY95]) that this organization is more appropriate for a basic course on the design and analysis of algorithms. There are three principal reasons for emphasis on algorithm design techniques. First, these techniques provide a student with tools for designing algorithms for new problems. This makes learning algorithm design techniques a very valuable endeavor from the practical standpoint. Second, they seek to classify multitudes of known algorithms according to an underlying design idea. Learning to see such commonality among algorithms from different application areas should be a major goal of computer science education. After all, every science considers classification of its principal subject as a major if not the central point of its discipline. Third, in my opinion, algorithm design techniques have considerable utility as general problem solving strategies, applicable to problems beyond computing.xvii

A

xvm

Preface Unfortunately, the traditional classification of algorithm design techniques has several serious shortcomings from both theoretical and educational points of view. The most significant of these shortcomings is the failure to classify many important algorithms. This limitation has forced the authors of textbooks to depart from the design technique organization and to include chapters dealing with specific problem types. Such a switch leads to a loss of course coherence and almost unavoidably creates confusion in students' minds.

New Taxonomy of Algorithm Design TechniquesMy frustration with the shortcomings of the traditional classification of algorithm design techniques has motivated me to develop a new taxonomy of them [Lev99], which is the basis of this book. Here are the principal advantages of the new taxonomy:111

Ill

'"

'"

The new taxonomy is more comprehensive than the traditional one. It includes several strategies-brute force, decrease-and-conquer, transform-andconquer, space and time tradeoffs, and iterative improvement-that are rarely if ever recognized as important design paradigms. The new taxonomy covers naturally many classic algorithms (Euclid's algorithm, heapsort, search trees, hashing, topological sorting, Gaussian elimination, Horner's rule, to name a few) that the traditional taxonomy cannot classify. As a result, the new taxonomy makes it possible to present the standard body of classic algorithms in a unified and coherent fashion. It naturally accommodates the existence of important varieties of several design techniques. (For example, it recognizes three variations of decreaseand-conquer and three variations of transform-and-conquer.) It is better aligned with analytical methods for efficiency analysis (see Appendix B).

Design Techniques as General Problem Solving StrategiesMost applications of the design techniques in the book are to classic problems of computer science. (The only innovation here is the inclusion of some material on numerical algorithms, which are covered within the same general framework.) But these design techniques can be considered general problem solving tools, whose applications are not limited to traditional computing and mathematical problems. Two factors make this point particularly important. First, more and more computing applications go beyond the traditional domain, and there are reasons to believe that this trend will strengthen in the future. Second, developing students' problem solving skills has come to be recognized as a major goal of college education. Among all the courses in a computer science curriculum, a

L

Preface

xix

course on the design and analysis of algorithms is uniquely suitable for this task because it can offer a student specific strategies for solving problems. I am not proposing that a course on the design and analysis of algorithms should become a course on general problem solving. But I do believe that the unique opportunity provided by studying the design and analysis of algorithms should not be missed. Toward this goal, the book includes applications to puzzles and puzzle-like games. Although using puzzles in teaching algorithms is certainly not a new idea, the book tries to do so systematically by going well beyond a few standard examples.

Textbook PedagogyMy goal was to write a text that would not trivialize the subject but would still be readable by most students on their own. Here are some of the things done toward this objective. " Sharing the opinion of George Forsythe expressed in the epigraph, I have sought to stress major ideas underlying the design and analysis of algorithms. In choosing specific algorithms to illustrate these ideas, I limited the number of covered algorithms to those that most clearly demonstrate an underlying design technique or analysis method. Fortunately, most classic algorithms satisfy this criterion. In Chapter 2, which is devoted to efficiency analysis, the methods used for analyzing nonrecursive algorithms are separated from those typically used for analyzing recursive algorithms. The chapter also includes sections devoted to empirical analysis and algorithm visualization. The narrative is systematically interrupted by questions to the reader. Some of them are asked rhetorically, in anticipation of a concern or doubt, and are answered immediately. The goal of the others is to prevent the reader from drifting through the text without a satisfactory level of comprehension. Each chapter ends with a summary recapping the most important concepts and results discussed in the chapter. The book contains about 700 exercises. Some of them are drills; others make important points about the material covered in the body of the text or introduce algorithms not covered there at all. A few exercises take advantage of Internet resources. More difficult problems-there are not many of them-are marked with a special symbol in the Instructor's Manual. (Because designating problems as difficult may discourage some students from trying to tackle them, problems are not marked in the book itself.) Puzzles, games, and puzzlelike questions are marked in the exercises with a special icon. The book provides hints to all the exercises. Detailed solutions, except for programming projects, are provided in the Instructor's Manual, available

"

"

"

"

r

XX

Preface to qualified adopters through Addison-Wesley's Instructor Resource Center. (Please contact your local Addison-Wesley sales representative or visit www.aw.com/irc to access this material.) Slides in PowerPoint are available to all readers of this book at www.aw.comfcssupport.

Changes for the Second EditionThe most important change in the second edition is a new chapter (Chapter 10) dealing with the iterative-improvement technique. There are several reasons for this addition.10

10

'" '"

Some of the most important algorithms in computer science, both from a theoretical and practical standpoint, are based on this technique. Iterative improvement is the principal alternative to the greedy strategy already covered in the book. Since linear programming is discussed in Chapter 6, it is natural to cover its principal algorithm (the simplex method) as well. Iterative improvement is the only design technique of the new taxonomy that was not included in the first edition. Therefore, its inclusion could also be justified by the sheer need for completeness.

The four sections of the new chapter cover the following topics: the simplex method, maximum network flows, maximum cardinality matching in bipartite graphs, and the stable marriage problem. Each of these sections can be studied independently of the rest, as is the case for the other chapters of the book. There are two other additions in this edition worth mentioning. The first is a new subsection about the game of Nim, included in Section 5.6 on variabledecrease algorithms. The second is an expanded coverage of approximation algorithms for the traveling salesman problem to give a better account of the remarkable progress achieved in this area. I also added about 80 new problems to the exercises. (A list of the new problems is available on the book's Web site at www.aw-bc.com/info/levitin.) About half of them are added to the old chapters, and a majority are puzzles or puzzle-like questions. Judging by the feedback received, the inclusion of puzzles and games in tbe exercises has been welcome by the book's readers, including instructors and students using it as a textbook. As in the first edition, only the puzzles related to the discussed topics were included in the exercises.

PrerequisitesThe book assumes that a reader has gone through an introductory programming course and a standard course on discrete structures. With such a background, he or she should be able to handle the book's material without undue difficulty. Still, fundamental data structures, necessary summation formulas, and recurrence relations are reviewed in Section 1.4, Appendix A, and Appendix B, respectively.

Preface

xxi

Calculus is used in only three sections (Sections 2.2, 11.4, and 12.4) and to a very limited degree; if students lack calculus as an assured part of their background, the portions of these three sections that involve calculus can be omitted without hindering their understanding of the rest of the material.

Use in the CurriculumThe book can serve as a textbook for a basic course on design and analysis of algorithms organized around algorithm design techniques. It might contain too much material for a typical one-semester course. By and large, portions of Chapters 3 through 12 can be skipped without the danger of making later parts of the book incomprehensible to the reader. Any portion of the book can be assigned for self-study. In particular, Sections 2.6 and 2.7 on empirical analysis and algorithm visualization, respectively, can be assigned in conjunction with projects. Here is a possible plan for a one-semester course; it assumes a 40-class meeting format. Lecture1 2,3 4 5,6 7 8 9-10 11 12-14 15 16 17-19 20 21 22-24 25-27 28-30

TopicIntroduction Analysis framework; 0, 8,

Sections

n notations

Mathematical analysis of nonrecursive algorithms Mathematical analysis of recursive algorithms Brute-force algorithms Exhaustive search Divide-and-conquer: mergesort, quicksort, binary search Other divide-and-conquer examples Decrease-by-one: insertion sort, DFS and BFS, topological sorting Decrease-by-a-constant-factor algorithms Variable-size-decrease algorithms Instance simplification: presorting, Gaussian elimination, balanced search trees Representation change: heaps and heapsort or Horner's rule and binary exponentiation Problem reduction Space-time tradeoffs: string matching, hashing, B-trees Dynamic programming algorithms Greedy algorithms: Prim's, Kruskal's, Dijkstra's, Huffman's

1.1-1.3 2.1,2.2 2.3 2.4, 2.5 (+ App. B) 3.1, 3.2 (+ 3.3) 3.4 4.1-4.3 4.4 or 4.5 or 4.6 5.1-5.3 5.5 5.6 6.1-6.3 6.4 or 6.5 6.6 7.2-7.4 3 from 8.1-8.4 9.1-9.4

xxii

Preface

Lecture31-33 34 35 36 37 38 39 40

TopicIterative improvement algorithms Lower-bound arguments Decision trees P, NP, and NP-complete problems Numerical algorithms Backtracking Branch-and-bound Approximation algorithms for NP-hard problems

Sections3 from 10.1-10.4 11.1 11.2 11.3 11.4 ( + 12.4) 12.1 12.2 12.3

AcknowledgmentsI would like to express my gratitude to the many people who have shared with me their own and their students' opinions about the first edition and suggested improvements and corrections. I have especially benefitted from the comments by Eric Bach (University of Wisconsin), Stephen Bique (University of AlaskaFairbanks), Walter Hower (Albstadt-Sigmaringen University), Dan Joyce (Villanova University), Steven Lindell (Haverford College), and Mary-Angela Papalaskari (Villanova University). My thanks also go to Edward L. Bosworth (Columbus State University), Philip Dorin (Loyola Marymount University), William Ford (University of the Pacific), Jonathan Goldstine (Penn State University), Chin-Cheng Hung (Southern Polytechnic State University), and Brian Rosmaita (Hamilton College )-the reviewers of the chapter on iterative-improvement algorithms-for supporting the inclusion of this chapter in the new edition and for their ideas about presenting these important algorithms. In addition, I would like to convey my appreciation to the following reviewers: David Avis (McGill University), Gordon Beavers (University of Arkansas), Dinesh Mehta (Colorado School of Mines), Feridoon Moinian (Cameron University), Stephan Olariu (Old Dominion University), Stanley Selkow (WPI), Jerry Trahan (Louisiana State University), and Yang Wang (Missouri State University). I gratefully acknowledge the help I have received from my former student, Rakhi Gandhi, whose assistance has certainly decreased the number of typos and errors in the book. I appreciate the contributions of all the reviewers of the first edition: Simon Berkovich (George Washington University), Richard Borie (University of Alabama), Douglas M. Campbell (Brigham Young University), Bin Cong (California State University, Fullerton), Steve Homer (Boston University), Roland Htibscher (Auburn University), Sukhamay Kundu (Louisiana State University), Sheau-Dong Lang (University of Central Florida), John C. Lusth (University of Arkansas), John F. Meyer (University of Michigan), Steven R. Seidel (Michigan

Preface

xxiii

Technological University), Ali Shokoufandeh (Drexel University), and George H. Williams (Union College). My thanks go to all the people at Addison-Wesley and their associates who worked on my book. I am especially grateful to my editor Matt Goldstein, Michelle Brown, Katherine Harutunian, Marilyn Lloyd, Joyce Wells, and Paul Anagnostopoulos of Windfall Software. I was lucky to have Richard Camp as the copy editor; his numerous suggestions have definitely improved the book. Finally, I am indebted to two members of my family. Living with a spouse writing a book is probably more trying than doing the actual writing. My wife, Maria, lived through several years of this, helping me any way she could. And help she did: over 400 figures in the book and the Instructor's Manual were created by her. My daughter Miriam has been my English prose guru over many years. She not only read large portions of the book, but she was also instrumental in finding the chapter epigraphs. Anany Levitin anany.levitin@villanova.edu December 2005

Introduction to

The Design & Analysis of Algorithms fiND EDFFIC>N .

i

,I:!

:iJ

II .Iiil!

I'il

:i![i

1IntroductionTwo ideas lie gleaming on the jeweler's velvet. The first is the calculus, the second, the algorithm. The calculus and the rich body of mathematical analysis to which it gave rise made modern science possible; but it has been the algorithm that has rnade possible the modern world.-David Berlinski, The Advent of the Algorithm, 2000

hy do you need to study algorithms? If you are going to be a computer professional, there are both practical and theoretical reasons to study algorithms. From a practical standpoint, you have to know a standard set of important algorithms from different areas of computing; in addition, you should be able to design new algorithms and analyze their efficiency. From the theoretical standpoint, the study of algorithms, sometimes called algorithmics, has come to be recognized as the cornerstone of computer science. David Hare!, in his delightful book pointedly titled Algorithmics: the Spirit of Computing, put it as follows: Algorithmics is more than a branch of computer science. It is the core of computer science, and, in all fairness, can be said to be relevant to most of science, business, and technology. [Har92], p. 6. But even if you are not a student in a computing-related program, there are compelling reasons to study algorithms. To put it bluntly, computer programs would not exist without algorithms. And with computer applications becoming indispensable in almost all aspects of our professional and personal lives, studying algorithms becomes a necessity for more and more people. Another reason for studying algorithms is their usefulness in developing analytical skills. After all, algorithms can be seen as special kinds of solutions to problems-not answers but precisely defined procedures for getting answers. Consequently, specific algorithm design techniques can be interpreted as problemsolving strategies that can be useful regardless of whether a computer is involved. Of course, the precision inherently imposed by algorithmic thinking limits the kinds of problems that can be solved with an algorithm. You will not find, for example, an algorithm for living a happy life or becoming rich and famous. On1

W

.

w.

'

2

Introduction

i!I

I'I

the other hand, this required precision has an important educational advantage. Donald Knuth, one of the most prominent computer scientists in the history of algorithmics, put it as follows: A person well-trained in computer science knows how to deal with algorithms: how to construct them, manipulate them, understand them, analyze them. This knowledge is preparation for much more than writing good computer programs; it is a general-purpose mental tool that will be a definite aid to the understanding of other subjects, whether they be chemistry, linguistics, or music, etc. The reason for this may be understood in the following way: It has often been said that a person does not really understand something until after teaching it to someone else. Actually, a person does not really understand something until after teaching it to a computer, i.e., expressing it as an algorithm ... An attempt to formalize things as algorithms leads to a much deeper understanding than if we simply try to comprehend things in the traditional way. [Knu96], p. 9. We take up the notion of algorithm in Section 1.1. As examples, we use three algorithms for the same problem: computing the greatest common divisor. There are several reasons for this choice. First, it deals with a problem familiar to everybody from their middle-school days. Second, it makes the important point that the same problem can often be solved by several algorithms. Quite typically, these algorithms differ in their idea, level of sophistication, and efficiency. Third, one of these algorithms deserves to be introduced first, both because of its age~it appeared in Euclid's famous treatise more than two thousand years ago~and its enduring power and importance. Finally, the middle-school procedure for computing the greatest common divisor allows us to highlight a critical requirement every algorithm must satisfy. Section 1.2 deals with algorithmic problem solving. There we discuss several important issues related to the design and analysis of algorithms. The different aspects of algorithmic problem solving range from analysis of the problem and the means of expressing an algorithm to establishing its correctness and analyzing its efficiency. The section does not contain a magic recipe for designing an algorithm for an arbitrary problem. It is a well-established fact that such a recipe does not exist. Still, the material of Section 1.2 should be useful for organizing your work on designing and analyzing algorithms. Section 1.3 is devoted to a few problem types that have proven to be particularly important to the study of algorithms and their application. In fact, there are textbooks (e.g., [Sed88]) organized around such problem types. I hold the view~shared by many others~that an organization based on algorithm design techniques is superior. In any case, it is very important to be aware of the principal problem types. Not only are they the most commonly encountered problem types in real-life applications, they are used throughout the book to demonstrate particular algorithm design techniques.

1:I

IIif1

'I

I

II

'Ii,'

~i

'I ''! !i

i'

I

'10 !J1

t

1.1

What is an Algorithm?

3

Section 1.4 contains a review of fundamental data structures. It is meant to serve as a reference rather than a deliberate discussion of this topic. If you need a more detailed exposition, there is a wealth of good books on the subject, most of them tailored to a particular programming language.

1.1

What is an Algorithm?Although there is no universally agreed-on wording to describe this notion, there is general agreement about what the concept means:An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

This definition can be illustrated by a simple diagram (Figure 1.1 ). The reference to "instructions" in the definition implies that there is something or someone capable of understanding and following the instructions given. We call this a "computer," keeping in mind that before the electronic computerwas invented, the word "computer" meant a human being involved in performing numeric calculations. Nowadays, of course, "computers" are those ubiquitous

electronic devices that have become indispensable in almost everything we do. Note, however, that although the majority of algorithms are indeed intended for eventual computer implementation, the notion of algorithm does not depend onsuch an assumption.

As examples illustrating the notion of algorithm, we consider in this section three methods for solving the same problem: computing the greatest common divisor of two integers. These examples will help us to illustrate several important points: "' The nonambiguity requirement for each step of an algorithm cannot be com promised.

problem

algorithm

1

input

---.[==~c~om~p~ut~e~r~=}----+ output

FIGURE 1.1 Notion of algorithm

4

Introduction

" ,. "II!

The range of inputs for which an algorithm works has to be specified carefully. The same algorithm can be represented in several different ways. Several algorithms for solving the same problem may exist. Algorithms for the same problem can be based on very different ideas and can solve the problem with dramatically different speeds.

Recall that the greatest common divisor of two nonnegative, not -both-zero integers m and n, denoted gcd(m, n), is defined as the largest integer that divides both m and n evenly, i.e., with a remainder of zero. Euclid of Alexandria (third century B.C.) outlined an algorithm for solving this problem in one of the volumes of his Elements, most famous for its systematic exposition of geometry. In modern terms, Euclid's algorithm is based on applying repeatedly the equality gcd(m, n) = gcd(n, m mod n) (where m mod n is the remainder of the division of m by n) until m mod n is equal to 0; since gcd(m, 0) = m (why?), the last value of m is also the greatest common divisor of the initial m and n. For example, gcd(60, 24) can be computed as follows: gcd(60, 24) = gcd(24, 12) = gcd(12, 0) = 12.(If you are not impressed by this algorithm, try finding the greatest common divisor of larger numbers such as those in Problem 5 of Exercises 1.1.) Here is a more structured description of this algorithm:

Euclid's algorithm for computing gcd(m, n) Step 1 If n = 0, return the value of m as the answer and stop; otherwise, proceed to Step 2. Step 2 Divide m by n and assign the value of the remainder to r. Step 3 Assign the value of n tom and the value of r ton. Go to Step 1. Alternatively, we can express the same algorithm in a pseudocode:ALGORITHM

Euclid(m, n)

//Computes gcd(m, n) by Euclid's algorithm !!Input: Two nonnegative, not -both-zero integers m and n //Output: Greatest common divisor of m and n while n oJ 0 do r +-m modn m +-n n +---- r return m How do we know that Euclid's algorithm eventually comes to a stop? This follows from the observation that the second number of the pair gets smaller with

1.1

What is an Algorithm?

5

each iteration and it cannot become negative. Indeed, the new value of n on the next iteration ism mod n, which is always smaller than n. Hence, the value of the second number in the pair eventually becomes 0, and the algorithm stops. Just as with many other problems, there are several algorithms for computing the greatest common divisor. Let us look at the other two methods for this problem. The first is simply based on the definition of the greatest common divisor of m and n as the largest integer that divides both numbers evenly. Obviously, such a common divisor cannot be greater than the smaller of these numbers, which we will denote by t = min{m, n ). So we can start by checking whether t divides both m and n: if it does, t is the answer; if it does not, we simply decrease t by 1 and try again. (How do we know that the process will eventually stop?) For example, for numbers 60 and 24, the algorithm will try first 24, then 23, and so on until it reaches 12, where it stops. Consecutive integer checking algorithm for computing gcd(m, n) Step 1 Assign the value of min{m, n) tot. Step 2 Divide m by t. If the remainder of this division is 0, go to Step 3; otherwise, go to Step 4. Step 3 Divide n by t. If the remainder of this division is 0, return the value of t as the answer and stop; otherwise, proceed to Step 4. Step 4 Decrease the value oft by 1. Go to Step 2. Note that unlike Euclid's algorithm, this algorithm, in the form presented, does not work correctly when one of its input numbers is zero. This example illustrates why it is so important to specify the range of an algorithm's inputs explicitly and carefully. The third procedure for finding the greatest common divisor should be familiar to you from middle school. Middle-school procedure for computing gcd(m, n) Step 1 Find the prime factors of m. Step 2 Find the prime factors of n. Step 3 Identify all the common factors in the two prime expansions found in Step 1 and Step 2. (If p is a common factor occurring Pm and Pn times in m and n, respectively, it should be repeated min{pm, p,) times.) Step 4 Compute the product of all the common factors and return it as the greatest common divisor of the numbers given. Thus, for the numbers 60 and 24, we get 60=2235 24=2223 gcd(60, 24) = 2. 2. 3 = 12.

'7

6

Introduction

Nostalgia for the days when we learned this method should not prevent us from noting that the last procedure is much more complex and slower than Euclid's algorithm. (We will discuss methods for finding and comparing running times of algorithms in the next chapter.) In addition to inferior efficiency, the middle-school procedure does not qualify, in the form presented, as a legitimate algorithm. Why? Because the prime factorization steps are not defined unambiguously: they require a list of prime numbers, and I strongly suspect that your middle-school teacher did not explain how to obtain such a list. You undoubtedly agree that this is not a matter of unnecessary nitpicking. Unless this issue is resolved, we cannot, say, write a program implementing this procedure. (Incidentally, Step 3 is also not defined clearly enough. Its ambiguity is much easier to rectify than that of the factorization steps, however. How would you find common elements in two sorted lists?) So let us introduce a simple algorithm for generating consecutive primes not exceeding any given integer n. It was probably invented in ancient Greece and is known as the sieve of Eratosthenes (ca. 200 B.C.). The algorithm starts by initializing a list of prime candidates with consecutive integers from 2 ton. Then, on the first iteration of the algorithm, it eliminates from the list all multiples of 2, i.e., 4, 6, and so on. Then it moves to the next item on the list, which is 3, and eliminates its multiples. (In this straightforward version, there is an overhead because some numbers, such as 6, are eliminated more than once.) No pass for number 4 is needed: since 4 itself and all its multiples are also multiples of 2, they were already eliminated on a previous pass. (By similar reasoning, we need not consider multiples of any eliminated number.) The next remaining number on the list, which is used on the third pass, is 5. The algorithm continues in this fashion until no more numbers can be eliminated from the list. The remaining integers of the list are the primes needed. As an example, consider the application of the algorithm to finding the list of primes not exceeding n = 25:2 2 2 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 17 19 5 7 9 11 13 15 21 23 25 3 5 7 11 17 19 23 25 13 3 5 7 11 17 19 23 3 13

For this example, no more passes are needed because they would eliminate numbers already eliminated on previous iterations of the algorithm. The remaining numbers on the list are the consecutive primes less than or equal to 25. In general, what is the largest number p whose multiples can still remain on the list? Before we answer this question, let us first note that if p is a number whose multiples are being eliminated on the current pass, then the first multiple we should consider is p p because all its smaller multiples 2p, ... , (p- l)p have been eliminated on earlier passes through the list. This observation helps to avoid

1.1

What is an Algorithm?

7

eliminating the same number more than once. Obviously, p p should not be greater than n, and therefore p cannot exceed ,jn rounded down (denoted ,jn using the so-called floor function). We assume in the following pseudocode that there is a function available for computing ,jn alternatively, we could check the inequality p p S n as the loop continuation condition there.

l J

l J;

ALGORITHM

Sieve(n)

//Implements the sieve of Eratosthenes //Input: An integer n ": 2 //Output: Array L of all prime numbers less than or equal ton for p-oo

(,Jit)

lim (Iogzn--+oo

_1_

e) ~

= 2 log

2

e

limn--+oo

Jii =n

0.

2.j/i

3.

The fourth case, in which such a limit does not exist, rarely happens in the actual practice of analyzing algorithms. Still, this possibility makes the limit-based approach to comparing orders of growth less general than the one based on the definitions of 0, ~,and e.

58

Fundamentals of the Analysis of Algorithm Efficiency

Since the limit is equal to zero, log 2 n has a smaller order of growth than fo. (Since lim Iogb" = 0, we can use the so-called little-oh notation: log2 n E o(fo). Unlike n---+oo -vn the big-oh, the little-oh notation is rarely used in analysis of algorithms.) Ill

EXAMPLE 3 Compare the orders of growth of 11! and 2". (We discussed this issue informally in the previous section.) Taking advantage of Stirling's formula, we getI = lim lim !!.:_/1----:>-00

211

v'2Jrn (~ e )"211

= limn-HX)

ll--+00

v'21fn-"-"= 211ell

lim11--+00

)" = v'21fn ( !!_ 2e

oo.

,I

Thus, though 2" grows very fast, 11! grows still faster. We can write symbolically that n! E rl(2"); note, however, that while big-omega notation does not preclude the possibility that n! and 2" have the same order of growth, the limit computed here certainly does. Ill

Basic Efficiency ClassesEven though the efficiency analysis framework puts together all the functions whose orders of growth differ by a constant multiple, there are still infinitely many such classes. (For example, the exponential functions a" have different orders of growth for different values of base a.) Therefore, it may come as a surprise that the time efficiencies of a large number of algorithms fall into only a few classes. These classes are listed in Table 2.2 in increasing order of their orders of growth, along with their names and a few comments. You could raise a concern that classifying algorithms by their asymptotic efficiency would be of little practical use since the values of multiplicative constants are usually left unspecified. This leaves open a possibility of an algorithm in a worse efficiency class running faster than an algorithm in a better efficiency class for inputs of realistic sizes. For example, if the running time of one algorithm is n3 while the running time of the other is 106n2 , the cubic algorithm will outperform the quadratic algorithm unless n exceeds 10. A few such anomalies are indeed known. For example, there exist algorithms for matrix multiplication with a better asymptotic efficiency than the cubic efficiency of the definition-based algorithm (see Section 4.5). Because of their much larger multiplicative constants, however, the value of these more sophisticated algorithms is mostly theoretical. Fortunately, multiplicative constants usually do not differ that drastically. As a rule, you should expect an algorithm from a better asymptotic efficiency class to outperform an algorithm from a worse class even for moderately sized inputs. This observation is especially true for an algorithm with a better than exponential running time versus an exponential (or worse) algorithm.

2.2 Asymptotic Notations and Basic Efficiency Classes TABLE 2.2 Basic asymptotic efficiency classes Class1

59

Nameconstant

CommentsShort of best-case efficiencies, very few reasonable examples can be given since an algorithm's running time typically goes to infinity when its input size grows infinitely large. Typically, a result of cutting a problem's size by a constant factor on each iteration of the algorithm (see Section 5.5). Note that a logarithmic algorithm cannot take into account all its input (or even a fixed fraction of it): any algorithm that does so will have at least linear running time. Algorithms that scan a list of size n (e.g., sequential search) belong to this class. Many divide-and-conquer algorithms (see Chapter 4), including mergesort and quicksort in the average case, fall into this category. Typically, characterizes efficiency of algorithms with two embedded loops (see the next section). Elementary sorting algorithms and certain operations on n-by-n matrices are standard examples. JYpically, characterizes efficiency of algorithms with three embedded loops (see the next section). Several nontriVial algorithms from linear algebra fall into this class. Typical for algorithms that generate all subsets of an n-element set. Often, the term "exponential" is used in a broader sense to include this and larger orders of growth as well. Typical for algorithms that generate all permutations of ann-element set.

Iogn

logarithmic

n

linear

n log n

"n-log-n"

n2

quadratic

n3

cubic

2"

exponential

n!

factorial

-----Exercises 2 . 2 - - - - - - - - - - - - - - - 1. Use the most appropriate notation among 0. 8, and efficiency class of sequential search (see Section 2.1) a. in the worst case. b. in the best case.Q

to indicate the time

c. in the average case.

60

Fundamentals of the Analysis of Algorithm Efficiency

2. Use the informal definitions of 0, El, and Q to determine whether the following assertions are true or false.

a. n(n+1)/2E O(n 3)c. n(n+l)j2EEl(n 3 )

b. n(n+l)j2E O(n 2 ) d. n(n+1)j2EQ(n)

3. For each of the following functions, indicate the class El(g(n)) the function belongs to. (Use the simplest g(n) possible in your answers.) Prove yourassertions.

a. (n 2 + 1) 10 c. 2n Jg(n + 2) 2 + (n + 2) 2 Jg!! e. Llog2 nj

b. Vl0n 2 + 7n + 3d. zn+l+3n-l

4. a. Table 2.1 contains values of several functions that often arise in analysis of algorithms. These values certainly suggest that the functionsog n, 1n, n

Iog n,

n2 , 3 n ,2 " ,1 n.

are listed in increasing order of their order of growth. Do these values prove this fact with mathematical certainty? b. Prove that the functions are indeed listed in increasing order of their order of growth. 5. Order the following functions according to their order of growth (from the lowest to the highest):(n- 2)!, Slg(n + 100) 10 , 22", 0.00ln 4 + 3n 3 + 1, Jn 2 n,

..y;J, 3".

6. a. Prove that every polynomial of degree k, p(n) = aknk + ak_ 1nk-l + + ao, with ak > 0 belongs to El(nk). b. Prove that exponential functions a" have different orders of growth for different values of base a > 0. 7. Prove (by using the definitions of the notations involved) or disprove (by giving a specific counterexample) the following assertions. a. If t(n) E O(g(n)), then g(n) E Q(l(n)). b. El(ag(n)) = El(g(n)) where a> 0. c. El(g(n)) = O(g(n)) n Q(g(n)). d. For any two nonnegative functions t(n) and g(n) defined on the set of nonnegative integers, either t(n) E O(g(n)), or t(n) E Q(g(n)), or hath. 8. Prove the section's theorem for a. Q-notation. b. 8-notation. 9. We mentioned in this section that one can check whether all elements of an array are distinct hy a two-part algorithm based on the array's presorting.

2.3

Mathematical Analysis of Nonrecursive Algorithms

61

a. If the presorting is done by an algorithm with the time efficiency in El(n log n), what will be the time efficiency class of the entire algorithm? b. If the sorting algorithm used for presorting needs an extra array of size n, what will be the space efficiency class of the entire algorithm?10. Door in a wall

You are facing a wall that stretches infinitely in both directions. There is a door in the wall, but you know neither how far away nor in which direction. You can see the door only when you are right next to it. Design an algorithm that enables you to reach the door by walking at most 0 (n) steps where n is the (unknown to you) number of steps between your initial position and the door. [Par95], #652

2.3 Mathematical Analysis ofNomecursive AlgorithmsIn this section, we systematically apply the general framework outlined in Section 2.1 to analyzing the time efficiency of nonrecursive algorithms. Let us start with a very simple example that demonstrates all the principal steps typically taken in analyzing such algorithms.

EXAMPLE 1 Consider the problem of finding the value of the largest element in a list of n numbers. For simplicity, we assume that the list is implemented as an array. The following is a pseudocode of a standard algorithm for solving the problem. ALGORITHM MaxE!ernent(A[O .. n -1]) //Determines the value of the largest element in a given array //Input: An array A[O .. n - 1] of real numbers //Output: The value of the largest element in Arnaxva[ +-- A[O]

fori +-lton-1doif A[i] > rnaxval maxval +-- A[i] return maxval The obvious measure of an input's size here is the number of elements in the

array, i.e., n. The operations that are going to be executed most often are in the algorithm's for loop. There are two operations in the loop's body: the comparison A[i] >max val and the assignment max val +-- A[i]. Which of these two operationsshould we consider basic? Since the comparison is executed on each repetition

of the loop and the assignment is not, we should consider the comparison to be the algorithm's basic operation. (Note that the number of comparisons will be the

62

Fundamentals of the Analysis of Algorithm Efficiency

l!

same for all arrays of size n; therefore, in terms of this metric, there is no need to distinguish among the worst, average, and best cases here.) Let us denote C(n) the number of times this comparison is executed and try to find a formula expressing it as a function of size n. The algorithm makes one comparison on each execution of the loop, which is repeated for each value of the loop's variable i within the bounds 1 and n - 1 (inclusively). Therefore, we get the following sum for C(n):C(n) =

I

l

I

L 1.i=l

n-l

IIlll

This is an easy sum to compute because it is nothing else but 1 repeated n - 1 times. Thus,C(n) =

1

L

Il-l

1=n -lE G(n).

i=l

J

Here is a general plan to follow in analyzing nonrecursive algorithms.

General Plan for Analyzing Time Efficiency of Nonrecursive Algorithmsl, 2.

3.

4. 5.

Decide on a parameter (or parameters) indicating an input's size. Identify the algorithm's basic operation. (As a rule, it is located in its innermost loop.) Check whether the number of times the basic operation is executed depends only on the size of an input. If it also depends on some additional property, the worst-case, average-case, and, if necessary, best-case efficiencies have to be investigated separately. Set up a sum expressing the number of times the algorithm's basic operation is executed.4 Using standard formulas and rules of sum manipulation, either find a closedform formula for the count or, at the very least, establish its order of growth.

l

I

!

I

I

! I i

l

Before proceeding with further examples, you may want to review Appendix A, which contains a list of summation formulas and rules that are often useful in analysis of algorithms. In particular, we use especially frequently two basic rules of sum manipulationui=l

u

Lcai=c Laiui=l u u

(Rl)

(R2)i=l i=f

i=l

4.

Sometimes, an analysis of a nonrecursive algorithm requires setting up not a sum but a recurrence relation for the number of times its basic operation is executed. Using recurrence relations is much more typical for analyzing recursive algorithms (see Section 2.4).

2.3

Mathematical Analysis of Nonrecursive Algorithms

63

and two summation formulas

" Li=l

1= u - l

+ 1 where l 1 that determines whether the loop's body will be executed. Since the number of times the comparison will be executed is larger than the number of repetitions of the loop's body by exactly 1, the choice is not that important. A more significant feature of this example is the fact that the loop's variable takes on only a few values between its lower and upper limits; therefore we have to use an alternative way of computing the number of times the loop is executed. Since the value of n is about halved on each repetition of the loop, the answer should be about log2 n. The exact formula for the number of times the comparison 11 > 1 will be executed is actually Llog2 11j + 1-the number of bits in the binary representation of 11 according to formula (2.1). We could also get this answer by applying the analysis technique based on recurrence relations; we discuss this technique in the next section because it is more pertinent to the analysis II! of recursive algorithms.

2.3

Mathematical Analysis of Nonrecursive Algorithms

67

-----Exercises 2 . 3 - - - - - - - - - - - - - - 1. Compute the following sums. a. 1+3+5+7++999 b. 2 + 4 + 8 + 16 + ... + 1024 c.f.

h. z::;~11/i(i 2. Find the order of growth of the following sums. 2 a. +1) 2 b. lg i 2

1 2.::}~1 31+

z::;:i 1

d.

z::;:i i I:;',..;i

e.

z::;,..;z i (i + 1)+ 1)

g. z::;~l 2.::}~1 ij

c. d. z::;,..;J I:}~1u + Use the El(g(n)) notation with the simplest function g(n) possible. 3. The sample variance of n measurements x1,... ,

z::;'~ 1 Ci + 1J2'-1

z::;,..;tu

n

xn can be computed as

"" ( -)2 where i = -"-~i=~1~x_; "" ~"-~i=~1~x-'...,-,-x_ n -1 n

orI:;'~, xf-

CI:7d x;)2 jn

n-1Find and compare the number of divisions, multiplications, and additions/ subtractions (additions and subtractions are usually bunched together) that are required for computing the variance according to each of these formulas. 4. Consider the following algorithm.ALGORITHM Mystery(n)

//Input: A nonnegative integer n s of- 0 for i +-- 1 to n do S 1 disks from peg 1 to peg 3 (with peg 2 as auxiliary), we first move recursively n - 1 disks from peg 1 to peg 2 (with peg 3 as auxiliary), then move the largest disk directly from peg 1 to peg 3, and, finally, move recursively n - 1 disks from peg 2 to peg 3 (using peg 1 as auxiliary). Of course, if n = 1, we can simply move the single disk directly from the source peg to the destination peg. Let us apply the general plan to the Tower of Hanoi problem. The number of disks n is the obvious choice for the input's size indicator, and so is moving one disk as the algorithm's basic operation. Clearly, the number of moves M(n) depends on n only, and we get the following recurrence equation for it:M(n) = M(n- 1)

+ 1 + M(n- 1)

for n > 1.

With the obvious initial condition M(1) = 1, we have the following recurrence relation for the number of moves M(n):

2.4

Mathematical Analysis of Recursive Algorithms

73

3

2FIGURE 2.4 Recursive solution to the Tower of Hanoi puzzle

M(n) = 2M(n- 1)

+1

for

11

> 1,

(2.3)

M(1)

=

1.

We solve this recurrence by the same method of backward substitutions:M(n)

= 2M(n- 1) + 12

sub. M(n-1)3

= 2M(n- 2) + 1

= 2[2M (n - 2) + 1] + 1 = 2 M (n - 2) + 2 + 1 = 2 [2M (n - 3) + 1] + 2 + 1 = 2 M (n - 3) +2

sub. M (n - 2) = 2M (n - 3) + 1

22

+ 2 + 1.

The pattern of the first three sums on the left suggests that the next one will be 24 M(n- 4) + 23 + 22 + 2 + 1 and, generally, after i substitutions, we getM(n) =

z' M(n- i) + 2i-l + z'- 2 +

+ 2 + 1=

z' M(n- i) + zi- 1.

Since the initial condition is specified for 11 = 1, which is achieved fori = n - 1, we get the following formula for the solution to recurrence (2.3):

M(n)=2"- 1M(n-(n-1))+2"- 1 -1 = 2"- 1M(1) + 2"- 1 -1 = 2"- 1 + 2"- 1 -1 = 2" -1.Thus, we have an exponential algorithm, which will run for an unimaginably long time even for moderate values of n (see Problem 5 in Exercises 2.4). This is not due to the fact that this particular algorithm is poor; in fact, it is not difficult to prove that this is the most efficient algorithm possible for this problem. It is

I

74

Fundamentals of the Analysis of Algorithm Efficiency

n-2

/~ n-2

--------------- n n-1

~ n-1 /~ n-2 n-2

FIGURE 2.5 Tree of recursive calls made by the recursive algorithm for the Tower of Hanoi puzzle

the problem's intrinsic difficulty that makes it so computationally hard. Still, this example makes an important general point:One should be careful with recursive algorithms because their succinctness may mask their inefficiency.

When a recursive algorithm makes more than a single call to itself, it is useful for analysis purposes to construct a tree of its recursive calls. In this tree, nodes correspond to recursive calls, and we can label them with the value of the parameter (or, more generally, parameters) of the calls. For the Tower of Hanoi example, the tree is given in Figure 2.5. By counting the number of nodes in the tree, we can get the total number of talls made by the Tower of Hanoi algorithm:C(n) =

L zit~o

n-1

(where lis the level in the tree above)= 2"- 1.!Ill

The number agrees, as it should, with the move count obtained earlier.

EXAMPLE 3 As our next example, we investigate a recursive version of the algorithm discussed at the end of Section 2.3. ALGORITHM BinRec(11) //Input: A positive decimal integer 11 //Output: The number of binary digits inn's binary representation if 11 = 1 return 1else return BinRec(L11/2J)

+1

Let us set up a recurrence and an initial condition for the number of additions A (11) made by the algorithm. TI1e number of additions made in computing

2.4 Mathematical Analysis of Recursive Algorithms

75

BinRec(Ln/2J) is A(Ln/2J), plus one more addition is made by the algorithm to

increase the returned value by 1. This leads to the recurrenceA(n) = A(Ln/2J)

+1

for

11

> 1.

(2.4)

Since the recursive calls end when n is equal to 1 and there are no additions made then, the initial condition isA(1) = 0.

The presence of Ln/2J in the function's argument makes the method of backward substitutions stumble on values of n that are not powers of 2. Therefore, the standard approach to solving such a recurrence is to solve it only for 11 = 2' and then take advantage of the theorem called the smoothness rule (see Appendix B), which claims that under very broad assumptions the order of growth observed for n = 2' gives a correct answer about the order of growth for all values of n. (Alternatively, after obtaining a solution for powers of 2, we can sometimes fine-tune

this solution to get a formula valid for an arbitrary n.) So let us apply this recipe to our recurrence, which for n = 2' takes the form A(2'J = A(2'- 1J + 1 A(2) = 0. Now backward substitutions encounter no problems: A(2'J = A(2'- 1J + 1= [A(2'- J + 1] + 1 = A(2'- 2 ) = (A(2'- 3J + 1] + 2 =2

fork> 0,

substitute A(2'- 1J = A(2'- 2J + J

+2 A(2'- 3J + 3

substitute A(2'- 2J = A(2'- 3 )

+1

Thus, we end up withA (2')

= A (1) + k = k

or, after returning to the original variable n = 2' and hence k = log2 n,A(n) = log2 n E G(log n).

In fact, we can prove (Problem 6 in Exercises 2.4) that the exact solution for an arbitrary value ofn is given by just a slightly more refined formula A(n) = Llog2 11j.!!I

This section provides an introduction to analysis of recursive algorithms. These techniques will be used throughout the book and expanded further asnecessary. In the next section, we discuss the Fibonacci numbers; their analysis

76

Fundamentals of the Analysis of Algorithm Efficiency

involves more difficult recurrence relations to be solved by a method different from backward substitutions.

-----Exercises 2 . 4 - - - - - - - - - - - - - - - 1. Solve the following recurrence relations. a. x(n)=x(n-1)+5 forn>1, x(l)=O b. x(n) = 3x(n- 1) for n > 1, x(1) = 4 c. x(n) = x(n- 1) + n for n > 0, x(O) = 0 d. x(n) = x(n/2) + n for n > 1, x(1) = 1 (solve for n = 2k) e. x(n) = x(n/3)

+1

for n > 1,

x(1) = 1 (solve for n = 3k)

2. Set up and solve a recurrence relation for the number of calls made by F(n), the recursive algorithm for computing n!.

3. Consider the following recursive algorithm for computing the sum of the first n cubes: S(n) = 13 + 23 + ... + n 3ALGORITHMS(n)

//Input: A positive integer n //Output: The sum of the first n cubes if n = 1 return 1 else return S(n - 1) + n * n * n a. Set up and solve a recurrence relation for the number of times the algorithm's basic operation is executed. b. How does this algorithm compare with the straightforward nonrecursive algorithm for computing this sum?4. Consider the following recursive algorithm.

ALGORITHM

Q(n)

//Input: A positive integer n if n = 1 return 1 else return Q(n - 1) + 2 * n - 1 a. Set up a recurrence relation for this function's values and solve it to determine what this algorithm computes. b. Set up a recurrence relation for the number of multiplications made by this algorithm and solve it. c. Set up a recurrence relation for the number of additions/subtractions made by this algorithm and solve it.

;' '''i,~,,-_, __

2.4

Mathematical Analysis of Recursive Algorithms

77

5. a. Tower of Hanoi In the original version of the Tower of Hanoi puzzle, as it was published by Edouard Lucas, a French mathematician, in the 1890s, the world will end after 64 disks have been moved from a mystical Tower of Brahma. Estimate the number of years it will take if mouks could move one disk per minute. (Assume that monks do uot eat, sleep, or die.) b. How many moves are made by the ith largest disk (1: 34 34 89 17 34

'

H

?

etc.FIGURE3.2 First two passes of bubble sort on the list 89, 45, 68, 90, 29, 34, 17. Anew line is shown after a swap of two elements is done. The elements to the right of the vertical bar are in their final positions and are not considered in subsequent iterations of the algorithm.

the sum for selection sort:n-2 n-2-i 11-2

C(n)=L Li=O )=0

l=L[(n-2-i)-0+1](n- l)n E El(n2).

n-2

= L(n -1-i)

2

The number of key swaps, however, depends on the input. For the worst case of decreasing arrays, it is the same as the number of key comparisons:Swm."(n) = C(n) =

(n- l)n

2

E El(n ).

2

As is often the case with an application of the brute-force strategy, the first version of an algorithm obtained can often be improved with a modest amount of effort. Specifically, we can improve the crude version of bubble sort given above by exploiting the following observation: if a pass through the list makes no exchanges, the list has been sorted and we can stop the algorithm (Problem 9a in Exercises 3.1). Though the new version runs faster on some inputs, it is still in 8(n 2) in the worst and average cases. In fact, even among elementary sorting methods, bubble sort is an inferior choice, and, if it were not for its catchy name, you would probably have never heard of it. However, the general lesson you just learned is important and worth repeating:

..,

n102Brute Force

A first application of the brute-force approach often results in an algorithm that can be improved with a modest amount of effort

-----Exercises 3.1 - - - - - - - - - - - - - - - 1. a. Give an example of an algorithm that should not be considered an application of the brute-force approach. b. Give an example of a problem that cannot be solved by a brute-force algorithm. 2. a. What is the efficiency of the brute-force algorithm for computing a" as a function of n? As a function of the number of bits in the binary representation of n?

b. If you are to compute a 11 mod m where a > 1 and n is a large positive integer, how would you circumvent the problem of a very large magnitude of a"?3. For each of the algorithms in Problems 4, 5, and 6 of Exercises 2.3, tell whether or not the algorithm is based on the brute-force approach. 4. a. Design a brute-force algorithm for computing the value of a polynomialp ( x ) = a11 x n

+a

11

_1x n-l + + a1x

+ ao

at a given point x 0 and determine its worst-case efficiency class.

b. If the algorithm you designed is in EJ (n 2 ), design a linear algorithm for this problem. c. Is it possible to design an algorithm with a better than linear efficiency for this problem? 5. Sort the list E, X, A, M, P, L, E in alphabetical order by selection sort. 6. Is selection sort stable? (The definition of a stable sorting algorithm was given in Section 1.3.) 7. Is it possible to implement selection sort for linked lists with the same EJ(n 2)efficiency as the array version?

8. Sort the list E, X, A, M, P, L, E in alphabetical order by bubble sort. 9. a. Prove that if bubble sort makes no exchanges on its pass through a list, the list is sorted and the algorithm can be stopped. b. Write a pseudocode of the method that incorporates this improvement c. Prove that the worst-case efficiency of the improved version is quadratic. 10. Is bubble sort stable?

@ Alternating disks

You have a row of 2n disks of two colors, n dark and n light. They alternate: dark, light, dark, light, and so on. You want to get all the dark disks to the right-hand end, and all the light disks to the left-hand end. The

3.2

Sequential Search and Brute-Force String Matching

103

only moves you are allowed to make are those that interchange the positions of two neighboring disks.

Design an algorithm for solving this puzzle and determine the number of moves it takes. (Gar99), p. 75

3.2 Sequential Search and Brute-Force String MatchingWe saw in the previous section two applications of the brute-force approach to the sorting problem. Here we discuss two applications of this strategy to the problem of searching. The first deals with the canonical problem of searching for an item of a given value in a given list. The second is different in that it deals with the string-matching problem.

Sequential SearchWe have already encountered a brute-force algorithm for the general searching problem: it is called sequential search (see Section 2.1). To repeat, the algorithm simply compares successive elements of a given list with a given search key until either a match is encountered (successful search) or the list is exhausted without finding a match (unsuccessful search). A simple extra trick is often employed in implementing sequential search: if we append the search key to the end of the list, the search for the key will have to be successful, and therefore we can eliminate a check for the list's end on each iteration of the algorithm. Here is a pseudocode for this enhanced version, with its input implemented as an array.

AlGORITHM Sequentia!Search2(A[O.. n ], K) //Implements sequential search with a search key as a sentinel //Input: An array A of n elements and a search key K //Output: The index of the first element in A[O .. n -1) whose value is II equal to K or -1 if no such element is foundA[n] 2>

COSt=9+4+ 1 +4= 18 cost=9+4+8+9=30 cost = 9 + 3 + 8 + 4 = 24 cost = 9 + 3 + 8 + 6 = 26 cost=9+7+8+9=33 cost = 9 + 7 + 1 + 6 = 23

!etc.

i

FIGURE 3.9 First few iterations of solving a small instance of the assignment problem by exhaustive search

~ lIIl

I ,

! , 1

3.4 Exhaustive Search

119

-----Exercises1. a. Assuming that each tour can be generated in constant time, what will be the efficiency class of the exhaustive-search algorithm outlined in the text for the traveling salesman problem? b. If this algorithm is programmed on a computer that makes 1 billion additions per second, estimate the maximum number of cities for which the problem can be solved in i. one hour. ii. 24-hours. iii. one year. iv. one century. 2. Outline an exhaustive-search algorithm for the Hamiltonian circuit problem. 3. Outline an algorithm to determine whether a counected graph represented by its adjacency matrix has a Eulerian circuit. What is the efficiency class of your algorithm? 4. Complete the application of exhaustive search to the instance of the assignment problem started in the text. 5. Give an example of the assignment problem whose optimal solution does not include the smallest element of its cost matrix. 6. Consider the partition problem: given n positive integers, partition them into two disjoint subsets with the same sum of their elements. (Of course, the problem does not always have a solution.) Design an exhaustive-search algorithm for this problem. Try to minimize the number of subsets the algorithm needs to generate. 7. Consider the clique problem: given a graph G and a positive integer k, determine whether the graph contains a clique of size k, i.e., a complete subgraph of k vertices. Design an exhaustive-search algorithm for this problem. 8. Explain how exhaustive search can be applied to the sorting problem and determine the efficiency class of such an algorithm. 9. Magic squares A magic square of order n is an arrangement of the numbers from 1 to n2 in an n-by-n matrix, with each number occurring exactly once, so that each row, each column, and each main diagonal has the same sum. a. Prove that if a magic square of order n exists, the sum in question must be equal to n(n 2 + 1)/2. b. Design an exhaustive-search algorithm for generating all magic squares of order n.

120

Brute Force

c. Go to the Internet or yonr library and find a better algorithm for generatingmagic squares.

d. Implement the two algorithms-the exhaustive search and the one you have found-and run an experiment to determine the largest value of n for which each of the algorithms is able to find a magic square of order n in less than one minute of your computer's time.

10. Famous alphametic A puzzle in which the digits in a correct mathematical expression, such as a sum, are replaced by letters is called a cryptarithm; if, in addition, the puzzle's words make sense, it is said to be an alphametic. The most well-known alphametic was published by renowned British puzzlist H. E. Dudeney (1857-1930):SEND +MORE MONEY

Two conditions are assumed: first, the correspndence between letters and decimal digits is one-to-one, i.e., each letter represents one digit only and different letters represent different digits; second, the digit zero does not appear as the left -most digit in any of the numbers. To solve an alphametic means to find which digit each letter represents. Note that a solution's uniqueness cannot be assumed and has to be verified by the solver. a. Write a program for solving cryptarithms by exhaustive search. Assume that a given cryptarithm is a sum of two words. b. Solve Dudeney's puzzle the way it was expected to be solved when it was first published in 1924.

SUMMARY"' Brute force is a straightforward approach to solving a problem, usually directly based on the problem statement and definitions of the concepts involved.

"'

The principal strengths of the brute-force approach are wide applicability and simplicity; its principal weakness is the subpar efficiency of most brute-force algorithms. A first application of the brute-force approach often results in an algorithm that can be improved with a modest amount of effort. The following noted algorithms can be considered as examples of the bruteforce approach:

" "

f .,,.

-----------------------

'

< r-

Summary

121

definition-based algorithm for matrix multiplication;selection sort; sequential search;

straightforward string-matching algorithm. "' Exhaustive search is a brute-force approach to combinatorial problems. It suggests generating each and every combinatorial object of the problem, selecting those of them that satisfy all the constraints, and then finding a desired object. The traveling salesman problem, the knapsack problem, and the assignment problem are typical examples of problems that can be solved, at least theoretically, by exhaustive-search algorithms. Exhaustive search is impractical for all but very small instances of problems it can be applied to.

"

"

m bd

(Analogous results hold for the 0 and

notations, too.)

For example, the recurrence equation for the number of additions A (11) made by the divide-and-conquer sum-computation algorithm (see above) on inputs of size n = 2k isA(11) = 2A(nj2)

+ 1.

Thus, for this example, a= 2, b = 2, and d

= 0; hence, since a> bd,

A(11) E 8(n 10g"") = 8(11log2 2 ) = G(n).

Note that we were able to find the solution's efficiency class without going through the drudgery of solving the recurrence. But, of course, this approach can only establish a solution's order of growth to within an unknown multiplicative constant, while solving a recurrence equation with a specific initial condition yields an exact answer (at least for 11 's that are powers of b).

4.1

MergesortMergesort is a perfect example of a successful application of the divide-andconquer technique. It sorts a given array A[O .. n - 1] by dividing it into two halves A[D .. L11/2J -1] and A[lnf2j .. n -1], sorting each of them recursively, and then merging the two smaller sorted arrays into a single sorted one.

126

Divide-and-Conquer

ALGORITHM

Mergesort(A[O .. n - 1])

//Sorts array A[O .. n - 1] by recursive mergesort //Input: An array A[O .. n - 1] of orderable elements //Output: Array A[O .. n - 1] sorted in nondecreasing order ifn > 1 copy A[O .. Ln/2J -1]to B[O .. Ln/2J -1] copy A[Ln/2J .. n -1] to C[o .. rn/21 -1] Mergesort(B[O .. Ln/2J - 1]) Mergesort(C[O .. rn/21 -1]) Merge(B, C, A) The merging of two sorted arrays can be done as follows. Two pointers (array indices) are initialized to point to the first elements of the arrays being merged. The elements pointed to are compared, and the smaller of them is added to a new array being constructed; after that, the index of the smaller element is incrementedto point to its immediate successor in the array it was copied from. This operation

is repeated until one of the two given arrays is exhausted, and then the remaining elements of the other array are copied to the end of the new array.ALGORITHM

Merge(B[O .. p- 1], C[O .. q -1], A[O.. p + q -1])

//Merges two sorted arrays into one sorted array //Input: Arrays B[O .. p -1] and C[O .. q -1] both sorted //Output: Sorted array A[O .. p + q -1] of the elements of Band Ci +--0; j +--0; k +--0

whilei 1, Hence, according to the Master Theorem, Cwm-,Jn) E 8(11 log n) (why?). In fact, it is easy to find the exact solution to the worst-case recurrence for n = 2k:cwa>.>l(n) = 11

logz n- n

+ 1.

The number of key comparisons made by mergesort in the worst case comes very close to the theoretical minimum 1 that any general comparison-based sorting algorithm can have. The principal shortcoming of mergesort is the linear amount1. As we shall see in Section 11.2, this theoretical minimum is rtog 2 n !l ~ rn log2 n - 1.44n 1.

128

Divide-and-Conquer

of extra storage the algorithm requires. Though merging can be done in place, the resulting algorithm is quite complicated and, since it has a significantly larger multiplicative constant, the in-place mergesort is of theoretical interest only.

-----Exercises 4.1 - - - - - - - - - - - - - - - 1. \;JWrite a pseudocode for a divide-and-conquer algorithm for finding a position of the l!'.!~~t..~ement in an array of n numbers.

b. What will be your algorithm's output for arrays with several elements of the largest value? c. Set up and solve a recurrence relation for the number of key comparisons made by your algorithm. d. How does this algorithm compare with the brute-force algorithm for this problem? 2. a. Write a pseudocode for a divide-and-conquer algorithm for finding values of both the largest and smallest elements in an array of n numbers. b. Set up and solve (for n = 2k) a recurrence relation for the number of key comparisons made by your algorithm. c. How does this algorithm compare with the brute-force algorithm for this problem? 3. a. Write a pseudocode for a divide-and-conquer algorithm for the exponentiation problem of computing a" where a > 0 and n is a positive integer. b. Set up and solve a recurrence relation for the number of multiplications made by this algorithm. c. How does this algorithm compare with the brute-force algorithm for this problem? 4. We mentioned in Chapter 2, that logarithm bases are irrelevant in most contexts arising in the analysis of an algorithm's efficiency class. Is this true .. for both assertions of the Master Theorem that include logarithms?

,

5. Find the order of growth for solutions of the following recurrences.a. T(n) = 4T(nf2)'I

b. T(n) = 4T(n/2)

c. T(n) =

+ n, T(1) = 1 + n 2 , T(1) = 1 4T(nf2) + n 3 , T(l) = 1

6. Apply mergesort to sort the list E, X, A, M, P, L, E in alphabetical order. 7. Is mergesort a stable sorting algorithm?

)S._Ja.':

Solve the recurrence relation for the number of key comparisons made by mergesort in the worst case. (You may assume that n = 2k.)

'\

\

\

I.

\

\ j\

\\

~

r

II

4.2 Quicksort

129

b. Set up a recurrence relation for the number of key comparisons made by mergesort on best-case inputs and solve it for n = 2k. c. Set up a recurrence relation for the number of key moves made by the version of mergesort given in Section 4.1. Does taking the number of key moves into account change the algorithm's efficiency class? 9. Let A[O .. n - 1] be an array of n distinct real numbers. A pair (A[i], A[j]) is said to be an inversion if these numbers are out of order, i.e., i < j but A[i] > A[j]. Design an 0 (n log n) algorithm for counting the number of inversions.10. One can implement mergesort without a recursion by starting with merging

adjacent elements of a given array, then merging sorted pairs, and so on. Implement this bottom-up version of mergesort in the language of your choice.11. Tromino puzzle

A tromino is an L-shaped tile formed by 1-by-1 adjacent squares. The problem is to cover any 2"-by-2" chessboard with one missing square (anywhere on the board) with trominos. Trominos should cover all the squares except the missing one with no overlaps.

I

itDesign a divide-and-conquer algorithm for this problem.

4.2 Quicksortj

I

Quicksort is another important sorting algorithm that is based on the divide-andconquer approach. Unlike mergesort, which divides its input's elements according to their position in the array, quicksort divides them according to their value. Specifically, it rearranges elements of a given array A(O .. n - 1] to achieve its partition, a situation where all the elements before some position s are smaller than or equal to A[s] and all the elements after positions are greater than or equaltoA[s]: A[O] ... A[s -1] A[s] A[sall are~A(s]

+ 1] ... A[n -1]all arc ::::A[s]

r

130

Divide-and-Conquer

Obviously, after a partition has been achieved, A[s] will be in its final position in the sorted array, and we can continue sorting the two subarrays of the elements preceding and following A[s] independently (e.g., by the same method).ALGORITHM

Quicksort(A[Lr]) //Sorts a subarray by quicksort //Input: A subarray A[Lr] of A[O,n -1], defmed by its left and right indices II l and r //Output: Subarray A[l .. r] sorted in nondecreasing order if l < r s +-Partition(A[l .. r]) lis is a split position Quicksort(A[l .. s- 1]) Quicksort(A[s + l..r])

A partition of A[O .. n- 1] and, more generally, ofits subarray A[l .. r] (0::: l < r ::: n - 1) can be achieved by the following algorithm. First, we select an element with respect to whose value we are going to divide the subarray. Because of its guiding role, we call this element the pivot. There are several different strategies for selecting a pivot; we will return to this issue when we analyze the algorithm's efficiency. For now, we use the simplest strategy of selecting the subarray's first element: p = A[l]. There are also several alternative procedures for rearranging elements to achieve a partition. Here we use an efficient method based on two scans of the subarray: one is left-to-right and the other right-to-left, each comparing the subarray's elements with the pivot. The left-to-right scan, denoted below by index i, starts with the second element. Since we want elements smaller than the pivot to be in the first part of the subarray, this scan skips over elements that are smaller than the pivot and stops on encountering the first element greater than or equal to the pivot. The right-to-left scan, denoted below by index j, starts with the last element of the subarray. Since we want elements larger than the pivot to be in the second part of the sub array, this scan skips over elements that are larger than the pivot and stops on encountering the first element smaller than or equal to the pivot. After both scans stop, three situations may arise, depending on whether or notthe scanning indices have crossed. If scanning indices i and j have not crossed, i.e.,

i < j, we simply exchange A[i] and A[j] and resume the scans by incrementing i and decrementing j, respectively:

i~

all are ,; p

I~pIf f

all are 2

p

4.2 Quicksort

131

If the scanning indices have crossed over, i.e., i > j, we will have partitioned

the array after exchanging the pivot with A[J]:

Finally, if the scanning indices stop while pointing to the same element, i.e.,i = j, the value they are pointing to must be equal top (why?). Thus, we have the array partitioned, with the split positions = i = j:----i=j-all are s p

Ip I

1- pI

all are

~p

We can combine the last case with the case of crossed-over indices (i > j) by exchanging the pivot with A[J] whenever i 2" j. Here is a pseudocode implementing this partitioning procedure.

ALGORITHM Partition(A[l .. r]) //Partitions a subarray by using its first element as a pivot //Input: A subarray A[l..r] of A[O .. n - 1], defined by its left and right II indices land r (! < r) //Output: A partition of A[l..r ], with the split position returned as II this ftmction's valuep 1, Cw"'""(1) = 1.

(4.2)

4.3

Binary Search

137

(Stop and convince yourself that 11j2 must be, indeed, rounded down and that the initial condition must be written as specified.) As we discussed in Section 2.4, the standard way of solving recurrences such as recurrence (4.2) is to assume that 11 = 2k and solve the resulting recurrence by backward substitutions or another method. We leave this as a straightforward exercise to obtain the solution(4.3)

Actually, one can prove that the solution given by formula (4.3) for 11 = 2k can be tweaked to get a solution valid for an arbitrary positive integer 11:(4.4)

Let us verify by substitution that cwm_,(11) = Llogz nJ + 1 indeed satisfies equation (4.2) for any positive even number 11. (You are asked to do this for odd 11's in Exercises 4.3). If n is positive and even, n = 2i where i > 0. The left-hand side of equation ( 4.2) for 11 = 2i iscwm-,(11)= (1 + Llog2 ij)

= L1og2 11J + 1 = Llog2 2iJ + 1 = Llog2 2 + log2 iJ + 1 + 1 = Llog2 iJ + 2.11

The right-hand side of equation (4.2) for

=

2i is

cwm-,1 Cln/2J) + 1 = cwm,(L2if2J) + 1 = cwm-,1(i) + 1 = (Llog2 ij + 1) + 1 = Llog2 ij + 2.Since both expressions are the same, we proved the assertion. Formula ( 4.4) deserves attention. First, it implies that the worst-case efficiency of binary search is in E>(log n). (Incidentally, we could get this fact by applying the Master Theorem, but this approach would not give us the value of the multiplicative constant.) Second, it is the answer we should have fully expected: since the algorithm simply reduces the size of the remaining array by about half on each iteration, the number of such iterations needed to reduce the initial size 11 to the final size 1 has to be about log2 11. Third, to reiterate the point made in Section 2.1, the logarithmic function grows so slowly that its values remain small even for very large values of 11. In particular, according to formula (4.4), it will take no more than Llog2 103J + 1 = 10 three-way comparisons to find an element of a given value (or establish that there is no such element) in any sorted array of 1000 elements, and it will take no more than Llog2 106 j + 1 = 20 comparisons to do this for any sorted array of size one million! What can we say about the average-case efficiency of binary search? A sophisticated analysis shows that the average number of key comparisons made by binary search is only slightly smaller than that in the worst case:

138

Divide-and-Conquer

(More accurate formulas for the average number of comparisons in a successful and an unsuccessful search are C~~~ (n) "" log2 n - 1 and C;:~ (n) ""log2 (n + 1), respectively.) Though binary search is an optimal searching algorithm if we restrict our operations only to comparisons between keys (see Section 10.2), there are searching algorithms (see interpolation search in Section 5.6 and hashing in Section 7.3) with a better average-case efficiency, and one of them (hashing) does not even require the array to be sorted! These algorithms do require some special calculations in addition to key comparisons, however. Finally, the idea behind binary search has several applications beyond searching (see, e.g., [BenOO]). In addition, it can beapplied to solving nonlinear equations in one unknown; we discuss this continuous

analogue of binary search, called the method of bisection, in Section 12.4. Before we leave this section, one other remark about binary search needs to be made. Binary search is sometimes presented as the quintessential example of a divide-and-conquer algorithm. This interpretation is flawed because, in fact, binary search is a very atypical case of divide-and-conquer. Indeed, according to the definition given at the beginning of this chapter, the divide-and-conquer technique divides a problem into several subproblems, each of which needs to be solved. That is not the case for binary search, where only one of the two subproblems needs to be solved. Therefore, if binary search is to be considered as a divide-and-conquer algorithm, it should be looked on as a degenerate case of this technique. As a matter of fact, binary search fits much better into the class of decrease-by-half algorithms, which we discuss in Section 5.5. Why then is this discussion of binary search in this chapter? Partly because of tradition and partly because a bad example can sometimes make a point that a good example cannot.

-----Exercises 4 . 3 - - - - - - - - - - - - - - 1. a. What is the largest number of key comparisons made by binary search in searching for a key in the following array?

b. List all the keys of this array that will require the largest number of key comparisons when searched for by binary search. c. Find the average number of key comparisons made by binary search in a successful search in this array. (Assume that each key is searched for with the same probability.) d. Find the average number of key comparisons made by binary search in an unsuccessful search in this array. (Assume that searches for keys in each of the 14 intervals formed by the array's elements are equally likely.)

~

I

4.4

Binary Tre