Book 3: Internal Design and Programming

Chapter 2: Algorithms

Introduction

• This chapter describes the basics of algorithms used for module logic design and the algorithms used to solve typical problems.

• Technology Engineers No.3 Internal Design and Programming Chapter 2

The basis of programming is algorithm design. In designing the logical structure of a program, it is important to use the most appropriate algorithm. In choosing a program from a library, what algorithm to use is one of the most important criteria for choosing the most appropriate program.

Overview

Objectives

• Understand the basics of algorithms– Algorithm definition, design, relationships with

data structures, representation methods, etc.

• Understand the characteristics and meaning of representative search, sort, character string processing, and file processing algorithms

What is an Algorithm?

• Definition of an algorithm– Defined in the Japanese Industrial Standard as:

• A set of a limited number of clearly defined rules that are applied a limited number of times to solve problems

– To be an algorithm, a set of rules must be unambiguous (to solve a problem effectively and efficiently) and have a clear stopping point

What is an Algorithm?

• Algorithm and Programming– Programming describes data and algorithms in

a programming language to execute its given tasks

– Algorithm vs Program• A program consists of algorithms and data

• An algorithm consists of logic and control

What is an Algorithm?– 4 different types of programming:

• Procedural programming– Most commonly used type of programming

– E.g. FORTRAN, COBOL, Pascal, C, etc.

– Characteristics:

» Divided into modules (coding is done for each module)

» Structured theorems are introduced

– Limitations (as it is procedure oriented)

» Beside declaring procedures, variables must be declared

» Instructions are executed one by one in sequential manner (parallel processing cannot be done)

» Comparisons and calculations must be made to solve problems

What is an Algorithm?

• Functional programming– A type of function-oriented programming

– E.g. used in field of Artificial Intelligence (AI), basic calculation theories, other research tasks, etc.

– Characteristics:

» Expressions are built by nesting and they are replaced with results of calculations to execute a program

» Recursive calls are described more easily

» Level of affinity with parallel processing is high

» Calculation process or procedure does not need to be considererd

What is an Algorithm?

• Logic programming– Predicate based on facts and inferences

– E.g. Prolog

– Characteristics:

» Facts are described based on the prejudice logic, making the process of programming easier

» Inferences and logical calculations can be made easily

• Object-oriented programming– System is considered a group of objects

– E.g. Smalltalk, C++, Java, etc.

Algorithm and the Data Structure• 1. Data Structure

– A procedure that a program follows to store data and perform given tasks

• Basic data structure– Most basic data structure unit– Data is stored in main memory unit– Data type must be declared

• Problem-oriented data structure– Built by combining basic data structures containing one or more of

the following:» List» Stack» Queue» Tree Structure

Algorithm and the Data Structure• 2. Relationships between algorithm and the data

structure– Array processing

• Linear search algorithm and the data structure• Most frequently used data search method• Data is searched while subscripts in an array are being incremented• Maximum number of times the search is repeated is the size of an

array

– File processing• Process of reading, editing and printing a file is repeated until there is

no data in a file

Algorithm and the Data Structure

– List processing• Data arranged in terms of logic

• Data does not need to be shifted during insertion or deletion

• Data can be inserted or deleted by simply manipulating a pointer

Various Algorithms

• Describes the representative algorithms that have until now been developed in relation to each type of problem

Search Algorithm• 1. Linear (or sequential) search method

– Exhaustive search method• A method of combining a table from the top to the end

• Data retrieved is collated with each data in a table

• Search is successful only if data matches data to retrieve

• Search ends when first successful match occurs

• Inappropriate for numerous data retrieval

• Comparisons are made (N x 2) times

15 3 2 9 6 1 7 8

1 2 3 4 5 6 7 8

TBL

6(Match)

Searching from the top to the end one by one

Search Algorithm– Sentinel search method

• Same data (sentinel) as the data to retrieve is placed at the end of a table

• Used when authenticity of data and data matches data to retrieve

• Comparisons are made (N + 1) times

• Loop does not need to carry out the “end-of-array” test, making search more efficient

H K A I S D E A

1 2 3 4 5 6 7 8

TBL

N pieces

The same data as data to retrieve is stored

N + 1 (sentinel) pieces

Search Algorithm

• 2. Binary search method– A method that narrows down target data while successively

dividing a search area into two parts– Requires elements to be arranged in ascending or

descending order– Data smaller than reference data does not need to be

searched if data being searched is larger than reference data– Average number of times = [log2N]– Maximum number of times = -[log2N] + 1– ** Decimal numbers of the value in [] are truncated

Binary Search Method• Step1: A total of the value of a subscript representing the top of a table and

that of a subscript representing the end of a table is a divided by 2.• Step 2: The elements having the value obtained in step 1 as a subscript are

compared with a target element.• Step 3: If there is an element that matches as target element the search is

successful.• Step 4: If the value of a target element is smaller than that of an element in

a table,1 is subtracted from the current subscript and the the end of table.If the value of a target element is larger than that of an element in a table,1 is added to the current subscript and the value is used as a subscript for representing the top of a table.

• Step 5: Step 1 through step 4 is repeated. If an element matching a target element cannot be found at the point where the value of a subscript representing the top of a table becomes larger than that of a subscript representing the end of a table, the search is unsuccessful. This completes the search.

Binary Search Method

1002 1005 1010 1024 1028 1052 1211 1322 1866 2132

TBL(1) TBL(2) TBL(3) TBL(4) TBL(5) TBL(6) TBL(7) TBL(8) TBL(9) TBL(10)

1052 1211 1322 1866 2132

1866 2132

1866

1866

1866

TBL

The first comparison

The second comparison

The third comparison

x

x

x

X>TBL(i)

X>TBL(i)

X=TBL(i)

Sort Algorithm• Reorganizing data in a specific order• Internal sorting

– Sorting data in a main memory unit

• External sorting– Sorting data stored on a magnetic disk and other auxiliary

storage unit

• Selection of sorting method– Speed of sort– How data is sorted– Computational complexity

Sort Algorithm

Sorting

Internal sorting

External sorting

Shaker sort method

Basic selection method

Shell sort method

Basic exchange method

Merge sort method

Quick sort method

Basic insertion method

Exchange Sort

• 1. Basic Exchange method (bubble sort)– One of the simplest sort methods– Used to compare a pair of data sequentially from the

head of an array– Efficiency is low as data items are unconditionally

compared even if they have been sorted correctly– Process is time-consuming when data volume is large– Maximum computational complexity: 0 (n2)– Average computational complexity: 0 (n2)

Exchange Sort– Steps of Basic Exchange (Bubble) method

• 1st & 2nd elements in a table are compared

• If 1st element is larger than 2nd, 1st is exchanged with 2nd

• If 2nd element is larger than 1st, no exchange occurs

• 2nd and 3rd elements are compared and steps 2 & 3 are repeated

• This routine operation is repeated to the last element in the table. Maximum value is stored in the last element in that table as it reaches the last element

• Steps 1 through 4 and 5 are executed until the operation reaches the last but one

• Steps 1 through 6 are repeated until only the 1st & 2nd elements in a table remain

Selection Sort

• 2. Basic selection method– Like bubble sort, it is one of the simplest sort method

– Smallest/largest value is first selected from all data items and it is exchanged with the data item at the head of an array, then same routine repeats its execution on all remaining data items

– Efficiency is low as data items are unconditionally compared even if they have been sorted correctly

– Process is time-consuming when data volume is large

– Maximum computational complexity: 0 (n2)

– Average computational complexity: 0 (n2)

Selection Sort– Steps of basic selection method

• Data item with the smallest value is detected in data items stored in a table

• Data item with the smallest detected is exchanged with the first data item in a table (space into which the data item with the smallest value can be temporarily in a table)

• Data item with the smallest value is detected in the 2nd item in a table• Data item with the smallest value is detected is exchanged with the 2nd• The same operation is repeatedly performed until the last data item but

one is reached. When the last data item but one is reached, data sorting is completed

Selection Sort

Insertion Sort

• 3. Basic insertion method– Like basic exchange method, this is one of the simplest

sort method

– While data items are being sorted, an unsorted data item is inserted into a proper position in the sequence of sorted data items

– Comparison and insertion speeds are fast as preceding data items are sorted

– Process is time-consuming when data volume is large

Insertion Sort

– Steps of the basic insertion method– 1st & 2nd elements in a table are compared– If 1st element < 2nd element, nothing is done– If 1st element > 2nd element, 1st element is exchanged with

2nd, i.e. 1st & 2nd elements are in correct order– 2nd & 3rd elements are compared– If 2nd element < 3rd element, nothing is done– If 2nd element > 3rd element, 2nd element is exchanged with

3rd element. Then this element is compared with the preceding element according to steps 2 and 3. These steps are repeated until it is placed in the right position

– Steps 4, 5 and 6 are repeated until the last element in the table is inserted into the correct position

Insertion Sort

Shaker Sort• Shaker sort method

– Algorithm is the same as that of basic exchange method (bubble sort)

– Data items are first compared from left to right, such that maximum (minimum) value is set at rightmost position

– Data items are then compared from right to left, such that minimum (maximum) value is set at leftmost position

– Operation is repeated to perform data sorting– Process is time-consuming when data volume is large

Shell Sort

• Shell sort method– Extended version of basic insertion method

– Execution speed is faster than basic insertion method

– Two items located away from each other a certain interval are picked out of data sequence and sorting is executed while picked data items are compared with each other

– Interval is called gap, such that gap = number of data (n) ÷ 2, which gradually made smaller and finally set to 1

– Pieces of data that are away from each other and located in different positions are quickly exchanged

Shell Sort

– Data items that are sorted in wrong positions can be resorted to correct positions in earliest stages of the operation

– If part of the data is already sorted, sorting can be completed very quickly

Shell Sort

Quick Sort• Quick sort method

– Fastest sort method using the recursive call method

– A method of dividing a large problem into small problems and solving each small problem individually

– Basic algorithm of quick sort method:– Choose a pivot (a key to partition the set of numbers into equal

halves)

– Divide all numbers into two partitions - All numbers in first partition are less than the pivot- All numbers in second partition are greater than or equal to the pivot

– Recursively sort numbers in the two partitions.

– The sorted list of numbers is obtained by putting the pivot at the beginning of the second partition and then appending the numbers in the second partition after the first partition

Quick Sort

Quick Sort– Example : Sort the list of numbers [15, 17, 8, 13, 9, 24, 18,

2]. • Let us choose 15 as the pivot(First element of the set) • The first partition is the list [8, 13, 9, 2] - all smaller than 15.

The second partition is the list [17, 24, 18] - all larger than 15.• The sorted first partition is [2,8,9,13]

The sorted second partition is [17, 18, 24]• The sorted list for the input numbers is obtained as follows :

– put the pivot 15 at the beginning of the sorted second partition - giving the list [15, 17, 18, 24]

– append this list at the end of the first partition giving [2,8,9,13, 15, 17, 18, 24]

• http://cs.oregonstate.edu/~minoura/cs162/javaProgs/sort/QuickSort.html

Sort Algorithm

• Merge sort method– The recursive call and divide-and-conquer methods are used, as in the

case of the quick sort method

– All data items are divided into sections and sorting is executed in each divided section. This is repeated until there is only one remaining element in a data sequence

– After a data sequence is divided, divided sections are sequentially merged

– The process overall is thus: • Split the original list into two halves

• sort each half (using merge sort)

• merge the two sorted halves together into a single sorted list W

Merge Sort

– Example

34 56 78 12 45 3 99 23

34 56 78 12 45 3 99 23

34 56 78 12 45 3 99 23

34 56 78 12 45 3 99 23

34 56 12 78 3 45 23 99

12 34 56 78 3 23 45 99

3 12 23 34 45 56 78 99

sorting 100 elements: bubble sort : 13.639008 sec/ 30.08% insertion sort: 13.368042 sec/ 29.48% (fast for sorted lists) shell sort : 13.478973 sec/ 29.73% (poor implementation) merge sort : 2.232971 sec/ 4.92% quick sort : 2.433960 sec/ 5.37%

Source: http://www.faqs.org/faqs/CAD/autolisp-faq/part1/section-9.html

Sort Algorithm

Recursive Algorithm

• A recursive algorithm is one which calls itself to solve “smaller” versions of an input problem

• Recursive computer programs require more memory and computation compared with iterative algorithms, but they are simpler and for many cases a natural way of thinking about the problem.

• Example: Algorithm for finding the k-th even

Algorithm -> Even(positive integer k) Input: k , a positive integer Output: k-th even natural number (the first even being 0)

Algorithm: if k = 1, then return 0; else return Even(k-1) + 2 .

Recursive Algorithm

• Now, the same problem can be solved by an iterative algorithm.

Algorithm ->   Even(positive integer k) Input: k, a positive integer Output: k-th even natural number (the first even being 0)

Algorithm: int   i, even; i := 1;even := 0;while( i < k ) {           even := even + 2;           i := i + 1; } return even .

Recursive Algorithm

• Example: Algorithm for testing whether or not a number x is a natural number

Algorithm -> Natural(a number x) Input: A number x Output: "Yes" if x is a natural number, else "No"

Algorithm: if x < 0,   then return "No" else     if x = 0,   then return "Yes"     else return Natural( x - 1 )

• http://www.cs.queensu.ca/home/cisc121/2002f/lecturenotes/malamb/CISC121A2002fset11-4up.pdf

Recursive Algorithm

Character String Processing

• Character string search– Simple string search

• Text is compared character by character sequentially from the head

– Boyer-Moore method (BM method)• The basis of the fastest known ways to find one string

of characters in another

• Data is collated while characters in the text are skipped

Simple collation• Text is compared character by character sequentially from the

head. • When a character string on which a search should be performed

is detected, the position where it is located is set as a return value.

• Principle: – comparison is repeatedly executed until the first character of a character

string to search matches a character in character strings in text. – If there is a match, each remaining character of the matched character

string is compared with each of a character string to search.

BM Method• Data is collated while characters in the text are

skipped.• Case 1: If there are no character string to search

– a character at the tail, and all other characters in the first text portion do not match any character of a character string to be searched. Therefore, a search point is moved by the length of a character string to be

searched to allow the next search to start from that point.

BM Method• Case 2: If there is a match with a character at the tail of a

character string to be searched– Because a character at the tail of a character string matches a character in the

text, characters that precede the matched character must be compared. If all characters match, the value of a subscript of the first character of that matched text pattern is returned.

BM Method• Case 3: If there is a match with one character

somewhere in a character string but unmatched with a character at the tail of a character string– Because a character at the tail of a character string matches a

character in the text, characters that precede the matched character must be compared. If all characters match, the value of a subscript of the first character of that matched text pattern is returned.

two

BM Method

• Distance of movement

Character String Processing• 2. Character string compression

– Making a character string short by replacing consecutive characters or blanks with the number of characters

I M A T A K

# 4 I M # 3 A T A # 5 K

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 K

Special character

TBL A

TBL B

The number of blank characters

If the number of blank characters is two or one, no compression takes place