universiti putra malaysia content-based image retrieval...

25
UNIVERSITI PUTRA MALAYSIA CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB- BLOCK TECHNIQUE MAS RINA BINTI MUSTAFFA FSKTM 2006 6

Upload: duonghuong

Post on 11-Apr-2019

237 views

Category:

Documents


0 download

TRANSCRIPT

UNIVERSITI PUTRA MALAYSIA

CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-

BLOCK TECHNIQUE

MAS RINA BINTI MUSTAFFA

FSKTM 2006 6

CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-BLOCK

TECHNIQUE

MAS RINA BINTI MUSTAFFA

MASTER OF SCIENCE UNIVERSITI PUTRA MALAYSIA

2006

CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-BLOCK TECHNIQUE

By

MAS RINA BINTI MUSTAFFA

Thesis Submitted to the School of Graduate Studies, Universiti Putra Malaysia, in Fulfilment of the Requirement for the Degree of Master of Science

August 2006

ii

TO MY HUSBAND, PARENTS, FAMILY, AND FRIENDS…

iii

Abstract of thesis presented to the Senate of Universiti Putra Malaysia in fulfilment of the requirement for the degree of Master of Science

CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-BLOCK TECHNIQUE

By

MAS RINA BINTI MUSTAFFA

August 2006

Chairman: Associate Professor Hajah Fatimah Ahmad, PhD

Faculty : Computer Science and Information Technology

Traditional Content-Based Image Retrieval (CBIR) systems mainly relied on the

extraction of features globally. The drawback of this approach is that it cannot

sufficiently capture the important features of individual regions in an image which

users might be interested in. Due to that, an extension of the CBIR systems is

designed to exploit images at region or object level. One of the important tasks in

CBIR at region or object level is to segment images into regions based on low-level

features. Among the low-level features, colour and location information are widely

used.

In order to extract the colour information, Colour-based Dominant Region

segmentation is used to extract a maximum of three dominant colour regions in an

image together with its respective coordinates of the Minimum-Bounding Rectangle

iv

(MBR). The Sub-Block technique is then used to determine the location of the

dominant regions by comparing the coordinates of the region’s MBR with the four

corners of the centre of the location map. The cell number that is maximally covered

by the region is supposedly to be assigned as the location index. However, the Sub-

Block technique is not reliable because in most cases, the location index assigned is

not the cell number that is maximally covered by the region and sometimes a region

does not overlap with the cell number assigned at all. The effectiveness of this

technique has been improved by taking into consideration the total horizontal and

vertical distance of a region at each location where it overlaps. The horizontal

distance from the left edge to the right edge of a region and the vertical distance

from the top edge to the bottom edge of a region are calculated. The horizontal and

vertical distances obtained for that region are then counted. The cell number with the

highest distance would be assigned as the location index for that region. The

individual colour and location index of each dominant region in an image is

extended to provide combined colour-spatial indexes. During retrieval, images in the

image database that have the same index as the query image is retrieved.

A CBIR system implementing the Improved Sub-Block technique is developed. The

CBIR system supports Query-By-Example (QBE). The retrieval effectiveness of the

improved technique is tested through retrieval experiments on six image categories

of about 900 images. The precision and recall is measured. From the experiments it

is shown that retrieval effectiveness has been significantly improved by 85.86%

through the Improved Sub-Block technique.

v

Abstrak tesis yang dikemukakan kepada Senat Universiti Putra Malaysia sebagai memenuhi keperluan untuk ijazah Master Sains

DAPATAN SEMULA IMEJ BERASASKAN KANDUNGAN MELALUI PEMBAIKKAN TEKNIK SUB-BLOK

Oleh

MAS RINA BINTI MUSTAFFA

Ogos 2006

Pengerusi: Profesor Madya Hajah Fatimah Ahmad, PhD

Fakulti : Sains Komputer dan Teknologi Maklumat

Sistem tradisional Dapatan Semula Imej Berasaskan Kandungan biasanya

bergantung kepada perwakilan ciri secara global. Pendekatan ini mempunyai

kelemahan di mana ia tidak dapat menangkap ciri-ciri penting bagi sesuatu

kelompok di dalam imej yang mungkin menjadi kepentingan kepada seseorang

pengguna. Oleh yang demikian, pembaikan terhadap Sistem Dapatan Semula Imej

berasaskan Kandungan telah direka bentuk untuk mengeksploitasi imej pada

peringkat objek atau kelompok. Salah satu tugas penting Sistem Dapatan Semula

Imej berasaskan Kandungan peringkat kelompok atau objek adalah untuk

membahagi imej kepada beberapa kelompok berdasarkan kepada ciri bertahap

rendah. Di antara ciri bertahap rendah, warna dan kedudukan digunakan secara

berleluasa.

vi

Bagi mengekstrak maklumat warna, segmentasi Kelompok Dominan berasaskan

Warna digunakan untuk mengekstrak sekurang-kurangnya tiga kelompok yang

mempunyai warna yang dominan beserta dengan koordinat pengsempadanan

minima segiempatnya. Teknik Sub-Blok kemudiannya digunakan untuk menentukan

kedudukan kelompok dominan dengan membandingkan sempadan minima

segiempat kelompok tersebut dengan empat pepenjuru bahagian tengah peta lokasi.

Dalam hal ini, nombor sel yang diliputi secara maksima oleh kelompok tersebut

sepatutnya ditetapkan sebagai indeks kedudukannya. Walau bagaimanapun,

kebergantungan teknik Sub-blok ini diragui kerana di dalam kebanyakan situasi,

indeks kedudukan yang ditetapkan bukanlah nombor sel yang diliputi oleh kelompok

tersebut secara maksima dan kadangkala, kelompok tersebut tidak langsung

menindih sel tersebut. Keberkesanan teknik ini telah diperbaiki dengan mengambil

kira jumlah jarak melintang dan menegak sesuatu kelompok di setiap kedudukan di

mana ia bertindih. Jarak melintang daripada sisi kiri ke sisi kanan dan jarak

menegak daripada sisi atas ke sisi bawah bagi sesuatu kelompok akan dikira. Jarak

melintang dan menegak yang diperoleh bagi kelompok tersebut akan dijumlahkan.

Nombor sel dengan jumlah jarak yang tertinggi akan ditetapkan sebagai indeks

kedudukan untuk kelompok tersebut. Indeks warna dan indeks kedudukan bagi

setiap kelompok di dalam sesuatu imej akan digunakan untuk menghasilkan

beberapa gabungan indeks warna-kedudukan. Sewaktu dapatan semula, imej di

dalam pangkalan data yang mempunyai indeks yang sama dengan imej carian akan

diambil (papar).

vii

Sistem Dapatan Semula Imej berasaskan Kandungan yang melaksanakan teknik

Pembaikan Sub-Blok telah dibangunkan. Sistem yang dibangunkan menyokong

carian berdasarkan contoh. Keberkesanan dapatan semula menggunakan teknik yang

diperbaiki telah diuji ke atas enam kategori imej yang mengandungi 900 imej

melalui eksperimen dapatan semula. Ketepatan dan perolehan telah diukur. Hasil

eksperimen menunjukkan peningkatan yang jelas dari segi keberkesanan dapatan

semula iaitu sebanyak 85.86% telah dapat dicapai melalui teknik Pembaikkan Sub-

blok.

viii

ACKNOWLEDGEMENTS

‘Alhamdulillah’, finally this thesis is completed. First and for most, I would like to

thank my supervisor, Associate Professor Dr. Hjh Fatimah Binti Dato’ Ahmad for

her constant support and guidance. She gave me enough time to start my research

and trusted that I would accomplish something. I am also grateful to my Supervisory

Committee Members, Dr. Rahmita Wirza O.K. Rahmat and Associate Professor Dr.

Ramlan Bin Mahmod for their guidance and valuable ideas given through out the

duration of this research study.

A big thank you to both of my parents, Mr. Mustaffa Mohd. Som and Mrs. Azizah

Zabidi and the rest of the family for giving me all the support that I need and have

been always encouraging me to pursue my studies until this far.

My thanks are also extended to Dr. Jehad and my friends for their willingness to

share knowledge on matters regarding to this research field.

Last but not least, a very big thank you to my beloved husband, Mr. Mohd Azrin for

all the ideas, attention, and encouragement that you gave during the ups and downs

of the ‘stressful’ moments. All of your kind deed will indeed be remembered.

Mas Rina Binti Mustaffa

2006

ix

I certify that an Examination Committee has met on 9 August 2006 to conduct the final examination of Mas Rina Binti Mustaffa on her Master of Science thesis entitled “Content-Based Image Retrieval Through Improved Sub-Block Technique” in accordance with Universiti Pertanian Malaysia (Higher Degree) Act 1980 and Universiti Pertanian Malaysia (Higher Degree) Regulations 1981. The Committee recommends that the candidate be awarded the relevant degree. Members of the Examination Committee are as follows: Md. Nasir Sulaiman, PhD Associate Professor Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Chairman) Abdul Rahman Ramli, PhD Associate Professor Faculty of Engineering Universiti Putra Malaysia (Internal Examiner) Shyamala C. Doraisamy, PhD Lecturer Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Internal Examiner) Siti Mariyam Hj. Shamsudin, PhD Associate Professor Faculty of Computer Science and Information Systems Universiti Teknologi Malaysia (External Examiner) _________________________________ HASANAH MOHD. GHAZALI, PhD Professor/Deputy Dean School of Graduate Studies Universiti Putra Malaysia Date:

x

This thesis submitted to the Senate of Universiti Putra Malaysia and has been accepted as fulfilment of the requirement for the degree of Master of Science. The members of the Supervisory Committee are as follows: Fatimah Dato’ Ahmad, PhD Associate Professor Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Chairman) Ramlan Mahmod, PhD Associate Professor Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Member) Rahmita Wirza O.K. Rahmat, PhD Lecturer Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Member) ______________________ AINI IDERIS, PhD Professor / Dean School of Graduate Studies Universiti Putra Malaysia Date:

xi

DECLARATION

I hereby declare that the thesis is based on my original work except for quotations and citations which have been duly acknowledged. I also declare that it has not been previously or concurrently submitted for any other degree at UPM or other institutions. ______________________________ MAS RINA BINTI MUSTAFFA Date:

xii

TABLE OF CONTENTS

Page

DEDICATION ii ABSTRACT iii ABSTRAK v ACKNOWLEDGEMENTS viii APPROVAL ix DECLARATION xi LIST OF TABLES xiv LIST OF FIGURES xvi LIST OF ABBREVIATIONS xviii CHAPTER

1 INTRODUCTION 1.1 Background 1 1.2 Problem Statement 3 1.3 Research Objectives 4 1.4 Scope of Research 4 1.5 Research Methodology 7 1.6 Contributions of Research 8 1.7 Thesis Organisation 8

2 LITERATURE REVIEW 2.1 Introduction 10 2.2 Content-Based Image Retrieval 10 2.3 Low-level Features 15

2.3.1 Colour Feature 15 2.3.2 Spatial Location Feature 20

2.4 Previous Work on Colour-Spatial Techniques 22 2.5 Sub-Block Technique 29 2.6 Summary 34

3 METHODOLOGY 3.1 Introduction 37 3.2 Methodology 37 3.3 Data Collection 40 3.4 Experimental Settings 41

3.4.1 Experiment Objective 43 3.4.2 Data 43 3.4.3 Task 43 3.4.4 Data Analysis and Representation 43

xiii

3.5 Summary 44

4 SYSTEM DESIGN 4.1 Content-Based Image Retrieval Framework 46

4.1.1 Image Initialisation 48 4.1.2 Colour-Based Dominant Region Segmentation 48 4.1.3 Improved Sub-Block Technique 51 4.1.4 Indexing 54

4.2 System Requirement 57 4.3 User Interface Design 59 4.4 Screen Link Design 64 4.5 Entity-Relationship Diagram (ERD) 65 4.6 Data Flow Diagram (DFD) 68 4.7 Data Dictionary 72 4.8 Summary 77

5 RESULTS AND DISCUSSION 5.1 Introduction 78 5.2 Example of CBIR System Inputs and Outputs 79 5.3 Average Precision and Recall for Sub-Block Technique 90

versus Improved Sub-Block Technique 5.3.1 Flowers category 91

5.4 Discussion 93

6 CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE RESEARCH WORKS 6.1 Summary and Conclusions 96 6.2 Future Research Works 98

REFERENCES 100 APPENDICES 103 BIODATA OF THE AUTHOR 116

xiv

LIST OF TABLES

Table Page

3.1 Research Objectives and Methodology 39 3.2 Images according to Image Category 40 4.1 Colour Look-up Table 49 4.2 Label according to Category 73 4.3 Structure for ImageIDXX 73 4.4 Structure for ImageRGBXX 73 4.5 Structure for ImageDiffRGBXX 73 4.6 Structure for ImageFreqRGBXX 73 4.7 Structure for ImageRGMXX 74 4.8 Structure for ImageLocXX 74 4.9 Structure for ImageRGMLocXX 74 4.10 Structure for AreaTableXX 74 4.11 Structure for MRLCheckXX 75 4.12 Structure for ImageIndexXX 75 4.13 Structure for ImageIndexOverAll_XX 76 4.14 Structure for ColourLook-upTable 76 4.15 Structure for Login 76 4.16 Structure for MAAI 76 4.17 Structure for PerMeasSummary 76 5.1 Average Precision of 10 Queries based on Sub-Block Technique 91 for Flowers Category

xv

5.2 Average Precision of 10 Queries based on Improved Sub-Block 91 Technique for Flowers Category 5.3 Precision Difference between Sub-Block and Improved Sub-Block 92

Technique according to Image Category

xvi

LIST OF FIGURES

Figure Page

2.1 Interactions of Major Components for a General CBIR System 12 2.2 Location Map 30 2.3 An Example of a Region Falls within One Grid Cell 31 2.4 An Example of a Region Overlaps more than One Grid Cell 32 4.1 Content-Based Image Retrieval System Framework 47 4.2 Algorithm for Region Growing Method 50 4.3 A Region with its Horizontal and Vertical Distance 51 4.4 Algorithm for Improved Sub-Block Technique 52 4.5 A Region consists of One Line 53 4.6 Possible Combination Indexes for Three Dominant Regions 56 4.7 Content-Based Image Retrieval System Requirement 58 4.8 Main Menu Interface Design 59 4.9 User Maintenance Interface Design 60 4.10 Change Password Interface Design 61 4.11 Category Database Maintenance Interface Design 62 4.12 Search for Images Interface Design 63 4.13 Screen Link Design 65 4.14 ERD at Level 1 for ‘Tools’ Category 67 4.15 ERD at Level 2 for ‘Tools’ Category 68

4.16 CBIR System Context Diagram 68

xvii

4.17 DFD Diagram 0 for ‘Tools’ Category 69 4.18 DFD at Level 1 for User Maintenance 69 4.19 DFD at Level 1 for Image Retrieval of ‘Tools’ Category 70 4.20 DFD at Level 1 for Performance Measurement of ‘Tools’ Category 70 4.21 DFD at Level 1 for Image Maintenance of ‘Tools’ Category 71 4.22 DFD at Level 2 for Add Images of ‘Tools’ Category 71 4.23 DFD at Level 3 for Identify Location of ‘Tools’ Category 72 5.1 User at Level 1 Entering User ID and Password in Login Screen 79 5.2 Invalid User ID or Password 80 5.3 Invalid Old Password 81 5.4 Password Successfully Changed 82 5.5 Level 1 User Adding a New Level 2 User 83 5.6 Level 1 User is Viewing the Added Users in the User Database 83 5.7 Choose Image Database Category Screen 85 5.8 Category Database Maintenance for ‘Flags’ Category 85 5.9 An Image is being Processed by the CBIR System 86 5.10 An Image is Deleted from the CBIR System 87 5.11 Selecting a Query Image 88 5.12 Image Retrieval based on Sub-Block Technique 89 5.13 Image Retrieval based on Improved Sub-Block Technique 89 5.14 Average Precision and Recall for Sub-Block Technique versus Improved 92

Sub-Block Technique for Flowers Category 5.15 Sample of Images Fully Filled in the MBR After Segmentation 95 5.16 Sample of Images Not Fully Filled in the MBR After Segmentation 95

xviii

LIST OF ABBREVIATIONS

CBIR Content-Based Image Retrieval

CLI Colour-Location Index

ISB Improved Sub-Block

MBR Minimum-Bounding Rectangle

QBE Query-By-Example

SB Sub-Block

CHAPTER 1

INTRODUCTION

1.1 Background

Content-Based Image Retrieval (CBIR) is an exciting and in-depth area of research,

which has garnered much interest over the past few years. The relevance of visual

information retrieval for many applications which includes art galleries, museum

archives, criminal investigation, medical, and geographic databases makes this

research field one of the important and fastest growing in information technology.

Image retrieval has come a long way where it started off with text-based retrieval.

However, there are many problems associated with retrieving images based on text

such as manual annotation of keywords, differences in perceptions and

interpretations, and a few others. Due to this, researchers came up with CBIR where

images are retrieved based on the information extracted from the content of the

image globally. Later, CBIR has been extended to exploit images at region or object

level, which gives more explicit information of the image structure and

configuration.

One of the important task in CBIR at region or object level is to segment images into

regions on which the user can perform queries, based on low-level features such as

colour, shape, texture, location, etc. Among these low level features, colour plays an

important role in CBIR because of its robustness to complex background and

independent of image size and orientation. However, where CBIR is concern, using

2

colour alone is not sufficient to characterise an image. For example, there are two

different images; one contains a red ball at the left side while another image contains

a red ball at the top. Based on human observer, these two images are different.

However a colour-based retrieval system will assume that these two different images

are the same since both of them do have the same colour composition despite that

their location is different. In order to rectify this problem and produce a better

retrieval system, multiple but related features are usually used. If we look at the

example above again, since everything is the same, the spatial pattern will possibly

distinguish the image content. There are several colour descriptors incorporated with

the location feature such as Colour Coherence Vectors (Pass et. al, 1996), Binary

Thresholded Histogram (Walczak, 2001), Spatial Chromatic Histogram (Cinque et.

al, 2001), Major Colour Set and Distribution Block Signature (Yoo et. al, 2005), and

a few others. Colour-spatial features can really help in differentiating images having

pixels with the same colour distribution but different location information. However,

the methods must be able to retrieve images that a user requires.

Prasad et. al (2004) acknowledged the importance of combining colour and spatial

information for better retrieval result. In their work, a maximum of three dominant

colour regions in an image is extracted using Colour-Based Dominant Region

segmentation (Ravishankar et. al, 1999) and the location of the regions is then

determined using the Sub-Block technique (Prasad et. al, 2004). The Sub-Block

technique works by finding the distance from the region�s Minimum-Bounding

Rectangle (MBR) to the coordinates of the centre of the location map. According to

3

them, the cell number that is maximally covered by a region would be assigned as

the location index. However, in most cases, the location index assigned is the cell

number having the highest distance from the region�s MBR to the coordinates of the

centre of the location map instead of the cell number that is maximally covered by

the region. This will result in the wrong location being assigned to the region, thus it

is not reliable.

The aim of this research work is to overcome the above-mentioned shortcoming

existed in the Sub-Block technique. Having found the colour-spatial information, the

information is then used for further image retrieval process through a CBIR system.

1.2 Problem Statement

According to Prasad et. al (2004), their approach is based on the Sub-Block

technique where the image space is divided into nine sub-locations numbered one to

nine. The classification is according to this numbered grid cell where the cell

number that is maximally covered by the region is assigned as the location index. In

order to determine the location of a region, the coordinates of the region�s MBR will

be compared with the four corners of the centre of the location map. If the MBR of a

region falls within one grid cell, then the location of a region can be determined

clearly and correctly. However most of the regions are likely to overlap more than

one grid cell in the image space. When this happens, the distance from the region�s

MBR to the coordinates of the centre of the location map will be used to determine

the region�s location. However, in most cases, the location index assigned is the cell

4

number having the highest distance from the region�s MBR to the coordinates of the

centre of the location map instead of the cell number that is maximally covered by

the region. This will usually result in the wrong location being assigned to the

region.

This research focused on eliminating the above-mentioned weakness in the Sub-

Block approach by taking into consideration the total horizontal and vertical distance

of a region at each location where it overlaps. This will result in an enhanced

technique that will be able to locate a region effectively thus, providing better

retrieval results when implementing the technique on a CBIR system.

1.3 Research Objectives

The objectives of this research work are as follows:

1. Improving the Sub-Block technique so that it will be able to detect the location

of a region in a more effective way.

2. Developing a colour-location CBIR system supporting Query-By-Example

(QBE) that implements the improved technique.

1.4 Scope of Research

This study focuses on developing a colour-spatial technique for CBIR. The colour

information is extracted using the existing Colour-Based Dominant Region

segmentation while the location information is extracted using the Improved Sub-

Block technique. This Improved Sub-Block technique is able to locate regions in an

5

image effectively compared to the original technique where the cell number that is

maximally covered by the region is indeed assigned as the location index. The

location index together with the colour index is combined to generate possible

colour-location indexes. These colour-location indexes are used for retrieval.

A CBIR system implementing the colour-spatial technique is developed. The

developed CBIR system consists of three main modules, which are the user

maintenance module, category database maintenance module and the image retrieval

and performance measurement module. User maintenance module is used to

maintain users who have the access to the category database maintenance module. In

the category database maintenance module, the system is able to extract colour and

location information from an image, store the necessary extracted information into

the database, and maintain the image database where new images can be added

while stored images can be deleted. In the image retrieval and performance

measurement module, querying and retrieval of images that have been processed by

the category database maintenance module, can be done. The module is also able to

automatically make the relevant judgement for each of the query image based on

Area technique as the benchmark.

The CBIR system supports Query-By-Example (QBE). During retrieval, images

having the same index as the indexes of the query image are retrieved. The CBIR

system focuses more towards similar retrieval rather than speedy retrieval. Each of

the query images is tested using the Sub-Block technique and the Improved Sub-