universiti putra malaysia content-based image retrieval...
TRANSCRIPT
UNIVERSITI PUTRA MALAYSIA
CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-
BLOCK TECHNIQUE
MAS RINA BINTI MUSTAFFA
FSKTM 2006 6
CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-BLOCK
TECHNIQUE
MAS RINA BINTI MUSTAFFA
MASTER OF SCIENCE UNIVERSITI PUTRA MALAYSIA
2006
CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-BLOCK TECHNIQUE
By
MAS RINA BINTI MUSTAFFA
Thesis Submitted to the School of Graduate Studies, Universiti Putra Malaysia, in Fulfilment of the Requirement for the Degree of Master of Science
August 2006
iii
Abstract of thesis presented to the Senate of Universiti Putra Malaysia in fulfilment of the requirement for the degree of Master of Science
CONTENT-BASED IMAGE RETRIEVAL THROUGH IMPROVED SUB-BLOCK TECHNIQUE
By
MAS RINA BINTI MUSTAFFA
August 2006
Chairman: Associate Professor Hajah Fatimah Ahmad, PhD
Faculty : Computer Science and Information Technology
Traditional Content-Based Image Retrieval (CBIR) systems mainly relied on the
extraction of features globally. The drawback of this approach is that it cannot
sufficiently capture the important features of individual regions in an image which
users might be interested in. Due to that, an extension of the CBIR systems is
designed to exploit images at region or object level. One of the important tasks in
CBIR at region or object level is to segment images into regions based on low-level
features. Among the low-level features, colour and location information are widely
used.
In order to extract the colour information, Colour-based Dominant Region
segmentation is used to extract a maximum of three dominant colour regions in an
image together with its respective coordinates of the Minimum-Bounding Rectangle
iv
(MBR). The Sub-Block technique is then used to determine the location of the
dominant regions by comparing the coordinates of the region’s MBR with the four
corners of the centre of the location map. The cell number that is maximally covered
by the region is supposedly to be assigned as the location index. However, the Sub-
Block technique is not reliable because in most cases, the location index assigned is
not the cell number that is maximally covered by the region and sometimes a region
does not overlap with the cell number assigned at all. The effectiveness of this
technique has been improved by taking into consideration the total horizontal and
vertical distance of a region at each location where it overlaps. The horizontal
distance from the left edge to the right edge of a region and the vertical distance
from the top edge to the bottom edge of a region are calculated. The horizontal and
vertical distances obtained for that region are then counted. The cell number with the
highest distance would be assigned as the location index for that region. The
individual colour and location index of each dominant region in an image is
extended to provide combined colour-spatial indexes. During retrieval, images in the
image database that have the same index as the query image is retrieved.
A CBIR system implementing the Improved Sub-Block technique is developed. The
CBIR system supports Query-By-Example (QBE). The retrieval effectiveness of the
improved technique is tested through retrieval experiments on six image categories
of about 900 images. The precision and recall is measured. From the experiments it
is shown that retrieval effectiveness has been significantly improved by 85.86%
through the Improved Sub-Block technique.
v
Abstrak tesis yang dikemukakan kepada Senat Universiti Putra Malaysia sebagai memenuhi keperluan untuk ijazah Master Sains
DAPATAN SEMULA IMEJ BERASASKAN KANDUNGAN MELALUI PEMBAIKKAN TEKNIK SUB-BLOK
Oleh
MAS RINA BINTI MUSTAFFA
Ogos 2006
Pengerusi: Profesor Madya Hajah Fatimah Ahmad, PhD
Fakulti : Sains Komputer dan Teknologi Maklumat
Sistem tradisional Dapatan Semula Imej Berasaskan Kandungan biasanya
bergantung kepada perwakilan ciri secara global. Pendekatan ini mempunyai
kelemahan di mana ia tidak dapat menangkap ciri-ciri penting bagi sesuatu
kelompok di dalam imej yang mungkin menjadi kepentingan kepada seseorang
pengguna. Oleh yang demikian, pembaikan terhadap Sistem Dapatan Semula Imej
berasaskan Kandungan telah direka bentuk untuk mengeksploitasi imej pada
peringkat objek atau kelompok. Salah satu tugas penting Sistem Dapatan Semula
Imej berasaskan Kandungan peringkat kelompok atau objek adalah untuk
membahagi imej kepada beberapa kelompok berdasarkan kepada ciri bertahap
rendah. Di antara ciri bertahap rendah, warna dan kedudukan digunakan secara
berleluasa.
vi
Bagi mengekstrak maklumat warna, segmentasi Kelompok Dominan berasaskan
Warna digunakan untuk mengekstrak sekurang-kurangnya tiga kelompok yang
mempunyai warna yang dominan beserta dengan koordinat pengsempadanan
minima segiempatnya. Teknik Sub-Blok kemudiannya digunakan untuk menentukan
kedudukan kelompok dominan dengan membandingkan sempadan minima
segiempat kelompok tersebut dengan empat pepenjuru bahagian tengah peta lokasi.
Dalam hal ini, nombor sel yang diliputi secara maksima oleh kelompok tersebut
sepatutnya ditetapkan sebagai indeks kedudukannya. Walau bagaimanapun,
kebergantungan teknik Sub-blok ini diragui kerana di dalam kebanyakan situasi,
indeks kedudukan yang ditetapkan bukanlah nombor sel yang diliputi oleh kelompok
tersebut secara maksima dan kadangkala, kelompok tersebut tidak langsung
menindih sel tersebut. Keberkesanan teknik ini telah diperbaiki dengan mengambil
kira jumlah jarak melintang dan menegak sesuatu kelompok di setiap kedudukan di
mana ia bertindih. Jarak melintang daripada sisi kiri ke sisi kanan dan jarak
menegak daripada sisi atas ke sisi bawah bagi sesuatu kelompok akan dikira. Jarak
melintang dan menegak yang diperoleh bagi kelompok tersebut akan dijumlahkan.
Nombor sel dengan jumlah jarak yang tertinggi akan ditetapkan sebagai indeks
kedudukan untuk kelompok tersebut. Indeks warna dan indeks kedudukan bagi
setiap kelompok di dalam sesuatu imej akan digunakan untuk menghasilkan
beberapa gabungan indeks warna-kedudukan. Sewaktu dapatan semula, imej di
dalam pangkalan data yang mempunyai indeks yang sama dengan imej carian akan
diambil (papar).
vii
Sistem Dapatan Semula Imej berasaskan Kandungan yang melaksanakan teknik
Pembaikan Sub-Blok telah dibangunkan. Sistem yang dibangunkan menyokong
carian berdasarkan contoh. Keberkesanan dapatan semula menggunakan teknik yang
diperbaiki telah diuji ke atas enam kategori imej yang mengandungi 900 imej
melalui eksperimen dapatan semula. Ketepatan dan perolehan telah diukur. Hasil
eksperimen menunjukkan peningkatan yang jelas dari segi keberkesanan dapatan
semula iaitu sebanyak 85.86% telah dapat dicapai melalui teknik Pembaikkan Sub-
blok.
viii
ACKNOWLEDGEMENTS
‘Alhamdulillah’, finally this thesis is completed. First and for most, I would like to
thank my supervisor, Associate Professor Dr. Hjh Fatimah Binti Dato’ Ahmad for
her constant support and guidance. She gave me enough time to start my research
and trusted that I would accomplish something. I am also grateful to my Supervisory
Committee Members, Dr. Rahmita Wirza O.K. Rahmat and Associate Professor Dr.
Ramlan Bin Mahmod for their guidance and valuable ideas given through out the
duration of this research study.
A big thank you to both of my parents, Mr. Mustaffa Mohd. Som and Mrs. Azizah
Zabidi and the rest of the family for giving me all the support that I need and have
been always encouraging me to pursue my studies until this far.
My thanks are also extended to Dr. Jehad and my friends for their willingness to
share knowledge on matters regarding to this research field.
Last but not least, a very big thank you to my beloved husband, Mr. Mohd Azrin for
all the ideas, attention, and encouragement that you gave during the ups and downs
of the ‘stressful’ moments. All of your kind deed will indeed be remembered.
Mas Rina Binti Mustaffa
2006
ix
I certify that an Examination Committee has met on 9 August 2006 to conduct the final examination of Mas Rina Binti Mustaffa on her Master of Science thesis entitled “Content-Based Image Retrieval Through Improved Sub-Block Technique” in accordance with Universiti Pertanian Malaysia (Higher Degree) Act 1980 and Universiti Pertanian Malaysia (Higher Degree) Regulations 1981. The Committee recommends that the candidate be awarded the relevant degree. Members of the Examination Committee are as follows: Md. Nasir Sulaiman, PhD Associate Professor Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Chairman) Abdul Rahman Ramli, PhD Associate Professor Faculty of Engineering Universiti Putra Malaysia (Internal Examiner) Shyamala C. Doraisamy, PhD Lecturer Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Internal Examiner) Siti Mariyam Hj. Shamsudin, PhD Associate Professor Faculty of Computer Science and Information Systems Universiti Teknologi Malaysia (External Examiner) _________________________________ HASANAH MOHD. GHAZALI, PhD Professor/Deputy Dean School of Graduate Studies Universiti Putra Malaysia Date:
x
This thesis submitted to the Senate of Universiti Putra Malaysia and has been accepted as fulfilment of the requirement for the degree of Master of Science. The members of the Supervisory Committee are as follows: Fatimah Dato’ Ahmad, PhD Associate Professor Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Chairman) Ramlan Mahmod, PhD Associate Professor Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Member) Rahmita Wirza O.K. Rahmat, PhD Lecturer Faculty of Computer Science and Information Technology Universiti Putra Malaysia (Member) ______________________ AINI IDERIS, PhD Professor / Dean School of Graduate Studies Universiti Putra Malaysia Date:
xi
DECLARATION
I hereby declare that the thesis is based on my original work except for quotations and citations which have been duly acknowledged. I also declare that it has not been previously or concurrently submitted for any other degree at UPM or other institutions. ______________________________ MAS RINA BINTI MUSTAFFA Date:
xii
TABLE OF CONTENTS
Page
DEDICATION ii ABSTRACT iii ABSTRAK v ACKNOWLEDGEMENTS viii APPROVAL ix DECLARATION xi LIST OF TABLES xiv LIST OF FIGURES xvi LIST OF ABBREVIATIONS xviii CHAPTER
1 INTRODUCTION 1.1 Background 1 1.2 Problem Statement 3 1.3 Research Objectives 4 1.4 Scope of Research 4 1.5 Research Methodology 7 1.6 Contributions of Research 8 1.7 Thesis Organisation 8
2 LITERATURE REVIEW 2.1 Introduction 10 2.2 Content-Based Image Retrieval 10 2.3 Low-level Features 15
2.3.1 Colour Feature 15 2.3.2 Spatial Location Feature 20
2.4 Previous Work on Colour-Spatial Techniques 22 2.5 Sub-Block Technique 29 2.6 Summary 34
3 METHODOLOGY 3.1 Introduction 37 3.2 Methodology 37 3.3 Data Collection 40 3.4 Experimental Settings 41
3.4.1 Experiment Objective 43 3.4.2 Data 43 3.4.3 Task 43 3.4.4 Data Analysis and Representation 43
xiii
3.5 Summary 44
4 SYSTEM DESIGN 4.1 Content-Based Image Retrieval Framework 46
4.1.1 Image Initialisation 48 4.1.2 Colour-Based Dominant Region Segmentation 48 4.1.3 Improved Sub-Block Technique 51 4.1.4 Indexing 54
4.2 System Requirement 57 4.3 User Interface Design 59 4.4 Screen Link Design 64 4.5 Entity-Relationship Diagram (ERD) 65 4.6 Data Flow Diagram (DFD) 68 4.7 Data Dictionary 72 4.8 Summary 77
5 RESULTS AND DISCUSSION 5.1 Introduction 78 5.2 Example of CBIR System Inputs and Outputs 79 5.3 Average Precision and Recall for Sub-Block Technique 90
versus Improved Sub-Block Technique 5.3.1 Flowers category 91
5.4 Discussion 93
6 CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE RESEARCH WORKS 6.1 Summary and Conclusions 96 6.2 Future Research Works 98
REFERENCES 100 APPENDICES 103 BIODATA OF THE AUTHOR 116
xiv
LIST OF TABLES
Table Page
3.1 Research Objectives and Methodology 39 3.2 Images according to Image Category 40 4.1 Colour Look-up Table 49 4.2 Label according to Category 73 4.3 Structure for ImageIDXX 73 4.4 Structure for ImageRGBXX 73 4.5 Structure for ImageDiffRGBXX 73 4.6 Structure for ImageFreqRGBXX 73 4.7 Structure for ImageRGMXX 74 4.8 Structure for ImageLocXX 74 4.9 Structure for ImageRGMLocXX 74 4.10 Structure for AreaTableXX 74 4.11 Structure for MRLCheckXX 75 4.12 Structure for ImageIndexXX 75 4.13 Structure for ImageIndexOverAll_XX 76 4.14 Structure for ColourLook-upTable 76 4.15 Structure for Login 76 4.16 Structure for MAAI 76 4.17 Structure for PerMeasSummary 76 5.1 Average Precision of 10 Queries based on Sub-Block Technique 91 for Flowers Category
xv
5.2 Average Precision of 10 Queries based on Improved Sub-Block 91 Technique for Flowers Category 5.3 Precision Difference between Sub-Block and Improved Sub-Block 92
Technique according to Image Category
xvi
LIST OF FIGURES
Figure Page
2.1 Interactions of Major Components for a General CBIR System 12 2.2 Location Map 30 2.3 An Example of a Region Falls within One Grid Cell 31 2.4 An Example of a Region Overlaps more than One Grid Cell 32 4.1 Content-Based Image Retrieval System Framework 47 4.2 Algorithm for Region Growing Method 50 4.3 A Region with its Horizontal and Vertical Distance 51 4.4 Algorithm for Improved Sub-Block Technique 52 4.5 A Region consists of One Line 53 4.6 Possible Combination Indexes for Three Dominant Regions 56 4.7 Content-Based Image Retrieval System Requirement 58 4.8 Main Menu Interface Design 59 4.9 User Maintenance Interface Design 60 4.10 Change Password Interface Design 61 4.11 Category Database Maintenance Interface Design 62 4.12 Search for Images Interface Design 63 4.13 Screen Link Design 65 4.14 ERD at Level 1 for ‘Tools’ Category 67 4.15 ERD at Level 2 for ‘Tools’ Category 68
4.16 CBIR System Context Diagram 68
xvii
4.17 DFD Diagram 0 for ‘Tools’ Category 69 4.18 DFD at Level 1 for User Maintenance 69 4.19 DFD at Level 1 for Image Retrieval of ‘Tools’ Category 70 4.20 DFD at Level 1 for Performance Measurement of ‘Tools’ Category 70 4.21 DFD at Level 1 for Image Maintenance of ‘Tools’ Category 71 4.22 DFD at Level 2 for Add Images of ‘Tools’ Category 71 4.23 DFD at Level 3 for Identify Location of ‘Tools’ Category 72 5.1 User at Level 1 Entering User ID and Password in Login Screen 79 5.2 Invalid User ID or Password 80 5.3 Invalid Old Password 81 5.4 Password Successfully Changed 82 5.5 Level 1 User Adding a New Level 2 User 83 5.6 Level 1 User is Viewing the Added Users in the User Database 83 5.7 Choose Image Database Category Screen 85 5.8 Category Database Maintenance for ‘Flags’ Category 85 5.9 An Image is being Processed by the CBIR System 86 5.10 An Image is Deleted from the CBIR System 87 5.11 Selecting a Query Image 88 5.12 Image Retrieval based on Sub-Block Technique 89 5.13 Image Retrieval based on Improved Sub-Block Technique 89 5.14 Average Precision and Recall for Sub-Block Technique versus Improved 92
Sub-Block Technique for Flowers Category 5.15 Sample of Images Fully Filled in the MBR After Segmentation 95 5.16 Sample of Images Not Fully Filled in the MBR After Segmentation 95
xviii
LIST OF ABBREVIATIONS
CBIR Content-Based Image Retrieval
CLI Colour-Location Index
ISB Improved Sub-Block
MBR Minimum-Bounding Rectangle
QBE Query-By-Example
SB Sub-Block
CHAPTER 1
INTRODUCTION
1.1 Background
Content-Based Image Retrieval (CBIR) is an exciting and in-depth area of research,
which has garnered much interest over the past few years. The relevance of visual
information retrieval for many applications which includes art galleries, museum
archives, criminal investigation, medical, and geographic databases makes this
research field one of the important and fastest growing in information technology.
Image retrieval has come a long way where it started off with text-based retrieval.
However, there are many problems associated with retrieving images based on text
such as manual annotation of keywords, differences in perceptions and
interpretations, and a few others. Due to this, researchers came up with CBIR where
images are retrieved based on the information extracted from the content of the
image globally. Later, CBIR has been extended to exploit images at region or object
level, which gives more explicit information of the image structure and
configuration.
One of the important task in CBIR at region or object level is to segment images into
regions on which the user can perform queries, based on low-level features such as
colour, shape, texture, location, etc. Among these low level features, colour plays an
important role in CBIR because of its robustness to complex background and
independent of image size and orientation. However, where CBIR is concern, using
2
colour alone is not sufficient to characterise an image. For example, there are two
different images; one contains a red ball at the left side while another image contains
a red ball at the top. Based on human observer, these two images are different.
However a colour-based retrieval system will assume that these two different images
are the same since both of them do have the same colour composition despite that
their location is different. In order to rectify this problem and produce a better
retrieval system, multiple but related features are usually used. If we look at the
example above again, since everything is the same, the spatial pattern will possibly
distinguish the image content. There are several colour descriptors incorporated with
the location feature such as Colour Coherence Vectors (Pass et. al, 1996), Binary
Thresholded Histogram (Walczak, 2001), Spatial Chromatic Histogram (Cinque et.
al, 2001), Major Colour Set and Distribution Block Signature (Yoo et. al, 2005), and
a few others. Colour-spatial features can really help in differentiating images having
pixels with the same colour distribution but different location information. However,
the methods must be able to retrieve images that a user requires.
Prasad et. al (2004) acknowledged the importance of combining colour and spatial
information for better retrieval result. In their work, a maximum of three dominant
colour regions in an image is extracted using Colour-Based Dominant Region
segmentation (Ravishankar et. al, 1999) and the location of the regions is then
determined using the Sub-Block technique (Prasad et. al, 2004). The Sub-Block
technique works by finding the distance from the region�s Minimum-Bounding
Rectangle (MBR) to the coordinates of the centre of the location map. According to
3
them, the cell number that is maximally covered by a region would be assigned as
the location index. However, in most cases, the location index assigned is the cell
number having the highest distance from the region�s MBR to the coordinates of the
centre of the location map instead of the cell number that is maximally covered by
the region. This will result in the wrong location being assigned to the region, thus it
is not reliable.
The aim of this research work is to overcome the above-mentioned shortcoming
existed in the Sub-Block technique. Having found the colour-spatial information, the
information is then used for further image retrieval process through a CBIR system.
1.2 Problem Statement
According to Prasad et. al (2004), their approach is based on the Sub-Block
technique where the image space is divided into nine sub-locations numbered one to
nine. The classification is according to this numbered grid cell where the cell
number that is maximally covered by the region is assigned as the location index. In
order to determine the location of a region, the coordinates of the region�s MBR will
be compared with the four corners of the centre of the location map. If the MBR of a
region falls within one grid cell, then the location of a region can be determined
clearly and correctly. However most of the regions are likely to overlap more than
one grid cell in the image space. When this happens, the distance from the region�s
MBR to the coordinates of the centre of the location map will be used to determine
the region�s location. However, in most cases, the location index assigned is the cell
4
number having the highest distance from the region�s MBR to the coordinates of the
centre of the location map instead of the cell number that is maximally covered by
the region. This will usually result in the wrong location being assigned to the
region.
This research focused on eliminating the above-mentioned weakness in the Sub-
Block approach by taking into consideration the total horizontal and vertical distance
of a region at each location where it overlaps. This will result in an enhanced
technique that will be able to locate a region effectively thus, providing better
retrieval results when implementing the technique on a CBIR system.
1.3 Research Objectives
The objectives of this research work are as follows:
1. Improving the Sub-Block technique so that it will be able to detect the location
of a region in a more effective way.
2. Developing a colour-location CBIR system supporting Query-By-Example
(QBE) that implements the improved technique.
1.4 Scope of Research
This study focuses on developing a colour-spatial technique for CBIR. The colour
information is extracted using the existing Colour-Based Dominant Region
segmentation while the location information is extracted using the Improved Sub-
Block technique. This Improved Sub-Block technique is able to locate regions in an
5
image effectively compared to the original technique where the cell number that is
maximally covered by the region is indeed assigned as the location index. The
location index together with the colour index is combined to generate possible
colour-location indexes. These colour-location indexes are used for retrieval.
A CBIR system implementing the colour-spatial technique is developed. The
developed CBIR system consists of three main modules, which are the user
maintenance module, category database maintenance module and the image retrieval
and performance measurement module. User maintenance module is used to
maintain users who have the access to the category database maintenance module. In
the category database maintenance module, the system is able to extract colour and
location information from an image, store the necessary extracted information into
the database, and maintain the image database where new images can be added
while stored images can be deleted. In the image retrieval and performance
measurement module, querying and retrieval of images that have been processed by
the category database maintenance module, can be done. The module is also able to
automatically make the relevant judgement for each of the query image based on
Area technique as the benchmark.
The CBIR system supports Query-By-Example (QBE). During retrieval, images
having the same index as the indexes of the query image are retrieved. The CBIR
system focuses more towards similar retrieval rather than speedy retrieval. Each of
the query images is tested using the Sub-Block technique and the Improved Sub-