Navidgator— Similarity Based Browsing for Image & Video Databases —

Damian Borth, Christian Schulze,Adrian Ulges, Thomas M. Breuel

Image Understanding and Pattern RecognitionDFKI & TU Kaiserslautern

25.Sep.2008

Borth: KI2008 1 25.Sep.2008

Outline

Borth: KI2008 2 25.Sep.2008

MotivationHow to retrieve & browse in large databases of images or videos?

I provide global view→ browse through allimages of last year

I find visual similar images→ find all pictures with abeach

I Movie scene finder→ find scenes of Batcave form“The Dark Knight”

I TV network archives→ need a desert video clip fornews report

Borth: KI2008 3 25.Sep.2008

Browsing Metaphor: Stack of images

Start Browsing

I no query is selected

I a random set of image is displayed

I user selects image which roughly represents this query

Borth: KI2008 4 25.Sep.2008

Browsing Metaphor: Stack of images

Zooming & Panning

I zooms into the database i.r.t. selected query

I displays a more detailed cluster of images

I panning = refinement of query

I further zooms will lead to different results

Borth: KI2008 5 25.Sep.2008

Related Work

Query-by-Example [Flickner95]

I database is searchedaccording to given image

I tend to provide a localizedview of the database

I How to get an overview?

ViBE System [Chen04]

I browsing based onSimilarity Pyramids

I browsing performed onlyat shoot level→ shot = one keyframe→ loosing visual details

Borth: KI2008 6 25.Sep.2008

Our Approach

System Pipeline

I Keyframe Extraction→ only for video databases

I Feature Extraction

I Balanced Hierarchical Clustering

I User Interface

Borth: KI2008 7 25.Sep.2008

Keyframe Extraction

Notation

I video databaseDBvideo = X1, ...,Xn

I every video Xi ∈ DBvideo

is represented by a set of{xi1, ..., xim} keyframes

I → resulting in a total setof {x1, ..., xk} keyframesfor the entire DBvideo

Borth: KI2008 8 25.Sep.2008

Keyframe Extraction

Two Step Method

I Shot Boundary DetectionI Color Layout Desc.

(CLD) differencesI Adaptive Thresholding

[Lienhart01]

I Intra-Shot Clustering[Hammoud00]

I CLD feature spaceI K-MeansI Number of clusters:

Bayesian InformationCriterion (BIC)

Borth: KI2008 9 25.Sep.2008

Feature Extraction & Hierarchical ClusteringFeature Extraction

I a feature vector zj ∈ RD is extracted for everykeyframe xj ∈ {x1, ..., xk}

I zj ∈ RD consists of equally weighted color & texture featuresI Color Histograms - 8× 8× 8 binning of RGB CubeI Tamura Texture Histograms [Tamura78]

Clustering Overview

(a) Feature Space of zj ∈ RD

(b) Clustering Result

(c) Dendrogram

(d) Generated Binary Tree

Borth: KI2008 10 25.Sep.2008

Feature Extraction & Hierarchical ClusteringFeature Extraction

I a feature vector zj ∈ RD is extracted for everykeyframe xj ∈ {x1, ..., xk}

I zj ∈ RD consists of equally weighted color & texture featuresI Color Histograms - 8× 8× 8 binning of RGB CubeI Tamura Texture Histograms [Tamura78]

Clustering Overview

(a) Feature Space of zj ∈ RD

(b) Clustering Result

(c) Dendrogram

(d) Generated Binary Tree

Borth: KI2008 10 25.Sep.2008

Balanced Hierarchical Clustering (cont)

Agglomerative Hierarchical Clustering

1. initialization: - each feature vector zj ∈ {z1...zk} representsa singelton cluster cj ∈ {c1...ck}

- compute Euclidean distance for singeltons

2. merge two nearest clusters: {ci} ∧ {cj} → {ci , cj}3. update distances i.r.t. merged clusters

4. go to step 2 until |c | = 1

Nearest = Used Linkage Method

I Singelton Linkage

I Complete Linkage

I Average Linkage

Borth: KI2008 11 25.Sep.2008

Balanced Hierarchical Clustering (cont)

Agglomerative Hierarchical Clustering

1. initialization: - each feature vector zj ∈ {z1...zk} representsa singelton cluster cj ∈ {c1...ck}

- compute Euclidean distance for singeltons

2. merge two nearest clusters: {ci} ∧ {cj} → {ci , cj}3. update distances i.r.t. merged clusters

4. go to step 2 until |c | = 1

Nearest = Used Linkage Method

I Singelton Linkage

I Complete Linkage

I Average Linkage

Borth: KI2008 11 25.Sep.2008

Balanced Hierarchical Clustering (cont)

Average Linkage

d(ci , cj) =1

|ci | ∗ |cj |∗

∑zi∈ci

∑zj∈cj

d(zi , zj)

I distance between “means” of cluster pairs

Clustering Result

I binary tree is linearized → not usable for browsing

Borth: KI2008 12 25.Sep.2008

Balanced Hierarchical Clustering (cont)

Average Linkage

d(ci , cj) =1

|ci | ∗ |cj |∗

∑zi∈ci

∑zj∈cj

d(zi , zj)

I distance between “means” of cluster pairs

Clustering Result

I binary tree is linearized → not usable for browsing

Borth: KI2008 12 25.Sep.2008

Balanced Hierarchical Clustering (cont)Modified Average Linkage

d(ci , cj) =1

|ci | ∗ |cj |∗

∑zi∈ci

∑zj∈cj

d(zi , zj) + α ∗ (|ci |+ |cj |)

I Balancing Term: α ∗ (|ci |+ |cj |), where 0 ≤ α ≤ ∞I empirically gave best results with α = 0.01

Clustering with Balancing Term

I balancing term forces clustering to prefer small clustersI binary tree is more balanced → preferred browsing structure

Borth: KI2008 13 25.Sep.2008

Balanced Hierarchical Clustering (cont)Modified Average Linkage

d(ci , cj) =1

|ci | ∗ |cj |∗

∑zi∈ci

∑zj∈cj

d(zi , zj) + α ∗ (|ci |+ |cj |)

I Balancing Term: α ∗ (|ci |+ |cj |), where 0 ≤ α ≤ ∞I empirically gave best results with α = 0.01

Clustering with Balancing Term

I balancing term forces clustering to prefer small clustersI binary tree is more balanced → preferred browsing structure

Borth: KI2008 13 25.Sep.2008

Balanced Hierarchical Clustering (tree building)

Binary Tree Generation

I postprocessing of dendrogram structure into a binary tree

I neglecting similarity measurement given by dendrogram

I singelton clusters ci define leafs of binary tree

I merging points {ci , cj} define inner nodes of binary tree

Borth: KI2008 14 25.Sep.2008

Balanced Hierarchical Clustering (tree building)

Question: How to arrange keyframes within a cluster?

I Elements within a cluster do not have a spatial arrangement

I Additional processing to provide spatial organization→ Quad-Tree to Pyramid Mapping [Chen98]

I Our Approach: Use the already available cluster hierarchy→ Inorder Traversal of tree structure

Borth: KI2008 15 25.Sep.2008

Navidgator User Interface

Borth: KI2008 16 25.Sep.2008

Navidgator User Interface

User Interface

1. Selected Query

2. Similarity Cluster (if query refinement is needed)

3. Navigation Menu

4. Browser History

5. Restart BrowsingBorth: KI2008 17 25.Sep.2008

Live Demo

...do we have Internet access here?...

(demo also available at the DFKI-IUPR booth)

Borth: KI2008 18 25.Sep.2008

Summary

Typical Browsing Scenario

I annotation (tags) are not always available

I user has a rough visual concept of the query in mind

Navidgator Prototype

I browsing with “query by example” paradigm

I use of dynamic query refinement

I allows to zoom into different levels of detail

Further Work

I dealing with growing databasesI online clusteringI tree merging

Borth: KI2008 19 25.Sep.2008

References

I [Flickner95]: M. Flickner et al. Query by image and video content:the QBIC system. IEEE Computer Journal, 1995.

I [Lienhart01]: R. Lienhart. Reliable Transition Detection in Videos:A Survey and Practitioner’s Guide. IJIG, 2001.

I [Hammoud00]: R. Hammoud, R. Mohr. A Probabilistic Frameworkof Selecting Effective Key-Frames for Video Browsing and Indexing2000.

I [Tamura78]: H. Tamura, S. Mori, T. Yamawaki. Textural featurescorresponding to visual perception. IEEE-SMC, 2000.

I [Chen04]: J.Y. Chen, C. Taskiran, A. Albiol, E.J.Delp, C.A.Bouman. Vibe: A compressed video database structured for activebrowsing and search, IEEE Transactions on Multimedia 2004.

I [Chen98]: J. Chen, C.A. Bouman, J.C. Dalton. Similarity pyramidsfor browsing and organization of large image databases, SPIE 1998.

Borth: KI2008 20 25.Sep.2008

Questions?

Borth: KI2008 21 25.Sep.2008