livre: a video extension to the lire content-based image retrieval system

LIVRE: A VIDEO EXTENSION TO THE LIRE CONTENT-BASED IMAGE RETRIEVAL SYSTEM

Degree’s Final Project DissertationTelecommunications Engineering

Gabriel de Oliveira

Supervisors:

Assoc. Prof. Mathias LuxAssoc. Prof. Xavier Giró

Outline of the Thesis

1. Introductioni. Motivationii. Overview and previous work

2. Proposed solution: The LIvRE system i. Parsingii. Indexingiii.Retrieval

3. Validationi. Dataset

- Stanford I2V Newscasts datasetii. Experiments

- Quantitative evaluation- Qualitative evaluation - The thinking-aloud test.

4. Conclusions and Further Work

March – October 2015

Slide 2

MotivationGoal: To develop an all-in-one open source system for CBVR.

• Server side requeriments:• Fast• Scalable• Flexible• Automated

• User interface requeriments: • Fast• OS and device independent• Mobile

Slide 3

Introduction · Overview · LIvRE CBVR system · Validation · Conclusions

Overview and previous work

Slide 6

• Open source CBIR Library in Java• Apache Lucene core• Solr plugin

• Supports parsing, indexing and retrieval• Global and local descriptors• Web-based interface

[1] Mathias Lux. LIRE: Open source image retrieval in java. In Proceedings of the 21st ACM international conference on Multimedia, pages 843{846.ACM, 2013.


Developed solution: The LIvRE CBVR system

Slide 7

CBVR system - concept and requirements.

database


Developed solution: The LIvRE CBVR system

Slide 7

User sideServer side

LIvRE CBVR system architecture.


Block 1: Parsing

System Architecture - Parsing

Slide 8

1. Find videos in any given folder structure.

2. Extract keyframes from those videos.

3. Parse extracted keyframes with selected image descriptors.- Color Layout, Edge Histogram, JCD and PHOG

4. Generate XML Documents with the Feature Vectors.

Tools are provided to:


Block 2: Indexing

Fig. System Architecture

Slide 8

1. Find XML Documents containing the Feature Vectors

(generated from Parsing Block).

2. Upload XML documents to Solr.

3. Commit changes in Solr core.

Tools are provided to:

Fig. System Architecture - Indexing


Block 3: Retrieval

System Architecture - Retrieval

Slide 8

1. Image search field.

2. Settings.

User web-based interface input:

Web-based user interface input as displayed on small screen devices.


Block 3: Retrieval

Slide 8

1. Candidate videos displayed using HTML5.

2. Thumbnails with other similar frames.

3. Time refinement.

4. Video information.

User web-based results presentation:

System Architecture - Retrieval Retrieval results presentation for small screen devices


Block 3: Retrieval

Slide 8

1. Candidate videos displayed using HTML5.

2. Thumbnails with other similar frames.

3. Time refinement and ranking.

4. Video information.

User web-based results presentation:

Fig. System Architecture - Retrieval Fig. Retrieval results presentation for small screen devices


LIvRE CBVR system demo

Introduction · Overview · LIvRE CBVR system · Dataset · Experiments · Conclusions

Slide 9

ValidationStanford I2V Dataset

Freely available data set. Large (~1TB Video)

• 23,443 video clips• Average video duration: 2,65min.• Keyframes @1fps: 3,808,760• Video hours: 1,035h

Ground-truth• 78 queries

Some query images and video frames from the Stanford I2V dataset.

Slide 14


ValidationExperiments

LIvRE CBVR system tested with 2 different evaluation methods:

Slide 16

12

Quantitative evaluation

Qualitative evaluation(Thinking-aloud Test)


Quantitative study:

• Use ground-truth provided with the dataset for:

• Scene Retrieval evaluation (finding the right video).

• Time Refinement evaluation (finding the right moment of time at the right video).

Qualitative study:

• Web-based user interface.

• Thinking-aloud Test (offline).

• Participants are expert and non expert users.

• 4 Non-expert users.

• 2 Expert users. Slide 17

1

2

ValidationExperiments


Quantitative study

Slide 18

LIvRE

Quantitative study evaluation process from provided ground-truth


Quantitative study

Slide 18

1st Stage: Scene Retrieval


Quantitative study

Slide 18

2nd Stage: Temporal Refinement

Temporal Refinement results for 100k candidates


Qualitative study

Thinking-aloud Test

• Volunteer participants perform specific tasks with the web-based user interface.

• LIvRE CBVR system is running locally (offline) on the machine.

• Participants show their thoughts in loud-voice.

• Sessions are recorded and evaluated.

Slide 18


Qualitative studyThinking-aloud Test

Slide 19

Sample input query frames Screenshots from Thinking-aloud test 1


Timing results for 50K candidates (in miliseconds)

Conclusions and Future Work

- New CBVR System, LIvRE, was developed as an extension of LIRE. - LIvRE is now a branch of the LIRE Solr Project.

Slide 28

Future work: • Local image descriptors.• Integration of sound descriptors.• Simplified set-up and deployment.• Demo paper at ICMR 2016.• Add Video annotation tool.• Integration with computer vision / deep learning projects.


http://www.icmr2016.com/

http://www.icmr2016.com/

Thank you for your attentionDo you have any question?

8 October 2015

LIvRE: A Video Extension to the LIREContent-Based Image Retrieval System.

Gabriel de Oliveira

livre: a video extension to the lire content-based image retrieval system

Technology