new approaches to interactive multimedia content retrieval from different sources

New Approaches to Interactive Multimedia Content Retrieval from

different SourcesJulián Moreno Schneider

LaBDA Group, Computer Science DepartmentUniversidad Carlos III de Madrid, Spain

jmschnei@inf.uc3m.es

Content

Motivation Background Objectives Proposal

Sports-domain Scenario and Validation Adaptation techniques and Validation Health-domain Scenario and Evaluation

Future directions Publications

Motivation (I) Multimedia content is increasing at staggering

Devices and formats are very diverse and move away from traditional modes.

New Approaches to Interactive Multimedia Content Retrieval from different Sources

Motivation (II)

Motivation (III) Problem description

Current Limitation: multimedia elements retrieved by textual metadata

Users need access in a transparent, faster and easier way to many independent sources containing information in different formats (such as video, text, audio, images, graphics, etc.).

Motivation (IV) Clarifying the problem

Seeking the album of a song having the audio file and the artist’s name

+ ‘I want you back’ The Jackson 5

Content

Formal model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation

Organization by the components Multimodal Information (Collections) Query Information Retrieval Approaches Retrieval Selection Fusion Interactions

Background (I)

Multimodal InformationBackground (II)

Image and Text

Image and Audio

Image and Video

TextandVideo

Multimodal

Federated Web Search Track

Jou et al. [2013]

Background (III) Query Modalities

Text (Monomodal)

Image (Monomodal)

Text and Image

Video and Image

Text and Audio

Multimodal

Yang et al. [2002]

de Vries [1998]

Marchand-Maillet et al. [2011]

Background (IV) Retrieval Approaches

Text Retrieval Low-level features Combined Indexes

Low-level featuresText-based(metadata)retrieval

Full text retrieval

Salton et al. [1975]

Romberg et al. [2012]

Lana-Serrano et al. [2011]

Background (V) Retrieval Engine Selection Strategy

Unknown StrategyBy Elements

By Query Terms

Probabilistic

Renaud and Azzopardi [2012]Demner-Fushman et al. [2012]Romberg et al. [2012].

Chernov et al. [2006]

Balog et al. [2012]

Background (VI) Result Fusion or Aggregation

Pre-RE fusion: Joint indexes (prior fusion)

Post-RE fusion Randomness Source or type Scores (unification)

Aggregated search

Arampatzis et al. [2011]Balog et al. [2012]Romberg et al. [2012]

Background (VII) Semantic Knowledge

Annotation-based Retrieval Multimedia Ontology Retrieval Combination of multimedia ontologies

Worring et al. [2007]Medina-Ramírez [2007] Castells et al. [2007]

Background (VIII) User Interactions

• Relevance Judgmentso Directo Indirect

Document browsing Clicks logging and analysisQuery history

• Log Analysis

• Surveys

• Dwell time• Eye tracking• Gestures, lip motion, speech and facial expression

Discussion Limitations

Handler strategy (specially adapted to the user experience)

Multimodality in query and results Multimodal semantically related collection Spanish

Out of the scope of this thesis Retrieval approaches Fusion algorithms Innovation in Interaction Logging

Content

Formal model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation

Objectives (I)

1 • Propose a formal model to define

multimodal information retrieval (IMR) systems.

2• Develop two multimodal prototypes

based on the proposed model and evaluate them

3• Design and define techniques to

adapt MIR System based on user experience.

Objectives (II) Methodology

Formal Model

Interactions

Sports Domain Scenario

Adaptation techniques

Evaluation

Health Domain Scenario

Content

Formal Model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation

Formal Model (I) Architecture is composed by the most

common components used in IR models.

Formal Model (II) Multimodal Information

{text, audio, video, image} SemanticRelations

Multimedia: isFrameOf(image17, video004)

Semantic: shows(image23,FC_Barcelona)

mentions(video12,FC_Barcelona)

Formal Model (III) Multimodal Query

RetrievalEngines (RE)

Example of RE:

Formal Model (IV) Handler

: set of rules

Example:

Formal Model (V) Results Fusion

Interactions

useridentifier, sessionidentifier, timestamp and additionalinformation

Visualizationusingexistingtechniques (clouds, lists, grouping, …)

Content

Proposal: Sports-Domain Prototype (XII) Architecture

Proposal: Sports-Domain Prototype (VI) Buscamedia Collection

Developed in the framework of the Buscamedia Project

Sports Domain Multimodal documents

10000 Texts 350 Images 15 Videos

Recruited in October 2010 Semantically Related

Proposal: Sports-Domain Prototype (VII) Multimodal Query

Text, Audio and Text + Image

Información sobre el accidente de la foto +

Proposal: Sports-Domain Prototype (VIII) Retrieval Engines

Question Answering (QA), Full Text Search (FT), Ontology-based Search (ONT), Object Detection in Image (ODI), OCR in Image (OCRI), Audio Transcription (AT)

RE selection (Handler) Simple Approach Expert-defined rule-based approach

Question {QA,FT} Txt(short)+img {ONT,FT,{ODI,OCRI}}

Proposal: Sports-Domain Prototype (X) Fusion Strategy: Round-Robin Approach

Proposal: Sports-Domain Prototype (XI) User Interactions

Searches Documents Browsing Relevance Judgments Visualizations

Validation and Results (I) Objective

User Preferences Requested sources? Preferred modes? Preferred visualizations? More used query modes?

Expert-defined Rules Validation Comparison with Baseline (Full Text Search Engine)

Web Interface to test with users 2 months 235 users

Validation and Results (II) What query types are used?

981 queries: 239 predefined and 742 user-generated.

Short, long and question queries more often than concepts.

Sources ‘usage’ by query type.

Visualizations Answer List, Answer / Concept Cloud, Concept

Groups, Individual Document

Validation and Results (III) Baseline: logs from users IR

performance Mean

Average Precision (MAP)

Mean Reciprocal Rank (MRR)

R-Precision

Adapting IR Functionality (I)

Adapting IR Functionality (II) Rule-Based MIR

(qmode=t, qtype=long) ont , qa , f t(qmode=t, qtype=question, qlength=14) qa , f t , ont(qmode=t, qtype=short, qlength=2, qentities=alonso) ont, qa, ft

Adapting IR Functionality (III)

Adaptation architecture

Adapting IR Functionality (IV) Classification Algorithms

Decision trees, multilayer perpectron and simple K-means

Query features Mode, type, length, number of entities, entities,

number of verbs, topic Ranking Scores Interaction-based

Lowest-position Average-position Iteration Mathematical

Validating IR Functionality Adaptation (I) Definition of SilverStandard

Example with 4 entity features:qmode=‘t’; qtype=‘short’; qlength=‘1’; qentities=‘Barcelona’ ft, ont, qa

Query: Barcelona

Validating IR Functionality Adaptation (II) The best combination is:

Query features: mtle Classification algorithms: J4.8 Ranking scores: Average Position Score

Monitoring health social media (I)

Monitoring health social media (I) Online: http://

trendminer.daedalus.es/views/dashboard.php

Monitoring health social media (III) Annotation Pipeline

Documents Index

Twitter Saluspot

Relations Manager

Disambiguation

Medical Events Filter

Topics Analyzer

Morpho-syntactic Parser

Language Identification

Resources• DrugsGaz• DrugsATC• AdrsMedDRA• DiseasesUMLS• SpanishDrugEffectDB

Monitoring health social media (IV) IMIR System

Monitoring health social media (V) Results’ Combination

Health-domain Prototype Evaluation No User Evaluation NER & Relation Extraction Performance

Relations extraction

Drugs R P F-mStrict 0,68 0,75 0,76Lenient

0,68 0,75 0,76

Effects R P F-mStrict 0,43 0,75 0,54Lenient 0,47 0,83 0,6

SpanishDrugEffectDB

Coocurrences

Wind. R P F-m R P F-m30 Strict 0,08 0,57 0,14 0,63 0,44 0,5230 Lenient 0,13 0,96 0,24 0,88 0,61 0,72

Conclusions (I) Formal model for IMIR systems

Two prototypes based on the formal model in two different scenarios: Sports domain Health social media

Scenario 1: Adaptation of multimodal IR Best result: NDCG=81,54% (2,81% gain)

Good RE performance Small improvements

Future Lines (I) Multimodal Query

Future Lines (II) Second Screen

Publications: Journals Bedmar, I. S., Martínez, P., Arenaz, R. R., and

Schneider, J. M. (2015). Exploring spanish health social media for detecting drug effects. BMC Medical Informatics and Decision Making, 15. 183, 216

Martínez, P., Fernández, J. L. M., Bedmar, I. S., Schneider, J. M., Luna, A., and Arenaz, R. R. (2015). Turning user generated health-related content into actionable knowledge through text analytics services. Computers in Industry.

Publications: Conferences SEPLN

Julián Moreno-Schneider, José Luis Martínez Fernández, Paloma Martínez, and Thierry Declerck. Prueba de Concepto de Expansión de Consultas basada en Ontologías de Dominio Financiero.

AMR Julián Moreno-Schneider, José Luis Martínez Fernández, and

Paloma Martínez. A Proof-of-Concept for Orthographic Named Entity Correction in Spanish Voice Queries.

González, M., Moreno Schneider, J., Martínez, J. L., and Martínez, P. (2013). An illustrated methodology for evaluating asr systems.

Schneider, J. M., Salazar, M. G., Martínez, P., and Fernández, J. L. M. (2011). Some experiments in evaluating asr systems applied to multimedia retrieval.

Publications: Conferences CLEF Conference

Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez, P. (2010a). Temporal information needs in respubliqa: an attempt to improve accuracy. the uc3m participation at clef 2010.

Vicente-Díez, M. T., De Pablo-Sanchez, C., Martínez, P., Moreno-Schneider, J., and Salazar, M. G. (2009). Are passages enough? the miracle team participation in qaclef2009.

SemEval Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez,

P. (2010b). Uc3m system: Determining the extent, type and value of time expressions in tempeval-2.

Research and Development (R&D) projects Trendminer (FP7-ICT 287863)

Buscamedia (CEN-20091026)

Bravo (Búsqueda de Respuestas Avanzada Multimodal y Multilingüe) (TIN2007-67407-C03-01)

MAVIR (S-0505/TIC-0267) and MAVIR2 (S-2009/TIC-1542)

‘‘New Approaches to Interactive Multimedia Content Retrieval from different Sources’’

Julián Moreno Schneiderjmschnei@inf.uc3m.es

Thank you for your attention

new approaches to interactive multimedia content retrieval from different sources

Technology

interactive learning multimedia

interactive search in image retrieval: a survey · keywords...

interoperable and unified multimedia...

creative computing 2: interactive multimedia volume 2...

drilldown: interactive retrieval of complex scenes using...

balancing the power of multimedia information...

interactive multimedia technology

interactive multimedia cd

multimedia information storage and retrieval techniques ......

interactive multimedia teaching

multimedia retrieval. outline audio retrieval spoken...

multimedia information retrieval · multimedia information...

dl-media: an ontology mediated multimedia information...

(multimedia retrieval) -...

content-based multimedia information retrieval: … acm...

semantic multimedia database information retrieval

interactive multimedia i

integrated multimedia indexing and retrieval

cse 635 multimedia information retrieval

multimedia retrieval (dcu 2016)