new approaches to interactive multimedia content retrieval from different sources
TRANSCRIPT
New Approaches to Interactive Multimedia Content Retrieval from
different SourcesJulián Moreno Schneider
LaBDA Group, Computer Science DepartmentUniversidad Carlos III de Madrid, Spain
Content
Motivation Background Objectives Proposal
Sports-domain Scenario and Validation Adaptation techniques and Validation Health-domain Scenario and Evaluation
Future directions Publications
Motivation (I) Multimedia content is increasing at staggering
rates
Devices and formats are very diverse and move away from traditional modes.
New Approaches to Interactive Multimedia Content Retrieval from different Sources
4
Motivation (II)
New Approaches to Interactive Multimedia Content Retrieval from different Sources
5
Motivation (III) Problem description
Current Limitation: multimedia elements retrieved by textual metadata
Users need access in a transparent, faster and easier way to many independent sources containing information in different formats (such as video, text, audio, images, graphics, etc.).
New Approaches to Interactive Multimedia Content Retrieval from different Sources
6
Motivation (IV) Clarifying the problem
Seeking the album of a song having the audio file and the artist’s name
+ ‘I want you back’ The Jackson 5
New Approaches to Interactive Multimedia Content Retrieval from different Sources
7
Content
Motivation Background Objectives Proposal
Formal model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation
Future directions Publications
New Approaches to Interactive Multimedia Content Retrieval from different Sources
8
Organization by the components Multimodal Information (Collections) Query Information Retrieval Approaches Retrieval Selection Fusion Interactions
Background (I)
New Approaches to Interactive Multimedia Content Retrieval from different Sources
9
Multimodal InformationBackground (II)
Image and Text
Image and Audio
Image and Video
TextandVideo
Multimodal
Federated Web Search Track
Jou et al. [2013]
New Approaches to Interactive Multimedia Content Retrieval from different Sources
10
Background (III) Query Modalities
Text (Monomodal)
Image (Monomodal)
Text and Image
Video and Image
Text and Audio
Multimodal
Yang et al. [2002]
de Vries [1998]
Marchand-Maillet et al. [2011]
New Approaches to Interactive Multimedia Content Retrieval from different Sources
11
Background (IV) Retrieval Approaches
Text Retrieval Low-level features Combined Indexes
Low-level featuresText-based(metadata)retrieval
Full text retrieval
Salton et al. [1975]
Romberg et al. [2012]
Lana-Serrano et al. [2011]
New Approaches to Interactive Multimedia Content Retrieval from different Sources
12
Background (V) Retrieval Engine Selection Strategy
Unknown StrategyBy Elements
By Query Terms
Probabilistic
Renaud and Azzopardi [2012]Demner-Fushman et al. [2012]Romberg et al. [2012].
Chernov et al. [2006]
Balog et al. [2012]
New Approaches to Interactive Multimedia Content Retrieval from different Sources
13
Background (VI) Result Fusion or Aggregation
Pre-RE fusion: Joint indexes (prior fusion)
Post-RE fusion Randomness Source or type Scores (unification)
Aggregated search
Arampatzis et al. [2011]Balog et al. [2012]Romberg et al. [2012]
New Approaches to Interactive Multimedia Content Retrieval from different Sources
14
Background (VII) Semantic Knowledge
Annotation-based Retrieval Multimedia Ontology Retrieval Combination of multimedia ontologies
Worring et al. [2007]Medina-Ramírez [2007] Castells et al. [2007]
New Approaches to Interactive Multimedia Content Retrieval from different Sources
15
Background (VIII) User Interactions
• Relevance Judgmentso Directo Indirect
Document browsing Clicks logging and analysisQuery history
• Log Analysis
• Surveys
• Dwell time• Eye tracking• Gestures, lip motion, speech and facial expression
New Approaches to Interactive Multimedia Content Retrieval from different Sources
16
Discussion Limitations
Handler strategy (specially adapted to the user experience)
Multimodality in query and results Multimodal semantically related collection Spanish
Out of the scope of this thesis Retrieval approaches Fusion algorithms Innovation in Interaction Logging
New Approaches to Interactive Multimedia Content Retrieval from different Sources
17
Content
Motivation Background Objectives Proposal
Formal model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation
Future directions Publications
New Approaches to Interactive Multimedia Content Retrieval from different Sources
18
Objectives (I)
1 • Propose a formal model to define
multimodal information retrieval (IMR) systems.
2• Develop two multimodal prototypes
based on the proposed model and evaluate them
3• Design and define techniques to
adapt MIR System based on user experience.
New Approaches to Interactive Multimedia Content Retrieval from different Sources
19
Objectives (II) Methodology
Formal Model
Interactions
Sports Domain Scenario
Adaptation techniques
Evaluation
Evaluation
1
2 3
4 5
Health Domain Scenario
6
New Approaches to Interactive Multimedia Content Retrieval from different Sources
20
Content
Motivation Background Objectives Proposal
Formal Model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation
Future directions Publications
New Approaches to Interactive Multimedia Content Retrieval from different Sources
21
Formal Model (I) Architecture is composed by the most
common components used in IR models.
New Approaches to Interactive Multimedia Content Retrieval from different Sources
22
Formal Model (II) Multimodal Information
{text, audio, video, image} SemanticRelations
Multimedia: isFrameOf(image17, video004)
Semantic: shows(image23,FC_Barcelona)
mentions(video12,FC_Barcelona)
New Approaches to Interactive Multimedia Content Retrieval from different Sources
23
Formal Model (III) Multimodal Query
RetrievalEngines (RE)
Example of RE:
New Approaches to Interactive Multimedia Content Retrieval from different Sources
24
Formal Model (IV) Handler
: set of rules
Example:
New Approaches to Interactive Multimedia Content Retrieval from different Sources
25
Formal Model (V) Results Fusion
Interactions
useridentifier, sessionidentifier, timestamp and additionalinformation
Visualizationusingexistingtechniques (clouds, lists, grouping, …)
New Approaches to Interactive Multimedia Content Retrieval from different Sources
26
Content
Motivation Background Objectives Proposal
Formal Model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation
Future directions Publications
New Approaches to Interactive Multimedia Content Retrieval from different Sources
27
Proposal: Sports-Domain Prototype (XII) Architecture
New Approaches to Interactive Multimedia Content Retrieval from different Sources
28
Proposal: Sports-Domain Prototype (VI) Buscamedia Collection
Developed in the framework of the Buscamedia Project
Sports Domain Multimodal documents
10000 Texts 350 Images 15 Videos
Recruited in October 2010 Semantically Related
New Approaches to Interactive Multimedia Content Retrieval from different Sources
29
Proposal: Sports-Domain Prototype (VII) Multimodal Query
Text, Audio and Text + Image
Información sobre el accidente de la foto +
New Approaches to Interactive Multimedia Content Retrieval from different Sources
30
Proposal: Sports-Domain Prototype (VIII) Retrieval Engines
Question Answering (QA), Full Text Search (FT), Ontology-based Search (ONT), Object Detection in Image (ODI), OCR in Image (OCRI), Audio Transcription (AT)
RE selection (Handler) Simple Approach Expert-defined rule-based approach
Question {QA,FT} Txt(short)+img {ONT,FT,{ODI,OCRI}}
New Approaches to Interactive Multimedia Content Retrieval from different Sources
31
Proposal: Sports-Domain Prototype (X) Fusion Strategy: Round-Robin Approach
New Approaches to Interactive Multimedia Content Retrieval from different Sources
32
Proposal: Sports-Domain Prototype (XI) User Interactions
Searches Documents Browsing Relevance Judgments Visualizations
New Approaches to Interactive Multimedia Content Retrieval from different Sources
33
Validation and Results (I) Objective
User Preferences Requested sources? Preferred modes? Preferred visualizations? More used query modes?
Expert-defined Rules Validation Comparison with Baseline (Full Text Search Engine)
Web Interface to test with users 2 months 235 users
New Approaches to Interactive Multimedia Content Retrieval from different Sources
34
Validation and Results (II) What query types are used?
981 queries: 239 predefined and 742 user-generated.
Short, long and question queries more often than concepts.
Sources ‘usage’ by query type.
Visualizations Answer List, Answer / Concept Cloud, Concept
Groups, Individual Document
New Approaches to Interactive Multimedia Content Retrieval from different Sources
35
Validation and Results (III) Baseline: logs from users IR
performance Mean
Average Precision (MAP)
Mean Reciprocal Rank (MRR)
R-Precision
New Approaches to Interactive Multimedia Content Retrieval from different Sources
36
Adapting IR Functionality (I)
Motivation Background Objectives Proposal
Formal Model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation
Future directions Publications
New Approaches to Interactive Multimedia Content Retrieval from different Sources
37
Adapting IR Functionality (II) Rule-Based MIR
(qmode=t, qtype=long) ont , qa , f t(qmode=t, qtype=question, qlength=14) qa , f t , ont(qmode=t, qtype=short, qlength=2, qentities=alonso) ont, qa, ft
New Approaches to Interactive Multimedia Content Retrieval from different Sources
38
Adapting IR Functionality (III)
Adaptation architecture
New Approaches to Interactive Multimedia Content Retrieval from different Sources
39
Adapting IR Functionality (IV) Classification Algorithms
Decision trees, multilayer perpectron and simple K-means
Query features Mode, type, length, number of entities, entities,
number of verbs, topic Ranking Scores Interaction-based
Lowest-position Average-position Iteration Mathematical
New Approaches to Interactive Multimedia Content Retrieval from different Sources
40
Validating IR Functionality Adaptation (I) Definition of SilverStandard
Example with 4 entity features:qmode=‘t’; qtype=‘short’; qlength=‘1’; qentities=‘Barcelona’ ft, ont, qa
Query: Barcelona
New Approaches to Interactive Multimedia Content Retrieval from different Sources
41
Validating IR Functionality Adaptation (II) The best combination is:
Query features: mtle Classification algorithms: J4.8 Ranking scores: Average Position Score
New Approaches to Interactive Multimedia Content Retrieval from different Sources
42
Monitoring health social media (I)
Motivation Background Objectives Proposal
Formal Model Sports-domain Scenario and Validation Adaptation techniques and validation Health-domain Scenario and Evaluation
Future directions Publications
New Approaches to Interactive Multimedia Content Retrieval from different Sources
43
Monitoring health social media (I) Online: http://
trendminer.daedalus.es/views/dashboard.php
New Approaches to Interactive Multimedia Content Retrieval from different Sources
44
Monitoring health social media (III) Annotation Pipeline
Documents Index
Twitter Saluspot
Relations Manager
Disambiguation
Medical Events Filter
Topics Analyzer
Morpho-syntactic Parser
Language Identification
Resources• DrugsGaz• DrugsATC• AdrsMedDRA• DiseasesUMLS• SpanishDrugEffectDB
Anot
atio
n Pi
pelin
e
New Approaches to Interactive Multimedia Content Retrieval from different Sources
45
Monitoring health social media (IV) IMIR System
New Approaches to Interactive Multimedia Content Retrieval from different Sources
46
Monitoring health social media (V) Results’ Combination
New Approaches to Interactive Multimedia Content Retrieval from different Sources
47
Health-domain Prototype Evaluation No User Evaluation NER & Relation Extraction Performance
NER
Relations extraction
Drugs R P F-mStrict 0,68 0,75 0,76Lenient
0,68 0,75 0,76
Effects R P F-mStrict 0,43 0,75 0,54Lenient 0,47 0,83 0,6
SpanishDrugEffectDB
Coocurrences
Wind. R P F-m R P F-m30 Strict 0,08 0,57 0,14 0,63 0,44 0,5230 Lenient 0,13 0,96 0,24 0,88 0,61 0,72
New Approaches to Interactive Multimedia Content Retrieval from different Sources
48
Conclusions (I) Formal model for IMIR systems
Two prototypes based on the formal model in two different scenarios: Sports domain Health social media
Scenario 1: Adaptation of multimodal IR Best result: NDCG=81,54% (2,81% gain)
Good RE performance Small improvements
New Approaches to Interactive Multimedia Content Retrieval from different Sources
49
Future Lines (I) Multimodal Query
New Approaches to Interactive Multimedia Content Retrieval from different Sources
50
Future Lines (II) Second Screen
New Approaches to Interactive Multimedia Content Retrieval from different Sources
51
Publications: Journals Bedmar, I. S., Martínez, P., Arenaz, R. R., and
Schneider, J. M. (2015). Exploring spanish health social media for detecting drug effects. BMC Medical Informatics and Decision Making, 15. 183, 216
Martínez, P., Fernández, J. L. M., Bedmar, I. S., Schneider, J. M., Luna, A., and Arenaz, R. R. (2015). Turning user generated health-related content into actionable knowledge through text analytics services. Computers in Industry.
New Approaches to Interactive Multimedia Content Retrieval from different Sources
52
Publications: Conferences SEPLN
Julián Moreno-Schneider, José Luis Martínez Fernández, Paloma Martínez, and Thierry Declerck. Prueba de Concepto de Expansión de Consultas basada en Ontologías de Dominio Financiero.
AMR Julián Moreno-Schneider, José Luis Martínez Fernández, and
Paloma Martínez. A Proof-of-Concept for Orthographic Named Entity Correction in Spanish Voice Queries.
González, M., Moreno Schneider, J., Martínez, J. L., and Martínez, P. (2013). An illustrated methodology for evaluating asr systems.
Schneider, J. M., Salazar, M. G., Martínez, P., and Fernández, J. L. M. (2011). Some experiments in evaluating asr systems applied to multimedia retrieval.
New Approaches to Interactive Multimedia Content Retrieval from different Sources
53
Publications: Conferences CLEF Conference
Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez, P. (2010a). Temporal information needs in respubliqa: an attempt to improve accuracy. the uc3m participation at clef 2010.
Vicente-Díez, M. T., De Pablo-Sanchez, C., Martínez, P., Moreno-Schneider, J., and Salazar, M. G. (2009). Are passages enough? the miracle team participation in qaclef2009.
SemEval Vicente-Díez, M. T., Moreno-Schneider, J., and Martínez,
P. (2010b). Uc3m system: Determining the extent, type and value of time expressions in tempeval-2.
New Approaches to Interactive Multimedia Content Retrieval from different Sources
54
Research and Development (R&D) projects Trendminer (FP7-ICT 287863)
Buscamedia (CEN-20091026)
Bravo (Búsqueda de Respuestas Avanzada Multimodal y Multilingüe) (TIN2007-67407-C03-01)
MAVIR (S-0505/TIC-0267) and MAVIR2 (S-2009/TIC-1542)
New Approaches to Interactive Multimedia Content Retrieval from different Sources
55
‘‘New Approaches to Interactive Multimedia Content Retrieval from different Sources’’
Julián Moreno [email protected]
Thank you for your attention