mediaeval 2015 - eurecom @ sava2015: visual features for multimedia search
TRANSCRIPT
Motivation System overview Results
Motivation
QUERY
Textualdescription
Textual descriptionof visual content
Videocollection
TranscriptTextual terms for
extracted visual concepts
Connection?Connection?
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8
Motivation System overview Results
Motivation
QUERY
Textualdescription
Textual descriptionof visual content
Videocollection
TranscriptTextual terms for
extracted visual concepts
Connection?Connection?
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8
Motivation System overview Results
Motivation
QUERY
Textualdescription
Textual descriptionof visual content
Videocollection
TranscriptTextual terms for
extracted visual concepts
Connection?Connection?
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8
Motivation System overview Results
Motivation
QUERY
Textualdescription
Textual descriptionof visual content
Videocollection
TranscriptTextual terms for
extracted visual concepts
Connection?Connection?
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 2 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
ResultsegmentsRerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visualRun type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
ResultsegmentsRerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visualRun type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
ResultsegmentsRerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visualRun type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
Resultsegments
Rerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visualRun type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
Resultsegments
Rerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visual
Run type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
Resultsegments
Rerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visual
Run type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
Resultsegments
Rerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visual
Run type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
System overview
QUERY
Textualdescription
Textual descriptionof visual content
Resultsegments
Rerankedresultsegments
Terrier 4.0IR System
Videocollection
TranscriptTextual terms for
extracted visual concepts
Indexingof
segments
Informationrequest
Confidence scorecalculation
Confidence scoresfor visual concepts
in shots correspondingto retrieved segments
Run type: visual
Run type: text
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 3 / 8
Motivation System overview Results
Confidence score calculation
Calculate new confidence scores for theoverlapping terms from query information andvisual concepts extracted from the shotscorresponding to the retrieved video segments.
Query Information
Query terms(text/visual):ConfScore = 1.0
Word2Vec terms:ConfScore =Word2VecVectorDistance
w1 .. wn w1 .. wm
Informationbased onVisual
ConceptsExtraction
forRetrievedSegments
Visual Conceptsterms:
ConfScore=Visual Concepts
Extraction ConfScore
w1∑
ConfScoresComposition
NumOfOverlappingTerms ∗ 0.6
∑ConfScoresComposition
NumOfOverlappingTerms ∗ 0.2..wk
Word2Vec terms:ConfScore=
Visual ConceptsExtraction ConfScore
*Word2Vec Vector
Distance
w1∑
ConfScoresComposition
NumOfOverlappingTerms ∗ 0.2 -..wm
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 4 / 8
Motivation System overview Results
Evaluation Results: Precision @5,10,20
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 5 / 8
Motivation System overview Results
Evaluation Results: MAP metrics
Query Visual MAP MAP_bin MAP_tol MAiSPfields conceptstext none 0.5511 0.3529 0.3089 0.3431text Oxford 0.3196 0.2739 0.2053 0.2978visual Oxford 0.3368 0.2958 0.2293 0.3092text Leuven 0.3227 0.2801 0.2187 0.2958visual Leuven 0.3394 0.2970 0.2222 0.3117text CERTH 0.2295 0.2027 0.1554 0.1983visual CERTH 0.2624 0.2375 0.1822 0.2380
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 6 / 8
Motivation System overview Results
Evaluation Results: MAP metrics
Query Visual MAP MAP_bin MAP_tol MAiSPfields conceptstext none 0.5511 0.3529 0.3089 0.3431text Oxford 0.3196 0.2739 0.2053 0.2978visual Oxford 0.3368 0.2958 0.2293 0.3092text Leuven 0.3227 0.2801 0.2187 0.2958visual Leuven 0.3394 0.2970 0.2222 0.3117text CERTH 0.2295 0.2027 0.1554 0.1983visual CERTH 0.2624 0.2375 0.1822 0.2380
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 6 / 8
Motivation System overview Results
Results Discussion and Future Work
We achieve higher scores when establishing theconnection between visual concepts in thesegments and visual query definition, overtextual fields.
We have shown that the visual concepts definedfor the other task and extracted for thiscollection can be transferred to be used inthis task.
We did not outperform the baseline yet.Further tuning of the confidence scorescalculation and segments reranking is planned.
Eskevich (EURECOM) EURECOM @ SAVA2015 Sept 14, 2015 7 / 8