fusing semantic, observability, reliability and diversity ...xiaoyong/papers/mm08.ppt.pdffusing...
TRANSCRIPT
![Page 1: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/1.jpg)
Fusing Semantic, Observability, Reliability and Diversity of Concept Detectors
for Video Search
Xiao-Yong WEI, Chong-Wah NgoDept. of Computer Science
City University of Hong Kong
ACM Multimedia 2008, Vancouver, Canada
![Page 2: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/2.jpg)
Find shots of military personnel or soldiers together with military vehicles or weapons
Which concepts are related to this
query?
![Page 3: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/3.jpg)
Find shots of military personnel or soldierstogether with military vehicles or weapons
![Page 4: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/4.jpg)
explosion, flag, (entertainment)
thinkingobserving
armored car, armed person, tank
(e.g., IS-A relation)
(occur together)
Find shots of military personnel or soldierstogether with military vehicles or weapons
explosionMilitary
vehicle
soldiers
![Page 5: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/5.jpg)
Find shots of military personnel or soldierstogether with military vehicles or weapons
Military personnel, soldier, military vehicle,
weaponWhat else?
How to model different types of relations among
concepts?
![Page 6: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/6.jpg)
ObservabilitySpace
thinkingobserving
Semantic Space
![Page 7: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/7.jpg)
Outline
IntroductionSemantic Space vs. Observability SpaceConcept Selection and FusionExperimental ResultsConclusions
![Page 8: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/8.jpg)
Video Search vs. Semantic Gap
User Level
Multimedia Level
Query Query Query Query
Introduction - Background
![Page 9: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/9.jpg)
Video Search vs. Semantic Gap
User Level
Multimedia Level
Query Query Query
Text Image Motion Audio
Low-Level Representations
Low-Level Features
Query
Semantic Gap
Natural language
Machine computable
Introduction - Background
![Page 10: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/10.jpg)
Video Search vs. Semantic GapConcept-based Video Search
User Level
Multimedia Level
Query Query Query
Semantic G
ap
Text Image Motion Audio
Low-Level Representations
Concept Concept Concept …….
Low-Level Features
Query
Introduction - Background
![Page 11: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/11.jpg)
Video Search vs. Semantic GapConcept-based Video Search
User Level
Multimedia Level
Query Query Query
Semantic G
ap
Text Image Motion Audio
Low-Level Representations
Concept Concept Concept …….
High-L
evel Sem
antic
Low-Level Features
High-Level Concepts
Query
Introduction - Background
![Page 12: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/12.jpg)
User Level
Multimedia Level
Query Query Query
Semantic G
ap
Text Image Motion Audio
Low-Level Representations
Data Flow
Concept Concept Concept …….
High-L
evel Sem
anticG
eneral V
ocabularies
Low-Level Features
High-Level Concepts
Vocabularies Set (General Knowledge)
Query
Video Search vs. Semantic GapConcept-based Video Search
Introduction - Background
![Page 13: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/13.jpg)
User Level
Multimedia Level
Query Query Query
Semantic G
ap
Text Image Motion Audio
Low-Level Representations
Data Flow
Concept Concept Concept …….
High-L
evel Sem
anticG
eneral V
ocabularies
Low-Level Features
High-Level Concepts
Vocabularies Set (General Knowledge)
Query
Video Search vs. Semantic GapConcept-based Video Search
Introduction - Background
Crowd … Banner
protest
![Page 14: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/14.jpg)
User Level
Multimedia Level
Query Query Query
Semantic G
ap
Text Image Motion Audio
Low-Level Representations
Data Flow
Concept Concept Concept …….
High-L
evel Sem
anticG
eneral V
ocabularies
Low-Level Features
High-Level Concepts
Vocabularies Set (General Knowledge)
Query
How many and which detectors should be developed?
Critical questions to answer
![Page 15: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/15.jpg)
User Level
Multimedia Level
Query Query Query
Semantic G
ap
Text Image Motion Audio
Low-Level Representations
Data Flow
Concept Concept Concept …….
High-L
evel Sem
anticG
eneral V
ocabularies
Low-Level Features
High-Level Concepts
Vocabularies Set (General Knowledge)
Query
How many and which detectors should be developed?
Which concepts should be selected to describe the query?
Introduction - Background
![Page 16: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/16.jpg)
User Level
Multimedia Level
Query Query Query
Semantic G
ap
Text Image Motion Audio
Low-Level Representations
Data Flow
Concept Concept Concept …….
High-L
evel Sem
anticG
eneral V
ocabularies
Low-Level Features
High-Level Concepts
Vocabularies Set (General Knowledge)
Query ⎫⎪⎪⎪⎬⎪⎪⎪⎭⎫⎪⎪⎪⎬⎪⎪⎪⎭
How many and which detectors should be developed?
Which concepts should be selected to describe the query?
How to answer the query with selected concepts ?
Introduction - Background
![Page 17: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/17.jpg)
Large scale concept ontology for multimedia (LSCOM)MediaMill – 101TRECVID
How many and which concepts should be developed?
![Page 18: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/18.jpg)
Query-to-concept mapping
Which concepts should be selected?
![Page 19: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/19.jpg)
Query-to-concept mappingOntology reasoning: Resnik, JCN, WUP
Which concepts should be selected?
Object
militarypersonnel
soldier
militaryvehicle
tank armoredcar
ontologyQueries: … military personnel
or military vehicles
…
concepts
animal soldier
bus tank
armored car car
explosion
![Page 20: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/20.jpg)
Query-to-concept mappingOntology reasoning: Resnik, JCN, WUPComparing to text descriptions (definitions) of concepts
Which concepts should be selected?
descriptionsSoldier: is a …military personnel
Tank: is a …military vehicle
Armored car: is a …military vehicle
concepts
animal soldier
bus tank
armored car carBus: is a …
Queries: … military personnel
or military vehicles
…explosion
![Page 21: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/21.jpg)
Query-to-concept mappingOntology reasoning: Resnik, JCN, WUPComparing to text descriptions (definitions) of conceptsStatistic-based (e.g., by Internet)
Which concepts should be selected?
Explosion and military vehicle frequently occur together…
concepts
animal soldier
bus tankarmored car car
explosion
Queries: … military personnel
or military vehicles
…
![Page 22: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/22.jpg)
Query-to-concept mappingOntology reasoning: Resnik, JCN, WUPComparing to text descriptions (definitions) of conceptsStatistic-based (e.g., by Internet)Example-based
[C. G. M. Snoek, IEEE Trans. on Multimedia, 2007]
Vector-based (for image and video query examples)[John R. Smith, ICME’03]
Which concepts should be selected?
![Page 23: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/23.jpg)
Query-to-concept mappingOntology reasoning: Resnik, JCN, WUPComparing to text descriptions (definitions) of conceptsStatistic-based (e.g., by Internet)Example-based
[C. G. M. Snoek, IEEE Trans. on Multimedia, 2007]
Vector-based (for image and video query examples)[John R. Smith, ICME’03]
None of existing methods jointly considers semantics and observablity
Problem of concept selectionSemantics
Observability
![Page 24: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/24.jpg)
Most are simply using linear fusionSemanticsReliabilityObservability?Diversity?
How to answer the query with selected concepts?
person, face, police, newspaper
people-related
![Page 25: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/25.jpg)
Framework
Rel
evan
t sho
t lis
t
![Page 26: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/26.jpg)
Outline
IntroductionSemantic Space vs. Observability SpaceConcept Selection and FusionExperimental ResultsConclusions
![Page 27: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/27.jpg)
Construction of Semantic Space
Semantic Space
Ontology
![Page 28: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/28.jpg)
Ontology-enriched Semantic Space (OSS)- Global Consistency [X.-Y Wei, MM07]
Conventional Ontology Reasoning
weapon
gun tank armored car
Query: tank
Sim (tank, gun) = Sim (tank, armored car)
gun ? armored car ?
![Page 29: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/29.jpg)
OSS - Global Consistency
Conventional Ontology ReasoningLocal measure
weapon
gun
vehicle
tank armored car
![Page 30: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/30.jpg)
Construction of Semantic Space
gun tank …
gun
tank
armored car
weaponvehicle
Ontologyenriched
Semantic Space
weapon vehicle
weapon
vehicle
Minimize redundancy
Space transformation
WordNet
weapon
gun
vehicle
tank armored car
B2
gun
armoredcar
B1
tank
vehicle
weapon
![Page 31: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/31.jpg)
Construction of Observability Space
Observability Space
LSCOM annotation
![Page 32: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/32.jpg)
Construction of Observability Space
road boat …
road
boat
watercar
vehicle
Pearson product-moment (PM)
Observability Space
road water
road
water
Minimize redundancy
Space transformation
B2
B1
boat
car
vehicle
road
sky
LSCOM and Concept Annotation
Observability
![Page 33: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/33.jpg)
Solving problem of missing annotation
road boat car …
road
PM(car, vehicle)
boat
watercar
vehicle Vehicle is easy to be ignored by annotators when they are annotating a keyframes with car presented.[J.R. Kender, ICME07]
road water
road
water
Minimize redundancy
LSCOM and Concept Annotation
Observability
carvehicle
… …… …
When car and vehicle are represented by road and water, their observabilityrelation is also transferred through the two concepts. This relation does not rely on PM(car,vehicle) .
![Page 34: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/34.jpg)
Semantic Space vs. Observability Space
Semantic SpaceSemantic Space Observability SpaceObservability Space
Dendrograms created by SS and OS
![Page 35: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/35.jpg)
Outline
IntroductionSemantic Space vs. Observability SpaceConcept Selection and FusionExperimental ResultsConclusions
![Page 36: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/36.jpg)
Concept Selection
Anchor concepts: represent the semantic aspects of a queryBridge concepts: represent the context of a queryPositive concepts: concepts frequently co-occur with the target conceptNegative concepts: concepts never co-occur with the target concept
![Page 37: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/37.jpg)
Concept Selection– Query-to-Concept Semantic MappingSelecting Anchor concepts in SS
One concept to each query termRepresenting the semantic aspect of the query
v1
v3
v2
Concept vector
Concept vector Concept
vector
Vector of a query item
SS
Query: Find vehicles on the way
Vehicle roadSS
![Page 38: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/38.jpg)
Concept Selection– Detector Mining in OS
Selecting Bridge Concepts in OSForming subspaces to represent the context of the queryObservability Gap between Anchor Concepts
More specific concepts in the context (car)Latent concept not defined in SS (car_on_road)
Find vehicles on the way
SS
Vehicle road
OS
Vehicle
road
CarCar_on_road
water
boat
![Page 39: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/39.jpg)
Concept Selection– Mining positive and negative concepts in OS
Vehicle
Car
Road
Truck
Outer Space
Tennis
Positive
Negative
OSRoad
Carvehicle
![Page 40: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/40.jpg)
Concept Fusion– Reliability-based fusion
Vehicle Truck
Car
Road+
Outer Space
Tennis
Positive
Negative
OS
Enrich target concepts with its positive conceptsRefine target concept’s detector scores with its negative concepts (filters)
![Page 41: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/41.jpg)
Concept Fusion– Reliability-based fusion
Enrich target concepts with its positive conceptsRefine target concept’s detector scores with its negative concepts (filters)
vehicle
cartruckroad
+
Outer
space
tennis
=
+ =
![Page 42: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/42.jpg)
Enrich anchor concepts with bridge concepts
Multi-level Detector Fusion– Observability-based Fusion (in OS)
Query: Find vehicle on the way
Vehicle Road
+
Anchor concepts selection in SS
Car, Car_on_road
Bridge concepts selection in OS
![Page 43: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/43.jpg)
Multi-level Detector Fusion– Observability-based Fusion (in OS)
vehiclevehicle
vehiclevehicle
vehicle
car
Car on road
+
car +
car ==
car
+ =
car + =car + =
vehiclevehicle
vehiclevehicle
vehicle
Car on road
car
Car on road
car
Car on road
car
Car on road
car
![Page 44: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/44.jpg)
Answer the query with the reliability improved and observablity enriched anchor concepts
Multi-level Detector Fusion– Semantic-based Fusion (in SS)
Find vehicle on the way
Vehicle
Road
+
Semantic(vehilce, Vehicle)
Semantic(way, Road)
![Page 45: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/45.jpg)
Consider diversity of anchor concepts in concept fusion
person, face, police, newspaper
Multi-level Detector Fusion– Diversity-based Fusion
people-related
clustering
person
facepolice
newspaper
![Page 46: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/46.jpg)
Outline
IntroductionSemantic Space vs. Observability SpaceConcept Selection and FusionExperimental ResultsConclusions
![Page 47: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/47.jpg)
Datasets from TRECVID 2005 to 2007 with more than 285 hours videos and 72 queriesVIREO-374 detectors trained using TRECVID
2005 development setTop 1000 shots in returned list are evaluated by
using Average precision (AP)
Experimental Results– Dataset and Evaluation
![Page 48: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/48.jpg)
Concept selections by using SS and OSSS: 572 concepts, WordNet, WUP -> 366 dimensionsOS: 374 concepts, LSCOM, PM -> 253 dimensions
Experimental Results– Space Construction
Find shots of a person walking or riding a bicycle
Anchor concepts
![Page 49: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/49.jpg)
Experimental Results– Video Search Performance
Semantic-based fusion (S)Reliability-based fusion (R)Observability-based fusion (O)Diversity-based fusion (D)
0
0.05
0.1
0.15
0.2
0.25
0.3
AP-30 AP-50 AP-100 AP-1000
S-only
S+O
S+OR
S+ORD
Top-k performance on TV07 dataset
![Page 50: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/50.jpg)
Experimental Results– Video Search Performance
Performance based on Query TypesEvent – 31 queriesPerson or Thing (PT) – 19 queriesPlace – 14 queriesName Entity (NE) – 12 queries
0
5
10
15
20
25
30
35
Event PT Place NE
# of queries
![Page 51: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/51.jpg)
Experimental Results– Video Search Performance
Performance based on Query TypesEvent – 31 queriesPerson or Thing (PT) – 19 queriesPlace – 14 queriesName Entity (NE) – 12 queries
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Event PT Place NE
MAP
S-only
S+O
S+OR
S+ORD
Observability-based
![Page 52: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/52.jpg)
Experimental Results– Video Search Performance
Performance based on Query TypesEventPerson or Thing (PT)PlaceName Entity (NE)
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Event PT Place NE
MAP
S-only
S+O
S+OR
S+ORD
Diversity-based
![Page 53: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/53.jpg)
Experimental Results– Video Search Performance
Performance based on Query TypesEvent – 31 queriesPerson or Thing (PT) – 19 queriesPlace – 14 queriesName Entity (NE) – 12 queries
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Event PT Place NE
MAP
S-only
S+O
S+OR
S+ORD
Reliability-based
![Page 54: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/54.jpg)
Experimental Results– Comparison to Ontology
Reasoning
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
S+ORD OSS RES JCN WUP Lesk
TV07
TV06
TV05
![Page 55: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/55.jpg)
Experimental Results– Comparison to Ontology
Reasoning
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
S+ORD OSS RES JCN WUP Lesk
Event
PT
Place
NE
![Page 56: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/56.jpg)
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
TV05 runs
Experimental Results– Compare to TRECVID Submissions
Our runs are Visual-OnlyTV05
S-onlyS-O
S-ORS-ORD
![Page 57: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/57.jpg)
Experimental Results– Compare to TRECVID Submissions
Our runs are Visual-OnlyTV06TV07
0
0.02
0.04
0.06
0.08
0.1
TV06 runs
0
0.02
TV07 runs
0.04
0.06
0.08
0.1
S-only S-O S-OR S-ORD
![Page 58: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/58.jpg)
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
TV08 runs (Type A)
Experimental Results– Compare to TRECVID
SubmissionsOur runs are Visual-Only
TV08
S-onlyS-ORD
![Page 59: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/59.jpg)
Outline
IntroductionSemantic Space vs. Observability SpaceConcept Selection and FusionExperimental ResultsConclusions
![Page 60: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/60.jpg)
ConclusionTwo spaces complement to each other in concept
selectionSS provides model for semantic reasoningOS provides model for observability reasoning
observablity gap, bridge conceptsMulti-level concept fusion addresses different
aspects of detectorsSemanticsReliability (helpful for all types of queries)Observability (helpful for person+thing and place queries) Diversity (helpful for event related queries)
![Page 61: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/61.jpg)
Future work
Concept FrequencyCausalityMulti-modality fusion
![Page 62: Fusing Semantic, Observability, Reliability and Diversity ...xiaoyong/papers/mm08.ppt.pdfFusing Semantic, Observability, Reliability and Diversity of Concept Detectors for Video Search](https://reader034.vdocuments.net/reader034/viewer/2022052612/5f0fb0717e708231d4456672/html5/thumbnails/62.jpg)
ThanksThanks !
Presented by Xiao-Yong WEI