RESEARCH POSTER PRESENTATION DESIGN © 2015

www.PosterPresentations.com

Hao Zhang, Lei Pang, Yi-Jie Lu, Chong-Wah NgoVideo Retrieval Group (VIREO), City University of Hong Kong

http://vireo.cs.cityu.edu.hk

VIREO@TRECVID 2016: Multimedia Event Detection, Ad-hoc Video Search,Video to Text Description

Multimedia Event Detection (MED) Video to Text Description (VTT)

Motivation

Framework

Experiments

Conv 1Max Pool

Conv 2Max Pool

Conv 3Conv 4Conv 5

Extarction Scaling

Scaling

RoI

RoI

Feature Map ofScaled RoI

Feature Map

Feature Map of RoI

Scaled Feature Map of RoI

Image

RoI

Dee

p F

eatu

re E

xtra

ctio

n

Conv 1Max Pool

Conv 2Max Pool

Conv 3Conv 4Conv 5

COS Similarity Only around 40% of Regional Information left

Discriminatory power

of deep features consistently improves

0.4

0.45

0.5

0.55

0.6

0.65

0.7

conv1 conv2 conv3 conv4 conv5

AlexNet

cos-sim

0.4 0.5 0.6 0.7 0.8 0.9

conv1-1conv1-2conv2-1conv2-2conv3-1conv3-2conv3-3conv3-4conv4-1conv4-2conv4-3conv4-4conv5-1conv5-2conv5-3conv5-4

VGG-19-Net

cos-sim

0.4

0.5

0.6

0.7

0.8

0.9

1

conv1Res2ARes2BRes2CRes3ARes3BRes3CRes3DRes4ARes4BRes4CRes4DRes4ERes4FRes5ARes5BRes5C

Deep-ResNet-50

cos-sim

Video Tim

eFrame

Selective search

Candidate Objects

Temporal line

Spatial & temporal features clustering

Primary Scenes

Regional objects

Deep features K-means & VLAD

Video Tim

e

Frame People

Concept Bank

Baby Cake People

Time

Sequential Features

Average pooling

Chi2 SVM

Concept Bank Vocabulary

Object-VLAD Spatial Pyramid Pooling VLAD Details

Deep Feature MapExtraction

Feature Map7 X 7

Spatial Pyramid Pooling

7 X 7 6 X 6 5 X 5 2 X 2

50

de

scri

pto

rs

VLAD

Max pool filter:

50

de

scri

pto

rs

Concept-Bank VS Object-VLAD

CNN-VLAD

CNN-VLAD VS Object-VLAD

0

5

10

15

20

25

30

35

40

45

Team

2

VIR

EO

Team

3

Team

4

Team

5

Team

6

Team

7

Team

8

Team

9

Team

10

Team

11

MED16-EvalSub (MinfAP200%)

25

27

29

31

33

35

37

39

41

Team2 VIREO Team3 Team4

MED16-EvalFull (MinfAP200%)

Evaluation: PS-10Ex (Object-VLAD + Concept-Bank)

Evaluation: AH-10Ex

(Object-VLAD + Concept-Bank)

25

30

35

40

45

50

Team2 VIREO Team3 Team4

MED16-EvalFull (MinfAP200%)

0

5

10

15

20

25

30

35

40

45

50MED16-EvalSub (MinfAP200%)

Evaluation: PS-100Ex (Object-VLAD)

Pri

mary

Ru

ns

Pri

mary

Ru

ns

Pri

mary

Ru

ns

Framework

Concept-based Text Description Matching

Stacked Attention Model

Pool5 Feature Map

Image embedding layer

Some guy is playing Frisbee with a dog

Text CNN Layer

Soft-m

ax

Soft-m

ax

Multi-modal embedding

layer

Multi-modal embedding

layer

tanh

hA P1

vT u

hA

(2)

PI

(2)

vI

u(2)

S(vI, vT )

: feature for each region

1st Attention Layer 2nd Attention Layer

Objective-Function

Training Dataset Deep Features

Experiments

Ad-hoc Video Search (AVS)

OnlineOnline

OfflineOffline

Semantic Query Generation

Event QueryCleaning an appliance Tokens

NLP Parser

Concept matching and selection

Shot Corpus

$

€

₤

ƒ

$

￥

Concept Bank

Shot Ranking

Concept-based Video Representation

cleaning

refrigeratordishwasherstovemicrowave

spray bottle

kitchen

Semantic Query

Vocabulary

Framework

Performance for different concept settings

in Video-Search 2008 dataset (MinfAP%)

Experiments

Performance for different concept settings in

AVS 2016 evaluation dataset (MinfAP%)

Performance with and without highlighted concept

ranking (HCR)

Highlighted Concept Reranking (HCR)

guitar (#RC)

acoustic guitar (#I-1K)

electric guitar (#I-1K)

daytime outdoor (#SIN)

outdoor (#SIN)

outdoor (#RC)

a person playing guitar outdoors

Rerank 1

500

Without reranking With reranking