Finding Time Together: Detection and Classification of Focused Interaction in Egocentric Video

Sophia Bano, Jianguo Zhang, Stephen J. McKenna, {s.bano, j.n.zhang, s.j.z.mckenna}@dundee.ac.uk

Computer Vision and Image Processing Group, Computing, School of Science and Engineering, University of Dundee, United Kingdom

International Conference on

Computer Vision 2017

1. Introduction

Focused Interaction (FI)

• Co-present individuals, having mutual focus of attention, interact by establishing face-to-face engagement and direct conversation [1]

Hypothesis

• Fusion of multimodal features can improve the overall FI detection

Challenges

• Face-to-face engagement often not maintained

• Conversational partner not always present in the video frame

• Varying illumination

• Varied scenes

Examples from our Focused Interaction Dataset

Existing methods

• Off-line processing of video clips or photo streams captured in quite constrained conditions and interacting people always in view [2, 3, 4]

2. Focused Interaction Detection using Multimodal Features

3. EvaluationFocused Interaction Dataset

• 19 egocentric videos (378 mins) captured using a shoulder-mounted GoPro Hero4 at 18 different locations and with 16 conversational partners

Observations

• Fusion of multimodal features is useful for discriminating no FI and FI (walk) when using SVM with RBF kernel

• Face track and VAD scores are significant for discriminating FI (non-walk)

Limitations

• Sound from nearby surroundings influenced the VAD

• Low illumination scenarios affected the face tracker

HOG: Histogram of Oriented Gradient

KLT: Kanade-Lucas-Tomasi Tracker

HOOF: Histogram of Oriented Optical Flow [5]

VAD: Voice Activity Detection [6]

References[1] E. Goffman. Encounters: Two studies in the sociology of interaction. Bobbs-Merrill, 1961.[2] M. Aghaei, M. Dimiccoli, P. Radeva. With whom do I interact? Detecting social interactions in egocentric photostreams. IEEE

ICPR, 2016.[3] S. Alletto, G. Serra, S. Calderara, F. Solera, R. Cucchiara. From ego to nos-vision: Detecting social relationships in first-person

views. IEEE CVPRW, 2014.[4] A. Fathi, J. K. Hodgins, J. M. Rehg. Social interactions: A first-person perspective. IEEE CVPR, 2012.[5] R. Chaudhry, A. Ravichandran, G. Hager, R. Vidal. Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear

dynamical systems for the recognition of human actions. IEEE CVPR, 2009.[6] M. Van Segbroeck, A. Tsiartas, S. Narayanan. A robust frontend for VAD: exploiting contextual, discriminative and spectral cues

of human voice. INTERSPEECH, 2013.

4. Conclusion and Future Work• Automatic online classification of Focused Interaction in continuous,

egocentric videos

• Multimodal features: face track, VAD and camera motion profile

• Best performance with multimodal feature fusion and SVM with RBF kernel

• Future work involves the use of recurrent neural network for classification and to extend this work to identify conversational partners

AcknowledgementsThis work is supported by the UKEngineering and Physical SciencesResearch Council (EPSRC) under grantEP/N014278/1: ACE-LP: AugmentingCommunication using EnvironmentalData to drive Language Prediction.

An outdoor night-time FI scenario with weak visual cues due to low illumination

Lin

ear

Ke

rne

lSV

M

https://ace-lp.ac.uk/

FI in which conversational partners are in the field of view of the camera

FI in which the conversational partner is no longer in the field of view as the interaction occurred while walking

RB

FK

ern

elS

VM

No Focused Interaction Focused Interaction (non-walk) Focused Interaction (walk)

Computer work FI-NW Searching for documents Walk- turn around-walk FI-NWCamera setup

HO

OF

(bin

s)

VAD score

0 20 40 60 80 100 120 140 160 180 200Time (sec)

0

0.5

1

VA

D s

core

s

0 20 40 60 80 100 120 140 160 180 2000

200

400

Tra

cke

r sc

ore

Camera motion feature (HOOF)

Input video

Video Stream

Audio Stream Audio-based

feature (VAD)

Face detection (HOG) and

tracking (KLT)

Feature concatenation

Temporal windowing

Classification using SVM

(Linear/RBF)

No FI

FI-NW(non-walk)

FI-W (walk)

M

T

V