bovdw: bag-of-visual-and-depth- words for gesture recognition all rights reserved hubpa© human pose...

BoVDW:Bag-of-Visual-and-Depth-Words for Gesture Recognition

All rights reserved HuBPA©

Human Pose Recovery and Behavior Analysis

Antonio Hernández-Vela1,2

Miguel Ángel Bautista1,2

Xavier Perez-Sala2,3

Victor Ponce Lopez1,2

Xavier Baro2,4

Oriol Pujol1,2

Cecilio Angulo3

Sergio Escalera1,2

1Dept. Applied Mathematics and Analysis, Universitat de Barcelona

2Computer Vision Center3CETpD, Universitat Politècnica de Catalunya

4EIMT, Universitat Oberta de Catalunya

1. Introduction2. Methodology3. Results4. Conclusion

BoVDW for Gesture Recognition



Outline




Bag of (Visual) WordsIntroduction Methodology Results Conclusion




Bag of Visual and Depth WordsIntroduction Methodology Results Conclusion

• In this work, we propose:

• Bag of Visual and Depth Words (BoVDW).

• A new depth descriptor.

• Comparison with state-of-the-art descriptors.

• Gesture recognition framework.




Standard pipelineIntroduction Methodology Results Conclusion

Point detection Point description Vocabulary & Representation Classification




Introduction Methodology Results Conclusion

Point detection Point description Vocab. & Represent. Classification

• Spatio-Temporal Interest Points (STIPs) [1]

[1] I. Laptev, "On Space-Time Interest Points", (2005) in International Journal of Computer Vision, vol 64, number 2/3, pp.107-123.

Temporal extension of the Harris operator






• Viewpoint Feature Histogram (VFH)[2]

[2] Rusu, R.B et al., "Fast 3D recognition and pose using the Viewpoint Feature Histogram", IROS, 2010

Figs. credit to [2] and Aitor Aldoma

Histogram of angles between surface normals and viewpoint direction

• Camera Roll Histogram (CRH)

Invariant to rotations in the roll axis of the camera!






• Concatenation of VFH and CRH






• Vocabulary building K-means clustering

• Spatio-temporal pyramids

…

Final histogram: Concatenation of 8+1

histograms





Point detection Point description Vocabulary Classification

• K-nearest neighbor classifier

• Distance function:

• Histogram intersection:




Chalearn datasetIntroduction Methodology Results Conclusion

• RGB-D video sequences.

• Organised in 20 batches:

• 47 sequences of 1-5 gestures each.

• Gestures from certain lexicon.

• Same actor.

•One-shot learning problem:

• Just 1 training sample for gesture.




ResultsIntroduction Methodology Results Conclusion

• Evaluation measurement: Levenshtein distance

Series1

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45

0.3452

0.4144

0.331400000000001

0.4021

0.306400000000001

VFHCRHVFHHOGHOFHOFHOG

Depth

RGB

Mean Levenshtein distance

(α= 1)

(α= 0)




Results (Late fusion)Introduction Methodology Results Conclusion

Late fusion approach Mean Lev. Dist.

HOGHOF/VFHCRH 0.2714

HOG/HOF/VFHCRH 0.2662

Batch number

Mea

n Le

vens

htei

n di

st.

(α= 0.8)




ConclusionIntroduction Methodology Results Conclusion

• We have presented:

• BoVDW approach for gesture recognition.

• VFHCRH, a new depth descriptor.

• Comparison of state-of-the-art descriptors .

• Analysis of Late fusion of RGB and Depth information.

• Future work:

• Test other methodologies for spatial coherence.

• Improve continuous gesture detection.

BoVDW:Bag-of-Visual-and-Depth-Words for Gesture Recognition



Antonio Hernández-Vela1,2

Miguel Ángel Bautista1,2

Xavier Perez-Sala2,3

Victor Ponce Lopez1,2

Xavier Baro2,4

Oriol Pujol1,2

Cecilio Angulo3

Sergio Escalera1,2

1Dept. Applied Mathematics and Analysis, Universitat de Barcelona2Computer Vision Center

3CETpD, Universitat Politècnica de Catalunya4EIMT, Universitat Oberta de Catalunya

Thank you!Questions?

bovdw: bag-of-visual-and-depth- words for gesture recognition all rights reserved hubpa© human pose...

Documents

gesture recognition

d recognition

depth words bovdw

conclusion bovdw

crh slide

catalunya slide

depth words

classification spatiotemporal