Slide 1

One-Shot Learning Gesture Recognition Students:Itay Hubara Amit Nishry Supervisor:Maayan Harel Gal-On

Slide 2

Introduction Reduced Problem Complete Problem ConclusionOutline 2

Slide 3

Background Gesture recognition is a strong upcoming field in computer vision Gesture recognition can be seen as a way for computers to begin to understand human body language 3

Slide 4

Motivation Existing Gesture recognition demand a long configuration and training Different Gestures are been solved using different approaches 4

Slide 5

Goals Learn and understand existing Gesture recognition algorithms. Compare different approaches Design Gesture recognition algorithm which reduces training time 5

Slide 6

Data The Data is compose from several set each contains: o Gesture vocabulary (learning set) which contain only one sample per gesture. o Test set which contain one or more gestures. Each of the sets has different vocabulary features such as large/small gesture hand/legs movement etc. 6

Slide 7

Data Train 7 Base gesture

Slide 8

Data Test 8 Multiple base gestures Large movements

Slide 9

Data - Test 9 Multiple base gestures Small movements

Slide 10

Challenges One shot learning - only one learning sample (unlike the common approach of multi class classification) Tests videos segmentation Same gesture can have different number of frames Each set has different features (small/big gestures) 10

Slide 11

Introduction Reduced Problem Complete Problem ConclusionOutline 11

Slide 12

Reduced Problem Assume that each of the test movies has only one gesture Goal: finding features space and distance function which have good separation of the features space 12

Slide 13

Problem Approach Classic machine learning problem Select Feature One Shot Match using similarity function 13

Slide 14

Features Motion Energy subtracting consecutive frames Space Quantization 14

Slide 15

Features Harris Corner Detector Find interest point in the difference image based on corner detection Space Time Interest Points Extend Harris to the time domain 15

Slide 16

Features Harris Corner Detector Find interest point in the image based on corner detection 16

Slide 17

Features Space Time Interest Points 17

Slide 18

Features 18 STIPHarris

Slide 19

Features Head Relative Interest Points 19

Slide 20

Features 20 Interest pointsHead Histogram

Slide 21

Distance Functions Good features space is defined not only by the features but also by the distance (similarity) function Different features need different distance functions 21

Slide 22

Principal Motion Using PCA Using principal component analysis (PCA), to find the main motion vectors. For test set - project feature onto each of train principals and evaluate similarity 22

Slide 23

Earth Moving Distance Given two sets of distribution, EMD will measure the minimum cost to shift dirt from one distribution to the other. 23

Slide 24

Perturbed Variations 24 Given two sets of distribution and predefined value of permitted variations optimally perturbs the distribution to best fit each other. Transportation problem under permitted variations constrain

Slide 25

Perturbed Variations 25

Slide 26

Levenshtein Distance Measure the difference between two sequences. Consider lengths and classification. 26

Slide 27

Results 27

Slide 28

Results 28 Top 10 Top 20

Slide 29

Results 29

Slide 30

Introduction Reduced Problem Complete Problem ConclusionOutline 30

Slide 31

Complete Problem Separate problems Basic Segmentation (equal/movement) Whole problem solving approach Moving Window Dynamic Time Warping (DTW) 31

Slide 32

Problem Approach Three different method to solve the problem: Basic Segmentation (equal/movement) Moving Window Dynamic Time Warping (DTW) 32

Slide 33

Moving Window Move a window along the test video. Assume each window frames has only one gesture Preform basic analysis as did before to and build the distance matrix 33

Slide 34

Moving Window 34

Slide 35

Moving Window After Sorting the Distance matrix we extract labels and cuts 35 Several other operation (such as smoothing) are done before extracting the final result

Slide 36

Dynamic Time Warping Create a state machine from train data: Module standing position Form standing position can move to start of base gestures Assume we can move forward, or stay in the same sate. For a given gesture find the best path along the sate machine 36

Slide 37

Dynamic Time Warping 37

Slide 38

Results 38

Slide 39

Results 39

Slide 40

Results 40

Slide 41

Results 41 Top 10 Top 20

Slide 42

Introduction Reduced Problem Complete Problem ConclusionOutline 42

Slide 43

Conclusions Each approach receive better results in different feature and similarity function Different algorithms has different strengths (segmentation\recognition) Segmentation require standing position model. 43

Slide 44

Conclusions 44 Pre-processing unsupervised algorithms help better representing the data. There is still allot left to do on the field

Slide 45

Future Work Try different models for the standing position to improve segmentation results Try combing DTW for segmentation and PCA for recognition. Use different unsupervised algorithms to better represent the data. 45

Slide 46

References Ivan Laptev, "On Space-Time Interest Points, 2005 Hugo Jair Escalantea and Isabelle Guyonb, "Principal motion: PCA-based reconstruction of motion histograms M.Harel, S.Manor, "The Perturbed Variation, NIPS 2012 Elizaveta Levina, Peter Bickel Department of Statistics, The EarthMovers Distance is the Mallows Distance: Some Insights from Statistics. Ofir Pele,Michael Werman, Fast and Robust Earth Movers Distances.2008 46