a robust method of detecting hand gestures using depth sensors
DESCRIPTION
A Robust Method of Detecting Hand Gestures Using Depth Sensors. Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on. Outline. Introduction Related Works The Proposed Method Experimental Results - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/1.jpg)
1
A Robust Method of Detecting Hand Gestures Using
Depth Sensors
Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang
Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on
![Page 2: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/2.jpg)
2
Outline
IntroductionRelated WorksThe Proposed MethodExperimental ResultsConclusion
![Page 3: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/4.jpg)
4
Introduction
In human-computer interaction(HCI) system, recognizing hand and finger gestures are significant. Medical system, computer games, and human-robot
Depth-sensing camera(Kinect, Xtion) add a dimension to increase accuracy.
Goal: detect hand gestures with color and depth information
![Page 5: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/5.jpg)
5
Related Works
![Page 6: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/6.jpg)
6
Related Works
Body [4][5] V.S. HandHand
Superiority: simple Inferiority: small scale, low resolution Strict condition: cluttered background, lighting
variation
![Page 7: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/7.jpg)
7
Related Works
Hand gesture recognition Only color[12]
Data glove[7]
Training process[9][10]
Earth Mover’s Distance(EMD)[11]
![Page 8: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/8.jpg)
8
References [7] R. M. Satava, “Virtual reality surgical simulator,” Surgical Endoscopy, vol. 7, pp.
203–205, 1993. [9] C. Keskin, F. Kirac, Y. Kara, and L. Akarun, “Real time hand pose estimation
using depth sensors,” in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, nov. 2011.
[10] P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, and V. Athitsos, “Comparing gesture recognition accuracy using color and depth information,” in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments, ser. PETRA ’11. New York, NY, USA: ACM, 2011, pp. 20:1–20:7.
[11] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera,” in Proceedings of the 19th ACM international conference on Multimedia, ser. MM ’11. New York, NY, USA: ACM, 2011, pp. 1093–1096.
[12] A. Argyros and M. Lourakis, “Real-time tracking of multiple skincolored objects with a possibly moving camera,” in Computer Vision -ECCV 2004, ser. Lecture Notes in Computer Science. Springer Berlin/ Heidelberg, 2004, vol. 3023, pp. 368–379.
![Page 9: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/9.jpg)
9
The Proposed Method
![Page 10: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/10.jpg)
10
The Proposed Method
Gesture Representation
Finger RecognitionFind convex hull Detect fingertip and direction
Hand SegmentationFind hands through color
Separate hands by k-means
Find palm center
![Page 11: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/11.jpg)
11
Hand Segmentation (I)
Find hands through color Train skin-color[12], detect face[15], image filtering[16],
color threshold L*a*b color space
b Operate AND on the two images Minimum depth (10cm)
![Page 12: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/12.jpg)
12
Hand Segmentation (I)--Find hands through color
RGB image Depth image L*a*b color space where b = 2
L*a*b color space where b = 3
Skin color images after ANDoperation
Binary image of hand segmentation
![Page 13: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/13.jpg)
13
Hand Segmentation (II)
Separate hands by k-means k=2 Assignment:
Update:
Threshold of distance between 2 clusters
![Page 14: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/14.jpg)
14
Hand Segmentation (II)--Separate Hands By K-means
![Page 15: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/15.jpg)
15
Hand Segmentation (III)
Find palm center Inscribed circle Minimum inner distance
Maximum element of inner distances set
![Page 16: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/16.jpg)
16
Finger Recognition (I)
Find convex hull Graham’s scan algorithm
P: the lowest y-coordinate Sort in increasing order of angle Point to point is left/right turn Left-turn: O ; right-turn: X
![Page 17: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/17.jpg)
17
Finger Recognition (II)
Detect fingertip and direction Fingers are long and narrow Find an isosceles triangle with V
V: Every vertex on the convex hull Set a maximum threshold to the vertex angle The direction vector is paralleled with the median
length of an isosceles triangle
![Page 18: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/18.jpg)
18
Finger RecognitionBlue point : cluster centroid Green point : palm centerRed points : fingertipsYellow curves : hand contourLong lines : finger directions Structures around the hand : convex hull
![Page 19: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/19.jpg)
19
The Proposed Method
Gesture Representation
Finger RecognitionFind convex hull Detect fingertip and direction
Hand SegmentationFind hands through color
Separate hands by k-means
Find palm center
![Page 20: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/20.jpg)
20
Gesture Representation
All information about hands Palm center location Finger number Fingertips location Finger direction vectors
Gestures Rock-paper-scissors game Drag images Grasping, releasing
![Page 21: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/21.jpg)
21
Experimental Results
![Page 22: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/22.jpg)
22
Experimental Results
Use Kinect as input of depth and color images The detection successful rate can reach 95%.No matter the hand is horizontally or vertically
placed.
![Page 23: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/23.jpg)
23
![Page 24: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/24.jpg)
24
Experimental Results
![Page 25: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/25.jpg)
25
Experimental Results--Shadow Puppetry
![Page 26: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/26.jpg)
26
Conclusion
![Page 27: A Robust Method of Detecting Hand Gestures Using Depth Sensors](https://reader036.vdocuments.net/reader036/viewer/2022062310/56816597550346895dd86c82/html5/thumbnails/27.jpg)
27
Conclusion
Present a new method to detect hands’ positions and gestures
NO training, NO database Future works
Set a threshold to the distance between the palm center and the fingers
Add additional sensor devices to overcome no palm detection
Shadow Puppetry project