1

Hands and face tracking for VR applications

Adviser: Chih-Hung Lin

Date:2010/12/14

Speaker: Chin He Hsu

Javier Varona, Jose’ M. Buades, Francisco J. Perales Unidad de Gra´ficos y Visio´n por Ordenador, Dept. de matematiques i Informatica, Universitat de les Illes Balears (UIB), crta. Valldemossa km. 7,5, 07122 Palma de Mallorca, Spain

2

Outline1.Introduction

2. Hands and face tracking algorithm

3.Visualization using H-Anim

4.Conclusion and future work

3

1.Introduction

• In order to allow a user to navigate in a 3D-space

4

Interactive 3D-space

5

• system must detect a new user– entering into the system’s environment– analyse him to set parameters– tracking interesting regions

6

2. Hands and face tracking algorithm

• tracking problem lies in identifying both hands and face in each image – detect skin-colour pixels– data association algorithm

7

2.1. Skin-colour segmentation module

• skin-colour detection– necessary to model the actor’s skin-colour in a pre

vious step

8

skin-colour sample

• transform these pixels from the RGB-space to HSL-space– hue and saturation values contain the chroma infor

mation

• two main problems– human skin hue values are near the red colour– saturation values are near 0

9

10

skin-colour distribution

•

• Gaussian model

•

•

1{ ,..., ,..., }, ( , )i n i i iX x x x x h s

1

1 n

ii

x xn

1

1( ), ( ) '

n

i ii

x x x xn

1

2

1 1( is skin) exp( ( )| |( )' )

2(2 ) | |p x x x x x

11

12

Contours of skin-colour blobs after the connected components process

13

2.2. Data association module

•

s (p , w , )l l l l

p ( , ):position in the 2D image

w ( , ) : size of the limb in pixels

: angle in the 2D image plane

x yp p

w h

14

Next linear scheme of prediction

•

• that an extreme limb will maintain the same velocity

p( ) p( ) p( 1)

p( ) p( ) p( 1)

t t t

t t t

15

Set of hypothesis

•

•

{h }, 3,

Where

h (p , w , )

l

l l ll

H l

1{ ,..., ,..., }, : blob with labeli M iB b b b b

16

• define an approximation to the distance from the x image pixel to the hypothesis h

• t=x p

n=R t',

where

cos sinR=

sin cos

17

calculating the angle

• Normalized image pixel and the hypothesis centre

•

atan( / )x yn n

c ( , ),crossing point

cos

sin

x y

x

y

c c

c w

c h

18

19

• distance between an image pixel and a hypothesis

•

• if d( x ,h)<=0 , then x is inside the hypothesis h ,if d( x

,h)>0 , then x is outside the hypothesis h

(x,h) || n || || c ||d

20

• a blob with empty intersection with all hypotheses

• a pixel x of a blob is inside a limb hypothesis

x , min{ (x,h)} 0h H

b d

x , x= iff { (x,h)} 0B l d

21

Occlusion case solved using multiple labelling

22

23

2.3. 3D-point reconstruction

24

Complete procedure: color segmentation, data association and 3D reconstruction

25

3. Visualization using H-Anim

• H-Anim (humanoid animation)

• we use the H-Anim standard, this way we can collaborate with standard VRML (Virtual Reality Modeling Language ) models

26

3D position

27

4. Conclusion and future work

• proposed a new system

• human–computer interaction

• future work