JN 05/03/23 Knowledge Systems Lab

Facilitating User Interaction with Complex Systems via

Hand Gesture Recognition

MCIS DepartmentKnowledge Systems Laboratory

Jacksonville State University

Joshua R. New, Erion Hasanbelliu, andMario Aguilar

JN 05/03/23 Knowledge Systems Lab

Outline

• Motivation• System Architecture• Implementation Overview• Proposed Approach• Demonstration• Future Directions

JN 05/03/23 Knowledge Systems Lab

Motivation

• Gesturing is a natural form of communication

• Interaction problems with the mouse– Have to locate cursor– Hard for some to control (Parkinsons or

people on a train)– Limited forms of input from the mouse

JN 05/03/23 Knowledge Systems Lab

Motivation (2)

• Interaction Problems with the Virtual Reality Glove– Reliability– Always connected – Encumbrance

JN 05/03/23 Knowledge Systems Lab

System Architecture

StandardWeb Camera

Rendering

User InterfaceDisplay

Hand

M

ovem

ent

User

Gesture Recognition

System

ImageCapture

Update Object

Image Input

JN 05/03/23 Knowledge Systems Lab

Implementation Overview• System:

• 1.6 Ghz AMD Athlon• OpenCV and IPL libraries (from Intel)

• Input:• 640x480 video image• Hand calibration measure

• Output:• Rough estimate of centroid• Refined estimate of centroid• Number of fingers being held up• Manipulation of 3D skull in QT interface in response to

gesturing

JN 05/03/23 Knowledge Systems Lab

Implementation Overview (2)• Hand Calibration Measure:

• Max hand size in x and y orientations in # of pixels

JN 05/03/23 Knowledge Systems Lab

Implementation Overview (3)

Saturation Channel Extraction (HSL space):

Original ImageOriginal Image Hue

Lightness

Saturation

JN 05/03/23 Knowledge Systems Lab

Proposed Approach

JN 05/03/23 Knowledge Systems Lab

Proposed Approach (2)

JN 05/03/23 Knowledge Systems Lab

Proposed Approach (3)

)(*19.0 HandSizeYHandSizeXRadius

• The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses

• A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance)

• Total fingers is number of fingers minus 1 for the hand itself

JN 05/03/23 Knowledge Systems Lab

Proposed Approach (4)

System Runtime:

• Current time – 41 ms for one image from camera

• Processing Capability on 1.6 Ghz Athlon:• 24 fps

ProcessSteps

Time (ms)Athlon

XP 1900(1.6 Ghz)

1) Extract Sat Channel 9

2) Threshold 3

3) Connected Contour Fill

14

4) Centroid 2

5) Segment Hand From Arm

9

6) Refined Centroid 4

7) Count Number of Fingers

0

Total Time 41

JN 05/03/23 Knowledge Systems Lab

Demonstration

SystemConfiguration

SystemGUI Layout

JN 05/03/23 Knowledge Systems Lab

Demonstration (2)Gesture to Interaction Mapping

Number of Fingers:

2 – Roll Left3 – Roll Right

4 – Zoom In5 – Zoom Out

JN 05/03/23 Knowledge Systems Lab

Demonstration (3)

JN 05/03/23 Knowledge Systems Lab

Demonstration (4)

JN 05/03/23 Knowledge Systems Lab

Future Directions

• Optimization• Calibration Phase• Defining Hand Orientation• Learning System• Interface Extensions

For additional information, please visit http://ksl.jsu.edu.