Gestures without Libraries, Toolkits or Training: A $1 Recognize r for Use r Inte rface Prototypes

Authors: Jacob O. Wobbrock, Andrew D. Wilson, Yang LiPresente r: Spence r Ne lson

Introduction: Background

Gestures becoming more re levant with the spread of touchscreen devices

Gesture recognition has gene rally been the fie ld of AI/Machine learning expe rts, rathe r than HCI/UX experts

This limits the adoption of gestures in consumer products

Previous solutions include librarie s and toolkits, as we ll as “ad-hoc” recognize rs tuned to a prede fined gesture se t

Introduction: Objectives

Create and present a gesture recognition algorithm that is easy to implement in any language , without advanced knowledge

Compare the algorithm ($1) to more theore tically advanced recognize rs

Analyze types of gestures for use with this recognize r

Related Work

Previous work used specialized algorithms to make classifications (Hidden Markov Mode ls, neural ne tworks)

Complex me thods gene rally fall to be ing difficult to work with (debugging, training time , computation cost, ove rfitting)

Othe rs (ad-hoc) have poor scalability and do not allow additional gestures

Toolkit based e fforts work we ll, but only in the language(s) in which they are available

$1 Recognizer: Challenges

$1 Recognizer: Algorithm

$1 is a four step algorithm, best summarized as:

1. Resampling2. Rotate by “indicative angle”3. Scale and Translate4. Optimize best score with rotation re finement

Step 1: Resampling

Goal: e liminate variability due to movement speed during gesture

Takes original points and transforms it into a se t of N equidistant points

Calculate s the length of the path, then steps through each point until length/N-1 distance is cove red, then adds a new point (using linear inte rpolation for be tween point distances)

Step 2: Rotate on Indicative Angle

Goal: e liminate variability on angle of gesture

Calculate s the centroid

Rotate s angle from centroid to first point to 0 degrees

Step 3: Scale and Translate

Goal: Eliminate Variability in size of gesture

Se lects a re fe rence square for all gestures

Bounds the gesture by a rectangle , then scale s rectangle to match square

Translate s gesture to a re fe rence point - moves centroid to origin

Step 4: Optimize angle

Calculate s ave rage distance for C and T for each Ti (Euclidean distance ), converts closest value to a score be tween 0 and 1

Step 4

To optimize the score , the candidate must be rotated to find the optimal angle

Attempted three me thods:

Brute -force with prede fined accuracy (.5 degrees)

Hill Climbing

Golden Section Search

Optimization methods

Hill Climbing was be tte r for similar gestures, but worse for dissimilar gestures when compared to Golden Section Search

Hill Climbing - Maximum 90 ite rations, minimum 0

GSS - Maximum 10 ite rations, minimum 10

Limitations

Rotation, Scale , and Position invariant - Use ful, but also means that two gestures that are represent any combination of these ope rations from each othe r are mutually exclusive

E.g square vs. rectangle , up and down arrows

1-D gestures also require additional e ffort, as any variation in the unused dimension will be scaled significantly

Flexibility

$1 allows de fining of multiple template s by the same name , which allows for more variation in a gesture that all maps to the same gesture function

Evaluation

Used 10 subjects, ente ring gestures on an HP iPAQ h4355 Pocke t PC

Performed 10 se ts of 16 gestures at each slow, medium, and fast speeds

Evaluation

Results

Accuracy of 99.02% (.98% recognition e rrors)

Beats Rubine , and is comparable to DTW

Strengths

Algorithm with ~100 lines of code to implement

Achieves 99+% recognition

Comparable to othe r successful, and more complex, classifie rs

Allows designe rs to add arbitrary gestures at any time

Highly flexible and extendable

Weaknesses

Limits the gesture space somewhat, but these limitations are re lative ly small

Claims that library/package based attempts are le ss usable