uwave: accelerometer-based personalized gesture
TRANSCRIPT
uWave: Accelerometer-based Personalized Gesture Recognition and Its ApplicationsRecognition and Its Applications
Jiayang Liu, Zhen Wang, and Lin Zhong Jehan Wickramasuriya and Venu Vasudevany g , g, gDepartment. Of Electrical Computer
Engineering Rice University, Houston TX 77005
[email protected], [email protected], lzhong@rice edu
yPervasive Platforms & Architectures Lab
Applications & Software Research Center, Motorola Labs
{jehan,venu}@motorola.com [email protected]
Matthew [email protected] 525w Mobile Computing
Introduction to uWaveIntroduction to uWave
• “efficient recognition algorithm”efficient recognition algorithm– Focus on Gestures/Physical Manipulation– User-dependent Gesture Recognitionp g– Dynamic Time Warping
• Goal: Support efficient personalized gesture recognition on a wide range of devices
2
Worcester Polytechnic Institute
2
Related WorkRelated Work
• Computer Vision/Vision BasedComputer Vision/Vision Based Techniques– Translates a “gesture” into “handwriting”– Fundamentally Limited by Hardware
Requirements• Hidden Markov Models
– Require extensive training data to be effectiveR i k l d f th b l i d– Require knowledge of the vocabulary in order to configure the model
3
Worcester Polytechnic Institute
3
Related WorkRelated Work
• Dynamic Time Warping (DTW)Dynamic Time Warping (DTW)– Algorithm for measuring similarity between
two sequences which may vary in time or speed
– Allows a computer to find an optimal match between two given sequences with certainbetween two given sequences with certain restrictions
4
Worcester Polytechnic Institute
4
Technical ChallengesTechnical Challenges
• Gesture Recognition lacks a standardizedGesture Recognition lacks a standardized “vocabulary”
• Spontaneous interaction requires p qimmediate engagement
5
Worcester Polytechnic Institute
5
uWave Algorithm Design: O iOverview
• Premise: “Human gestures can bePremise: Human gestures can be characterized by the time series of forces applied to the handheld device”
• Template Library– Store of one or more time series of known
identities for every vocabulary gesture• Input: Time series of acceleration
id d b th i l tprovided by a three-axis accelerometer
6
Worcester Polytechnic Institute
6
uWave Algorithm Design: O iOverview
7
Worcester Polytechnic Institute
7
uWave Algorithm Design: Q i iQuantization
8
Worcester Polytechnic Institute
8
uWave Algorithm Design: D i Ti W iDynamic Time Warping
9
Worcester Polytechnic Institute
9
uWave Algorithm Design:T l Ad iTemplate Adaptation
• Variation between gesture samplesVariation between gesture samples by same user
• Should adapt templates to• Should adapt templates to accommodate variationsU d ti S h• Updating Schemes:– Positive Update– Negative Update
10
Worcester Polytechnic Institute
10
Prototype ImplementationPrototype Implementation
• Wii remote prototypeWii remote prototype– Accelerometer range: -3g to 3g– Noise below 3.5mgg
• Recognition results returned without perceptible delay on PCs (template library of 8 gestures)– 2ms on Lenovo T60– 4ms on T-Mobile MDA Pocket PC– 300ms on 16-bit microcontroller in the Rice
Orbit sensor 11Orbit sensor
Worcester Polytechnic Institute
11
Gesture VocabularyGesture Vocabulary
12
Worcester Polytechnic Institute
12
Evaluation: SetupEvaluation: Setup
• Uses the gesture vocabulary fromUses the gesture vocabulary from previous slide
• 8 Participants• 8 Participants– 2 undergraduate, 8 graduate
7 l 1 f l– 7 male, 1 female– All 20s or early 30s, right handed
13
Worcester Polytechnic Institute
13
Evaluation: Data CollectionEvaluation: Data Collection
• Gestures are collected from 7 daysGestures are collected from 7 days within a period of about 3 weeks
• Each day the participant uses the Wii• Each day the participant uses the Wii remote and performs the 8 gestures, 10 times each10 times each
• Database at the end consists of 4480 t t t l d 560 f hgestures total and 560 for each
participant14
Worcester Polytechnic Institute
14
Evaluation: Recognition i h Ad iwithout Adaptation
• Evaluate uWave using the gestures fromEvaluate uWave using the gestures from each subject separately
• Use Bootstrapping to improve statistical pp g psignificance
• Use the collected samples to generate 70 p gtests of uWave– Produces 70 confusion matrixes– Averaged into 1 confusion matrix per subject– Average confusion matrixes of the 8 subjects
combined into a final confusion matrix 15combined into a final confusion matrix
Worcester Polytechnic Institute
15
Evaluation: Recognition i h Ad iwithout Adaptation
16
Worcester Polytechnic Institute
16
Evaluation: Recognition i h Ad iwithout Adaptation
• Average Accuracy of 93.5%g y– Gestures 1,2,6 and 7 have lower accuracy due
to similar hand movements• Large variation (9%) among participants
– “The participant with the highest accuracy performed the gestures in larger amplitude andperformed the gestures in larger amplitude and slower speed compared to other participants”
• Temporal Compression of the data speeds p p pup recognition by more than 9 times without negatively affecting accuracy
17
Worcester Polytechnic Institute
17
Evaluation: Recognition i h Ad iwithout Adaptation
Evaluation Using Samples from theEvaluation Using Samples from the Same Day
• Significantly Higher Accuracy (98 4%) when usingSignificantly Higher Accuracy (98.4%) when using only samples from the same day
• Results reported in previous reports may have been overly optimistic
• “The difference between Figure 4 (Left) and Figure 4 (Right) highlights the possible variationsFigure 4 (Right) highlights the possible variations for the same gesture from the same user over multiple days and the challenge it poses to
iti ” 18recognition.”
Worcester Polytechnic Institute
18
Evaluation: Recognition with Ad iAdaptation
19
Worcester Polytechnic Institute
19
Evaluation: Recognition with Ad iAdaptation
• Produced 7 confusion matrixes for each participants
• Averaged into confusion matrix on previous slide
• Accuracy:– Positive Update: 97.4%– Negative Update: 98.6%
• Accuracy is much better than without adaptation
Cl t d 20– Close to same day accuracy
Worcester Polytechnic Institute
20
uWave-Enhanced Applications: Gesture based Light WeightGesture-based Light-Weight
User Authentication• Prioritizes Ease-of-use over hard security• Privacy InsensitivePrivacy Insensitive• Enables authentication based on physical
manipulation of the devicep• Ran studies that showed uWave can
recognize user-defined gestures with g ghigher than 99.5% accuracy
21
Worcester Polytechnic Institute
21
uWave-Enhanced Applications: Gesture based 3D Mobile UserGesture-based 3D Mobile User
Interface• Intuitive and Convenient to navigate a 3D
interface with 3D hand gesturesS i l N ki b d id h i• Social Networking-based video-sharing serviceR t ti Ri I t f• Rotating Ring Interface– Employed uWave to navigate the interface
Uses a series of specific movements such as– Uses a series of specific movements such as tilting or slight shaking
22
Worcester Polytechnic Institute
22
Discussion of uWaveDiscussion of uWave
• Gestures and Time Series of ForcesGestures and Time Series of Forces– Diverse opinions on what is a unique gesture– Closer to speech than handwritingp g
• Challenge of Tilt– uWave uses a single three-axis accelerometer– Tilt can change the readings of force applied– Opportunity for detecting tilt is limited with a
i l l tsingle accelerometer– Extra Sensors needed to fully address problem
23
Worcester Polytechnic Institute
23
Discussion of uWaveDiscussion of uWave
• User-Dependent vs. User IndependentUser Dependent vs. User Independent Recognition– Much Lower Accuracy for User Independent
Recognition (75.4% down from 98.4%)– No commonly accepted gestures for
InteractionsInteractions• Gesture Vocabulary Selection
More Complicated Gestures may have higher– More Complicated Gestures may have higher accuracy
– Number of Complicated Gestures Users can 24use may be small
Worcester Polytechnic Institute
24
ConclusionsConclusions
• Employs a single accelerometer so it canEmploys a single accelerometer so it can be readily implemented on current devices
• Uses DTW to measure similarities between two time series of forces
• Tests show uWave achieves 98.6% accuracy with one training sample– Comparable to HMM-based methods with 12
i i ltraining samples• Challenges of Variation across Time and
Users 25Users
Worcester Polytechnic Institute
25
Video DemonstrationVideo Demonstration
• uWave DemonstrationuWave Demonstration
26
Worcester Polytechnic Institute
26
Questions?
27
Worcester Polytechnic Institute
27