final presentation (revision 2)

Motion Recognition Using

Accelerometers

Walter Hudgens

Chad Buckallew

Jay Won

Sarah Duncan

Matt Smith

Bobby Onyeukwu

Advisor: Dr. Fei Hu

1ECE 494 University of Alabama

Agenda

• Accelerometer Background

• Wii-motes

• Extraction Program

• MATLAB Implementation

• Acceleglove

• Validation

• Administrative information


Project overview

• Our capstone group was given the task of creating a

virtual environment. This environment was to utilize

accelerometers in order to recognize dynamic gestures

performed by an unknown user. In the process of

recognizing these gestures, the environment was to also

be able to train and store data from past trails, creating a

library of gestures.


Deliverables

The deliverables expected upon completion of this project

are as follows:

1. Demonstration of Wii-motes functioning with

corresponding limb recognition software

2. Demonstration of the Acceleglove functioning

with hand recognition software

3. Generation of a group website

4. Final report


Full system overview

5

MATLAB User

Interface

ANNMATLAB Data

Manipulation

Acceleglove SDKExtraction

Program

Wired Digital

GloveBluetooth Adapter

Wii-motes: Wii-mote Group

: Glove Group

Legend

ECE 494 University of Alabama

Accelerometers

• Accelerometers are sensors that are capable of measuring

acceleration.

• Used to convert linear or angular accelerations into output

signals

• The sensor then splits the signal into two distinct signals

• Acceleration due to gravity

• Allows the sensor to determine which direction is down

• External acceleration

• By factoring out the external acceleration, orientation

of accelerometer can be factored.


Sample accelerometer spring

system

7[1]ECE 494 University of Alabama

Accelerometer theory

• Output signals are obtained by an application of Newton’s

law.

• When in equilibrium, the restorative force is equal to the

force of acceleration acting on the mass.

• The displacement of the spring (x) is then converted into

an output signal.

• Measurement of piezoresistave material

• Measurement of changing capacitance between

moving and fixed elements.

8ECE 494 University of Alabama[1]

Wii-motes

Bobby Onyeukwu

9University of AlabamaECE 494

Wii-mote comparison

10[2] University of AlabamaECE 494

Old Wii-mote New Wii-mote

Accelerometer ADXL330 ADXL330

Communication Bluetooth HID Bluetooth HID

Name Nintendo RVL-CNT-01 Nintendo RVL-CNT-01-TR

Product ID 0X0306 0X0330

Vendor ID 0X057e 0X057e

Costs $10.00 $40.00

MotionPlus Not Included Included

(X,Y,Z) axes


ADXL330 accelerometer


Why Wii-motes?

• Budget friendly

• $10.00 each

• Readily available

• User friendly

• Utilizes a triple axis accelerometer

• Coincides with research being conducted by Dr. Hu


Wii-mote validation

Test controller functionality

PC connectivity via Bluetooth adapter

Connectivity to C# data extraction program

Inputs data into MATLAB environment

C# user interface

MATLAB user interface


Bluetooth adapter (dongle)

15

• Rocketfish Micro Bluetooth

adapter

• RF-MRBTAD

• Size: 0.2 x 0.6.x 0.8 inches

• Cost: $ 7.00

[4] University of AlabamaECE 494

Wii-mote flowchart

Train

Complete

N

Y

N

Y

C# Testing

C# Training MATLABSuccessful

Recognition

Wii-mote

Expected

Output

University of AlabamaECE 494 16

C# program

Chad Buckallew


Data acquisition codes for

Wii-mote• C# API

• WiimoteCS by Brian Peek.

• C++ API

• Wiiyourself!

• Fwiine

• Window 7 platform.


Choosing C#

• Familiar platform – Visual Studio 2010

• Delegates and Events

• Runtime Generic Support

• Open source code with least modification required


Main window

• MultipleWiimoteForm.cs

• Initial User Interface

• Device status

• Access to Training/Testing functionality

• Invokes:

• Wiimoteinfo.cs

• Train.cs

• Trainusercontrol.cs

• Test.cs

• Testusercontrol.cs


Training


• Training Interface

• Trials Accuracy

• Sampling Rate 40Hz

• Stores results to Train.txt

• Math Utilized - MATLAB

• Smoothing

• Dynamic Time

Warping

• Affinity Propagation

• Invokes:

• Wiimoteinfo.cs



Training


Training Library

• Output – Train.txt

• Serves as reference

library

• Stores training data

(x,y,z)

• Wii-mote data is set of 3

columns

• Gestures denoted by ‘$’

• “%x%” Identifies Trials

• File accumulates all

gestures


Trial 1

Sampling

• 40Hz Sampling Rate

• 200 traces distributed

for single or multiple

Wii-motes

• Testing verified

successful recognition

Number of

Wii-motes

Number of traces

(per Wii-mote)

1 200

2 100

3 66


Test main window

• Test.cs

• Verify successful gesture

recognition

• Generate Test.txt

• Invokes:

• Wiimoteinfo.cs

• Testusercontrol.cs

• Math Utilized -MATLAB

• Smoothing

• DTW

• Random Projection University of AlabamaECE 494 25

Sample gesture (bicep curl)

University of AlabamaECE 494 26[17]

Wii-Mote 1 Wii-Mote 2 Wii-Mote 3 *Omit Dumbbells

C# program validation plan

Test Wii-mote compatibility (single/multi-mode)

Acquire Wii-mote data

Train simple gesture to obtain ‘Train.txt’

Obtain ‘Test.txt’

Ensure ‘Train.txt’ and ‘Test.txt’ valid for gesture

recognition


MATLAB Training

Jay Won


MATLAB overview


Training Data DTW AP Exemplars

Smoothing

Testing Data DTW RPL-1

Minimization

Recognized

Gesture: Training Phase

: Testing Phase

Legend

Training phase


Training Data DTW AP Exemplars

[16,18,17]

Dynamic Time Warping

(DTW)

• Finding the similarity

cost between two graphs.

• The lower the cost, the

most similar the graphs

are.


Mathematical representation


DTW Example


Cost:

3.3084e+05

Cost:

2.7239e+06

Affinity propagation (AP)

• Finds exemplars among the

data

• Operates using matrix

similarities

• Four Basics of AP

• Responsibility

• Availability

• Similarity

• Preference


AP basics

• Responsibility- message between exemplars and clusters

to decide which traces are exemplars.

• Availability- message to decide which clusters belong to

which exemplars.

• Similarity- DTW costs.

• Preference- a minimum requirement for exemplars to be

optimal exemplars.


Mathematical representation

• Responsibility

• Availibility

• Similarity

• Preference


MATLAB Testing

Chad Buckallew


Testing phase

Exemplars

Recognition of Y

R

DTW

Y

Test Data

L1 Minimization

Solving for (X,Y,Z)

(Rx,Ry,Rz)


RP

DTW


Exemplars

Recognition of Y

R

DTW

Y

Test Data

L1

Minimization

Solving for

(X,Y,Z)

(Rx,Ry,Rz)

Equation:

[17]

Random projection (RP)

• Utilized to reduce a high

dimension matrix to a

lower dimension by:


Exemplars

Recognition of Y

R

DTW

Y

Test Data

L1

Minimization

Solving for

(X,Y,Z)

(Rx,Ry,Rz)

[17]

RP mathematical representation


Exemplars

Recognition of Y

R

DTW

Y

Test Data

L1

Minimization

Solving for

(X,Y,Z)

(Rx,Ry,Rz)Generation of Matrix

[17]

Further mathematical

representationExemplars

Recognition of Y

R

DTW

Y

Test Data

L1

Minimization

Solving for

(X,Y,Z)

(Rx,Ry,Rz)


MATLAB GUI


Data acquisition


MATLAB GUI


Compute


MATLAB validation

Train and test data acquisition

Understand and implement DTW, AP, RP

Recognize unknown gesture

Make MATLAB GUI and connect with C# Program

Work in 32-bit, 64-bit, and Mac Operating system


Digital Glove

Matt Smith


Digital glove

• Java software

• MEM Accelerometers

• USB cable (wired)

• Supply Voltage: 5V

• Range: ± 1.5g

• Sampling Rate: 120 Hz

49University of AlabamaECE 494 [20]

Accelerometer placement

50University of AlabamaECE 494 [20]

SDK visualizer


Artificial neural networks

• Artificial neural networks

(ANN) are computer codes

that act like a human brain.

• They consist of cells which

have inputs and outputs.

• The output (y) of a cell is a

linear combination of its

inputs (x), with different

weights (w) assigned to each

one.

[8] 52University of AlabamaECE 494

Training the network

• In order to work, the neural network must be trained.

• Training a neural network assigns the value of the weights.

• The weights represent the network’s “knowledge.”

[9] 53University of AlabamaECE 494

How training works

• The network is given some sample inputs and target outputs.

• This data is divided into three classes.

• Training data

• This is the data the network will use to learn.

• Validation data

• This data is used so the network knows when to stop learning.

• Testing data

• This data is used as a final test of the network’s accuracy.


How training works

1. Give weights some random

initial value.

2. Perform calculations on the

testing data and the validation

data.

3. Calculate error for both sets

4. Change weights to reduce

training data error.

5. Repeat 2-4 until error from

validation data is a minimum.


Flowchart


Input Matrix

GeneratorNetwork Trainer

Target Matrix

Generator

RecognizerPerformanc

e

Acceptable?

Y

N

Gestures used


Demo


Validation, testing, and challenges.

Sarah Duncan


Validation plan (visualizer)

Confirm glove functionality with

SDK/Visualizer software through

existing ASL gesture library (A-Z).

Build numerical library (1-10).


Visualizer testing data

0

20

40

60

80

100

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Probability Filter Set on 90


Building the numerical library

• Used the SDK from the Acceleglove

• Created a new library to train the numerical gestures into

• Determined an average number of gestures using pre-

existing library (ASL)

• Trained numerical gestures into the new library

• Determined accuracy


Accuracy from trained library

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Su

ccess (

%)

One through Ten


Validation plan (MATLAB)

Confirm that AcceleGlove data can be accessed in

readable form through USB.

Access glove data directly through MATLAB.

Generate data to train network.

Read in and recognize gestures with the neural network.

Recognize gestures made by a user with 90% accuracy.


USB data access


Serial through MATLAB

• This test was undertaken to determine whether we could

access serial ports through MATLAB

• We wrote test code to access the AcceleGlove’s data

directly in MATLAB.

• Data is read in from the glove as a character vector.

• Code converts the data to a vector of type “double.”


MATLAB-serial code

port=serial('com13','Baudrate',38400,'Terminator','CR');

fopen(port);

for k = 1:50

fprintf(port,'a');

ASCII_data = fscanf(port);

double_data = sscanf(ASCII_data,'%d')

display(double_data)

end

fclose(port);


Generating data


Accuracy of the trainer

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Su

ccess (

%)

Matt


Accuracy comparison

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Su

ccesses (

%)

Sarah

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

Su

ccess

(%

)

Chad

• Low: 15

• High:100

• Total Accuracy : 70

• Low: 0

• High: 100

• Total Accuracy:

65.5


Obtaining 90% accuracy

• Modify code

• Expand the reference library

• Test results as each new reference is added to the ANN

library

• Determine if accuracy increases for trainers and non-

trainers


Logs and backups

• The code is being stored on personal computers, flash

drives, and Crimson Google Drive.

• The code versions are organized by date modified.


Challenges and issues

• Glove configures to different COM ports on different

computers.

• This requires changing the code every time it is used on

a new computer.

• IRB approval currently limits the amount of testing that

can be conducted

• Team has completed IRB Certification as of 9/13/2012

• Request submitted 10/15/2012;re-submitted 12/7/2012

• Glove is wired.

• This restricts the user.

• Cable can be disconnected. 73University of AlabamaECE 494

Budgeting and Impacts

Walter Hudgens


Budgeting

75

0%

100%

0%

Wii-motes

Bluetooth Adapter

Acceleglove

Equipment Price per unit Quantity Total

Wii-motes $10.00 3 $30.00*

Accelglove $650.00 1 $650.00

Bluetooth adapter $7.00 2 $14.00*

Total: $650.00

*=Donated

ECE 494 University of Alabama

Schedule


Societal impacts

• If implemented into hospitals, reduced cost of teletherapy

will decrease the medical costs and improve the quality of

life for stroke victims.


Environmental impacts

• Telerehabilitation

therapy will reduce the

amount of gasoline used

in the transportation of

patients to and from their

respective medical

centers.


Economical impacts

• According to the World Health Organization (WHO), the

total healthcare spending in the year 2010 was $2.3

trillion, which is 17.9% of our nation’s Gross Domestic

Product (GDP). It is estimated to reach 19.5% by the year

2017.


Healthcare spending (2010)

Hosipital Care31%

Physician/Clinal Services20%

Pharmaceuticals10%

Dental4%

Nursing Homes6%

Home Health Care3%

Govt Public Health Activites

3%

Administrative Costs

7%Investments

7%

Other (Physical Therapists)

6%

Other Retail Products3%


Political impacts

• Under our nations

current healthcare

reforms, reimbursement

for members of the

Medicare Advantage

program will be reduced.

• If implemented, we

believe that our system

could reduce the cuts in

benefits and services the

seniors experience.


References

[1] T. R. Burchfield, S. Venkatesan, “Accelerometer-Based Human Abnormal Movement Detection in Wireless Sensor

Networks,” Computer Engineering Program, University of Texas at Dallas, Dallas, TX.

[2] http://www.symbolicsound.com/cgi-bin/bin/view/Learn/UsingTheNintendoWiimoteWithKyma

[3] “ADXL330 Datasheet”. http://www.analog.com/static/imported-files/data_sheets/ADXL330.pdf

[4] “Rocketfish Bluetooth Micro Dongle”. http://www.rocketfishproducts.com/products/computer-accessories/RF-

MRBTAD.html

[5] Yao, B. and Zhu S. "Learning Deformable Action Templates from Cluttered

Videos". UCLA.http://www.stat.ucla.edu/~zyyao/data/Action_template_iccv09.pdf

[6] Frau, D. "Pattern Matching Techniques". http://cyber.felk.cvut.cz/gerstner/teaching/zbd/dtw.pdf.

[7] http://max-muscle-gain.blogspot.com/

[8] http://mybrainnotes.com/brain-cell-neuron.gif

[9] http://photos1.blogger.com/blogger/82/1614/320/fig121.JPG

[10] http://libbyjohnson.hubpages.com/hub/Whats-Behind-the-High-Gas-Prices-More-Than-You-Think

[11] http://castro21.files.wordpress.com/2010/10american_sign_language_numbers_at_coloringpageskidsboys.gif

[12] http://www.mathworks.com/help/nnet/examples/cancerdetectdemonnet_02.png

[13] http://seandenoyer.com/wp-content/uploads/2012/10/president_obama_lightsaber.jpg

[14] http://www.kaiseredu.org/Issue-Modules/US-Health-Care-Costs/Background-

Brief.aspx#How%20is%20the%20U.S.%20health%20care%20dollar%20spent?

[15] http://en.wikipedia.org/wiki/Health_care_in_the_United_States#cite_note-29

[16] http://www.stat.ucla.edu/~zyyao/projects/action.htm

[17] http://vimeo.com/1821063

[18] A. Ahmad, and F. Chen, “A Novel Accelerometer-based Gesture Recognition system” IEEE Trans. On Signal

Processing, VOL. 59, NO. 12, Dec. 2012

[19] http://3.bp.blogspot.com/-m2dY2-t1-Xg/TvzAgcehhOI/AAAAAAAAAHA/IOOGLTl3Sfw/s1600/end-of-semester-

student-studying-finals-week-grading-essays.jpg

[20] http://www.acceleglove.com/


http://www.stat.ucla.edu/~zyyao/data/Action_template_iccv09.pdf

http://cyber.felk.cvut.cz/gerstner/teaching/zbd/dtw.pdf

http://max-muscle-gain.blogspot.com/

http://mybrainnotes.com/brain-cell-neuron.gif

http://photos1.blogger.com/blogger/82/1614/320/fig121.JPG

http://libbyjohnson.hubpages.com/hub/Whats-Behind-the-High-Gas-Prices-More-Than-You-Think

http://www.mathworks.com/help/nnet/examples/cancerdetectdemonnet_02.png

http://www.kaiseredu.org/Issue-Modules/US-Health-Care-Costs/Background-Brief.aspx

http://en.wikipedia.org/wiki/Health_care_in_the_United_States

http://vimeo.com/1821063

Questions


• [email protected]

final presentation (revision 2)

Documents