dewristified: handwriting inference using wrist …...2019-05-16 spritelab @ utsa 7 our research...

deWristified: Handwriting Inference Using Wrist-Based

Motion Sensors Revisited

Raveen Wijewickramaraveen.wijewickrama@utsa.edu

Anindya Maitia.maiti@ieee.org

Murtuza Jadliwalamurtuza.jadliwala@utsa.edu

University of Texas at San Antonio

Wrist Wearables

• Extends the functionality of traditional wristwatches beyond timekeeping.

• Captures rich contextual information about the wearer.• Enables several novel context-based applications.

2019-05-16 SPriTELab @ UTSA 2

Motion Sensors

• Two main types of motion or inertial sensors:• Accelerometer: records device acceleration.• Gyroscope: records device angular rotation.

• Accessing motion sensors on wearable devices:

• All applications have access to motion sensors by default (also referred to as zero-permission sensors) on most wearable OSs.

• Applications’ access to motion sensors cannot be regulated on most wearable OSs –we can’t turn them off!

• Can an adversary take advantage of motion sensor data from a wrist-wearable device to infer private information inputted by the user’s device-wearing hand?

2019-05-16 SPriTELab @ UTSA 3

Inferring Private User Inputs (Using Wrist Wearables)

2019-05-16 4SPriTELab @ UTSA

State-of-the-Art in Handwriting Recognition (Using Wrist Wearables)

2019-05-16 5SPriTELab @ UTSA

Airwriting (Amma et al.) Whiteboard writing (Arduser et al.)

Finger writing (Xu et al.) Pen(cil) writing (Xia et al.)

Adversary Model

• Adversary has knowledge of the type of handwriting.

• Adversary is able to record data from the target smartwatch’s accelerometer and gyroscope sensors.• Could employ a Trojan app for this!

• Adversary’s Goal: To infer handwritten information using target user’s smartwatch sensors.

2019-05-16 SPriTELab @ UTSA 6

Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables)

• Airwriting (Amma et al.)• Custom-designed hand glove with very

high precision sensors.• Our adversary relies on target user’s

smartwatch or fitness band.

• Only uppercase words.

• Whiteboard writing (Arduser et al.)• Not generalized (training and testing

data not from different participants).

• Only uppercase alphabets.

• No handwriting activity detection.

• Finger writing (Xu et al.)• Use of Shimmer, a specialized sensing

device intended for lab studies.

• Not generalized (training and testing data not from different participants).

• Pen(cil) writing (Xia et al.)• Only lowercase alphabets.

• Controlled data collection.

• No handwriting activity detection.

2019-05-16 SPriTELab @ UTSA 7

Our Research

• How practical is handwriting inference when• Using consumer-grade wrist

wearables,

• Using generalized training and testing,

• Writing in a uncontrolled and unconstrained manner, and

• Both upper and lowercase alphabets are modeled ?

2019-05-16 SPriTELab @ UTSA 8

New Uncontrolled and Unconstrained

Writing Data

Existing Models

Handwriting Inference Framework

2019-05-16 9SPriTELab @ UTSA

Experimental Setup

• 28 participants for the four writing scenarios.• 18 to 30 years of age• 13 male, 15 female

• Two different wrist-wearables.• Sony Smartwatch 3, LG Watch Urbane

• Accelerometer and gyroscope recorded at 200Hz.

• Participants provided with appropriate writing apparatus.

2019-05-16 SPriTELab @ UTSA 10

Writing Tasks (In-Lab)

• Alphabets. • Individual alphabets one at a time.

• Covered all 26 English alphabets in random order.

• Each alphabet was written 10 times.

• Both upper and lower cases.

• Words.• 4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012).

• Each participant wrote 20 words, in both upper and lower cases.

• Sentence.• "the five boxing wizards jump quickly" in both upper and lower cases.

2019-05-16 SPriTELab @ UTSA 11

Writing Activity Recognition (Out of Lab)

• 2 participants.

• Wore a smartwatch for an entire day.

• Performed the four writing scenarios at random times.

• Adversary’s Goal: To infer handwriting activity first, and then classify the handwritten text.

2019-05-16 SPriTELab @ UTSA 12

Replicated Inference Frameworks

• Airwriting • Hidden Markov Model (HMM)

• Whiteboard writing • Dynamic Time Warping (DTW)

• Finger writing • Naive Bayes, Logistic Regression and Decision Tree classifiers

• Pen(cil) writing• Random Forest classifier

2019-05-16 SPriTELab @ UTSA 13

Personalized Inference Accuracy

2019-05-16 SPriTELab @ UTSA 14

Writing Activity Detection:56% recall and 57% precision for air and finger writing39% recall and 47% precision for pencil writing23% recall and 34% precision for whiteboard writing

Personalized Inference Accuracy(Whiteboard Writing)

2019-05-16 SPriTELab @ UTSA 15

Lowercase Uppercase

Generalized Inference Accuracy

2019-05-16 SPriTELab @ UTSA 16

Writing Activity Detection:35-40% recall for airwriting, whiteboard writing and pencil writingOnly 8% recall for finger writing

Factors Affecting Inference Accuracy

• Number of Strokes.

2019-05-16 SPriTELab @ UTSA 17

Factors Affecting Inference Accuracy

2019-05-16 SPriTELab @ UTSA 18

Number of strokes for the same letter for different participants (lowercase).

Number of strokes for the same letter for different participants (uppercase).

Factors Affecting Inference Accuracy

2019-05-16 SPriTELab @ UTSA 19

Lowercase Uppercase

Variance in number of strokes per alphabet per participant, averaged for all participants

Factors Affecting Inference Accuracy

• Number of Strokes.

• Order of Strokes.

• Direction of Strokes.

2019-05-16 SPriTELab @ UTSA 20

Factors Affecting Inference Accuracy

2019-05-16 SPriTELab @ UTSA 21

Factors Affecting Inference Accuracy

• Number of Strokes.

• Order of Strokes.

• Direction of Strokes.

• Uppercase vs Lowercase.

• Specialized Devices.

2019-05-16 SPriTELab @ UTSA 22

Airwriting (Amma et al.)

Conclusion

• We investigated how wrist-wearable based handwriting inference attacks perform in realistic day-to-day writing situations.

• Such inference attacks are unlikely to pose a substantial threat to users of current consume-grade smartwatches and fitness bands.• Primarily due to highly varying nature of handwriting.

• Replicable artifacts: https://sprite.utsa.edu/art/dewristified

2019-05-16 SPriTELab @ UTSA 23