HeadScan: A Wearable System for Radio-based

Sensing of Head and Mouth-related Activities

Biyi Fang

1 Apr. 13, 2016IPSN

Nicholas D. Lane

Department of Electrical and Computer Engineering

Michigan State University

Mi Zhang Fahim KawsarBiyi Fang

2

The age of wearables is upon us

VisionMicAccelerometer Physiology

Wearables provide possibilities to understand oneself as well as the world.

3

Limitation of Current Sensors

Accelerometer

Complex human activities need multiple Accelerometers.

4

Audio + Video

Audio and Video are privacy intrusive.

Limitation of Current Sensors

5

Physiology

Most Physiology sensors require firm skin contact.

Limitation of Current Sensors

6

We need a novel sensing modality for wearables that is non-contact, privacy-

preserving while still providing richinformation.

7

Radio as a Sensing Modality

All of these applications use radios/WiFi deployed in the ambient environment such as homes and public places.

Radio sensing has attracted considerable attention:

Crowd Counting [3]

Track Indoor Movement [2]

Indoor Activity [1]

…

8

What will radio sensing meanfor wearables?

1. Can it sense at a distance? People? Object? Material?

2. Can it sense complex movements? Transportation mode?

3. Can it sense physiological signals? Breath? Heart? Muscle?

9

Comparison of Wearable Sensing Modalities

Multiple Privacy Contact

Accelerometer

Audio + Video

Physiology

Radio

We envision radio has a significant potential to provide new sensing capabilities for next-gen wearables.

10

This work represents our first step to realize our vision.

11

HeadScan Overview

HeadScan solves all the problems.

HeadScan is a wearable system that uses radio to sense head and mouth-related activities

Rationale:

─ Eat, Drink, Cough, Speak

─ Important, e.g., healthcare, social computing etc.

─ Existing wearable technologies have their limitations

multiple privacy contactdiet monitoring audio sociometer acoustic sensing

12

Key Principle

Tx Rx

HeadScan

Wearable

Unit

Radio

Signal

Static Moving

13

Observations

Eat, Drink, Cough, Speak have distinctive shapesand periodicities.

One Bite

One Sip

One Cough One Consonant

eat drink

cough speak

Radio signal contains rich information of mouth/head movements.

Hardware Prototype

• One wearable unit that contains two HummingBoard Pro. Each board connects with one Intel WiFi card for measuring CSI (3).

Hardware contains two parts:

• Two omni-directional antennas as transmitter (1) and receiver (2) antenna.

14

1

2

3

Radio Signal Processing Pipeline

Wearable

Unit

Filtered

Data

Principal

Components

Feature

Vector

Sparse

Coefficient

Vector

SegmentationLow Pass

Filter

Segmented

DataPCA

Feature

Extraction

SparseRepresentation

Classifier

15

Raw

CSI Data

Recognized

Activity

Noise Removal

16

Low Pass Filter:─ Radio signal is noisy.

─ Remove noise and outliers that are not caused by the

targeted activities.

─ We selected 20 Hz as our cut-off frequency.

17

Principal Component Analysis (PCA) on subcarriers:─ CSI has 30 subcarriers and they are highly correlated.

─ Select the first projection of CSI data

─ Remove redundant information

Principle Component Analysis

18

Sparse Representation

Feature Extraction:─ Similar to kNN with difference in the distance definition

─ Extract features based on previous observations

eat drink

cough speak

19

Sparse Representation

Residual Determination (Distance):─ Construct overcomplete dictionary A:

─ Sparse Representation of new sample

─ Residual (distance) calculate

20

Sparse Representation

Compare the distance:

21

Evaluation Setup

Methodology:─ Compare Radio vs. Audio (2) [4]

─ Examine different factors (5)

Data Collection:─ Collected data from 7 participants

─ Radio and Audio collected simultaneously

─ Lab Setting

─ Scripted Activities

─ A total of 7.2 hours – 5,171 samples -- collected

Result

Radio performs better with average accuracy 86.3% compared to 81.7% of Audio

Comparison between Radio and Audio-based Sensing Wearable

─ Experiments

conducted in clean

environment

─ Leave-one-subject-

out Cross Validation

22

Result

Minimum requirement to maintain above 80% of sampling rate is 10 Hz, which takes tiny part of wireless bandwidth.

Impact of Radio Signal Transmission Rate

─ Configured at multiple

sampling rate.

─ The average accuracy

of 100, 50, 10, 8 and 5

Hz are 86.2%, 85.7%,

80.6%, 67.4%, and

63.7% respectively.

23

67.4%86.2% 85.7% 80.6% 63.7%

Result

Shoulder and Collar (SC) are the best deploy locations of receiver and transmitter antenna, respectively.

Impact of Radio Transmitter/Receiver On-body Locations

─ Abbreviation: C,

Collar; S, Shoulder;

24

92.0% 82.7% 69.2% 68.0%

─ Left/Right symbol

represents receiver/

transmitter antenna

location.

─ This is because Rx is

much more sensitive to

movements that occur at

a closer location to Rx.

Collar

Shoulder

Result

HeadScan wearable system is robust to the interferencecaused by nearby people.

Impact of Interference Caused by Nearby People

─ Operated when there

are 0, 1 and 2 people

nearby the subject.

─ Average accuracy are

90%, 93% and 90%

when 0, 1 and 2 people

nearby, respectively,.

25

90% 93% 90%

Limitation & Future Work

─ All the data collected are in the lab environment. We will

conduct in-the-wild experiments in the future.

─ Our prototype is a little bit bulky. We are designing a

new prototype to make it more wearable.

─ Our radio signal processing pipeline is heavyweight and

thus runs offline. We will design an lightweight pipeline

to make it online.

26

Conclusion

─ We designed and implemented a radio based system

with competitive performance, with features of privacy-

preserving and non-contact.

─ We believe our research opens up a new direction.

─ We have discovered more possibilities along this

direction. Welcome to join us at MobiSys 2016 ☺.

27

28

Reference

1. Wei, Bo, et al. "Radio-based device-free activity recognition with radio frequency interference." Proceedings of the 14th International Conference on Information Processing in Sensor Networks. ACM, 2015.

2. Adib, Fadel, Zachary Kabelac, and Dina Katabi. "Multi-person localization via rf body reflections." 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). 2015.

3. http://www.ece.ucsb.edu/~ymostofi/HeadCountingWithWiFi.html

4. K. Yatani et al. BodyScope: A wearable acoustic sensor for activity recognition. In ACM Conference on Ubiquitous Computing, 2012.

29

Q & A

Thank You