Download - When Machine Learning Meets Wi-Fi
![Page 1: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/1.jpg)
![Page 2: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/2.jpg)
Robust Device-Free Indoor Localization
When Machine Learning Meets Wi-Fi
Steve Liu, VP R&D, Chief-Scientist
Samsung AI Center – Montreal &
Professor & William Dawson Scholar
McGill University
Nov. 5th, 2019
Disclaimer: Any views or opinions presented in this talk are personal and do not represent those of people, institutions, organizations that the presenter may or may not be associated with in professional or personal capacity.
![Page 3: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/3.jpg)
Outline
Robust Device-Free Indoor Localization
• Indoor Localization & Background
– Fingerprinting-based Device-Free Localization Approach
• A Major Challenge: Environment Changes
• AutoFi
• Experiments
• More Recent Advances: Toward Robustness in Real World
• Conclusions & Future Work Directions
3
![Page 4: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/4.jpg)
Indoor Localization & Background
![Page 5: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/5.jpg)
Why this Topic?
The next big thing!
• Internet of Things
• AI and Machine Leaning (=> Intelligent)
• Advanced Communications like Wi-Fi 6 & 5G (=> Connected )
5
![Page 6: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/6.jpg)
Indoor Localization
GPS signal not available
6
![Page 7: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/7.jpg)
Applications
What can we do with location information?
Indoor navigation
Smart home automation
Automated industry
7
![Page 8: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/8.jpg)
State-of-the-Art
Device-Oriented
• Pro: High accuracy: 10cm-level (e.g., [1],[2],[3])
• Con: Complicated design– Customized hardware
– Antenna arrays (10+ antenna)
– Usually heavy computational overhead
• Locating devices, NOT users
[1] M. Kotaru, K. R. Joshi, D. Bharadia, and S. Katti, “Spotfi: Decimeter level localization using WiFi”, Sigcomm ‘15. [2] X. Li, D. Zhang, Q. Lv, J. Xiong, S. Li, Y. Zhang, and H. Mei, “Indotrack: Device-free indoor human tracking with commodity WiFi,” IMWUT’ 17[3] K. Qian, C. Wu, Y. Zhang, G. Zhang, Z. Yang, and Y. Liu, “Widar2.0: Passive human tracking with a single WiFi link,” MobiSys ‘18.
8
![Page 9: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/9.jpg)
Why Device-free Localization?
People are not always carrying devices
• Cannot, or Do not want to carry
9
![Page 10: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/10.jpg)
State of the Art
Device-Oriented
Device-Free
• Triangulation methods– Utilize user’s blocking/reflection of the signals
– Calculating Time-of-Fight (ToF), Angle-of-Arrival (AoA), Propagation features, etc.
– Dedicated devices required
10
![Page 11: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/11.jpg)
Disadvantages of Dedicated Devices
• Cost– Equipment cost
– Deployment
– Triangulation: positions of devices must be known in advance
• Scalability
• Range limitation (motion sensors, RFID, …)
• Blocked by walls
• …
Motion Sensors CamerasRFID
Tag & Reader
11
![Page 12: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/12.jpg)
State of the Art
Device-Oriented
Device-Free
• Triangulation methods
• Fingerprinting-based method– Correlating environment features with human locations
– Environment features:
• Wireless signals, e.g., Wi-Fi
• Light or sound
• Magnetic field
• …
12
![Page 13: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/13.jpg)
Benefits of Wi-Fi based DfP Localization
Work through walls!
Available Everywhere!
13
![Page 14: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/14.jpg)
Wi-Fi Based DfP Localization
Fingerprinting: Associate Wi-Fi features such as CSI (Channel State Information) or
RSSI (Received Signal Strength Indication) with users’ locations.
14
![Page 15: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/15.jpg)
Wi-Fi Channel State Information
Channel State Information (CSI)
• Describe the Wi-Fi channel properties
• Sensitive to human locations
• Provides much more information than RSSI
• Readily available from commercial Wi-Fi cards
15
1 RSSI → 1 integer
56 subcarriers with CSI values(complex numbers) → 56 magnitudes + 56 phases
→ 112 floating point numbers
![Page 16: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/16.jpg)
More Background on Wi-Fi CSI
Wi-Fi Multiple-Input Multiple-Output (MIMO)
e.g., Intel 5300 NICe.g., Linksys WRT160N16
![Page 17: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/17.jpg)
More Background on Wi-Fi CSI
MIMO in math
Received signals
Streams (Spatial Channels)CSI
Transmittedsignals
Noise
Source: http://www.sharetechnote.com/html/Communication_ChannelModel.html
𝒚 = 𝑯 𝒙 + 𝒏
17
![Page 18: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/18.jpg)
How does CSI look like?
CSI profiles at h11
• Captures the math representation of channel characteristics from transmitter antenna 1 to receiver antenna 1
• Time/frequency variant
Time (seconds)
Frequency(Subcarriers)
CSIMagnitude
18
![Page 19: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/19.jpg)
CSI Profiles are Location-Dependent
Subcarriers
CSI
Magnitude
19
![Page 20: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/20.jpg)
CSI Profiles to CSI Fingerprints
A CSI fingerprint of location L is a CSI profile associated with this location
• i.e., fingerprint = CSI profile at location L
20
![Page 21: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/21.jpg)
How to use ML for Localization?
Use CSI fingerprints to train ML classifiers
• Locations – labels (output), CSI profiles (input)
• SVM, Random Forest, KNN, Neural Networks, etc.
MLClassifiers
For training
21
![Page 22: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/22.jpg)
Classification for Localization
Trained models to classify the labels of new (online measured) CSI profiles
• Predicted labels = estimated locations
Trained ML
Classifiers
10 locations, (sub-)meter-level resolution - 99.7% accuracy.22
![Page 23: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/23.jpg)
A Major Challenge
- Environment Changes
![Page 24: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/24.jpg)
A Major Challenge
The changes in the indoor environment make the (old) fingerprints inconsistent with
the current situation (new).
24
![Page 25: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/25.jpg)
A Major Challenge
The CSI fingerprints will be “contaminated” by environment changes.
• The recorded fingerprints (old) no longer represent the changed environment (new)
25
![Page 26: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/26.jpg)
A Major Challenge
The CSI fingerprints will be “contaminated” by environment changes.
• The recorded fingerprints (old) no longer represent the changed environment (new)
Before After
26
![Page 27: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/27.jpg)
A Major Challenge
Localization accuracy drops significantly
• A case study:
Record the fingerprints again, and retrain the model?
27
Impractical: Inconvenient & time-consuming
![Page 28: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/28.jpg)
AutoFi
![Page 29: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/29.jpg)
Idea: Can We Reuse the Fingerprints?
Reconstruct vectors measured from the changed domain to the old fingerprint domain, so we can reuse the already trained model
𝑀𝑛𝑒𝑤−𝑜𝑙𝑑
-------------------------------------------------------------------------------------
New vector domain(Contaminated CSI profiles)
Old vector domain(fingerprints)
𝑀𝑛𝑒𝑤−𝑜𝑙𝑑
𝑉𝑒𝑚𝑝𝑡𝑦𝑜𝑙𝑑 𝑉𝑝1
𝑜𝑙𝑑
𝑉𝑒𝑚𝑝𝑡𝑦𝑛𝑒𝑤 𝑉𝑝1
𝑛𝑒𝑤
29
![Page 30: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/30.jpg)
The Mapping Functions
Map CSI from the new domain to the old domain
Contaminated CSI profiles(locations unknown)
Unknown mapping How to determine?
𝑉𝑝1𝑜𝑙𝑑 = 𝑀𝑝1 × 𝑉𝑝1
𝑛𝑒𝑤 + 𝑛𝑝1,
𝑉𝑝2𝑜𝑙𝑑 = 𝑀𝑝2 × 𝑉𝑝2
𝑛𝑒𝑤 + 𝑛𝑝2 ,
⋮
𝑉𝑝𝑁𝑜𝑙𝑑 = 𝑀𝑝𝑁 × 𝑉𝑝𝑁
𝑛𝑒𝑤 + 𝑛𝑝𝑁 .
Fingerprints in Old Domain(associated with locations)
30
![Page 31: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/31.jpg)
The Mapping Functions
Approximation: the mapping functions are the same for different locations
• Domain-to-domain mapping
𝑀𝑝1 = 𝑀𝑝2 = ⋯ = 𝑀𝑝𝑁 = 𝑀.
Contaminated domain (New) Fingerprint domain (Old)
M: New →Old
31
![Page 32: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/32.jpg)
The Mapping Function
Mapping function: M
𝑉𝑝1𝑜𝑙𝑑 = 𝑀 × 𝑉𝑝1
𝑛𝑒𝑤 + 𝑛𝑝1,
𝑉𝑝2𝑜𝑙𝑑 = 𝑀 × 𝑉𝑝2
𝑛𝑒𝑤 + 𝑛𝑝2 ,
⋮
𝑉𝑝𝑁𝑜𝑙𝑑 = 𝑀 × 𝑉𝑝𝑁
𝑛𝑒𝑤 + 𝑛𝑝𝑁 .
32
Fingerprints in Old Domain(associated with locations)
Contaminated CSI profiles(locations unknown)
![Page 33: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/33.jpg)
The Mapping Functions
How to automatically determine the mapping function M?
𝑉?𝑜𝑙𝑑 = 𝑀 × 𝑉?
𝑛𝑒𝑤 + 𝑛?
Both unknown!
• In order to get M, we need to automatically detect a location p as a
reference point such that both 𝑉𝑝𝑛𝑒𝑤𝑎𝑛𝑑 𝑉𝑝
𝑜𝑙𝑑are known
33
![Page 34: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/34.jpg)
Determine M
Observation: The status of an empty area can be detected using a rule-based algorithm
• Reference point (state) = Detecting Empty
• Both before and after the environmental changes
• How? Using variance of CSI magnitude (can be detected automatically)
Associate 𝑉?𝑛𝑒𝑤 with ?=Empty => 𝑉𝑒𝑚𝑝𝑡𝑦
𝑜𝑙𝑑 = 𝑀 × 𝑉𝑒𝑚𝑝𝑡𝑦𝑛𝑒𝑤 + 𝑛
34
![Page 35: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/35.jpg)
Contaminant Removal
𝑀 = 𝑀E𝑚𝑝𝑡𝑦
Contaminated profile (raw CSI measured in the newly changed environment)
𝑀
? ?
De-contaminated (purified) profile• Transformed CSI• In the old domain, i.e. when
there is no environ changes
Trained ML model
Determine Location
35
![Page 36: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/36.jpg)
Contaminant Removal Example
An example of contaminant removal result @ P1
Still some residual errors!
36
![Page 37: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/37.jpg)
Autoencoder
A neural network learns efficient data representation (encoding) of the input data
• Encoder
• Decoder
– To (re)generate data
37
![Page 38: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/38.jpg)
Training the Autoencoder
The autoencoder tries to reconstruct the input
• Training: Inputs and outputs are the same – the fingerprint profiles recorded
• Learns an efficient representation in the recorded domain (fingerprint profiles)
• Use BP to train
Fingerprint profiles
The encoding
Fingerprint profiles
38
![Page 39: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/39.jpg)
Denoising with the Autoencoder (Inference)
• The autoencoder denoises measured new CSI profiles
– Using the (already) purified profiles as input
– Old fingerprint domain features are identified through the coding/decoding
– Features related to the environment changes are diluted/omitted
New (purified) profiles Reconstructed profiles
The encoding
39
![Page 40: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/40.jpg)
Putting All Components Together
The architecture of AutoFi (Training in red, localization in blue).
40
![Page 41: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/41.jpg)
Experiments
![Page 42: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/42.jpg)
Experiment Setup
A Linksys WRT160N router, a laptop with Intel 5300 NIC.
• Wi-Fi traffic: 10 – 20 pings per second.
• Meter-level resolution (minimum distance 80cm)
42
![Page 43: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/43.jpg)
Experiment Setup
Introducing the “contaminants”
43
![Page 44: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/44.jpg)
Experiment Results
Baseline, no mapping was applied, Random Forest (RF).
Opening windows Opening doors
Accuracy Mean Min Mean Min
Baseline 18.8% 0% 41.7% 0%
44
![Page 45: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/45.jpg)
Experiment Results
Using only contaminant removal technique
Opening windows Opening doors
Accuracy Mean Min Mean Min
Baseline 18.8% 0% 41.7% 0%
Contaminant Rmv 69.0% 0% 70.0% 0%
45
![Page 46: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/46.jpg)
Experiment Results
AutoFi (using both contaminant removal and autoencoder)
Opening windows Opening doors
Accuracy Mean Min Mean Min
Baseline 18.8% 0% 41.7% 0%
Contaminant Rmv 69.0% 0% 70.0% 0%
AutoFi 84.9% 47.6% 90.2% 71.3% 46
![Page 47: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/47.jpg)
Brief Summary
• Problem for robust localization
– Small variations in the environment may significantly contaminate the
fingerprints
• Solution - AutoFi
– Reuse the fingerprints and the trained ML model with a contaminant
removal technique
automatically maps the contaminated profiles back to the fingerprint
domain
– Utilize an autoencoder to further denoise the purified profiles
47
![Page 48: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/48.jpg)
More Recent Advances
- Toward Robustness in Real World
![Page 49: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/49.jpg)
More Robust Motion Detection
AutoFi used CSI variance as an indicator of human motion
• CSI variance is not a robust indicator of human motion
• In some empty rooms, CSI variance caused by noise can be even higher than that caused by human motion
• Simply applying white noise filtering does not work
– Because this also removes the variance caused by human motion
Our solution MoFi [to appear in RTSS@Work 2019]
49
![Page 50: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/50.jpg)
More Robust Motion Detection
• Robustness Study on Motion Detection
– Different users
– Different rooms
– Different AP/device placements
– Environment changes, e.g., moved furniture, open/close doors/windows
– Interference from other 2.4GHz devices, e.g., microwave ovens and Bluetooth beacons
50
![Page 51: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/51.jpg)
Addressing User Diversity
Different body shapes yield different CSI profiles
• A system trained on one user may not work for a new user
• Labelling CSI profiles for a new user is mostly impractical
– Requiring user involvement
– Time consuming
• Work under submission and review
51
![Page 52: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/52.jpg)
Wi-Fi Sensing
A new IEEE TIG (Topic Interest Group) on Wi-Fi sensing:
52
![Page 53: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/53.jpg)
Wi-Fi based Motion Detection
Photo credit: Linksys: https://www.linksys.com/us/linksys-aware/ 53
![Page 54: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/54.jpg)
Conclusions & Future Directions
![Page 55: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/55.jpg)
Conclusions and Future Directions
• Wi-Fi based device-free localization is very promising
• Robust ML solutions have been developed for real-life scenarios
– Robust to environment changes: AutoFi
– Robust to random noises: MoFi
– Robust to user diversity: Under submission
• Future: Many more to explore!
– Multi-user localization
– Fast bootstrap/adaptation
– …
55
![Page 56: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/56.jpg)
![Page 57: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/57.jpg)
BACKUP SLIDES
57
![Page 58: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/58.jpg)
Physics Behind This
Signals affected by human body
58
![Page 59: When Machine Learning Meets Wi-Fi](https://reader031.vdocuments.net/reader031/viewer/2022012102/6169f55611a7b741a34d41ee/html5/thumbnails/59.jpg)
Physics Behind This
Different body locations introduce different effects
59