geo-location-aware emulations for performance evaluation ...€¦ · • notifications, network...
TRANSCRIPT
![Page 1: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/1.jpg)
@ WONS 2014, Obergurgl, Austria, April 2014
Geo-location-aware Emulations
for Performance Evaluation of
Mobile Applications
Alberto García Estévez
University of Alcala
Niklas Carlsson
Linköping University
![Page 2: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/2.jpg)
• Access to Internet everywhere
• Wireless connectivity
• Increasingly mobile users
• Smart phones and tablets
• Connected (close to) all the time
• Powerful customized applications
• Location-aware app
• Customized services based on location
Customized service
2
![Page 3: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/3.jpg)
• New emerging location-based services
and applications for mobile users
• Many alternative implementations
• Need fair evaluation methodology
Evaluation methodology
3
![Page 4: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/4.jpg)
• New emerging location-based services
and applications for mobile users
• Many alternative implementations
• Need fair evaluation methodology
Evaluation methodology
4
![Page 5: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/5.jpg)
• New emerging location-based services
and applications for mobile users
• Many alternative implementations
• Need fair evaluation methodology
Evaluation methodology
5
![Page 6: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/6.jpg)
• New emerging location-based services
and applications for mobile users
• Many alternative implementations
• Need fair evaluation methodology
Evaluation methodology
6
• Fair head-to-head comparisons …
…. under realistic scenarios
• Repeatable experiments
• Quick and low price
![Page 7: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/7.jpg)
7
Approach Comment Choice
Field tests
Modeling Simulations
Emulation
Evaluation Methodology
![Page 8: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/8.jpg)
8
Approach Comment Choice
Field tests Expensive and does not allow repeatable experiments
Modeling Simulations
Emulation
Evaluation Methodology
![Page 9: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/9.jpg)
9
Approach Comment Choice
Field tests Expensive and does not allow repeatable experiments
Modeling Simulations
Difficult to ensure that abstraction matches reality
Emulation
Evaluation Methodology
![Page 10: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/10.jpg)
10
Approach Comment Choice
Field tests Expensive and does not allow repeatable experiments
Modeling Simulations
Difficult to ensure that abstraction matches reality
Emulation Relatively cheap, real hardware, but we still need methodology for repeatable location-based evaluation …
Evaluation Methodology
![Page 11: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/11.jpg)
11
• Repeatable experiments: Allow head-to-head comparison
• Quick and low price: Can be done in-house
• Realistic scenarios: Use of real mobility patterns and network conditions
… develop simple methodology that allow …
Approach Comment Choice
Field tests Expensive and does not allow repeatable experiments
Modeling Simulations
Difficult to ensure that abstraction matches reality
Emulation Relatively cheap, real hardware, but we still need methodology for repeatable location-based evaluation …
Evaluation Methodology
![Page 12: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/12.jpg)
• Location-aware download scheduler
based on notification service
• Google Cloud Messaging (GCM)
• Mobile app
• HTC wildfire with Android
• Wi-Fi and location service (GPS and
network)
• Application server
• PHP + MySql
• Notifications, network conditions
Example application
12
![Page 13: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/13.jpg)
13
Example application
Registration Notifications
![Page 14: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/14.jpg)
14
Example application
Registration Notifications
![Page 15: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/15.jpg)
15
Example application
1. The server sends a notification to GCM
2. GCM notifies the mobile that an update is available
3. The mobile requests the update
4. The server sends the update
![Page 16: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/16.jpg)
16
Example application
1. The server sends a notification to GCM
2. GCM notifies the mobile that an update is available
3. The mobile requests the update [** geoSmart Scheduler** ]
4. The server sends the update
![Page 17: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/17.jpg)
GeoSmart Scheduler
-- Design and Proof-of-concept Implementation
17
Performance Network Map + Smart Scheduler
![Page 18: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/18.jpg)
GeoSmart Scheduler
-- Design and Proof-of-concept Implementation
18
Performance Network Map + Smart Scheduler
![Page 19: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/19.jpg)
Throughput-location pairs
• HTTP throughput prediction
1. Passively measure throughput when data is downloaded
2. Update prediction using EWMA
• UTM location:
1. Obtain location in latitude/longitude when data is downloaded
2. Convert location to UTM coordinates
Performance Network Map
19
![Page 20: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/20.jpg)
GeoSmart Scheduler
-- Design and Proof-of-concept Implementation
20
Performance Network Map + Smart Scheduler
![Page 21: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/21.jpg)
21
GeoSmart Scheduler
Basic implementation
FIFO Notifications queue using
Threshold based on average path throughput
![Page 22: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/22.jpg)
TRACE-BASED EMULATION
EVALUATION
Evaluation and results
22
![Page 23: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/23.jpg)
Trace-driven emulation
23
Client location and bandwidth conditions
Traces obtained from dataset of real measurements
E.g., commuter traces: bus, ferry, car, train, etc.
(i) Timestamp, (ii) Latitude/longitude, and (iii) bandwidth
Location mocking using Android API features
Create test location service
Network conditions emulated with Dummynet
Server-driven workload
Traces collected using Twitter API
E.g., rate of 3 to 12 notifications per minute
(i) time stamp and (ii) unique ID
![Page 24: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/24.jpg)
Trace-driven emulation
24
Client location and bandwidth conditions
Traces obtained from dataset of real measurements
E.g., commuter traces: bus, ferry, car, train, etc.
(i) Timestamp, (ii) Latitude/longitude, and (iii) Bandwidth
Location mocking using Android API features
Create test location service
Network conditions emulated with Dummynet
Server-driven workload
Traces collected using Twitter API
E.g., rate of 3 to 12 notifications per minute
(i) time stamp and (ii) unique ID
![Page 25: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/25.jpg)
Bandwidth, location, and and workload traces
25
Bus scenario
H. Riiser, P. Vigmostad, C. Griwodz, and P. Halvorsen, “Commute path bandwidth traces
from 3g networks: Analysis and applications,” in Proc. ACM MMSys, Feb/Mar. 2013.
![Page 26: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/26.jpg)
Bandwidth, location, and and workload traces
26
Bus scenario
(a) Bus (b) Ferry (c) Metro (d) Tram
![Page 27: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/27.jpg)
Bandwidth, location, and and workload traces
27
Bus scenario
#topicX
#topicY
Notification traces …
Time
1 2 3 4 5 6 78 9
![Page 28: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/28.jpg)
Naive download speeds
28
Sample file size 100KB
Bus scenario Ferry scenario
![Page 29: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/29.jpg)
GeoSmart Scheduler Results
- Example measure: Average download time
- Three (3) alternative approaches (or grid sizes)
- Four (4) alternative file sizes
Bus scenario Ferry scenario
29
![Page 30: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/30.jpg)
GeoSmart Scheduler Results
- Example measure: Average download time
- Three (3) alternative approaches (or grid sizes)
- Four (4) alternative file sizes
Bus scenario Ferry scenario
30
![Page 31: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/31.jpg)
GeoSmart Scheduler Results
- Example measure: Average download time
- Three (3) alternative approaches (or grid sizes)
- Four (4) alternative file sizes
Bus scenario Ferry scenario
31
![Page 32: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/32.jpg)
GeoSmart Scheduler Results
Bus scenario Ferry scenario
32
- Relatively small improvements (e.g, 10-20%)
![Page 33: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/33.jpg)
GeoSmart Scheduler Results
Bus scenario Ferry scenario
33
- Relatively small improvements
- Better improvements in scenarios with significant
location differences in network performance
![Page 34: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/34.jpg)
• Our emulation framework provides fair-head-to-head
protocol/service comparisons
• Real hardware and realistic mobile scenarios
• Repeatable experiments
• Relatively low cost
• Regards to our proof-of-concept implementation
• GeoSmart scheduler perform better in scenarios with significant location
differences in network performance
• Limited accuracy of EWMA estimator for HTTP throughput
• Choose correct resolution is important
• Future work will consider
• Higher order stochastic models for estimation, adaptive map resolution
(e.g., based on speed of user) with richer information (e.g., based on
network data technology)
Conclusions
34
![Page 35: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/35.jpg)
• Our emulation framework provides fair-head-to-head
protocol/service comparisons
• Real hardware and realistic mobile scenarios
• Repeatable experiments
• Relatively low cost
• Regards to our proof-of-concept implementation
• GeoSmart scheduler perform better in scenarios with significant location
differences in network performance
• Limited accuracy of EWMA estimator for HTTP throughput
• Choose correct resolution is important
• Future work will consider
• Higher order stochastic models for estimation, adaptive map resolution
(e.g., based on speed of user) with richer information (e.g., based on
network data technology)
Conclusions
35
![Page 36: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/36.jpg)
• Our emulation framework provides fair-head-to-head
protocol/service comparisons
• Real hardware and realistic mobile scenarios
• Repeatable experiments
• Relatively low cost
• Regards to our proof-of-concept implementation
• GeoSmart scheduler perform better in scenarios with significant location
differences in network performance
• Limited accuracy of EWMA estimator for HTTP throughput
• Choose correct resolution is important
• Future work will consider
• Higher order stochastic models for estimation, adaptive map resolution
(e.g., based on speed of user) with richer information (e.g., based on
network data technology)
Conclusions
36
![Page 37: Geo-location-aware Emulations for Performance Evaluation ...€¦ · • Notifications, network conditions Example application 12 . 13 Example application ... Basic implementation](https://reader035.vdocuments.net/reader035/viewer/2022071212/6025dc578621f542a54ca78e/html5/thumbnails/37.jpg)
www.liu.se
Alberto García Estévez (UA)
Niklas Carlsson (LiU)
Geo-location-aware Emulations
for Performance Evaluation of
Mobile Applications
Software: www.ida.liu.se/~nikca/papers/wons14.html