its world congress 2014 - performance evaluation of transit data formats on a mobile device
DESCRIPTION
Presented at the 2014 ITS World Congress in Detroit, MI. Over the last decade mobile devices have changed how we access transportation information. However, many past data formats used in information services have been designed without the processing and energy constraints of mobile devices in mind. Newer standards, such as the Service Interface for Real-time Information (SIRI) v2.0, are including mobile-friendly features that reduce the processing overhead on mobile devices. Coupled with the advances in mobile device processing capabilities, many believe that performance is no longer an issue of concern in modern mobile phones. This paper presents an evaluation of the SIRI data format on a mobile device, which indicates that not only is performance still an issue, but that app developers should carefully consider certain software design choices to avoid exposing mobile users to extensive wait times (e.g., wait for real-time transit arrival information). The results also demonstrate that information service providers should always offer mobile-friendly interfaces (i.e., RESTful web services with JSON encoding) when possible. The benchmarking software is made available as an open-source application so that others can perform their own experiments, and so that app developers can use this library as a foundation for building new applications based on the SIRI format.TRANSCRIPT
Center for Urban Transportation Research | University of South Florida
Performance Evaluation of Transit Data Formats on a Mobile Device
Sean J. Barbeau, Ph.D.Tatu Saloranta
10101
10101
10101
2
Overview
• Why does performance matter?
• Evaluation
• Conclusions
3
Choose an app for real-time info:
A B
SLOW
FAST
4
Performance matters
• Users can perceive time differences of 100ms when waiting for a response[1]
• Slower response => more communication with server => shorter battery life
• For time-sensitive tasks, and apps that are run frequently, we want good performance
[1] R. Jota, A. Ng, P. Dietz, and D. Wigdor, "How Fast is Fast Enough? A Study of the Effects of Latency in Direct-Touch Pointing Tasks," presented at the CHI 2013, Paris, France, 2013.
5
Typical architecture for real-time apps
Mobile device requests info from server, server responds
Request(StopId=1)
Response(Arrivals)
6
Typical architecture for real-time apps
Request(StopId=1)
Response(Arrivals)
• What are the response field names?• How is the response encoded?
7
Response Contents
• Field names– Define the names
of data elements– Usually dictated by
(defacto) standards• GTFS• GTFS-realtime• SIRI• TCIP
Bus Stop 1
stop_id
stopId
stop_id
stopId
Field names
8
Response Contents
• Encoding– Defines how the
data is formatted
Bus Stop 1
<stop_id>1</stop_id>
<stopId>1</stopId>
stop_id
stopId
XML
Field names Encodings
JSON
{stop_id: 1}
{stopId: 1}• Focus – XML vs. JSON
encoding• Use MTA Bus Time SIRI
StopMonitoring REST API [1]
[1] http://bustime.mta.info/wiki/Developers/SIRIStopMonitoring
9
Cold vs. Warm starts
• Cold start = first execution
• Warm start = subsequent executions
Initialize Make HTTP Request
Parse Response
Make HTTP Request
Parse Response
No initialization required for warm starts
10
Results
0
2
4
6
8
10
12
14
16
18
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
Elap
sed
Tim
e (s
)XML vs. JSON Parsing Time - All Requests
JSON XML
Request #
Cold Start
11
Min.
Max
.Avg
.
50th perce
ntile
68th perce
ntile
95th perce
ntile
Std dev
.0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
JSON XML
Elap
sed
Tim
e (m
s)XML vs. JSON Parsing Time –
Samsung Galaxy S3
• ~4.3 times longer to parse the first response using XML
• Cold start time is critical -application state is often destroyed when user multitasks (checks email, etc.) in mobile apps
• Samsung Galaxy S5 – similar results
-Using Jackson 2.1.2-Using MTA SIRI REST API StopMonitoring
-See http://goo.gl/EhYSl for details
12
How can we improve cold starts?
• Cold start = first execution
• Instead of initializing, read prev. persisted data
Initialize Make HTTP Request
Parse Response
Read cached data structures
Make HTTP Request
Parse Response
“Pseudo-warm start”
13
0
5
10
15
20
25
Min. Max. Avg. 50th percentile 68th percentile 95th percentile Std dev.
Elap
sed
Tim
e (s
)
XML - Cold, Pseudo-Warm, and Warm Starts - Summary(n = 30)
Cold Starts Pseudo-Warm Starts Warm Starts
14
0
1
2
3
4
5
6
Min. Max. Avg. 50th percentile 68th percentile 95th percentile Std dev.
Elap
sed
Tim
e (s
)
JSON - Cold, Pseudo-Warm, and Warm Starts - Summary(n = 30)
Cold Starts Pseudo-Warm Starts Warm Starts
Noticeable difference to user!
15
Other benefits of Pseudo-warm starts
• Cold start
• Reading cache can begin when app starts!
Initialize Make HTTP Request
Parse Response
Read cached data structures
Make HTTP Request
Parse Response
Can be hidden from user!
Must be done at the time of request
16
0
1
2
3
4
5
6
Min. Max. Avg. 50th percentile 68th percentile 95th percentile Std dev.
Elap
sed
Tim
e (s
)
JSON - Cold, Pseudo-Warm without cache read time, and Warm Starts - Summary
(n = 30)
Cold Starts Pseudo-Warm Starts Warm Starts
17
Future work – XML improvements?
0
1
2
3
4
5
6
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
Elap
sed
Tim
e (s
)
XML vs. JSON Parsing Time - Cached XML List Wrapper Deserializers
JSON XML
Request #
Samsung Galaxy S5 using Jackson v2.4.1
1818
4 15 30 600
5
10
15
20
25
30
7.02
12.68
16.76
19.37
9.44
17.77 18.62
24.01
Using HTTP Increases Battery Life by 28% on Avg.
JAX-RPC HTTP-POSTInterval Between Wireless Transmissions (s)
Batt
ery
Life
(hou
rs)
SOAP
Energy overhead of XML
Motorola i580 phone -See http://goo.gl/hq6nE for details
19
Conclusions
• Performance matters (time and battery life)• Use efficient formats with mobile devices
(e.g., RESTful web services with JSON)• Further optimizations (pseudo-warm starts)
can help hide latency from users• Future work – further XML vs. JSON evaluation
20
Thanks!
Sean J. Barbeau, [email protected] for Urban Transportation ResearchUniversity of South Florida
Tatu [email protected], LLC