prioritizing the devices to test your app on: a case study of android game apps
TRANSCRIPT
![Page 1: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/1.jpg)
Prioritizing The Devices To Test Your App On:A Case Study Of Android Game Apps
Hammad Khalid
Emad Shihab
Mei Nagappan
AhmedHassan
![Page 2: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/2.jpg)
The Android Ecosystem has huge number of Stakeholders
1B+ users$4B+ revenues
2
1M+ apps150K+ developers
![Page 3: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/3.jpg)
3
One Challenge faced by developers
![Page 4: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/4.jpg)
4
Android Device Fragmentation
As of Aug 2014 – 19K devices
![Page 5: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/5.jpg)
5
Android Device Fragmentation
As of Aug 2014 – 19K devices
94% of developers not working on Android cited fragmentation as their
primary reason
![Page 6: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/6.jpg)
6
Which Devices should I test my App on?
![Page 7: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/7.jpg)
7
Which Devices should I test my App on?
![Page 8: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/8.jpg)
8
Current Solution
![Page 9: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/9.jpg)
9
Problem with Market Share
![Page 10: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/10.jpg)
10
Market Share does not care about what people are saying about your app.
![Page 11: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/11.jpg)
11
Example - 100 Doors 2013
![Page 12: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/12.jpg)
12
Examine reviews from Motorola Droid X
![Page 13: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/13.jpg)
13
77% of the reviews from Motorola Droid X are Bad reviews
![Page 14: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/14.jpg)
14
77% of the reviews from Motorola Droid X are Bad reviews
The rating for the app is brought down by users of Motorola Droid X
![Page 15: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/15.jpg)
15
Samsung G
alaxy S
3
Samsung G
alaxy S2
Samsung G
alaxy S
Samsung G
alaxy Ace
Samsung G
alaxy Note
Samsung G
alaxy Y
HTC Desire HD
Asus Nexu
s 7
Samsung G
alaxy Tab 10.1
Motorola Droid RAZR02468
10
However Motorola Droid X is not in the top 10 devices by Market Share
![Page 16: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/16.jpg)
16
![Page 17: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/17.jpg)
17
Ratings from Reviews correlated with Downloads
![Page 18: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/18.jpg)
Since Ratings are important
Prioritize Devices based on
Review Share
18
![Page 19: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/19.jpg)
19
Definition
Review Share – The percentage of reviews that an app gets
from a device.
![Page 20: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/20.jpg)
20
Approach to Calculate Review Share
![Page 21: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/21.jpg)
21
Approach to Calculate Review Share
![Page 22: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/22.jpg)
22
Approach to Calculate Review Share
Review Share
3/5 2/5
![Page 23: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/23.jpg)
23
Case Study on 99 Game Apps in Google Play
![Page 24: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/24.jpg)
24
Case Study on 99 Game Apps in Google Play
But why Game Apps?
![Page 25: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/25.jpg)
25Books & Reference
Comics
Education
Finance
Librarie
s & Demo
Media & Video
Music & Audio
Personaliza
tion
Productivity
Socia
lTools
Travel & Lo
cal
Games0
5000100001500020000250003000035000
Most Popular - There are 35K Game Apps with > 500K downloads
![Page 26: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/26.jpg)
26
144K+ Most Useful Reviews
![Page 27: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/27.jpg)
27
Device info present in ~ 90K reviews
![Page 28: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/28.jpg)
28
187 Devices – Fragmentation does exist
Device info present in ~ 90K reviews
![Page 29: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/29.jpg)
29
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
![Page 30: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/30.jpg)
30
Predict the devices to test
your new app on
Do some devices give worse ratings?
% of reviews from a device
![Page 31: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/31.jpg)
31
~20% of the Devices Account for 80% of the Reviews
![Page 32: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/32.jpg)
32
Total - Min 38, Max 132 Devices
![Page 33: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/33.jpg)
33
80% of reviews from just 13 - 45 devices
![Page 34: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/34.jpg)
34
On average, 33% of all devices account for 80% of reviews.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
![Page 35: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/35.jpg)
35
On average, 33% of all devices account for 80% of reviews.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
![Page 36: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/36.jpg)
36
% of bad ratings to all ratings from a device to an app
![Page 37: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/37.jpg)
37
% of bad ratings to all ratings from a device to an app
![Page 38: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/38.jpg)
38
7/10 reviews from Motorola Droid X to 100 Doors app are 1 or 2 Star Ratings
![Page 39: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/39.jpg)
39
% of bad to all ratings from a device to an app for all apps and all devices
Statistical Test
Scott-Knott
![Page 40: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/40.jpg)
40
% of bad to all ratings from a device to an app for all apps and all devices
Statistical Test
Scott-Knott
Grouping of Devices
![Page 41: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/41.jpg)
41
Some devices give worse ratings
![Page 42: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/42.jpg)
42
Some devices give worse ratings
![Page 43: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/43.jpg)
43
Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2
![Page 44: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/44.jpg)
44
Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2
12% - Performance
![Page 45: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/45.jpg)
45
Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2
12% - Performance 6% - UI
![Page 46: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/46.jpg)
46
On average, 33% of all devices account for 80% of reviews.
Statistical evidence that some devices give worse ratings than others.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
![Page 47: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/47.jpg)
47
On average, 33% of all devices account for 80% of reviews.
Statistical evidence that some devices give worse ratings than others.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
![Page 48: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/48.jpg)
48
Learn what devices review 98 of the 99 Game apps
![Page 49: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/49.jpg)
49
Apply it to the remaining app
![Page 50: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/50.jpg)
50
Compare Top 10 devices we predict vs Top 10 actual devices
![Page 51: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/51.jpg)
51
For most apps 7 out of
top 10 devices with most reviews are common
Compare Top 10 devices we predict vs Top 10 actual devices
![Page 52: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/52.jpg)
52
For most apps only 7% of reviews are
missed
Compare Top 10 devices we predict vs Top 10 actual devices
![Page 53: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/53.jpg)
53
On average, 33% of all devices account for 80% of reviews.
Statistical evidence that some devices give worse ratings than others.
App developer can focus testing even before first release
% of reviews from a device
Do some devices give worse ratings?
Can we predict the devices?
![Page 54: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/54.jpg)
54
Take Away
![Page 55: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/55.jpg)
55
Take Away
1. Android Fragmentation is not as bad one would think (in practice).
![Page 56: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/56.jpg)
56
Take Away
2. App developers could use device information from reviews to prioritize their testing efforts
1. Android Fragmentation is not as bad one would think (in practice).
![Page 57: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/57.jpg)
57
All results Generalize to Paid Game Apps…
![Page 58: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/58.jpg)
58
… and to 4 other categories of apps
![Page 59: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/59.jpg)
59
Summary
![Page 60: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/60.jpg)
60
![Page 61: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/61.jpg)
61
![Page 62: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/62.jpg)
62
![Page 63: Prioritizing the Devices to Test Your App On: A Case Study of Android Game Apps](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f1b3ad1a28abd54c8b45e9/html5/thumbnails/63.jpg)
63