mobile app testing secrets
DESCRIPTION
Most app teams aim for 4 stars. Why not 5? Because delivering and maintaining a high-quality app becomes more challenging every day. The requirements of agile and continuous integration put more pressure than ever on testers and quality-focused developers. Add to that the raw complexity of device and platform fragmentation, new sensors, app store processes, star ratings and reviews, increased app competition, mobile automation frameworks that only half work, and users who expect the app to not just work flawlessly but also be intuitive and even beautiful and fun. Jason Arbon shares app testing secrets gleaned from thousands of testers on hundreds of the world’s most popular apps, and data analytics on millions of apps and hundreds of millions of reviews. Learn how the best teams manage their star ratings and app store reviews. Learn what has been tried and failed. Join Jason to get a glimpse of the future of app testing.TRANSCRIPT
T4
Mobile Testing
5/8/2014 9:45:00 AM
Mobile App Testing Secrets
Presented by:
Jason Arbon
Applause
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Jason Arbon
Applause
Jason Arbon currently leads the labs engineering team at uTest.com and applause.com. In the past, Jason worked on Google Chrome, ChromeOS, Google+, several Microsoft teams including Bing and most importantly WinFS. He coauthored the book How Google Tests Software. Jason has started several open source projects, more recently the opentipmap.com project to codify mobile best practices. Jason is the cofounder and mobile app developer for a personalized discovery, search, and sharing app at herecandy.com. In his free time he’s working on a mobile app quality book, OnAppQuality.com. Free time? Wait, Jason doesn’t actually have free time.
4/26/2014
1
On App Quality
Jason Arbon
Dir. Product, applause.com/utest.com
window on humanity
4/26/2014
2
shaped my thinking
coding eng director
Analytics
1,000,000+ apps
100,000,000+ reviews
Crowd Testing
1,000+ apps
100,000+ Testers
4/26/2014
3
author
most aim for 4 stars
...and then stop shipping.
4/26/2014
4
monsters
“agile”
“continuous
integration”
app store monster
4/26/2014
5
app quality: difficult
user expectations
reviews/stars
location/network
few best practices
fragmented tools, platforms, devices
awesome future
rich appstore data
cloud devices
cloud humans
best practices
moving to agile and CI (finally!)
finally quantify quality
4/26/2014
6
quantify/compare
4/26/2014
7
4/26/2014
8
4/26/2014
9
window on humanity
competitive info
where winning
what to focus on
4/26/2014
10
devices in the cloud
humans in the cloud
4/26/2014
11
agile doesn’t mean stars
Correlation between number
of days since release and star
rating
x-axis: star rating
y-axis: number of days since
release
Pearson correlation: -0.31
(threshold for weak/medium
correlation is 0.3)
top 1k apps (>10k reviews)
agile doesn’t mean stars
Correlation between number
of versions and star rating
x-axis: star rating
y-axis: number of versions
Pearson correlation: -0.28
(threshold for weak/medium
correlation is 0.3)
top 1k apps (>10k reviews)
4/26/2014
12
open tip map
encoding best practices
programmatic access
data triggered tips
open
4/26/2014
13
window on humanity
contact
@jarbon
best practices for book
tips
4/26/2014
14
far future: no backlog
4/26/2014
15
app-differencing
monkey-bots compare every build with the last
screenshots of differences sent to developers
and testers in real time
Analytics brought to you by...
Matthew, Heidi and Jason Stredwick and Joe