cloud-based, automated mobile app testing for the enterprise
TRANSCRIPT
6/2/15
1
Cloud-‐Based, Automated Mobile App Tes5ng for the Enterprise
Joe Schulz, AVP Emerging Technology
Today’s Speaker
Joe Schulz Orasi So:ware
AVP, Emerging Technology [email protected]
6/2/15
2
Today’s Agenda z What is Mobile Tes<ng? z What is Cloud Tes<ng? z How do I Automate for Mobile?
z What is Con<nuous Mobile? z Ques<ons?
What is Mobile Tes5ng?
6/2/15
3
Mobile is BIG
Source: Forbes Magazine, 3/23/2014
7 Billion People in the World
6 Billion have a Mobile Phone
Only 4.5 Billion Have What…?
6/2/15
4
“Only 16% of people will try out an app more than twice…” Digital Trends Magazine, March 2013
“80-‐90% of all apps are used once and then deleted…” Tech Crunch Magazine, March 2013
Source: Dimensional Research, 2014
The New Benchmark: 48% expect a mobile app to respond in two seconds or less
6/2/15
5
Maintaining con<nuity is a huge challenge Layout issue
Device incompaJbility
OS issue Network issue AuthenJcaJon issue
Impact of the network on test results
Capacity > 500+ users
Capacity < 200 users
By not considering actual network condi<ons, testers can get trapped
Without real network condi<ons Real network condi<ons
VS
• Performance tests under ideal lab condiJons can result in a false sense of security that an applicaJon will perform well aQer deployment
• By accurately emulaJng real-‐world network condiJons, organizaJons can improve the load Jme of web assets to mobile devices by up to 80% while driving remediaJon costs down by 30%
6/2/15
6
Why did this happen?
Is it AT&T’s fault?
Is it Apple’s fault?
Is it YOUR fault?
The Cost of Not Tes<ng
Source: New York Times Online, February 2012
“…customers using iPads…actually paid twice”
“…problem began in July but went undetected…unRl December”
“Other mobile apps…immune from the problem”
6/2/15
7
What is Cloud Tes5ng?
Introducing the Orasi Mobile TesRng Maturity Model
Is this YOUR mobile lab?
6/2/15
8
Orasi Mobile Tes<ng Maturity
1: Handheld Manual tesJng on borrowed or purchased mobile devices
5 Stages of Mobile Tes<ng Maturity
1: Handheld � Easy � Expensive
� No AutomaJon
2: Crowdsourcing
Manual tesJng by consumers in return for a small reward
6/2/15
9
5 Stages of Mobile Tes<ng Maturity
1: Handheld � Easy � Expensive
� No AutomaJon
2: Crowdsourcing
� Global � Unreliable � No AutomaJon
3: Emulators SoQware version of mobile OS running on a workstaJon
5 Stages of Mobile Tes<ng Maturity
1: Handheld � Easy � Expensive
� No AutomaJon
2: Crowdsourcing
� Global � Unreliable � No AutomaJon
3: Emulators � Cheap � AutomaJon � Unreliable � Market Delays
4: Tethered
ConnecJng physical devices to a workstaJon with USB or WiFi
6/2/15
10
5 Stages of Mobile Tes<ng Maturity
1: Handheld � Easy � Expensive
� No AutomaJon
2: Crowdsourcing
� Global � Unreliable � No AutomaJon
3: Emulators � Cheap � AutomaJon � Unreliable � Market Delays
4: Tethered
� Real Devices � AutomaJon
� Local � Expensive
5: Cloud ConnecJng to physical devices hosted by a cloud service on a subscripJon basis
5 Stages of Mobile Tes<ng Maturity
1: Handheld � Easy � Expensive
� No AutomaJon
2: Crowdsourcing
� Global � Unreliable � No AutomaJon
3: Emulators � Cheap � AutomaJon � Unreliable � Market Delays
4: Tethered
� Real Devices � AutomaJon
� Local � Expensive
5: Cloud � Real Devices & Networks � Emulators � Scalable � Global Secure � AgnosJc AutomaJon � Cost-‐effecJve � Up-‐Front Cost
6/2/15
11
Device Cradle Device Cradle
Perfecto Mobile
ScriptOnce AutomaRon for Device-‐AgnosRc ScripRng
UFT Mobile plug-‐in for naRve HP ALM integraRon
Secure Web Interface
Internet or WAN
Device Cradle MobileCloud Service
Mobile Labs
6/2/15
12
How do I Automate for Mobile?
Thousands of combina<ons
32 versions of Android released since 2007 450
acJve models of Android mobile phones
21 versions of iOS released since 2007 7
Common operaJng plaborms
5 Major carrier
Networks in the US
14 primary cell phone
manufacturers worldwide
4 wildly popular
browser technologies
12 common mobile scripJng and programming and languages
6/2/15
13
How Big Is The Challenge? Manual Test Cases to Execute 90 iOS Devices to Test (iPhone 5/5S/5C/6/6S iPad 2/3/4/Mini) 9 iOS Versions to Test (iOS 6/7/8) 3 Test Case ExecuJons 2,430 Average Minutes per Test Case ExecuJon 8
40.5 days 4
162 days = 8 FTE
146 44 FTE
52 FTE
Total iOS Manual Test ExecuJon Time Per Build Testable Releases per sprint Total iOS Manual Test Execu<on Time Per Sprint
Android Device VariaJons to Test Total Android Manual Test Execu<on Time Per Sprint Total Manual Execu<on Time for iOS and Android =
x
x
x
Automa<on Objects
Objects are the building blocks of automaJon
Visual (OCR) vs. NaJve object recogniJon
AutomaJon invokes methods and agributes
6/2/15
14
Hybrid Mobile Automa<on § Objects in Automa<on
w NaJve objects preferred but limiJng w Visual (OCR) objects more flexible but
problemaJc
§ Best alterna<ve is hybrid approach mixing na<ve and visual objects to cover all scenarios w Provides direct access to applicaJon objects w Allows interacJon with the device outside
of the custom app
UFT Mobile Automa<on
Full support for advanced UFT funcJonality like run-‐Jme parameters
and data tables
Device-‐agnosJc scripJng allows test reuse across devices
Integrated UFT reporJng provides run results with recordings and
checkpoint screen shots
6/2/15
15
What is Con5nuous Mobile?
Con<nuous Integra<on Frequently merging all developer changes into a shared mainline to avoid merge conflicts
Con<nuous Tes<ng Applying test automaRon to the conRnuous integraRon process
Con<nuous Delivery Extending the automaRon paradigm across the enRre so:ware delivery chain
6/2/15
16
CI/CT/CD for Mobile
Code IDE
Unit Test
Build Tool
Developer Machine
Test Repo
Code Repo
Defect Repo
Development Server
Unit Test
Build Tool
Deploy App
CI Server
Build Server
Mobile Server
Test Tool
Test Server
CI/CT/CD for Mobile
Code IDE
Unit Test
Build Tool
Developer Machine
Test Repo
Code Repo
Defect Repo
Development Server
Unit Test
Build Tool
Deploy App
CI Server
Build Server
Mobile Server
Test Tool
Test Server
Eclipse w/ADT
JUnit Ant
HP ALM
SVN
Jira
JUnit Ant
Cloud Server
Jenkins
Cloud Server
HP UFT
6/2/15
17
C2
Automation never lags by more than one sprint
The “s+1” rule of test automation
Sprint 4 1 Sprint 3 Sprint 2 Sprint 1 Sprint 5
C1
C Component (non-GUI)
G GUI element
B Business process
Stable
C1
G1
C1
G1
G2
C1
G1
C2
G2
C3
B1
C1
G1
C2
G2
C3
B1
G1
C2
Other Considera5ons
6/2/15
18
Public, Private or Mixed? Public Cloud Private Cloud § Devices available first-‐come, first-‐served
§ Select specific handset as needed § ReservaRons can be made in advance
§ Ac<vity cleansing important since devices are shared
§ IniRal state not guaranteed
§ Purchase blocks of minutes
§ Device “cradles” reserved for secure, exclusive use
§ Specify any device to be placed in cradle § Can “swap” devices on request
§ Automa<on requires private cloud § Ac<vity cleansing less important
§ IniRal state consistent with previous session
§ Purchase cradles by the month
How Many Do I Need? § “Must” Devices
§ 2-‐5 devices used for essenJal and regression tesJng § AutomaJon is criJcal
§ “Major” Devices § 6-‐12 devices used for compaJbility tesJng
– Vary models, specs, manufacturers, and networks § AutomaJon less useful but cost-‐effecJve for key tests
§ “Market” Devices § 20-‐40 devices used for acceptance tesJng § Primarily manual tesJng
6/2/15
19
Ques<ons?
If you have any further quesJons, please contact: Orasi Mobile [email protected]
Orasi Somware At A Glance
Our Company · Focus on enterprise soQware quality management
· 1800+ Customers · Partners with Mercury/HP since 1994
Our Technology · HP Test Management and AutomaJon · HP Performance TesJng · HP Mobile Center, NV · Perfecto Mobile, Mobile Labs
2012 HP Support Partner of Year 2013 HP Somware Partner of Year
Our Services · ImplementaJon Services · Install and Mentoring
· Mobile, SAP, Security and Performance TesJng
· Onsite, Rural or Offshore · SoQware Support