automating hybrid apps- which tool and why_unicom
TRANSCRIPT
www.unicomlearning.com
Next Gen Testing Summit-2014
14th Nov, 2014 & Pune
Automating Hybrid Apps Testing: Which tools , Why & how does it affect what you already have ??Speaker Name : Indira Pai, Ayan Modak
Speaker Designation : Lead SQA Engineer, Sr. SQA Engineer
www.nextgentesting.org
Clarice Technologies
www.unicomlearning.com
The Question
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
• You own and run the automation framework of a web based app which is widely used.
• To increase the reach of a widely used web app, the clients decide to
• “Go mobile” and launch a mobile version of the app. • Use “Hybrid App” technology to build the apps.
• As the automation testing owner, you are faced with the challenge of finding the right testing solution for the app in question. How do you go about it?
www.unicomlearning.com
What are Hybrid Apps
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Hybrid apps: • Are written with web technologies (HTML5, CSS
and JavaScript).
• Run inside a native container and leverage the device’s browser engine (but not the browser) to render the HTML and process the JavaScript locally.
• Use a web-to-native abstraction layer that enables access to device capabilities that are not accessible in Mobile Web applications, such as the accelerometer, camera and local storage.
• Are a union of web technology and native execution.
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
• Automation Scope: Unit level or UI and Functional Level?
• Automation Support for WebView
• Support for Various Mobile App Gestures
• Mode of Locating Various Elements in the App
• Support for Multiple OS versions
• Cross Platform Support for Hybrid Apps
• Technological Considerations of the Automation Tool
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
• Automation Scope: Unit level or UI and Functional Level?
• Unit Level: Testing the smallest possible unit which can be isolated in the system.
• GUI and Functional level:
• Not just testing certain apis or GUI flows,
• Testing a collection of related technologies and APIs that together form the new programmatic accessibility story.
• GUI and functional component -based flows which are user centric are automated.
• We have conducted our study for GUI and Functional level automation.
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Automation Support for WebView
• Hybrid apps use a WebView control (UIWebView on iOS, WebView on Android and others) to present the HTML and JavaScript files in a full-screen format, using the native browser rendering engine (not the browser itself).
• WebKit is the browser rendering engine that is used on iOS, Android, Blackberry and others.
• No constraint on using HTML and JavaScript for only in-browser implementations on mobile devices.
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
• Support for Various Mobile App Gestures
• Touch and gesture interactions make mobile experiences easier and more fun to use.
• Core and commonly used gestures like Tap, Double Tap, Drag, Flick, Pinch, Spread, Press, Press and Tap, Press and Drag, Multi-finger Drag should be supported by the tool.
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Mode of Locating Elements in the App
• For automation scripts, the following considerations need to be taken into account.
• Using ID-based automation only may not be the most favourable approach as some popular hybrid app development frameworks may cause dynamic non pattern based ID generation for apps.
• In case there are restrictions as stated above for ID-based automation, XPath and CSS-Selectors are more preferred locators.
• The tool should be flexible in terms of locator support and support maximum locators such as ID, XPath and CSS-Selectors.
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Support for Multiple OS versions
• As the apps would be running on multiple devices with different OS versions, the tool should support the same.
• Limited support for certain OSs will limit benefits of automation.
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Cross Platform Support for Hybrid Apps
• The main advantage of an hybrid app is the cross platform support.
• The tool chosen for automation must therefore, support the same.
• Otherwise, the ROI for time and effort for automating the hybrid app is not optimal.
www.unicomlearning.com
Points to Consider
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Technological Considerations of the Automation Tool
Once the tool has met all of the considerations above, you can further evaluate it on the following parameters:
• Is the tool Open Source?
• Does it have Record and PlayBack support or is hands-on coding needed?
• Are there chances of false alarms (False Negative and False Positive)?
• Is re-usability of scripts on release candidate builds possible?
• Does it provide flexibility of technological know-how ?
www.unicomlearning.com
How we did it @Clarice
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
• At Clarice, we develop and test numerous mobile apps.
• Based on the key parameters mentioned, we evaluated the capabilities of a couple of automation frameworks, like
• Appium
• Calabash
• MonkeyTalk
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Appium
• An open source test automation framework
• Supports native and hybrid mobile apps automation
• It drives iOS and Android apps using the WebDriver JSON wire protocol.
• Technical “need to knows” for this framework:• Need a good understanding of WebDriver.
• For Details refer : http://appium.io/
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Appium Pros:
• No compulsory source code access needed for automation (apart from getting XPaths, which can be done with access to www folder, or using Appium Inspector).
• Element selection using Selenium WebDriver (Selendroid for Android): Support for XPath and CSS selectors, not limited to ID-based automation.
• Final release candidate can be tested without any changes in the .apk/.ipa.
• Since it uses WebDriver, portability of scripts in multiple languages is also possible, which makes it very flexible.
• Supports maximum gestures like tap, flip, swipe, pinch, zoom, and tilt.
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Appium Cons:
• For iOS app, Appium uses Instruments for initialization.
• Instruments itself is a little buggy during initialization, which would cause problems for Appium.
• Reinstalling Xcode, setting dev path correctly can solve this problem… Not exactly an Appium con, more of a configuration overhead
• Record and Playback support for Android:• Although “Record and Playback” support is there for iOS via Appium
Inspector, but
• For Android, it is still issue prone. Appium team encourages the use of uiautomatorviewer, which is an inspector tool provided by Google, that provides similar functionality to the Appium inspector tool.
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Calabash:
• Enables Automated UI Acceptance Tests written in Cucumber to be run on iOS and Android applications.
• Tightly with a Xamarin.iOS and Xamarin.Android project
• The actual tests are written in Gherkin, backed with Ruby code, and run in the context of the Cucumber Framework.
• Initial support for C# and JVM-based languages is also there.
• Technical “need to knows” for this framework:• Elementary Ruby• Elementary Cucumber• BDD knowledge
• For details refer: http://calaba.sh/
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Calabash Pros:
• Powerful WebView support (API support for helper functions like wait, double tap, css selectors query were very good).
• Support for XPath and CSS selectors, not limited to ID-based automation.
• Cucumber support helps to keep the script easy to write and read in terms of user scenarios.
• Supports gestures.
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Calabash Cons:
• Compulsory source code (XCode project) access needed for iOS automation.
• No independent “Record and Playback” feature available (Can be done when integrated with UIAutomation Framework in Instruments for iOS for platforms under iOS7 and ADB screen record in Android).
• Final release candidate for iOS apps cannot be tested as the tests need to run on the build generated by the project (else certain events may not work), and then the final archived build is generated.
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
MonkeyTalk
• A cross-platform testing tool
• Records and plays back test scripts for• Native iOS • Native Android apps• Mobile web apps • Hybrid apps.
• Technical “need to knows” for this framework:• Elementary JS.• Respective framework knowledge in which App was developed
(elementary level).
• For details please refer : https://www.cloudmonkeymobile.com/monkeytalk
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
MonkeyTalk Pros
• No compulsory source code access needed for automation.
• Record and Playback feature.
• Can help in automating redundant steps.
• Can help in tracing back steps to an issue while exploratory testing.
www.unicomlearning.com
Tools Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
MonkeyTalk Cons:
• Issues specific to the MonkeyTalk agent build are encountered:
• App when running with respective MonkeyTalk Agent can lead to issues like random crashes/ performance issues, which do not get encountered when the app is running without the MonkeyTalk Agent.
• Such issues are very difficult to debug and can lead to wastage of time and effort if the issue turns out to be MonkeyTalk build-specific.
• Final release build cannot be tested• The build has to be built with the respective platform’s agent,
• Hence the release candidate will have to be rebuilt without MonkeyTalk agents.
www.unicomlearning.com
Appium
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
MonkeyTalk Cons (continued)• The framework is still very young and issue prone:
• Sometimes the Swipe action was interpreted as Tap, which has to be changed, else it gives error on playback.
• Does not record the correct classes when text was being typed in text fields (needs manual correction).
• Needs code-level knowledge as much as unit level test cases knowledge.
• More suitable for plain JQuery app, not much suited for Angular JS, and Sencha apps.
• Suitable for basic functions, but for adding more scenarios and logic, requires coding, which may not be high return of investment.
• ID based and initial Xpath based automation only:• Difficult to retrieve elements as CSS Selectors are not supported.• XPath support initial level, cannot support all cases in Webview. • Dynamic IDs cannot be automated if they are not pattern based.
www.unicomlearning.com
Case Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
The Subject
• A hybrid app developed in Angular JS and Node JS, built using Apache Cordova for Android and iOS platforms.
• The app has basic screen navigations and mobile gestures like swipe, scroll, pinch, and tap.
• During app development, some UI elements have been assigned an ID, some have dynamic ids, which do not get generated in a pattern.
• Certain sanity flows, for example, login and navigation between screen 1 and screen 2 , and certain gestures like tap, swipe, scroll need to be automated.
• These flows needs to be run on the external release candidates for both Android and iOS builds.
www.unicomlearning.com
Case Study
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Automation Activity Observations and RemarksAppium Calabash MonkeyTalk
Was recording a script as initial automation PoC for the iOS app and playing it on the same app successful?
Yes Yes No
Was recording a script as initial automation PoC for the iOS app and playing it on the Android app successful?
Yes Yes No
Could all Sanity Flows be automated? Yes Yes No
Could we run our sanity scripts on the potential Release Candidate ?
Yes Android: YesiOS: No
No
Conclusion: Did the tool meet all requirements for automating the app in question?
Yes Almost No
www.unicomlearning.com
Conclusion
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
The Conclusion
Although Appium and Calabash both are optimal candidates for hybrid app automation, we recommend to go ahead with Appium due to the following fine winning points:• Webdriver support which allows portability of the test script in multiple
frameworks.
• No dependability on project setup (apart from access to www folder helps in getting XPaths).
• Flexibility of running scripts on app: scripts can be run on the release candidate for the app.
Calabash is a promising alternative, if one is comfortable in Cucumber, Ruby-based test framework. C# and JVM-based languages are also supported.
www.unicomlearning.com
You’ve chosen the tool, now what ?
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Re-Using existing frameworks
After selecting the correct framework for your needs, you need to have the ability to re-use the existing framework you may have created earlier.
This may have been for a Web App or Desktop based app.
How can we make the best of this existing framework and still use the latest capabilities of a framework like Appium ?
www.unicomlearning.com
Clarice Framework – Appium Integration
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
www.unicomlearning.com
Questions
UNICOM PresentsNext Gen Testing Summit-2014
www.nextgentesting.org
Questions
www.unicomlearning.com
THANK YOU
Next Gen Testing Summit-2014Organized by
UNICOM Trainings & Seminars Pvt. Ltd. [email protected]
Speaker name:Email ID: