+

Testing Mobile Apps

Joanne and Xidong

+Content

Types of Application on Mobile Browser Application and its testing strategy Extra: current iOS and Android Automation Testing

Software Client Application and its testing strategy Common Testing Technology

+What is the focus of this topic?

To find an proper testing automation that significantly reduces the overall time needed to test each software release.

+3 Categorized Applications

Client Application: Applications are installed on a mobile device and run on

that device, may be written to look and feel like a native application for specific phone models.

Messaging Application: The servers need to receive and respond to the

specifications of SMS messages.

Browser Application: Browser applications are server-based applications that can

be accessed through a web browser via a URL from a mobile device.

+Aspects need to consider for testing (in general) The human languages (e.g., French, Kanji); The locales (e.g., UK English, Australian, and US

English), which affect things like formatting numbers and currency symbols;

Who pays for updates to be downloaded (users may be unwilling to pay to download updates OTA);

How the software is installed on the device (e.g., in terms of security permissions); and

The number of applications and versions you need to support in parallel.

+Goals when testing

To ensure we deliver attractive, easy-to-use, working applications for as many users as practical. Lots of happy, frequent users help show our software is successful and useful.

To have justified confidence in the quality of our software. Ideally, most of the bugs would be found and fixed before the software is widely used.

Fast feedback to developers. Fast feedback helps them to fix the code while it is still “warm,” while they are still intimately familiar with it.

To quickly detect issues so they can be addressed. This is particularly relevant when the problem is related to external factors (e.g., an operator’s network configuration or a specific handset model).

+Markup Language Application on Mobile Application is written HTML and run by browser in

general. But it has revolution for many years…

+Markup Language Application on Mobile (cont.) Flash Lite is a system that enables devices like phones

to make use of a subset of the capabilities offered by Adobe Flash. Flash Lite is used on some smartphones and feature phones to enable things like embedded video viewing on sites such as YouTube.

XHTML Mobile Profile (XHTML MP) is a hypertextual computer language standard designed specifically for mobile phones and other resource-constrained devices. Comparison of XHTML Mobile profile and XHTML basic. http://mobiforge.com/designing/story/comparison-xhtml-mobile-profile-and-xhtml-basic

+Testing Markup Language Application on Mobile Strategy: Using a program or script since Markup

languages are designed to be processed by programs (such as the web browser) rather than being directly interpreted by a human.

Problems: Cookies are sometimes intercepted by a gateway, or a proxy server, provided by the carrier. Automated tests may need to communicate across multiple distinct connections, in parallel, probably across several devices or computers, in order to trigger cookie sharing issues.

+AJAX Mobile Application

Ajax (Asynchronous JavaScript and XML) is a group of interrelated web development techniques used on the client-side to create asynchronous web applications. With Ajax, web applications can send data to, and retrieve data from, a server asynchronously (in the background) without interfering with the display and behavior of the existing page.

+Testing Strategy for AJAX Mobile Application Be able to test rich UI interactions of the application

running within a web browser. (e.g., for an email application new emails may arrive from time to time from the server without any action by the user.)

Testing how well an application copes with intermittent network connectivity can be convoluted and may require an intermediate device. (e.g., to intercept and modify network packets.)

Test specific features that are differentiated from applications running on desktops. (e.g. rotation events)

+What is Client Application?

Applications are installed onto mobile wireless devices. They are able to provide users with more functionality,

better integration, and better performance than browser applications.

Client applications can be divided into two categories: portable applications and native applications. Portable applications run in (or on) a virtual machine, such

as the Java Virtual Machine. (e.g. Android) Native applications are written to run directly on particular

architecture or platform of the target devices. (e.g. iOS)

+iOS Automation Testing

ios-driver Test any IOS native, hybrid, or mobile web application using the Selenium

/ Webdriver API. IOS automation is as easy as automation for a browser, due to reuse of the well known API.

High level features : Implements the JSON wire protocol Runs on emulators and devices Can be run as a regular node in a Selenium Grid Can be used in an existing Selenium Grid to manage device / simulator

farm First class support for localized native apps ( 1 test works for all the

localized versions ) No need to change the source No need to jailbreak the device No ios-driver app to install on the phone

+iOS Automation Testing (cont.)

Appium is an open source test automation framework for use with native and hybrid mobile apps. It drives iOS and Android apps using the WebDriver JSON wire protocol.

+Android Automation Testing

MonkeyRunner: Written in Python; Has APIs for connecting to a device, installing/uninstalling

apps, running apps, taking screenshots, comparing images.

Robotium: A framework provides functional, system and acceptance

tests running across multiple activities; S Supports Activities, Dialogs, Toasts, Menus and Context

Menus, and it is integrated with Maven and Ant for running continuous integration tests.

+Android Automation Testing (cont.) Robolectric (taken a different path):

Uses shadow objects and runs the tests on a regular workstation/server JVM avoiding dexing, packaging, deploying and running the application tested on an emulator or a real device, reducing the time needed to perform tests.

LessPainful (a step further): Offering a service for automated testing of applications on

multiple real devices. The user uploads the application (*.apk) and a test file written in Cucumber, a business-readable DSL, selects the configuration of devices he wants the tests to run on, and then the tests are automatically executed and a report is generated. The list of devices supported include Garmin Asus, several HTCs, LGs, Samsung Galaxy, Sony Xperia, and Motorola Motodefy.

+Techniques for Client Application Automated Unit Tests

iOS application - unit testing framework is available on code.google.com (http://code.google.com/p/google-toolbox-for-mac/) that even supports basic UI testing (http://code.google.com/p/google-toolbox-for-mac/wiki/iPhoneUnitTesting).

Client applications are harder to automate Ideally the test automation would include interaction with

the native system events (e.g., for keyboard and other inputs) and the ability to query the GUI layer (e.g., to read the contents of a text box). Extra features could be provided to enhance our ability to test the application (e.g., where system calls can be intercepted and modified to force certain conditions to be triggered.)

+

Automated Unit Tests (cont.) Example of Unit Tests

Running Unit Tests in Emulator Challenging – capturing output, automating execution of

the tests in emulator

Code Injection Used to obtain information, e.g., statement coverage,

timing data, etc., or it may include automated tests

protected void testBasicAuthForEmptyStrings() {String authResult = BasicAuth.encode("", "");assertEquals("EmptyStrings should result in 4 byte result", 4,

authResult.length());}

Techniques for Client Application

+

Code Injection (cont.) Unique features:

Source code is not required for some forms of code injection, e.g., to add code coverage instrumentation.

The application’s source code does not need modifying Injecting code requires:

The application, The code to inject (which may be tightly coupled to the

implementation details of the application), A way to inject the code, and Instructions on where and how to inject the code.

Techniques for Client Application

+

Test Tools for Client Applications On-Device debugging

Not really a way to automate the testing Emulators

Blackberry Emulator: supports automation Command-line tools are used to enter commands, and to

create screenshots Challenges:

Driving the user-interface. Interpreting screen-responses, including stitching together

images for pages that require scrolling, pattern matching where the content varies from request to requests.

Latency in the device interaction

Techniques for Client Application

+Common Testing Techniques

GUI-level Automation Challenges (in terms of making the tests relatively reliable):

The need for high contrast, opaque, block colors for backgrounds, etc., to make images

easier to match for menus, etc. Deciding whether to use image pattern-matching or OCR (Optical

Character Recognition) . Structuring test resources, e.g., you may need a set of images

per device.

Image Stitching Used to assemble a number of screenshots into a composite single

image that represents the original page. Useful when the page is too large to be displayed on a physical

phone

+

Image Stitching (cont.) Step 1: Image retrieval:

Step 2: Image alignment:

Common Testing Techniques

+

Image Stitching (cont.) Step 3: Image subdivision and election:

Step 4: Image reconstruction:

Common Testing Techniques

+ Image Stitching (cont.)

Challenges of image matching: Captured images may vary significantly from device

model to model: Content varies across screen sizes and resolutions

Common Testing Techniques

+ Image Stitching (cont.)

Optical Character Recognition (OCR): to extract text from images

Encoding Data in Pixels: Each character of ASCII: 8 bits =>HEX “0xNN” (N= 0~F)

E.g. “A” = 0x41, “a” = 0x61

Common Testing Techniques

+ “Contact Sheet”

multicolumn format of rows of small images: screenshots of results generated by automated “user” input

Using Transparency Masking:

Common Testing Techniques

+ “Contact Sheet” (cont.)

Using Transparency Masking:

Common Testing Techniques

+ “Contact Sheet” (cont.)

Assisting test automation with human decision making:

Common Testing Techniques

+ Model Based Testing (MBT)

used to test other software domains such as web sites, desktop applications, etc.

helps to automate longer-running automated tests used MBT techniques to test a number of mobile wireless

applications, including a search engine and client applications.

Common Testing Techniques

+When to test manually

UI and rendering problems: alignment issues, flickering, etc.

When the UI is changing frequently, test scripts tend to break and need repairing, requiring lots of work

When content is dynamic and hard to predict When the cost of automating is likely to significantly

exceed the costs of manual testing

+How to improve Testability Generally:

Provide a complete API to enable the test automation code to test “below” (without) the GUI.

In programming languages such as Java consider making methods and data protected.

Browser based application: Add ID tags to the main elements the automated tests will

interact with

Client Applications Consider using high-contrast colors, visual markers, and even

pixel-encoding to make GUI-based automation more reliable

SMS Applications Provide a scriptable library or interface (e.g., available from

Python) to make tests easier to create.

+Reference

http://mobiforge.com/designing/story/comparison-xhtml-mobile-profile-and-xhtml-basic

http://www.mobileburn.com/definition.jsp?term=Flash+Lite http://en.wikipedia.org/wiki/AJAX http://ios-driver.github.io/ios-driver/index.html http://appium.io http://appium.io/getting-started.html#hacking-with-appiu

m-ios

http://www.infoq.com/news/2011/09/Android-Testing