Addressing Mobile App Testing Challenges

Notes from webinar by Lee Barnes

hosted by QAI onFebruary 13th 2013

This presentation by Maira Bay de Souza is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

Before we begin ...

Items in this font are the notes I took from what the presenter said

Items in this font are my own comments

Mobile testing challenges

Testing in mobile environments has the same challenges of any testing: you need to prioritize.

I like how he's able to have a big picture view of testing.

I've worked in several industries and in companies from large to small. I've noticed that the problems in IT projects are very similar no matter where you go. And the same applies for the problems in testing projects.

Target test environment

When defining the target test environment, consider the difference between the 2 main OSs:

Android: installation of a new OS version is optional and up to the user. There are lots of users still with very old versions installed. There is a wide variety of environments.

iOS: the system auto-updates to new versions. This means that Apple has complete control over the OS in their devices.

He didn't mention it, but the Blackberry OS is like the Android. I know because I have one.

What does this mean for testing?

This means that when you test in iOS, you know that the application will work the same way in all devices (all hardware versions). But the same cannot be said for Android.

If you want your app to work on Android, you have to set up several test environments, with all the different OS versions.

That makes sense. Maybe the solution is to target just one version of Android OS.

Other environment considerations

Devices can also vary in terms of: How fast it the battery consumed? Does it have a camera? What is the screen size?

Unique mobile scenarios (while your app is running)

Incoming call

Low battery warning comes to the foreground, sound is played

Power is lost due to drained battery

An alarm from the calendar app comes to the foreground

Loss of mobile network signal

Transition between mobile networks (including roaming)

Transition from mobile network signal to wi-fi network signal

Keyboard slide

Change of screen orientation (landscape/portrait)

... and much more!

More on environments

Consider the: Type of app. If it's native, it has more/easier access

to the device (integration with camera, GPS, etc). If it runs on a virtual machine (like Java) things may be different.

App usage: if it's used very frequently, it will require more data, which means more network demand and faster battery drain.

Wow, there's so many things to consider when testing mobile apps!

Test Case selection

You don't need to write specific steps because the devices/OS/app changes a lot. It's better to write high-level test cases/test scenarios.

I was a bit surprised by that approach. But on one hand, it makes sense.

Test environment maturity

Tools for desktop testing are vary mature: we have tools to automate, to communicate with each other, to control versions, to capture screenshots, etc.

That is not the case for mobile (yet).

Test environment options

Type of environments: Remote device control Mobile cloud

Both require jailbroken devices.

And for some devices, jailbreaking means loss of warranty.

Continues in Part 2

Disclaimer

The notes presented here are what I understood from what the presenter communicated. They might not be 100% accurate, as I was taking notes and listening to the presentation at the same time.

All the information I am quoting from the presenter is their intellectual property. I am reproducing it here under the fair use policy, for quoting purposes only.