4/23/15  

1  

Designing a Robust Test Strategy for

Mobile Apps

Parimala Hariprasad Tester, Speaker, Author

Who am I?

u  A Curious Tester

u  12+ years experience in eCommerce, Mobile and UX

u  Domain Expertise in CRM, Remote Automation, eCommerce, Real Estate and Healthcare

u  Contributing Author to ‘More Agile Testing’ book

u  First Recipient of ‘Bach Brothers Legion of Testing Merit’ award from James and Jon Bach

u  Rock Star Mom of 2 girls, Foodie and a Horrible Cook

   

4/23/15  

2  

Agenda

•  Mobility Trends •  Four Dimensional Mobile Test Strategy

•  Functional Tests •  Code Level Tests •  Specialized Tests •  Analytical Tests

•  Fifth Dimension •  Real World Testing

•  Summary u  Q & A

4/23/15  

3  

Mobile Vs. Desktop

What is Test Strategy?

“Set of ideas that guide test design and execution“

~ James Bach

4/23/15  

4  

Four Dimensional Mobile Test Strategy

Functional

• Installation • Features • Platforms • Networks • Guidelines

Code Level

• Memory Footprint Analysis • Performance Benchmark • APK De-Compilation • IPA Analysis • Others

Analytical

• Field Testing • Analytics Driven • Social Media • Competitor Analysis • Customer Touch Points

Specialized

• Load / Performance • User Experience • Accessibility • Security • Localization

Functional

8

Installation Features

• App Exploration

• Feature Coverage

• LONG FUN CUP

Platforms

• Platforms / OS •  iOS • Android • Windows

• Device Manufacturers & Models

• Screen Sizes & Resolutionn

• Fragmentation

Network

• Network Speed • 2G, 3G, 4G • WiFi • No Network • Switching • Shared

Network • Cellular • Dead Spots

• Network Carriers

Guidelines

• HIG Design Guidelines

• Android Design Guidelines

• UI Design Do's and Dont's

•  Installability • Onboarding • Uninstallation

4/23/15  

5  

Functional > Features > App Exploration

Functional > Features > LONG FUN CUP

4/23/15  

6  

11  

•  LONG FUN CUP is a heuristic based test model E.g. If an app doesn’t support orientation, tester can skip that step based on the context

•  LONG FUN CUP is a good way to test User Interfaces of mobile apps

•  LONG FUN CUP is the topmost layer of mobile app testing

Functional > Features > LONG FUN CUP

Browsers  

Opera2ng  Systems   Device  Models  

Functional > Platforms > Fragmentation

4/23/15  

7  

Fragmentation

Functional > Platforms > Selecting from an ocean of devices?

•  Singular Approach ü One Device

•  Proportional Approach ü Multiple Devices

•  Shotgun approach

•  Outsourced Approach

hBp://www.tes2ngcircus.com/tes2ng-­‐trapeze-­‐2014-­‐december-­‐edi2on/  

4/23/15  

8  

Functional > Network

• Hardware Ø Smartphone

Ø Tablet

Ø Others?

• Operating System Versions Ø Main and Sub Versions

• Network Providers

15  

•  iOS HIG Guidelines

•  Android Design Guidelines

•  Android UI Design Do’s and Don’ts

Functional > Guidelines

4/23/15  

9  

Code Level

17

Memory Footprint Analysis

• Memory • CPU • Crash • Network

Performance Benchmark

• Battery Usage • Network

Performance

Android De-Compilation

Testing

• Manifest File Testing

• White box Testing • Apps Settings • Developer

Options

Code Level > Android Decompilation Testing

4/23/15  

10  

Code Level > Memory Footprint Analysis

Analytical

20

Field Testing

• Field Testing • Mobile

Personas • User Scenarios

Analytics Driven Testing

• User Analytics • Usage

Analytics • Analytical Tools

• Google Analytics

• Mix Panel • Omniture • Others

Social Media Testing

• Social Sharing • Social

Sentiments

Competitor Analysis

• Feature Comparison

• Platform Comparison

• Technology Comparison

• Search • SWOT

Customer Touch Points

• Supportability Factors

4/23/15  

11  

Analytical > Field Testing

21  

•  80 kms bike trip while the app was streaming music all along the journey

•  3 hour road trip between Bangalore and Mysore

•  Using the app in Garuda Mall, ladies toilet where none of the streaming apps work on 3G

•  Streaming music in Old Airport Road, Bangalore where network gets disconnected automatically in one specific location

•  Testing the app on Wi-Fi about 50m away from the building that hosted the Wi-Fi infrastructure

•  Moving from 2G tower to 3G tower and vice versa over a distance of several kilometers

22  

•  Novice

•  Trespasser

•  First time user

•  Impatient user

•  Tech Savvy •  User of competitor apps / different platform

•  Who else?

Analytical > Field Testing > Mobile Personas

4/23/15  

12  

23  

A person looking to rent on a <Real Estate Mobile App> is a

male graduate, between 25- 30 years on

age looking for a place in Bangalore. He is a

salaried employee of a private company,

earning between <XXX> USD to Rs <YYY> USD

Analytical > Field Testing > User Scenarios

24  

Analytical > User Analytics

4/23/15  

13  

25  

•  What does data tell? ü  Analytics

•  How are apps used? ü  Surveys

•  What do users say? ü  Reviews

Analytical > Usage Analytics

u  Social Sharing u  Context-driven sharing

u  Login/Logout – App Login vs. Social Media Login

u  App Sharing Compatibility across Platforms

u  Data Privacy & Security

u  Social Sentiments u  How do users feel about the product?

u  How do users feel about the social media presence?

Analytical > Social Media Testing

4/23/15  

14  

27  

•  Platform level comparison •  Feature level comparison

•  Technology level comparison

•  Quality Criteria level comparison

Analytical > Competitor Analysis

What’s Sticky About YOUR App when compared to other apps?

4/23/15  

15  

29  

Analytical > Customer Touch Points Testing

Supportability Factors

•  Calls / SMSes

•  Email

•  Chat

•  Service Requests

•  Feedback

•  Field Visits

•  Operational Documentation

•  Service Level Agreements 30  

4/23/15  

16  

Measuring Customer Touch points

Specialized Tests Load /

Performance

• Concurrency • Throughput • Response

Times •  Network

Utilization •  Server

Utilization • Page Load &

Rendering Times

• Performance across Networks

User Experience

• Usability / UX • Real User

Surveys • Think Aloud

Videos • Emotions

Testing •  Supportability

Testing •  Claims Testing •  Paper

Prototyping

Security

• OWASP Guidelines

• Code Review • Threat

Modeling • Vulnerability

Scanning • Tools based

Security Testing

• Crowd Security Experts

Accessibility

• Feature Coverage

• Language Coverage

• Serves Different Types of Specially-abled Users

• User Demographics

• Feature and Behavior Consistency

Localization

• Location based customization

• Language testing

4/23/15  

17  

Introducing Fifth Dimension

Real World Testing

• Platforms • Networks • App Behavior • Localization • User Groups • Security

App ready to Test

Ready to Go-Live

Five dimensional Mobile Test Strategy Framework

Real World Field Testing • Platforms • Networks • App Behavior • Localization • User Groups • Security

4/23/15  

18  

How do you know your Test Strategy is Good Enough?

What type of fish you catch depends on the type of net you use

Selber Schult

Never, under any circumstances, blame yourself for feeling confused by the technology

~ Jonathan Kohl ~

4/23/15  

19  

•  Dhanasekar Subramaniam

•  Jonathan Kohl

•  Karen Johnson

•  Julian Harty

•  Rob Sabourin •  Mayank Mittal

•  Anne Marie Charrett

•  Pradeep Soundararajan

•  Image Credits – The World Wide Web

Acknowledgements

38

@CuriousTester  

Parimala.Shankaraiah@gmail.com  

hBp://curioustester.blogspot.in  

Thank You