mobile app test attacks to efficiently explore software

Mobile App Test Attacks to Efficiently Explore Software

Jon D. Hagar, Consultant, Grand Software Testing

[email protected]: Software Test Attacks to Break

Mobile and Embedded Devices

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 1

Gaming Testing Story

It only takes a few minutes using an App before users like or hate it

Worse than that. . . Many users will post a social media review of the app

You don’t want to be a BAD

Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”

2

The Mobile Opportunity

Depth

Passion

Speed

What Does it Take to be a Great

Mobile App Tester?

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices

3Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices

3

As the names imply, these are devices—small, held in the hand, connected to communication networks, including Cell and smart phones – apps Tablets Medical devices

Typically have: Many of the problems of classic embedded systems The power of PCs/IT More user interface (UI) than classic embedded systems Fast and frequent updates

However, mobile devices are “evolving” with more power, resources, apps, etc.

Mobile is the “hot” area of computers/software Testing rules and concepts are still evolving Now starting to include IoT

You know what they are right?

Mobile?


Requirements verification checking Necessary but not sufficient

Risk–based testing Tried and true in many contexts including mobile, but we

need more

We need to do more as testers

We Need Better App Testing


Management directed “No testing”

Dev-ops without enough “thinking” of context and risk to find the BUGS that “count”

Stupid requirements verification checking without GOOD supporting test activities

Testing without thinking of cost schedule users

Current Situation in Mobile Projects

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”

6

From Wikipedia:

Taxonomy is the practice and science of classification. The word finds its roots in the Greek τάξις, taxis (meaning 'order', 'arrangement') and νόμος, nomos ('law' or 'science'). Taxonomy uses taxonomic units, known as taxa (singular taxon). In addition, the word is also used as a count noun: a taxonomy, or taxonomic scheme, is a particular classification ("the taxonomy of ..."), arranged in a hierarchical structure.

The attacks of this session are based on a researched TaxonomyCopyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”

7

Lets look for bugs, but where?

A pattern (of testing) based on a common mode of failure seen over and over Part of Exploratory Testing May be seen as a negative, when it really is a positive Goes after the “bugs” that may be in the software May include or use classic test techniques and test

concepts Lee Copeland’s book on test design Many other good books

A Pattern (more than a process) which must be modified for the context at hand to do the testing

Testers learn mental attack patternsworking over the years in a specific domain

Apply Attack-based TestingWhat is an attack?


A Sampling of Attacks (from Software Test Attacks to Break Mobile and Embedded

Devices) Attack 1: Static Code Analysis Attack 2: Finding White–Box Data Computation Bugs

Attack 3: White–Box Structural Logic Flow Coverage Attack 4: Finding Hardware–System Unhandled Uses in

Software Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs Attack 6: Long Duration Control Attack Runs Attack 7: Breaking Software Logic and/or Control Laws Attack 8: Forcing the Unusual Bug Cases Attack 9 Breaking Software with Hardware and System

Operations 9.1 Sub–Attack: Breaking Battery Power Attack 10: Finding Bugs in Hardware–Software

Communications Attack 11: Breaking Software Error Recovery Attack 12: Interface and Integration Testing 12.1 Sub–Attack: Configuration Integration Evaluation

Attack 13: Finding Problems in Software–System Fault Tolerance

Attack 14: Breaking Digital Software Communications

Attack 15: Finding Bugs in the Data Attack 16: Bugs in System–Software Computation Attack 17: Using Simulation and Stimulation to Drive

Software Attacks Attack 18: Bugs in Timing Interrupts and Priority Inversion Attack 19: Finding Time Related Bugs

Attack 20: Time Related Scenarios, Stories and Tours

Attack 21: Performance Testing Introduction Attack 22: Finding Supporting (User)

Documentation Problems Sub–Attack 22.1: Confirming Install–ability Attack 23: Finding Missing or Wrong Alarms Attack 24: Finding Bugs in Help Files Attack 25: Finding Bugs in Apps Attack 26: Testing Mobile and Embedded Games

Attack 27: Attacking App–Cloud Dependencies

Attack 28 Penetration Attack Test Attack 28.1 Penetration Sub–Attacks:

Authentication — Password Attack Attack 28.2 Sub–Attack Fuzz Test Attack 29: Information Theft—Stealing Device

Data Attack 29.1 Sub Attack –Identity Social

Engineering Attack 30: Spoofing Attacks Attack 30.1 Location and/or User Profile Spoof

Sub–Attack Attack 30.2 GPS Spoof Sub–Attack Attack 31: Attacking Viruses on the Run in

Factories or PLCs Attack 32: Using Combinatorial Tests Attack 33: Attacking Functional Bugs


Attack 1: Static Code Analysis (testing)

When to apply this attack? After/during coding

What faults make this attack successful? Many Example: Issues with pointers

Who conducts this attack? Developer, tester, independent

party Where is this attack conducted?

Tool/test lab How to determine if the attack

exposes failures? Review warning messages and find

true bugs

How to conduct this attack? Obtain and run tool Find and eliminate false

positive Identify and address real

bugs Repeat as code evolves

Single unit/object Class/Group Component Full system

10

Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

Attack 2: Finding White–Box Data Computation Bugs


What faults make this attack successful? Mistakes associated with data Example: Wrong value of Pi

Who conducts this attack? Developer, tester, independent party

Where is this attack conducted? Development Tool/test lab

How to determine if the attack exposes failures? Structural-data test success criteria

not met

How to conduct this attack? Obtain tool Determine criteria and

coverage Create test automation with

specific values (really a programing problem) NOT NICE NUMBERS

Run automated test cases Resolve failures Peer check test cases Repeat as code evolves


Attack 3: White–Box Structural Logic Flow Coverage


What faults make this attack successful? Many Example: Statement coverage

Who conducts this attack? Developer, tester, independent

Where is this attack conducted? Tool/test lab

How to determine if the attack exposes failures? Coverage not met and/or success

criteria fails

How to conduct this attack? Obtain tool Determine criteria and

coverage Create test automation with

specific values to drive logic flow within code

Run automated test cases Resolve failures Peer check test cases Repeat as code evolves


Attack 4: Finding Hardware–System Unhandled User Cases

When to apply this attack? Starting at system-software analysis

What faults make this attack successful? Lack of understand of the world Example: Car braking on ice

Who conducts this attack? Developer, tester, analyst

Where is this attack conducted? Environments, simulations, field

How to determine if the attack exposes failures? An unhandled condition exist Note: data explosion problem

How to conduct this attack? Knowledge Out-of-box thinking Operation Concepts Analysis Modeling Lab testing Field testing Feedback Repeat


Attack 22 and 24: Finding Supporting (User) Documentation and Help File

Problems

14

When to apply this attack? As soon as user documents exist

What faults make this attack successful? Incorrect information about how

to “use” the app Who conducts this attack?

Tester, independent party, stakeholders

Where is this attack conducted? Conduct on the online or

hardcopy documents How to determine if the

attack exposes failures? Follow the instructions exactly

and determine if system works

How to conduct this attack? Access the documentation Use instructions to create a user

story Play the role of different

personas Consider giving the

documentation to a independent party

Repeat as document and systems change


Attack 22.1: Confirming Installability

15

When to apply this attack? When installation is available

What faults make this attack successful? “Missing” part and/or incorrect

configurations Configurations of hardware and

software may not support the app (Device fragmentation)

Who conducts this attack? Tester, independent party

Where is this attack conducted? Tool/test lab, field

How to determine if the attack exposes failures? System fails to install or run

correctly after install

How to conduct this attack? Obtain “clean” device/system (s) Identify load procedures Note: if doing device configuration

operability test use of techniques such as combinatorial or market penetration identification may be needed

Define test strategy and plan Define test design Automate if needed Execute test (follow load procedures) Confirm load and use configuration Repeat as needed


Attack 23: Finding Missing or Wrong Alarms

When to apply this attack? Device has alarms or

information notifications to drive user interaction

What faults make this attack successful? Time or other interactions cause

notification-alarm to be missed Who conducts this attack?

Tester, independent party Where is this attack

conducted? Tool/test lab, field

How to determine if the attack exposes failures? Alarm is missed or wrong

How to conduct this attack? Define alarms and conditions Define risks of alarms in usage

and time Define strategy and test plan Define use cases Define test design within

environments including time Run tests Review for missing/wrong alarms

and cases to “force” Leap year


When to apply this attack? …all the time What faults make this attack successful? …apps can be quite

complex Example: Games-Entertainment ( 40-60 % of downloads)

Who conducts this attack? Test Team A-B “user” testing (crowd, Beta, early releases in continuous

integration/Deployment, etc) Where is this attack conducted? …throughout lifecycle and in

environments How to determine if the attack exposes failures?

Unhappy “users” Bugs found See checklist

Attack : Testing Usability

Credit to Jean Ann Harison2013Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

The developer(s)—see Attacks 1, 2, and 3. The app architect or director On-team tester(s) In-company “dog food” testers Independent test players Mass beta trials Not a tester—Finally, consider who should not be

playing

Note on roles: During the testing effort and as it progresses, don’t forget that there are many different user roles

Exercise: WHAT ARE THE ROLES?

Roles in Usability


Refine checklist to context scope Define a role

Watch what is happening with this role Define a usage (many different user roles)

Guided explorations or ad hoc Stress, unusual cases, explore options Capture understanding, risk, observations,

etc. Checklist (watch for confusion)

Run Exploratory Attack(s) Run A-B statistical Test with monitoring

Learn Re-plan/design

Watch for Bias Switch testers

Repeat

Usability Attack Pattern


Apply when the device is mobile and has Account numbers User-ids and passwords Location tags Restricted data

Current authentication approaches in use on mobile devices Server-based

Registry (user/password) Location or device-based Profile-based

My Personal Pet CauseSecurity Attacks


Attack 28 Penetration Attack Test Attack 28.1 Penetration Sub–Attacks: Authentication —

Password Attack 28.2 Sub–Attack Fuzz Test Attack 29: Information Theft—Stealing Device Data Attack 29.1 Sub Attack –Identity Social Engineering Attack 30: Spoofing Attacks Attack 30.1 Location and/or User Profile Spoof Sub–Attack Attack 30.2 GPS Spoof Sub–Attack

Security Attacks (Con: only a starting point, a checklist of things to

start with)


Security attacks must be done with the knowledge and approval of owners of the system and software

Severe legal implications exist in this area Many of these attacks must be done in a lab (sandbox) In these attacks, I tell you conceptually how to “drive a car very fast

(150 miles an hour) but there are places to do this with a car legally (a race track) and places where you will get a ticket (most public streets)”

Be forewarned - Do not attack you favorite app on your phone or any connected server without the right permissions due to legal implications

Warnings When Conducting Security

Attacks


There will always be Good, Bad, and Ugly Work with the Good Work to over come the Bad Change the Ugly into good

Understanding your local context and error patterns is important

(one size does NOT fit all)

Attacks are patterns…you must still THINK and tailor

Wrap Up of this Session


James Whittaker (attacks) Elisabeth Hendrickson (simulations) Lee Copeland (techniques) Brian Merrick (testing) James Bach (exploratory and tours) Cem Kaner (test thinking) Jean Ann Harrison (her thinking and help)

Many teachers Generations past and future Books, references, and so on

Notes: Thank You (ideas used from)


“Software Test Attacks to Break Mobile and Embedded Devices”

– Jon Hagar

“How to Break Software” James Whittaker, 2003 And his other “How To Break…” books

“A Practitioner’s Guide to Software Test Design” Copeland, 2004 “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al.,

1993 “Computer Related Risks”, Neumann, 1995 “Safeware: System Safety and Computers”, Leveson, 1995 Honorable mentions:

“Systems Testing with an Attitude” Petschenik 2005 “Software System Testing and Quality Assurance” Beizer, 1987 “Testing Computer Software” Kaner et. al., 1988 “Systematic Software Testing” Craig & Jaskiel, 2001 “Managing the Testing Process” Black, 2002

Book/Notes List (my favorites)


• www.stickyminds.com – Collection of test info• www.embedded.com – info on attacks www.sqaforums.com - Mobile Devices, Mobile

Apps - Embedded Systems Testing forum

• Association of Software Testing– BBST Classes

http://www.testingeducation.org/BBST/

• Your favorite search engine

More Resources

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices

mobile app test attacks to efficiently explore software

Software

mobile devices

software jon

mobile app test attacks

embedded devices copyright

mobile opportunity

hagar grand software

llc software test attacks

mobile projects copyright