proficience - sqam - module 3b

8/14/2019 Proficience - SQAM - Module 3b

1/26

SQAM Course, Proficience, IISc 1

Module 3: Software TestingModule 3: Software Testing

3.1 Fundamentals of Software Testing

3.2 Testing Strategy and Planning

3.3 Testing Methods3.4 Economics of Testing

3.5 Testing Metrics

3.6 Defect Tracking3.7 Templates and Checklists

3.8 Case Study


2/26


TestingTesting

Goal is to find faults

What kind of faults?

Algorithmic

Computation and Precision

Documentation

Overload

Capacity

Timing

Performance Recovery

System

Standards


3/26


Codes logic does not produce the proper output for a

given input Some examples

Branch under wrong conditionsFail to set loop invariant

Fail to initialize variablesComparing inappropriate data types

Formula is implemented incorrectly

Computation not performed to the required accuracy

Data structures filled past their capacity Buffer overflow

Dimension of tables larger than code can handle

Length of queue too great

ExamplesExamples


4/26


Some more..Some more.. Performance becomes unacceptable as activity reaches system limit

E.g., system capable of handling 1,000,000 records becomes slower andslower as more records are added

May be examined in relation to disk access, number of processes, etc. System does not perform at the speed prescribed by the requirements Problem with underlying hardware or operating system

Disk failure Out of memory Process table full

Software doesnt adequately handle the system fault Interface

Fault in interacting with another component or driver Checking

Fault in logic that fails to validate data properly before it is used Build/Package/Merge

Fault that occurs because of problems in repositories and version control


5/26


Testing TypesTesting Types

Goal is to find the faults

Different types of testing

Unit testing Integration testing

Functional testing

Performance testing Acceptance testing

Installation testing


6/26


Testing PhasesTesting Phases

Requirements

Specification

System

Specification

System

Design

Detailed

Design

Unit code and

Test

Sub-system

Integration test

System

Integration test

Acceptance

TestService

Acceptance

Test Plan

System

Integration

Test Plan

Sub-system

Integration

Test Plan


7/26


Unit TestingUnit Testing

The first opportunity to test whether a particular module orcomponent is complete

Often performed by the developer

Can choose test cases in many ways. Some sample to get

broad test coverage: Statement testing : Every statement is executed at least once

Branch testing : Every decision point is seen at least once.

Path testing : Variation of above, each path through the

code tested at least once (visualize code as a flow chart) Definition-use path testing : Every path from every

definition of every variable to every use of that definition is

exercised


8/26


Integration TestingIntegration Testing

Integration

Piecing together

individual components

to make a workingsystem

Individual components

should already have

passed their unit tests

Integration Testing Tests to ensure the

system functions when

components pieced

together

Bottom-up

Low-level first, with test

harness

Top-Down Stubs

Big-Bang

Put them together and see if

it works

Sandwich System viewed as three

layers

Top-down in top layer,

bottom-up in lower,

converge in the middle


9/26


Top-Down Integration TestingTop-Down Integration Testing

Modules are integrated by

moving downward

through control hierarchy

Modules subordinate to

main control module are

incorporated

Depth-first

Breadth-first

M1

M3 M4M2

M7M6M5

M1


10/26


Top-Down Integration TestingTop-Down Integration TestingStrategy:

Verifies major control or decision early in the test process. Allows early recognition of problems Depth first strategy allows a complete function of the software to be

implemented Stubs replace lower-level modules so no significant data can flow upward

Delay some tests until have actual modules Add code to simulate module

Steps:

The main control module is used as a test driver, and stubs are substituted forall components directly subordinate to the main control module.

Depending on the integration approach selected, subordinate stubs arereplaced one at a time with actual components.

Tests are conducted as each component is integrated. On completion of each set of tests, another stub is replaced with a real

component . Regression testing may be conducted to ensure that new errors have not been

introduced.


11/26


Bottom-Up IntegrationBottom-Up Integration Begins construction and testing with atomic modules

Need for stubs is eliminated

Operational modules tested thoroughly

Steps Low-level components are combined into clusters (builds) to perform specific sub-function

A driver is written to coordinate test case input and output

Cluster is tested

Drivers are removed and clusters are combined moving upward in program structure.

Level N Level N Level N Level N Level N

Level N-1 Level N-1 Level N-1

Test drivers

Test

drivers

Testing

sequence


12/26


13/26


Comparison of Integration StrategiesComparison of Integration Strategies

MediumEasyHardEasyAbility to test

particular paths

MediumHighLowMediumWork parallelism at

beginning

YesNoYesNoStubs needed

YesYesNoYesComponent drivers

needed

EarlyLateEarlyLateTime to basic working

program

EarlyLateEarlyEarlyIntegration

SandwichBig-BangTop-DownBottom-Up


14/26


System TestingSystem Testing

Software incorporated with hardware, people, information to form

system

System testing is a series of different tests whose primary purpose is to

fully exercise the computer-based system

Types Recovery testing

Security testing

Stress testing

Performance testing


15/26


Validation (Product) andValidation (Product) and

Acceptance TestingAcceptance Testing

Validation test

Series of tests that demonstrate conformity with requirements

Test plan and procedure ensure that

all functional requirements are satisfied

all behavioral characteristics achieved all performance requirements attained

documentation is correct

usability and other requirements are met

Acceptance test Alpha test: test at developers site by end-users

Beta test: test conducted at end-user sites


16/26


Results of Acceptance TestsResults of Acceptance Tests

Ideally: acceptance testing uncovers discrepancies in requirements

Reality: by working with the system users discover aspects of the

problem which they were not aware

New requirements New development and features

Other direction:

Some requirements may not be needed


17/26


Installation TestingInstallation Testing

Installing the system at the user site

System-specific configuration

Co-existence test with other software

E.g. does it work with Office software?

Does anything break if the software is

removed?


18/26


Regression TestingRegression Testing

Adding new or changingmodule impacts the system

New data flow paths

established

New I/O may occur

New control logic invoked

Regression testing is re-

execution of subset of tests that

have already been conducted

Ensures changes have notpropagated unintended side

effects

Approaches:Manual testing

Capture/Playback tools: capture test

cases and results for subsequent

playback and comparison

Test suite contains following classes of

test cases:

Representative sample of tests that

exercises all software functions

Focus on functions likely affectedby change

Focus on components that have been

changed


19/26


Test CasesTest Cases

Want a broad set of test cases to cover the range of possible values and

code paths

Closed Box

Apply all possible inputs, compare with expected output accordingto requirements

Includes out of range inputs

Open Box

View codes internal structure

Generate tests based on this structure, e.g. give values on both

sides of an if-then else test


20/26


Who Should Test?Who Should Test?

In your case, you (the developer) will test

But the developer is often too close to the code to be objective

and recognize some subtle faults

Difficult for developer to separate implementation structure andrequired function

Independent tester or test team is preferable

Familiar with how to break software

Familiar with testing methods and tools

Users should be involved throughout the process

Might catch missing requirements at early stages


21/26


Automated Testing ToolsAutomated Testing Tools

Programs that will help test your code

Static Analysis, e.g. lint

Code Analysis

Check syntax, of a construct is fault-prone, etc.

Structure Checker

Tool to depict logic flow, check for structural flaws Data analyzer

Review data structures, illegal data usage, improper linkage

Sequence checker

Highlight events in the wrong sequence

Dynamic Analysis, e.g. VTune

Monitor and collect statistics as program is running

How many times a statement was executed, branches covered, memory use,

etc.

Useful for performance analysis to find the hotspots of the code


22/26


How Much Testing?How Much Testing?

How do we know when to stop?

When all tests pass?

When enough tests pass meeting some threshold

Myers (1979)

As the number of detected faults increases, the probability of the

existence of more undetected faults increases

Suggests that if the number of detected faults is decreasing, we

may not have too many left and can stop soon

Fault Seeding

Deliberately add faults to the process

Use their detection as a measure of the remaining faults


23/26


Test MetricsTest Metrics

Effective TestMetrics


24/26


Defect TrackingDefect Tracking

ug defect life cycl

Defect RemovalEffectiveness


25/26


Test Plan Outline

esting Health Car

Testing CRMTesting ERP

Web Testing

Road map fortesting

Case studies, Checklists, Templates

reating Checklistfor testing

oad Plan templat

V&V Template


26/26


ReferencesReferences

Effective Methods of Software Testing,

W.Perry

Managing Testing Process, Rex Black

proficience - sqam - module 3b

Documents