software testing (satyam) by jai raj yadav.ppt

7/28/2019 Software Testing (Satyam) by Jai Raj Yadav.ppt

1/105

Software Testing Ver 2.0 Jan, 20011

Software Testing

Jai Raj Yadav


2/105


Software TestingSoftware Testing

What are your expectationsfrom the course?


3/105



Course CoverageCourse Coverage Software Testing Fundamentals Testing & Debugging Role of Testing What Hinders Effective Testing Testing Everything ? Can it be replaced ?

Testing Principles Software testing Requirements Testing Levels/Phases Alpha & Beta Testing Exception & Suspicion Testing

Testing Methodology Testing Strategy Software Testing Phases


4/105



Course CoverageCourse Coverage

How Much to Test Testing Techniques

Testing for Specialized Environments & Applications

Regression Testing

Unit Testing

Integration Testing

System Testing

Acceptance Testing

Causal analysis

S f i


5/105



DefinitionsDefinitions Defect

A deviation from specification or standard Anything that causes customer dissatisfaction

Verification

All Quality Control activities throughout the life cycle that ensure that interim

deliverables meet their input specification Validation

The test phase of life cycle which assures that the end product meets the user

needs.

Testing

Act of showing that the program has bugs / does not have bugs

Debugging

Debugging is the act of attempting to determine the cause of symptoms of

malfunctions detected by testing or by frenzied user complaints.

S f T i


6/105



DefinitionsDefinitions Static Testing

Verification performed without executing the systems codes Code inspection

Reverse engineering

Dynamic Testing

Verification or validation performed by executing the systems code Black Box Test

Testing based on external specifications (functional) without knowledge of how

the system is constructed

White Box Test Testing based on knowledge of internal structure and logic

Usually logic driven

S f iS f T i


7/105Software Testing Ver 2.0 Jan, 20017


DefinitionsDefinitions Functional Test

Tests that validate business functional requirements (what the systemisupposed to do)

Structural Test

Tests that validate the system architecture (how the system was designed /

implemented)

Test Drivers / Stubs

A test driver simulates a calling component or external environmenet

a test stub simulates a called component

S f T iS ft T ti




Testing & DebuggingTesting & DebuggingTesting Debugging

Starts with known conditions, userpredefined procedures, predictable

outcomes

Unknown initial conditions, end cannot be predicted

Should be planned, designed,

scheduled

Can not be constrained

Is demonstration of error / apparent

correctness

Is deductive process

Proves a programmers / designers

failure

Vindicates a Programmer / Designer

Should strive to be predictable, dull,

rigid, inhuman

Demands intuitive leaps,conjectures,

experimentation,freedom

Much can be done withoutknowledge Impossible without design knowledge

Can be done by outsider Must be done by insider

Theory of testing available No theory of debugging is available

Much of test design and execution

can be automated

Can not be automated

S f T iS ft T ti




StatisticsStatistics Introduction of Defects

Requirements - 56% Design - 27%

Coding - 7%

Others - 10%

Testing Time breakup Testing - 20%

Debugging - 80%

80-20 of the Defects -80% of all defects occur in 20% of the work

The Pesticide Paradox First Law - Every method you use to prevent or find bugs leaves a residue

of subteler bugs against which those methods are ineffective

Second Law - Software complexity grows to the limits of our ability to

manage that complexity

S ft T tiS ft T ti




TestingTesting

Testing is the process of exercising or evaluating a system or

system component by manual or automated means to verify that

it satisfies specifed requirements (IEEE 83A)

Testing is a process of executing a program with the intent of

finding an error. Software testing is a critical element of software quality

assurance and represents the ultimate review of system

specification, design and coding.

Testing is the last chance to uncover the errors / defects in thesoftware and facilitates delivery of quality system.

S ft T tiS ft T ti




Role ofTestingTesting Primary Role of Testing

Determine whether system meets specificatios

Determine whether system meets needs

Secondary Role of Testing

Instil confidence

Provide insight into the software development process

Continuously improve the testing process

Why Test

Developer not falliable

Bugs in compilers, languages, DBs , Operating Systems Certain bugs easier to find in testing

Dont want customers to find bugs

Post release debugging is expensive

Good test designing is challenging & rewarding

S ft T tiS ft T ti




What Hinders Effective TestingWhat Hinders Effective Testing What Hinders Effective Testing

Optimism

Belief that the system works

Negative attitude towards effective testing

Ego

Dont want to fail Conflict between testers and developers

testing is least structured

testing is expensive

Delivery commitments

S ft T tiS ft T ti




Testers MaturityTesters Maturity Testers Maturity (as perBiezer)

Phase 0 - no difference between testing and debugging. Test only to

support debugging

Phase 1 - purpose is to show that the software works

Phase 2 - purpose is to show that the software does not work

Phase 3 - purpose is not to prove anything, but to reduce perceived risk ofnot working to an acceptable value

Phase 4

Testing is a mental discipline resulting in low risk software without

much testing effort

Testing as state of mind The goal is testability

Reduce the labour of testing

Testable code has less bugs

S ft T tiS ft T ti




Testing Everything ? Can it be replaced ?Testing Everything ? Can it be replaced ? Is Testing Every Thing ?

There are other approaches to create good software

Effective methods are

inspection

design style

static analysis language checks

development environment

Can Testing be Replaced ?

No, not yet Even if we use methods, cannot do away with the testing

review, inspect, read, walkthrough, better methodologies and then

Test

S ft T tiS ft T ti




Testing PrinciplesTesting Principles Testing Principles

A good test case is one that has a high probability of finding an as-yet

undiscovered error.

A successful test is one that uncovers an as-yet-undiscovered error.

All tests should be traceable to the customer requirements

Tests should be planned long before testing begins Testing should begin in the small and progress towards testing in the large

Exhaustive testing is not possible

Testing Involves

Generate test conditions, cases

Create required test environments

Execute test by initiating application under test & applying inputs that are

specified in the already generated test cases

Compare actual results with expected results

Results : Test passed or failed

S ft T tiS ft T ti




Software Testing RequirementsSoftware Testing Requirements Software testing is carriedout thru all the phases of development.

An effective testing begins with a proper plan from the user requirements

stage itself.

Software testability is the ease with which a computer program is tested.

Metrics can be used to measure the testability of a product

Operability The better the software works, the more efficiently it can be tested.

ObservabilityWhat is seen is what is tested

ControllabilityThe better the software is controlled, the more the testing can be automated

and optimised.

S ft T tiS ft T ti




Software Testing RequirementsSoftware Testing Requirements Decomposability

By controlling the scope of testing, problems can be isolated quickly, and

smarter testing can be performed.

SimplicityThe less there is to test, the more quickly it can be tested

StabilityThe fewer the changes, the fewer the disruptions to testing

Understandability

The more information we have, the smarter we will test.

S f iS ft T ti



Software TestingSoftware TestingTesting Strategies , Levels/PhasesTesting Strategies , Levels/Phases

Testing Strategies

Testing begins at the unit level and works outward toward the integrationof the entire system

Different testing techniques are appropriate at different points of S/W

development cycle.

Testing Levels/Phases

Unit / Component Testing

Focuses on individual software units ( programs) and group of related units

Integration Testing

Focuses on combining units to evaluate the interaction among them

System TestingFocuses on complete integrated system to evaluate compliance with specificified

requirements (test characterstics that are present only when entire system is run)

Acceptance Testing

Done from users perspective to evaluate fitness of use

Soft are TestingSoft are Testing



Software TestingSoftware TestingSoftware Testing PhasesSoftware Testing Phases

IntegrationTest

ReviewedSource

Code

Unit TestedSource

Code

IntegratedSource

CodeUnit Test

DDD UTP

Unit Test Cases

HLD ITP

Integration Test Cases

Review

UTRDefect

Log

Review

ITR



20/105


Software TestingSoftware TestingSoftware Testing PhasesSoftware Testing Phases

URD ATP

AcceptanceTest Cases

AcceptanceTest

IntegratedSource

Code

SystemTestedSource

Code

AcceptedProductSystem Test

SRS STP

System Test Cases

Review

STRDefect

Log

Review

ATR



21/105



Relation of Development and Testing PhasesSSAD Projects:

S.No Development Cycle Phase Type of Testing to be planned1 URD Acceptance Testing2 SRS System Testing3 HLD Integration Testing4 DD Unit Testing

OOAD and Web based Projects:S.No Development Cycle Phase Type of Testing to be planned1 URD Acceptance Testing2 SRS System Testing3 OOADD Class Integration Testing, andClass Testing



22/105



Relation of Development and Testing Phases

Project Management

Software

Development

Quality

ssurance

Coding

Contract Warranty

Users SoftwareRequirements

AcceptanceTesting

SoftwareRequirementsSpecifications

SystemTesting

High LevelDesign

IntegrationTesting

DetailedDesign

UnitTesting

V MODELV MODEL

S ft T tiS ft T ti


23/105


Software TestingSoftware TestingAlpha and Beta TestingAlpha and Beta Testing

Alpha Test At developers site by customer

Developer

looking over the shoulder

recording errors, usage problems

controlled environment

Beta Test at one / more customer sites by end user

developer not present

live situation, developer not in control

customer records problems and reports to developer

Software Testingo tware est ng


24/105


Software Testingo tware est ngException & Suspicion TestingException & Suspicion Testing

Exception Testing

Exceptions handling often crucial for reliable operations, should be tested file errors (empty, missing, overflow)

i-o error handling

arithmetic operations (overflow)

resource allocation (memory)

task communication, creation, purging Suspicion Testing

Consider modules where

Programmer is less experienced

Module having high failure rate

Failed inspection Late change order

Designer / programmer feels uneasy

More extensive tests such as

Multi- condition / Multi-branch coverage

S ft T tiSoftware Testing


25/105


Software TestingSoftware TestingTesting Exercise - 1Testing Exercise - 1

ABC Ltd is a software development company in area of software consultancy &

export. Company had been collecting and analysing project manpower costsbased on time sheets. The processing had been normal till now. Currently, one

of the project group is in development of new integrated computerised system

for Insurance Membership system for overseas client.

The project team has now completed unit & integration testing and are aboutto start system testing phase.

The project Manager has decided to use one months live data as the only data

in system test. The system testers will not be making test plans and preparing

test data based on test plan, but will use the input data of the old computerised

system & compare the outputs of the new system with the outputs of the old

system.

Discuss the advantages & disadvantages of using live data for

system testing as against using planned data.



26/105


Software Testingo tware est ngTesting Methodologies

Testing methodologies are used for designing test cases which provide the

developer with a systematic approach for testing.

Any software product can be tested in one of the two ways:

Knowing the specific function the product has been designed to perform,

tests can be planned and conducted to demonstrate that each function is

fully operational, and to find and correct the errors in it (Blackbox Testing).

Knowing the internal working of a product, tests can be conducted to ensure

that the internal operation performs according to specification and all

internal components are being adequately exercised and in the process,

errors if any are eliminated (Whitebox Testing).



27/105



The attributes of both black-box and white-box testing can be combined

to provide an approach that validates the software interface and also

selectively assures that internal structures of software are correct.

The black-box and white-box testing methods are applicable across all

environments, architectures and applications but unique guidelines and

approaches to testing are warranted in some cases.

The testing methodologies applicable to test case design in different

testing phases are as given below:White-box

Testing

Black-box

Testing

Testing for spe

environments & applicati

Unit Testing IntegrationTesting

System Testing Acceptance

Testing



28/105



Black-box Testing

Black-box tests are used to demonstrate that

the software functions are operational;

input is properly accepted and output is correctly produced;

the integrity of external information (e.g., data files) is maintained.

enables the developer to derive sets of input conditions (test cases) that will fully

exercise all functional requirements for a program.

Black-box testing uncovers errors of the following categories: In-correct or missing functions

Interface errors

Errors in the data structures or external data base access

Performance errors Initialisation and termination errors

Black-box testing is applied during the later stages of the testing as it purposely

disregards control structure and attention is focused on the problem domain.



29/105



Black-box Testing (contd) Test cases are to be designed to answer the following questions:

How is functional validity tested?

What categories of input will make good test case?

Is the system particularly sensitive to certain input values?

How are the boundaries of data input isolated?

What data rates and data volume can the system tolerate? What effect will specific combinations of data have on system operation?

The following black-box testing methods are practically feasible and

adopted depending on the applicability: Graph-based testing methods

Equivalence partitioning

Boundary value analysis



30/105



White-box Testing

White-box testing is designed for close examination of procedural detail byproviding test cases that exercise specific sets of conditions and/or loops tests

logical paths through the software.

White box testing uses the control structure of the procedural design to derive

test cases.

The test cases derived from white-box testing methods will: Guarantee that all independent paths within a module have been exercised at leastones

Exercise all logical decisions on their true and false sides

Execute all loops at their boundaries and within their operational bounds

Exercise internal data structures to ensure their validity.

White box testing need to be adopted under Unit level testing strategy & canbe adapted to a limited extent under integration testing if situation warrants it.

Basis path testing and control structure testing are some of the most widely

used white-box testing techniques



31/105


Software TestingSoftware TestingTest StrategyTest Strategy

Should be developed for each project

Defines the scope & general direction of testing in project High level, prepared together with the project plan

Should answer

When will testing occur ?

What kind of testing will occur ?

What are the risks ? What are the critical success factors ?

What are the testing objectives ?

What are the trade offs ?

Who will conduct the testing ?

How much testing will be done ? What tools, if any, will be used ?



32/105


Software TestingSoftware TestingTest Strategy - SampleTest Strategy - Sample

1. Name of the Project :

Insurance Management System (IMS)

2. Brief Description :

Process new insuarance membership & revisions to existing memberships.

3. Type of Project :

New development(Batch / online application)4. Type of Software :

PL/I, COBOL, IMS DB/DC, DB2 on Mainframe

5. Critical Success Factors Delivery on time

Exceptional user friendliness

Attractive user interface

Reasonable response time for online part



33/105



6. Risk Factors : Development team members new to environment

Limited automated testing tools Thorough Knowledge of American Insurance Business

7. Test Objectives : Insure that no more 1 bug per 10 function point. Max 3 seconds response time

User friendliness of screen / menues / documentations8. Trade Offs :

Objectives stated under 7 to be achieved at any cost. Delivery on time takes precedence over other aspects

9.1 AcceptanceTest (Plan,Test Preparation,Execution) Responsibility - Client/PM/PL Resource budgeted - 6 person month, 3 terminals Planned start date - Jan 16,2001 Planned end date - Feb 28, 2001 Stop criteria - Review and approval



34/105



9.2 System Test (Plan,Test Preparation,Execution) Responsibility - PM/PL

Resource budgeted - 6 person month, 3 terminals

Planned start date - Dec 15,2000 Planned end date - Jan 31, 2001

Stop criteria - Review , approval, all URD functionalty covered

9.3 IntegrationTest Plan (Plan,Test Preparation,Execution)

Responsibility - PL/ML Resource budgeted - 3 person month, 3 terminals

Planned start date - Nov 15,2000 Planned end date - Dec 14, 2000

Stop criteria - Review , approval, 100 design features covered

9.4 Unit Test (Plan,Test Preparation,Execution)

Responsibility - ML/TM Resource budgeted - 6 person month, 3 terminals

Planned start date - Nov 15,2000 Planned end date - Dec 14, 2000

Stop criteria - Review , approval, 100% code coverage proved



35/105


Software TestingSoftware TestingExercise 2 (Test Strategy )Exercise 2 (Test Strategy )

For sample project, estimated LOC is 100,000. Client has specified that number of

major (defined without ambiguity) bugs found during acceptance testing should

not exceed 50. In case it exceeds 50, system will not be accepted, no payment &future projects from the client.

Acceptance test plan will be prepared by client before the start of acceptance test and

will be made available to the team before acceptance starts. Requirement

specifications are provided by client , reviewed by team and queries clarified byclient.

Client will not be willing to be associated in internal reviews of Unit, Integration and

System testing before acceptance phase.

Define a test strategy for the Project IMS



36/105


Software TestingSoftware TestingHow Much to TestHow Much to Test

Is complete Testing Possible ?To prove that a program is free of bug is

Practically impossible, and

Theoretically a mammoth exercise

Barriers We can never be sure of the verification system has been implemented

without any bug No verification system can confirm absence of bugs

We aim at Not absolute proof, but

A suitable , convincing demonstration

Quantitative measures - statistical measure of software reliability Judgement of enough



37/105


Software TestingSoftware TestingHow Much to TestHow Much to Test

Stop Criteria Time runs out - Poor Criteria

Requires specified test design methods e.g. test cases must be derived using

class partitioning and boundry value analysis. Testing stops when all tests

execute without producing any error.

A certain number of errors found - until N numbers of errors found and

corrected.

Requires a certain coverage - e.g. testing stops when all statements and branches

are executed and all test cases execute without failure.

Stop when testing becomes unproductive - e.g.system testing stops when the

number of errors detected per testing person day drops under one.

Use error prediction - e.g. Stop testing when number of bugs left over reduces

to X / 1000 LOC



38/105


Software TestingSoftware TestingExercise 3 ( How Much to Test)Exercise 3 ( How Much to Test)

For sample project, estimated LOC is 100,000. Client has specified that number ofmajor (defined without ambiguity) bugs found during acceptance testing should

not exceed 50. In case it exceeds 50, system will not be accepted, no payment &future projects from the client.

Acceptance test plan will be prepared by client before the start of acceptance test andwill be made available to the team before acceptance starts. Requirementspecifications are provided by client , reviewed by team and queries clarified by

client. Client will not be willing to be associated in internal reviews of Unit,Integration and System testing before acceptance phase.

To be safe side , ABC decided to release software for acceptance when no ofunknown bugs < 35.

Devise a scheme to prdict remaining unknown bugs inthe software.



39/105


Software TestingSoftware TestingTesting techniquesTesting techniques

Testing Techniques are means by which test conditions / cases are

identified. Three broad types of techniques :1. Flow / Coverage based

2. Domain Based

3. Population analysis

Flow / Coverage based Statement coverage

Branch coverage

Condition coverage (unitary )

Multiple condition coverage (compound)

Full path coverage



40/105


Software Testingo tware est ngExercise-4 (Testing techniques)Exercise-4 (Testing techniques)

Illustrate logic coverage in logic below :

IF CODE IS BLANK OR NOT IN DATABASEDISABLE ACCOUNT HISTORY

ELSE

IF NO CREDIT AND AMOUNT < 100

DISABLE ACCOUNT HISTORY

ELSEENABLE ACCOUNT HISTORY

ENDIF

ENDIF

DISABLE SPECIAL REGION

IF REGION = NORTH EASTENABLE SPECIAL REGION

ENDIF



41/105



Domain Based

Domain based testing techniques look at inputs and outputs and derive test casesbased on the analysis of the input & output domains. Steps are :

Identify all inputs

Identify all outputs

Identify equivalence class for each input

Identify equivalence class for each output Equivalence partitioning :Ensure that test cases test each input & output

equivalence class at least once Boundary Value Analysis :For each input equivalence class, ensure that

test cases include

one interior point all extrme point

all epsilon points

Decision Table : Identify combinations of input that result in different

output values



42/105


Software TestingSoftware TestingExercise - 5 (Testing techniques)Exercise - 5 (Testing techniques)

A subroutine takes in subject marks, student type and returns the grades for the

students in that subject.

Inputs : Marks - (0-100)

Student type - First Time / Repeat

Outputs : Grades - F, D, C, B, A

Rules :Student type First Time : 0-40 = F, 41-50 = D, 51-60 = C, 61-70 = B, 71-100 = A

Student type Repeat : 0-50 = F, 51-60 = D, 61-70 = C, 71-80 = B, 81-100 = A

1. Identify test cases that satisfy equivalence

partitioning rules

2. Identify cases based on boundry value analysis

3. Identify cases that satisfy decision tables



43/105



Population analysis Used to identify kinds and frequency of data in production

environment Uses existing production data, could be in various forms

production files, tables manual files files from other similar systems

Could give the tester things not clear in / different form/ additionalto specifications

codes/values used in production

unusual data conditions type, frequency of transactions types of in correct transactions

Production environment also gives model for flow of scripts



44/105


Software TestingSoftware TestingUnit TestingUnit Testing

Unit testing focuses verification effort on the smallest unit of

software code. Five aspects are tested under Unit testingconsiderations:

module interface. local data structure Boundary conditions independent paths error-handling paths are tested.

Unit can be compiled / assembled/ linked/ loaded and put undertest

Unit testing is done to show that the unit does not satisfy thefunctional specification and / or its implemented structure doesnot match the intended design structure

Unit Test PlanUnit Test Plan


45/105


Unit Test PlanUnit Test PlanScope of TestingScope of Testing

Plan for Overall Acceptance Testing in the project

Scope of Testing

Unit test plan identifies the Units/Sub-Units of the software which will be

considered for Unit testing.

Test CoverageLinks each unit being tested with the associated unit test report.

Unit ID ASSOCIATED UNIT TEST REPORT (ID)

Unit Test PlanUnit Test Plan


46/105


Unit Test PlanUnit Test PlanTest CoverageTest Coverage

Test Coverage

Test Coverage Goals

Test Completion Criteria

Specify the test completion criteria to be applied. These may be different

for different units

Test Coverage or X Remarks (If any)Path Coverage

Statement Coverage

Decision (Logic/Branch) Coverage

Condition Coverage

Decision/Condition Coverage

Multiple-Condition Coverage

Unit Test Plan (UTP)n t est an


47/105


Unit Test Plan (UTP)n t est anTest DeliverablesTest Deliverables

Test Deliverables

Test deliverables(documents, reports that must be provided at the end of testing). For example,Test Plan, Test Cases, Input and Output Data (Screen Dumps, Reports etc.) and

any other items used.

Submitted to Whom

the agencies to whom they should be submitted. For example, Project Team, ,Customer etc.

Metrics Collection

The Unit Test Report containing the details of the test runs and defect analysisshall be prepared by each tester. The data from all the Unit Test Reports shall beconsolidated in the Defect Log.

Goto : UTPTemplate

U it T t R t (UTR)U it T t R t (UTR)
http://utp.doc/http://utp.doc/


48/105


Unit Test Report (UTR)Unit Test Report (UTR)

Individual Unit Test Report (UTR) for each of the Units (as

mentioned in DDD) Tested

Test Prerequisites

Execution Details

Wrap-up Details

Test Cases

U it T t C D iU it T t C D i


49/105


Unit Test Case DesignUnit Test Case Design

Test Case Id

Unique identification number which relates to unit or sub-unit

Test Case Description

Identifies test coverage(s) undertaken for this unit test case

Path Coverage

Statement Coverage

Decision (Logic/branch) Coverage

Condition Coverage

Test Case Structure

Test Category

Test No.

Expected Result

Actual result

U it T t C D iU it T t C D i


50/105


Unit Test Case DesignUnit Test Case Design

Test Case Structure (Contd.) Test Category

Type of test

screen layout

data input, etc.

Test No.

Identifies unique serial no. for sub test case required to test individualtest category

Expected Result

Output to be produced

Actual ResultActual result after executing the test case

Goto: Unit Test ReportTemplate

http://utr.doc/http://utr.doc/


51/105


Software Testingo tware est ngIntegration TestingIntegration Testing

Integration testing is a systematic technique for verifying the software structure

while conducting tests to uncover errors associated with interfacing. Integration testing is done to show that even though components were

individually satisfactory, the combination is in correct / inconsistent.

Black-box test case design techniques are the most prevalent during integration,

although limited amount of white box testing may be used to ensure coverage of

major control paths. Top-Down Integration Testing

Top-Down integration is an incremental approach to construction of program

structure.

Modules are integrated by moving downward through the control hierarchy,

beginning with the main control module (main program).

Modules subordinate to the main control module are incorporated into the

structure in either a depth-first or breadth-first manner



52/105



Bottom-Up Integration Testing

Bottom-Up integration testing begins construction and testing with atomicmodules

Since modules are integrated from the bottom up, processing required for

modules sub-ordinate to a given level is always available and the need for

stubs is eliminated.

Integration Testing for Web applications Collaboration diagrams, screens and report layouts are matched to OOADD

and associated class integration test case report is generated

Regression Testing

Testing after changes have been made to ensure that no unwanted changeswere introduced.

Regression testing is the re-execution of some subset of tests that have

already been conducted to ensure that changes have not propagated

unintended side effects.



53/105



Types of integration Problems

Configuration / version control

I/O format , protocol mismatchs

Conflicting data views / usage

data integrity violated

wrong call order / parameters Missing / overlapping functions

Resource problems (memory etc)

I t ti T t PlIntegration Test Plan


54/105


Integration Test PlanIntegration Test Plan

Plan for Overall Acceptance Testing in theproject

Scope of TestingTest Coverage

Test Deliverables

ITPTemplate

http://itp.doc/http://itp.doc/


55/105


teg at o est ateg at o est aScope of TestingScope of Testing

Scope of Testing

ITP identifies the modules, sub modules, interfaces among

modules or sub-modules, and the specific functional,

performance, and internal design characteristics that are to be

tested.

Test Coverage

Test Modules and Associated Reports

Specifies details about each module / sub module and associated

interfaces, and the related Integration Test Reports.Module ID / Sub Module ID/ Interface ID

HLD Reference (ID) Associated Integration

Test Case Report (ID)



56/105


ggTest CoverageTest Coverage

Test Coverage (contd)

Test Coverage Goals

Identify the testing coverage that will be followed


Test Features (as per URD) and associated Test Data

Source of Test Data (Supplied by Customer or generated by

Satyam)

Test Coverage or X Remarks (If any)Equivalence partitioningBoundary-value analysis

Cause-effect graphing

Error guessing



57/105


ggTest DeliverablesTest Deliverables

Deliverables Test Reports

Defect Summary

Severity details

(test deliverables (documents, reports that must be provided at

the end of testing eg, Test Plan, Test Case(s), Input and Output

Data (Screen Dumps, Reports etc.) and any other items used)

Submitted to Whomthe agencies to whom they should be submitted. For example,

Project Team, Quality group, Customer etc.

Integration Test Report (ITR)Integration Test Report (ITR)


58/105


Integration Test Report (ITR)Integration Test Report (ITR)

Individual Acceptance Test Report (ATR) for eachof the Modules / Sub-Modules / Interfaces Tested

(as mentioned in HLDD)

Test Prerequisites

Execution Details

Wrap-up Details

Test Cases

Integration Test Case DesignIntegration Test Case Design


59/105



Test Case ID


Test Case Structure



60/105



Test Case IDUnique identification number which relates to

individual elements of HLD

Test Case DescriptionIdentifies test coverage(s) undertaken

Boundary-value Analysis

Error Guessing

Interface Integrity

Functional Validity

Informal Content



61/105



Test Case StructureTest Category

Test No.

Expected ResultActual Result



62/105



Test Case StructureTest Category

Type of test (Screen Layout, Data Input, etc.)

Test No.

identifies unique serial no. of the sub test case

required to test individual test category



63/105



Test Case Structure (Contd.)Expected Result


Actual Result

Actual result after executing the Test Case

Integration Test ReportTemplate

http://itr.doc/http://itr.doc/


64/105


ggSystem TestingSystem Testing

System testing verifies that all elements mesh properly and the overall

system function/performance is achieved. The aim is to verify all system elements and validate conformance

against SRS.

System testing is aimed at revealing bugs that can not be attributed to

a component as such, to inconsistencies between components orplanned interactions between components.

Concerns : issues , behaviours that can only be exposed by testing the

entire integrated system (eg. Performance, security, recovery etc)

System testing is categorised into the following 15 types.The type(s)

of testing is to chosen depending on the customer / system

requirements.



65/105



Compatibility / Conversion Testing

Where the software developed is a plug-in into an existing system, thecompatibility of the developed software with the existing system has to be

tested.

Configuration Testing

Configuration testing includes either or both of the following: testing the software with the different possible hardware configurations

testing each possible configuration of the software

Documentation Testing

Documentation testing is concerned with the accuracy of the user documentation Facility Testing

Facility Testing is the determination of whether each facility / functionality

mentioned in SRS is actually implemented.



66/105



Installability Testing

Certain software systems will have complicated procedures for installing the

system e.g. the system generation (sysgen) process in IBM Mainframes.

Performance Testing

Performance testing is designed to test run-time performance of software within

the context of an integrated system.

Performance testing occurs throughout all phases testing.

Performance Testing for Web Applications

Performance testing has five manageable phases:

architecture validation, performance bench marking, performance regression

performance tuning and acceptance, and the continuous performance monitoring

necessary to control performance and manage growth.



67/105



Procedure Testing

When the software forms a part of a large and not completely automated system,the interfaces of the developed software with the other components in the larger

system shall be tested.

Recovery Testing

Recovery testing is a system test that forces the software to fail in a variety ofways and verifies that recovery is properly performed.

Reliability Testing

Test any specific reliability factors that are stated explicitly in the SRS.

Security Testing

Verify that protection mechanisms built into a system will protect it from

improper penetration.

Design test cases that try to penetrate into the system using all possible

mechanisms.



68/105



Serviceability Testing

Serviceability testing covers the serviceability or maintainability characteristics ofthe software. The requirements stated in the SRS may include :-

service aids to be provided with the system, e.g., storage-dump programs,

diagnostic programs

the mean time to debug an apparent problem

the maintenance procedures for the system the quality of the internal-logic documentation

Storage TestingStorage testing is to ensure that the primary & secondary storage requirements are

within the specified bounds.

Stress Testing

Stress tests are designed to confront programs with abnormal situations.

Stress testing executes a system in a manner that demand rescues in abnormal

quantity, frequency or volume.



69/105



Usability Testing

Attempt to uncover the software usability problems involving the human-factor

Volume Testing

Volume Testing is to ensure that

the software can handle the volume of data as specified in the SRS

Does not crash with heavy volumes of data, but gives an appropriate message

and/or makes a clean exit.

System Test Plan for the Project

System Test Report

System Test PlanSystem Test Plan


70/105


yyScope of TestingScope of Testing

Plan for Overall System Testing in the project

Scope of Testing

System Test Plan identifies which portions of the system are covered under

the test and the features being tested.

Test Coverage

Test Features (as per SRS)

Links each requirement in the SRS (identified by means of a brief

description) with specific test cases.

Feature Being

Tested

Functional Requirement ID in

SRS

Associated System Test C

(ID)



71/105


yyTest CoverageTest Coverage

Test Coverage Goals

Identifies the testing coverage that will be followed.


Test Coverage or X Remarks (If any)Facility Testing

Volume Testing

Stress Testing

Usability Testing

Security TestingPerformance Testing

Storage Testing

Configuration Testing

Compatibility / Conversion Testing

Installability Testing

Reliability Testing

Recovery TestingServiceability Testing

Documentation Testing

Procedure Testing



72/105


Test DeliverablesTest Deliverables

Test Deliverables

Deliverables

Test Reports

Defect Summary

Severity details

(test deliverables (documents, reports that must be provided at the end oftesting). For example, Test Plan, Test Case(s), Input and Output Data

(Screen Dumps, Reports etc.) and any other items used.)

Submitted to Whom

the agencies to whom they should be submitted. For example, Project Team,Quality department, Customer etc.

Goto:STPTemplate

System Test Report (STR)System Test Report (STR)
http://stp.doc/http://stp.doc/


73/105


System Test Report (STR)System Test Report (STR)

Individual System Test Report (STR) for each of the Features (as

mentioned in SRS) Tested

Test Prerequisites

Execution Details

Wrap-up Details Test Cases

System Test Case DesignSystem Test Case Design


74/105



Test Case ID

Unique identification number which relates to individual elements of SRS


Identifies test coverage(s) undertaken

facility testing

volume testing



75/105



Test Case Structure

Test Category

Type of test

facility test

volume test

Test No.Identifies unique serial no. of sub test case related to individual test category

Expected Result


Actual ResultActual result after executing the test case

Goto :System Test ReportTemplate

Software Testingo tware est ngA t T tiA t T ti
http://str.doc/http://str.doc/


76/105


Acceptance TestingAcceptance Testing

Acceptance tests are conducted at the development site or at the

customer site depending upon the requirements and mutuallyagreed principles to enable the customer to validate all the

requirements as per Requirement Document(URD).

Aims at uncovering implied requirements

Aims at evaluating fitness for use Should not find bugs which should have been found in earlier

phases.

Acceptance Test Plan for the Project

Acceptance Test ReportOne for each Feature to be Tested (as mentioned in URD)

Acceptance Test PlanAcceptance Test Plan


77/105


Acceptance Test PlanAcceptance Test Plan

Plan for Overall Acceptance Testing in the project

Scope of Testing Test Coverage

Test Features (as per URD) and associated Test Data

Source of Test Data (Supplied by Customer or generated by Satyam)

Test Coverage Goals


Test Deliverables Deliverables

Test Reports

Defect Summary Severity details

Submitted to Whom

Goto :ATPTemplate

Acceptance Test ReportAcceptance Test Report
http://atp.doc/http://atp.doc/


78/105


Acceptance Test ReportAcceptance Test Report

Individual Acceptance Test Report (ATR) for each of the

Features (as mentioned in URD) Tested

Test Prerequisites

Execution Details

Wrap-up Details Test Cases

Acceptance Test Case Design


79/105


p g

Test Case ID

Unique identification number which relates to individual Functional-IDs of URD


Identifies test coverage(s) undertaken for this Test case e.g.

Data Entry

Report Generation Test Case Structure

Test Category

Test Number

Expected Result Actual Result

Acceptance Test Case DesignAcceptance Test Case Design


80/105


Acceptance Test Case DesignAcceptance Test Case Design

Test Case Structure

Test Category

Type of test

data entry

report generation

Test NumberIdentifies unique serial no.for each sub-test case related to individual test

category

Test Case Structure

Expected Result ( output to be produced)

Actual Result (actual result after executing the test case)

GO TO :Acceptance Test Report Template

Software Testingo tware est ngRegression TestingRegression Testing
http://atr.doc/http://atr.doc/


81/105


Regression TestingRegression Testing

Re-running of test cases after fix / change / enhancement re-verify all functions of each build of application

no new problem introduced by fix / change (ripple effect)

more complete testing

confidence in quality and stability of build

Bugs remain undetected without full regression test Repaeting the full test catches inadvertently introduced bugs

Regression Testing Testing after changes have been made to ensure that no unwanted changes

were introduced.

Regression testing is the re-execution of some subset of tests that have

already been conducted to ensure that changes have not propagated

unintended side effects.



82/105


Regression TestingRegression Testing Regression Testing recommendations

During Unit Testing

Re-run unit test after every change

During Integration Testing

Re-run unit tests of all changed programs and re-run full

integration tests During System Testing

Re-run unit tests of all changed programs, and

re-run full integration tests & system tests

During Acceptance Testing

Re-run unit tests of all changed programs, and

re-run full integration tests , system tests and acceptance

tests



83/105


Regression TestingRegression Testing

During Post Acceptance Testing

Create special regression Test packs at unit level,

integration level,

system test levels

Consider time for re-running regression tests

To build regression packs (post acceptance) , use

earlier tests (unit, integration)

tests that found bugs

additional tests for bugs found in productions

Software TestingSoftware TestingTesting for Specialized Environments & Applications (WEB)Testing for Specialized Environments & Applications (WEB)


84/105


Testing for Specialized Environments & Applications (WEB)Testing for Specialized Environments & Applications (WEB)

Usability Testing for Web Applications

The intended audience will determine the "usability" testing needs of the Web

site.Take into account the current state of the Web and Web culture.

Link testing forWeb ApplicationsTesting determines if the site's links to internal and external Web pages areworking.

HTML validation for Web applicationsTesting will be determined by the intended audience, the type of browser(s)expected to be used, whether the site delivers pages based on browser type ortargets a common denominator.

Link testing forWeb ApplicationsTesting determines if the site's links to internal and external Web pages areworking.



85/105



HTML validation for Web applications

Testing will be determined by the intended audience, the type of browser(s)expected to be used, whether the site delivers pages based on browser type or

targets a common denominator.

Load testing for Web applications

If there is a large number of interactions per unit time on the Web site testingmust be performed under a range of loads to determine at what point the

system's response time degrades or fails.

When the Web server software and configuration settings, CGI scripts, database

design, and other factors can all have an impact.



86/105




Performance testing has five manageable phases: architecture validation

performance bench marking

performance regression

performance tuning and acceptance

and the continuous performance monitoring necessary to controlperformance and manage growth

. .



87/105




Performance testing must be an integral part of designing, building, andmaintaining Web applications.

Automated testing tools play a critical role in measuring, predicting, and

controlling application performance.

The final goal for any Web application set for high-volume use is for

users to consistently have

continuous availability

consistent response timeseven during peak usage times.



88/105



Performance Testing for Web Applications (contd)

1. Architecture Validation: Use performance tests to validate that the software architecture will deliver

the necessary performance & it will scale linearly as hardware is added to

accommodate future growth.

Testing the scalability of the architecture must happen at the start ofdevelopment, after a prototype of the application has been created and is

able to generate transactions to touch all tiers of the application.

Walkthrough of the logic of the front-end of the application must be done

to ensure that its easy to navigate.

There should be provision for enough memory and processors for theselayers of the architecture to handle the expected user volumes.

If the application requires custom components, the scalability of the core

technologies involved should be tested.



89/105




2. Performance Benchmarking: Consider and define the types of performance tests needed.

Create, run, and analyze the first round of performance tests against theinitial version of the application to provide a set of performance metricscommonly referred to as performance benchmarks.

While considering the types of performance tests to create, start by definingall of the possible types of transactions user audience can initiate.

3. Performance Regression Testing : Perform regression testing by which the Web application is continuously

re-tested and measured against the established benchmark tests to ensure

that application performance hasnt been degraded because of the changesmade.

These tests should be performed when architectural modifications have beenmade during the development of the application .



90/105




4. Performance Tuning and Acceptance : This is the final load-testing phase prior to the acceptance testing in which

all of the different pieces of the Web application are integrated, andperformance is validated.

Different transaction scenarios of real-life usage are emulated, and the

scalability of the final configuration is validated. Ensure that server-clustering techniques provide adequate failover and

bandwidth to handle the amount of concurrent users planned.

5. Continuous Performance Monitoring:

24/7 Performance testing must continue even after the application isdeployed. Typical thresholds for returned requests range anywhere betweensix and ten seconds. Anything outside the response time range shouldtrigger some type of alert.

o tware Test ngo tware est ngGUI - WindowsGUI - Windows


91/105


Testing for Specialized Environments & ApplicationsTesting for Specialized Environments & Applications

Testing GUIs

GUIs consist of number of reusable components to simplify system development ,but their inherent complexity results in the necessity of carefully designed test

cases.

For Windows Will window open properly based on related typed or menu-based commands?

Can it be resized, moved and scrolled? Is all data content contained within the window properly addressable with

mouse, and keyboard?

Are all functions that relate to window available when needed?

Are all functions that relate to window operational?

Are all relevant pull down menus, tool bars, dialogue boxes, and buttons, andother controls available and properly displayed for the window?

When multiple windows are displayed, is the name of window properly

represented?

o tware Test ngo tware est ngGUI - WindowsGUI - Windows


92/105



For Windows (contd)

Is the active window properly highlighted? If multitasking is used, are all windows updated at appropriate time?

Do multiple or incorrect mouse picks within the window cause unexpected

side effects?

Are audio and/or colour prompts within the window or as a consequence of

window operations presented according to the specification? Does the window properly close?

o tware Test ngo tware est ngGUI - Pull Down Menues & Mouse OperationsGUI - Pull Down Menues & Mouse Operations


93/105



For pull down menus and mouse operations:

Is the appropriate menu bar displayed in the appropriate context? Does the application menu bar display system related features?

Do pull-down operations work properly?

Do breakaway menus, pallettes, and tool bars work properly?

Are all menu functions and pull-down sub functions properly listed?

Are all menu functions properly addressable by mouse? Is text typeface, size and format correct?

Is it possible to invoke each menu function using its alternative text based

command?

Are menu functions highlighted based on the context of current operations

within a window? Does each menu function perform as required?

Are the names of menu functions self-explanatory?

o tware Test ngo tware est ngGUI - Pull Down Menues & Mouse OperationsGUI - Pull Down Menues & Mouse Operations


94/105



For pull down menus and mouse operations (contd)

Is context sensitive help available for each menu item? Are mouse operations properly recognised throughout the interactive context?

If multiple clicks are required, are they properly recognised in the context?

Do the cursor, processing indicator, and mouse pointer properly change as

different operations are invoked?

Software TestingSoftware TestingGUI - Data EntryGUI - Data Entry


95/105



For Data Entry:

Is alphanumeric data entry properly echoed and input to the system? Do graphical models of data entry work properly?

Is invalid data properly recognised?

Are data input messages intelligible?

o tware Test ngo tware est ngTesting of Client/ Server ArchitecturesTesting of Client/ Server Architectures


96/105


Testing for Specialized Environments & ApplicationsTesting for Specialized Environments & Applications Testing of client/server software occurs at three different levels:

Individual client applications are tested in disconnected mode - the operation of the

server and the underlying network are not considered. The client software and associated server applications are tested in concert, but

network operations are not explicitly exercised.

The complete C/S architecture, network operation and performance, is tested.

Commonly testing approaches for C/S applications: Application function tests

Server tests

Database tests

Transaction testing

Network communication testing.

Derive an operation profile from client/server user scenario to indicate the

multiple user inter-operation with the C/S system and to provide the pattern ofusage, so that the tests are planned and executed.

Initially a single client application is tested, integration of the clients, server,

and the network is tested next. Finally, the entire system is tested.



97/105


Information to be Recorded in each Testing Phase

Each run shall be recorded in the corresponding Test Report as

Observed behavior

Severity of each defect

Cause of each defect Defect Summary shall be recorded inDefect Analysis Log

Test Reports shall be Peer Reviewed and Approved

Software TestingSoftware TestingCAUSAL ANALYSIS AND DEFECT PREVENTIONCAUSAL ANALYSIS AND DEFECT PREVENTION


98/105


CAUSAL ANALYSIS AND DEFECT PREVENTIONCAUSAL ANALYSIS AND DEFECT PREVENTION

Let us recollect

Defect : A defect is what was wrong or incorrect in the workproduct.

Symptom : A Symptom is an outwardly manifestation of a defect inthe work product.

Defect Type : A type is category of defect which is indicative of the

nature of the defect.

Cause :A cause is some preceding incident that brought about thedefect.

Software TestingSoftware TestingDefect TypesDefect Types


99/105


Defect TypesDefect Types Syntax: General Syntax Problems, Spelling, punctuation, General format

problem, Did not properly delimit the operation

Assignment: Value(s) assigned incorrectly or not assigned at all. Interface: Communication problems between modules, components, device

drivers, objects, functions via macros, call statements, control blocks, parameterlists.

Checking: Errors caused by missing or incorrect validation of parameters ordata in conditional statements.

Data: Errors cause in data definitions and handling. Includes errors related toStructure, content

Function: General logic, Pointers, strings, Off-by-one, incrementing,recursion, Computation, algorithmic

System:Necessary serialization of shared resource was missing, the wrong

resource was serialized, or the wrong serialization technique was employed. Documentation: Errors associated with Non-conformance to standards,

Comments, Messages, Manual etc.

o tware est ngo ware es ngCause CategoriesCause Categories


100/105


Cause CategoriesCause Categories

COMM: Communications Failure. e.g., Incorrect information, lost information,

failure to communicate a change in information.

OVER: Oversight. For example, something is overlooked or forgotten, all cases

and conditions are not considered

EDUC: Education. i.e., lack of knowledge or understanding about something.

For example, not understanding a new functionality, not understanding some

aspect of the existing system, inadequate knowledge of programming language,

programming standards, compiler, tool, etc.

TRAN: Transcription Error. Transcribed or copied something incorrectly.

Knew what to do but just made a mistake

PROC: Inadequacy of the Process. The process definition did not cover

something and that led to a defect in the work product

Software TestingSoftware TestingCausal AnalysisCausal Analysis


101/105


Causal AnalysisCausal Analysis

Assigning Defect Type to defects

Assigning Cause Categories to defects

Causal Analysis

Causal Analysis Meetings

Preventive Actions Evaluation of the effectiveness of defect preventive actions in

projects

Software TestingSoftware TestingTest ToolsTest Tools


102/105


Test ToolsTest Tools Why Test Tools ?

To do tedius work

Free tester for creative work Do things Manually impossible

Eliminate errors in manual testing

Ability to test more

Specially useful for regression testing

But Tools can not repair effects of poor management

Typical Toolkit - Broad classification defect management

test coverage

test execution & regression testing test planning

test data generation



103/105


Test Toolses oo s File Comparators

Compares contents of two files and highlight differences

Used to compare out files of test runs

before,after changes

expected / actual results

Examples

IMSCOMPR (IMS Files) from IBM CA-Verify (IBM) from Computer Associates

Comparex from sterling Software

EXPEDITOR (IBM) from Compuware

BTS (IBM)

IDT (IBM)

Tools for keystroke capture in test scripts and script playback



104/105


Test Tools Defect Management Tools

Compares contents of two files and highlight differences

Used to compare out files of test runs

before,after changes

expected / actual results

Examples

IMSCOMPR (IMS Files) from IBM CA-Verify (IBM) from Computer Associates

Comparex from sterling Software



105/105

Any Questions / clarifications ?

Thank You

software testing (satyam) by jai raj yadav.ppt

Documents