software testing - anuradha bhatia testing is as old as the hills in the history of digital...

Software Engineering 2. Project Scheduling

Software

Testing

THIRD YEAR – SUBJECT CODE – 17624

COMPUTER TECHNOLOGY

COMPUTER ENGINEERING

INFORMATION TECHNOLOGY

2015-2016

MRS.ANURADHA BHATIA

G-SCHEME

M.E. Computer Engineering

MSBTE

Software Testing

Anuradha Bhatia 0

Table of Contents

1. Basics of Software Testing ............................................................................................................... 2

2. Types of Testing .............................................................................................................................. 21

3. Levels of Testing and Special Tests.............................................................................................. 41

4. Test Management ............................................................................................................................ 60

5. Defect Management Objectives ....................................................... Error! Bookmark not defined.

6. Testing Tools and Measurement .................................................................................................. 87

Software Testing

Anuradha Bhatia 1

Disclaimer

The content of the book is the copyright property of the author, to be used by the students for

the reference for the subject “Software Testing”, Sixth Semester, for the Final Year Computer

Technology, Computer Engineering and Information Technology department.

The complete set of the e-book is available on the author’s website www.anuradhabhatia.com,

and students are allowed to download it free of charge from the same.

The author does not gain any monetary benefit from the same, it is only developed and designed

to help the teaching and the student feternity to enhance their knowledge with respect to the

curriculum prescribed by MSBTE.

http://www.anuradhabhatia.com/

Software Testing

Anuradha Bhatia 2

1. Basics of

Software Testing CONTENTS

1.1 Software Quality, Definition of testing, Role of testing.

1.2 Failure, Error, fault, Defect, Bug Terminology

1.3 Objectives of Testing

1.4 When to start and Stop Testing of software.

1.5 Skills for Software Tester.

1.6 Quality Assurance, Quality Control, Verification and Validation, V

Model

Software Testing 1. Basics of Software Testing

Anuradha Bhatia 3

1.1 Software Quality, Definition of testing, Role of testing

1. Software Quality

Software quality is the degree of conformance to explicit or implicit requirements

and expectations.

i. Explicit: clearly defined and documented

ii. Implicit: not clearly defined and documented but indirectly suggested

iii. Requirements: business/product/software requirements

iv. Expectations: mainly end-user expectations

2. Definition of Testing

(Question: Explain testing with software quality- 2 Marks each, 4 Marks)

Testing can be defined in simple words as “Performing Verification and Validation of

the Software Product” for its correctness and accuracy of working.

i. Software testing is as old as the hills in the history of digital computers.

ii. The testing of software is an important means of assessing the software to

determine its quality.

iii. Since testing typically consumes 40% to50% of development efforts, and

consumes more effort for systems that require higher levels of reliability, it is

a significant part of the software engineering.

iv. With the development of fourth generation languages (4GL), which speeds up

the implementation process, the proportion of time devoted to testing

increased.

v. As the amount of maintenance and upgrade of existing systems grow,

significant amount of testing will also be needed to verify systems after

changes are made.


Anuradha Bhatia 4

3. Role of Testing

(Question: Explain the role of Testing – 6 Marks)

i. Software testing is to identify errors in order to reveal and spot it.

ii. The extent of software testing consists of implementation of that code in

different domain and also to look at the features of the code does the software

do what it is should be done and methods respect to the condition.

iii. It is proposed to begin testing from the first phase of the software

development.

iv. It saves time as well as cost. It is a continuous method, which is probably

nonstop but has to be stopped anywhere, for the need of time and resources.

v. The basic need of the testing is to provide best quality product without taking

so much time and money.

vi. The test engineer has to pursue some technical way by which he/she can

review that all the points of necessity for testing have been covered or not.

In SDLC stage there are some most importance things as described below:

i. Recognition of Error and Faults

Testing step is one step which resolves the errors and faults in the software

application.

These errors may be in unit level or in system level.

After going through so many testing the application will be free of errors that

may be disturbing the application.

ii. Statistics to Shareholders and Status of Organization

Testing stage helps to know the condition of product and work standards.

The stakeholders get better data through testing stage about utility value too.

iii. Enhancement in Product Standards


Anuradha Bhatia 5

Testing can help to know the real result and the probable result.

It also helps to pick up the standards of the software.

With proper testing an application can come out of bugs and build up ideal

software for the end-users.

iv. Technical Significance

Testing segment is significant for technical characteristics of any SDLC, as the

software then completed with technically satisfied.

v. To Succeed of any Contentious Programmers

Ideal testing functions and tools aid to evolve up the product in business and

keep programmers away from the other contestant.

Going through all stages of testing, the software application will be more bugs

free, protected and technically sound.

vi. Free from any Risk

Whenever going to develop any software, testing is an essential part.

When develop software without any testing then it may cause lots of risks to

the end users.

To free everyone from any risk, it is essential that to go under all testing stages.

vii. Enhanced Standards

Appropriate tested application provides additional assurance of build up with

best software.

Moreover, it refines standards of application as incessant and all types of

testing stages have prepared a protected and harmless software application

that could be worn by the end users.

viii. Confirmation and Corroboration


Anuradha Bhatia 6

One of the major targets of testing stage in SDLC is for confirmation and

corroboration.

Testing is greatly used in confirmation and corroboration method.

Depending on the result we can compare among standards of several software

application.

ix. Credibility Evaluation

Testing stage also insist this important issue.

If the software application has gone through all the testing types (like unit

testing, regression testing etc.), the application will surely be a reliable one.

So, testing evaluate credibility of software application.

Testing provides the greatest analytical process to give equipped testing on

product ensuing in a credible product.

x. Demonstrate Accessibility and Feasibility

One of the most significant targets of testing is to demonstrate the product is

both accessible and functional.

Accessibility testing is where the application is delivering to a select assembly

of users and their functioning with the application is noticed.

All type of a user's communication with the application, like easiness of applies

and whenever users are getting troubles, are preserved and examined.

xi. Avoid Fault Immigration

In the first stage of SDLC, most of the faults have been found. If the faults can

be noticed earlier, then these may be prohibited from immigrating to the

following progress stage.

If the errors could be discover previously then the saving of software

development cost will be vast.


Anuradha Bhatia 7

xii. Commercial Significance

A full tested software application will have excellent business aspects.

As all are like to work with reliable and trusted application in commercial.

1.2 Failure, Error, fault, Defect, Bug Terminology

(Question: Define Bug and list the terms related to software failure: – definition -

1 mark, list – 1 mark, explanation – 2 marks = 4 marks)

Definition: A bug can be defined in simple term as any error or mistake that leads to the

failure of the product or software either due to the specification problem or due to

communication problem, regarding what is developed and what had to be developed.

1. Bug Terminology: Failure, Error, Fault

The various terms related to software failure with respect to the area of

application are listed as Defect, Variance, Fault, Failure, Problem, Inconsistency,

Error, Feature, Incident, Bug, and Anomaly.

i. Problem, error, and bug are probably the most generic terms used.

ii. Anomaly, incident, and variance don’t sound quite so negative and infer

more unintended operation than an all-out failure.

iii. Fault, failure, and defect tend to imply a condition that’s really severe,

maybe even dangerous. It doesn’t sound right to call an incorrectly colored

icon a fault. These words also tend to imply blame: “It’s his fault that the

software failed.

iv. As all the words sound the same they are distinguished based on the

severity and the area in which the software failure has occurred.

v. When we run a program the error that we get during execution is termed

on the basis of runtime error, compile time error, computational error, and

assignment error.


Anuradha Bhatia 8

vi. The error can be removed by debugging, if not resolved leads to a problem

and if the problem becomes large leads to software failure.

vii. A bug can be defined as the initiation of error or a problem due to which

fault, failure, incident or an anomaly occurs.

viii. The program to find the factorial of a number given below lists few errors

problem and failure in a program.

For example

1. #include<stdio.h>

2. void main()

3. {

4. int i , fact, n;

5. printf(“enter the number “);

6. scanf(“%d”,&n);

7. for(i =1 ;i <=n;i++)

8. fact = fact * i;

9. printf (“the factorial of a number is ”%d”, fact);

10. }

As in line number 4 the fact is not initialized to 1, so it takes garbage value

and gives a wrong output, this is an example of a bug.

If fact is initialized to zero (fact = 0) than the output will be zero as anything

multiplied by zero will give the output as zero. This is a bug which can be

removed by initializing fact = 1 during initializing.

As the fact is declared as integer, for the number till 7! will work perfectly.

When the number entered is 8, the output is garbage value as the integer

limit is from – 32767 to +32768, so in declaration change the initialization

to long int fact;


Anuradha Bhatia 9

1.3 Objective of Testing

(Question: Explain the objective of Testing- 6 Marks)

1. Finding defects which may get created by the programmer while developing the

software.

2. Gaining confidence in and providing information about the level of quality.

3. To prevent defects.

4. To make sure that the end result meets the business and user requirements.

5. To ensure that it satisfies the BRS that is Business Requirement Specification and

SRS that is System Requirement Specifications.

6. To gain the confidence of the customers by providing them a quality product.

1.4 When to start and stop testing of software

1. When to Start Testing

(Question: Give the steps when should software testing start in project

development- 6 Marks)

i. An early start to testing reduces the cost, time to rework and error free

software that is delivered to the client.

ii. Software Development Life Cycle (SDLC) testing can be started from the

Requirements Gathering phase and lasts till the deployment of the

software.

iii. It also depends on the development model that is being used. For example

in Water fall model formal testing is conducted in the Testing phase, but

in incremental model, testing is performed at the end of every

increment/iteration and at the end the whole application is tested.

iv. Testing is done in different forms at every phase of SDLC like during

Requirement gathering phase, the analysis and verification of

requirements are also considered testing.

http://istqbexamcertification.com/what-is-defect-or-bugs-or-faults-in-software-testing/


Anuradha Bhatia 10

v. Reviewing the design in the design phase with intent to improve the design

is also considered as testing.

vi. Testing performed by a developer on completion of the code is also

categorized as Unit type of testing.

2. When to Stop Testing

(Question: Give the steps when should software testing stop in project

development- 6 Marks)

i. Unlike when to start testing it is difficult to determine when to stop testing,

as testing is a never ending process and no one can say that any software

is 100% tested.

ii. Following are the aspects which should be considered to stop the testing:

Testing Deadlines.

Completion of test case execution.

Completion of Functional and code coverage to a certain point.

Bug rate falls below a certain level and no high priority bugs are

identified.

Management decision.

iii. Testing should be stopped when it meets the completion criteria.

iv. Completion criteria should be based on Risks.

v. Testing should be stopped when :

Test cases completed with certain percentage passed and test

coverage is achieved.

There are no known critical bugs

Coverage of code, functionality, or requirements reaches a specified

point;


Anuradha Bhatia 11

Bug rate falls below a certain level, now testers are not getting any

priority 1, 2, or 3 bugs.

vi. The risk can be measured by Risk analysis but for small duration / low

budget / low resources project, risk can be deduced by simply: -

Measuring Test Coverage.

Number of test cycles.

Number of high priority bugs.

1.5 Skills of a software tester

(Question: List and explain skills of software tester- 8 Marks)

1. Analytical and logical thinking

i. The major objective of testing is to identify the hidden errors, not simply

prove that the software works.

ii. For a tester to be effective in his role, he must be able to analyze the given

business situation and judge all the possible scenarios.

iii. He should have the capacity to identify and tackle unfamiliar problems and

should develop a strategy to validate it.

iv. Creating situations and validating the application under test, before

presenting it to customers, can be done effectively only by a person who

has strong analytical skills.

ii. A tester should be able to separate the whole into logical parts -- to

examine a complex problem, its elements and their relationships.

iii. He should be able to develop a logical argument based on relationships

between elements and propositions, as well as identify implications,

relationships, redundancies and contradictions without leaving any room

for inconsistency and ambiguity.


Anuradha Bhatia 12

iv. He should be consistent in analyzing and solving complex, multi-step

problems.

2. The ability to envision business situations

i. A tester should be able to envisage real-time business situations through

mental mapping, abstracting the idea inferred from the specifications.

ii. The real-time business scenarios should crystallize in a tester's mind, and

he should think about what the case is rather than what ought to be the

case or what he believes the case is.

iii. A tester should be able to anticipate complex problems, in addition to

visualizing and articulating them.

iv. He should be able to do a complete system simulation rapidly and

accurately.

v. In the present software development environments, it is hard to believe

that teams/individuals will get enough time to do a series of conventional

brainstorming sessions to finalize the concept mapping.

vi. Therefore, it is vital that a tester develop his conceptualization skills

through mental mapping.

3. A sense of intellectual curiosity and creativity

i. A tester should understand that being an intellectual and being

intellectually curious are not the same.

ii. A tester should arguably be the latter one -- intellectually curious -- which

is all about asking questions and not about having answers.

iii. He should believe in the pursuit of knowledge as a value in and of itself.

iv. He should love asking questions and should not consider it a blow to his

ego if he is wrong about something.

v. It is intellectual curiosity that motivates and prompts a tester to identify

interesting questions about the software being tested.

http://searchsoftwarequality.techtarget.com/feature/Software-testing-trends-2012-Business-alignment-not-bug-fixes


Anuradha Bhatia 13

vi. Thus, a tester should develop the skill to see what everyone else hasn't

seen, to think what no one else has thought of and to do what no one else

has dared.

4. A "glocal" approach

i. Software systems have become extremely complex.

ii. Most of the time, the system designed involves multiple stakeholders, and

dealing with such systems is not always easy.

iii. A tester should be able to deal effectively with business situations marked

by complexity and the number of interactions with third-party systems.

iv. He should be able to identify how the system under test interacts with

other constituents of the system.

v. He should also be able to isolate the minutest units of the application

under test and do the validation, keeping in mind the behavior of the

system as a whole.

vi. A tester will stand out among his peers if he is able to detach himself from

the system, look at it as an outsider, and present his findings on the

relationship between the individual components convincingly in a logical

fashion.

vii. At the same time, he should keep his eyes and ears open so as not to ignore

issues that may be inherent in individual components.

viii. Thus, a "glocal" (global + local) approach is essential.

5. Critical thought and rational enquiry

i. The quality of life of an individual and the quality of what he

produces/delivers depends largely on the quality of his thought process.

ii. The thought process of a tester should be undistorted, impartial and

without any prejudices.

iii. A tester should be able to take charge of the inherent structures and

impose intellectual standards upon the software under test.


Anuradha Bhatia 14

iv. He should be able to raise vital questions precisely and clearly, gather and

assess relevant information, interpret it effectively in order to come to

well-reasoned conclusions and solutions, and test those conclusions

against the given criteria and standards.

v. A tester should be open minded, suspend any judgment in the absence of

sufficient evidence to support a decision, and always abide by logical

reasoning.

6. The ability to apply basic and fundamental knowledge

i. Knowledge in the context of testing can be attributed as the fluid mix of

experience, values, contextual information and expert insight.

ii. Those things provide a framework for evaluating the system under test.

One can attain knowledge by so many means, but that knowledge is

worthwhile only when it adds value to situations encountered.

iii. A smart tester should be able to apply the knowledge attained over years

of experience with the domain, process, product, customers, mistakes and

successes in his testing.

iv. He should be able to make use of fundamental communication,

mathematical and software application skills.

v. He should also be able to effectively apply the skills he has attained to

practical situations.

7. Continue to learn

i. Organizations and business environments change rapidly, which means

the approaches and processes that work well today will be outdated

tomorrow.

ii. Therefore, it is imperative that a tester place priority on noticing, adapting

and learning from change that is happening around him.


Anuradha Bhatia 15

iii. That doesn't mean a tester should continually undergo training or

certification, rather he should be open to learning from everything in life

that comes he across.

iv. If he has gained basic and fundamental knowledge, then the rest can be

achieved through self-directed learning.

v. Learning should be a lifelong habit.

8. Respect for truth and intellectual integrity

i. A tester should be able to examine the piece of software under test and

the resulting processes, with focus on the given specification, and

understand the behavior of the software.

ii. Being human, a tester may have severe biases, prejudices and intolerances

that prevent him from performing well.

iii. He should possess the intellectual integrity to correct those barriers in

order to efficiently understand the nature of the software under test.

iv. He should also be willing to shrug off the set of practices and character

traits that undermine his intellectual integrity.

v. And he should be able to exemplify intellectual virtues such as honesty,

impartiality and openness to the views of others.

9. Planning, time management skills

i. Planning is nothing but writing the story of the future.

ii. A tester needs to have a thorough plan and must develop a well-thought

test strategy/approach.

iii. And that plan must be in place before work begins on any software testing

assignment.

iv. It should describe the items and features to be tested, the test strategy

and levels of testing, pass/fail criteria, suspension/resumption criteria,

schedule, etc.

http://searchsoftwarequality.techtarget.com/answer/Creating-clear-software-requirements-specifications


Anuradha Bhatia 16

v. The plan developed should be monitored continually, and validations

should be done through organized system feedback.

vi. Sticking to the plan and monitoring the progress in order to ensure timely

delivery is key to any software testing assignment's success.

10. Effective communication skills

i. A tester must be able to communicate his thoughts and ideas effectively,

using a variety of tools and media.

ii. He needs to develop and use this skill throughout his career and should

learn to communicate effectively to the stakeholders so as to avoid

ambiguities and inconsistencies.

iii. For example, printed presentations should be concise and to the point and

should follow logically.

iv. The language should be pragmatic rather than philosophical, and

arguments should be supported by facts.

v. In the case of oral presentations, the voice, body language and appearance

of the presenter are as important as the content and visual aids.

vi. A tester should develop his skills to overcome shyness and any fear of

speaking. He should also have good listening skills.

1.6 Quality Assurance, Quality Control, Verification and Validation,

V Model

1. Quality Assurance

(Question: Explain the concept of Quality Assurance and Quality control in software

testing. - 2 marks + 2 marks =4 marks)


Anuradha Bhatia 17

i. Quality Assurance: A part of quality management focused on providing

confidence that quality requirements will be fulfilled.

ii. All the planned and systematic activities implemented within the quality

system that can be demonstrated to provide confidence that a product or

service will fulfill requirements for quality

iii. Quality Assurance is fundamentally focused on planning and documenting

those processes to assure quality including things such as quality plans and

inspection and test plans.

iv. Quality Assurance is a system for evaluating performance, service, of the

quality of a product against a system, standard or specified requirement

for customers.

v. Quality Assurance is a complete system to assure the quality of products

or services. It is not only a process, but a complete system including also

control. It is a way of management.

2. Quality Control

i. A part of quality management focused on fulfilling quality requirements.

ii. The operational techniques and activities used to fulfill requirements for

quality.

iii. Quality Control on the other hand is the physical verification that the product

conforms to these planned arrangements by inspection, measurement etc.

iv. Quality Control is the process involved within the system to ensure job

management, competence and performance during the manufacturing of the

product or service to ensure it meets the quality plan as designed.

v. Quality Control just measures and determines the quality level of products or

services.

3. Precision and Accuracy


Anuradha Bhatia 18

(Question: Explain with diagram the difference between precision and accuracy. :-

diagram – 1 mark, explanation – 3 marks = 4 marks)

i. The Accuracy of a measurement system is the degree of closeness of

measurements of a quantity to that quantity's actual (true) value.

ii. The Precision of a measurement system, also called reproducibility or

repeatability, is the degree to which repeated measurements under

unchanged conditions show the same results.

iii. Although the two words precision and accuracy can be synonymous in

colloquial use, they are deliberately contrasted in the context of the scientific

method.

Diagram for Accuracy and Precision

Figure 1.1: Accuracy and Precision

For example –

1. A measurement system can be accurate but not precise, precise but not accurate,

neither, or both.

2. If an experiment contains a systematic error, then increasing the sample inputs

generally increases precision but does not improve accuracy.

3. The result would be a consistent yet inaccurate string of results from the flawed

software.

4. Eliminating the systematic error improves accuracy but does not change precision.

http://en.wikipedia.org/wiki/Measurement

http://en.wikipedia.org/wiki/Quantity

http://en.wikipedia.org/wiki/Value_(mathematics)

http://en.wikipedia.org/wiki/Reproducibility

http://en.wikipedia.org/wiki/Repeatability

http://en.wikipedia.org/wiki/Result

http://en.wikipedia.org/wiki/Synonymous

http://en.wikipedia.org/wiki/Colloquial

http://en.wikipedia.org/wiki/Scientific_method

http://en.wikipedia.org/wiki/Scientific_method

http://en.wikipedia.org/wiki/Experiment

http://en.wikipedia.org/wiki/Systematic_error

http://en.wikipedia.org/wiki/Sample_size


Anuradha Bhatia 19

4. Verification and Validation

(Question: Explain the concept of verification and validation in software

testing. - 2 marks + 2 marks = 4 marks)

Verification

i. It makes sure that the product is designed to deliver all functionality to the

customer.

ii. Verification is done at the starting of the development process. It includes

reviews and meetings, walkthroughs, inspection, etc. to evaluate documents,

plans, code, requirements and specifications.

iii. It answers the questions like: Am I building the product right?

iv. Am I accessing the data right (in the right place; in the right way).

v. It is a Low level activity

vi. Performed during development on key art facts, like walkthroughs, reviews

and inspections, mentor feedback, training, checklists and standards.

vii. Demonstration of consistency, completeness, and correctness of the software

at each stage and between each stage of the development life cycle.


Anuradha Bhatia 20

Validation

1. Determining if the system complies with the requirements and performs

functions for which it is intended and meets the organization’s goals and user

needs.

2. Validation is done at the end of the development process and takes place after

verifications are completed.

3. It answers the question like: Am I building the right product?

4. Am I accessing the right data (in terms of the data required to satisfy the

requirement).

5. It is a High level activity.

6. Performed after a work product is produced against established criteria

ensuring that the product integrates correctly into the environment.

7. Determination of correctness of the final software product by a development

project with respect to the user needs and requirements.

5. V Model

(Question: Draw the diagram of V Model – 4 Marks)

Figure 1.2: The V Model

Software Testing 2. Types of Testing

Anuradha Bhatia 21

2. Types of Testing CONTENTS

I. White Box Testing: Classification of White Box.

1. Static Testing- Inspections, Structured Walkthroughs, Technical

Review.

2. Structural Testing-Code Functional Testing, Code Coverage Testing,

Code Complexity Testing.

II. Black Box Testing: Techniques for Black Box Testing

1. Requirement Based Testing.

2. Positive and Negative Testing.

3. Boundary Value Analysis.

4. Decision Tables.

5. Equivalence Partitioning.

6. User Documentation Testing.

7. Graph Based Testing.

8. Sample Examples on White and Black Box Testing.


Anuradha Bhatia 22

I. White Box Testing: Classification of White Box.

(Question: Explain white box testing. – 4 marks)

1. This is also known as glass box, clear box, and open box testing.

2. In white box testing, test cases are created by looking at the code to detect any

potential failure scenarios.

3. The suitable input data for testing various APIs and the special code paths that

need to be tested by analysing the source code for the application block.

4. Therefore, the test plans need to be updated before starting white box testing and

only after a stable build of the code is available.

5. White box testing assumes that the tester can take a look at the code for the

application block and create test cases that look for any potential failure scenarios.

6. During white box testing, analyze the code of the application block and prepare

test cases for testing the functionality to ensure that the class is behaving in

accordance with the specifications and testing for robustness.

7. A failure of a white box test may result in a change that requires all black box

testing to be repeated and white box testing paths to be reviewed and possibly

changed.


Anuradha Bhatia 23

Figure 2 .1 : White Box Test ing

i . Static Testing- Inspections, Structured Walkthroughs,

Technical Review.

1. Inspection

(Question: Explain the concept of inspection = 4 marks)

i. Inspections are the most formal type of reviews.

ii. They are highly structured and require training for each participant.

iii. Inspections are different from peer reviews and walkthroughs in that the

person who presents the code, the presenter or reader, isn’t the original

programmer.

iv. These forces someone else to learn and understand the material being

presented, potentially giving a different slant and interpretation at the

inspection meeting.

v. The other participants are called inspectors.

vi. Each is tasked with reviewing the code from a different perspective, such as a

user, a tester, or a product support person.


Anuradha Bhatia 24

vii. This helps bring different views of the product under review and very often

identifies different bugs.

viii. One inspector is even tasked with reviewing the code backward—that is, from

the end to the beginning—to make sure that the material is covered evenly

and completely.

2. Walkthrough

(Question: Explain the concept of Walkthrough = 4 marks)

i. Walkthroughs are the next step up in formality from peer reviews.

ii. In a walkthrough, the programmer who wrote the code formally presents

(walks through) it to a small group of five or so other programmers and testers.

iii. The reviewers should receive copies of the software in advance of the review

so they can examine it and write comments and questions that they want to

ask at the review.

iv. Having at least one senior programmer as a reviewer is very important.

3. Technical Review

(Question: Explain Technical review - 6 Marks)

i . Formal Review

A formal review is the process under which static white-box testing is

performed.

A formal review can range from a simple meeting between two programmers

to a detailed, rigorous inspection of the code.

There are four essential elements to a formal review

1. Identify Problems: - .The goal of the review is to find problems with the

software—not just items that are wrong, but missing items as well. All

criticism should be directed at the code, not the person who created it.


Anuradha Bhatia 25

Participants shouldn’t take any criticism personally. Leave your egos,

emotions, and sensitive feelings at the door.

2. Follow Rules: - A fixed set of rules should be followed. They may set the

amount of code to be reviewed (usually a couple hundred lines), how much

time will be spent (a couple hours), what can be commented on, and so

on. This is important so that the participants know what their roles are and

what they should expect. It helps the review run more smoothly.

3. Prepare: - Each participant is expected to prepare for and contribute to

the review. Depending on the type of review, participants may have

different roles. They need to know what their duties and responsibilities

are and be ready to actively fulfill them at the review. Most of the

problems found through the review process are found during preparation,

not at the actual review.

4. Write a Report: - The review group must produce a written report

summarizing the results of the review and make that report available to

the rest of the product development team. It’s imperative that others are

told the results of the meeting—how many problems were found, where

they were found, and so on.

ii. Peer Reviews

(Question: Explain the concept of peer review = 4 marks)

The easiest way to get team members together and doing their first formal

reviews of the software is through peer reviews, the least formal method.

Sometimes called buddy reviews, this method is really more of a

discussion.

Peer reviews are often held with just the programmer who wrote the code

and one or two other programmers or testers acting as reviewers.

Small group simply reviews the code together and looks for problems and

oversights.


Anuradha Bhatia 26

To assure that the review is highly effective all the participants need to

make sure that the four key elements of a formal review are in place: Look

for problems, follow rules, prepare for the review, and write a report.

As peer reviews are informal, these elements are often scaled back. Still,

just getting together to discuss the code can find bugs.

ii. Structural Testing-Code Functional Testing, Code

Coverage Testing, Code Complexity Testing.

1. Data Flow (Code Functional Testing)

i. Data flow coverage involves tracking a piece of data completely through the

software.

ii. At the unit test level this would just be through an individual module or

function.

iii. The same tracking could be done through several integrated modules or even

through the entire software product—although it would be more time-

consuming to do so.

iv. During data flow, the check is made for the proper declaration of variables

declared and the loops used are declared and used properly.

For example


2. void main()

3. {

4. int i , fact= 1, n;


6. scanf(“%d”,&n);

7. for(i =1 ;i <=n;i++)

8. fact = fact * i;


10. }


Anuradha Bhatia 27

2. Data Coverage (Code Coverage Testing)

(Question: Explain the concept of data coverage in software testing- 4 Marks)

i. The logical approach is to divide the code just as you did in black-box testing

into its data and its states (or program flow).

ii. By looking at the software from the same perspective, you can more easily

map the white-box information you gain to the black-box cases you’ve already

written.

iii. Consider the data first. Data includes all the variables, constants, arrays, data

structures, keyboard and mouse input, files and screen input and output, and

I/O to other devices such as modems, networks, and so on.

For example


12. void main()

13. {

14. int i , fact= 1, n;


16. scanf(“%d”,&n);

17. for(i =1 ;i <=n;i++)

18. fact = fact * i;


20. }

The declaration of data is complete with the assignment statement and the variable

declaration statements. All the variable declared are properly utilized.

3. Program Statements and Line Coverage (Code Complexity Testing)

(Question: Explain the concept of code complexity testing with example- 6 Marks)

i. The most straightforward form of code coverage is called statement coverage

or line coverage.


Anuradha Bhatia 28

ii. If you’re monitoring statement coverage while you test your software, your

goal is to make sure that you execute every statement in the program at least

once.

iii. With line coverage the tester tests the code line by line giving the relevant

output.

For example


2. void main()

3. {

4. int i , fact= 1, n;


6. scanf(“%d”, &n);

7. for(i =1 ;i <=n; i++)

8. fact = fact * i;


10. }

4. Branch Coverage (Code Complexity Testing)

i. Attempting to cover all the paths in the software is called path testing.

ii. The simplest form of path testing is called branch coverage testing.

iii. To check all the possibilities of the boundary and the sub boundary conditions

and it’s branching on those values.

iv. Test coverage criteria requires enough test cases such that each condition in a

decision takes on all possible outcomes at least once, and each point of entry

to a program or subroutine is invoked at least once.

v. Every branch (decision) taken each way, true and false.

vi. It helps in validating all the branches in the code making sure that no branch

leads to abnormal behavior of the application.


Anuradha Bhatia 29

5. Condition Coverage (Code Complexity Testing)

i. Just when you thought you had it all figured out, there’s yet another

complication to path testing. Condition coverage testing takes the extra

conditions on the branch statements into account.

II. Black Box Testing: Techniques for Black Box Testing (Question: Explain the black box testing technique – 4 marks)

Figure 2 .2 . Black Box Tes ting

1. This approach tests all possible combinations of end -user actions.

2. Black box testing assumes no knowledge of code and is intended to

simulate the end-user experience.

3. The tester can use sample applications to integrate and test the

application block for black box testing.

4. Planning for black box testing immediately after the requirements

and the functional specifications are available.

5. Black box testing should be conducted in a test environment close to

the target environment.


Anuradha Bhatia 30

1. Requirement Based Testing.

(Question: Explain requirement based testing stages - 4 marks)

Requirements-based testing is a testing approach in which test cases, conditions

and data are derived from requirements. It includes functional tests and also non-

functional attributes such as performance, reliability or usability.

i. Stages in Requirements based Testing:

Defining Test Completion Criteria - Testing is completed only when all the

functional and non-functional testing is complete.

Design Test Cases - A Test case has five parameters namely the initial state

or precondition, data setup, the inputs, expected outcomes and actual

outcomes.

Execute Tests - Execute the test cases against the system under test and

document the results.

Verify Test Results - Verify if the expected and actual results match each

other.

Verify Test Coverage - Verify if the tests cover both functional and non-

functional aspects of the requirement.

Track and Manage Defects - Any defects detected during the testing

process goes through the defect life cycle and are tracked to resolution.

Defect Statistics are maintained which will give us the overall status of the

project.

ii. Requirements testing process:

Testing must be carried out in a timely manner.

Testing process should add value to the software life cycle, hence it needs

to be effective.

Testing the system exhaustively is impossible hence the testing process

needs to be efficient as well.


Anuradha Bhatia 31

Testing must provide the overall status of the project, hence it should be

manageable.

2. Positive and Negative Testing.

i. Positive Testing:

Positive Testing is testing process where the system validated against the

valid input data.

In this testing tester always check for only valid set of values and check if

an application behaves as expected with its expected inputs.

The main intention of this testing is to check whether software application

not showing error when not supposed to & showing error when supposed

to.

Such testing is to be carried out keeping positive point of view & only

execute the positive scenario.

Positive Testing always tries to prove that a given product and project

always meets the requirements and specifications.

Under Positive testing is test the normal day to day life scenarios and check

the expected behaviour of application.

Example of Positive testing:

To test the text box for integer numbers only. As we are testing for positive

testing the input range is valid numbers only.

Figure 2.3: Positive Testing

http://www.softwaretestingclass.com/wp-content/uploads/2013/10/postitive-testing.jpg


Anuradha Bhatia 32

ii. Negative Testing:

Negative Testing is testing process where the system validated against the

invalid input data.

A negative test checks if an application behaves as expected with its

negative inputs.

The main intention of this testing is to check whether software application

not showing error when supposed to & showing error when not supposed

to. Such testing is to be carried out keeping negative point of view & only

execute the test cases for only invalid set of input data.

Negative testing is a testing process to identify the inputs where system is

not designed or un-handled inputs by providing different invalid. The main

reason behind Negative testing is to check the stability of the software

application against the influences of different variety of incorrect

validation data set.

3. Boundary Value Analysis.

(Question: Explain the concept of boundary conditions and sub boundary

condition s–2 marks each - 4 marks)

i. Boundary conditions are special because programming, by its nature, is

susceptible to problems at its edges.

ii. The boundary conditions are defined as the initial and the final data ranges of

the variables declared.

iii. If an operation is performed on a range of numbers, odds are the programmer

got it right for the vast majority of the numbers in the middle, but maybe made

a mistake at the edges.

iv. The edges are the minimum and the maximum values for that identifier.


2. void main()


Anuradha Bhatia 33

3. {

4. int i , fact=1, n;


6. scanf(“%d”,&n);

7. for(i =1 ;i <=n;i++)

8. fact = fact * i;


10. }

The boundary condition in the above example is for the integer variable.

Sub-Boundary Conditions

i. They’re the ones defined in the specification or evident when using the

software.

ii. Some boundaries, though, that are internal to the software aren’t necessarily

apparent to an end user but still need to be checked by the software tester.

iii. These are known as sub-boundary conditions or internal boundary conditions.

iv. In the given example the sub boundary condition is the value of factorial

For example


2. void main()

3. {

4. int i , fact=1, n;


6. scanf(“%d”,&n);

7. for(i =1 ;i <=n;i++)

8. fact = fact * i;


10. }


Anuradha Bhatia 34

4. Decision Tables.

i. Decision table testing is black box test design technique to determine the

test scenarios for complex business logic.

ii. Decision tables provide a systematic way of stating complex business rules,

which is useful for developers as well as for testers.

iii. Decision tables can be used in test design whether or not they are used in

specifications, as they help testers explore the effects of combinations of

different inputs and other software states that must correctly implement

business rules.

iv. It helps the developers to do a better job can also lead to better relationships

with them.

v. Testing combinations can be a challenge, as the number of combinations can

often be huge.

vi. Testing all combinations may be impractical if not impossible.

vii. We have to be satisfied with testing just a small subset of combinations but

making the choice of which combinations to test and which to leave out is also

important.

viii. If you do not have a systematic way of selecting combinations, an arbitrary

subset will be used and this may well result in an ineffective test effort.

5. Equivalence Partitioning.

i. Type of black box testing that divides that divides the input domain

of the program into classes of data from which the test cases can be

divided.

ii. This method is typically used to reduce the total number of test cases to a

finite set of testable test cases, still covering maximum requirements.

iii. It is the process of taking all possible test cases and placing them into classes.

One test value is picked from each class while testing.

For example:


Anuradha Bhatia 35

If you are testing for an input box accepting numbers from 1 to 1000 then there is

no use in writing thousand test cases for all 1000 valid input numbers plus other

test cases for invalid data.

i. Using equivalence partitioning method above test cases can be divided into

three sets of input data called as classes. Each test case is a representative of

respective class.

ii. For the above example we can divide our test cases into three equivalence

classes of some valid and invalid inputs.

Test cases for input box accepting numbers between 1 and 1000 using

Equivalence Partitioning:

One input data class with all valid inputs. Pick a single value from range 1 to 1000

as a valid test case. If you select other values between 1 and 1000 then result is

going to be same. So one test case for valid input data should be sufficient.

1. Input data class with all values below lower limit. i.e. any value below 1, as an

invalid input data test case.

2. Input data with any value greater than 1000 to represent third invalid input

class.

3. So using equivalence partitioning all possible test cases is partitioned into

three classes. Test cases with other values from any class should give you the

same result.

4. We have selected one representative from every input class to design our test

cases. Test case values are selected in such a way that largest number of

attributes of equivalence class can be exercised.

5. Equivalence partitioning uses fewest test cases to cover maximum

requirements.

6. User Documentation Testing.

i. Documentation testing is a non-functional type of software testing.

ii. It is a type of non-functional testing.

http://istqbexamcertification.com/what-is-a-software-testing/

http://istqbexamcertification.com/what-is-non-functional-testing-testing-of-software-product-characteristics/


Anuradha Bhatia 36

iii. Any written or pictorial information describing, defining, specifying, reporting,

or certifying activities, requirements, procedures, or results’. Documentation

is as important to a product’s success as the product itself.

iv. If the documentation is poor, non-existent, or wrong, it reflects on the quality

of the product and the vendor.

v. As per the IEEE Documentation describing plans for, or results of, the testing

of a system or component, Types include test case specification, test incident

report, test log, test plan, test procedure, test report. Hence the testing of all

the above mentioned documents is known as documentation testing.

vi. This is one of the most cost effective approaches to testing. If the

documentation is not right: there will be major and costly problems.

vii. The documentation can be tested in a number of different ways to many

different degrees of complexity.

viii. These range from running the documents through a spelling and grammar

checking device, to manually reviewing the documentation to remove any

ambiguity or inconsistency.

ix. Documentation testing can start at the very beginning of the software process

and hence save large amounts of money, since the earlier a defect is found the

less it will cost to be fixed.

7. Graph Based Testing.

i. Each and every application is buildup of some objects.

ii. All such objects are identified and graph is prepared.

iii. From this object graph each object relationship is identified and test cases

written accordingly to discover the errors.

8. State Testing

(Question: Explain the concept state testing in dynamic black box testing –4

marks)

http://istqbexamcertification.com/what-is-defect-or-bugs-or-faults-in-software-testing/


Anuradha Bhatia 37

i. The data gets tested on the numbers, words, inputs, and outputs of the

software.

ii. The product or the software should be tested for the program’s logic flow

through its various states.

iii. A software state is a condition or mode that the software is currently in.

iv. Consider Figures 4, which illustrates the software mode of the paint software

in airbrush mode.

Figure 2.4: Paint Mode

Testing Software’s Logic Flow

1. Testing the software’s states and logic flow has the same problems. It’s

usually possible to visit

2. Check for the all possible states for test to pass condition

3. The complexity of the software, especially due to the richness of today’s

user interfaces, provides so many choices and options that the number of

paths grows exponentially.

Creating a State Transition Map

(Question: Explain the concept of creating a state transition map. Diagram –

1 mark, Explanation 3 marks –4 marks)


Anuradha Bhatia 38

i. There are various techniques for state transition diagrams.

ii. Figure 5 shows two examples. One uses boxes and arrows and the other uses

circles (bubbles) and arrows.

iii. The sate transition diagram should be easily understandable and should be

able to explain clearly about the various stages the software passes through.

Figure 2.5: State Transition Map

A state transition map should show the following items:

Each unique state that the software can be in.

A good rule of thumb is that if the state is unsure whether something is a

separate state, it probably is.

Always collapse it into another state if you find out later that it isn’t.

The input or condition that takes it from one state to the next.

This might be a key press, a menu selection, a sensor input, a telephone

ring, and so on.

A state can’t be exited without some reason. The specific reason is what

you’re looking for here.

Set conditions and produced output when a state is entered or exited.

This would include a menu and buttons being displayed, a flag being set, a

printout occurring, a calculation being performed, and so on.

It’s anything and everything that happens on the transition from one state

to the next.


Anuradha Bhatia 39

9. Sample Examples on White and Black Box Testing.

A simple website application as an example to explain the white and the black box

testing. . The end user is simply access the website, Login & Logout, this is very

simple and day 2 days life example. As end users point of view user able to access

the website from GUI but inside there are lots of things going on to check the

internal things are going right or not, the white box testing method is used. To

explain this we have to divide this part in two steps. This is all is being done when

the tester is testing the application using White box testing techniques.

Step 1) UNDERSTAND OF THE SOURCE CODE

The first & very important steps is to understand the source code of the

application is being test.

As tester should know about the internal structure of the code which helps

to test the application.

Better knowledge of Source code will helps to identify & write the

important test case which causes the security issues & also helps to cover

the 100% test coverage.

Before doing this the tester should know the programming language what

is being used in the developing the application.

As security of application is primary objective of application so tester

should aware of the security concerns of the project which help in testing.

If the tester is aware of the security issues then he can prevents the

security issues like attacks from hackers & users who are trying to inject

the malicious things in the application.


Anuradha Bhatia 40

Step 2) CREATE TEST CASES AND EXECUTE

In the second step involves the actual writing of test cases based on

Statement/Decision/Condition/Branch coverage & actual execution of test

cases to cover the 100% testing coverage of the software application.

The Tester will write the test cases by diving the applications by

Statement/Decision/Condition/Branch wise.

Other than this tester can include the trial and error testing, manual testing

and using the Software testing tools as we covered in the previous article.

Software Testing

Anuradha Bhatia 41

3. Levels of Testing

and Special Tests CONTENTS

I. Unit Testing: Driver, Stub II. Integration Testing

i. Decomposition Testing: Top Down and Bottom up

Integration.

ii. Bi-Directional Integration.

iii . Incremental Integration.

iv. Non- Incremental Integration.

III. System testing i. Recovery Testing ii. Security Testing iii. Performance Testing. iv. Load Testing. v. Stress Testing. vi. Usability Testing. vii. Compatibility Testing.

IV. Acceptance Testing. i . Acceptance Criteria. ii . Alpha Testing. iii . Beta Testing.

V. Special Tests i. Smoke Testing and Sanity Testing, Regression Testing. ii . Usability Testing, GUI Testing. iii . Object Oriented Application Testing, Client Server Testing. iv. Web Based Testing.

Software Testing 3.Level of Testing and Special Tests

Anuradha Bhatia 42

I. Unit Testing: Driver, Stub (Question: Explain the concept of stubs and drivers in unit testing. - 6 Marks)

1. Unit is the smallest testable part of the software system.

2. Unit testing is done to verify that the lowest independent entities in any

software are working fine.

3. The smallest testable part is isolated from the remainder code and tested to

determine whether it works correctly.

4. When developer is coding the software it may happen that the dependent

modules are not completed for testing, in such cases developers use stubs and

drivers to simulate the called (stub) and caller (driver) units.

5. Unit testing requires stubs and drivers, stubs simulates the called unit and

driver simulates the calling unit.

STUBS: i. Assume you have 3 modules, Module A, Module B and module C.

ii. Module A is ready and we need to test it, but module A calls functions from

Module B and C which are not ready, so developer will write a dummy module

which simulates B and C and returns values to module A.

iii. This dummy module code is known as stub.

Figure 3.1: The Stub Flow Diagram

iv. The above diagrams clearly states that Modules 1, 2 and 3 are available for

integration, whereas, below modules are still under development that cannot

be integrated at this point of time.

v. Hence, Stubs are used to test the modules.


Anuradha Bhatia 43

DRIVERS: i. Now suppose you have modules B and C ready but module A which calls

functions from module B and C is not ready so developer will write a dummy

piece of code for module A which will return values to module B and C.

ii. This dummy piece of code is known as driver.

II. Integration Testing 1. Decomposition Testing: Top Down and Bottom up

Integration. (Question: explain top down and bottom up integration with

advantages and disadvantages. - 6 Marks)

i. Top down Testing: In this approach testing is conducted from main module to

sub module.

ii. If the sub module is not developed a temporary program called STUB is used

for simulate the sub module.

Advantages

Advantageous if major flaws occur toward the top of the program.

Once the I/O functions are added, representation of test cases is easier.

Early skeletal Program allows demonstrations and boosts morale.

Disadvantages:

Stub modules must be produced

Stub Modules are often more complicated than they first appear to be.

Before the I/O functions are added, representation of test cases in stubs

can be difficult.

Test conditions may be impossible, or very difficult, to create.

Observation of test output is more difficult.

Allows one to think that design and testing can be overlapped.

Induces one to defer completion of the testing of certain modules.

i. Bottom up testing: In this approach testing is conducted from sub module to main

module, if the main module is not developed a temporary program called DRIVERS

is used to simulate the main module.

Advantages:

Advantageous if major flaws occur toward the bottom of the program.

Test conditions are easier to create.

Observation of test results is easier.

Disadvantages:


Anuradha Bhatia 44

Driver Modules must be produced.

The program as an entity does not exist until the last module is added.

Figure 3.2: Integration Diagram

2. Bi-Directional Integration. (Question: Explain bi directional integration. - 4 Marks)

i. Bi-directional Integration, is a kind of integration testing process that combines top-down and bottom-up testing.

ii. With an experience in delivering Bi-directional testing projects custom software development services provide the best quality of the deliverables right from the development of software process.

iii. Bi-directional Integration testing is a vertical incremental testing strategy that tests the bottom layers and top layers and tests the integrated system in the computer software development process.

iv. Using stubs, it tests the user interface in isolation as well as tests the very lowest level functions using drivers.

v. Bi-directional Integration testing combines bottom-up and top-down testing. vi. Bottom-up testing is a process where lower level modules are integrated and

then tested.


Anuradha Bhatia 45

vii. This process is repeated until the component of the top of the hierarchy is analyzed. It helps custom software development services find bugs easily without any problems.

viii. Top down testing is a process where the top integrated modules are tested and the procedure is continued till the end of the related module.

ix. Top down testing helps developers find the missing branch link easily.

3. Incremental Integration.

(Question: Explain the features of incremental integration. - 4 Marks)

i. After unit testing is completed, developer performs integration testing. ii. It is the process of verifying the interfaces and interaction between

modules. iii. While integrating, there are lots of techniques used by developers and one

of them is the incremental approach. iv. In Incremental integration testing, the developers integrate the modules

one by one using stubs or drivers to uncover the defects. v. This approach is known as incremental integration testing.

vi. To the contrary, big bang is one other integration testing technique, where all the modules are integrated in one shot.

Features i. Each Module provides a definitive role to play in the project/product

structure ii. Each Module has clearly defined dependencies some of which can be

known only at the runtime. iii. The incremental integration testing's greater advantage is that the defects

are found early in a smaller assembly when it is relatively easy to detect the root cause of the same.

iv. A disadvantage is that it can be time-consuming since stubs and drivers have to be developed for performing these tests.

4. Non- Incremental Integration.

(Question: Explain the non -incremental integration technique

or the big band testing technique. – 4 Marks)

i. The non-incremental approach is also known as “Big-Bang” Testing. ii. Big Bang Integration Testing is an integration testing strategy wherein all units are linked

at once, resulting in a complete system. iii. When this type of testing strategy is adopted, it is difficult to isolate any errors found,

because attention is not paid to verifying the interfaces across individual units.


Anuradha Bhatia 46

Figure 3.3: Non-Incremental Integration

Disadvantages

i. Defects present at the interfaces of components are identified at very late stage as all components are integrated in one shot.

ii. It is very difficult to isolate the defects found. iii. There is high probability of missing some critical defects, which might pop up in the

production environment. iv. It is very difficult to cover all the cases for integration testing without missing even a single

scenario.

III. System Testing (Question: List the various system testing approaches and explain any two. – 8 Marks) 1. System Testing (ST) is a black box testing technique performed to evaluate the

complete system the system's compliance against specified requirements. 2. In System testing, the functionalities of the system are tested from an end-to-

end perspective. 3. System Testing is usually carried out by a team that is independent of the

development team in order to measure the quality of the system unbiased. 4. It includes both functional and Non-Functional testing.


Anuradha Bhatia 47

1. Recovery Testing

i. Recovery testing is a type of non-functional testing technique performed in order to determine how quickly the system can recover after it has gone through system crash or hardware failure.

ii. Recovery testing is the forced failure of the software to verify if the recovery is successful.

Steps:

Determining the feasibility of the recovery process. Verification of the backup facilities. Ensuring proper steps are documented to verify the compatibility of backup

facilities. Providing Training within the team. Demonstrating the ability of the organization to recover from all critical

failures. Maintaining and updating the recovery plan at regular intervals.

2. Security Testing

i. Security testing is a testing technique to determine if an information system protects data and maintains functionality as intended.

ii. It also aims at verifying 6 basic principles as listed below:

Confidentiality Integrity Authentication Authorization Availability Non-repudiation

Techniques:

Injection Broken Authentication and Session Management Cross-Site Scripting (XSS) Insecure Direct Object References Security Misconfiguration Sensitive Data Exposure Missing Function Level Access Control Cross-Site Request Forgery (CSRF) Using Components with Known Vulnerabilities


Anuradha Bhatia 48

Invalidated Redirects and Forwards

3. Performance Testing. i. Performance testing, a non-functional testing technique performed to

determine the system parameters in terms of responsiveness and stability under various workload.

ii. Performance testing measures the quality attributes of the system, such as scalability, reliability and resource usage.

Techniques:

Load testing - It is the simplest form of testing conducted to understand the behaviour of the system under a specific load. Load testing will result in measuring important business critical transactions and load on the database, application server, etc., are also monitored.

Stress testing - It is performed to find the upper limit capacity of the system and also to determine how the system performs if the current load goes well above the expected maximum.

Soak testing - Soak Testing also known as endurance testing, is performed to determine the system parameters under continuous expected load. During soak tests the parameters such as memory utilization is monitored to detect memory leaks or other performance issues. The main aim is to discover the system's performance under sustained use.

Spike testing - Spike testing is performed by increasing the number of users suddenly by a very large amount and measuring the performance of the system. The main aim is to determine whether the system will be able to sustain the workload.

Performance Testing Process:

Figure 3.4: Performance Testing Process


Anuradha Bhatia 49

Attributes of Performance Testing:

Speed Scalability Stability Reliability

4. Load Testing. i. Load testing is performance testing technique using which the response of

the system is measured under various load conditions. ii. The load testing is performed for normal and peak load conditions.

Load Testing Approach:

Evaluate performance acceptance criteria Identify critical scenarios Design workload Model Identify the target load levels Design the tests Execute Tests Analyse the Results

Objectives of Load Testing:

Response time Throughput Resource utilization Maximum user load Business-related metrics

5. Stress Testing.

i. Stress testing a Non-Functional testing technique that is performed as part of performance testing.

ii. During stress testing, the system is monitored after subjecting the system to overload to ensure that the system can sustain the stress.

iii. The recovery of the system from such phase (after stress) is very critical as it is highly likely to happen in production environment.

Reasons for conducting Stress Testing:

It allows the test team to monitor system performance during failures. To verify if the system has saved the data before crashing or NOT. To verify if the system prints meaning error messages while crashing or did

it print some random exceptions. To verify if unexpected failures do not cause security issues.


Anuradha Bhatia 50

Stress Testing - Scenarios:

Monitor the system behaviour when maximum number of users logged in at the same time.

All users performing the critical operations at the same time. All users accessing the same file at the same time. Hardware issues such as database server down or some of the servers in a

server park crashed.

6. Usability Testing.

i. Usability testing, a non-functional testing technique that is a measure of how easily the system can be used by end users.

ii. It is difficult to evaluate and measure but can be evaluated based on the below parameters: Level of Skill required to learn/use the software. It should maintain the

balance for both novice and expert user. Time required to get used to in using the software. The measure of increase in user productivity if any. Assessment of a user's attitude towards using the software.

Usability Testing Process:

Figure 3.5: Usability Testing Process


Anuradha Bhatia 51

7. Compatibility Testing.

i. Compatibility testing is a non-functional testing conducted on the application to evaluate the application's compatibility within different environments.

ii. It can be of two types - forward compatibility testing and backward compatibility testing. Operating system Compatibility Testing - Linux , Mac OS, Windows Database Compatibility Testing - Oracle SQL Server Browser Compatibility Testing - IE , Chrome, Firefox Other System Software - Web server, networking/ messaging tool, etc.

IV. Acceptance Testing. 1. Acceptance Criteria.

(Question: Explain the criteria for acceptance testing. - 4 Marks) i. Comparison testing comprises of comparing the contents of files, databases,

against actual results. ii. They are capable of highlighting the differences between expected and actual

results. iii. Comparison test tools often have functions that allow specified sections of the

files be ignored or masked out. iv. This enables the tester to mask out the date or time stamp on a screen or field

as it is always different from the expected ones when a comparison is performed.

2. Alpha Testing. (Question: Explain alpha testing. - 4 Marks)

Alpha testing takes place at the developer's site by the internal teams, before release to external customers. This testing is performed without the involvement of the development teams.

i. Alpha Testing - In SDLC

The following diagram explains the fitment of Alpha testing in the software development life cycle.


Anuradha Bhatia 52

Figure 3.6: Alpha Testing - SDLC

ii. How do we run it?

In the first phase of alpha testing, the software is tested by in-house developers during which the goal is to catch bugs quickly.

In the second phase of alpha testing, the software is given to the software QA team for additional testing.

Alpha testing is often performed for Commercial off-the-shelf software (COTS) as a form of internal acceptance testing, before the beta testing is performed.

3. Beta Testing.

(Question: Explain Beta testing- 4Marks)

i. Beta testing also known as user testing takes place at the end users site by the end users to validate the usability, functionality, compatibility, and reliability testing.

ii. Beta testing adds value to the software development life cycle as it allows the "real" customer an opportunity to provide inputs into the design, functionality, and usability of a product. These inputs are not only critical to the success of the product but also an investment into future products when the gathered data is managed effectively.


Anuradha Bhatia 53

Beta Testing - In SDLC

The following diagram explains the fitment of Beta testing in the software development life cycle:

Figure 3.7: Beta Testing SDLC

V. Special Tests 1. Smoke Testing and Sanity Testing

(Question: Differentiate smoke testing and sanity testing. – 6 Marks) i. Smoke Testing is a testing technique that is inspired from hardware testing,

which checks for the smoke from the hardware components once the hardware's power is switched on.

ii. In Software testing context, smoke testing refers to testing the basic functionality of the build.

iii. If the Test fails, build is declared as unstable and it is NOT tested anymore until the smoke test of the build passes.

Smoke Testing - Features:

i. Identifying the business critical functionalities that a product must satisfy. ii. Designing and executing the basic functionalities of the application.

iii. Ensuring that the smoke test passes each and every build in order to proceed with the testing.

iv. Smoke Tests enables uncovering obvious errors which saves time and effort of test team.

v. Smoke Tests can be manual or automated.


Anuradha Bhatia 54

i. Sanity testing, a software testing technique performed by the test team for some basic tests. The aim of basic test is to be conducted whenever a new build is received for testing. The terminologies such as Smoke Test or Build Verification Test or Basic Acceptance Test or Sanity Test are interchangeably used, however, each one of them is used under a slightly different scenario.

ii. Sanity test is usually unscripted, helps to identify the dependent missing functionalities. It is used to determine if the section of the application is still working after a minor change.

iii. Sanity testing can be narrow and deep. Sanity test is a narrow regression test that focuses on one or a few areas of functionality.

2. Regression Testing.

(Question: Explain Regression Testing. – 4 Marks)

i. Regression testing a black box testing technique that consists of re-executing those tests that are impacted by the code changes.

ii. These tests should be executed as often as possible throughout the software development life cycle.

Types of Regression Tests:

i. Final Regression Tests: - A "final regression testing" is performed to validate the build that hasn't changed for a period of time. This build is deployed or shipped to customers.

ii. Regression Tests: - A normal regression testing is performed to verify if the build has NOT broken any other parts of the application by the recent code changes for defect fixing or for enhancement.

3. Usability Testing. (Question: Explain Usability Testing. – 4 Marks)

i. Usability testing, a non-functional testing technique that is a measure of how easily the system can be used by end users.

ii. It is difficult to evaluate and measure but can be evaluated based on the below parameters:

iii. Level of Skill required to learn/use the software. It should maintain the balance for both novice and expert user.

iv. Time required to get used to in using the software. v. The measure of increase in user productivity if any.

vi. Assessment of a user's attitude towards using the software. vii. The usability process is shown in Figure


Anuradha Bhatia 55

Figure 3.8: Usability Testing

4. GUI Testing.

(Question: Explain GUI Testing. – 4 Marks) i. GUI testing is a testing technique in which the application's user interface is

tested whether the application performs as expected with respect to user interface behaviour.

ii. GUI Testing includes the application behaviour towards keyboard and mouse movements and how different GUI objects such as toolbars, buttons, menu bars, dialog boxes, edit fields, lists, behaviour to the user input.

GUI Testing Guidelines

i. Check Screen Validations ii. Verify All Navigations

iii. Check usability Conditions iv. Verify Data Integrity v. Verify the object states

vi. Verify the date Field and Numeric Field Formats

GUI Automation Tools

Following are some of the open source GUI automation tools in the market:


Anuradha Bhatia 56

Product Licensed Under URL

AutoHotkey GPL http://www.autohotkey.com/

Selenium Apache http://docs.seleniumhq.org/

Sikuli MIT http://sikuli.org

Robot Framework Apache www.robotframework.org

Water BSD http://www.watir.com/

Dojo Toolkit BSD http://dojotoolkit.org/

Table 3.1:GUI Automation

Following are some of the Commercial GUI automation tools in the market.

Product Vendor URL

AutoIT AutoIT http://www.autoitscript.com/site/autoit/

EggPlant TestPlant www.testplant.com

QTP Hp http://www8.hp.com/us/en/software-solutions/

Rational

Functional

Tester

IBM http://www-

03.ibm.com/software/products/us/en/functional

Infragistics Infragistics www.infragistics.com

iMacros iOpus http://www.iopus.com/iMacros/

CodedUI Microsoft http://www.microsoft.com/visualstudio/

http://sikuli.org/

http://www.testplant.com/

http://www.infragistics.com/


Anuradha Bhatia 57

Sikuli Micro Focus

International http://www.microfocus.com/

Table 3.2:Commercial GUI Automation

5. Object Oriented Application Testing. (Question: Explain object oriented application testing. - 6 Marks)

i . The Full-Lifecycle Object-Oriented Testing (FLOOT) methodology is a collection of testing techniques to verify and validate object-oriented software.

i i . The FLOOT lifecycle is depicted in Figure 9, indicating a wide variety of techniques (described in Table 9 are available to you throughout all aspects of software development.

i i i . The list of techniques is not meant to be complete: instead the goal is to make it explicit that you have a wide range of options available to you.

iv. It is important to understand that although the FLOOT method is presented as a collection of serial phases it does not need to be so: the techniques of FLOOT can be applied with evolutionary/agile processes as well.

v. The reason why I present the FLOOT in a "traditional" manner is to make it explicit that you can in fact test throughout all aspects of software development, not just during coding.

Figure 3.9: FLOOT Diagram


Anuradha Bhatia 58

6. Client Server Testing.

(Question: Explain Client Server Testing. – 4 Marks) i. This type of testing usually done for 2 tier applications (usually developed

for LAN) Here we will be having front-end and backend. ii. The application launched on front-end will be having forms and reports

which will be monitoring and manipulating data.E.g: applications developed in VB, VC++, Core Java, C, C++, D2K, PowerBuilder etc.,

iii. The backend for these applications would be MS Access, SQL Server,

Oracle, Sybase, Mysql, Quadbase. iv. The tests performed on these types of applications would be– User

interface testing Manual support testing– Functionality testing– Compatibility testing & configuration testing – Intersystem testing.

7. Web Based Testing. (Question: Explain web based testing. - 4 Marks)

i. Web application testing, a software testing technique exclusively adopted to test the applications that are hosted on web in which the application interfaces and other functionalities are tested.

Web Application Testing - Techniques: 1. Functionality Testing - The below are some of the checks that are performed

but not limited to the below list: Verify there is no dead page or invalid redirects. First check all the validations on each field. Wrong inputs to perform negative testing. Verify the workflow of the system. Verify the data integrity.

2. Usability testing - To verify how the application is easy to use with. Test the navigation and controls. Content checking. Check for user intuition.

3. Interface testing - Performed to verify the interface and the dataflow from one system to other.

4. Compatibility testing- Compatibility testing is performed based on the context of the application. Browser compatibility Operating system compatibility Compatible to various devices like notebook, mobile, etc.


Anuradha Bhatia 59

5. Performance testing - Performed to verify the server response time and throughput under various load conditions. Load testing - It is the simplest form of testing conducted to understand

the behaviour of the system under a specific load. Load testing will result in measuring important business critical transactions and load on the database, application server, etc. are also monitored.

Stress testing - It is performed to find the upper limit capacity of the system and also to determine how the system performs if the current load goes well above the expected maximum.

Soak testing - Soak Testing also known as endurance testing, is performed to determine the system parameters under continuous expected load. During soak tests the parameters such as memory utilization is monitored to detect memory leaks or other performance issues. The main aim is to discover the system's performance under sustained use.

Spike testing - Spike testing is performed by increasing the number of users suddenly by a very large amount and measuring the performance of the system. The main aim is to determine whether the system will be able to sustain the work load.

6. Security testing - Performed to verify if the application is secured on web as data theft and unauthorized access are more common issues and below are some of the techniques to verify the security level of the system. Injection Broken Authentication and Session Management Cross-Site Scripting (XSS) Insecure Direct Object References Security Misconfiguration Sensitive Data Exposure Missing Function Level Access Control Cross-Site Request Forgery (CSRF) Using Components with Known Vulnerabilities Invalidated Redirects and Forwards

Software Testing

Anuradha Bhatia 60

4. Test Management CONTENTS

I. Test Planning

i. Preparing a Test Plan

ii. Scope Management

iii. Deciding Test Approach

iv. Setting Up Criteria for Testing

v. Identifying Responsibilities

vi. Staffing, Training Needs, Resource Requirements

vii. Test Deliverables

viii. Testing Tasks

II. Test Management

III. Test Process

i. Base Lining a Test Plan

ii. Test Case Specification

iii. Update of Traceability Matrix

iv. Executing Test Cases

v. Collecting and Analyzing Metrics

vi. Preparing Test Summary Report

IV. Test Reporting: Recommending Product Release.

Software Testing 4.Test Management

Anuradha Bhatia 61

I. Test Planning i. Test plan is the project plan for the testing work to be done.

ii. It is not a test design specification, a collection of test cases or a set of test

procedures; in fact, most of our test plans do not address that level of detail.

iii. Many people have different definitions for test plans.

1. Preparing a Test Plan

(Question: Write steps to create test plan- 4 Marks)

i. Test Plan is a document that is the point of reference based on which testing

is carried out within the QA team.

ii. Test plan is not static and is updated on an on demand basis.

iii. The more detailed and comprehensive the Test plan, the more successful the

testing activity.

iv. The test plan keeps track of possible tests that will be run on the system after

coding.

v. The test plan is a document that develops as the project is being developed.

vi. Record tests as they come up

vii. Test error prone parts of software development.

viii. The initial test plan is abstract and the final test plan is concrete.

ix. The initial test plan contains high level ideas about testing the system without

getting into the details of exact test cases.

x. The most important test cases come from the requirements of the system.

xi. When the system is in the design stage, the initial tests can be refined a little.

xii. During the detailed design or coding phase, exact test cases start to

materialize.

xiii. After coding, the test points are all identified and the entire test plan is

exercised on the software.

2. Purpose of Software Test Plan

(Question: Write steps for purpose of software test plan- 4 Marks)

i. Test Plan Ensures all Functional and Design Requirements are implemented

as specified in the documentation.

ii. To provide a procedure for Unit and System Testing.

iii. To identify the documentation process for Unit and System Testing.

iv. To identify the test methods for Unit and System Testing.

http://istqbexamcertification.com/what-is-fundamental-test-process-in-software-testing/

http://istqbexamcertification.com/what-is-test-design-technique/


Anuradha Bhatia 62

3. Advantages of test plan

i. Serves as a guide to testing throughout the development.

ii. We only need to define test points during the testing phase.

iii. Serves as a valuable record of what testing was done.

iv. The entire test plan can be reused if regression testing is done later on.

v. The test plan itself could have defects just like software!

4. Scope Management

(Question: Explain Scope Management of test plan- 4 Marks)

i. Before starting Test Planning activity, scope of the test activities should be

known.

ii. The information on what features will be tested and what will not be tested.

iii. The information on what areas your team is owning?

iv. Are you taking care of all the types of testing that is required for the product

including Performance, Security, globalization etc.?

v. Defining scope for your testing project is very important for the management

as well.

vi. If scope is properly defined, everyone will have clear understanding about

what is tested and what is not.

5. Setting Up Criteria for Testing

(Question: Explain the setting up criteria for test plan- 4 Marks)

i. Criteria for Entry and Exit should be clearly defined for every activity of your

testing project.

ii. Well defined entry/exit criteria for starting, stopping, suspending and

resuming test activities.

iii. The criteria should be defined for specifying when testing is complete.

6. Staffing, Training Needs, Resource Requirements

i. Mostly, testing project follows development activities.

ii. So for estimation and scheduling you should have information on the

development plan and milestone.

iii. Once you have information on the development plan, you can schedule your

testing activities accordingly.

iv. Resources in testing projects include hardware, software and people

management.

7. Test Deliverables

i. The information about what tools you are using to manage your testing

activities.

http://www.softwaretestingtimes.com/2010/05/purpose-advantages-of-test-plan.html


Anuradha Bhatia 63

ii. The information on the configuration management for test artifacts, test case

management tool, defect tracking system, tools for automation etc.

iii. Ideally, test automation should be treated as separate project and should have

brief information here along with the link to automation plan.

8. Identifying Responsibilities

Roles and Responsibilities

=>

Team members are listed

Who is to do what

module owners are listed and their contact info

Deliverables => What documents(test artifacts) are going to produce at what

time frames

What can be expected from each document

Environment => What kind of environment requirements exist

Who is going to be in charge

What to do in case of problems

Tools => For example: JIRA for bug tracking

Login

How to use JIRA

Defect Management => Who are we going to report the defects to

How are we going to report

What is expected- do we provide screenshot?

Risks and Risk Management

=>

Risks are listed

Risks are analyzed- likelihood and impact is documented

Risk mitigation plans are drawn

Exit criteria => When to stop testing


Anuradha Bhatia 64

9. Testing Tasks

i . The information on how execution will be managed for the various

testing activities

ii . What kind of reports you are planning to generate from the data

that you gather from test activities.

iii . This should have information on the various matrixes and how

they should be interpreted.

II. Test management i . Test management most commonly refers to the activity of managing

the computer software testing process.

ii . A test management tool is software used to manage tests (automated

or manual) that have been previously specified by a test procedure.

iii . It is often associated with automation software.

iv. Test management tools often include requirement and/or

specification management modules that al low automatic generation

of the requirement test matrix (RTM), which is one of the main

metrics to indicate functional coverage of a system under test (SUT).

1. Creating tests definitions in a database[edit]

i . Test definit ion includes: test plan, association with product

requirements and specifications.

ii . Eventually, some relationship can be set between tests so that

precedence can be established.

E.g. if test A is parent of test B and if test A is fail ing, then it

may be useless to perform test B. Tests should also be associated

with priorities. Every change on a test must be versioned so that

the QA team has a comprehensive view of the history of the test.

2. Preparing test campaigns

i . This includes building some bundles of test cases and execute

them (or scheduling their execution).

ii . Execution can be either manual or automatic.

3. Manual execution

i . The user will have to perform all the test steps manually and

inform the system of the result.

ii . Some test management tools includes a framework to interface the

user with the test plan to facilitate this task.


Anuradha Bhatia 65

iii . There are several way to run tests. The simplest way to run a test

is to run a test case.

iv. The test case can be associated with other test artifacts such as

test plans, test scripts, test environments, test case execution

records, and test suites.

4. Automatic execution

i . There are numerous ways of implementing automated tests.

ii . Automatic execution requires the test management tool to be

compatible with the tests themselves.

iii . To do so, test management tools may propose proprietary

automation frameworks or APIs to interface with third -party or

proprietary automated tests.

5. Generating reports and metrics

i . The ultimate goal of test management tools is to deliver sensitive

metrics that will help the QA manager in evaluating the quality of

the system under test before releasing.

ii . Metrics are generally presented as graphics and tables indic ating

success rates, progression/regression and much other sensit ive

data.

6. Managing bugs

i . Eventually, test management tools can integrate bug tracking

features or at least interface with well -known dedicated bug

tracking solutions (such as Bugzilla or Mantis) efficiently link a

test failure with a bug.

7. Planning test activities

i . Test management tools may also integrate (or interface with third -

party) project management functionalities to help the QA manager

planning activities ahead of time.

8. Test management tools

i . There are several commercial and open source test management

tools available in the market today which include HP Quality

Center and IBM Rational Quality Manager.

ii . Most test management tools are web -served applications that need

to be installed in-house. Others can be accessed as Software as a

service.[citation needed]


Anuradha Bhatia 66

III. TEST PROCESS i . Testing is a process rather than a single activity.

ii . This process starts from test planning then designing test cases,

preparing for execution and evaluating status till the test closure.

iii . Divide the activities within the fundamental test process into the

following basic steps:

1. Planning and Control

Test planning has following major tasks:

i . To determine the scope and risks and identify the objectives of

testing.

ii . To determine the test approach.

iii . To implement the test policy and/or the test strategy . (Test

strategy is an outl ine that describes the testing portion of the

software development cycle. It is created to inform PM, testers

and developers about some key issues of the t esting process. This

includes the testing objectives, method of testing, total t ime and

resources required for the project and the testing environments.).

iv. To determine the required test resources like people, test

environments, PCs, etc.

v. To schedule test analysis and design tasks, test implementation,

execution and evaluation.

vi. To determine the Exit criteria we need to set criteria such as

Coverage criteria. (Coverage criteria are the percentage of

statements in the software that must be executed during test ing.

This will help us track whether we are completing test activities

correctly. They will show us which tasks and checks we must

complete for a particular level of testing before we can say that

testing is finished.)

Test control has the following major tasks:

i . To measure and analyze the results of reviews and testing.

ii . To monitor and document progress, test coverage and exit criteria.

iii . To provide information on testing.

iv. To initiate corrective actions.

v. To make decisions.


Anuradha Bhatia 67

2. Analysis and Design

Test analysis and Test Design has the following major tasks:

i . To review the test basis. (The test basis is the information we

need in order to start the test analysis and create our own test

cases. Basically it’s a documentation on which test cases are

based, such as requirements, design specifications, product risk

analysis, architecture and interfaces. We can use the test basis

documents to understand what the system should do once buil t.)

ii . To identify test conditions.

iii . To design the tests.

iv. To evaluate testabil ity of the requirements and system.

v. To design the test environment set -up and identify and required

infrastructure and tools.

3. Implementation and Execution

i . During test implementation and execution, take the test conditions

into test cases and procedures and other test ware such as scripts

for automation, the test environment and any other test

infrastructure.

ii . (Test cases is a set of conditions under which a tester will determine

whether an application is working correctly or not.)

iii . (Test ware is a term for all utilities that serve in combination for

testing a software like scripts, the test environment and any other

test infrastructure for later reuse.)

Test implementation has the following major task:

i . To develop and prioritize our test cases by using techniques and

create test data for those tests.

ii . (In order to test a software application you need to enter some

data for test ing most of the features.

iii . Any such specifically identified data which is used in tests is

known as test data.)

iv. We also write some instructions for carrying out the tests which

is known as test procedures.

v. We may also need to automate some tests using test harness and

automated tests scripts.

vi. (A test harness is a collection of software and test data for test ing

a program unit by running it under different conditions and

monitoring its behavior and outputs.)


Anuradha Bhatia 68

vii. To create test suites from the test cases for efficient test

execution.

viii. (Test suite is a collection of test cases that are used to test a

software program to show that it has some specified set of

behaviors.

ix. A test suite often contains detailed instructions and information

for each collection of test cases on the system configuration to be

used during testing.

x. Test suites are used to group similar test cases toge ther.)

xi. To implement and verify the environment.

Test execution has the following major task:

i . To execute test suites and individual test cases following the test

procedures.

ii . To re-execute the tests that previously failed in order to confirm

a fix. This is known as confirmation testing or re-testing.

i ii . To log the outcome of the test execution and record the identi ties

and versions of the software under tests.

iv. The test log is used for the audit trial.

v. (A test log is nothing but, what are the test cases that we executed,

in what order we executed, who executed that test cases and what

is the status of the test case (pass/fail).

vi. These descriptions are documented and called as test log.).

vii. To compare actual results with expected results.

viii. Where there are differences between actual and expected results,

it report discrepancies as Incidents.

4. Evaluating Exit criteria and Reporting

i . Based on the risk assessment of the project we will set the criteria

for each test level against which we will measure the “enough

testing”. These criteria vary from project to project and are known

as exit criteria .

i i . Exit criteria come into picture, when:

Maximum test cases are executed with certain pass percentage.

Bug rate falls below certain level.

When achieved the deadlines.

iii . Evaluating exit criteria has the following major tasks

To check the test logs against the exit cri teria specified in test

planning.


Anuradha Bhatia 69

To assess if more test are needed or if the exit criteria specified

should be changed.

To write a test summary report for stakeholders.

5. Test Closure activities:

Test closure activities are done when software is delivered. The

testing can be closed for the other reasons also l ike:

When all the information has been gathered which are needed

for the testing.

When a project is cancelled.

When some target is achieved.

When a maintenance release or update is done.

Test closure activities have the following major tasks:

To check which planned deliverables are actually delivered and

to ensure that all incident reports have been resolved.

To finalize and archive test ware such as scripts, test

environments, etc. for later reuse.

To handover the test ware to the maintenance organization.

They will give support to the software.

To evaluate how the testing went and learn lessons for future

releases and projects.

Software Testing

Anuradha Bhatia 70

5. Defect

Management

Objectives CONTENTS

I. Introduction, Defect Classification, Defect Management Process

II. Defect Life Cycle, Defect Template III. Estimate Expected Impact of a Defect, Techniques for

Finding Defects, Reporting a Defect.

Software Testing 5. Defect Management Objectives

Anuradha Bhatia 71

I. Introduction i. Software defects are expensive.

ii. The cost of finding and correcting defects represents one of the most expensive software development activities.

iii. While defects may be inevitable, we can minimize their number and impact on our projects.

iv. To do this development teams need to implement a defect management process that focuses on preventing defects, catching defects as early in the process as possible, and minimizing the impact of defects.

v. A little investment in this process can yield significant returns.

1. Defect Classification (Question: Explain the defect classification. – 8 Marks)

i. A Software Defect / Bug is a condition in a software product which does not

meet a software requirement (as stated in the requirement specifications) or

end-user expectations (which may not be specified but are reasonable).

ii. In other words, a defect is an error in coding or logic that causes a program to

malfunction or to produce incorrect/unexpected results.

iii. A program that contains a large number of bugs is said to be buggy.

iv. Reports detailing bugs in software are known as bug reports.

v. Applications for tracking bugs are known as bug tracking tools.

vi. The process of finding the cause of bugs is known as debugging.

vii. The process of intentionally injecting bugs in a software program, to estimate

test coverage by monitoring the detection of those bugs, is known

as bebugging.

There are various ways in which we can classify.

Severity Wise: i. Major: A defect, which will cause an observable product failure or departure

from requirements.

ii. Minor: A defect that will not cause a failure in execution of the product.

iii. Fatal: A defect that will cause the system to crash or close abruptly or effect other applications.

Type of Errors Wise i. Comments: Inadequate/ incorrect/ misleading or missing comments in the

source code

ii. Computational Error: Improper computation of the formulae / improper business validations in code.


Anuradha Bhatia 72

iii. Data error: Incorrect data population / update in database

iv. Database Error: Error in the database schema/Design

v. Missing Design: Design features/approach missed/not documented in the design document and hence does not correspond to requirements

vi. Inadequate or sub optimal Design: Design features/approach needs additional inputs for it to be complete Design features described does not provide the best approach (optimal approach) towards the solution required

vii. In correct Design: Wrong or inaccurate Design

viii. Ambiguous Design: Design feature/approach is not clear to the reviewer. Also includes ambiguous use of words or unclear design features.

ix. Boundary Conditions Neglected: Boundary conditions not addressed/incorrect

x. Interface Error: Internal or external to application interfacing error, Incorrect handling of passing parameters, Incorrect alignment, incorrect/misplaced fields/objects, un friendly window/screen positions

xi. Logic Error: Missing or Inadequate or irrelevant or ambiguous functionality in source code

xii. Message Error: Inadequate/ incorrect/ misleading or missing error messages in source code

xiii. Navigation Error: Navigation not coded correctly in source code

xiv. Performance Error: An error related to performance/optimality of the code

xv. Missing Requirements: Implicit/Explicit requirements are missed/not documented during requirement phase

xvi. Inadequate Requirements: Requirement needs additional inputs for to be complete

xvii. Incorrect Requirements: Wrong or inaccurate requirements

xviii. Ambiguous Requirements: Requirement is not clear to the reviewer. Also includes ambiguous use of words – e.g. Like, such as, may be, could be, might etc.

xix. Sequencing / Timing Error: Error due to incorrect/missing consideration to timeouts and improper/missing sequencing in source code.

xx. Standards: Standards not followed like improper exception handling, use of E & D Formats and project related design/requirements/coding standards

xxi. System Error: Hardware and Operating System related error, Memory leak

xxii. Test Plan / Cases Error: Inadequate/ incorrect/ ambiguous or duplicate or missing - Test Plan/ Test Cases & Test Scripts, Incorrect/Incomplete test setup

xxiii. Typographical Error: Spelling / Grammar mistake in documents/source code

xxiv. Variable Declaration Error: Improper declaration / usage of variables, Type mismatch error in source code


Anuradha Bhatia 73

Status Wise:

i. Open

ii. Closed

iii. Deferred

iv. Cancelled

II. Defect Management Process

(Question: Explain the defect management process in software testing with a neat diagram. – 4 Marks)

Figure 5.1: Defect management Process

i. Defect Prevention-- Implementation of techniques, methodology and standard processes to reduce the risk of defects.

ii. Deliverable Baseline-- Establishment of milestones where deliverables will be considered complete and ready for further development work. When a deliverable is base lined, any further changes are controlled. Errors in a deliverable are not considered defects until after the deliverable is base lined.

iii. Defect Discovery-- Identification and reporting of defects for development team acknowledgment. A defect is only termed discovered when it has been documented and acknowledged as a valid defect by the development team member(s) responsible for the component(s) in error.

iv. Defect Resolution-- Work by the development team to prioritize, schedule and fix a defect, and document the resolution. This also includes notification back to the tester to ensure that the resolution is verified.


Anuradha Bhatia 74

III. Defect life cycle and Defect Template 1. Defect life cycle

(Question: Explain the defect life cycle or a bug life cycle in software

testing. – 8 Marks) i. Defect Life Cycle (Bug Life cycle) is the journey of a defect from its

identification to its closure.

ii. The Life Cycle varies from organization to organization and is governed by the

software testing process the organization or project follows and/or the Defect

tracking tool being used.

Nevertheless, the life cycle in general resembles the following:

Figure 5.2: Bugs Life Cycle

Status Alternative Status

NEW

ASSIGNED OPEN

DEFERRED

DROPPED REJECTED


Anuradha Bhatia 75

COMPLETED FIXED, RESOLVED, TEST

REASSIGNED REOPENED

CLOSED VERIFIED

Table 5.1: Defect Status

Defect Status Explanation

i. NEW: Tester finds a defect and posts it with the status NEW. This defect is yet

to be studied/approved. The fate of a NEW defect is one of ASSIGNED,

DROPPED and DEFERRED.

ii. ASSIGNED / OPEN: Test / Development / Project lead studies the NEW defect

and if it is found to be valid it is assigned to a member of the Development

Team. The assigned Developer’s responsibility is now to fix the defect and have

it COMPLETED. Sometimes, ASSIGNED and OPEN can be different statuses. In

that case, a defect can be open yet unassigned.

iii. DEFERRED: If a valid NEW or ASSIGNED defect is decided to be fixed in

upcoming releases instead of the current release it is DEFERRED. This defect is

ASSIGNED when the time comes.

iv. DROPPED / REJECTED: Test / Development/ Project lead studies the NEW

defect and if it is found to be invalid, it is DROPPED / REJECTED. Note that the

specific reason for this action needs to be given.

v. COMPLETED / FIXED / RESOLVED / TEST: Developer ‘fixes’ the defect that is

ASSIGNED to him or her. Now, the ‘fixed’ defect needs to be verified by the

Test Team and the Development Team ‘assigns’ the defect back to the Test

Team. A COMPLETED defect is either CLOSED, if fine, or REASSIGNED, if still

not fine.

vi. If a Developer cannot fix a defect, some organizations may offer the following

statuses:

Won’t Fix / Can’t Fix: The Developer will not or cannot fix the defect due

to some reason.

Can’t Reproduce: The Developer is unable to reproduce the defect.

Need More Information: The Developer needs more information on the

defect from the Tester.

vii. REASSIGNED / REOPENED: If the Tester finds that the ‘fixed’ defect is in fact

not fixed or only partially fixed, it is reassigned to the Developer who ‘fixed’ it.

A REASSIGNED defect needs to be COMPLETED again.

viii. CLOSED / VERIFIED: If the Tester / Test Lead finds that the defect is indeed

fixed and is no more of any concern, it is CLOSED / VERIFIED. This is the happy

ending.


Anuradha Bhatia 76

IV. Defect Template (Question: Create the bug template for a login form. – 4Marks)

i. Reporting a bug/defect properly is as important as finding a defect.

ii. If the defect found is not logged/reported correctly and clearly in bug tracking

tools (like Bugzilla, ClearQuest etc.) then it won’t be addressed properly by the

developers, so it is very important to fill as much information as possible in the

defect template so that it is very easy to understand the actual issue with the

software.

Sample defect template

Abstract :

Platform :

Testcase Name :

Release :

Build Level :

Client Machine IP/Hostname :

Client OS :

Server Machine IP/Hostname :

Server OS :

Defect Type :

Priority :

Sevierity :

Developer Contacted :

Test Contact Person :

Attachments :

Any Workaround :

Steps to Reproduce

1.

2.

3.

Expected Result:

Actual Result:


Anuradha Bhatia 77

Defect report template i. A defect reporting tool is used and the elements of a report can vary.

ii. A defect report can consist of the following elements.

ID Unique identifier given to the defect. (Usually Automated)

Project Project name.

Product Product name.

Release Version Release version of the product. (e.g. 1.2.3)

Module Specific module of the product where the defect was detected.

Detected Build

Version

Build version of the product where the defect was detected

(e.g. 1.2.3.5)

Summary Summary of the defect. Keep this clear and concise.

Description

Detailed description of the defect. Describe as much as possible

but without repeating anything or using complex words. Keep

it simple but comprehensive.

Steps to Replicate

Step by step description of the way to reproduce the defect.

Number the steps.

Actual Result The actual result you received when you followed the steps.

Expected Results The expected results.

Attachments Attach any additional information like screenshots and logs.

Remarks Any additional comments on the defect.


Anuradha Bhatia 78

Defect Severity Severity of the Defect.

Defect Priority Priority of the Defect.

Reported By The name of the person who reported the defect.

Assigned To

The name of the person that is assigned to analyze/fix the

defect.

Status The status of the defect. (See Defect Life Cycle)

Fixed Build Version

Build version of the product where the defect was fixed (e.g.

1.2.3.9)

Table 5.2: Defect Report Template

Defect tracking tools

Following are some of the commonly used defect tracking tools:

i. Bugzilla - Open Source Bug Tracking.

ii. Testlink - Open Source Bug Tracking.

iii. ClearQuest – Defect tracking tool by IBM Rational tools.

iv. HP Quality Center– Test Management tool by HP.

VI. Estimate Expected Impact of a Defect, Techniques for Finding Defects, Reporting a Defect.

1. Estimate Expected Impact of a Defect i. There is a strong relationship between the number of test cases and the

number of function points. ii. There is a strong relationship between the number of defects and the

number of test cases and number of function points. iii. The number of acceptance test cases can be estimated by multiplying the

number of function points by 1.2. iv. Acceptance test cases should be independent of technology and

implementation techniques. v. If a software project was 100 function points the estimated number of test

cases would be 120. vi. To estimate the number of potential defects is more involved.

http://softwaretestingfundamentals.com/defect-life-cycle/


Anuradha Bhatia 79

a) Estimating Defects i. Intuitively the number of maximum potential defects is equal to the

number of acceptance test cases which is 1.2 x Function Points.

b) Preventing, Discovering and Removing Defects i. To reduce the number of defects delivered with a software project an

organization can engage in a variety of activities. ii. While defect prevention is much more effective and efficient in reducing

the number of defects, most organization conduct defect discovery and removal.

iii. Discovering and removing defects is an expensive and inefficient process. iv. It is much more efficient for an organization to conduct activities that

prevent defects.

c) Defect Removal Efficiency i. If an organization has no defect prevention methods in place then they are

totally reliant on defect removal efficiency.

Figure 5.3: Defect Removal Efficiency

1. Requirements Reviews up to 15% removal of potential defects. 2. Design Reviews up to 30% removal of potential defects. 3. Code Reviews up to 20% removal of potential defects. 4. Formal Testing up to 25% removal of potential defects.


Anuradha Bhatia 80

d) Defect Discovery and Removal

Size in Function Points

Totals Defects Remaining

Max Defects Perfect Medium Poor

100 120 12 66 102

200 240 24 132 204

500 600 60 330 510

1,000 1,200 120 660 1,020

2,500 3,000 300 1,650 2,550

5,000 6,000 600 3,300 5,100

10,000 12,000 1,200 6,600 10,200

20,000 24,000 2,000 13,200 20,400

Table 3: Defect Removal and Recovery

i. An organization with a project of 2,500 function points and was about medium at defect discovery and removal would have 1,650 defects remaining after all defect removal and discovery activities.

ii. The calculation is 2,500 x 1.2 = 3,000 potential defects. iii. The organization would be able to remove about 45% of the defects or 1,350 defects. iv. The total potential defects (3,000) less the removed defects (1,350) equals the remaining

defects of 1,650.

e) Defect Prevention

If an organization concentrates on defect prevention (instead of defect detection) then the number of defects inserted or created is much less. The amount of time and effort required to discover and remove this defects is much less also.

i. Roles and Responsibilities Clearly Defined up to 15% reduction in number of defects created

ii. Formalized Procedures up to 25% reduction in number of defects created iii. Repeatable Processes up to 35% reduction in number of defects created iv. Controls and Measures in place up to 30% reduction in number of defects

created.


Anuradha Bhatia 81

2. Techniques to find defects (Question: Explain any two techniques to find the defect with strength and weakness. – 8 Marks)

a) Quick Attacks:

i. Strengths

The quick-attacks technique allows you to perform a cursory analysis

of a system in a very compressed timeframe.

Even without a specification, you know a little bit about the software,

so the time spent is also time invested in developing expertise.

The skill is relatively easy to learn, and once you've attained some

mastery your quick-attack session will probably produce a few bugs.

Finally, quick attacks are quick.

They can help you to make a rapid assessment. You may not know the

requirements, but if your attacks yielded a lot of bugs, the

programmers probably aren't thinking about exceptional conditions,

and it's also likely that they made mistakes in the main functionality.

If your attacks don't yield any defects, you may have some confidence

in the general, happy-path functionality.

ii. Weaknesses

Quick attacks are often criticized for finding "bugs that don't matter"—

especially for internal applications.

While easy mastery of this skill is a strength, it creates the risk that

quick attacks are "all there is" to testing; thus, anyone who takes a two-

day course can do the work.

b) Equivalence and Boundary Conditions

i. Strengths

Boundaries and equivalence classes give us a technique to reduce an

infinite test set into something manageable.

They also provide a mechanism for us to show that the requirements

are "covered".

ii. Weaknesses

The "classes" in the table in Figure 1 are correct only in the mind of the

person who chose them.


Anuradha Bhatia 82

We have no idea whether other, "hidden" classes exist—for example, if a numeric number that represents time is compared to another time as a set of characters, or a "string," it will work just fine for most numbers.

c) Common Failure Modes

i. Strengths

The heart of this method is to figure out what failures are common for

the platform, the project, or the team; then try that test again on this

build.

If your team is new, or you haven't previously tracked bugs, you can

still write down defects that "feel" recurring as they occur—and start

checking for them.

ii. Weaknesses

In addition to losing its potency over time, this technique also entirely

fails to find "black swans"—defects that exist outside the team's recent

experience.

The more your team stretches itself (using a new database, new

programming language, new team members, etc.), the riskier the

project will be—and, at the same time, the less valuable this technique

will be.

d) State-Transition Diagrams

Figure 5.4: State Transition Map


Anuradha Bhatia 83

i. Strengths

Mapping out the application provides a list of immediate, powerful test

ideas.

Model can be improved by collaborating with the whole team to find

"hidden" states—transitions that might be known only by the original

programmer or specification author.

Once you have the map, you can have other people draw their own

diagrams, and then compare theirs to yours.

The differences in those maps can indicate gaps in the requirements,

defects in the software, or at least different expectations among team

members.

ii. Weaknesses

The map you draw doesn't actually reflect how the software will

operate; in other words, "the map is not the territory."

Drawing a diagram won't find these differences, and it might even give

the team the illusion of certainty.

Like just about every other technique on this list, a state-transition

diagram can be helpful, but it's not sufficient by itself to test an entire

application.

e) Use Cases and Soap Opera Tests

Use cases and scenarios focus on software in its role to enable a human being

to do something.

i. Strengths

Use cases and scenarios tend to resonate with business customers, and

if done as part of the requirement process, they sort of magically

generate test cases from the requirements.

They make sense and can provide a straightforward set of confirmatory

tests. Soap opera tests offer more power, and they can combine many

test types into one execution.


Anuradha Bhatia 84

ii. Weaknesses

Soap opera tests have the opposite problem; they're so complex that

if something goes wrong, it may take a fair bit of troubleshooting to

find exactly where the error came from!

f) Code-Based Coverage Models

Imagine that you have a black-box recorder that writes down every single line of

code as it executes.

i. Strengths

Programmers love code coverage. It allows them to attach a number—

an actual, hard, real number, such as 75%—to the performance of their

unit tests, and they can challenge themselves to improve the score.

Meanwhile, looking at the code that isn't covered also can yield

opportunities for improvement and bugs!

ii. Weaknesses

Customer-level coverage tools are expensive, programmer-level tools

that tend to assume the team is doing automated unit testing and has

a continuous-integration server and a fair bit of discipline.

After installing the tool, most people tend to focus on statement

coverage—the least powerful of the measures.

Even decision coverage doesn't deal with situations where the decision

contains defects, or when there are other, hidden equivalence classes;

say, in the third-party library that isn't measured in the same way as

your compiled source code is.

Having code-coverage numbers can be helpful, but using them as a

form of process control can actually encourage wrong behaviours. In

my experience, it's often best to leave these measures to the

programmers, to measure optionally for personal improvement (and

to find dead spots), not as a proxy for actual quality.

g) Regression and High-Volume Test Techniques

People spend a lot of money on regression testing, taking the old test

ideas described above and rerunning them over and over.


Anuradha Bhatia 85

This is generally done with either expensive users or very expensive

programmers spending a lot of time writing and later maintaining

those automated tests.

i. Strengths

For the right kind of problem, say an IT shop processing files through a

database, this kind of technique can be extremely powerful.

Likewise, if the software deliverable is a report written in SQL, you can

hand the problem to other people in plain English, have them write

their own SQL statements, and compare the results.

Unlike state-transition diagrams, this method shines at finding the

hidden state in devices. For a pacemaker or a missile-launch device,

finding those issues can be pretty important.

ii. Weaknesses

Building a record/playback/capture rig for a GUI can be extremely

expensive, and it might be difficult to tell whether the application

hasn't broken, but has changed in a minor way.

For the most part, these techniques seem to have found a niche in

IT/database work, at large companies like Microsoft and AT&T, which

can have programming testers doing this work in addition to traditional

testing, or finding large errors such as crashes without having to

understand the details of the business logic.

While some software projects seem ready-made for this approach,

others...aren't.

You could waste a fair bit of money and time trying to figure out where

your project falls.

3. Reporting defects effectively (Question: Explain how defects can be effectively reported. – 4 Marks)

It is essential that you report defects effectively so that time and effort is not

unnecessarily wasted in trying to understand and reproduce the defect. Here are

some guidelines:

i. Be specific:

Specify the exact action: Do not say something like ‘Select Button B’.

Do you mean ‘Click Button B’ or ‘Press ALT+B’ or ‘Focus on Button B and

click ENTER’.


Anuradha Bhatia 86

In case of multiple paths, mention the exact path you followed: Do not say

something like “If you do ‘A and X’ or ‘B and Y’ or ‘C and Z’, you get D.”

Understanding all the paths at once will be difficult. Instead, say “Do ‘A and

X’ and you get D.” You can, of course, mention elsewhere in the report that

“D can also be got if you do ‘B and Y’ or ‘C and Z’.”

Do not use vague pronouns: Do not say something like “In Application A,

open X, Y, and Z, and then close it.” What does the ‘it’ stand for? ‘Z’ or, ‘Y’,

or ‘X’ or ‘Application A’?”

ii. Be detailed:

Provide more information (not less). In other words, do not be lazy.

Developers may or may not use all the information you provide but they

sure do not want to beg you for any information you have missed.

iii. Be objective:

Do not make subjective statements like “This is a lousy application” or “You

fixed it real bad.”

Stick to the facts and avoid the emotions.

iv. Reproduce the defect:

Do not be impatient and file a defect report as soon as you uncover a

defect. Replicate it at least once more to be sure.

v. Review the report:

Do not hit ‘Submit’ as soon as you write the report.

Review it at least once.

Remove any typing errors.

Software Testing

Anuradha Bhatia 87

6. Testing Tools and

Measurement CONTENTS

I. Limitation of Manual Testing and Need for Automated Testing Tool.

II. Features of Test Tool: Guidelines for Static and Dynamic Testing Tools.

III. Advantage and Disadvantage of Using Tools. IV. Selecting a Testing Tool. V. When to Use Automated Test Tools, Testing Using

Automated Tool. VI. What are Metrics and Measurements: Types of Metrics,

Project Metrics, Progress and Productivity Metrics?

Software Testing 6. Testing Tools and Measurement

Anuradha Bhatia 88

I. Limitation of Manual Testing and Need for Automated Testing Tool.

i. Manual testing is slow and costly. ii. It is very labor intensive, it takes a long time to complete tests.

iii. Manual tests don’t scale well. As the complexity of the software increases the complexity of the testing problem grows exponentially. This leads to an increase in total time devoted to testing as well as total cost of testing.

iv. Manual testing is not consistent or repeatable. Variations in how the tests are performed as inevitable, for various reasons. One tester may approach and perform a certain test differently from another, resulting in different results on the same test, because the tests are not being performed identically.

v. Lack of training is the common problem, although not unique to manual software testing.

vi. GUI objects size difference and color combinations are not easy to find in manual testing.

vii. Not suitable for large scale projects and time bound projects. viii. Batch testing is not possible, for each and every test execution Human user

interaction is mandatory. ix. Comparing large amount of data is impractical. x. Processing change requests during software maintenance takes more time.

NEED of automated testing

i. An automated testing tool is able to playback pre-recorded and predefined actions, compare the results to the expected behavior and report the success or failure of these manual tests to a test engineer.

ii. Once automated tests are created they can easily be repeated and they can be extended to perform tasks impossible with manual testing.

iii. Because of this, savvy managers have found that automated software testing is an essential component of successful development projects.

1. Automated Software Testing Saves Time and Money

i. Software tests have to be repeated often during development cycles to ensure quality. Every time source code is modified software tests should be repeated.

ii. For each release of the software it may be tested on all supported operating systems and hardware configurations.

iii. Manually repeating these tests is costly and time consuming. Once created, automated tests can be run over and over again at no additional cost and they are much faster than manual tests.

iv. Automated software testing can reduce the time to run repetitive tests from days to hours.

v. A time savings that translates directly into cost savings.

http://smartbear.com/products/qa-tools/automated-testing-tools


Anuradha Bhatia 89

2. Testing Improves Accuracy

i. Even the most conscientious tester will make mistakes during monotonous manual testing.

ii. Automated tests perform the same steps precisely every time they are executed and never forget to record detailed results.

3. Increase Test Coverage

i. Automated software testing can increase the depth and scope of tests to help improve software quality.

ii. Lengthy tests that are often avoided during manual testing can be run unattended.

iii. They can even be run on multiple computers with different configurations. iv. Automated software testing can look inside an application and see memory

contents, data tables, file contents, and internal program states to determine if the product is behaving as expected.

v. Automated software tests can easily execute thousands of different complex test cases during every test run providing coverage that is impossible with manual tests.

vi. Testers freed from repetitive manual tests have more time to create new automated software tests and deal with complex features.

4. Automation Does What Manual Testing Cannot

i. Even the largest software departments cannot perform a controlled web application test with thousands of users.

ii. Automated testing can simulate tens, hundreds or thousands of virtual users interacting with network or web software and applications.

5. Automated QA Testing Helps Developers and Testers

i. Shared automated tests can be used by developers to catch problems quickly before sending to QA.

ii. Tests can run automatically whenever source code changes are checked in and notify the team or the developer if they fail.

iii. Features like these save developers time and increase their confidence.

6. Team Morale Improves

i. This is hard to measure but we’ve experienced it first hand, automated software testing can improve team morale.

ii. Automating repetitive tasks with automated software testing gives your team time to spend on more challenging and rewarding projects.

iii. Team members improve their skill sets and confidence and, in turn, pass those gains on to their organization.


Anuradha Bhatia 90

7. Test Complete is a Powerful and Affordable Automated

Software Testing Tool

i. Test Complete addresses a full range of software testing challenges facing corporate IT departments, product developers, QA engineers, and consultants.

ii. Test Complete enhances the software testing process by increasing efficiency, removing complexity and lowering costs

ADVANTAGES

Automated Testing Manual Testing

• If you have to run a set of tests repeatedly automation is a huge gain

• If Test Cases have to be run a small number of times it's more likely to perform manual testing

• Helps performing "compatibility testing" - testing the software on different configurations

• It allows the tester to perform more ad-hoc (random testing)

• It gives you the ability to run automation scenarios to perform regressions in a shorter time

• Short term costs are reduced

• It gives you the ability to run regressions on a code that is continuously changing

• The more time tester spends testing a module the grater the odds to find real user bugs

• Can be run simultaneously on different machines thus decreasing testing time

• Long term costs are reduced

DISADVANTAGES

Automated Testing Manual Testing

• It's more expensive to automate. Initial investments are bigger than manual testing

• Manual tests can be very time consuming

• You cannot automate everything, some tests still have to be done manually

• For every release you must rerun the same set of tests which can be tiresome

OTHER FACTORS

• The performance of yout test tools

• The knowledge level of your testing team

• The continous growth of software to be tested

• Number of necessary regressions

II. Features of Test Tool: Guidelines for Static and Dynamic Testing Tools.

1. Static Testing Tool


Anuradha Bhatia 91

i. Static analysis tools are generally used by developers as part of the development

and component testing process. The key aspect is that the code (or other artefact)

is not executed or run but the tool itself is executed, and the source code we are

interested in is the input data to the tool.

ii. These tools are mostly used by developers.

iii. Static analysis tools are an extension of compiler technology – in fact some

compilers do offer static analysis features. It is worth checking what is available

from existing compilers or development environments before looking at

purchasing a more sophisticated static analysis tool.

iv. Other than software code, static analysis can also be carried out on things like,

static analysis of requirements or static analysis of websites (for example, to

assess for proper use of accessibility tags or the following of HTML standards).

v. Static analysis tools for code can help the developers to understand the structure

of the code, and can also be used to enforce coding standards.

2. Dynamic Testing Tool

i. Dynamic analysis tools are ‘dynamic’ because they require the code to be in

a running state. They are ‘analysis’ rather than ‘testing’ tools because they

analyze what is happening ‘behind the scenes’ that is in the code while the

software is running (whether being executed with test cases or being used in

operation).

ii. Let us take an example of a car to understand it in a better way. If you go to a

showroom of a car to buy it, you might sit in the car to see if is comfortable

and see what sound the doors make – this would be static analysis because

the car is not being driven. If you take a test drive, then you would check that

how the car performs when it is in the running mode e.g. the car turns right

when you turn the steering wheel clockwise or when you press the break then

how the car will respond and can also check the oil pressure or the brake fluid,

this would be dynamic analysis, it can only be done while the engine is running.

Features or characteristics of dynamic analysis tools are as follows:

i. To detect memory leaks;

ii. To identify pointer arithmetic errors such as null pointers;

iii. To identify time dependencies.


Anuradha Bhatia 92

Eventually when your computer’s response time gets slower and

slower, but it get improved after re-booting, this may be because of

the ‘memory leak’, where the programs do not correctly release blocks

of memory back to the operating system. Sooner or later the system

will run out of memory completely and stop. Hence, rebooting restores

all of the memory that was lost, so the performance of the system is

now restored to its normal state.

These tools would typically be used by developers in component

testing and component integration testing, e.g. when testing

middleware, when testing security or when looking for robustness

defects.

Another form of dynamic analysis for websites is to check whether

each link does actually link to something else (this type of tool may be

called a ‘web spider’). The tool does not know if you have linked to the

correct page, but at least it can find dead links, which may be helpful.

III. Advantage and Disadvantage of Using Tools.

There are many benefits that can be gained by using tools to support testing.

i. Reduction of repetitive work: Repetitive work is very boring if it is done

manually. People tend to make mistakes when doing the same task over

and over. Examples of this type of repetitive work include running

regression tests, entering the same test data again and again (can be done

by a test execution tool), checking against coding standards (which can be

done by a static analysis tool) or creating a specific test database (which can

be done by a test data preparation tool).

ii. Greater consistency and repeatability: People have tendency to do the

same task in a slightly different way even when they think they are

repeating something exactly. A tool will exactly reproduce what it did

before, so each time it is run the result is consistent.

iii. Objective assessment: If a person calculates a value from the software or

incident reports, by mistake they may omit something, or their own one-

sided preconceived judgments or convictions may lead them to interpret

that data incorrectly. Using a tool means that subjective preconceived


Anuradha Bhatia 93

notion is removed and the assessment is more repeatable and consistently

calculated. Examples include assessing the cyclomatic complexity or nesting

levels of a component (which can be done by a static analysis tool),

coverage (coverage measurement tool), system behavior (monitoring tools)

and incident statistics (test management tool).

iv. Ease of access to information about tests or testing: Information

presented visually is much easier for the human mind to understand and

interpret. For example, a chart or graph is a better way to show

information than a long list of numbers – this is why charts and graphs in

spreadsheets are so useful. Special purpose tools give these features

directly for the information they process. Examples include statistics and

graphs about test progress (test execution or test management tool),

incident rates (incident management or test management tool) and

performance (performance testing tool).

Disadvantage

i. Unrealistic expectations from the tool: Unrealistic expectations may be

one of the greatest risks to success with tools. The tools are just software

and we all know that there are many problems associated with any kind of

software. It is very important to have clear and realistic objectives for what

the tool can do.

ii. People often make mistakes by underestimating the time, cost and effort

for the initial introduction of a tool: Introducing something new into an

organization is hardly straightforward. Once you purchase a tool, you want

to have a number of people being able to use the tool in a way that will be

beneficial. There will be some technical issues to overcome, but there will

also be resistance from other people – both need to be handled in such a

way that the tool will be of great success.

iii. People frequently miscalculate the time and effort needed to achieve

significant and continuing benefits from the tool: Mostly in the initial

phase when the tool is new to the people, they miscalculate the time and

effort needed to achieve significant and continuing benefits from the tool.

Just think back to the last time you tried something new for the very first

time (learning to drive, riding a bike, skiing). Your first attempts were

unlikely to be very good but with more experience and practice you


Anuradha Bhatia 94

became much better. Using a testing tool for the first time will not be your

best use of the tool either. It takes time to develop ways of using the tool

in order to achieve what is expected.

iv. Mostly people underestimate the effort required to maintain the test

assets generated by the tool: Generally people underestimate the effort

required to maintain the test assets generated by the tool. Because of the

insufficient planning for maintenance of the assets that the tool produces

there are chances that the tool might end up as ‘shelf-ware’, along with

the previously listed risks.

v. People depend on the tool a lot (over-reliance on the tool): Since there

are many benefits that can be gained by using tools to support testing like

reduction of repetitive work, greater consistency and repeatability, etc.

people started to depend on the tool a lot. But the tools are just a software

they can do only what they have been designed to do (at least a good

quality tool can), but they cannot do everything. A tool can definitely help,

but it cannot replace the intelligence needed to know how best to use it,

and how to evaluate current and future uses of the tool. For example, a

test execution tool does not replace the need for good test design and

should not be used for every test – some tests are still better executed

manually. A test that takes a very long time to automate and will not be

run very often is better done manually.

IV. Selecting a Testing Tool. i. While introducing the tool in the organization it must match a need within

the organization, and solve that need in a way that is both effective and

efficient.

ii. The tool should help in building the strengths of the organization and should

also address its weaknesses. The organization needs to be ready for the

changes that will come along with the new tool. If the current testing

practices are not good enough and the organization is not mature, then it is

always recommended to improve testing practices first rather than to try to

find tools to support poor practices. Automating chaos just gives faster

chaos!

i. Certainly, we can sometimes improve our own processes in parallel with

introducing a tool to support those practices and we can always pick up some


Anuradha Bhatia 95

good ideas for improvement from the ways that the tools work. However, do

not depend on the tool for everything, but it should provide support to your

organization as expected.

The following factors are important during tool selection:

i. Assessment of the organization’s maturity (e.g. readiness for change);

ii. Identification of the areas within the organization where tool support will help

to improve testing processes;

iii. Evaluation of tools against clear requirements and objective criteria;

iv. Proof-of-concept to see whether the product works as desired and meets the

requirements and objectives defined for it;

v. Evaluation of the vendor (training, support and other commercial aspects) or

open-source network of support;

vi. Identifying and planning internal implementation (including coaching and

mentoring for those new to the use of the tool).

V. When to Use Automated Test Tools, Testing Using Automated Tool.

1. When Does Test Automation Make Sense? i. When there are many repetitive tests

ii. When there are frequent regression testing iterations

iii. When you need to simulate large number of users who are using the

application resources

iv. When AUT is having comparatively stable UI

v. When you have large set of BVT cases

vi. When you can’t rely solely on manual test execution for critical

functionality

Here are the criteria you need to consider before selecting any testing tool: i. Do you have necessary skilled resource to allocate for automation tasks?

ii. What is your budget? iii. Does the tool satisfy your testing needs? Is it suitable for the project

environment and technology you are using? Does it support all tools and objects used in the code? Sometime you may get stuck for small tests due to inabilities of the tool to identify the objects used in the application.

http://www.softwaretestinghelp.com/performance-testing-tools-load-testing-tools/


Anuradha Bhatia 96

VI. What are Metrics and Measurements: Types of Metrics, Project Metrics, Progress and Productivity Metrics?

In software testing there are three main areas which needs to be considered while

thinking about metrics and measurement.

i. Defining the Metrics Small and quality set of metrics should be chosen, large set of metrics

should be avoided as it is very confusing to understand large set of metrics. Metrics should also be uniform and everybody in team should agree with

it. ii. Tracking test metrics

After defining the metrics the next step is to track the metrics. Since tracking is a constant activity so its always nice to automate the

tracking part. Automation reduces time required to track the metrics, analyze them and

measure them. iii. Reporting

Reporting of the metrics is the most important step, you should report test metrics to stakeholders so that they have clear picture of project progress.

A Metric is a quantitative measure of the degree to which a system, system component, or process possesses a given attribute. Metrics can be defined as “STANDARDS OF MEASUREMENT”. Software Metrics are used to measure the quality of the project. Simply, Metric is a unit used for describing an attribute. Metric is a scale for measurement.

“How many issues are found in thousand lines of code?”, here No. of issues is one measurement & No. of lines of code is another measurement. Metric is defined

from these two measurements. Test metrics example:

How many defects are existed within the module?

How many test cases are executed per person? What is the Test coverage %?

1. What is Software Test Measurement?

i. Measurement is the quantitative indication of extent, amount, dimension, capacity, or size of some attribute of a product or process.

ii. Test measurement example: Total number of defects.


Anuradha Bhatia 97

iii. Please refer below diagram for clear understanding of the difference between Measurement & Metrics.

2. Why Test Metrics?

Generation of Software Test Metrics is the most important responsibility of the Software Test Lead/Manager.

Test Metrics are used to, i. Take the decision for next phase of activities such as, estimate the cost &

schedule of future projects. ii. Understand the kind of improvement required to success the project

iii. Take decision on process or technology to be modified etc. Importance of Software Testing Metrics:

i. As explained above, Test Metrics are the most important to measure the quality of the software.

ii. Now, how can we measure the quality of the software by using Metrics? iii. Suppose, if a project does not have any metrics, then how the quality of

the work done by a Test analyst will be measured?

For Example: A Test Analyst has to, i. Design the test cases for 5 requirements

ii. Execute the designed test cases iii. Log the defects & need to fail the related test cases

http://cdn2.softwaretestinghelp.com/wp-content/qa/uploads/2014/09/difference-between-Measurement-Metrics.jpg


Anuradha Bhatia 98

iv. After the defect is resolved, need to re-test the defect & re-execute the corresponding failed test case.

In above scenario, if metrics are not followed, then the work completed by the test analyst will be subjective i.e. the test report will not have the proper information to know the status of his work/project. If Metrics are involved in the project, then the exact status of his/her work with proper numbers/data can be published.

I.e. in the Test report, we can publish: 1. How many test cases have been designed per requirement? 2. How many test cases are yet to design?

3. How many test cases are executed? 4. How many test cases are passed/failed/blocked?

5. How many test cases are not yet executed? 6. How many defects are identified & what is the severity of those defects? 7. How many test cases are failed due to one particular defect? etc. Based on the project needs we can have more metrics than above mentioned list, to know the status of the project in detail.

Based on the above metrics, test lead/manager will get the understanding of the below mentioned key points.

a) %ge of work completed b) %ge of work yet to be completed c) Time to complete the remaining work d) Whether the project is going as per the schedule or lagging? etc. Based on the metrics, if the project is not going to complete as per the schedule, then the manager will raise the alarm to the client and other stake holders by providing the reasons for lagging to avoid the last minute surprises.


Anuradha Bhatia 99

3. Metrics Life Cycle:

4. Types of Manual Test Metrics:

Testing Metrics are mainly divided into 2 categories. 1. Base Metrics Base Metrics are the Metrics which are derived from the data gathered by

the Test Analyst during the test case development and execution. This data will be tracked throughout the Test Life cycle. I.e. collecting the

data like, Total no. of test cases developed for a project (or) no. of test cases need to be executed (or) no. of test cases passed/failed/blocked etc.

2. Calculated Metrics Calculated Metrics are derived from the data gathered in Base Metrics.

These Metrics are generally tracked by the test lead/manager for Test Reporting purpose.

http://cdn.softwaretestinghelp.com/wp-content/qa/uploads/2014/09/Metrics-Life-Cycle.jpg


Anuradha Bhatia 100

5. Examples of Software Testing Metrics:

Let’s take an example to calculate various test metrics used in software test reports:

Below is the table format for the data retrieved from the test analyst who is actually involved in testing:

6. Importance of metrics and measurement in SDLC

i. During all the software development life cycle it is very important to apply

metrics and measurement because metrics and measurement set

expectations. If there are well established metrics and measurements in

project then the test analyst can easily track and report quality results to

the management.

ii. If the metrics & measurements are not established properly then the

assessment of software quality is purely subjective which arises disputes

at the end of development life cycle. You can consider some the following

http://www.softwaretestingmentor.com/articles/importance-of-metrics-and-measurement-in-sdlc/

http://cdn2.softwaretestinghelp.com/wp-content/qa/uploads/2014/09/Example-of-Software-Testing-Metrics.jpg


Anuradha Bhatia 101

areas where you can apply metrics and measurement. This list is not

exhaustive, you can have metrics for lot more things. Schedule of project Coverage Planned & actual cost Workload & resource usage Product risk & project risk Defects

iii. While doing test planning we set the expectations for the stakeholders or

we set the baselines for them. If you have established the baselines the

test reporting is consistent to the management and you can avoid

subjective assessment of testing.

software testing - anuradha bhatia testing is as old as the hills in the history of digital...

Documents