creating test reports test report documents the result of software testing

©NIIT

Test Reporting

Lesson 1A / Slide 1 of 25Managerial Aspects of Testing

Creating Test Reports• Test report documents the result of software testing.• Test reporting involves reporting the results of test execution of a software

application to relevant stakeholders.• Test report data is used to fix defects and track the status of the software

application under test.

©NIIT

Test Reporting


Test Reporting Tools• Help a software tester to effectively generate a test report• These tools enable the software tester to do the following:

• Generate test report formats• Provide statistical output of test results• Track the defects found in a software application during testing• Record the result of a test

• The following are some test reporting tools:• Analysis Tools

• Represent the result of a test in statistical form• Help in analysis of test data• Examples:JMP 5.1 and SysStat 11

• Database Tools• Store the result of a software test

©NIIT

Test Reporting


Test Reporting Tools(Contd.)• Defect Tracking Tools

• Help to track the resolution of defects • Example:Bugzilla

• Test Reporting Formats• Software packages like word processors are used to generate formats for

test reports• Sample Test Report

Microsoft Word Document

©NIIT

Test Reporting


Test Reporting Standards• Each test report should provide the following information:

• Details of the defect detected• Type of test performed• Scope of the test• Environment and condition under which defect is detected• Inconsistency between actual and expected result

• Sample test report


©NIIT

Test Reporting


Test Reporting Standards(Contd.)• Problem Statement:Consider a software application, AutoCal that performs the

function of a calculator. A test is done on the application to check the add and subtract functions of the application. While testing the following errors are found:• Subtraction of two digit numbers gives the product of the numbers.• Subtracting a greater number from a lesser number does not show a

negative result.• Adding two three digit numbers gives the square of the first number.

Develop a test report for the software test of AutoCal application.

©NIIT

Test Reporting


Test Reporting Standards(Contd.)

• Solution

Microsoft Word

Document

©NIIT

Test Reporting


Statistical Analysis Tools• Pareto Charts

• Ranks the causes of a problem from the most significant to the least significant

• Analyzing data using a pareto chart enables a software tester to:• Focus on critical issues of a software application• Prioritize the problems

©NIIT

Test Reporting


Statistical Analysis Tools(Contd.)• Histogram

• Displays the frequency of occurrence of data• Provides an easy way to analyze and interpret data

©NIIT

Test Reporting


Statistical Analysis Tools(Contd.)• Scatter Plot

• Shows the relationship between two variables

©NIIT

Test Reporting



©NIIT

Test Reporting


Statistical Analysis Tools(Contd.)• Cause and Effect Diagram

• Careful analysis of a cause and effect diagram enables a software tester to:

• Identify the various causes of a problem• Determine the root cause of a problem

©NIIT

Test Reporting


Statistical Analysis Tools(Contd.)• Cause and Effect Diagram

©NIIT

Test Reporting


Statistical Analysis Tools(Contd.)• Problem Statement:Virgtech Inc., a software development team is

developing a voice recognition system. The project has already crossed its first deadline and does not seem to meet the forthcoming deadline. Use a cause and effect diagram to determine the causes for the delay in completion of the project.

©NIIT

Test Reporting


Statistical Analysis Tools(Contd.)• Solution

©NIIT

Test Reporting


Analyzing Statistical Data• Exploratory Analysis

• Uses graphical and descriptive statistical analysis to:• Maximize insight into a dataset• Learn about the structure of a dataset• Extract information related to variables in a dataset• Test underlying assumptions• Highlight unusual conditions

©NIIT

Test Reporting


Analyzing Statistical Data(Contd.)• Exploratory Analysis

• Example: Consider the data set in the following table:

©NIIT

Test Reporting



• After quantitative computation of the preceding dataset the following result is obtained:

• Mean of X is 9• Mean of Y is 7.5• Intercept of the line XY is 3• Slope of the line XY is 0.5• Correlation of X and Y is 0.816

©NIIT

Test Reporting



• Graphical representation of the dataset helps to understand its underlying structure.

• Analysis of dataset in the preceding scatter plot infers that the tested variables are not closely related

©NIIT

Test Reporting


Analyzing Statistical Data(Contd.)• Cluster Analysis

• Classifies a set of observations into two or more mutually exclusive groups • Example:The budget and schedule of projects of a company are plotted on

a scatter plot. • Analysis of the following scatter plot infers that projects behind schedule

and over budget share similar characteristics

©NIIT

Test Reporting


Analyzing Statistical Data(Contd.)• Regression Analysis

• Used to estimate the nature of relationship between two variables • Example:The scatter plot below represents the relationship between the

complexity of a software application and the cost to test the same.

©NIIT

Test Reporting


Analyzing Statistical Data(Contd.)• Regression Analysis

• The following information can be gathered after analyzing the preceding scatter plot:

• As the complexity of the software application increases the cost to test the same also increases.

• The cost to test any software application depends upon the complexity of the software application.

©NIIT

Test Reporting

Lesson 1B / Slide 23 of 26Managerial Aspects of Software Testing

Risk Management Process• Risks are events which may or may not occur and whose occurrence may have

harmful effects on the project• A risk has the following two features associated with it:

• Probability• Loss

• Risk Management is the process of identifying, controlling and eliminating or minimizing risks that may affect a project

• Risk Management comprises of the following:• Risk Identification • Risk Analysis• Risk Mitigation

©NIIT

Test Reporting


Risk Management Process(Contd.)

• The following are the two major decisions taken during risk management:• Deciding on actions that can prevent risks from occurring• Deciding on actions to tackle the risks that materialize

©NIIT

Test Reporting


Identifying Risks in Software Testing• Risk Identification is the process of identifying risks that might affect the

project and its test process and documenting their features.

©NIIT

Test Reporting


Classifying Risks • Software Risks

• Risks associated with a software application being developed • The following are the factors that lead to software risks:

• Skill of developers• Disgruntled employees• Poorly defined project objectives

• Software risks are categorized into the following:• Project Risks:threaten the project plan as they affect the project

schedule and the cost of the project• Technical Risks:threaten the quality of the software being developed

©NIIT

Test Reporting


Classifying Risks(Contd.)• Business Risks

• Threaten the feasibility of the software being developed• Some contenders of business risks are:

• Market risk: Developing an excellent software product that has a few takers

• Technology change: Underlying technology of the software product being superseded by a new technology

• Product competition: Launch of a competitive product in the market before the completion of the project

• Management risk: Loss of management support due to difference in opinion

• Budget risk: Project cost estimated to exceed the planned budget

©NIIT

Test Reporting


Classifying Risks(Contd.)• Premature Release Risks

• Occur when software application is released in the market before it is totally complete

• The following can be the reasons for premature release risks:• Deadline for the completion of the project has exceeded• Equally competitive product is about to be released by another

company • Testing Risks

• Affect the process of testing a software application• The following factors can lead to testing risks:

• Insufficient budget • Incompatible test environment• Inappropriate tools and procedures• Inexperienced testers

©NIIT

Test Reporting


Tools and Techniques for Identifying Risks • Checklists

• Used to identify potential risks to a project and test process • Include a list of most probable risks that can be encountered by any

project and a test process• The following are the generic categories that can be included in a risk

identification checklist:• Product size• Business impact• Development environment• Technology used• Staff characteristics• Customer characteristics

©NIIT

Test Reporting


Tools and Techniques for Identifying Risks(Contd.)• Interviews

• People directly or indirectly associated with the project are contacted on a one to one basis to collect information

• Software testers who have tested a project or worked on a test environment similar to the current project can also be interviewed

• Questionnaires • Contain a list of questions related to a particular risk • Help in the identification of a majority of risks associated with a test

process and the project • Sample questionnaire


©NIIT

Test Reporting


Tools and Techniques for Identifying Risks(Contd.)• Brainstorming

• Used to generate a large number of ideas in a short span of time• It is used in the following cases:

• Participation of the entire group is desired• Broad range of options or ideas are desired• Intuitive, creative, or unconventional ideas are desired

• Observations • Effective method used to collect information related to risks associated

with a project and its test process • Reviews

• Frequent and periodic reviews of projects are done to identify the potential risks to similar projects and their test processes

©NIIT

Test Reporting


Analyzing Risks in Software Testing • Risks analysis is used to assess the following:

• Probability of occurrence of the identified risks• Impact of the identified risks on the project and test process

• Risk analysis comprises of three main tasks: • Risk Probability• Risk Impact• Risk Factor

©NIIT

Test Reporting


Risk Probability• Determines the probability of occurrence of risks identified for the project and

its test process • A risk may have a high probability in one project and a low probability in

another project depending on the nature the project. • During the risk analysis process, the result of each task is logged in a risk

analysis table.

Risk Description

Probability of Occurrence

Risk Impact Risk Factor

Risk Analysis Table

©NIIT

Test Reporting


Risk Impact • Degree of impact that a risk has on a project and test process • Determined on the basis of the probability of occurrence of the risk in the

project and during the testing of the project • The impact of the risk on the following aspects of the project need to quantified

and graded:• Cost• Schedule • Quantity

©NIIT

Test Reporting


Risk Factor• Used to prioritize the risk to a project relative to the other risks identified for

the project• Determined on the basis of the probability of occurrence and impact of the risk

on the project

Risk Factor = Probability of risk * Impact of risk

©NIIT

Test Reporting


Risk Mitigation• Risk mitigation is the planning process wherein alternative ways are considered

to do the following:• Avoid the identified risks from occurring • Reduce the impact of the risk on the project

• There are three possibilities for which the risk mitigation strategies are developed:• Risk Avoidance• Risk Monitoring• Contingency Planning

©NIIT

Test Reporting


Risk Avoidance

• The following are the tasks that a project team performs to avoid the occurrence of risks:• Preparation of the risk plan before the commencement of the project• Identification of potential risks to a project and its process of testing• Prioritization of the identified risks to a project and its process of testing.• Preparation of a plan for managing risks. This plan is called the risk

management plan • The risk management plan is a document that describes how the risk

management processes will be carried out during the project

©NIIT

Test Reporting


Risk Avoidance(Contd.)Risk Probability Risk

ImpactRisk

FactorResponsibi

lityMitigation

StepsStart Date

End Date

Risk Management Plan

©NIIT

Test Reporting


Risk Monitoring

• Risk monitoring has three primary objectives:• To assess whether the predicted risks really occur• To ensure that the risk avoidance steps defined for a particular risk are

being implemented• To collect information that can be used for risk analysis in future

• Risk monitoring activities commence as soon as the project development begins

• Risk monitoring is done periodically

©NIIT

Test Reporting


Contingency Planning • Contingency planning refers to the development of a plan that identifies

alternative strategies used to ensure project success if specified risk occurs • The following are the steps to develop a contingency plan:

• Identify and prioritise risks for which contingency planning is necessary• Group similar risks together• Consider possible actions and select the most appropriate action.• Divide actions into the following categories:

• Taken prior to the event• Taken in the event of a failure• Necessary if the event does not occur

• List possible causes of the risk• Create documentation to be followed in the event of occurrence of a risk• Consider training the concerned people

creating test reports test report documents the result of software testing

Documents