sonata software limited...technical white paper risk-based testing 2 sonata software limited...

Ananthakrishnan J

Architect, Sonata Software

Sonata Software Limitedwww.sonata-software.com

Risk-Based Testing: Implementation of Risk-Based

Approach for Quality & Cost Optimization

Technical White Paper

Author

Kalyanam Kannan

Technical White Paper www.sonata-software.com

Risk-Based Testing 1 Sonata Software Limited

STATEMENT OF CONFIDENTIALITY

Information included in this document, in its entirety, is considered both confidential and proprietary to

Sonata Software and may not be copied or disclosed to any other party without its prior written

consent.

All logos used in this document are registered trademarks of the respective organizations.



Abstract

As a practiced trend in IT projects, Testing is performed only towards the end of a project. Teams

dedicate hours to test possible risks and flaws after the project is ready to run. As software testing at

this level invites several last minute modifications that can cause discomfort, or sometimes even refute

the very concept of the project, it has become the need of the hour to come up with a way to ensure

detection and reduction of risks, at an early stage of the project. Risk-Based Testing, or RBT as referred

to in this paper, is a procedure in software testing which is used to prioritize the development and

execution of tests based upon the impact and likelihood of failure of the functionality or aspect being

tested based on existing patterns of risk.

Taking a cue from the age-old saying of ‘Precaution is better than cure’, RBT aims to find areas where

possibility of risk or defect is most likely to occur. Through this testing technique, a software test

engineer can now select tests based on risk even before the initiation of the project. Example, through

software testing, one can detect 200 errors by testing 5000 defects. RBT on the other hand, enables the

software tester to pick only 500 probable defects areas and conclude with 190 defects, thereby saving

the effort and time of the software tester.

This paper outlines the Risk-Based Testing approach and describes how Risk-Based Testing can positively

impact the development life-cycle based on business-oriented factors, offering organizations an

actionable plan for starting a Risk-Based Testing approach for projects.

About the Author

Kalyanam Kannan has been in the software industry and testing for the past 14 years and has managed

testing projects using different engagement models, Currently, as a Practice Director in testing, he is

responsible for controlling the quality of releases and delivery with optimum cost. He is also involved in

providing testing solutions using the latest technology, tools and operating models, which enable

projects to minimize cost to quality. His current areas of interest include Risk Based Testing, Test Driven

Development and Open Source Based Testing.

If you would like to interact with the author of this White Paper, feel free to contact us.

mailto:[email protected]



Contents

Abstract .................................................................................................................................................... 2

About the Author...................................................................................................................................... 2

Risk-Based Testing .................................................................................................................................... 4

Generic Approach for Risk-Based Testing ................................................................................................. 5

Statistical Models ..................................................................................................................................... 7

Illustration ................................................................................................................................................ 9

Workflow for Risk-Based approach: ......................................................................................................... 10

Results ...................................................................................................................................................... 11

Inferences of the concept ......................................................................................................................... 13

Open Source Test Management ................................................................................................................ 13

Summary .................................................................................................................................................. 14

To read more about our views on technology, do visit www.sonatablogs.com


White_Paper_Rbt 4 Sonata Software Limited

Risk – Based Testing

In today’s scenario, the quality of software is becoming a matter of concern. With this issue creating

conflicting challenges, the industry is testing and trying different measures to tackle it. Innovative

techniques, tools, technologies and ideas are being implemented to ensure availability of standard

software. One of the popular measures adapted is Risk-Based Testing – a technique through which a

certain amount of testing can be done without covering an entire gamut of available test cases.

According to Industry Experts, 80% of applications are either not tested or are manually tested before

being delivered to production. This leaves the quality of such software open for speculation and hence,

several software projects cost high due to the risks related to it.

Although a lot of mandatory regression and end-to-end testing is being done, the earlier the defect is

detected; the lower is the cost of solving the issues. To address these issues, Sonata has developed a

statistical model which would provide us a required methodology for Risk-Based Testing.

Sonata with its unparallel experience of product quality assurance services has understood that Risk-

based testing is vital in today's competitive market. With this Risk-based approach, a reduction in cost

per quality with a faster time to market is achieved.

Diagram 1



Risk-Based Testing

Risk-Based Testing is a methodology which after identification of risks and their possible impact on

system allows you to prioritize and plan your test strategy in accordance to the risk rating and mitigation

plans.

These provide us with faster time-to-market that gives us more time to fix the defects. The defects are

not detected at the end of the release; in fact the defects can be detected in the early stages of

Application Development itself. This is a scientific and data-based approach which results in cost

optimization and enhancement of quality. It can identify and execute high risk data hence providing

more time for defect fixes.

Generic Approach for Risk-Based Testing

Going ahead with our Risk-Based approach, a Risk Analysis is performed before starting the testing

activities. The prime objective is to take control over the problems before problems take over the

situation.

The following figure shows the activities involved in Risk Analysis when a project is performed. The

diagram below discusses this in detail:



Diagram 2

Diagram2: Risk analysis activity model - This model is taken from Karolak’s book “Software Engineering Risk Management”,

1996 [6] with some additions made (the oval boxes) to show how this activity model fits in with the test process.

The first step is the Test Planning. In this phase, the risks need to be identified and a Risk Strategy should

be created. A risk can be of many types. One of the key important types could be the complexity in the

available applications, the type of resources and available tools. A clear Risk Strategy needs to be

defined before getting into the Test Planning activities.

Subsequently, the Risk Mitigation plan must be prepared. This plan clearly states for a particular type of

risk what the risk mitigation is. For example, if it is going to be a very complex application the risk

mitigation plan would be “dissecting the application into several components as smaller modules, and

fill up each of those components with more capable resources”.

Once the Risk Mitigation plan is completed, other important areas like the Risk Reporting can be

focused on. Risk Reporting is very important because it provides complete transparency across the

entire stakeholders of the project to gauge and act on the risk area. With all of the testing and

inspection techniques and capturing all test metrics, the risks that get reported are identified. At this

stage, one can predict the risks.



In the Risk Prediction stage there is an entire set of data from which the risk-prone area is identified. In

this model risk prediction feedback is again fed into the risk identification area and it is a cyclic process.

By undergoing several iterations of this cyclic process the Risk Strategy as well as Risk Prediction model

can be refined. The areas with certain defects or minimum defects or no defects can easily be predicted.

This is the core idea of Risk Based Technology.

Statistical Models

In the statistical model the importance lies on the characterization of numerical data. Also, it is very

important to estimate the probability in terms of the behavior of system.

The entire testing activity is nothing but probability. It is the probability of finding out a particular defect

or a particular section failing on a particular area or on a particular type of environment. These all may

be useful in deciding the type of testing required and to ascertain the focus in areas of testing. With this

focus, extrapolation or interpolation of the existing data can be conducted and the best fit for that can

be identified. The best fit will provide the critical path for the defects. It will clearly provide areas that

require testing in that particular application or system.

There is another model called Spectral Analysis of data or model generated output which is an industry

standard. Here the focus is on the algorithms that have been used for this Risk-Based Testing. The

algorithm is extended to suit the current situations where the best fit for various factors is exercised.

This helps in calculating the risk as well as the probability of failure.

The statistical model is based on the probability of defects and a consequence of defect. These two are

very critical in defining the Risk Exposure of the system. One of the important parameters is quality of

the code. It may be suffering from poor designing or it may have been coded by an inexperienced

programmer, it may be to a complex functionality. The probability of defects is defined as P(f),

consequence of defect pertaining to the customer C(c) and consequence of defect to the vendor as C(v).



Diagram 3

Consequences of defect for a customer (which is a cost to the customer) may be:

o The probability of a legal threat

o Losing a market place

o Not fulfilling regulations or FDI regulations

The consequence of defect towards the vendor gives a negative credibility to the vendor or it would

increase the maintenance cost because of the functions with faults.

The combination of these two factors leads to a formula: Risk exposure [Re(f)] is calculated as a product

of the probability of failure and the consequence effect.

Diagram 4

The probability of failure again characterized. It is a combination of multiple probabilities of failures.

Normally, the consequences value weighted between 1-3, and it would count the production fault loss

of revenue impact and incurs cost change impact also. The probability of failure is always weighed

between 0-1.



The weighted average of the probability of failure is dependent upon following factors:

• Changed functionality

• New functionality

• Design quality

• Size of the project

• Complexity

• Programmers’ experience

Illustration

The following calculation of Risk-Based matrix is with a live example from one of the Sonata's projects.

For the sample calculation 18 factors of probability were taken into account. The customer is Europe’s

largest holiday company, serves more than 23 million guests every year. Operates own resorts, hotels,

airlines, travel agencies and cruise ships.

The front-end was a Web Selling platform and the back-end is a mainframe system which supports the

entire set of operations. The backend system is capable of running 600-1000 batch programs on a daily

basis and transacts 5-6 million records every day. The entire system has around 10 interfacing systems

(e.g. Amadeus, Alamo etc). They have multiple staging areas where they present their data and do a

focused analysis for the next quarter or the next season coming through.

The Risk-Based Testing was conducted for this particular engagement because there were nearly 8000-

10,000 test cases for the entire set of Enterprise Applications. While doing so, around 2 to 3 releases are

gathered on a monthly basis at the Enterprise Level. If a complete set of testing or end-to-end testing is

required, it is important to cover all sets of test cases across the enterprise chain. In such scenarios, over

2000 to 3000 test cases are run per release. This consumes a lot of effort and hence the cost, as well as a

delay to market.

In order to handle this situation several testing techniques are adopted, test automation in terms of tool

level as well as in terms of data level which would also test the testing data integrity. In terms of

approach a Risk Based Approach is followed because only a certain amount of test cases or certain

types of test cases catch errors, rest of the test cases were defect-free. Based on our observation it was

found that out of 2000-3000 test cases only around 200-300 test cases were capturing defects. This is



because only these test cases are clearly attached to the risk prone areas. Hence, the statistical model

was adopted to capture most of the defects and implement Risk-Based approach.

Various kinds of testing methods have been implemented in this regard; starting with System Testing

which involves testing of the Web selling platform, testing of their core system, testing of their business

intelligence areas, Integration Testing, Regression Testing, Performance Testing to Security Testing etc. A

specialized testing on Data Integrity on volume testing was also conducted. In compiling all these

methods of testing several areas with defects were identified.

The inputs required for fitting the statistical model are:

o The number of defects

o Types of defect: Database defect, Staging Area defect, Web area defect

o Classification of defects: Defects originated of database, originated from the application server

or from the functionality

o Effort required for defect identification

o Effort required for fixing defects

o Weightage for the probability functions in terms of failure and the consequence

All inputs in the algorithms and routines for iterative runs were rigorously followed. This has resulted in

the probability of failure as well as the risk exposure co-efficient.

Workflow for Risk-Based approach:

As the first step, all defects are classified. Once the defects have been classified the probabilities of

various factors which affect the quality of the release are obtained, post which the risk exposure co-

efficient is derived. Once the Risk Exposure co-efficient is identified, the co-efficients are fed into the

iterative algorithms for various values of probability of failures. From this the type and number of test

cases that will be utilized for Risk Based Testing are obtained.

Example, in a live environment, if there are 2000-3000 test cases and it is known from an existing

analysis that the 23rd or 33rd test case are going to yield defects in a particular area which comes under

the sampling techniques, and there are other defects from a different area, then it is convenient to

sample them on a common algorithm and feed it into algorithm. This enables the identification of the



Risk exposure coefficient. Wherever the Risk exposure co-efficient is high, those are the areas that need

100% coverage.

In this specific example the probable failures in terms of change requests, test interfaces, inexperienced

developer, field validations, business rules validations, positive and negative scenarios, third party

interfaces, system integration testing, backend verifications, UI elements testing, content verifications,

content validations, error messages verifications & validations, cross browser testing, platform

compatibility testing, functional end-to-end flow testing have been taken into consideration. Certain

weights have been assigned to these areas, to calculate the Risk exposure for various iterations from a

value of 0.5 to 2.0 and to achieve a constant consequence as 2 to arrive at the probability of failure.

Results:

There are 3 different depictions:

Iterations:

Re(f) {0.5, 1.1, 1.2…..2.0}

having C(c&v) = 2

Note: Open Source TM (Algorithms and IP) was used for this study

Defects in Releases

Sample: 1200 TC - Continuous

Graph 1Defects in Releases

Sample: 1200 TC - Random

Graph 2



Graph 1: It is a sample of 1200 test cases and it is continuous. A particular release has undergone 10-12

iterations and the test cases have been run 1200 continuously. The test cases have been run by an

automated test suite which has been developed on an Open Source Framework. The defects in Graph 1

found in the different iterations and how the applications are stabilized over a period of iteration. In this

case all the 1200 test cases on all releases in an automated way.

Graph 2: Test cases were selected at random, without any logic or reason. This particular method also

provides the defects but the amount of the defect captured is lesser when compared to the amount of

defect captured while the entire set of test cases is run.

Graph 3: The statistical-based algorithm is run and 400 out of the 1200 test cases are sampled. Only a

minimum number of test cases are run but they capture the maximum number of defects. There is no

variation in results as compared to the number of defects identified when all 1200 test cases are run.

This concept saves a lot of effort and cost that leads to an impressive turnaround time.

Defects in Releases

Sample: 1200 TC - RBT

Graph 3



Inferences of the concept

As a result, a 50 – 60% reduction on testing effort (400 test cases out of 2000 test cases) is achieved,

generating data for multiple set of defects scenario. These defect scenarios or data will be applicable in

subsequent releases. In a similar project or a similar type of release the respective test cases can be

pulled straightaway. There is no impact on the critical factors that is on the database side or in the

performance side or in the quality side by doing this.

This has resulted in the 40-50% of the testing cost reduction. In the enterprise environment when

multiple projects in multiple streams are run, each product needs to be tested on a particular day or a

particular time segment. To do this testing continuously in all these areas by using risk-based approach a

greater bandwidth is required to run these tests in spite of running less amount of test cases and achieve

more amount of coverage.

The other important advantage is Defect Predictability. Number of defects can be predicted for a certain

size of application. This helps us in estimating the time for defect fixes. In a project lifecycle analysis,

requirements, developments, testing and release need to be planned. Often, the element that is missed

out is time and effort required for the defect fixes. If a decent estimate of defect fixes can be identified,

then it is easier to estimate the time required to complete it.

Since the developers or the programmers required for this program are selected right at the beginning,

there can be optimized use of relevant expertise and hence the risks can be handled efficiently.

Open Source Test Management

In this activity of statistical model or the iterative algorithm and selecting the relevant test cases and

then running those test cases for execution, you require a proper test management system. Either it can

be a quality center or a QA director or any other tool which is capable of doing that. Sonata has

developed an Open Source Test Management System which is integrated with defect and test

management areas. It houses the entire Risk-Based Testing algorithm and the data for various values of

probabilities of failure in different areas in terms of classification. It has become easy to do the

automated test cases predictability and organize the test cases according to the functionality and defect



areas. Customized simulation for various resources can be obtained. This reduces around 70% of the

regression test cost and 50% improvement in controlled releases.

Summary

The statistical approach for Risk Based Testing is a proven model. It is capable of simulating error

injection, analyzed impacts associated with failures. More importantly, it is simple and cost effective. As

an added advantage an Open Source System supporting it is also available. The algorithms are scalable

and iterative and these algorithms can be used or extended for any type of testing (Web testing, Data

Testing, testing of ERP systems). Customized reports in terms of the available number of risk prone test

cases are generated which definitely need to run in a particular release. All the data is available in a

report format. The Open Source Test Management System houses all of these activities and functions

together and being provided to the customer as a package.



CORPORATE OFFICE APS Trust Building Bull Temple Road, N. R. Colony Bangalore 560 019, India Tel: 91-80-3097 1999, Fax: 91-80-2661 0972 Email: [email protected] WORLDWIDE OFFICES Dubai Office # 507, Thurraya Tower No.1 P O Box 502818, Dubai Internet City Dubai, United Arab Emirates Tel: 971-4-375-4355, Fax: 971-4-424-0132 Email: [email protected] Germany TUI InfoTec GmbH Karl-Wiechert-Allee 4 30625 Hannover, Germany Tel: 49-511-567 5296 Email: [email protected] India 6, Richmond Road Bangalore - 560 025, India Tel: 91-80-3097 3299, Fax: 91-80-2248 4045 Email: [email protected] 193, R.V. Road, Basavanagudi, Bangalore - 560 004, India Tel: 91-80-3097 2999, Fax: 91-80-2656 7487 Email: [email protected] Sonata Towers, Global Village, Pattenegere & Mylasandra, RVCE Post, Mysore Road, Bangalore - 560 059, India Tel: +91-80-3097 1499 Email: [email protected] 1-10-176, Begumpet Main Road Opp. Hyderabad Public School Hyderabad - 500 016, India Tel: 91-40-3981 3899, Fax: 91-40-2776 4831 Email: [email protected]

Singapore 1, North Bridge Road, #19-04/05 High Street Center Singapore – 179094, Singapore Tel: 65-633-724-72, Fax: 65-633-740-70 Email: [email protected] UK 5, Churchill Court 58, Station Road, North Harrow Middlesex HA2 7SA, UK Tel: 44-20-8863 8833, Fax: 44-20-8863 5533 Email: [email protected] USA 39300 Civic Center Drive, Suite 270, Fremont, CA 94538, USA Tel: 510-791-7220, Fax: 510-791-7270 Email: [email protected] 2018 156th Ave NE, Suite 100, Building F, Bellevue, WA 98007, USA Tel: 425-372-2167, Fax: 425 484 7799 Email: [email protected] 1901 North Roselle Road, Suite 800, Schaumburg, IL 60195, USA Tel: 847-517-6310, Fax: 847-517-6313 Email: [email protected] 11330 Lakefield Drive, Bldg #2, Suite 200 Duluth, GA 30097, USA Tel: 770-814-4213, Fax: 678-623-0236 Email: [email protected] 275 Grove Street, Suite 2-400 Newton, MA 02466, USA Tel: 617-663-4866, Fax: 617-663-6127 Email: [email protected] 212, Carnegie Center, Suite 206 Princeton, NJ 08540, USA Tel: 609-919-6325, Fax: 617-663-6127 Email: [email protected]

If you have any experiences related to Risk Based Testing that you would like to share with us, please

write in to us on [email protected]



mailto:�[email protected]













sonata software limited...technical white paper risk-based testing 2 sonata software limited...

Documents