analytical risk-based and specification-based testing - bui duy tam

Analytical:

Risk-based and Specification-

based Testing

SPEAKER: TAM BUI

About me

• B. A in English in 1996 – University of Hue.

• B. Eng in Computer Science in 2005 – Hanoi

University of Science and Technology.

• ISTQB – Advanced Level – Test Manager in USA in

2013.

• Over 14 year experiences in software testing.

• Strong knowledge and experiences in automation

testing, performance testing and security testing.

• Great passion in software testing: learning and

sharing knowledge and experiences.

2

Objectives

• What is a Test Strategy?

• What is about Analytical Test Strategy?

• Risk-based Testing.

• Specification-based Testing.

• Benefits of Analytical Test Strategy.

3

What are Test Objectives?

• Finding Defects.• Preventing Defects.• Providing Quality Information.• Gaining confidence about the level of quality.• Satisfying the needs of stakeholders: Scope, Time, and

Budget.• Satisfying the Business/System Requirement

Specification.

4

What is a Test Strategy?

A Test Strategy is a document to be developed to inform project managers, testers, developers how to achieve test objectives.

5

Types of Test Strategies

AnalyticalModel-based

MethodicalStandard-compliant

Dynamic ConsultativeRegression-

averse

6

Test objectives in real projects

Project 1You are a Test Manager. You are assigned to work with a new project with some difficult objectives: large scope with short time and resource shortage.

Project 2You are a Test Manager. You are assigned to work with a new project to develop a safety critical system which will be used in the surgery field.

Which Test Strategy is selected?

Analytical

7

What is about Analytical

Strategy?

All test planning activities are based on data and analysis of data

Risk-based TestingSpecification-based Testing

8

Risk-based Testing

• What is Software Risk?

• Risk Management.

9

What is Software Risk?

A risk is something that has not happened yet and it may never happen. It is a potential problem.

Risk is the possibility of a negative or undesirable outcome.

A risk has some likelihood between 0% and 100%.

10

Software Risk Areas

Functional complexity

Performance

Safety

Data Selections

Recoverability

New Technology

Scalability and Reliability

Scope of Use

Environment

Security

Reliability

Usability

Interface Complexity Technical Complexity

11

Risk Management

Risk Identification

Risk Control, Mitigation

Risk Analysis

12

Risk Identification

Expert interviews

Independent assessments

Use of risk templates

Lessons learned• Metrics and data from past projects.• Develop a risk database or repository

13

Risk Identification (Cont.)

Formal analysis methods• Failure modes and effects analysis (FMEA)• Hazard analysis, Cost of failure, Other

Risk workshops

Brainstorming

Checklists

Calling on past experience

14

Risk Analysis – Likelihood

(Technical Factors)

Complexity of technology

Personnel issues

Trainings

Intra-team and inter-team issues

Supplier and Vendor contractual problems

Geographical distribution of the development organization

New Technologies and designs

Bad quality of tools and technologies used

Bad management and technical leadership

Time, resource, and management pressure

Late testing and bad quality assurance

Requirements, designs, code changes

High defect rate

The frequency of use of the affected feature

15

Risk Analysis – Impact (Business

Factors)

Potential damage to the company image

Loss of Customers and Business

Financial and social losses or liability

Civil or criminal legal sanction

Loss of licenses or permits

The lack of reasonable workarounds

The visibility of failure and the associated negative publicity

16

Risk Analysis – Risk Priority

Number• Apply 5-point-scale for Likelihood and Impact for

each risk:

– 1 = Very High

– 2 = High

– 3 = Medium

– 4 = Low

– 5 = Very Low

Risk Priority Number = Likelihood * Impact

17

Risk Analysis – Extent of

Testing1-5: Extensive: run large number of tests, both broad and deep, combine and vary interesting conditions, use all relevant techniques with strong coverage criteria.

6-10: Broad: run medium number of tests, exercise many different interesting conditions use most relevant different interesting conditions, use most relevant techniques with medium coverage criteria.

11-15: Cursory: run small number of tests, sample most interesting conditions, use efficient techniques with weak interesting conditions, use efficient techniques with weak coverage criteria

16-20: Opportunity: leverage other tests or activities to test 1-2 interesting conditions, investing very little time and effort, using reactive techniques especially.

21-25: Report bugs only: allocate only a small amount of extra time to report and manage these accidental bugs.

18

ExampleRISK LIST WITH EXTENT OF TESTING

No Quality Risk Likehood Impact Risk Pri. # Extent of Testing Tracing

1 Functional Risks

1.1 Function 1 2 5 10 Broad 100-100-100

1.2 Function 2 1 1 1 Extensive 100-100-101

1.3 Function 3 3 5 15 Cursory 100-100-102

1.4 Function 4 4 5 20 Opportunity 100-100-103

1.5 Function 5 5 5 25 Report Bugs 100-100-104

2 Performance Risks


2.2 Function 7 2 4 8 Broad 100-100-106

2.3 Function 8 3 3 9 Broad 100-100-107



3 Usabilty Risks

3.1 Function 11 4 4 16 Opportunity 100-100-110

3.2 Function 12 2 3 6 Broad 100-100-111




19

Risk Control

Mitigate: Take steps in advance to reduce the possibility and impact of the risk.

Contingency: Have a plan in place to reduce the possibility of the risk to become an outcome.

Transfer: Convince some other member of the team or project stakeholder to reduce the probability or accept the impact of the risk.

Ignore: Ignore the risk, which is usually a good option only when there is little that can be done or when the possibility and impact of that risk are low in the project.

20

Bottom line of Risk-based

Testing

Plan Testing Activities such as design tests,

implement tests, execute tests, report test results according to Risk

Priority Number

21

Specification-based Testing

• What is a Specification?

• What are implicit Specifications?

• How to learn from the Specification.

• Clarify and fix ambiguities – Review techniques.

• Prioritize Software Requirement specification.

• Bottom line of Specification-based Testing.

22

What is a Specification?

A detailed description of work to be done or materials to be used in a project

- Marriam-Webster

A software requirements specification (SRS) is a description of a software system to be developed. It lays out functional and non-functional requirements and may include a set of use cases that describe user interactions that the software must provide.

- Wikipedia

23

What kinds of Specification can

be tested?

Software Requirement Specification

Software Design Specification

User Interface Description

User Manual

24

What are implicit

Specifications?

Characteristics of the product are not mentioned in the

specification.

25

What are implicit Specifications

(Cont.)

Business Culture

Technical Norms

Design/Code Standards

Guidelines

Regulations

Marketing/Sale Presentations

Memos (Software changes)

26

What are implicit Specifications

(Cont.)

Competing products

Related products

Email discussions within the project

Customer comments

Bug Reports

Test Results

Prototypes

27

How to learn from the

Specification?

SQ3R Reading Strategy

Skim Question Read

Respond Review

28

Clarify and Fix Ambiguities –

Review Techniques

Walkthrough

Technical ReviewPeer Review

Inspection

29

Prioritize Requirement

Specifications

MoSCoW

MUSTMandatory

SHOULDHigh Priority

COULDDesired but not necessary

WOULDCan be delayed and proposed

for future releases

30

Example

SOFTWARE REQUIREMENT LIST

Req. ID Requirement Requirement Description Priority

1 Funtionality Requirement

1.1 Requirement 1 Description of Requirement 1 Must

1.2 Requirement 2 Description of Requirement 2 Could

1.3 Requirement 3 Description of Requirement 3 Would


1.5 Requirement 5 Description of Requirement 5 Should




1.9 Requirement 9 Description of Requirement 9 Would


2 Performance Requirements






31

Bottom line of Specification-

based Testing

Study Specifications which include implicit Specifications

Review Specifications and update them if necessary

Specifications are prioritized

Plan testing activities such as design tests, implement tests, execute tests and report test results based on prioritized Specifications

32

Benefits of Analytical Strategy

Risk-base Testing Specification-based Testing

• Exhausted testing is impossible. Important features/functions will be addressed early. Important problems can be discovered early.

• Defects/bugs can be prevented by running static testing.

• Test Managers properly distribute test efforts, schedule time, use budget to mitigate risks and make contingency plan when risks occur.

• Prioritized features/functions will be addressed early. Quality will be improved early.

• Defects/bugs can be prevented by running static testing.

• Tests can be derived easily.

• Customer’s needs and expectations can be satisfied correctly.

• Training new team members is easily.

33

Summary

• Understand common test strategies and select appropriate test strategy for each project.

• Comprehend risk-based testing and how to plan risk-based testing activities.

• Comprehend specification-based testing and how to plan specification-based testing activities.

34

Reference

• Lessons Learned in Software Testing – Cem

Kaner, James Bach, Bret Pettichord.

• Specification-based Testing – Testing Education

and BBST – Cem Kaner.

35