CS 217 Software Verification and

Validation

Week 6, Summer 2014

Instructor: Dong Sihttp://www.cs.odu.edu/~dsi

REVIEW OF MIDTERM

Statistic of Midterm

Highest Score = 78

65 – 78 = A

50 – 64 = B

3

Testing Principles (1) – p54 of week1

Testing can demonstrate only the presence of defects and not their absence– Testing can show that defects are present, but cannot

prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness.

Exhaustive testing is impossible– Exhaustive testing (all combinations of inputs and

preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts.

Testing Principles (3) – p56 of week1

Testing is context dependent– Testing is done differently in different contexts. For

example, military software is tested differently from an business site.

Motivation – p21 of week2 Logic became popular in the early 20th century

among philosophers and mathematicians

What constitutes a correct proof in Mathematics?

Some ‘correct’ proofs were later disproved by other mathematicians

Concept of logic helps us to figure out what constitutes a correct argument and what constitutes a wrong argument

6

p28 of week2

Propositions - a statement that is either true or false.

For every proposition p, either p is T or p is F For every proposition p, it is not the case that

p is both T and F

7

Validation & Verification – p14 of week1

Validation : Have we built the right software? i.e., do the requirements satisfy the customer? (This is dynamic process for checking and

testing the real product. Software validation always involves with executing the code)

Verification : Have we built the software right? i.e., does it implement the requirements? This is static method for verifying design, code.

Software verification is human based checking of documents and files

Introduction to Software Testing, Edition 2 (Ch 1) © Ammann & Offutt 8

Introduction to Software Testing, Edition 2 (Ch 1) © Ammann & Offutt 9

Software Fault : A static defect in the software

Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior

Software Error : An incorrect internal state that is the manifestation/expression of some fault

Faults in software are equivalent to design mistakes in hardware.

Software does not degrade.

Software Faults, Errors & Failures – p21 of week1

Unit Test (Component Level Test) – p65 of week1

Individual components are tested independently to ensure their quality. The focus is to uncover errors in design and implementation, including

10

The focus is to uncover errors in:- data structure in a component- program logic and program structure in a component- component interface- functions and operations of a component

Operations and Functions with I/O

White-box

interfaceinput

output

Internal logic, data, structure

output

inputinterface

operation

Black-box

Integration Testing – p66 of week1 A group of dependent components are tested

together to ensure their the quality of their integration unit.

11

- Design and construction of software architecture- Integrated functions or operations at sub-system level- Interfaces and interactions between them- Resource integration and/or environment integration

Component #1

Operations and Functions with I/O

input

interface

operation

Component #2

Operations and Functions with I/O

output

interface

operation

System Testing - p67 of week1 The system software is tested as a whole. It

verifies all components mesh properly to make sure that all system functions and performance are achieved in the target environment.

12

The focus areas are:- System functions and performance - System reliability and recoverability (recovery test)- System installation (installation test)- System behavior in the special conditions (stress and load test)- System user operations (acceptance test/alpha test)- Hardware and software integration and collaboration- Integration of external software and the system

When a system is to be marketed as a software product, a testing process called beta testing is often used.

System

(Operations &Functions

& Behavior)

Userinterface

User External interfaces

Rules of Natural Deduction – p45 of week2

FR 3 (De Morgan’s law)~(p v q) = (~p) ^ (~q)~(p ^ q) = (~p) v (~q)

FR 4 (Law of contrapositive)p q = (~q ~p)

FR 5 (Double Negation)~(~p) = p

13

Examples of Arguments – p46 of week2

If a baby is hungry, then the baby cries. If the baby is not mad, then he does not cry. If a baby is mad, then he has a red face. Therefore, if a baby is hungry, then he has a red face.

Model this problem!!

h: a baby is hungryc: a baby criesm: a baby is madr: a baby has a red face

14

h c~m ~cm r

. . . h r

h cc mm r

. . . h r

Bonus Question – p39 of week3 What happened if switch AND with OR logic in

the program:

fun (x, y){ z = 1; IF ((x>z) OR (y>z)) THEN z = 0; Output z; }

15

1. Fun (0, 0) 2. Fun (2, 0) 3. Fun (0, 2) 4. Fun (2, 2) 5. Fun (8, 9)

Function coverage:

Statement coverage:

Branch coverage:

Condition coverage:

Example 1: compare two numbers – p50 of week3

Function ‘compare (x, y)’

Inputs: Two numbers – x and y Outputs: A larger number between x and y

16

z = Compare (x, y)

(x, y) z

– p51 of week3

17

• Equivalence Classes:

{ (x, y) | x < y }

{ (x, y) | x > y }

{ (x, y) | x = y }

{ input other than a pair of numbers,

“as&%dfget^$(&w” }

Valid inputs

Invalid inputs

18

Valid (x, y) Input Space

x = y

x < y

x > y

Three test cases:

(1, 2) --- 2

(8, 8) --- 8

(100, 30) --- 100

Plus one test cases:

(^&%*) --- ERROR

– p52 of week3

P53 of week3 - Example 2: Loan application

19

Customer Name

Account number

Loan amount requested

Term of loan

Monthly repayment

Term:

Repayment:

Interest rate:

Total paid back:

6 digits, 1stnon-zero

$500 to $9000

1 to 30 years

Minimum $10

2-64 chars.

Choosing (or defining) partitions seems easy, but is easy to get wrong…

20

Customer name

Number of characters:

2 64 65invalid valid invalid

1

Conditions ValidPartitions

InvalidPartitions

ValidBoundaries

InvalidBoundaries

Customername

2 to 64 charsvalid chars

< 2 chars> 64 charsinvalid chars

2 chars64 chars

1 chars65 chars0 chars

Valid characters:Anyother

A-Za-z-’

space

21

Loan amount

500 9000 9001

invalid valid invalid

499

Conditions ValidPartitions

InvalidPartitions

ValidBoundaries

InvalidBoundaries

Loanamount

500 - 9000 < 500>90000non-numericnull

5009000

4999001

WHITE-BOX TESTING

Styles of Testing Testing traditionally can be conducted in two

styles

Black-Box testingTry to choose "smart" tests based on the requirements,without looking at the code.

White-Box testingTry to choose "smart" tests based on the structure of thecode, with minimal reference to the requirements.

23

Definition of White-Box Testing

Testing based on analysis of internal logic (design, code, etc.). (But expected results still come from requirements.)

Use different techniques to check every visible path of the source code to minimize errors.

It is the ability to know which line of the code is being executed and being able to identify what the correct output should be.

24

Characters of WB testing Exploits the structure, control or data flow of

programs in order to design test cases.

Typically performed during coding.

Allows to test parts of the program, since you know the structure. (with black‐box is not possible)

Allows to cover systematically every part of the program.

25

WB Testing Techniques

Logic coverage: (learned in previous class) Statement coverage Branch coverage Condition coverage …

Dataflow based testing / Path coverage: all program paths have been traversed at least once

26

Pseudo code & Control/Dataflow Graphs

27

“nodes”

“edges”Input

output

Absolute (x){ IF (x>=0) THEN y = x;

ELSE IF (x<0) THEN y = -x;

Output y;}

x

IF (x>=0) ELSE IF (x<0)

y = x; y = -x;

y

Example

28

Fun (x, y){ z = 1;

IF ((x>z) OR (y>z)) THEN z = 0;

Output z; }

Test cases:1. Fun (0, 0) 2. Fun (2, 0) 3. Fun (0, 2) 4. Fun (2, 2) 5. Fun (8, 9)

Control/Dataflow of the example

29

Fun (x, y){ z = 1;

IF ((x>z) OR (y>z)) THEN z = 0;

Output z; }

Input x, y

z = 1;

IF ((x>z) OR (y>z))

z = 0;

ELSE

z = 1;

Output z;

Test cases:1. Fun (0, 0) 2. Fun (2, 0) 3. Fun (0, 2) 4. Fun (2, 2) 5. Fun (8, 9)

Statement coverage

Has each statement in the program been executed?

30

√ √

√

Branch coverage

Has each branch of each control structure been executed?

For example, given an if statement, have both the T and F branches been executed?

Another way of saying this is, has every edge in the program been executed?

31

Condition coverage

Has each Boolean sub-expression evaluated both to true (T) and false (F) ?

32

Condition coverage

Fun (x, y){ z = 1;

IF ((x>z) OR (y>z)) THEN z = 0;

Output z; }

Test cases:1. Fun (0, 0) 2. Fun (2, 0) 3. Fun (0, 2) 4. Fun (2, 2) 5. Fun (8, 9)

X>1 Y>1

T , F ? T , F ?

Exercise Consider a program ‘Compare’ which compare

the value between two numbers. The inputs are two real numbers x and y, the program outputs the larger number between x and y.

34

Compare (x, y){ IF (x>y) THEN z = x;

ELSE IF (x==y) THEN z = x;

ELSE IF (x<y) THEN z = y;

Output z;}

Exercise - Control/Data Flow Graphs

I have provided you the Pseudo code,

Can you draw a Control/Dataflow Graphs?

Try it! This is a bonus question with +5 points towards to FINAL!!

35

Exercise - Statement Coverage

Statement Coverage requires that each statement will have been executed at least once.

What is the minimum number of test cases required to achieve statement coverage for the program Compare (x, y) ?

36

Exercise - Branch Coverage

Branch Coverage requires that each branch will have been traversed at least once.

What is the minimum number of test cases required to achieve branch coverage for the program Compare (x, y) ?

37

Condition Coverage

Condition Coverage requires that each condition (sub-expression) will have been True at least once and False at least once.

What is the relationship between Branch and Condition Coverage?

Condition / branch coverage

IF ( AND )

THEN …

ELSE …

39

X>0 Y>0

T F

T , F ? T , F ?

Condition Coverage example – Combination !

IF A or B THEN s1;

ELSE s2;

A B Branch

test 1 T F true

test 2 F F false

test 3 T T true

test 4 F T true

Combinations of condition values: TT, TF, FT, FF

Exercise

IF (A or B or C) THEN s1;

ELSE s2;

What is the minimum number of test cases required to achieve condition coverage for the above pseudo code?

Dataflow based testing -> Path coverage

All program paths have been traversed at least once.

4242

Input x, y

z = 1;

IF ((x>z) OR (y>z))

z = 0;

ELSE

z = 1;

Output z;

Two paths !

Path coverage - Example 1

43

if a then s1else if b then s2 else if c then s3 else s4 end_if_then_else end_if_then_elseend_if_then_else

# Paths: ___

# Branches: ___

WB Testing Techniques

Logic coverage: (learned in previous class) Statement coverage Branch coverage Condition coverage …

Dataflow based testing / Path coverage: all program paths have been traversed at least once

44

Coming Up Next week…

Lab 2: White-box Testing

Preparation: Go over the slides of WB testing Understand “Code coverage” & “Data flow /

Path”