the dynamic interaction of passed and failed requirements during software testing

The dynamic interaction of passed and failed requirements during software testing

Copyrights (c) 2011-2013 Pragmatic Cohesion Consulting

1

Motivation • In my professional experience, I have

repeatedly witnessed systemic issues during Software Verification that prompted me to develop the model presented in this paper.

• It is my hope that the non-intuitive results that the model yields help software development team better understand certain dynamic behavior embedded in the software development process.


2

Requirements Statuses • We are interested in 2 simple sets of

requirements. All requirements within each set share one of the following statuses:

– The first one represents requirements that have been implemented i.e. requirements for which the development team has created actual software code. Let’s call this set of requirements ISR.

– The second one represents requirements that have failed testing after they were implemented by the development team. Let’s call this second set of requirements FSR.


3

Requirement States Probabilities

• Let’s call SFR the probability that a requirement that has been implemented fails testing. When this failure happens the requirement becomes part of the FSR set otherwise it remains in the ISR set.

• Let’s call SFXR the probability that a failed requirement gets fixed. When the fix takes place, the requirement is no longer part of the FSR set but it moves into the ISR set.


4

Requirement States Probabilities • The following probability constraints exist:

0 <= SFR <= 1

0 <= SFXR <=1

• We can derive a probability transition matrix that captures the possible transitions of a requirement from one set to the other:

From ISR to ISR, probability = 1-SFR

From ISR to FSR, probability = SFR

From FSR to ISR, probability = SFXR

From FSR to FSR, probability = 1-SFXR


5

Transition probability matrix


6

To ISR To FSR

From ISR 1-SFR SFR

From FSR SFXR 1-SFXR

Let M be the transition probability matrix,

M=

Two-Step Transition probabilities

• The following question is particularly interesting:

– given the transition probability matrix M and the fact that a requirement has been added to the ISR set, what is the probability that this same requirement will be in the ISR set two transitions in the future?


7


• The problem is not as obvious as it seems; here is is why:

– When a Requirement enters set ISR it has a SFR probability of being moved to set FSR and a 1-SFR probability of staying in ISR. Once in FSR, it has a SFXR probability of moving into ISR and a 1-SFXR probability of staying in FSR.

– We can represent the possible paths with the following tree


8



9

ISR

FSR

ISR

ISR

FSR

FSR

ISR SFR

1-SFR

SFXR

1-SFXR

1-SFR

SFR

P(ISR,ISR,ISR) = (1-SFR).(1-SFR)

P(ISR,ISR,FSR) = (1-SFR).SFR

P(ISR,FSR,ISR) = SFR.SFXR

P(ISR,FSR,FSR) = SFR.(1-SFXR)


• So the probability that a Requirement that started in ISR is in ISR after two transitions is:

P(ISR,ISR,ISR) + P(ISR,FSR,ISR) = (1-SFR).(1-SFR) + SFR.SFRX

• All other similar two-step transition probabilities can be calculated by M2


10



11

To ISR To FSR

From ISR (1-SFR).(1-SRF)+SFR.SFXR (1-SFR).SFR+SRF.(1-SFXR)

From FSR SFXR.(1-SFR)+(1-SFXR).SFXR SFXR.SFR+(1-SFXR).(1-SFXR)

M2 =

Steady State Analysis

• We are interested in knowing in the long run, the probability P1 that any given requirement will be in the ISR set. The probability that such requirement will be in the FSR set is P2=1-P1.

• Markov chains can help us solve this problem by calculating the Steady State probabilities of Matrix M:

P1 = SFXR/(SFR+SFXR) [e1]

P2 =1- SFXR/(SFR+SFXR) [e2]


12

http://en.wikipedia.org/wiki/Markov_chain

Steady State Analysis- Example 1 • if:

P1 = SFXR/(SFR+SFXR) [e1]

• Then P1 = 0.5 = 50% when SFR = SFXR.

• This implies that even if there is a low probability that a requirement that has been implemented fails testing (SFR); if an equally low fixing rate takes place, given sufficient time, a requirement gets closer and closer to having a 50% chance of being in the set that failed testing!


13

Steady State Analysis – Example 2

• Let’s look at another illustration:

– P1 is the probability of having in the long run, a non-failed requirement. Let’s say that we want P1 to be 90%. If we know that an implemented requirement has an SFR=20% probability of failing testing then what should SFXR (the rate at which failed requirements are fixed) be to assure P1=90%?


14


• From [e1] we can say that:

[e1] -> 0.9 = SFXR/(0.2+SFXR)

[e1] -> 0.9.(0.2+SFXR) = SFXR

[e1] -> 0.18 + 0.9.SFXR = SFXR

[e1] -> 0.18/(1-0.9) = SFXR

[e1] -> SFXR = 1.8 > 1 which is impossible!


15


• If SFXR was equal to its maximum value of 100% then the maximum value of P1 can only be 1/(1+0.2) = 83.3%=P1

• This result shows that there is a clear upper bound to the probability that in the long run a requirement will not fail testing given the value of SFR. SFR is the probability that a given requirement that has just been implemented fails testing.


16


• The only way to achieve P1 = 90% when SFXR = 100% is to decrease SFR from 20% so that:

0.9 = 1/(SFR+1)

SFR + 1 = 1/0.9

SFR = 1/0.9 – 1

SFR = 11.11%

• This lower value of SFR implies a better quality of software code that properly implements more requirements.


17

Transient Analysis • It is possible to graph the values of P1 and P2

over several transitions, this is called transient analysis.

• Transient analysis can show how fast the steady states values for P1 and P2 are reached.

• Transient analysis can also show instability in P1 and P2 values over several transitions due to higher values of SFR and SFRX (see examples 4 and 5 in subsequent slides).


18

Transient Analysis – Example 1 • SFR = 10% and SFXR = 50%


19



20



21



22



23


24

For Comments and Questions contact [email protected]

the dynamic interaction of passed and failed requirements during software testing

Technology

transient analysis example

thevalue of sfr

pragmatic2cohesion consulting

pragmatic4cohesion consulting

pragmatic18cohesion

pragmatic23cohesion

lower value of sfr

software testingcopyrights