cse 8314 - sw measurement and quality engineering copyright © 1995-2003, dennis j. frailey, all...

version 3.09

Slide 1CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14

SMU CSE 8314 / NTU SE 762-N

Software Measurement and Quality Engineering

Module 14Software Reliability

Models - Part 2

version 3.09


Contents• Requirements Volatility• RADC Model• Summary

version 3.09


Requirements Volatility[NOT an IEEE Metric, but Similar]

Goal: Determine the stability of the requirements, so you can decide: – How far you really are in your

development,– How reliable your software is likely to

be, and– What type of process to use for

software development

version 3.09


Requirements VolatilityGeneral Rules of Thumb

• If requirements are stable, use “waterfall” or similar processes

• If requirements are unstable, use incremental or evolutionary development

version 3.09


Requirements VolatilityPrimitive Data Collected

R = # of original requirements– In original specification, for example

C = # of changes to original requirements

A = # of additions to original requirements

D = # of deletions from original requirements

version 3.09


Requirements VolatilityEquation

V = (C + A + D) / R

•Very large V means unstable requirements

•You measure periodically to see if things are stabilizing

version 3.09


Typical Graph of Volatility

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10

Volatility

version 3.09


Requirements VolatilityUsage Notes - 1

• In a mature development effort for a production product, V should flatten out in the design phase, indicating stabilization of requirements

• If it continues to rise, it means you have an unstable development and should not be proceeding to later phases yet, unless this is a prototype effort

version 3.09


Requirements VolatilityUsage Notes - 2

• If V is large, the implication is that the current software development effort is really a requirements definition effort, which suggests a prototype, incremental or evolutionary development approach– If intended to be final development, do

not go on to next step of process yet

version 3.09


Requirements VolatilityVariation

T = Number of “TBD” (“to be determined”) requirements in original specification

• This gives you more insight on changes that MUST happen (TBDs)

• It also gives more insight on stability over time

V = (C + A + D + T) / R

version 3.09


Typical Graphs of V

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10

With TBDs Without TBDs

version 3.09


You Can Also Learn a Lot by Graphing the Individual Factors of the Equation

R = # of original requirements– In original specification, for example

C = # of changes to original requirementsA = # of additions to original requirementsD = # of deletions from original

requirementsT = # of TBDs

version 3.09


Requirements Volatility Factors

Sample GraphRequirements Volatility

020406080

100120

1 2 3 4 5 6 7 8 9

Week

C A D T R V

R

VT

C AD

version 3.09


Thresholds and Targets• The nature of the development

determines what thresholds should be established

• In a supposedly stable development, thresholds for stability should be very low - instability indicates development effort may be being wasted -- lots of rework ahead

Continued ...

version 3.09


Thresholds and Targets• In a development that is expected

to be volatile, thresholds might be high and targets would be established to determine when stability has been achieved.

• Historical data is essential for establishing reliable thresholds and targets

version 3.09


RADC Measurements

Rome Air Development CenterUS Air Force

Rome Air Force Base

version 3.09


RADC Measurements• These are based on a large amount

of data collected from U.S.A.F. Projects:– 5 million lines of code– 59 projects– Dating back to 1976

• 24 reliability models were studied• Used as the basis for several

government standards

version 3.09


Like IEEE, These Measurements Break the

Process into PhasesPredict

ReliabilityEstimate Reliability

Start Codin

g

Release Software

Requirements

Design

CodeTest

version 3.09


Background of RADC MeasurementsAssumptions:

– # of faults is fixed at the start of formal test– # of faults correlates to # of failures (failures

are easier to measure and are the things the customer cares about)

Goals:– Get the number of faults as low as possible– Predict number of faults as early as possible– Use Predictions to Manage the Process

version 3.09


Basic Approach to RADC Measurements for Reliability

(one variant)• Each factor that influences reliability is

expressed as a number N 0 < N < 1

N = reliability impact of the individual factorN near 0 means it lowers reliability N near 1 means higher reliability

• The product of all these factors is the net reliability

• Each factor may be defined as the product of other, more detailed factors

version 3.09


RADC Concept

R

F1 F3F2

F21 F22 F23

R = F1 * F2 * F3

F2 = F21 * F22 * F23

Assumptions: Factors are Bayesian, Independent, Homogeneous

version 3.09


Use of RADC Formula - I• At start of project, you compute R and

use it as the “current reliability prediction”

• As you go through the project, you try to improve the factors represented by the Fis, thus improving the value of R

version 3.09


Use of RADC Formula - II• e.g. if F3 represents programmer

capability and it has a value of 0.6, you could improve it to 0.7 or 0.8 by hiring more capable programmers or by training your staff in defect reduction techniques

• Eventually, you base your values on actual results rather than on predictions

version 3.09


Reliability Expectation Improves Throughout the

Lifecycle

0

0.2

0.4

0.6

0.8

1

1.2

Requirements PreliminaryDesign

Detailed Design Code Test Release

Historical Actual

Estimates, based on Actual Code

(RE)

Predictions, based on Factors Known at This

Time

(RP)

Goal (based on Specific

System

version 3.09


Note: The Whole Thing Can Also Be Done in Terms of Other Factors

• Mean time between failures• Probability of failure• Hazard function• Defects, etc.• Regardless of how it is expressed, the

idea is to:– Set goals based on system requirements– Determine the indicators for reliability– Improve early to achieve desired goals

version 3.09


What Factors does RADC Recommend?

• RADC has studied many software development efforts and has developed a recommended set of factors to use

version 3.09


Predicted Reliability

RP = A * D * S

• A, D and S are factors known before you start developing the software

version 3.09


Predicted ReliabilityFactors

A = Application type– Similar to Cocomo estimation model– Worse for embedded, real time, etc.

D = development environment– Tools, turnaround, etc– Personnel capability also included

S = software development methods & process– Factors included for each phase

version 3.09


S = Software Characteristics

S = S1 * S2

version 3.09


S = Software CharacteristicsS1 = Requirements & design methods &

process– Structured Analysis, OO, etc. score higher– Less Formal techniques score lower– Process Management is a Big Factor

S2 = Implementation methods & process– Language– Coding standards– etc.

version 3.09


S1 = Requirements and Design Methods

S1 = SA * ST * SQ

version 3.09


S1 = Requirements and Design Methods

SA = Anomaly management»Corrective action process»Risk tracking and contingency»etc.

ST = Traceability»Ability to trace design to requirements,

etc.SQ = Results of quality reviews

»Number of defects found

version 3.09


SQ = Results of Quality Reviews

Design

Design Inspection

Design Repair

Design Inspection

27 defects

SQ = .6 (too low)

3 defects

SQ = .9 (OK)

to Next Phase

Note: values of SQ shown are illustrations. Actual values depend on size of code, defect definitions.

version 3.09


General Algorithm

Do Somethin

g

SiToo Low?

Go On

No

YesRedo

version 3.09


S2 = Software Implementation

Methods and Process

S2 = SL * SS * SM * SU * SX * SR

version 3.09


S2 = Software Implementation

Methods and ProcessSL = Language type– Higher order languages are better– Ada better than C due to Discipline, etc.

SS = Program sizeSM = ModularitySU = Extent of reuseSX = McCabe complexity (of Design)SR = Review results (Defects Detected)

version 3.09


Examples of Improving S2

• Reduce design complexity• Reduce number of defects allowed

before exiting a review or inspection• Use a better language• Reuse proven code• Make software more modular

version 3.09


Estimation Measurements(Based on Actual Code)

F = Failure rate during testingT = Test environment (for Software)E = Operational environmentDuring software test:

RE = F * T

During system test:RE = F * E

version 3.09


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Rqmts & Design Coding andSoftware Test

Begins

System TestBegins

Release

Diagram of Estimation Measurements

Predictive Measurements

RE = F * T RE = F * E

version 3.09


T = Test Environment

TE = Test Effort -- amount of work spent testing

TM = Test Methods -- sophistication, etc.

TC = Test Coverage -- percent of paths tested, etc.

T = TE * TM * TC

version 3.09


E = Operating EnvironmentEW = Workload

EV = Input Variability

RE = F * E = F * EW * EV

version 3.09


Summary of RADC Measurement Usage - At Start

of Project1) Establish a reliability goal based on the

objectives of the product2) Using organization’s history, characterize

the process in terms of the correlation between expected defect or reliability level and the various RADC parameters

3) Use this information to predict the reliability or defect level and use that information to affect the planning process– e.g. use to decide what language, CASE tools,

etc.

version 3.09


Summary of RADC Measurement Usage - During

Project Execution4) Track actuals and compare with

historical data and plans5) Adjust behavior if actuals are

inadequate to meet goals6) Record actuals for future use

version 3.09


Summary• Requirements volatility is easy to

measure early in the project and it can give you a useful prediction of reliability and stability

• RADC and IEEE reliability models use different techniques for different phases of development

• RADC uses facts about the development process to predict reliability

version 3.09


References• Bowen, C., et al., Methodology for Software

Reliability Prediction, RADC-TR-87-171, Vol I & II, Rome Air Development Center, 1987.

• Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS00030. ISBN 0-07-039400-8.

• Musa, John, Software Reliability Engineering: More Reliable Software, Faster Development and Testing, McGraw Hill. ISBN: 0-07-913271-5.

• Xie, M. Software Reliability Modeling, World Scientific, London, 1991. ISBN 981-02-0640-2.

version 3.09


END OFMODULE 14

cse 8314 - sw measurement and quality engineering copyright © 1995-2003, dennis j. frailey, all...

Documents