cse 8314 - sw metrics and quality engineering copyright 1995-2001, dennis j. frailey, all rights...
DESCRIPTION
CSE SW Metrics and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M12 8/20/2001Slide 3 IntroductionTRANSCRIPT
8/20/2001
Slide 1CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
SMU CSE 8314 / NTU SE 762-N
Software Metrics and Quality Engineering
Module 12Software Reliability
Overview
8/20/2001
Slide 2CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Contents
• Introduction• Measuring Reliability• Summary
8/20/2001
Slide 3CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Introduction
8/20/2001
Slide 4CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
What can be Measured• It does what was specified• Failure rates are low• etc.
End-User’s Perspective
• It does what I want• It never fails• etc.
Not a Perfect Match
Reliability is the “Bottom Line” of Software Quality
• Reliability is the most conspicuous attribute of quality
• But what do we mean by reliability?
8/20/2001
Slide 5CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
• Assumption: failure results from physical changes– breakage, wearout, fatigue
• Or incorrect manufacturing processes
• Theory of reliability is founded on the statistical behavior of random molecules
Hardware Reliability Focuses on Materials and Production
8/20/2001
Slide 6CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
But Product Development can also be a Factor in Quality &
Reliability• Bulletproof Vest
– 2” thick steelvs.
– lightweight synthetic material
• Quality?• Reliability?• Suitability for Purpose?
8/20/2001
Slide 7CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Poor Development Practices can Lead to Failure
• What if the design puts undue strain on a part?– Was the failure due to the part or the
design?• What if the product wasn't properly
tested?– Car overheats in the desert (never
tested that severely)
8/20/2001
Slide 8CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Poor Software Development Practices can lead to Failure
• Software Failures are almost always attributable to product development practices
–Design - Testing–Coding - CM
8/20/2001
Slide 9CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Software Reliability
• Failure: when it does not do what it is supposed to do
• Defect: the reason for the failure– Bad code/data/design/requirements– Bad configuration control– etc.
“The extent to which software correctly performs the functions
assigned to it”
8/20/2001
Slide 10CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Improving Software ReliabilityOption 1
Design software to be fault tolerant– Redundancy– Multiple algorithms
• This approach has been shown to have very little effect on overall reliability
• It is a better fit to the hardware paradigm that involves fatigue of parts
8/20/2001
Slide 11CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Improving Software ReliabilityOption 2
Develop software to be free of defects– Prevention activities– Detection activities
• This is where we will concentrate• Traditionally this has been viewed as
a lost cause• But using modern techniques of quality
improvement, there is hope for success
8/20/2001
Slide 12CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Measuring Reliability
8/20/2001
Slide 13CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
DefinitionsFailure
– When the product does not do what it is expected to do for a given set of input or operating conditions.
Fault (depends on author) – A condition that causes failures.
Defect (depends on author):– A fault found before / after product release– Any cause of failure– Any error, regardless of whether it is caught
before release– Other terms: bug, mistake, malfunction, etc.
8/20/2001
Slide 14CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
What is Reliability?• Reliability is the probability that
software will operate for a given time interval (usually denoted by t) under given conditions (usually denoted by C or E) with no failures– t is a random, failure free time interval. – The question we are trying to answer is:
how long is t? – But since we cannot know this, we can only
estimate the probability for a given value of t.
8/20/2001
Slide 15CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Reliability is not Correctness• Reliability means that it does what
you want it to do often enough to be satisfactory
whereas• Correctness is a binary, “yes or no”
condition• Software is almost never perfectly
correct• But it can be highly reliable
8/20/2001
Slide 16CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
This definition is usually expressed as a function:
R(t) = probability of operation without
failures in time t(i.e., in the interval 0-t)
• For example, with an exponential distribution, the classic reliability equation is: R(t) = e -t/
8/20/2001
Slide 17CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Graph of a Reliability Function
Reliability Function for Exponential Distribution
0
0.2
0.4
0.6
0.8
1 2 3 4 5 6 7 8 9
t = time since product release
R(t) = e -t/
8/20/2001
Slide 18CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Measures Reliability as a Constant
• For large values of , the probability of operation without failure remains high for a longer period of time
• For small values of , the probability of operation without failure deteriorates quickly
This works with an exponential distribution function. Otherwise, there may not be a simple
constant to measure reliability of the total product.
8/20/2001
Slide 19CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Additional Notes about Reliability
• The desired value of t depends a lot on the application and the priorities
• Commercial application– t is large & the priority is to have
few defects over the life of the application in order to keep maintenance cost low
• Real time application – e.g. an aircraft application, t is relatively
short & failures in operation are critical
8/20/2001
Slide 20CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Failure Function• Another popular approach is to look at
the probability of a failure:F(t) = 1 - R(t) = probability of failure in
time t• The latter is called a failure function.• It is the cumulative distribution
function of the time interval 0-t.• For the exponential distribution, the
failure function is:F(t) = 1 - e -t/
8/20/2001
Slide 21CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Graph of a Failure Function
Failure Function for Exponential Distribution
00.20.40.60.8
11.2
1 2 3 4 5 6 7 8 9
t = time since product release
F(t) = 1 - e -t/
8/20/2001
Slide 22CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Probability Density Function• Attempts to put it in another form
that means something to a user: “(approximately) how many defects
are left in the software?”
For the exponential distribution, the formula is:
f(t) = dF(t)/dt
f(t) = -1e -t/
8/20/2001
Slide 23CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Graph of Density Function
00.050.1
0.150.2
0.250.3
0.35
1 2 3 4 5 6 7 8 9
t = time since product release
f(t) = -1e -t/
8/20/2001
Slide 24CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Post Release DefectsPost- Release Defects
Product XYZ Release 2.3
020406080
0 1 2 3 4 5 6 7 8 9 10 11 12Months Since Release
Defe
cts
Defects Found Total Remaining
8/20/2001
Slide 25CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
All three functions( = 2)
00.20.40.60.8
11.2
1 2 3 4 5 6 7 8 9t = time since product release
R(t) F(t) f(t)
8/20/2001
Slide 26CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Failure Rate ()(Hazard Function; Failure
Intensity)• This is an attempt to estimate the
probability of a failure at a given time, assuming no prior failures.
(t) = f(t)/R(t) = -dR(t)/dt
8/20/2001
Slide 27CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
How to Determine • For an exponential distribution, is a constant
• I.e., the higher the reliability, the lower the failure rate
= 1/
This works with an exponential distribution. Otherwise, may be a more complex function.
8/20/2001
Slide 28CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Failure Ratevs Number of Defects
• Most models assume the failure rate is directly related to the number of defects remaining in the software.
Is this a reasonable assumption? (Discussion)
8/20/2001
Slide 29CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Problems with the Assumptions for Classic
Definitions• Defects are random with respect to
the structure of the software– But some parts are harder to write than
others• Testing is uniform with respect to
software– But some parts are better tested than
others
8/20/2001
Slide 30CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
More Problems with Assumptions
• All defects are equally likely to occur– But it actually depends on the paths
taken most often• All defects produce equally serious
failures– Clearly not the case for software
• Testing correctly simulates normal, stressful and unusual conditions– Generally this is very hard to do
8/20/2001
Slide 31CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Error ProbabilityHardware vs. Software
Probability of Error vs. Time
00.20.40.60.8
1
1 2 3 4 5 6 7 8 9 10 11 12
Hardware Software
8/20/2001
Slide 32CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Another View
Def ect Rate af ter Product Release
05
10152025
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Hardware Software
Infant Mortality
Wearout
8/20/2001
Slide 33CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Software May Not FitClassical Models of Reliability• There is very little solid data for any
of the models in use for software• Conditions that differ may produce
very different failure rates, even though defect count remains the same
8/20/2001
Slide 34CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
How to Measure Time
• The measure of time is a matter of considerable dispute
• This may dramatically affect how you determine reliability
8/20/2001
Slide 35CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Three Ways to Measure Timefor a Software Product
• Real Time (Calendar Time)– Number of weeks or months since some
event• Use Time
– Number of hrs the software is in actual use• CPU Time
– Number of hours using the CPU• Each of these produces very different
results and fits very different models
8/20/2001
Slide 36CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
The Nature of The Application
• Different applications can have very different notions of reliability
8/20/2001
Slide 37CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Different Applications - Different Reliability ImplicationsApplication: Financial TransactionsProblem: Floating Point Round off ErrorsNot a Problem: Excessive Time for Calculations
Application: Space Craft Flight Path Calculations (ground
based)See Financial Transactions
Application: Space Craft Local NavigationProblem: Excessive Time for CalculationsNot a Problem: Floating Point Round off Errors
8/20/2001
Slide 38CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
Summary• Reliability is important, but hard to
measure in a way that relates to customer expectations
• Software reliability is mainly determined by development practices rather than by manufacturing or materials
• Reliability, failure rate, and failure density are related measures
8/20/2001
Slide 39CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
References
• Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS00030. ISBN 0-07-039400-8.
• Musa, John, Software Reliability Engineering: More Reliable Software, Faster Development and Testing, McGraw Hill. ISBN: 0-07-913271-5.
• Xie, M. Software Reliability Modeling, World Scientific, London, 1991. ISBN 981-02-0640-2.
8/20/2001
Slide 40CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12
END OFMODULE 12