Software Reliability

Aaron Hoff

OverviewCompare and hardware and software

reliabilityDiscuss why software should be reliable?Describe MLE (Maximum Likelihood

Estimation)Show two specific reliability models

Mill’s Error Seeding ModelJelinski-Moranda Model

Software Reliability ToolsTrainingConclusion

ReliabilityWebster’s Dictionary defines reliability as:

1. The quality or state of being suitable or fit to be relied on, dependable2. The extent to which an experiment, test, or measuring procedure yields the same result on repeated trials.

Hardware Reliability

Hardware Reliability(cont.)Failure rate is very high during the burn in

period.Many faults are found on all components.Thorough testing of all components cuts

down on the number of faults.Enters the useful life with small amount of

faults.After time, wears down and quickly increases

in failure rate.

Software Reliability

Software Reliability(cont.)Starts with many faults in system when first

createdAfter much testing/debugging enters useful

life cycleUseful life includes upgrades made to system

which bring about new faults.System needs to then be tested to reduce

faults.Eventually evens out into the Obsolescence

cycle where software is usually quite reliable.

Compare/Contrast HW & SWBoth start out with large number of faultsBoth need to be tested completely

Hardware faults are physical, software faults are not

Hardware stays at steady reliability level in useful life, software needs constant testing after upgrades.

Hardware wears out over time, software does not.

Hardware failure is random, software failure is systematic.

Why should software be reliable?Examples of software failure

Therac 25 (1985 – 1986)Ariane 5 (1996)Mars Lander (1999)

Lives can be put in dangerMoney and time can be lostLost trust of customers.

Software Reliability ModelsWhy do we need them?

Predict probability of failure of a component or system

Estimate the mean time to the next failure Predict number of (remaining) failures

Software Reliability ModelsMany models have come out to help increase

reliabilityAll models can be grouped into these

categories: Error seeding, Failure rate, Curve fitting, Reliability growth, Program structure, Input domain, Execution path, Nonhomogenous Poisson process, Bayesian and unified, and Markov.

MLE(Maximum likelihood estimation) is method used for fitting statistical model to data

MLEPrinciple is to estimate parameter values of

models which make observed data most likely to occur

Probability itself cannot be used for estimation of parameters

MLE uses likelihood instead to estimate parameter values

Uses sample data set to estimate different parameter values.

Coin Flip ExampleProbability of flipping heads over 100 flips is

0.5Say sample set is: Heads was flipped 56 times

Plug variety of probability values into this model to obtain data that can be graphed.

Graph for MLE of Coin Flip example

Error SeedingEstimates total number of errors by

introducing known errors into software.Terminology

Inherent error – one found in software that causes failure regardless of what user does.

Induced(seeded) error – one intentionally inserted into piece of software to estimate total number of errors

Mill’s error seeding modelProposed in 1970 by MillUsed during testing phaseDevelopers/Testers insert errors in places

they think errors would occur (error prone locations)

Test whole systemGather data on all errors found during testing

processTotal number of errors can then be estimated

Mill’s Error Seeding Model(cont.)Hypergeometric distribution to find probability of k induced

errors

where N = total number of inherent errorsn1 = total number of induced errorsr = total number of errors removed during debuggingk = total number of induced errors in r removed errorsr – k = total number of inherent errors in r removed errors

Mill’s Error Seeding Model(cont.)The hypergeometric distribution can be

simplified greatly to achieve an equation for total # of inherent errors

The lower the total number of errors estimated, the higher the reliability.

Advantages/DisadvantagesA - The fault representation is easy to apply

with fault-seeding tool.A - Can be used to predict the fault

distribution of particular software.

D - This model is very time consumingD - Can not be applicable in large programsD - Always chance of human error when

deciding where to put all induced error.

Failure Rate(Terminology)Failure Rate – the frequency with which an

engineered system or component fails.Failure – Occurs when the user perceives that

the program ceases to deliver the expected service

Fault – The cause of the failure or internal error of software

Basic premise of failure rate is that successive failure rates will get longer as faults are removed from the software system.

Jelinski-Moranda ModelOne of the earliest models(1972) proposed when

looking into software reliability.Six assumptions:

Program contains N initial faults which is unknown but a fixed constant

Each fault is independent and equally likely to cause failure

Time intervals between failures are independentWhen failure occurs, corresponding fault is removedFault is assumed to be instantaneously removed, no new

faults are inserted during removalSoftware failure rate is constant and proportional to

number of faults remaining in software

Jelinski-Moranda ModelThe six assumptions set ground rules for

model.Intensity fail rate function:

where φ = a proportional constant, the contribution

any one fault makes to the overall program;N = the number of initial faults in the

program;ti = the time between the (i-1)th and the ith

failures.

Jelinski-Moranda ModelThe intensity function is used to obtain magnitude

of the failure rate at a certain failure interval.Infer: After first failure rate where fault is removed

with certainty, intensity shall be lowered in proportion to number of faults remaining.

Jelinski-Moranda used this information to obtain a reliability function.

MLE is used to estimate values such as the number of initial faults(N) or proportional constant(φ)

Software Reliability ToolsSMERFS(Statistical Modeling and Estimation

of Reliability Functions for Software) - allows user to perform complete software reliability analysis

SARA(Software Assurance Reliability Automation) - incorporates both reliability growth modeling and design code metrics for analyzing software time between failure data

TrainingTraining organizations

RACReliaSoftSoftRel - www.softrel.com/prod03.htmSoHaR

ConclusionSW reliability is similar to HW reliability but

must be treated differentlyReliability of software is something to strive

forCan prevent major faults that have possibility

of taking human life, money, time, and customers

Useful to have model or something to measure

Many models have been proposedTraining is available to those who want to

learn more about reliability engineering

References[1] Ensuring Software Reliability. Neufelder, Ann Marie. New York: Marcel

Dekker, Inc., 1992. 1-242.

 [2] Goddard Space Flight Center. Overview of Software Reliability. February 16,

2005. <http://sw-assurance.gsfc.nasa.gov/disciplines/reliability/index.php> [3] Handbook of Software Reliability Engineering. Ed. Michael R. Lyu.

New York: McGraw-Hill Companies, Inc., 1996. 3-779. [4] Lloyd, Robin. Metric mishap caused loss of NASA orbiter. September

30,1999. <http://cnn.com/TECH/space/9909/30/mars.metric.02/>

 [5] Pan, Jiantao. Software Reliability. Spring 1999.

<http://www.ece.cmu.edu/~koopman/des_s99/sw_reliability/#metrics>

References(cont.)[6] Purcell, S. Maximum Likelihood Estimation. May 20, 2007.

<http://statgen.iop.kcl.ac.uk/bgim/mle/sslike_1.html> [7] Software Reliability. Pham, Hoang. Singapore: Springer-Verlag Singapore

Pte. Ltd., 2000. 1-219. [8] Software Reliability and Testing. Pham, Hoang. Piscataway: The Institute of

Electrical and Electronics engineers, Inc., 1995. 1-133. [9] Software Reliability Models. Malaiya, Yashwant K., and Pradip K. Srimani.

New York: Institute of Electrical and Electronics Engineers, Inc., 1990. 1-121.

 [10] Software Safety and Reliability. Herrmann, Debra S. Piscataway: The

Institute of Electrical and Electronics engineers, Inc., 1999. 5-466.