cse 8314 - sw measurement and quality engineering copyright © 1995-2005, dennis j. frailey, all...

version 5.09

Slide 1CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

SMU CSE 8314 / NTU SE 762-N

Software Measurement and Quality Engineering

Module 15Six Sigma and Zero Defects - Overview

version 5.09

Slide 2CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Contents• Zero Defects Overview• Six Sigma Overview• Summary

version 5.09

Slide 3CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Zero Defects and Six Sigma

General Concepts

version 5.09

Slide 4CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Zero Defects and Six Sigma• Each of these is a measure of

quality• And also a set of principles and

concepts that can be used to establish a program for quality improvement

version 5.09

Slide 5CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

GoalsA measure of quality that can be applied to anything you produce and that will result in unmatched quality across the board

• Dimensionless• Fosters & motivates quality improvement• Provides insight into the quality

improvement process• Correlates to our innate sense of quality

version 5.09

Slide 6CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Zero Defects

See Schulmeyer in reference list

version 5.09

Slide 7CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Average Defects per Product

Zero Defects is a Stretch GoalStretch Goal

No defects in delivered productsExpectation

Number of defects will diminish» (asymptotically approach zero)

time

version 5.09

Slide 8CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Experience with Zero Defects

0

2

4

6

8

10

Cost of Non- Conformance Cost of Conformance Net Cost

Many quit about here

version 5.09

Slide 9CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Another Experience withZero Defects

Number of Products with Indicated Number of Defects

0100200300

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Number of Defects

Last Year This Year

version 5.09

Slide 10CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

How Do You Know How Well You Are Doing Relative to

What Is Possible?• The “Zero Defects” measure is not

directly related to “degree of goodness” or “degree of quality” or “what is possible”

• It only measures defects on an absolute scale

version 5.09

Slide 11CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

• What is needed for some products is not needed for others -- you need to know what the customer requires

How Do You Correlate the Defect Rate to What is

Possible or to Relative Cost?

version 5.09

Slide 12CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

How Many Products Must Be Defect Free?

• 99%– would mean 1 typo per 100 words of

course notes which is fairly good– but --- 200,000 wrong drug

prescriptions per year - very bad• 99.9%

– 1 typo per page - good– 500 surgical errors per week

• 99.99%– 2000 mail delivery errors per hour

version 5.09

Slide 13CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

What about Variance?• Average product vs worst case

product– Which one matters more?– Is it better to have an average

product with .3 defects or to have a worst case product with 2 defects?

• In its simplest form, “zero defects” does not tell us the answer to this kind of question

version 5.09

Slide 14CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Other Problems with Zero Defect Programs

• How do you gain insight into the nature of the problems?– The measure says nothing about the

causes of the defects or how to cure them

• How do you justify continuous improvements in defect removal?– There is no good way to know if you

can justify the cost

version 5.09

Slide 15CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Six Sigma

version 5.09

Slide 16CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Premise of Six Sigma Programs

• You use a process to produce something

• The process can vary as well as the product

• Average number of defects is not an acceptable measure . . .– You need to understand the worst case

and why it happens– You need to control process variance as

well as defects

version 5.09

Slide 17CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Product Quality Depends on:1) The design of the product2) The materials used to construct it3) The process used to produce it

1) Design 2) Materials

3) ProductionProcess

(Outputs)(Inputs)Products

version 5.09

Slide 18CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Applying to Software• For software, the product is

essentially a design!• So the three factors become:

– Inputs:1) The architecture of the software2) The requirements of the software

– Process:3) The software development process

version 5.09

Slide 19CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Software Quality Depends on:

1) The architecture of the software2) The requirements used to

construct it3) The process used to develop it

3) DevelopmentProcess1) Architecture

2) Requirements

(Outputs)(Inputs)Software

version 5.09

Slide 20CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Basic Units ofSix Sigma Programs

• Opportunity– Any step of the process, type of

material, or design element that can cause a defect

• Defect– Anything wrong with the product

• Defects per million opportunities (DPMO)– This is the basic measure of quality

version 5.09

Slide 21CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

DPMO• Defects are problems with the

product• Opportunities are steps in the

process, materials, or design elements where you can make a mistakeDPMODPMO = Number of Defects * 106

Number of Opportunities

version 5.09

Slide 22CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

To Improve Quality• Decrease the number of defects or• Increase the number of opportunities?

A common mistake when first learning about six sigma concepts is to assume that increasing opportunities is a good

thing. In fact, it is a bad thing.

version 5.09

Slide 23CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Count opportunities in such a way as to inflate the number, so that DPMO goes down.

Make the process more complex so as to increase the number of opportunities

Correct the process to reduce defects Simplify the process so that

opportunities go down and total defects go down as well

Uses and Abuses of DPMO

version 5.09

Slide 24CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Software Defects & Opportunities

• A defect is a failure to meet requirements

• An opportunity is a process step, architecture element, or requirement where a defect could originate

• In a complex product like software, it is very hard to determine all of the opportunities

version 5.09

Slide 25CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Characteristics of an Opportunity

• Independent from other opportunities• Constant total number for the

product/process if there is no change in the process

• Consistent measuring method is more important than exact definition

The real objective is to measure & improve, not to count opportunities

perfectly.

version 5.09

Slide 26CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Software Opportunities• One shortcut measure of

opportunities is lines of code (if you measure size in lines of code)

• Another might be steps of the process, if your process description is very detailed down to the individual work stepDo NOT get overly concerned about the

definition of an opportunity or defect

version 5.09

Slide 27CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Goals

• This goal is derived from a specific analysis of processes and has specific rationale behind it– Details in later slides and modules

• Other goals (more or less aggressive) can be established, based on understanding the rationale

6 Sigma goal: DPMO < 3.4

version 5.09

Slide 28CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Techniques to Achieve 6 Sigma

1) Design the product for producability

2) Improve the quality of the materials

3) Design the process to produce quality products

1) Design 2) Materials

3) ProductionProcess

(Outputs)(Inputs)Products

version 5.09

Slide 29CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Techniques to Achieve 6 Sigma Quality for Software

1) Architect the software for ease of development and maintenance

2) Improve the quality of the requirements3) Design the development process to produce quality software

1) Architecture 2) Requirements

3) DevelopmentProcess

(Outputs)(Inputs)Software

version 5.09

Slide 30CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Process Analysis forManufactured Products

• Each product or raw material has certain measurable characteristics

• Example: a bolt could be measured in terms of:– Length– Diameter– Spacing of threads– Diameter of head– Weight– Strength of material under specific conditions

version 5.09

Slide 31CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

For Each Measurable Property

You Establish SpecificationsExample: “Diameter should be 0.4 inches, plus

or minus .002 inches”• Notice that there are two

components:–A target value

• specification–A range of values that are acceptable

• specification limits

version 5.09

Slide 32CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Responsibility of Designer• Specify tolerances• Design products with reasonable

tolerances relative to the manufacturing capability

• Create designs that can be produced within the stated tolerances

version 5.09

Slide 33CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Responsibility of Software Architect

• Specify bounds of acceptable software behavior

• Architect software with reasonable tolerances relative to the design and programming capabilities of the staff

• Create software architectures that can be produced and maintained

version 5.09

Slide 34CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

SEI Claim0

5

10

15

20

1 3 5 7 9 11 13 15 17 19

0

5

10

15

20

25

1 3 5 7 9 11 13 15 17 19

0

5

10

15

20

25

30

1 3 5 7 9 11 13 15 17 19

0

5

10

15

20

25

30

1 3 5 7 9 11 13 15 17 19

0

10

20

30

40

1 3 5 7 9 11 13 15 17 19

1

2

3

4

5

As you increase

maturity level, the cost and

schedule decrease and the variance goes down

version 5.09

Slide 35CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Problems with Software and Sigma

• Software generally has low product volume compared with manufactured products– But what if we measure units, tests,

objects, screens, functions, etc?• Software development process has

very high variance– Does it need to?– Is that necessarily bad?

version 5.09

Slide 36CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Problems with Software (continued)

• We don’t have the statistical data to back up applying these techniques– But maybe that does not matter if the

ideas for improvement still apply• NIH (Not Invented Here)

version 5.09

Slide 37CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

Summary• Six Sigma and Zero Defects

programs involve– Measures of quality, such as– Goals that minimize quality problems– Methods of quality improvement

• Six Sigma focuses on variability as well as overall qualityThe next module will cover six sigma

principles and applications.

version 5.09

Slide 38CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

References• Harry, Mikel J. and J. Ronald Lawson, Six Sigma

Producibility Analysis and Process Characterization, Motorola University Press, Addison-Wesley, ISBN 0-201-63412-0

• Schulmeyer, G. Gordon. Zero Defect Software. McGraw Hill, 1990. ISBN 0-07-055663-6.

• Schulmeyer, G. Gordon and James McManus. Handbook of Software Quality Assurance, Second Edition (especially chapter 17). Van Nostrand Reinhold, New York, 1992. ISBN 0-442-00796-5.

version 5.09

Slide 39CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15

END OFMODULE 15