version 5.09
Slide 1CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
SMU CSE 8314 / NTU SE 762-N
Software Measurement and Quality Engineering
Module 15Six Sigma and Zero Defects - Overview
version 5.09
Slide 2CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Contents• Zero Defects Overview• Six Sigma Overview• Summary
version 5.09
Slide 3CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Zero Defects and Six Sigma
General Concepts
version 5.09
Slide 4CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Zero Defects and Six Sigma• Each of these is a measure of
quality• And also a set of principles and
concepts that can be used to establish a program for quality improvement
version 5.09
Slide 5CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
GoalsA measure of quality that can be applied to anything you produce and that will result in unmatched quality across the board
• Dimensionless• Fosters & motivates quality improvement• Provides insight into the quality
improvement process• Correlates to our innate sense of quality
version 5.09
Slide 6CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Zero Defects
See Schulmeyer in reference list
version 5.09
Slide 7CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Average Defects per Product
Zero Defects is a Stretch GoalStretch Goal
No defects in delivered productsExpectation
Number of defects will diminish» (asymptotically approach zero)
time
version 5.09
Slide 8CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Experience with Zero Defects
0
2
4
6
8
10
Cost of Non- Conformance Cost of Conformance Net Cost
Many quit about here
version 5.09
Slide 9CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Another Experience withZero Defects
Number of Products with Indicated Number of Defects
0100200300
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Number of Defects
Last Year This Year
version 5.09
Slide 10CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
How Do You Know How Well You Are Doing Relative to
What Is Possible?• The “Zero Defects” measure is not
directly related to “degree of goodness” or “degree of quality” or “what is possible”
• It only measures defects on an absolute scale
version 5.09
Slide 11CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
• What is needed for some products is not needed for others -- you need to know what the customer requires
How Do You Correlate the Defect Rate to What is
Possible or to Relative Cost?
version 5.09
Slide 12CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
How Many Products Must Be Defect Free?
• 99%– would mean 1 typo per 100 words of
course notes which is fairly good– but --- 200,000 wrong drug
prescriptions per year - very bad• 99.9%
– 1 typo per page - good– 500 surgical errors per week
• 99.99%– 2000 mail delivery errors per hour
version 5.09
Slide 13CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
What about Variance?• Average product vs worst case
product– Which one matters more?– Is it better to have an average
product with .3 defects or to have a worst case product with 2 defects?
• In its simplest form, “zero defects” does not tell us the answer to this kind of question
version 5.09
Slide 14CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Other Problems with Zero Defect Programs
• How do you gain insight into the nature of the problems?– The measure says nothing about the
causes of the defects or how to cure them
• How do you justify continuous improvements in defect removal?– There is no good way to know if you
can justify the cost
version 5.09
Slide 15CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Six Sigma
version 5.09
Slide 16CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Premise of Six Sigma Programs
• You use a process to produce something
• The process can vary as well as the product
• Average number of defects is not an acceptable measure . . .– You need to understand the worst case
and why it happens– You need to control process variance as
well as defects
version 5.09
Slide 17CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Product Quality Depends on:1) The design of the product2) The materials used to construct it3) The process used to produce it
1) Design 2) Materials
3) ProductionProcess
(Outputs)(Inputs)Products
version 5.09
Slide 18CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Applying to Software• For software, the product is
essentially a design!• So the three factors become:
– Inputs:1) The architecture of the software2) The requirements of the software
– Process:3) The software development process
version 5.09
Slide 19CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Software Quality Depends on:
1) The architecture of the software2) The requirements used to
construct it3) The process used to develop it
3) DevelopmentProcess1) Architecture
2) Requirements
(Outputs)(Inputs)Software
version 5.09
Slide 20CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Basic Units ofSix Sigma Programs
• Opportunity– Any step of the process, type of
material, or design element that can cause a defect
• Defect– Anything wrong with the product
• Defects per million opportunities (DPMO)– This is the basic measure of quality
version 5.09
Slide 21CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
DPMO• Defects are problems with the
product• Opportunities are steps in the
process, materials, or design elements where you can make a mistakeDPMODPMO = Number of Defects * 106
Number of Opportunities
version 5.09
Slide 22CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
To Improve Quality• Decrease the number of defects or• Increase the number of opportunities?
A common mistake when first learning about six sigma concepts is to assume that increasing opportunities is a good
thing. In fact, it is a bad thing.
version 5.09
Slide 23CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Count opportunities in such a way as to inflate the number, so that DPMO goes down.
Make the process more complex so as to increase the number of opportunities
Correct the process to reduce defects Simplify the process so that
opportunities go down and total defects go down as well
Uses and Abuses of DPMO
version 5.09
Slide 24CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Software Defects & Opportunities
• A defect is a failure to meet requirements
• An opportunity is a process step, architecture element, or requirement where a defect could originate
• In a complex product like software, it is very hard to determine all of the opportunities
version 5.09
Slide 25CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Characteristics of an Opportunity
• Independent from other opportunities• Constant total number for the
product/process if there is no change in the process
• Consistent measuring method is more important than exact definition
The real objective is to measure & improve, not to count opportunities
perfectly.
version 5.09
Slide 26CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Software Opportunities• One shortcut measure of
opportunities is lines of code (if you measure size in lines of code)
• Another might be steps of the process, if your process description is very detailed down to the individual work stepDo NOT get overly concerned about the
definition of an opportunity or defect
version 5.09
Slide 27CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Goals
• This goal is derived from a specific analysis of processes and has specific rationale behind it– Details in later slides and modules
• Other goals (more or less aggressive) can be established, based on understanding the rationale
6 Sigma goal: DPMO < 3.4
version 5.09
Slide 28CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Techniques to Achieve 6 Sigma
1) Design the product for producability
2) Improve the quality of the materials
3) Design the process to produce quality products
1) Design 2) Materials
3) ProductionProcess
(Outputs)(Inputs)Products
version 5.09
Slide 29CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Techniques to Achieve 6 Sigma Quality for Software
1) Architect the software for ease of development and maintenance
2) Improve the quality of the requirements3) Design the development process to produce quality software
1) Architecture 2) Requirements
3) DevelopmentProcess
(Outputs)(Inputs)Software
version 5.09
Slide 30CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Process Analysis forManufactured Products
• Each product or raw material has certain measurable characteristics
• Example: a bolt could be measured in terms of:– Length– Diameter– Spacing of threads– Diameter of head– Weight– Strength of material under specific conditions
version 5.09
Slide 31CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
For Each Measurable Property
You Establish SpecificationsExample: “Diameter should be 0.4 inches, plus
or minus .002 inches”• Notice that there are two
components:–A target value
• specification–A range of values that are acceptable
• specification limits
version 5.09
Slide 32CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Responsibility of Designer• Specify tolerances• Design products with reasonable
tolerances relative to the manufacturing capability
• Create designs that can be produced within the stated tolerances
version 5.09
Slide 33CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Responsibility of Software Architect
• Specify bounds of acceptable software behavior
• Architect software with reasonable tolerances relative to the design and programming capabilities of the staff
• Create software architectures that can be produced and maintained
version 5.09
Slide 34CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
SEI Claim0
5
10
15
20
1 3 5 7 9 11 13 15 17 19
0
5
10
15
20
25
1 3 5 7 9 11 13 15 17 19
0
5
10
15
20
25
30
1 3 5 7 9 11 13 15 17 19
0
5
10
15
20
25
30
1 3 5 7 9 11 13 15 17 19
0
10
20
30
40
1 3 5 7 9 11 13 15 17 19
1
2
3
4
5
As you increase
maturity level, the cost and
schedule decrease and the variance goes down
version 5.09
Slide 35CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Problems with Software and Sigma
• Software generally has low product volume compared with manufactured products– But what if we measure units, tests,
objects, screens, functions, etc?• Software development process has
very high variance– Does it need to?– Is that necessarily bad?
version 5.09
Slide 36CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Problems with Software (continued)
• We don’t have the statistical data to back up applying these techniques– But maybe that does not matter if the
ideas for improvement still apply• NIH (Not Invented Here)
version 5.09
Slide 37CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
Summary• Six Sigma and Zero Defects
programs involve– Measures of quality, such as– Goals that minimize quality problems– Methods of quality improvement
• Six Sigma focuses on variability as well as overall qualityThe next module will cover six sigma
principles and applications.
version 5.09
Slide 38CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
References• Harry, Mikel J. and J. Ronald Lawson, Six Sigma
Producibility Analysis and Process Characterization, Motorola University Press, Addison-Wesley, ISBN 0-201-63412-0
• Schulmeyer, G. Gordon. Zero Defect Software. McGraw Hill, 1990. ISBN 0-07-055663-6.
• Schulmeyer, G. Gordon and James McManus. Handbook of Software Quality Assurance, Second Edition (especially chapter 17). Van Nostrand Reinhold, New York, 1992. ISBN 0-442-00796-5.
version 5.09
Slide 39CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M15
END OFMODULE 15