software quality overview - practices€¦ · software quality overview: ... dmaic and dmadv. –...

4/2/2012 Contents copyright © Operational Excellence Networks 2012 11

Software Quality Overview:Quality Frameworks

Duvan Luong Ph.D.Operational Excellence Networks


Release/Project Success ProfilesThe reasons projects succeed:

– User Involvement– Executive Management Support (Active interest)– Clear Statement of Requirements– Proper Planning– Realistic Expectations– Smaller Project Milestones (Visibility)– Competent Staff– Ownership– Clear Vision and Objectives– Risk Planning, Identification and Mitigation


Release/Project Obstacles ProfilesThe reasons projects are challenged:

– Lack of User Input– Incomplete Requirements and Specifications– Changing Requirements and Specifications– Lack of Executive Support– Technical Incompetence– Lack of Resources– Unrealistic Expectations– Unclear Objectives– New Technology– Team Commitment


The reasons releases/projects are impaired and canceled:

• Incomplete Requirements• Lack of User Involvement• Lack of Resources• Unrealistic Expectations• Lack of Executive Support• Changing Requirements• Lack of Planning• Obsolete before it is

complete

• Lack of IT Management Oversight

• Technology Illiteracy• Does not solve business

problem• Unrealistic schedule

requirements• Lack of Program

Management training• Poor Estimating


Quality frameworks

• ISO 9000 series• 6 Sigma• SEI CMM/CMMI


ISO 9000 standards• Maintained by the International Organization for

Standardization (ISO) as the model for implementing company quality systems

• Original standards published in 1987 which are based on the British quality standards BS5750

• Focus on the manufacturing environment which emphasizes the control of process variation to ensure quality


ISO 9000 family• ISO 9000 QMS-Fundamentals and vocabulary• ISO 9001 QMS-Requirements• ISO 9004 QMS-Guidelines for performance

improvements• ISO 19011 Guidelines on quality and

environmental management systems auditing• ISO 10012 Measurement control system• Company certified on ISO 9001 standards


ISO 9001 principles

Say what you do

Do what you say

And where necessary

Show what you did


ISO 9001-1994 elements1. Management Responsibility2. Quality System3. Contract Review4. Design Control5. Document Control6. Purchasing7. Purchaser Supplied Product8. Product Identification and

Traceability9. Process Control10. Inspection and Testing

11. Inspection Measuring and Test Equipment

12. Inspection and Test Status13. Control of Non-Conforming

Product14. Corrective Action15. Handling, Storage, Packaging

and Delivery16. Quality Records17. Internal Quality Audits18. Training19. Servicing20. Statistical Techniques


ISO 9001-2008 latest updateISO 9001-2008 update reorganized the requirements into 5 groupings:

• Quality Management System:4.1 General requirements4.2 Documentation requirements

• Management Responsibility 5.1 Management Commitment5.2 Customer focus5.3 Quality policy5.4 Planning5.5 Responsibility, authority and

communication5.6 Management review

• Resource Management6.1 Provision of resources6.2 Human resources6.3 Infrastructure6.4 Work environment

• Product Realization7.1 Planning of product realization7.2 Customer related processes7.3 Design and development7.4 Purchasing7.5 Production and service provision7.6 Control of monitoring and

measuring devices• Measurement, Analysis and Improvement

8.1 General8.2 Monitoring and measurement8.3 Control of non-conforming product8.4 Analysis of data8.5 Improvement


ISO 9000 implementation for Software companies

• ISO 9000-3 guidelines was developed in 1991 as a guidance for the implementation of the ISO-9001 standards in the software environment


6 Sigma• Six Sigma is a disciplined, data-driven approach and methodology for

eliminating defects in any process. • To achieve Six Sigma, a process must not produce more than 3.4

defects per million opportunities.• There are two Six Sigma process models: DMAIC and DMADV.

– The Six Sigma DMAIC process (define, measure, analyze, improve, control) is an improvement system for existing processes falling below specification and looking for incremental improvement.

– The Six Sigma DMADV process (define, measure, analyze, design, verify) is an improvement system used to develop new processes or products at Six Sigma quality levels.

– Both Six Sigma processes are executed by Six Sigma Green Belts and Six Sigma Black Belts, and are overseen by Six Sigma Master Black Belts.


6 Sigma history

1980– Established by Motorola in response to foreign

competition and Baldridge Quality Award goal.1990’s

– Focus on process improvement using statistical tools.2005

– “Lean Six Sigma” combines the structured management methodology and problem solving tools of Six Sigma with the business strategy, objectives and goals of “Lean Manufacturing”.


6 Sigma DMAIC• DMAIC

– Define the project goals and customer (internal and external) deliverables

– Measure the process to determine current performance – Analyze and determine the root cause(s) of the defects – Improve the process by eliminating defects – Control future process performance

• When To Use DMAIC– The DMAIC process, instead of the DMADV process, should be

used when a product or process is in existence at your company but is not meeting customer specification or is not performing adequately.


6 Sigma DMADV• DMADV

– Define the project goals and customer (internal and external) deliverables – Measure and determine customer needs and specifications – Analyze the process options to meet the customer needs – Design the process to meet the customer needs – Verify the design performance and ability to meet customer needs

• When To Use DMADV– A product or process is not in existence at your company and one needs to

be developed – The existing product or process exists and has been optimized (using

either DMAIC or not) and still doesn't meet the level of customer specification or six sigma level

DMADV

Define Measure

Analyze

Design

Verify

6 Sigma Statistical Tools


• Voice of the Customer (VOC)- QFD

• XY Matrix• Process Analysis and FMEA• Data Collection Methods• Data Integrity and Accuracy• Basic Statistics

- Common Distributions- Central Limit Theorem- Sampling Distribution of Mean- Basic Probability Concepts

• Decision-Making• Non-Normal Data Graphical• Short vs. Long Term Capability• Data Analysis (B7)

- Cause and Effect Diagram- Check Sheets- Control Charts- Flowcharts- Histogram- Pareto Chart- Scatter Diagram

• Measurement System (Gage) Analysis

• Process Capability Assessment

• Graphic Data Analysis• Multi-Vari Analysis• Inferential Statistics

- Confidence Intervals- Sample Size

• Hypothesis Tests- Means, Variances and Proportions- ANOVA- T-Tests- Test for Equal Variance- Paired Comparison Tests- Chi-Square- Proportion- Contingency Tables- Point and Interval Estimation- Non-Parametric Tests

• Simple Linear Correlation• Regression-Simple and Multiple

- Measure and Model Relationships Between Variables

- Binary Logistics • Process Modeling and Simulation

• Hypothesis Tests- ANOVA- Non-Parametric tests- Contingency Tables

• Design of Experiments- Terminology- Plan and Organize Experiments- Design Principles - One Factor- Full-Factorial- Two Level Fractional- Mixture Experiments- Taguchi Robustness Concepts

• Response Surface Methodology- Steepest Ascent/Decent- High Order Experiments

• Evolutionary Optimization (EVOP)

Design for Six Sigma• Quality Function Deployment (QFD)• Robust Design and Process

- Functional Requirements- Noise Strategies- Tolerance Design- Tolerance and Process Capability

• Design Analysis and FMEA• Reliability Fundamentals

• Control Charts- Theory, Objectives, Benefits- Variable Selection- Rational Sub-grouping- Selection/Application- Analysis- Moving Average- Exponentially Weighted

Moving Average (EWMA)- CuSum

• Process Capability Assessment

6 Sigma Management Tools


• Business Case• Problem/Objective Statement• Project Scope• Project Goals• Project Definition• Project Charter• Identify Owners and Stakeholders• Identify Customers• Financial Benefits• Benchmarking• Team Leadership• Team Dynamics and Performance• Change Agent• Performance Measurements• Gap Analysis• High Level Process Mapping/ SIPOC(Suppliers, Inputs, Process, Output, and Customers)

Design for Six Sigma• Design for (X)

- Manufacturing (DFM)- Cost (DFC)- Test (DFT)- Maintainability (DFMA)- Quality/Reliability (DFQ)

• Special Design Tools- Inventive Problem Solving- Axiomatic Design

• Lean Enterprise- Lean Thinking- Theory of Constraints- Continuous Flow Manufacturing- Non-Value Added Activities- Cycle Time Reduction- Cost of Quality- Tools

Visual Factory, Kanban, Poka-Yoke • Solution Implementation

- Mistake-Proofing (Poka-Yoke)- Visible Enterprise

The Six Sigma Journey• Enterprise View• Leadership• Business Strategy• Organizational Goals/Objectives• Project Management Tools (N7)

- Activity Network Diagram - Affinity Diagram- Interrelationship Digraph- Matrix Diagram- Prioritization Matrix/Grid- Tree Diagram- Process Decision Program Chart

• What’s Next-Six Sigma Evolution

• Control Plans- Training- Documentation- Monitoring- Response- Systems and Structures

• Total Productive Maintenance (TPM)• Best Practice Sharing/Translation


6 Sigma belt description

• Yellow Belts: Entry Level team member. Understands the fundamentals of Six Sigma. Data collection

• Green Belts: Lead and execute process-level improvement projects

• Black Belts: Implement the principles, practices, and techniques of Six Sigma for maximum cost reductions. (Lead the project teams)

• Master Black Belts: Successfully completed 20 or more Six Sigma projects.


SEI CMM/CMMI

“The quality of a product is largely determined by the quality of the process

that is used to develop and maintain it.”

Process Improvement Premise


SEI CMM

• The SEI (Software Engineering Institute) is a Federally Funded Research and Development Center Located at Carnegie Mellon University in Pittsburgh, Pennsylvania

• CMM (Capability Maturity Model) “A framework that describes the key elements of an effective software process.”


Five Levels of CMMResult

Level Characteristic

Optimizing Continuous process Process change management(5) capability improvement Technology change management

Defect prevention

Managed(4)

Defined Software process defined(3) and institutionalized to

provide product qualitycontrol

Repeatable(2)

Initial(1)

Product quality planning; Software quality managementtracking of measured Quantitative process managementsoftware process

Management oversightand tracking project;stable planning andproduct baselines

Key Process Areas

Ad hoc(success dependson heroes)

"People"

Software configuration management Software quality assurance Software subcontract management Software project tracking & oversightSoftware project planningRequirements management

Peer reviews Intergroup coordinationSoftware product engineering Integrated software managementTraining programOrganization process definitionOrganization process focus

Risk

Productivity& Quality


Key Process Area (KPA) Structure

Each of the 18 CMM Key Process Areas (KPAs) has:– Purpose– Goals– Common Features:

• Commitment to perform: sponsorship and policies

• Ability to perform: resources, organization, training

• Activities to perform• Measurement of performance• Verification of performance


Process Maturity Profile of Software CommunityMaturity Level 1987-91 1997 1999 2001 December

2002 1- Initial 80% 61% 48% 38% 32%

2- Repeatable 12% 23% 30% 34% 37%

3- Defined 7% 14% 16% 20% 21%

4- Managed 0% 2% 4% 5% 5%

5- Optimizing 1% 1% 2% 4% 5%

Organizations reporting to SEI

130 795 1179 1641 1998

Source: http://www.sei.cmu.edu/sema/profile.html

0

500

1000

1500

2000

1987-91

1992-96

1997 1998 1999 2000 2001 2002

5-Optimizing4-Managed3-Defined2-Repeatable1-Initial


SEI CMMI• SEI CMM retired in December 2003 • Being replaced by the SEI’s CMM Integration (CMMI)• CMMI is a process improvement model that provides a set of

industry-recognized practices to address productivity, performance, costs and stakeholder satisfaction in the systems engineering and software development process.

• Vastly improved version of the CMM • Emphasis on business needs, integration and institutionalization

• There are two representations in CMMI models• Staged• Continuous


CMMI Staged vs. Continuous Model Representations

Staged

ML 1

ML2ML3

ML4

ML5

. . .for an established set of process areas across anorganization

Continuous

. . .for a single process areaor a set of process areas

PA PA

Proc

ess

Are

a C

apab

ility

PA


Five Staged Levels of CMMI

Requirements DevelopmentTechnical SolutionProduct IntegrationVerificationValidationOrganizational Process FocusOrganizational Process DefinitionOrganizational TrainingIntegrated Project Mgmt Risk ManagementDecision Analysis and ResolutionIntegrated Teaming (IPPD only)Org. Environment for Integration Integrated Supplier Management

Level Focus Process Areas

5 OptimizingContinuousProcess Improvement

Organizational Innovation and DeploymentCausal Analysis and Resolution

4 Quantitatively Managed

Quantitative Management

Organizational Process PerformanceQuantitative Project Management

3 Defined ProcessStandardization

Requirements ManagementProject PlanningProject Monitoring and ControlSupplier Agreement ManagementMeasurement and AnalysisProcess and Product Quality AssuranceConfiguration Management

2 ManagedBasicProject Management

1 Initial

QualityProductivity

RiskRework


Generic Goals in Continuous Representation

Capability Level Generic Goal

CL0 No Goal

CL1 GG1: The project supports and enables achievement of the specific goals of the process area by transforming identifiable input work products to produce identifiable output work products

CL 2 GG 2: The process is institutionalised as a managed process

CL 3 GG 3: The process is institutionalised as a defined process

CL 4 GG 4: The process is institutionalised as a quantitatively managed process

CL 5 GG 5: The process is institutionalised as an optimising process


CMMI Process Area (PA) Structure• Each of the CMMI Process Areas (PAs) has:

– Purpose Description– Specific Goals (SG)– Specific Practices (SP) for each goal– Generic Goal (GG): Institutionalize a Managed Process– Generic Practices (GP)

• 1. Establish an Organizational Policy• 2. Plan the Process• 3. Provide Resources• 4. Assign Responsibility• 5. Train People• 6. Manage Configurations• 7. Identify and Involve Relevant Stakeholders• 8. Monitor and Control the Process• 9. Objectively Evaluate Adherence• 10. Review Status with Higher-Level Management


CMMI Process Maturity ProfileMaturity Level Feb 2003 Sept 2003 1- Initial 15% 11% 2- Managed 18% 27% 3- Defined 21% 38% 4- Quantitatively Managed 6% 5% 5- Optimizing 12% 19%

No rating assigned 27%

Organizations reporting to SEI 33 100

1 2 3 4 5

05

10

15

20

25

30

35

40

Orgs Reporting

Maturity Level

Source: http://www.sei.cmu.edu/sema/profile.html


Summary• ISO 9000

– Provides guidelines for establishing a quality system

• 6 Sigma methodology provides the solid foundation for fine tuning company process for

– Budget, – Schedule, and– Quality product

• SEI CMM/CMMI– Developed specially for software industry.– CMMI: Integrates existing quality techniques

software quality overview - practices€¦ · software quality overview: ... dmaic and dmadv. –...

Documents