copyright 1995-2008, dennis j. frailey cse7315 – software project management cse7315 m34 - version...
TRANSCRIPT
Copyright 1995-2008, Dennis J. Frailey CSE7315 – Software Project ManagementCSE7315 M34 - Version 8.01
SMU CSE 7315Planning and Managing a Software Project
Module 34
Software Quality Assurance
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 2
Objectives of This Module
To examine the techniques of software quality assurance
To look at some techniques of software quality engineering
– Notably, cost of quality analysis
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 3
Software Quality Engineering and Assurance
References to text:
Managing Quality 16
Engineering Quality 9,10
Software Testing 11
Other Relevant Material 8
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 4
Evolution of Software Quality
QualityControl
QualityAssurance
QualityEngineeringThis Module Introduced
in This Module
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 5
Software Quality Assurance
Purposes:
– To provide management and developers with visibility into the process being followed and the products being built So they can manage the outcome
– To provide a more effective quality control mechanism
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 6
Software Quality Assurance
Typical Practices:
– Inspections
– Reviews
– Audits
– Communication
– Measurements
– Independent confirmation of compliance Standards, requirements, procedures
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 7
Quality Assurance
Goal: To assure that quality is achieved
-- Add quality during development
-- Improve processes, standards, and up-front evaluations of the product as it is being developed
QualityControl
QualityAssurance
QualityEngineering
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 8
Quality Assurance Looks at the Entire Process
Requirements
Development QC InspectionPass
Fail
Standards ofQuality
Process and Design Standards
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 9
Each Step of the Process Has an Effect on The Final Defect Rate
ProcessStep
I = Defects Input O = Defects Output
F = Defects Found and Fixed
C = Defects Created
O = I + C - F
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 10
Typical Responsibilities of QA Review all development plans for
completeness
Participate as inspection moderators in design and code inspections
Make sure the tests are appropriate and are run correctly
– Note – true quality assurance does not develop or run tests – it assures they are done correctly
Review a significant sample of all test results to determine adherence to standards
Periodically audit SCM performance to determine adherence to standards
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 11
At Each Step, QA must...
Inspect the product for
– Conformance with standards
– Satisfaction of requirements
Evaluate the process for
– Conformance with standards
– Opportunities to improve
– Process defects that may cause problems later on
Evaluate the support processes as well
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 12
At Each Step, QA must...
Monitor reviews, inspections, walkthroughs, etc. to see that they are accomplishing their objectives
Trace back to prior steps when defects are found
In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each
new project
In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each
new project
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 13
At the Requirements Analysis Step ...
Trace software requirements or specifications to original system or customer requirements
Inspect or evaluate software requirements against standards to see that they are
-- complete,
-- correct,
-- consistent,
-- testable, and
-- understandable
“Clean Room” technique takes this farther
PROGRAM SAMPLE (IN1,IN2)INTEGAR IN1,IN2,A,B,C;FOR I = 1 STEP 3 UNTIL 99 DO IF IN1 < A*B THEN IN2 := C + A*B ELSE IN2 := C + IN1 ENDIFENDDO** THE NEXT PART WILL SORTDO FOREVER Ii := I + 1 TEMP := ARRAY[I] - ARRAY[I+1] IF TEMP < 0, SWAP (ARRAY, i)ENDLOOPCALL CHECKORDER(ARRAY)** NEXT WE DO THE i/oCALL PRINT(ARRAY)CALL DEBUG (ARRAY, I)CASE I OF CALL SUB1 CALL SUB1
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 14
“Clean Room” Software Development Technique
Motive: don’t let the bugs get in
Method: filter them out from the beginning
-- Define the requirements using a formal notation
– Reduces ambiguity
– Some consistency/correctness issues can be checked automatically
-- Carry out rigid inspections
Benefits: fewer problems later in the development process (e.g., reduced testing).
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 15
At the Design Step ...
Trace the design to the requirements specification (plus “derived requirements”)
Evaluate the design against standards
-- complexity
-- cohesion
-- understandability
-- etc.
Evaluation of design effectiveness, correctness, consistency, completeness
Evaluate plans for testing and test cases
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 16
At the Coding Step ...
Trace code to design specifications
Evaluate code against coding standards (example: give code to a new maintainer and see if she or he can understand it)
Code walkthroughs or inspections to check on coding mistakes
Test coverage analysis
-- Do the tests address all requirements? (black box tests)
-- Do the tests adequately evaluate the design and its implementation? (glass box tests)
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 17
At the Module Testing Step ...
Test procedures and code analysis
-- Evaluate test code like any other code
-- Are the procedures documented?
-- Are the expected results documented?
Test results analysis
-- Were the tests run?
-- Were the results correct?
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 18
Two Philosophies of Testing
CodeTest
a LittleReview
ThoroughTest
Code ReviewThorough
Test
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 19
During Integration ...
Make sure that regression tests are performed
-- Retest all modules potentially affected by any change
Do independent tests
-- Someone other than the person who developed the software
Formal qualification tests (acceptance tests)
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 20
Planning for QA
Each development project should have a documented Software Quality Assurance Plan (SQAP)
– May be part of SW Development Plan
The SQAP documents tasks to be performed, standards of verification, procedures and organizational structure
There is an IEEE standard for SQAPs
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 21
Pitfalls in Implementing QA
Lack of sufficiently skilled staff in QA
– Software professionals are typically more interested in development jobs than QA
QA management cannot effectively negotiate with the development team
Senior management commitment may fade under schedule pressures
– The developers stop caring about QA & action revolves around insignificant issues
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 22
Pitfalls in Implementing QA(continued)
QA organization operates without suitably documented and approved development standards and procedures
Development team does not produce verifiable quality plans
– Eventually results in a debate over which bugs are more important to fix rather than whether the product has sufficient quality or not
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 23
Pitfalls in Implementing QA(continued)
Disputes over who has responsibility for product quality and for performing QA functions
– The software developers and managers are responsible for product quality
– Quality assurance is a method to help them achieve quality through independent evaluation of their work (like a coach)
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 24
Reporting Chain for QA Staff
Except in the highest maturity organizations, it is strongly recommended that the QA staff have a separate reporting chain from the project team
– This improves their objectivity
– And assures that disputes are resolved at a level in the organization that has responsibility for the possible consequences
Vice President
Quality Mgr
Project Mgr
Software Mgr
SoftwareEngineer
Quality Engineer
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 25
Benefits Achieved by Quality Assurance over Quality Control
Significant insight is gained into the process that can be used to gain long term benefits
– Development can focus on specific weak points in the quality
– Process improvements can be identified
– Steps that are difficult to perform can be automated
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 26
Problems with Quality Assurance
Can still be adversarial (although perhaps less than with quality control)
Motivation to fix the process is still weak
Can be more costly than quality control
It can be hard to show benefits until long after the product has been shipped
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 27
Quality Engineering
Goal: Build quality in as part of the software engineering process
A philosophical change that relies on the professional pride of the software
engineer
Engineering staff define and execute the quality assurance tasks
Team approach to quality, with rewards based on quality
Quality professionals are more like coaches and educators than evaluators
QualityControl
QualityAssurance
QualityEngineering
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 28
Quality Engineering Focuses on People As Well As Process
Finding errors is good -- it keeps them from leaking through to the customer
Problems result in process changes, not punishment of people
Everyone appreciates that a competitive process is the way to remain a competitor
Measurements are used so that decisions are based on fact (in addition to intuition)
Independent tests are welcome -- they tell you if you are as good as you think you are
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 29
Peer Reviews
Motivation: how to find problems in your work in a non-threatening way
Method: evaluation of one individual’s work by others at the same organizational level (i.e., team members, not supervisors)
Software Developers Software Managers
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 30
Peer Reviews - Cost
Cost: may require 10-20% of the total development time
-- this must be planned into the schedule
-- it must also be planned into progress measures and reward systems
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 31
Peer Reviews - Benefits
Benefits: reduced rework and faster development due to a more effective evaluation
-- the peers understand the software best
-- and they will not be as much of a threat when it comes to promotions and raises
-- reciprocity -- you help them and they help you
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 32
Big Problem with Peer Reviews
People don’t want to attend others’ peer reviews because they are behind in their own schedules
Solutions?
Class discussion
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 33
Benefits of the Quality Engineering Approach
Less adversarial
Motivation and information to improve
Flexibility to change the process in response to a problem
-- you understand the problem and its cause
-- you understand the consequences of a change in the process
Knowledge is the foundation of successful quality engineering
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 34
The Cost ofQuality Engineering
Your process may be more complex, costly and time consuming - at least when you start
But you gain knowledge
-- what and where to automate
-- how and where to improve the process
And you gain higher quality
And you can begin to optimize the process, just as you optimize software
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 35
Cost of Quality Analysis
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 36
Quality (Fewer Defects; Customer
Satisfaction)
Productivity Cycle Time
CustomerValue
Quality Improvements Cost Money
How can we justify them?
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 37
Managers and Technical Staff
must be convinced that
Quality problems are serious
Poor quality costs them money
Quality is worth fixing
Quality can be fixed by proper techniques
In Order to Justify Quality Improvements ...
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 38
The Fundamental Prejudice
Quality improvement is a non-value-added activity
– It costs overhead resources
– The benefits are not necessarily real
So we seek to avoid it
Cost of Quality (COQ) Analysis is used to address this prejudice
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 39
The First Question
This is what management and technical staff will typically ask when approached with a
proposal to improve quality
This is what management and technical staff will typically ask when approached with a
proposal to improve quality
“What does it costto improve quality?”
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 40
The Right Questions
“What’s the payoff?”
“What is the return on investment for quality
improvements?
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 41
Categorizing Quality-Related Costs
Quality Related Costs
Cost of Conformance
Cost of Non-Conformance
Prevention Costs
Appraisal Costs
Internal Failure Costs
External Failure Costs
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 42
Cost of Conformance
Things that improve quality
– Prevention costs - prevent poor quality
Predictive measurements, training, root cause analysis, process improvement
– Appraisal costs - detect poor quality
Tests, inspections, audits
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 43
Cost of Non-Conformance
The price you pay when you fail to achieve quality
– Internal failures
Costs incurred before product shipment
– External failures
Costs incurred after product shipment
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 44
Effects of Maturity on Costs
SEI Maturity Level
Cost
as
a P
erc
ent
of
Develo
pm
en
t C
ost
10
20
30
40
50
60
1 2 3 4 5PreventionAppraisal
InternalFailures
ExternalFailures
Total COQ
As reported by Knox (see references)
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 45
Module Summary
Quality Control focuses on the end product
Quality Assurance focuses on the process
Quality Engineering focuses on process and people, integrating QA with the software engineering process
Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 46
References
Knox, 1993, Raytheon studies reported by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute
Meyer, Bertrand, 1997, Object Oriented Software Construction
Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview.
Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN 0-932633-22-6.