January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 1
CHAPTER 10
MEASURING AND
MONITORINGNOTICE: This material is copyrighted and may be copied
or downloaded ONCE ONLY by students who are registered in this course at Southern Methodist University or National Technological University.
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 2
Outline10.1 Overview10.2 Basic Issues - Levels of Metrics
and Attributes Measured10.3 Selecting Metrics10.4 Recommended Metrics10.5 Metrics Issues
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 3
10.1 Overview
• Where metrics fit in the larger planning, tracking and oversight picture
• Basic reasons for measurement
• Possible results - good and bad
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 4
Metrics are used to MonitorMonitoring is used to Manage
RiskManage Risks
Definethe Approach
GenerateDetailed Plans
Understandthe Need
Monitor Executiontext, chapters 6, 15
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 5
SW Project MonitoringTypical Symptoms of a Problem
We’re six months behind schedule, and
nobody knew it!
Why did it take us so long to find
out?
January
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 6
SW Project MonitoringSymptoms of Another Problem
They’ve beenworking on that module for eight weeks and everyone
else is waiting for it.
Did thedevelopers in charge
know how many people depend on
themodule?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 7
SW Project MonitoringOther Examples
• The project manager promised a new feature to the customer -- but never told any of the programmers!– “I thought you knew about this!”
• The software takes up too much disk space.– Nobody ever thought it would get that big.
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 8
SW Project Tracking and Oversight
PurposeTo provide adequate visibility into actual progress so that
management can take effective actions when the
software project’s performance deviates
significantly from the software plans
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 9
SW Tracking and Oversight Goals from the SEI CMM
1) Actual results and performance are tracked against software plans– Plans are revised to reflect actual
performance and changes in requirements or commitments
2) Corrective actions are taken and managed to closure when actual performance deviates significantly from software plans
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 10
3PM Today
SW Tracking and Oversight Goals from the SEI CMM
3) Changes to software commitments are communicated to all affected groups and individuals
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 11
SW Project Tracking & Oversight
Practices Recommended by SEI CMM• Use a software development plan for
tracking and communicating status• Track schedule, size, effort, computer
resources, technical activities and risks• Hold periodic reviews and take
corrective actions• Revise plans and schedules to reflect
changes -- using a defined procedure• Review customer commitments on a
regular basis
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 12
Things you can Estimate and Monitor
• Costs• Sizes• Quality• Reliability• Schedules• Staffing• etc.
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 13
Establish a Data Base• Know about your organization
– Performance on past projects– Lessons Learned
• Know about your industry and competitors– What is best in class?– Improvement rates
HistoricalData Base
- Metrics- Lessons- etc.
Facts to help you manage
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 14
Example of Experience vs. History
• History: for C++ doing your kind of software, you should be generating– 25 lines of code per day during the coding
phase, with– 3 errors per 1000 lines of code during
module test• Actual experience on your project:
– 40 lines of code per day, with– 0.5 errors per 1000 lines of code during
module test
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 15
Optimist’s Conclusion
Do you have a solid reason to explain this difference?
Ask questions. Why are you better?– Is the process different?– Are the people a lot better?– Are the tools better?
We are doing much better than in the past!
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 16
Pessimist’s Conclusion
Ask questions. Find out what is really happening.– Are the tests being performed?– Is the coverage adequate?– Are there higher rates of customer complaints after
shipment?
Our testing is no good (perhaps because it is being
rushed due to deadlines)
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 17
Knowing the Competition Can Give You Insights
• But what if the norm in your industry is an improvement of 25%?
• And what if your competitors have all switched to Java and are 50% more productive as a result?
Weimproved our “C” language
productivity by 15%
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 18
Every Metric should have a Purpose -- You want to get
Information
Data Analysis Information
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 19
But for Every Analysis there are Two Possible Results
Information - tells you something right– We are (or are not) on schedule– Our risks are (or are not) under control
Misinformation - tells you something wrong – We are (or are not) on schedule– Our risks are (or are not) under control
And there will always be changes in the organization when you measure it
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 20
Key Issues• Define how to interpret measurements
– To form a basis of consistent analysis• Choose consistent graphing techniques
– So people know how to interpret the data• Define how to use each measurement
– You must also demonstrate that you are using it that way, so people will believe you
– Given any measure, people will change to make it look to their advantage• you want to make their behavior change in a
positive way
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 21
Organizational Framework• There must be an Organizational
Framework for understanding the importance of metrics– i.e., people do not sabotage the
metrics collection effort– And people do not abuse metrics– And people do not draw wrong
conclusions
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 22
Achieving an Effective Organizational Framework
• Educate everyone in the proper use of measurements
• Develop the right metrics– Involve those who are being measured– Measure only what you can benefit from
• Use the measurements– To make decisions about the product
and the process– But NOT to make decisions about people
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 23
You Want to Achieve Optimal Performance
• Don’t over-measure or under-measure• Don’t over-test or under-test• Don’t over-inspect or under-inspect• etc.
• Track the things that represent your greatest risks and concerns
• Remember that it costs time and money to track - make it worthwhile
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 24
10.2 Basic Issues
• Levels of Metrics• What to Measure• Who Cares about What
Metrics
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 25
10.2.1: Levels of Metrics
• What do we mean by a metric?• Does everyone mean the
same thing?• How do we resolve
discrepancies?• Consider the example on the
next page:
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 26
Metrics may be assessed or measured in many ways!
Cost is best
measured in staff days!
Why not
dollars?
I measure it by
drop in stock
price.
However they measure it, they had better figure out how to reduce cost!
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 27
Metrics, Measures and Data
Process in Execution
Data Data Data DataData
Metric
Measure Measure Measure
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 28
Example
Process in Execution
UnitsProduced
Head-count
Lines of Code $ SpentMonths
Productivity
Units Per Month LOC perStaff Month $ per Line of Code
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 29
Definitions: Data
Data (or Primitive Measures)The fundamental, factual quantities that
characterize a process or product
– Specific Facts that are countable or otherwise obtainable
– Minimum amount of Analysis
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 30
Definitions: Data
A Key Issue is Consistent Definitions Another Key Issue is Accuracy of
CollectionExamples:
– Hours worked– Number of employees– Lines of code
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 31
Data(or Primitive Measures)Factual Information with Minimal
Interpretation -- Examples: days, lines of code,
people, dollars, defects, variance, tests completed, complaints
-- You must define what you want -- How many hours per day? -- Which lines to include in LOC?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 32
Data(or Primitive Measures)Factual Information with Minimal
Interpretation -- Can you collect accurate and consistent
data?-- Can you collect it efficiently?-- How will the organization and the
process change when you collect this data?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 33
How Long is a Staff Day?
Salaried Staff
Staff Day
OvertimeRegular Time
Hourly StaffSalaried Staff Hourly Staff
Unpaid
PaidUnpaid
Paid
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 34
Definitions: MeasuresMeasure (sometimes called Compound
Measure)The result of counting or otherwise
quantifying an attribute of a process or product
– How we quantify – Something we can analyze and interpret
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 35
Definitions: MeasuresA Key Issue is Proper Interpretation
How Measures are Graphed is Also Important
Examples:– lines of code per hour– ratio of current to historical test
performance– turnover rate
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 36
Measures(or Compound Measures)
Measures are values computed from data or other measures
-- Generally associated with specific graphs and rules of interpretation
-- Examples: lines of code per day; defects per thousand lines of code; return on investment
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 37
Measures(or Compound Measures)
• You must define how you will use it• You must also demonstrate that you
are using it that way, so people will believe you
• Given any measure, people will change to make it look to their advantage -- so you want to make their behavior change in a positive way
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 38
A Typical Graphof a Measure
Modules Tested
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10 11 12
HistoricalPlanActual
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 39
Example: A Measure & Its Impact
Metric: ProductivityMeasure 1: Lines of code per dayUse: reward those who produce the most lines of code per dayResult: people may produce bloated, inflated code in order to look goodMeasure 2: requirements met and tested, weighted by complexity of requirementUse: track against history and use to identify
process bottlenecksResult: people may use the data to make the process more efficient, resulting in lower cost
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 40
Definitions: MetricsMetric
A measure or combination of measures that provides insight into a software
issue or concept.
– What we Measure– An Indicator – A Standard of Measurement that ties to a
Goal or Purpose
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 41
Definitions: MetricsA key issue is selecting a measure
that provides insight or information r.e. the goal or
purposeExamples:
– Comparisons (planned vs. actual)– Indices (performance ratio,
productivity, return on investment)
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 42
MetricsGroups or Categories or Systems of
MeasuresFocus on the Goals
-- Examples: productivity, quality, cycle time, defect density, customer satisfaction
-- Each metric is usually associated with some higher level objective, such as “reducing cost” or “on time delivery”
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 43
MetricsGroups or Categories or Systems of
MeasuresFocus on the Goals
-- There are many ways to measure a metric and different organizations measure differently
-- The key is to spend time to select the right measure, so you achieve the higher level objective(s)
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 44
Good and Not So Good Measures
Goal: Produce software more efficientlyMetric: Efficiency
Measure 1: tests completed per weekResult: easy tests done first; corners cut in testing; hard problems ignored or deferred
Measure 2: reworkResult: process and methods are improved to reduce rework, resulting in more efficient software development
But rework is a lagging indicator - it does not spot problems in advance
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 45
10.2.2 What Should we Measure?
Product ProjectProcess determines
success of
determines
quality of
root causesroot causes
Process Metrics–Effectiveness of the process–How well are we following the process?–Risk monitoring
Product Metrics–Performance and quality–How well is the product meeting its requirements?
Project Metrics–The state of the project–How are we doing relative to cost, schedule, staffing, etc.?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 46
Product ProjectProcessAttributes
What
Resources
Quality
Time Are We On Schedule?Expenses
vs. Budget?
How Fast can we
Manufacture?
What Is our Cycle Time?
Post-release
Defects?
What will it Cost?
What is our Productivity
?
Customer Satisfaction
?
In-process Defects?
Performance
Meets Perf. Goals?
Meets Mgt. Goals?
Does it Work?
What Attributes can we Measure?
• We want attributes that relate to our goals – time, resources, performance, quality etc.
• The following type of matrix can help:
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 47
10.2.3: Who Cares about What?• Managers usually care about project
metrics - that’s what they are measured by
-- But if the project is in trouble they need to know more
• Developers usually care about product metrics
-- that’s what they are measured by• Both should care about process metrics -- this is usually where you learn the
reasons why a project or a product is in trouble
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 48
10.3 Selecting Metrics• Goal/Question/Metric• Mapping to the Process• Tying to Risks
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 49
Selecting Metrics: The Goal-Question-Metric Paradigm (1)
GOAL(what do we want to accomplish)
QUESTIONS(the answers tell us whether we are meeting the goals
or help us understand how to meet them)
METRICS(measurements that answer the questions)
(1) Victor Basili & David Weiss, “A Methodology for Collecting Valid Software Engineering Data,” IEEE Transactions on Software Engineering, Vol SE-10, no. 6, Nov., 1984, p728.
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 50
Steps of the GQM Paradigm1 Determine your Goals -- and
Prioritize Them– Don’t Select Too Many– Make Sure they are your Real Goals
2 For each Goal, Determine What Questions would Help you Understand Whether you are Achieving the Goal or How to Achieve it
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 51
Steps of the GQM Paradigm3 For each Question, Determine if it
can be Answered by Measurement– If so, determine what can be
measured– And what it would cost to measure– And the side effects of the
measurement4 Select the Metrics with the Best
Payoff
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 52
Remember that Measures are Not Goals
• If the measure becomes the goal, you usually end up at the wrong place (example: lines of code per day)
• The measure tells you how you are doing, but there may be other measures that tell you other aspects of how you are doing
You Get What You Measure!
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 53
Prioritize Goals• You cannot always measure everything• Determine which goals are most important• Also make sure that the goals are consistent
Are people getting consistent direction?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 54
It Helps to Have a VisionVision
Key Features or Goals
Prioritized List of Goals
Focus Metrics on Top Priority Goals
1. Learn to Sail2. Select the Best Beer3. Save $
1. Save $2. Learn to Sail3. ....
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 55
Forms of Payoff for MetricsI.e., Which Metrics to Use
• Low Cost of Collection and Storage• Minimal Impact of Collection (morale,
disruption, etc.)• High Information Communicated• One Metric answers Multiple Questions• One Metric supports Multiple Goals
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 56
Look for Questions and Metrics
that Do Dual DutyGoal
1Goal
2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 57
Other Attributes of a Good Set of Metrics
• Completeness - Addresses All Issues of Concern
• Value is High relative to Cost of Collection, Storage and Analysis
• There is an Organizational Framework for Understanding the Importance of Metrics– I.e., people do not sabotage the metrics
collection effort– And people do not draw wrong conclusions
• All Parts of the Organization Benefit• Alternative Metrics are Available if Primary
Choices are Not Available
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 58
Example: GQM Approach• Goal:
Improve Efficiency by Reducing the Cost of Rework
• Question 1: What does rework cost us?
• Question 2: How many defects are in the software when we release it?
• Question 3: What are the sources of defects and rework?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 59
Possible Metrics for Question 2How Many Defects are In the Software when
we Release It?• Number of Defects in Product -- this is impossible to know -- but we can approximate it with other metrics• Number of Known Defects at Release --
incomplete, but perhaps correlated to the total• Number of Defects Detected After Release• Total Defects over the Life of the Product
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 60
Analyzing Defect Data
Release
12 Months
Retirement
TotalDefect
sKnown at Release
Known after 12 Months
Total Defects in Product Life
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 61
Tie Metrics to the Process -- A Measure for Every Phase
PhaseMetric
Requiremen
tsTestPla
nning
Design
Integratio
nCodin
gStaffing
X
X
X X X
XRequirements
Stability
X XX
Etc.
X
Design Complexi
tyX
Code Complexi
ty
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 62
Risk: “Running out of memory space”
Possible Measures:– Predicted memory space used
(update each week or month, depending on risk)
– Size of code produced by compiler– Level of experience of average staff
Selecting Measures• Any risk may result in measures for monitoring • Choice depends on anticipated causes of risk
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 63
Another Example Risk Measure
Risk: “Not meeting schedule”Cause: Not enough experienced staffPossible Measures:
• Staffing level• Unplanned turnover • Average staff experience, etc.
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 64
10.4 Recommended Metrics• Problem Reports• Customer Satisfaction• Requirements Stability• Rate Charts• Earned Value• Resource Use• Defects• Rework
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 65
Problem ReportsComplaints Help you Improve
Reports should be categorized by such things as: -- Kind of problem -- Severity -- Source of the problem (process step) -- Effort required to correct -- When discoveredThis can tell you where you need to focus
improvement effortsProblem reports are a direct measure of customer
satisfaction
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 66
Problem Report Analysis
0
10
20
30
40
50
60
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
SeriousNormalMinor
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 67
Some Issues with Measuring Problem
Reports• How to handle duplicates• How to handle different symptoms
caused by the same root problem• How to handle reports that are
the result of multiple root problems
• Urgency vs cost vs other factors in deciding on priority
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 68
Customer Satisfaction-- as measured by the
customer• Surveys• Degree of cooperation• Do they buy your product?• etc.
Business survival depends on satisfying the customer better than the competition
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 69
Requirements StabilityTBDs + New + Changes
Total Requirements
00.10.20.30.40.50.6
Start Review 1 Review 2 Review 3 Review 4
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 70
Requirements StabilityAnother Way to Display
01020304050607080
Start Review 1 Review 2 Review 3 Review 4
TotalChangesTBDs
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 71
Rate Charts• These tell you the rate at which
something is happening - usually compared with some plan
• What you measure is something discrete that represents genuine progress toward the objectives of the project
• If you measure well, rate charts can give you a better handle on where you are than something like “hours spent”
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 72
Rate Charts ExampleCompletion vs. history and plans
Requirements Fully Tested
05
1015202530
0 1 2 3 4 5 6 7 8 9 10 11 12
HistoricalPlanActual
Today Deadline
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 73
Staffing: actual vs. plan
02468
10121416
J an Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec
PlanActual
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 74
Staffing: actual vs. plan
02468
1012141618
J an Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec
PlanActualProjection
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 75
Staffing: actual vs. plan
02468
1012141618
J an Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec
PlanActualProjectionBehind
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 76
Rate Charts can be used for Many Things
• Examples: -- Modules
designed/coded/tested/integrated -- Requirements
designed/coded/tested/integrated -- Problems corrected -- High priority complaints resolved
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 77
Characteristics of Applications for which Rate Charts are a Good Choice• Discrete Events• Easily Measured• Clear Definition of Completion• Many Occurrences
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 78
Earned ValueQuestions: “How much of the work have we actually
done?” “Are we likely to meet the deadline?”Metric: “Amount of work actually performed, and
projection for duration of project.”Measure: Earned Value = % of total work performed
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 79
Earned Value Uses These Data
• ACWP (Actual Cost of Work Performed)– How much money has been spent?
• BCWP (Budgeted Cost of Work Performed)– How much money should have been spent
for the work actually performed?• BCWS (Budgeted Cost of Work Scheduled)
– How much work should have been done by now according to the schedule?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 80
Units• “Money” and “work” can be
measured in dollars, effort, or other units
• Effort is usually the easiest way to measure
• But dollars can be adjusted to include overhead costs and non-labor costs, so it may be more accurate
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 81
Questions You Can Answer with Earned Value
• Are we getting the work done on schedule?
• Are we overspending?• Will we overrun or under-run the
budget?• What is a realistic end date?• What performance level is required to
meet the budget or the deadline?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 82
Earned Value Provides Insight Early in the Project
• You can tell if there are problems after a few weeks or months
• You have time to make adjustments• You can communicate the problems
and their scope using quantitative methods rather than guesswork
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 83
Rate Charts Can Display Earned Value in a Visual
Manner• Rate charts give a visual picture of
how much “value” you have “earned”• Consider some examples ...
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 84
Requirements Implemented
0
20
40
60
80
100
120
Total RqmtsPlanActual
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 85
A Rate Chart forIterative Lifecycles
01020304050607080
1 2 3 4 5 6 7 8
Rqmts DesignedRqmts CodedRqmts Tested
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 86
Earned Value Metrics Quantify Progress and Help
Predict and Manage
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 87
Top Level View ofEarned Value Metric
EarnedValue
•$ Spent vs $ Planned
•$ Spent vs Thresholds of what should have been spent
Budget
“Are we overspending?
”
WBSor
Requirements“Are we
getting the work done?”
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 88
A Simple Way to do Earned Value
• This method avoids a lot of the jargon and just focuses on the substance
• If you understand this method you will find it easier to understand the definitions of earned value metrics found in other places
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 89
Develop a Micro Schedule for the Next Part of the Project
Top Level Schedule
Micro Schedule for the Next Phase
Micro Schedule is the locally-
managed schedule,
defined by those doing the work
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 90
What Is a Micro Schedule• A schedule of small tasks whose
duration can be measured in days ora few weeks
• This represents the work tasks assigned to individuals
• Have the individuals develop their micro schedule
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 91
Earned Value Micro ScheduleBasic Task Estimates
Task Effort Planned Complete Date Resp.(days) (week #)
Set Up 3 1 JoeGet Specs 2 2 MaryDesign Output 10 5 Pete & JoePlan Tests 3 6 JoeCode 5 7 MaryUnit Test 3 8 JoeIntegrate 2 9 MaryBeta Test 3 10 PeteTotal 31
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 92
Convert to Running Total of Effort Planned
Week Total Effort Planned to be Complete1 32 53 7 (*)4 11 (*)5 156 187 238 269 28
10 31
(*) Assumes partial
progress on “design
output” task
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 93
The Total Planned Effort is the Budgeted Cost of Work
ScheduledWeek Total Effort Planned to be Complete
1 32 53 74 115 156 187 238 269 28
10 31
BCWS
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 94
Earned Value Plan Graph (BCWS)
Some systems estimate BCWS by drawing a straight line from 0 at start to total project budget
at end.
05
101520253035
1 2 3 4 5 6 7 8 9 10
TotalPlan
BCWS
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 95
Budget• Your actual budget should match
this plan• If the actual budget does not
match, you can negotiate what to do before beginning the work– Reduce plan to match available budget– Increase budget to match plan– Somewhere in between– Renegotiate plan with developers
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 96
Task Effort (days) % Complete EarnedSet Up 3 100 3Get Specs 2 50 1Design Output 10 25 2.5Plan Tests 3 0 0Code 5 0 0Unit Test 3 0 0Integrate 2 0 0Beta Test 3 0 0Total 31 6.5
BCWP
Earned Value Datafor a Given Week
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 97
Earned Value Procedure• Each week you compute a new “%
complete” value for each task– Perhaps allow only 0 or 100%– Or perhaps allow 50% or 33% or 25%,
but you should avoid smaller increments• Then compute the total work
performed so far• Plot the total each week in the plan
graph, representing an “actual” line
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 98
05
101520253035
1 2 3 4 5 6 7 8 9 10
TotalPlanActual
Earned ValueTypical Graph
BCWP
BCWS
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 99
Earned Value Applications• You can spot deviations from plan
and project completion dates• You can also plan overtime or
other contingency activities• You can show your management
that you know where you are and are not suffering from the “always 90% done” syndrome
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 100
Earned Value Advantages• By using only small tasks, it is easier to
tell what is actually complete• Developers can use the data to
manage their work - it is their plan– The data gives them insight on what they
are accomplishing• The detailed work sequence does not
matter. All completed work is “earned” – Developers can make the right decisions
about work sequencing
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 101
ACWPThe Third Number
• Actual Cost of Work Performed is what you have actually spent
• Whereas BCWP is what you have “earned” or accomplished
BCWP = Work Completed
(earned value)
ACWP = Money Spent
(or effort spent)
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 102
Typical Graph Showing Work Behind but On Budget
05
101520253035
1 2 3 4 5 6 7 8 9 10
TotalPlanActual$ Spent
BCWS
BCWP
ACWP
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 103
Typical Graph Showing Work Behind but On Budget
This project is probably understaffed.
05
101520253035
1 2 3 4 5 6 7 8 9 10
TotalPlanActual$ Spent
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 104
Typical Graph Showing Work Behind and Over
Budget
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7 8 9 10
TotalPlanActual$ Spent
This project is not as productive as planned.
ACWP
Slide # 105
Summary of the Three Numbers
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 106
BCWS(Budgeted Cost of Work
Scheduled)• The amount of effort you had
planned to expend (i.e., the amount you budgeted)
• “How much work should have been done”
05
101520253035
1 2 3 4 5 6 7 8 9 10
TotalPlanActual$ Spent
BCWS
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 107
BCWP(Budgeted Cost of Work
Performed)• The value produced for the effort
you have actually expended• “How much work was actually
done”
05
101520253035
1 2 3 4 5 6 7 8 9 10
TotalPlanActual$ Spent
BCWP
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 108
ACWP(Actual Cost of Work Performed)
• The amount of money you have spent so far
• “What did your work actually cost?”
05
101520253035
1 2 3 4 5 6 7 8 9 10
TotalPlanActual$ SpentACWP
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 109
What Can We Tell About the Schedule
From These Three Numbers?SV - Schedule Variance
SV = BCWP - BCWS
SPI - Schedule Performance IndexSPI = BCWP / BCWS
Negative means behind schedule
Less than 1 means behind schedule
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 110
SPI Indicates Whether You are Ahead of or Behind
Schedule
0.5
0.7
0.9
1.1
1.3
1.5
J F M A M J J A S O N D
SPI Target
Ahead of Schedule
Behind Schedule
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 111
What Can We Tell About the Cost
From These Three Numbers?CV - Cost Variance
CV = BCWP - ACWP
CPI - Cost Performance Index CPI = BCWP / ACWP
Negative means over budget
Less than 1 means over budget
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 112
CPI Indicates Whether You are Under or Over Budget
0.5
0.7
0.9
1.1
1.3
1.5
J F M A M J J A S O N D
CPI Target
Under Budget
Over Budget
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 113
Two Useful NumbersBAC - Budget at Completion
– The total budget for the project– BCWS at project completion
SCHED - Schedule for the project– The total number of weeks or months
or whatever
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 114
Estimated Budget At Completion
EAC - Estimate at Completion– The estimated actual cost of project
EAC = BAC / CPI
VAC - Variance at CompletionVAC = BAC - EAC
Negative means over budget by this much
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 115
Schedule At CompletionSAC - Schedule at Completion
– The estimated actual schedule of projectSAC = SCHED / SPI
SVAC - Variance at CompletionSVAC = SCHED - SAC
Negative means behind schedule by this much
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 116
Thus You Can Tell ...• How far behind or ahead of schedule
you are• How far over or under budget you are• What your eventual schedule and cost
are likely to be
You also have an early warning of trouble that you can use to try to
fix the underlying problems
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 117
A Handy Graph of SPI and CPIIndicates Your Overall Status
Behind Schedulebut Under Budget(shortage of staff )
Ahead of Scheduleand Under Budget
Behind Scheduleand Over Budget
Ahead of Schedulebut Over Budget(staff surplus?)
SPI
CPI
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 118
Reasons for Being Behind Schedule
• If you are also Under Budget– There is not enough work being done
• Not enough people• People being used for other tasks
• If you are at or Over Budget– There is a productivity problem
• Underestimation of the effort• Excessive “non-value-added” work
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 119
Resource Use Metrics• Staffing on project• Staff-hours worked• Memory requirements for product• CPU capacity required• Amount of reused code
You can predict beforehand and then measure actuals
When actuals deviate from plan, you know that you need to act
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 120
Example of Memory Requirements Metric
Memory Use
020406080
100120140
1 2 3 4 5 6 7 8 9 10
Total Available50% MarginProjected
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 121
Example of Reuse Measurement
Code Size Analysis
010203040506070
Start Milestone 1 Milestone 2 Milestone 3 Milestone 4 Milestone 5
TotalNewReused
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 122
The Estimation Spreadsheet (Assignments
4/5) Provides Initial Plan Data• Staffing levels
• Headcount• Dollars spent• Size in Lines of Code• Size in Memory• Reuse• etc.
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 123
The Estimation Spreadsheet (Assignments
4/5) Provides Initial Plan DataPeriodic updates can be
represented by additional rows and/or columns
Actuals and history can also be represented by additional rows and/or columns
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 124
Measuring Defects• Defects are one of the most valuable
things you can measure• They indicate fundamental process
problems that are causing you to waste time and money and produce lower quality products
• But people resist measuring defects because they represent “bad news”
• It takes a cultural change, sometimes, to get defect measurement viewed as a positive thing
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 125
Kinds of Defects to MeasureAny product or byproduct can have
defects.• Requirements analysis defects - collect at
inspections and reviews• Design defects - collect at design
walkthroughs, inspections, reviews• Coding defects - collect during peer
reviews and tests• Testing defects (defects in testing
procedures)• etc.
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 126
Rework• Rework is work you do over because the
first time was wrong, unneeded, solved the wrong problem, etc.
• Every process has some rework, but rework adds cost and time -- and it reflects quality problems
• Most organizations resist measuring rework because it makes them look bad
• But rework is an excellent metric for evaluating process health and effectiveness
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 127
Rework Loops
ProcessStep
Undetected
DefectSeveral Steps
ProcessStep
Defect Detected
Rework costs money and time
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 128
Measuring ReworkSimple Method: count problems• Count problem reports and estimate
effort to fix each problem• The sum of all the effort is an estimate of
total rework• Compare with total effort on project
This only works if you identify problems, document them as problems, and
estimate the work required to fix themMore Advanced Method: phase
containment (see next slide)
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 129
In-Phase vs. Out-of-Phase Defects
Defect Containment
Chart
Integrate
Test
Code
Design
Rqmts
Input6
42
22
3
2
12
1
22 113
14
16
3 82
23
18
5
14
12
Rqmts
Design
IntegrateTestCod
e
Phase where Defect was Inserted
PhaseDetected
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 130
Cost of Rework• In-phase defects are often not
counted as rework -- they can be viewed as a part of the process (more on this in a later course)
• Out-of-phase defects cost a lot more to fix, in general, and that effort is usually classified as rework
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 131
10.5 Metrics Issues• Frequency• Synchronization• Updating estimates• Margin• Peak vs. average• Units• Interplay of resources
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 132
FrequencyHow Often to Measure
• Too often results in high cost, disruption of process
• Too seldom results in failure to see problems soon enough
• Frequency should depend on degree of risk and cost of measurement
05
101520
1/yr 1/month 1/week 1/day 1/hour
EffectivenessCost
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 133
SynchronizationGetting the True Picture
Delay due to System Load
Hour
020406080
100120
8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 134
UpdateProjections and Estimates
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
ActualForecast
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 135
MarginA Frequently Misunderstood
TermWhat does “leave 50% margin” mean?
Code
Margin66 2/3
50
100
100
Code
Margin
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 136
Peak vs. Average“Software must be able to handle
100 screens per minute”
Peak? (Worst Case)
Average over Some Interval?
How Often to Measure?
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 137
UnitsOften Subject to Misinterpretation
• Memory Size: words, bytes, bits???• Program Execution Time:
– Hours since shipped? {was it delivered?}– Hours since received by customer? {was it installed?}– Hours since installed? {was it used?}– CPU execution time since installed?
• CPU Time: cycles, instructions, MIPS, FLOPS, ?
Beware of innocent misinterpretations
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 138
Interplay of ResourcesFew Things Happen in
IsolationSometimes, system resources interact with each other, giving misleading
evidence of performance
Memory
Capacity:80 cycles
per micro-second
CPU50 MIPS
I/O Channel20 MIPS
Display40 MIPS
CPU may run at only 20 MIPS because of
memory saturation
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 139
Summary1) Know WHAT you are measuring - attributes,
etc.2) Know WHY you are measuring - goals, risks,
etc.3) Select wisely
– Complete - something for each part of the process
– Reasonable cost of collection– High information content– Minimal organizational disruption
4) Beware of incorrect interpretations5) Beware of misuse
“Nobody believes statistics anyway.”
Adams, The Dilbert Principle
January 20, 2000 CSE 7315 - SW Project Management /
Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,
All Rights Reserved
Slide # 140
End of CHAPTER
10