january 20, 2000 cse 7315 - sw project management / chapter 10 – measuring and monitoring...

January 20, 2000 CSE 7315 - SW Project Management /

Chapter 10 – Measuring and MonitoringCopyright © 1995-2000, Dennis J. Frailey,

All Rights Reserved

Slide # 1

CHAPTER 10

MEASURING AND

MONITORINGNOTICE: This material is copyrighted and may be copied

or downloaded ONCE ONLY by students who are registered in this course at Southern Methodist University or National Technological University.



All Rights Reserved

Slide # 2

Outline10.1 Overview10.2 Basic Issues - Levels of Metrics

and Attributes Measured10.3 Selecting Metrics10.4 Recommended Metrics10.5 Metrics Issues



All Rights Reserved

Slide # 3

10.1 Overview

• Where metrics fit in the larger planning, tracking and oversight picture

• Basic reasons for measurement

• Possible results - good and bad



All Rights Reserved

Slide # 4

Metrics are used to MonitorMonitoring is used to Manage

RiskManage Risks

Definethe Approach

GenerateDetailed Plans

Understandthe Need

Monitor Executiontext, chapters 6, 15



All Rights Reserved

Slide # 5

SW Project MonitoringTypical Symptoms of a Problem

We’re six months behind schedule, and

nobody knew it!

Why did it take us so long to find

out?

January



All Rights Reserved

Slide # 6

SW Project MonitoringSymptoms of Another Problem

They’ve beenworking on that module for eight weeks and everyone

else is waiting for it.

Did thedevelopers in charge

know how many people depend on

themodule?



All Rights Reserved

Slide # 7

SW Project MonitoringOther Examples

• The project manager promised a new feature to the customer -- but never told any of the programmers!– “I thought you knew about this!”

• The software takes up too much disk space.– Nobody ever thought it would get that big.



All Rights Reserved

Slide # 8

SW Project Tracking and Oversight

PurposeTo provide adequate visibility into actual progress so that

management can take effective actions when the

software project’s performance deviates

significantly from the software plans



All Rights Reserved

Slide # 9

SW Tracking and Oversight Goals from the SEI CMM

1) Actual results and performance are tracked against software plans– Plans are revised to reflect actual

performance and changes in requirements or commitments

2) Corrective actions are taken and managed to closure when actual performance deviates significantly from software plans



All Rights Reserved

Slide # 10

3PM Today

SW Tracking and Oversight Goals from the SEI CMM

3) Changes to software commitments are communicated to all affected groups and individuals



All Rights Reserved

Slide # 11

SW Project Tracking & Oversight

Practices Recommended by SEI CMM• Use a software development plan for

tracking and communicating status• Track schedule, size, effort, computer

resources, technical activities and risks• Hold periodic reviews and take

corrective actions• Revise plans and schedules to reflect

changes -- using a defined procedure• Review customer commitments on a

regular basis



All Rights Reserved

Slide # 12

Things you can Estimate and Monitor

• Costs• Sizes• Quality• Reliability• Schedules• Staffing• etc.



All Rights Reserved

Slide # 13

Establish a Data Base• Know about your organization

– Performance on past projects– Lessons Learned

• Know about your industry and competitors– What is best in class?– Improvement rates

HistoricalData Base

- Metrics- Lessons- etc.

Facts to help you manage



All Rights Reserved

Slide # 14

Example of Experience vs. History

• History: for C++ doing your kind of software, you should be generating– 25 lines of code per day during the coding

phase, with– 3 errors per 1000 lines of code during

module test• Actual experience on your project:

– 40 lines of code per day, with– 0.5 errors per 1000 lines of code during

module test



All Rights Reserved

Slide # 15

Optimist’s Conclusion

Do you have a solid reason to explain this difference?

Ask questions. Why are you better?– Is the process different?– Are the people a lot better?– Are the tools better?

We are doing much better than in the past!



All Rights Reserved

Slide # 16

Pessimist’s Conclusion

Ask questions. Find out what is really happening.– Are the tests being performed?– Is the coverage adequate?– Are there higher rates of customer complaints after

shipment?

Our testing is no good (perhaps because it is being

rushed due to deadlines)



All Rights Reserved

Slide # 17

Knowing the Competition Can Give You Insights

• But what if the norm in your industry is an improvement of 25%?

• And what if your competitors have all switched to Java and are 50% more productive as a result?

Weimproved our “C” language

productivity by 15%



All Rights Reserved

Slide # 18

Every Metric should have a Purpose -- You want to get

Information

Data Analysis Information



All Rights Reserved

Slide # 19

But for Every Analysis there are Two Possible Results

Information - tells you something right– We are (or are not) on schedule– Our risks are (or are not) under control

Misinformation - tells you something wrong – We are (or are not) on schedule– Our risks are (or are not) under control

And there will always be changes in the organization when you measure it



All Rights Reserved

Slide # 20

Key Issues• Define how to interpret measurements

– To form a basis of consistent analysis• Choose consistent graphing techniques

– So people know how to interpret the data• Define how to use each measurement

– You must also demonstrate that you are using it that way, so people will believe you

– Given any measure, people will change to make it look to their advantage• you want to make their behavior change in a

positive way



All Rights Reserved

Slide # 21

Organizational Framework• There must be an Organizational

Framework for understanding the importance of metrics– i.e., people do not sabotage the

metrics collection effort– And people do not abuse metrics– And people do not draw wrong

conclusions



All Rights Reserved

Slide # 22

Achieving an Effective Organizational Framework

• Educate everyone in the proper use of measurements

• Develop the right metrics– Involve those who are being measured– Measure only what you can benefit from

• Use the measurements– To make decisions about the product

and the process– But NOT to make decisions about people



All Rights Reserved

Slide # 23

You Want to Achieve Optimal Performance

• Don’t over-measure or under-measure• Don’t over-test or under-test• Don’t over-inspect or under-inspect• etc.

• Track the things that represent your greatest risks and concerns

• Remember that it costs time and money to track - make it worthwhile



All Rights Reserved

Slide # 24

10.2 Basic Issues

• Levels of Metrics• What to Measure• Who Cares about What

Metrics



All Rights Reserved

Slide # 25

10.2.1: Levels of Metrics

• What do we mean by a metric?• Does everyone mean the

same thing?• How do we resolve

discrepancies?• Consider the example on the

next page:



All Rights Reserved

Slide # 26

Metrics may be assessed or measured in many ways!

Cost is best

measured in staff days!

Why not

dollars?

I measure it by

drop in stock

price.

However they measure it, they had better figure out how to reduce cost!



All Rights Reserved

Slide # 27

Metrics, Measures and Data

Process in Execution

Data Data Data DataData

Metric

Measure Measure Measure



All Rights Reserved

Slide # 28

Example

Process in Execution

UnitsProduced

Head-count

Lines of Code $ SpentMonths

Productivity

Units Per Month LOC perStaff Month $ per Line of Code



All Rights Reserved

Slide # 29

Definitions: Data

Data (or Primitive Measures)The fundamental, factual quantities that

characterize a process or product

– Specific Facts that are countable or otherwise obtainable

– Minimum amount of Analysis



All Rights Reserved

Slide # 30

Definitions: Data

A Key Issue is Consistent Definitions Another Key Issue is Accuracy of

CollectionExamples:

– Hours worked– Number of employees– Lines of code



All Rights Reserved

Slide # 31

Data(or Primitive Measures)Factual Information with Minimal

Interpretation -- Examples: days, lines of code,

people, dollars, defects, variance, tests completed, complaints

-- You must define what you want -- How many hours per day? -- Which lines to include in LOC?



All Rights Reserved

Slide # 32

Data(or Primitive Measures)Factual Information with Minimal

Interpretation -- Can you collect accurate and consistent

data?-- Can you collect it efficiently?-- How will the organization and the

process change when you collect this data?



All Rights Reserved

Slide # 33

How Long is a Staff Day?

Salaried Staff

Staff Day

OvertimeRegular Time

Hourly StaffSalaried Staff Hourly Staff

Unpaid

PaidUnpaid

Paid



All Rights Reserved

Slide # 34

Definitions: MeasuresMeasure (sometimes called Compound

Measure)The result of counting or otherwise

quantifying an attribute of a process or product

– How we quantify – Something we can analyze and interpret



All Rights Reserved

Slide # 35

Definitions: MeasuresA Key Issue is Proper Interpretation

How Measures are Graphed is Also Important

Examples:– lines of code per hour– ratio of current to historical test

performance– turnover rate



All Rights Reserved

Slide # 36

Measures(or Compound Measures)

Measures are values computed from data or other measures

-- Generally associated with specific graphs and rules of interpretation

-- Examples: lines of code per day; defects per thousand lines of code; return on investment



All Rights Reserved

Slide # 37

Measures(or Compound Measures)

• You must define how you will use it• You must also demonstrate that you

are using it that way, so people will believe you

• Given any measure, people will change to make it look to their advantage -- so you want to make their behavior change in a positive way



All Rights Reserved

Slide # 38

A Typical Graphof a Measure

Modules Tested

0

5

10

15

20

25

30

0 1 2 3 4 5 6 7 8 9 10 11 12

HistoricalPlanActual



All Rights Reserved

Slide # 39

Example: A Measure & Its Impact

Metric: ProductivityMeasure 1: Lines of code per dayUse: reward those who produce the most lines of code per dayResult: people may produce bloated, inflated code in order to look goodMeasure 2: requirements met and tested, weighted by complexity of requirementUse: track against history and use to identify

process bottlenecksResult: people may use the data to make the process more efficient, resulting in lower cost



All Rights Reserved

Slide # 40

Definitions: MetricsMetric

A measure or combination of measures that provides insight into a software

issue or concept.

– What we Measure– An Indicator – A Standard of Measurement that ties to a

Goal or Purpose



All Rights Reserved

Slide # 41

Definitions: MetricsA key issue is selecting a measure

that provides insight or information r.e. the goal or

purposeExamples:

– Comparisons (planned vs. actual)– Indices (performance ratio,

productivity, return on investment)



All Rights Reserved

Slide # 42

MetricsGroups or Categories or Systems of

MeasuresFocus on the Goals

-- Examples: productivity, quality, cycle time, defect density, customer satisfaction

-- Each metric is usually associated with some higher level objective, such as “reducing cost” or “on time delivery”



All Rights Reserved

Slide # 43

MetricsGroups or Categories or Systems of

MeasuresFocus on the Goals

-- There are many ways to measure a metric and different organizations measure differently

-- The key is to spend time to select the right measure, so you achieve the higher level objective(s)



All Rights Reserved

Slide # 44

Good and Not So Good Measures

Goal: Produce software more efficientlyMetric: Efficiency

Measure 1: tests completed per weekResult: easy tests done first; corners cut in testing; hard problems ignored or deferred

Measure 2: reworkResult: process and methods are improved to reduce rework, resulting in more efficient software development

But rework is a lagging indicator - it does not spot problems in advance



All Rights Reserved

Slide # 45

10.2.2 What Should we Measure?

Product ProjectProcess determines

success of

determines

quality of

root causesroot causes

Process Metrics–Effectiveness of the process–How well are we following the process?–Risk monitoring

Product Metrics–Performance and quality–How well is the product meeting its requirements?

Project Metrics–The state of the project–How are we doing relative to cost, schedule, staffing, etc.?



All Rights Reserved

Slide # 46

Product ProjectProcessAttributes

What

Resources

Quality

Time Are We On Schedule?Expenses

vs. Budget?

How Fast can we

Manufacture?

What Is our Cycle Time?

Post-release

Defects?

What will it Cost?

What is our Productivity

?

Customer Satisfaction

?

In-process Defects?

Performance

Meets Perf. Goals?

Meets Mgt. Goals?

Does it Work?

What Attributes can we Measure?

• We want attributes that relate to our goals – time, resources, performance, quality etc.

• The following type of matrix can help:



All Rights Reserved

Slide # 47

10.2.3: Who Cares about What?• Managers usually care about project

metrics - that’s what they are measured by

-- But if the project is in trouble they need to know more

• Developers usually care about product metrics

-- that’s what they are measured by• Both should care about process metrics -- this is usually where you learn the

reasons why a project or a product is in trouble



All Rights Reserved

Slide # 48

10.3 Selecting Metrics• Goal/Question/Metric• Mapping to the Process• Tying to Risks



All Rights Reserved

Slide # 49

Selecting Metrics: The Goal-Question-Metric Paradigm (1)

GOAL(what do we want to accomplish)

QUESTIONS(the answers tell us whether we are meeting the goals

or help us understand how to meet them)

METRICS(measurements that answer the questions)

(1) Victor Basili & David Weiss, “A Methodology for Collecting Valid Software Engineering Data,” IEEE Transactions on Software Engineering, Vol SE-10, no. 6, Nov., 1984, p728.



All Rights Reserved

Slide # 50

Steps of the GQM Paradigm1 Determine your Goals -- and

Prioritize Them– Don’t Select Too Many– Make Sure they are your Real Goals

2 For each Goal, Determine What Questions would Help you Understand Whether you are Achieving the Goal or How to Achieve it



All Rights Reserved

Slide # 51

Steps of the GQM Paradigm3 For each Question, Determine if it

can be Answered by Measurement– If so, determine what can be

measured– And what it would cost to measure– And the side effects of the

measurement4 Select the Metrics with the Best

Payoff



All Rights Reserved

Slide # 52

Remember that Measures are Not Goals

• If the measure becomes the goal, you usually end up at the wrong place (example: lines of code per day)

• The measure tells you how you are doing, but there may be other measures that tell you other aspects of how you are doing

You Get What You Measure!



All Rights Reserved

Slide # 53

Prioritize Goals• You cannot always measure everything• Determine which goals are most important• Also make sure that the goals are consistent

Are people getting consistent direction?



All Rights Reserved

Slide # 54

It Helps to Have a VisionVision

Key Features or Goals

Prioritized List of Goals

Focus Metrics on Top Priority Goals

1. Learn to Sail2. Select the Best Beer3. Save $

1. Save $2. Learn to Sail3. ....



All Rights Reserved

Slide # 55

Forms of Payoff for MetricsI.e., Which Metrics to Use

• Low Cost of Collection and Storage• Minimal Impact of Collection (morale,

disruption, etc.)• High Information Communicated• One Metric answers Multiple Questions• One Metric supports Multiple Goals



All Rights Reserved

Slide # 56

Look for Questions and Metrics

that Do Dual DutyGoal

1Goal

2

Q1 Q2 Q3 Q4 Q5

M1 M2 M3 M4 M5 M6 M7



All Rights Reserved

Slide # 57

Other Attributes of a Good Set of Metrics

• Completeness - Addresses All Issues of Concern

• Value is High relative to Cost of Collection, Storage and Analysis

• There is an Organizational Framework for Understanding the Importance of Metrics– I.e., people do not sabotage the metrics

collection effort– And people do not draw wrong conclusions

• All Parts of the Organization Benefit• Alternative Metrics are Available if Primary

Choices are Not Available



All Rights Reserved

Slide # 58

Example: GQM Approach• Goal:

Improve Efficiency by Reducing the Cost of Rework

• Question 1: What does rework cost us?

• Question 2: How many defects are in the software when we release it?

• Question 3: What are the sources of defects and rework?



All Rights Reserved

Slide # 59

Possible Metrics for Question 2How Many Defects are In the Software when

we Release It?• Number of Defects in Product -- this is impossible to know -- but we can approximate it with other metrics• Number of Known Defects at Release --

incomplete, but perhaps correlated to the total• Number of Defects Detected After Release• Total Defects over the Life of the Product



All Rights Reserved

Slide # 60

Analyzing Defect Data

Release

12 Months

Retirement

TotalDefect

sKnown at Release

Known after 12 Months

Total Defects in Product Life



All Rights Reserved

Slide # 61

Tie Metrics to the Process -- A Measure for Every Phase

PhaseMetric

Requiremen

tsTestPla

nning

Design

Integratio

nCodin

gStaffing

X

X

X X X

XRequirements

Stability

X XX

Etc.

X

Design Complexi

tyX

Code Complexi

ty



All Rights Reserved

Slide # 62

Risk: “Running out of memory space”

Possible Measures:– Predicted memory space used

(update each week or month, depending on risk)

– Size of code produced by compiler– Level of experience of average staff

Selecting Measures• Any risk may result in measures for monitoring • Choice depends on anticipated causes of risk



All Rights Reserved

Slide # 63

Another Example Risk Measure

Risk: “Not meeting schedule”Cause: Not enough experienced staffPossible Measures:

• Staffing level• Unplanned turnover • Average staff experience, etc.



All Rights Reserved

Slide # 64

10.4 Recommended Metrics• Problem Reports• Customer Satisfaction• Requirements Stability• Rate Charts• Earned Value• Resource Use• Defects• Rework



All Rights Reserved

Slide # 65

Problem ReportsComplaints Help you Improve

Reports should be categorized by such things as: -- Kind of problem -- Severity -- Source of the problem (process step) -- Effort required to correct -- When discoveredThis can tell you where you need to focus

improvement effortsProblem reports are a direct measure of customer

satisfaction



All Rights Reserved

Slide # 66

Problem Report Analysis

0

10

20

30

40

50

60

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

SeriousNormalMinor



All Rights Reserved

Slide # 67

Some Issues with Measuring Problem

Reports• How to handle duplicates• How to handle different symptoms

caused by the same root problem• How to handle reports that are

the result of multiple root problems

• Urgency vs cost vs other factors in deciding on priority



All Rights Reserved

Slide # 68

Customer Satisfaction-- as measured by the

customer• Surveys• Degree of cooperation• Do they buy your product?• etc.

Business survival depends on satisfying the customer better than the competition



All Rights Reserved

Slide # 69

Requirements StabilityTBDs + New + Changes

Total Requirements

00.10.20.30.40.50.6

Start Review 1 Review 2 Review 3 Review 4



All Rights Reserved

Slide # 70

Requirements StabilityAnother Way to Display

01020304050607080

Start Review 1 Review 2 Review 3 Review 4

TotalChangesTBDs



All Rights Reserved

Slide # 71

Rate Charts• These tell you the rate at which

something is happening - usually compared with some plan

• What you measure is something discrete that represents genuine progress toward the objectives of the project

• If you measure well, rate charts can give you a better handle on where you are than something like “hours spent”



All Rights Reserved

Slide # 72

Rate Charts ExampleCompletion vs. history and plans

Requirements Fully Tested

05

1015202530

0 1 2 3 4 5 6 7 8 9 10 11 12

HistoricalPlanActual

Today Deadline



All Rights Reserved

Slide # 73

Staffing: actual vs. plan

02468

10121416

J an Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec

PlanActual



All Rights Reserved

Slide # 74


02468

1012141618


PlanActualProjection



All Rights Reserved

Slide # 75


02468

1012141618


PlanActualProjectionBehind



All Rights Reserved

Slide # 76

Rate Charts can be used for Many Things

• Examples: -- Modules

designed/coded/tested/integrated -- Requirements

designed/coded/tested/integrated -- Problems corrected -- High priority complaints resolved



All Rights Reserved

Slide # 77

Characteristics of Applications for which Rate Charts are a Good Choice• Discrete Events• Easily Measured• Clear Definition of Completion• Many Occurrences



All Rights Reserved

Slide # 78

Earned ValueQuestions: “How much of the work have we actually

done?” “Are we likely to meet the deadline?”Metric: “Amount of work actually performed, and

projection for duration of project.”Measure: Earned Value = % of total work performed



All Rights Reserved

Slide # 79

Earned Value Uses These Data

• ACWP (Actual Cost of Work Performed)– How much money has been spent?

• BCWP (Budgeted Cost of Work Performed)– How much money should have been spent

for the work actually performed?• BCWS (Budgeted Cost of Work Scheduled)

– How much work should have been done by now according to the schedule?



All Rights Reserved

Slide # 80

Units• “Money” and “work” can be

measured in dollars, effort, or other units

• Effort is usually the easiest way to measure

• But dollars can be adjusted to include overhead costs and non-labor costs, so it may be more accurate



All Rights Reserved

Slide # 81

Questions You Can Answer with Earned Value

• Are we getting the work done on schedule?

• Are we overspending?• Will we overrun or under-run the

budget?• What is a realistic end date?• What performance level is required to

meet the budget or the deadline?



All Rights Reserved

Slide # 82

Earned Value Provides Insight Early in the Project

• You can tell if there are problems after a few weeks or months

• You have time to make adjustments• You can communicate the problems

and their scope using quantitative methods rather than guesswork



All Rights Reserved

Slide # 83

Rate Charts Can Display Earned Value in a Visual

Manner• Rate charts give a visual picture of

how much “value” you have “earned”• Consider some examples ...



All Rights Reserved

Slide # 84

Requirements Implemented

0

20

40

60

80

100

120

Total RqmtsPlanActual



All Rights Reserved

Slide # 85

A Rate Chart forIterative Lifecycles

01020304050607080

1 2 3 4 5 6 7 8

Rqmts DesignedRqmts CodedRqmts Tested



All Rights Reserved

Slide # 86

Earned Value Metrics Quantify Progress and Help

Predict and Manage



All Rights Reserved

Slide # 87

Top Level View ofEarned Value Metric

EarnedValue

•$ Spent vs $ Planned

•$ Spent vs Thresholds of what should have been spent

Budget

“Are we overspending?

”

WBSor

Requirements“Are we

getting the work done?”



All Rights Reserved

Slide # 88

A Simple Way to do Earned Value

• This method avoids a lot of the jargon and just focuses on the substance

• If you understand this method you will find it easier to understand the definitions of earned value metrics found in other places



All Rights Reserved

Slide # 89

Develop a Micro Schedule for the Next Part of the Project

Top Level Schedule

Micro Schedule for the Next Phase

Micro Schedule is the locally-

managed schedule,

defined by those doing the work



All Rights Reserved

Slide # 90

What Is a Micro Schedule• A schedule of small tasks whose

duration can be measured in days ora few weeks

• This represents the work tasks assigned to individuals

• Have the individuals develop their micro schedule



All Rights Reserved

Slide # 91

Earned Value Micro ScheduleBasic Task Estimates

Task Effort Planned Complete Date Resp.(days) (week #)

Set Up 3 1 JoeGet Specs 2 2 MaryDesign Output 10 5 Pete & JoePlan Tests 3 6 JoeCode 5 7 MaryUnit Test 3 8 JoeIntegrate 2 9 MaryBeta Test 3 10 PeteTotal 31



All Rights Reserved

Slide # 92

Convert to Running Total of Effort Planned

Week Total Effort Planned to be Complete1 32 53 7 (*)4 11 (*)5 156 187 238 269 28

10 31

(*) Assumes partial

progress on “design

output” task



All Rights Reserved

Slide # 93

The Total Planned Effort is the Budgeted Cost of Work

ScheduledWeek Total Effort Planned to be Complete

1 32 53 74 115 156 187 238 269 28

10 31

BCWS



All Rights Reserved

Slide # 94

Earned Value Plan Graph (BCWS)

Some systems estimate BCWS by drawing a straight line from 0 at start to total project budget

at end.

05

101520253035

1 2 3 4 5 6 7 8 9 10

TotalPlan

BCWS



All Rights Reserved

Slide # 95

Budget• Your actual budget should match

this plan• If the actual budget does not

match, you can negotiate what to do before beginning the work– Reduce plan to match available budget– Increase budget to match plan– Somewhere in between– Renegotiate plan with developers



All Rights Reserved

Slide # 96

Task Effort (days) % Complete EarnedSet Up 3 100 3Get Specs 2 50 1Design Output 10 25 2.5Plan Tests 3 0 0Code 5 0 0Unit Test 3 0 0Integrate 2 0 0Beta Test 3 0 0Total 31 6.5

BCWP

Earned Value Datafor a Given Week



All Rights Reserved

Slide # 97

Earned Value Procedure• Each week you compute a new “%

complete” value for each task– Perhaps allow only 0 or 100%– Or perhaps allow 50% or 33% or 25%,

but you should avoid smaller increments• Then compute the total work

performed so far• Plot the total each week in the plan

graph, representing an “actual” line



All Rights Reserved

Slide # 99

Earned Value Applications• You can spot deviations from plan

and project completion dates• You can also plan overtime or

other contingency activities• You can show your management

that you know where you are and are not suffering from the “always 90% done” syndrome



All Rights Reserved

Slide # 100

Earned Value Advantages• By using only small tasks, it is easier to

tell what is actually complete• Developers can use the data to

manage their work - it is their plan– The data gives them insight on what they

are accomplishing• The detailed work sequence does not

matter. All completed work is “earned” – Developers can make the right decisions

about work sequencing



All Rights Reserved

Slide # 101

ACWPThe Third Number

• Actual Cost of Work Performed is what you have actually spent

• Whereas BCWP is what you have “earned” or accomplished

BCWP = Work Completed

(earned value)

ACWP = Money Spent

(or effort spent)

Slide # 105

Summary of the Three Numbers



All Rights Reserved

Slide # 106

BCWS(Budgeted Cost of Work

Scheduled)• The amount of effort you had

planned to expend (i.e., the amount you budgeted)

• “How much work should have been done”

05

101520253035

1 2 3 4 5 6 7 8 9 10


BCWS



All Rights Reserved

Slide # 107

BCWP(Budgeted Cost of Work

Performed)• The value produced for the effort

you have actually expended• “How much work was actually

done”

05

101520253035

1 2 3 4 5 6 7 8 9 10


BCWP



All Rights Reserved

Slide # 108

ACWP(Actual Cost of Work Performed)

• The amount of money you have spent so far

• “What did your work actually cost?”

05

101520253035

1 2 3 4 5 6 7 8 9 10

TotalPlanActual$ SpentACWP



All Rights Reserved

Slide # 109

What Can We Tell About the Schedule

From These Three Numbers?SV - Schedule Variance

SV = BCWP - BCWS

SPI - Schedule Performance IndexSPI = BCWP / BCWS

Negative means behind schedule

Less than 1 means behind schedule



All Rights Reserved

Slide # 111

What Can We Tell About the Cost

From These Three Numbers?CV - Cost Variance

CV = BCWP - ACWP

CPI - Cost Performance Index CPI = BCWP / ACWP

Negative means over budget

Less than 1 means over budget



All Rights Reserved

Slide # 113

Two Useful NumbersBAC - Budget at Completion

– The total budget for the project– BCWS at project completion

SCHED - Schedule for the project– The total number of weeks or months

or whatever



All Rights Reserved

Slide # 114

Estimated Budget At Completion

EAC - Estimate at Completion– The estimated actual cost of project

EAC = BAC / CPI

VAC - Variance at CompletionVAC = BAC - EAC

Negative means over budget by this much



All Rights Reserved

Slide # 115

Schedule At CompletionSAC - Schedule at Completion

– The estimated actual schedule of projectSAC = SCHED / SPI

SVAC - Variance at CompletionSVAC = SCHED - SAC

Negative means behind schedule by this much



All Rights Reserved

Slide # 116

Thus You Can Tell ...• How far behind or ahead of schedule

you are• How far over or under budget you are• What your eventual schedule and cost

are likely to be

You also have an early warning of trouble that you can use to try to

fix the underlying problems



All Rights Reserved

Slide # 117

A Handy Graph of SPI and CPIIndicates Your Overall Status

Behind Schedulebut Under Budget(shortage of staff )

Ahead of Scheduleand Under Budget

Behind Scheduleand Over Budget

Ahead of Schedulebut Over Budget(staff surplus?)

SPI

CPI



All Rights Reserved

Slide # 118

Reasons for Being Behind Schedule

• If you are also Under Budget– There is not enough work being done

• Not enough people• People being used for other tasks

• If you are at or Over Budget– There is a productivity problem

• Underestimation of the effort• Excessive “non-value-added” work



All Rights Reserved

Slide # 119

Resource Use Metrics• Staffing on project• Staff-hours worked• Memory requirements for product• CPU capacity required• Amount of reused code

You can predict beforehand and then measure actuals

When actuals deviate from plan, you know that you need to act



All Rights Reserved

Slide # 122

The Estimation Spreadsheet (Assignments

4/5) Provides Initial Plan Data• Staffing levels

• Headcount• Dollars spent• Size in Lines of Code• Size in Memory• Reuse• etc.



All Rights Reserved

Slide # 123

The Estimation Spreadsheet (Assignments

4/5) Provides Initial Plan DataPeriodic updates can be

represented by additional rows and/or columns

Actuals and history can also be represented by additional rows and/or columns



All Rights Reserved

Slide # 124

Measuring Defects• Defects are one of the most valuable

things you can measure• They indicate fundamental process

problems that are causing you to waste time and money and produce lower quality products

• But people resist measuring defects because they represent “bad news”

• It takes a cultural change, sometimes, to get defect measurement viewed as a positive thing



All Rights Reserved

Slide # 125

Kinds of Defects to MeasureAny product or byproduct can have

defects.• Requirements analysis defects - collect at

inspections and reviews• Design defects - collect at design

walkthroughs, inspections, reviews• Coding defects - collect during peer

reviews and tests• Testing defects (defects in testing

procedures)• etc.



All Rights Reserved

Slide # 126

Rework• Rework is work you do over because the

first time was wrong, unneeded, solved the wrong problem, etc.

• Every process has some rework, but rework adds cost and time -- and it reflects quality problems

• Most organizations resist measuring rework because it makes them look bad

• But rework is an excellent metric for evaluating process health and effectiveness



All Rights Reserved

Slide # 128

Measuring ReworkSimple Method: count problems• Count problem reports and estimate

effort to fix each problem• The sum of all the effort is an estimate of

total rework• Compare with total effort on project

This only works if you identify problems, document them as problems, and

estimate the work required to fix themMore Advanced Method: phase

containment (see next slide)



All Rights Reserved

Slide # 129

In-Phase vs. Out-of-Phase Defects

Defect Containment

Chart

Integrate

Test

Code

Design

Rqmts

Input6

42

22

3

2

12

1

22 113

14

16

3 82

23

18

5

14

12

Rqmts

Design

IntegrateTestCod

e

Phase where Defect was Inserted

PhaseDetected



All Rights Reserved

Slide # 130

Cost of Rework• In-phase defects are often not

counted as rework -- they can be viewed as a part of the process (more on this in a later course)

• Out-of-phase defects cost a lot more to fix, in general, and that effort is usually classified as rework



All Rights Reserved

Slide # 132

FrequencyHow Often to Measure

• Too often results in high cost, disruption of process

• Too seldom results in failure to see problems soon enough

• Frequency should depend on degree of risk and cost of measurement

05

101520

1/yr 1/month 1/week 1/day 1/hour

EffectivenessCost



All Rights Reserved

Slide # 137

UnitsOften Subject to Misinterpretation

• Memory Size: words, bytes, bits???• Program Execution Time:

– Hours since shipped? {was it delivered?}– Hours since received by customer? {was it installed?}– Hours since installed? {was it used?}– CPU execution time since installed?

• CPU Time: cycles, instructions, MIPS, FLOPS, ?

Beware of innocent misinterpretations



All Rights Reserved

Slide # 138

Interplay of ResourcesFew Things Happen in

IsolationSometimes, system resources interact with each other, giving misleading

evidence of performance

Memory

Capacity:80 cycles

per micro-second

CPU50 MIPS

I/O Channel20 MIPS

Display40 MIPS

CPU may run at only 20 MIPS because of

memory saturation



All Rights Reserved

Slide # 139

Summary1) Know WHAT you are measuring - attributes,

etc.2) Know WHY you are measuring - goals, risks,

etc.3) Select wisely

– Complete - something for each part of the process

– Reasonable cost of collection– High information content– Minimal organizational disruption

4) Beware of incorrect interpretations5) Beware of misuse

“Nobody believes statistics anyway.”

Adams, The Dilbert Principle

january 20, 2000 cse 7315 - sw project management / chapter 10 – measuring and monitoring...

Documents