management & measurement & business value “oh my”...measurement needs to produce value...

Management & Measurement & Business Value – “Oh My”

Creating Solutions with Measurement in Changing Times

Dan Galorath

[email protected]

Copyright 2011 Galorath Incorporated

mailto:[email protected]

Key Points

Measurement needs to produce value not cost in the

eyes of management

and other stakeholders

Measurement results should include confidence and risk so leaders can understand and include in decisions

Measurement needs to think like business leaders… provide information understandable to management and other stakeholders

Human Nature: Humans Are Optimists

HBR Article explains this Phenomenon:

• Humans seem hardwired to be optimists

• We routinely exaggerate benefits and discount costs

Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)

3

Solution - Temper with “outside view”:

Past Measurement Results, traditional forecasting,

and statistical parametrics can help

Don‟t remove optimism, but balance optimism and

realism

http://hbr.org/search/Dan Lovallo/

http://hbr.org/search/Dan Lovallo/

http://hbr.org/search/Daniel Kahneman/

http://hbr.org/search/Daniel Kahneman/

Example of Tempering: German Mark V Tanks (Source: How To Measure Anything)

• Allies estimated production by analyzing serial numbers

• Used the captured tanks as a random sample and predicted probability of various production levels

The Gap Between IT & Management Needs (Source: Fraunhofer)

© 2011 Copyright Galorath Incorporated 5

Financial

Customer Business Process

Innovation / Growth

Vision & Strategy

Goals

Data Collection

Measurement Metrics

Answers Questions

Goal Attainment

Software Development & Maintenance

6σ

GQM

PSM

CoBIT…

Business/ Organization Level

IT Governance

IT Governance

Customer Focus

Software Projects Must Return Business Value “Economics is primarily a science of choice…

software economics should provide methods for analyzing the choices software projects must make.” Leon Levy

“Base choices on those providing the maximum business value to the organization” Eli Goldratt

Measurement and its uses such as estimating and defect analysis help this science of choice

• Some say business value is not our problem

• While others generally need to perform benefit analysis

• We need to build systems that optimize the business

• Make IT part of the solution

If IT & measurement don’t generate sufficient profit money will go elsewhere

Major Questions / Viewpoints By Role Executives Project

Managers Other Stakeholders

Team Members

Why should we fund this (what is the ROI)

My project is most important

Possibly what is the ROI

This is interesting, etc.

Is there something better to fund

Is there something more interesting

What if we do nothing

How are we doing on quality and what can I do about it

How good will it be (quality)

How long will it take

How long will it take

How long do I have Can I get more

How much will it cost



Don’t talk to me about function points

Function points don’t give “credit” for all my work © 2011 Copyright Galorath Incorporated 7

Our Issues In Measurement and Communicating Value Problem Solution

We speak in geekese Learn to speak leaders language Show value, not just what

We measure for the sake of measuring

If a measurement doesn’t have a customer it should not be produced If a measurement doesn’t provide quantifiable value it probably shouldn’t be produced

Business processes don’t change to use measurement conclusions

Champion process change as well as measurement

We measure without clear definitions

Define, normalize, validate

We fail to measure results so executives can relate

Quantify results: ROI, IRR, NPV, Payback, Etc.


Change the Results, Not the Debate

Measurement Manifesto For Software & IT

Measurement Perspective

• We measure to ultimately produce business value to the organization

• MEASURED Measurement should not ultimately be a cost

• The analysis of measurements produces decisions that produce business value

Management & Stakeholder Perspective

• Speak so I can understand

• Give me actionable items

• Don’t just give me problems… Give me solutions

• Help me make the best decisions so we can produce business value


Measurement itself is not the answer…. Management decision making, better performance, quality improvements and better serving stakeholders is

Software Process Stresses Measurement… But Do We get the Return on Measurement? Source: CMMI

Measure

&

Analyze

Monitor

&

Control

Estimate

&

Plan

A Foundation of Risk Management

Quantitative Project Management

10

Measurement & estimation go hand in hand

Software & IT Systems Are About Business Value

Value (expected consequential effects)

• Tangible benefits

• Return on investment

• Internal rate of return

• Net present value

• Intangibles

Cost

• Development

• Maintenance

• IT infrastructure

• IT services

• Other direct costs


Thinking Like a Leader

• First allegiance is to shareholder value

• Must make the decisions that maximize shareholder value

• You will make some wrong decisions & they can mean losing your bonus or your job

• You think in terms of the language of finance… Even if your are a techie by training


Return on Investment (How much Money the investment will return)

Net Present Value (VALUE: How much money will be made on the investment)

Internal Rate of Return (RATE: Yearly percentage returned on the funds used for the investment)

Payback (Time: When will the initial investment be recaptured)

Generating the Business Value Side of the Equation (Benefits) • The business owns benefit calculations

• IT should participate

• Exception: projects solely improving internal IT

• Beware of subjectivity translating soft benefits

• Use probability and risk


Increased Revenue

Increased Profit

Reduction in cost (people,

processes, cash out)

Internal benefits

Intangible Benefits

Quantify Intangible Software Benefits & Costs When Possible … But Be Realistic; Examples

• Improved Customer Satisfaction: May yield

Improved Customer Retention

• Higher Quality Product: May yield lower rework and

higher customer satisfaction

• Improved Decision Making: May yield many things

• Improved Win Ratio: Yields additional revenue

• Time to Market: May improve profit/market share


Reasons Management Doesn’t Believe the Software Metrics… Or IT ROI

• CAD Systems were supposed to improve productivity… But did not for many years… Learning curve

• MRP Systems spend millions with no ROI

• False business cases: If a 10,000 person organization installs a new timecard system…

• And system saves every employee 1 minute per day

• People work a day each day… 1 minute won’t help

• That is not a 10,000 minute per day savings


More Lack of ROI: (Source: Nucleus

Research, 2003)

• 57% of SAP Reference customers: no positive ROI

• Average about 3 years spending $5.4M

• Issues

• Excessive customization

• Extensive consulting costs over $3.6 million

• When consulting is 2 times software cost, positive ROI is unlikely

• Where customers did derive benefits

• Increased productivity

• Reduced headcount

• Improved operations management

• Improved information organization and access for decision making


Return on Investment (ROI)

• Basic assessment of projects value

• Ratio of money gained or lost relative to the investment

Be sure to consider risk on investment and return


Net Present Value of a Cash Stream

• PV = Net Present Value of future cash

• It = Future value of period t

• Rt = interest rate in period t

• Simplified equation should be iterated per period

Compares discounted value of dollars today to the same dollar in the future

• Highest NPV brings the most value to the company

• Theoretically if NPV > 0 project is worth doing


Internal Rate of Return (IRR)

• Rate of growth expected from project

• Higher is better return

• IRR =interest rate where net present value = 0


An ROI Analysis of A New System: Should We Fund This?


• Can we do better? • Will stakeholders tolerate a

loss for 3 years? • What is the risk?

Specification of ROI Should Be A Model, Not Just a Number (Adapted from Andy Pols)

• Technical baseline define product

• Assumptions: REVENUE = 20% existing product sales ($10m /year)

• Assumption: Product beats competitors to market

• Assumption: Can release 2 months before year end

• Estimate product cost, schedule & cost risk

• DEV = Development least $4m, likely $5m, most $8m

• MAINT = 5 years maintenance $3-5m total

• IT = IT infrastructure & services $1m annually

• Likely cost = (Revenue *5 – (Dev + Maint + IT)) / Dev + Maint + IT

• ROI likely = 3.57


Benefits model: 1. Repeatable & auditable 2. Can revaluate if things change

Pitfalls in ROI (Adapted from Andy Pols) • Project will generate $10m revenue

Can’t be evaluated if business conditions change

• Project X will automate the current manual process

So what? What is the value to the business?

• Project x will provide the most advanced solution

So What? Solutions for their own sake mean nothing

How will this impact the business?

• Regulators will shut us down if we do not do this

Should be easy to get cost to business

• Project X allows us to achieve real time processing

Solution, not business value statement


•Cutter Consortium Software Project Survey:

• 62% overran original schedule by more than 50%

• 64% more than 50% over budget

• 70% had critical product quality defects after release

• Standish Group CHAOS Report

• 46% challenged

• 19% failed

• 35% successful

~$875 billion spent on IT

~$300 billion spent on IT projects

~$57 billion wasted annually

ROI of better planning huge

IT Failures Are Pervasive: And Even Successful Projects May Not Be

Some suggest 80% of systems NEVER produce a positive ROI

24

Success Story: Data Improves Estimates For New Programs (Source: John Vu, Boeing)

.

0 %

140%

-140%

..

. .

.

..

. .

.

. .

.

. .

. . . .

.

. . .

. .

.

.

. . . . . . . . . . . . ..

. . . .. .

.

. .

. ..

.

.

.

. . . .. .... .. . .. . .. . . . . .

. . .

.

Without Historical Data With Historical Data

Variance between + 20% to - 145% Variance between - 20% to + 20%

(Efforts = Labor Hours)

(Mostly Level 1 & 2) (Level 3)

Ove

r/U

nd

er

Pe

rce

nta

ge

.

(Based on 120 projects in Boeing Information Systems)

.

. . . .

.

. .

..

.

. .

. .

. .

. .

. .

. . .

. . . . . . . . . . . . . . . . .

.. . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

John Vu, Boeing, keynote talk at SEPG „97,

“Software Process Improvement Journey (From Level 1 to Level 5)”

Success Stories: Affordability & Project Success


• Fortune 100 Company

• Savings $10m - Measurement & estimation discovered overcharging by outsource supplier

• ROI over 1,000%

• IRS (Source Bostelman) Evaluates over 120 investments worth about $6B (USD) TCO annually

• Speed, credibility, and transparency of resource estimates is of paramount importance

• Portfolio planning and on budget performance achieved

• Avoided major failures

More Example Success Stories

• Honeywell

• “Should Cost” models reduced outsourcing costs within the first two to three months

• Expected Savings between $457,500 and $1,372,500

• ROI, IRR, NPV confidential but significant

• Social Security

• Users can simulate project design, labor, technology, make-buy and compliance scenarios and monitor and control project progress to ensure that software projects are delivered on time, on budget and as scheduled


Example of Analysis of Estimation Process & Tools Analysis


Analysis Provided To Corporations On The Value of Better Estimating


• Results of implementation savings from $413,000 to $622,000

• Expected savings $501,000

• NPV of savings $641,000

Other Key Software Specific Metrics Can Help Track Project Performance

Track defect

discovery and

removal rates

against expected

rates

Increased defect

reporting rate

shows a

worsening trend

Heath and Status Indicator

shows status and trends from

the previous snapshot

Thresholds are user definable

Exuding Confidence or Concern With Measurement


Not All C Level Executives Understand Risk But If They Can They are Better Off (Source Q. Redman)

• A Point Estimate is never correct!

• There is always potential variability in inputs.

• But…

• 26% of program managers do not accept risk assessment at all, not even “slightly”.*

• 44% of risk ranges are intuitive judgments, without historical data or guided-survey.*

*U.S. Aerospace Risk Analysis Survey, Hollis M. black, III

Low Nominal High

Advanced Development - Total 68,348 97,132 257,492

Government In-House Costs @ 35% 17,720 25,182 66,757

Advanced Development - Contractor 50,628 71,950 190,735

Program Management 2,448 5,103 19,100

System Engineering 4,805 7,731 33,425

Design Engineering 4,526 9,980 33,193

Hardware Manufacturing (Prototype) 24,150 25,421 27,963

Software Development 14,064 14,805 16,285

Tooling and Test Equipment 45 1,034 13,847

System Integration 0 1,880 11,937

System Test and Evaluation 453 3,541 20,532

Data 136 1,013 6,526

Training 0 288 2,547

Facilities 0 627 2,865

Spares 0 528 2,515

Low Nominal High

System Develop. & Demo. - Total 184,881 273,557 361,292

Government In-House Costs @ 35% 47,932 70,922 93,668

System Develop. & Demo. - Contractor 136,949 202,635 267,623



Design Engineering 16,093 21,952 33,193

Hardware Manufacturing (Prototype) 85,865 90,385 99,423


Tooling and Test Equipment 45 2,866 13,847

System Integration 0 6,368 11,937

System Test and Evaluation 453 11,993 20,532

Data 136 3,432 6,526

Training 0 6,623 2,547

Facilities 0 2,123 2,865

Spares 0 2,323 2,515

Low Nominal High

Procurement - Total 2,149,874 3,162,081 6,722,475

Government In-House Costs @ 25% 429,975 632,416 1,344,495

Procurement - Contractor 1,719,899 2,529,665 5,377,980



Design Engineering 0 28,274 124,935

Hardware Manufacturing 1,497,878 1,576,714 1,734,385


Tooling and Test Equipment 18,003 83,576 367,455

System Integration 18,003 107,276 367,455

System Test and Evaluation 23,404 144,698 492,389

Data 3,601 33,680 110,236

Training 9,002 30,769 80,840

Facilities 0 7,484 36,745

Spares 18,003 142,703 551,182

Operational Site Activation 0 77,838 503,413

Low Nominal High

O&S Costs - Partial Total 4,241,190 5,173,716 8,377,451

Fielding 747,220 913,742 1,485,838

Sustainment (O&S) 2,689,990 3,289,472 5,349,016

Spares and Repair Parts 747,220 913,742 1,485,838

Personnel 56,760 56,760 56,760

Understanding & Tracking Defects, Growth And Other Metrics

Heath and Status Indicator

shows status and trends from

the previous snapshot

•Including Size Growth and Defect

Discovery/Removal Rate

•User defined control limits to

control the transition between

red-yellow-green

Track defect

discovery and

removal

rates against

expected

rates

Increased defect

reporting rate

shows a

worsening trend

Track

software size

growth

32

Software Is Getting More Complex (Source: Impact of

Weapon System Complexity on Systems Acquisition by Robert A. Dietrick, Major, USAF)

Why should we care: Software is already difficult, costly, and error prone. More complexity means more problems MEASUREMENT IDENTIFIES THESE

An Estimate Defined: Measurement & Estimation Go Hand In Hand

• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:

• Effort / Cost

• Schedule

• Staffing

• Risk

• Reliability

• Estimates more precise with progress

• A WELL FORMED ESTIMATE IS A

DISTRIBUTION

34

35

10 Step Software Estimation Process: Consistent Processes = Reliable Estimates

1. Establish

Estimate Scope

2. Establish Technical

Baseline, Ground

Rules, Assumptions

3. Collect Data

4. Estimate and Validate

Software Size

5. Prepare

Baseline

Estimates

7. Quantify Risks and

Risk Analysis

6. Review, Verify

and Validate

Estimate

8. Generate a

Project Plan

9. Document Estimate

and Lessons

Learned

10. Track Project

Throughout

Development

0 4 8 12 16 20

Schedule Probability Example Application 1

Probability

Time (calendar months)

1%

10%

20%

30%

40%

50%

60%

70%

80%

90%

99%

Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs)

0 1800 3600 5400 7200 9000

Effort (person-hours)

1%

10%

20%

30%

40%

50%

60%

70%

80%

90%

99%

Effort Probability Example Application 1

Probability

0 12 24 36 48 60

Defects Probability Example Application 1 Probability

Defects (count)

1%

10%

20%

30%

40%

50%

60%

70%

80%

90%

99%

36

Considering Maintenance During Planning Can Yield More Successful Long Term Results

Staff Vs Maintenance Rigor

0

500

1000

1500

2000

2500

3000

3500

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85

Time

sta

ff h

ou

rs p

er

mo

nth

develop

Rigor vhi+

Rigor nom

Rigor vlo

37

The IT Debt

• Gartner says $500 billion and going to $1 Trillion

• Cast Software says cost of the IT debt (Technical Debt) of an average-sized application of 374,000 lines of code is $1,055,000

• Future cost of replacement of decaying software

• Software Entropy

• Either evolution & innovation is managed & effort invested or

• Software / system become less useful to the point where it needs replacing


The means short term under spending is building up large future costs Management can understand the future costs of such approaches

The 7 Habits Of Highly Effective Measurement (Adapted From: Covey)

• Be Proactive: Identify Targets, Find Ways Of Meeting Conflicting Goals

• Begin With the End In Mind: Understand What the Key Issues Are… Measure what matters…

• Report what decision makers need

• Put First Things First: Planning

• Think Win/ Win: Ensure measurement helps those being measured as well as business leaders

• Seek First to Understand, Then to be Understood

• Synergize: Combine Knowledge/Resources /Data with Models and Processes

• Sharpen the Saw: Spend Some Resources To Improve Processes & Tools, Training

Key Points

Measurement needs to produce value not cost in the

eyes of management

and other stakeholders

Measurement results should include confidence and risk so leaders can understand and include in decisions

Measurement needs to think like business leaders… provide information understandable to management and other stakeholders

42

If you always do

what you always do

then

you will always get

what you always get DO THINGS DIFFERENTLY

The Rule of Inertia

Probable Consumers of Measurement Information

Executives

Program Managers

Customers internal / external

Team members


Development Resources To Generate Value (Source: Tom Gilb)


The Components of Measurement for Business Decision Making

Measurement of Actuals

Estimation of Cost

Estimation of Value

Quantification of Risk


Watch Total System Requirements: IT Systems

Total Ownership Costs; 60+% Can Be Infrastructure & Services

• Software Development

• Software Maintenance

• IT Infrastructure

• IT Services

• Other

Measurement Results must provide real analysis

Mission of IT & Measurement Is To Produce Business Value • IT exists to make a profit

• IT Measurement CAN BE A MAJOR CONTRIBUTOR

• If IT & measurement don’t generate sufficient profit money will go elsewhere

• IT generally focuses on delivering working software, features, functionality or a service

• Much measurement focuses on improving this feature / function / reliability


IT and hence measurement should focus on delivering business value

Economics Is a Key Driver In Software (Source: Royce)


Benefit Types

Tangible Benefits Actual benefit that can be measured (e.g. reduced headcount)

Intangible Benefits Contribute to quality & performance (e.g. improved employee morale, better customer satisfaction)

Intangible Quantified Benefits Numbers applied to intangibles when one will happen


An ROI Analysis of A New System: Should We Fund This? (Part 1 of 3)


Cost of capital 8.0%

Initial Investment Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Total

Ownership

Investment $100,000 $100,000

Increase/(dec.) in

revenue ($40,000) $60,000 $110,000 $100,000 $100,000 $150,000 $150,000 $630,000

Increase/(dec.) in

op. exp. $90,000 $70,000 $70,000 $22,000 $24,000 $27,000 $28,000 $331,000

Cash Flow ($100,000) ($130,000) ($10,000) $40,000 $78,000 $76,000 $123,000 $122,000 $199,000

PV of Cash Flow ($100,000) ($120,370) ($8,573) $31,753 $57,332 $51,724 $77,511 $71,186 $60,563

NPV 60,563 $60,563

IRR 13.5% 13.5%

ROI 121% 121.1%

An ROI Analysis of A New System: Should We Fund This? (Part 3)



Initial Investment Year 1 Year 2 Year 3

Year 4 Year 5 Year 6 Year 7 Total

Ownership

Investment $100,000

$100,000

Increase/(dec.) in revenue ($40,000) $60,000 $110,000

$100,000 $100,000 $150,000 $150,000 $630,000

Increase/(dec.) in op. exp. $90,000 $70,000 $70,000

$22,000 $24,000 $27,000 $28,000 $331,000

Cash Flow ($100,000) ($130,000) ($10,000) $40,000

$78,000 $76,000 $123,000 $122,000 $199,000

PV of Cash Flow ($100,000) ($120,370) ($8,573) $31,753

$57,332 $51,724 $77,511 $71,186 $60,563

NPV 60,563 $60,563

IRR 13.5% 13.5%

ROI 121% 121.1%

• Now we can say “this system has a 121% ROI • It returns 13.5% annually • And its net present value is $60,563 to the bottom line • The questions and answers are completely different:

• Can we do better? • Will stakeholders tolerate a loss for 3 years? • What is the risk?

An ROI Analysis of A New System: Should We Fund This?



Initial Investment Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Total

Ownership

Investment $100,000 $100,000

Increase/

(dec.) in

revenue ($40,000) $60,000 $110,000 $100,000 $100,000 $150,000 $150,000 $630,000

Increase/

(dec.) in op.

exp. $90,000 $70,000 $70,000 $22,000 $24,000 $27,000 $28,000 $331,000

Cash Flow ($100,000) ($130,000) ($10,000) $40,000 $78,000 $76,000 $123,000 $122,000 $199,000

PV of Cash

Flow ($100,000) ($120,370) ($8,573) $31,753 $57,332 $51,724 $77,511 $71,186 $60,563

NPV 60,563 $60,563

IRR 13.5% 13.5%

ROI 121% 121.1%

Year 3 Payback

• Can we do better? • Will stakeholders tolerate a

loss for 3 years? • What is the risk?

54

The RISK Problem: We Need TO Express Risk / Uncertainty To Leaders

"It's tough to make predictions, especially about the future." -- Yogi

Berra.

55

Functional Focus Example: Ladies Purse

• Function ………………………………………... Hold stuff

• Cost…………………………………………… $400 at Nordstrom

• What else will perform the function?…………Paper bag -cost = $0.05

• Go to plastic bag for more durability…………Cost = $0.10

• Add color………………………………………..Cost = $0.15

• Add strap.……………………………………….Cost = $0.25

Uncertainty and Risk (Source Q Redman)

• Deterministic Approach

• Single or “most likely” costs and other inputs are used to derive a point estimate

• Parameters bounded by “best case” and “worst case” scenarios or iterated a finite number of times

• Can quantify robustness of the outcome and highlight the degree to which the LCC is dependent on initial assumptions

• Stochastic Approach

• Most inputs are characterized by uncertainty

• These uncertainties are modeled as probability distributions (Normal, triangular, etc.)

• 69% of Cost Risk Analysis

• uses the triangular distribution.*

• Monte Carlo may be used.

• Randomly generates values for

uncertain variables over and over

looking for central tendency.

Mandatory Inputs Nominal Low High

Average Unit Production Cost = 4,583$ 0.95 1.10

Production Quantity = 344

ADM Quantity 1.5

Years in ADM Phase 3 (3 is norm, more = greater costs)

SDD Quantity = 4

Years in SDD Phase 4 (3 is norm, more = greater costs)

Average Unit Prototype Cost = 16,947$ 3.70 Step Down Calc.Learning Curve Slope (%) = 92.0 0.00 Step Down User Value - enter 0 to use calculated value, enter a value > 0.0 to enter your own value you wish to use for development'

Software Lines Of Code - ADM 80,000 Enter if known Effective Lines created in Phase (Color of money)

- SDD 80,000 Enter if known Effective Lines created in Phase (Color of money)

- Production 80,000 Enter if known Effective Lines created in Beginning of Phase (Color of money)

Fielding 2.5%

Annual Sustainment (O&S) 9.0% 18.00 Net Years of O&S

Spares and Repair Parts 2.5%

Personnel Assigned to each Vehicle 3 55$ Cost per soldier per year K$ - Total TOE level personnel (includes security and support)

RULES OF THUMB COST MODEL

Performance Measurement: Earned Value Analysis Can Make Projects More Successful

• “Earned Value Analysis” = industry standard way to:

1. Measure a project’s progress

2. Forecast its completion date and final cost

3. Provide schedule and budget variances along the way

• Provides consistent, numerical indicators with which you can evaluate and compare projects

• Earned value (Performance Measurement) techniques are just as viable using effort for root level project management

• Effort Value – units of labor; e.g.:

• person-hours, staff-hours, effort-hours, labor-hours

• person-months, staff-months, effort-months, labor-months

management & measurement & business value “oh my”...measurement needs to produce value...

Documents