galorath - it data collection, analysis and benchmarking: from processes and benchmarks into real...

IT Data Collection, Analysis and Benchmarking: From Processes and

Benchmarks into Real Business Value

Dan Galorath

[email protected]

Copyright 2013 Galorath Incorporated

mailto:[email protected]

http://itconfidence2013.wordpress.com/

Key Points

Measurement needs to think like business

leaders… provide

information understandable to management

and other stakeholders

Measurement needs to produce value not cost in the eyes of management and other stakeholders

Measurement results should include confidence and risk so leaders can understand and include in decisions

Context

© 2011 Copyright Galorath Incorporated

3

•Cutter Consortium Software Project Survey:• 62% overran original schedule by more than 50%

• 64% more than 50% over budget

• 70% had critical product quality defects after release

• Standish Group CHAOS Report• 46% challenged

• 19% failed

• 35% successful

~$875 billion spent on IT~$300 billion spent on IT projects ~$57 billion wasted annually

ROI of better planning huge

IT Failures Are Pervasive: And Even Successful Projects May Not Be

Some suggest 80% of systems NEVER produce a positive ROI

Human Nature: Humans Are Optimists

HBR Article explains this Phenomenon:

• Humans seem hardwired to be optimists

• We routinely exaggerate benefits and discount costs

Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)

5

Solution - Temper with “outside view”:Past Measurement Results, traditional forecasting,

and statistical parametrics can help

Don’t remove optimism, but balance optimism and realism

http://hbr.org/search/Dan%20Lovallo/

http://hbr.org/search/Daniel%20Kahneman/

Example of Tempering: German Mark V Tanks (Source: How To Measure Anything)

• Allies estimated production by analyzing serial numbers

• Used the captured tanks as a random sample and predicted probability of various production levels

Some of Dan’s Heroes

• Frederick Taylor: The Principals of Scientific Management 1901 “Let data and facts do the talking”

• W. Edwards Demming: “In God We Trust… All Others Bring Data”

• Frederick Brooks: “There is an incremental person when added to a software project that makes it take longer”

• Ed Yourdon: “Avoiding Death Marches in Software Projects”

• Steven Covey: “Sharpen the Saw” Focus on improvement

http://images.google.com/imgres?imgurl=http://www.trainingabc.com/covey.gif&imgrefurl=http://www.windley.com/archives/2003/03/productive_know.shtml&h=227&w=237&sz=20&hl=en&start=37&sig2=tFfuznXvEZRPH_lI57l5VA&um=1&tbnid=G3E1jI_VerXYnM:&tbnh=104&tbnw=109&ei=Y0uZRo7PF52mggORyIH9CA&prev=/images?q=steven+covey+picture&start=21&ndsp=21&svnum=10&um=1&hl=en&safe=active&rls=com.microsoft:*:IE-SearchBox&rlz=1I7DKUS&sa=N

http://www.nhne.com/y2kgrassrootssurvey/senate/yourdon.gif

http://images.google.com/imgres?imgurl=http://staff.imsa.edu/socsci/jvictory/taylor_fred/frederick_taylor.gif&imgrefurl=http://staff.imsa.edu/socsci/jvictory/taylor_fred/indx_taylor.htm&h=424&w=286&sz=98&hl=en&start=3&sig2=bgpMP_8lVYj04o_CyHxa4g&um=1&tbnid=Jo65JE-vQ4GQhM:&tbnh=126&tbnw=85&ei=gFCZRpTfFo_KgAOd-NyuCQ&prev=/images?q=frederick+taylor&svnum=10&um=1&hl=en&safe=active&rls=com.microsoft:*:IE-SearchBox&rlz=1I7DKUS

Measurement Manifesto For Software & IT

Measurement Perspective

• We measure to ultimately produce business value to the organization

• MEASURED Measurement should not ultimately be a cost

• The analysis of measurements produces decisions that produce business value

Management & Stakeholder Perspective

• Speak so I can understand

• Give me actionable items

• Don’t just give me problems… Give me solutions

• Help me make the best decisions so we can produce business value

© 2011 Copyright Galorath Incorporated 8

Measurement itself is N ENABLER….

Management decision making, better performance, quality improvements and better serving stakeholders is

Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR)

• Case Study: Levi Strauss

• $5m ERP deployment contracted

• Risks seemed small

• Difficulty interfacing with customers systems

• Had to shut down production

• Unable to fill orders for 3 weeks

• $192.5M charge against earnings on a $5m IT project failure


“IT projects touch so many aspects of organization they pose a new singular risk”

http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar/1

Exuding Confidence or Concern With Measurement


Frederick Brooks Classic Paper “No Silver Bullets”• “There is no single development, in either

technology or management technique, which by itself promises even one order-of magnitude improvement within a decade in productivity, in reliability, in simplicity.”

• -- Fred Brooks, 1986

• i.e. There is no magical cure for the “software crisis”

• Not software measurement

• Not better tools

• Not ---- (fill in the blank… the current great hope)

Size & Productivity


12

Fundamental Metrics For Estimation, Planning & Control

• Size

• AKA Volume, Mass

• Units: Source Lines of Code (SLOC); Function Points (FP) Use Cases

• New versus rework

• COTS & Packages

• Technology

• AKA Productivity Potential, Efficiency

• Units: none

• Time

• AKA Duration, Schedule

• Units: Calendar Months, Calendar Weeks

• Effort

• AKA Work, Labor

• Units: Staff Months, Staff Hours

• Cost

• AKA Budget, Money

• Units: $, other currencies

• Staffing

• AKA Manpower Loading

• Units: FTE People

• Defects

• AKA Reliability, Quality

• Units: Defect Count

Many Size Measures Are Viable

• We know about:

• Lines of Code &

• COSMIC Both provide a method for Sizing Software

• Story Points (different since their values are team defined) provide the same capability for User Stories or Features or anything that can sized

• Many others

• Function Points


A1-15

Size Is More Than Just Total Application

NewNew

Effective Effort Units

Lines, Functions, Objects, Use Cases, Etc.Lines, Functions, Objects, Use Cases, Etc.

Pre-existingPre-existing

Reuse: Watch Out For Low Cost Assumptions on “Heritage”

• Reuse or Heritage: applying existing software to a new mission (or additional innovation in its current mission)

• Effort to reuse software is routinely under estimated

Design

Test

ImplementationWhy should we care: Heritage is often underestimated and causes major schedule / cost overruns

Sample Effort Factors for Reuse


Software Design For Reuse (Lower Cost Heritage)

• Designed for reuse• Reusability requirements during original development

• Significant extra coding and documentation effort during original development to insure reusability

• Minor to moderate work (mostly retest) required when reused IF NOT MODIFIED

• Not designed for reuse• Developed without extra effort to ensure reusability

• Sources include any legacy code not specifically designed for reuse

• Other projects & programs

• Prior builds of the current program

• Moderate to major rework required• Redesign

• Reimplementation

• Retest

Rework Causes Software Project Issues

• Rework: Doing the same work over again because it was incorrect the first time

• Not prototyping

• Not refactoring (tuning to make software better)

• Between 40% and 50% of the total effort on software projects is spent on rework.. Barry Boehm

• Initiatives to reduce rework can save significantly

Why should we care: Unplanned rework can drive cost up dramatically AND reducing rework can be a boon to projects

Customization

COTS or GOTS Hoped For Reuse

• Developmental Software:

• Functionality developed specifically for the project at hand

• May include customization of COTS

• “Glue” Code:• Code written to bind

COTS to developmental software

• Development effort must be captured

• COTS Software:• Purchased functionality

• Direct Cost component of COTS integration

• COTS Cognition: • Required functionality within the COTS

software that must be understood

• Effort component of COTS integration

DevelopmentalSoftware

COTS Software

“COTS Cognition”

Glue

Why should we care: COTS promises are often not accurate meaning additional development cost & Schedule

Packaged Applications Are Costly To Organizations

• “Commercial application program or collection of programs developed to meet the needs of a variety of users, rather than custom designed for a specific organization”

• Many are enterprise applications

• Often allows / requires customization

• Examples: SAP; Rational PPM, SEER for Software; Microsoft Excel, CA Clarity, Oracle Business Suite

Why should we care: Packages sometimes comprise solutions for parts of complicated systems and can be trouble

"One-third [of the budget] has to go to testing. Don’t ever short change testing. Everyone always underestimates it, and says it’s the last thing to worry about. Don’t do that!“

- Jim Larson, consultant for communications solutions provider

Software Productivity Laws

• LAW 1 – Smaller teams are more efficient: The smaller the team the higher the productivity of each individual person.

• LAW 2 – SOME schedule compression can be bought: Adding people to a project, to a point, decreases the time and increases the cost.

• LAW 3 – Every project has a minimum time: There is an incremental person that consumes more energy than he/she produces. Team size beyond this point decreases productivity and increases time. (aka Brooks’ Law--”Adding staff to a late software project makes it later.”)

• LAW 4 – Productivity is scalable: Projects of larger software size can use larger teams without violating LAW 3.

• LAW 5 – Complexity limits staffing: As complexity increases, the number of people that can effectively work on the project and the rate at which they can be added decreases.

Software Productivity Laws (continued)• LAW 6 – Staffing can be optimized: There exists an optimal

staffing function (shape) that is effectively modeled by the Rayleigh function. Flat (level load) staffing is rarely optimal.

• LAW 7 – Projects that get behind, stay behind: It is extremely difficult to bring a project that is behind schedule back on plan.

• LAW 8 – Work expands to fill the available volume: It is possible to allow too much time to complete a project (aka Parkinson’s Law).

• LAW 9 – Better technology yields higher productivity: More capable teams, better tools, and advanced, stable processes yield higher productivity.

• LAW 10 – No “silver bullets”: There is no methodology, tool, or process improvement strategy out there that yields revolutionary improvements in project efficiency.

The Bottom Line

• Effort for reuse and rework must be considered

• Standard definitions of productivity e.g.

• 18 hours per function point…

• Is necessary but not sufficient for detailed estimates that must be viable

• Use a range

• Identify key cost drivers that impact productivity

• Ensure you know what is IN and NOT IN the hours


Relative Sizing Can Provide Quick ROM Numbers E.G. ISBSG

• War and Peace is 587,287 words

• How long will it take you to read it?

• Using a different reference try…

• Girl with the Dragoon Tattoo =144,000

• The Bourne Identity = 80,000

• Harry Potter and the Philosopher's Stone = 76,944


Data Collection, Normalization, Usage, Caution


Before You Start - Some Basic Rules

1. Know from where the data comes (total, effective, new, etc?)

2. Know what is included (Phases, People, Processes, …)

3. Know the different sizing schemas and how applied

4. Know what was intentionally left out (and why)

5. Consider the hindsight – if things happen they should be accounted for in the data collection

6. Data needs to be consistent or normalized

Galorath’s Data Review Process

Management is excluded from actual

Concept is excluded

from actual

CM / QA is excluded

from actual

…management from the estimate

…concept from the estimate

…CM / QA from the estimate

THE ACTUAL You should exclude...

Productivity Reasonable ?

Is Effort Accounting By Phase Probably

Complete?

Summary EffortMatch Accounting

By Phase?No No No

Adjust For Missing Phases

Is There Effort Accounting By

Phase?

Yes

YesYesYes

No

Do Not Adjust

Do Not Adjust

Do Not Adjust

Is“Project Activity

Scope” fieldComplete?

No

Do Not AdjustYes

Productivity Reasonable ?

Is Effort Accounting By Phase Probably

Complete?

Summary EffortMatch Accounting

By Phase?No No No

Adjust For Missing Phases

Is There Effort Accounting By

Phase?

Yes

YesYesYes

No

Do Not Adjust

Do Not Adjust

Do Not Adjust

Is“Project Activity

Scope” fieldComplete?

No

Do Not AdjustYes

Normalization

Mapping

29

Data Must Be Used With Caution

• Run sanity checks on data

• 10,000 Function Points can’t be developed in 3 months

• Ongoing issue between statisticians and engineers

• Some Statisticians claim.. “That is what the data says so it must be right”

• Sometimes even if it is obviously wrong

Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable

• 80 Calories per serving

• 2.5 Servings per can

• 4 Ounces, Condensed, 8 Ounces With Water

31

Data Improves Estimates For New Programs Source: John Vu, Boeing SEPG 1997

.

0 %

140%

-140%

..

..

.

..

..

.

..

.

. .

. . . .

.

. . .

. .

.

.

. . . .. . . . . . . ...

. . . .. .

.

. .

. ..

.

.

.

. .. .. ...... . .. . ... . ...

.. .

.

Without Historical Data With Historical Data

Variance between + 20% to - 145% Variance between - 20% to + 20%

(Efforts = Labor Hours)

(Mostly Level 1 & 2) (Level 3)

Ove

r/Und

er P

erce

ntag

e

.

(Based on 120 projects in

Boeing Information Systems)

.

. . .

.

.

..

..

.

. .

..

.

.

. .

.

.

.. .

. . .. . . . . .. . . . . .. .

..

. . .. . .. . . .

. . . .. . . . .

. . . . .

. . . . .. . . . . .

. . . . .. . . . . .

. . . . . .. . . . . . . .

. . .. . . . .

. . . . .. . . .

. . . . . .

. . . . . .

. . . . . .

John Vu, Boeing, keynote talk at SEPG ‘97,

“Software Process Improvement Journey (From Level 1 to Level 5)”

Just the Top Level Data Isn’t Sufficient For Estimation

Hours Versus UFPs

1

10

100

1000

10000

100000

1000000

1 10 100 1000 10000 100000UFPs

Hou

rs

Use Historical Data to evaluate your estimate!

The Error of Causal Analysis Is often encountered in data analysis!

NORMALIZE:There is no correlation that estimate of size ‘X’ is an

equivalent of historical projects of about ‘X’.

Code and Unit Test Only

System and SW Requirements

were done separately

This point does not include SW Testing Effort

Fallacy of Silent EvidenceWhat about what we don’t know?

How confident would you feel if the Silent Evidence was visible?

36

Software Has Additional Characteristics That Should Be Considered

Track defect discovery and removal rates

against expected rates

Increased defect reporting rate

shows a worsening trend

Heath and Status Indicator shows status and trends from

the previous snapshot

Thresholds are user definable

Watch Total System Requirements: IT Systems Total Ownership Costs; 60+% Can Be Infrastructure & Services

• Software Development

• Software Maintenance

• IT Infrastructure

• IT Services

• Other

Measurement Results must provide real analysis

Risk and Uncertainty


39

Statistician Drowns in River with Average Depth of 3 Feet!

Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5

• Ariane 5 video

• Note: $500 Million payload

• Failure due to reused software from (Ariane 4)

• with incompatible assumptions for exception condition that was not required

ariane_5_explosion_video.mp4

“the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”“the view had been taken that software should be considered correct until it is shown to be at fault”!

Why should we care: Software cost & schedule should be sufficient for successful missions

Minimum

Most Likely:

Maximum

If everything goes well (optimistic

estimate)

If everything goes badly

(pessimistic estimate)

Using Ranges To Address Uncertainty

pert beta Incremental Likelihood

Pert Beta

Adding Reality to Schedules Example (Source: SEI)

Process DurationsStep Expected1 302 503 804 505 906 257 358 459 7010 25

500

What would you forecast the schedule duration to be?

Adding Reality to Schedules – Example - 2

Process DurationsStep Best Expected Worst1 27 30 752 45 50 1253 72 80 2004 45 50 1255 81 90 2256 23 25 637 32 35 888 41 45 1139 63 70 17510 23 25 63

500

What would you forecast the schedule duration to be now?


With 90% confidence, the project will be under 817 days duration.

The project is almost guaranteed to miss the 500 days duration 100% of the time.


With only 50% confidence, the project will be under 731 days duration.

Typical Growth Range From Initial Sizing To Delivery

• Functional sizing for estimates should include growth range to mitigate risk


Dealing With the “Problem of Assumptions”• Assumptions are essential but…

• Incorrect assumptions can drive an estimate to uselessness

• Use an assumption verification process


Estimates must have assumptions defined but…Bad assumptions should not be justification for

bad estimatesASSUMPTIONS MAY BE RISKS

Business Value Is Part of Measurement & Estimation


48

Software Projects Must Return Business Value“Economics is primarily a science of choice…

software economics should provide methods for analyzing the choices software projects must make.” Leon Levy

“Base choices on those providing the maximum business value to the organization” Eli Goldratt

Measurement and its uses such as estimating and defect analysis help this science of choice

• Some say business value is not our problem

• While others generally need to perform benefit analysis

• We need to build systems that optimize the business

• Make IT part of the solutionIf IT & measurement don’t generate sufficient profit money will go elsewhere

The Gap Between IT & Management Needs (Source: Fraunhofer)


Financial

Customer Business Process

Innovation / Growth

Vision & Strategy

Goals

Data Collection

MeasurementMetricsAnswersQuestions

Goal Attainment

Software Development & Maintenance

6σ

GQM

PSM

CoBIT…

Business/OrganizationLevel

ITGovernance

ITGovernance

CustomerFocus

Software & IT Systems Are About Business Value


Generating the Business Value Side of the Equation (Benefits)• The business owns benefit calculations

• IT should participate

• Exception: projects solely improving internal IT

• Beware of subjectivity translating soft benefits

• Use probability and risk


Quantify Intangible Software Benefits & Costs When Possible … But Be Realistic; Examples

• Improved Customer Satisfaction: May yield Improved Customer Retention

• Higher Quality Product: May yield lower rework and higher customer satisfaction

• Improved Decision Making: May yield many things

• Improved Win Ratio: Yields additional revenue

• Time to Market: May improve profit/market share


An ROI Analysis of A New System


54

Cost of capital 8.0%

Initial Investment Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7Total

Ownership

Investment $100,000 $100,000

Increase/(dec.) in revenue ($40,000) $60,000 $110,000 $100,000 $100,000 $150,000 $150,000 $630,000

Increase/(dec.) in op. exp. $90,000 $70,000 $70,000 $22,000 $24,000 $27,000 $28,000 $331,000 Cash Flow ($100,000) ($130,000) ($10,000) $40,000 $78,000 $76,000 $123,000 $122,000 $199,000

PV of Cash Flow ($100,000) ($120,370) ($8,573) $31,753 $57,332 $51,724 $77,511 $71,186 $60,563

NPV 60,563 $60,563

IRR 13.5% 13.5%

ROI 121% 121.1%

The Rule of Measurement

When performance is measured Performance improves…

When performance is measured and reported the rate of improvement accelerates..

Thomas Monson

The work you do is a root cause of program success