Estimation, Planning & Control Processes Can Make the Difference Between Project success and FailureDan Galorath
Copyright 2011 Galorath Incorporated
Introduction
• Estimating is critical for all kinds of systems
• Yet many treat is as a second rate process
• Having a repeatable estimation process is critical to both estimating AND to successful projects
• Estimation and measurement go hand in hand
• While Galorath addresses Hardware, Software, IT , and manufacturing
• Context of this talk will be largely from software
© 2009 Copyright Galorath Incorporated 2
Delusions of Success: How Optimism Undermines Executives' Decisions (Source: Richard Hartley, HBR)
• Problem:
Humans seem hardwired to be optimists
• Optimism from cognitive biases & organizational pressures• Exaggerate talents & degree of control
• Attribute negative consequences to external factors
• Anchoring (relying too heavily on one piece of information) magnifies optimism
• Most pronounced for new initiativesSolution: Temper with “outside view”
Supplement traditional forecasting w/ statistical Parametrics
Don’t remove optimism, but balance optimism & realism
Example of Tempering: (Source How To Measure Anything)
• German Mark V Tanks
• Allies estimated production by analyzing serial numbers
• Used the captured tanks as a random sample and predicted probability of various production levels
An Estimate Defined
• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:
• Effort / Cost
• Schedule
• Staffing
• Risk
• Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A DISTRIBUTION
5
Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5
• Ariane 5 video
• Note: $500 Million payload
• Failure due to reused software from (Ariane 4)
• with incompatible assumptions for exception condition that was not required
ariane_5_explosion_video.mp4
“the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”“the view had been taken that software should be considered correct until it is shown to be at fault”!
Why should we care: Software cost & schedule should be sufficient for successful missions
Software Is A Key Risk Item In Weapons Systems
• Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to drastically underused… GAO
• GAO identified 42 programs at risk for cost & schedule 1. military requirements changes
2. software development challenges
3. workforce issues
• National Institute of Standards and Technology (NIST)
• Software defects cost nearly $60 Billion Annually
• 80% of development costs involve identifying and correcting defects
Software, not Hardware or technology readiness levels were called out
Software Is Getting More Complex Source: (Impact of Weapon System Complexity on Systems Acquisition by Robert A. Dietrick, Major, USAF)
Why should we care: Software is already difficult, costly, and error prone. More complexity means more problems
Software Is Providing an Increasing Percentage of Functionality
• Software development has been problematic for military aircraft developers
• Consuming about 40 percent of the USAF’s RDT&E budget
• Average software schedule overrun was 222 percent of the original estimate
• Overruns often justified by stating weapon systems performance requirements are evermore demanding
• And thus cause greater reliance on software
• Software provided about 10 percent of an F-4’s functionality in the 1960s but provides over 80 percent of an F-22’s
Why should we care: “unrealistic cost estimates lead to unrealistic budgets and un-executable programs” Meier Study… DAU
Software Failures Are Costly and Dangerous
• December 23, 2004, internal memo by Deputy Secretary of Defense called for the cancellation of the U.S. Air Force’s C-130J transport aircraft to cut its losses on the overpriced, unneeded, and problem-plagued C-130J. Estimated cancellation cost were $1.78 Billion – in 2004 dollars!
• The loss of a classified satellite after only 7 seconds on orbit prompted the review of software and processors that has caused the most recent delay and a potential $1 billion overrun in Lockheed Martin's Space-Based Infrared System (SBIRS). Strategic Command is worried about potential gaps in coverage for some mission areas in part because satellites are being delivered later than planned.
• Iraqi Scud missile slipped through Patriot missile defenses a year ago and hit U.S. Army barracks in Dhahran, Saudi Arabia, killing 28 servicemen.
• Cutter Consortium Software Project Survey:
• 62% overran original schedule by more than 50%
• 64% more than 50% over budget;
• 70% had critical product quality defects after release
• Standish Group CHAOS Report
• 46% challenged
• 19% failed
• 35% successful
~$875 billion spent on IT~$300 billion spent on IT projects ~$57 billion wasted annually
IT Has Similar Failures
Estimate Total Ownership Costs; 60+% Can Be Infrastructure & Services
• Software Development
• Software Maintenance
• IT Infrastructure
• IT Services
© 2009 Copyright Galorath Incorporated 12
Poor Estimates Effect on Projects• Inaccurate estimates can reduce project success:
• Poor implementations
• Critical processes don’t scale
• Emergency staffing
• Cost overruns caused by underestimating project needs
• Scope creep
• Forever changing project goals
• Frustration
• Customer dissatisfaction
• Cost overruns and missed schedules
• Project Failures
• Important project business decisions made early with minimum knowledge & maximum uncertainty
13
Why should we care: Poor estimates and plans are a root cause of program failure
Manual Estimates: Human Reasons For Error (Metrics Can Help)
• Manual Task estimates yield SIGNIFICANT error
• Desire for “credibility” motivates overestimate behavior (80% probability?)
• So must spend all the time to be “reliable”
• Better approach force 50% probability & have “buffer” for overruns
• Technical pride sometimes causes underestimates
14
SEER: Software Analysis Tools A Complete Software Project Management Solution
Kno
wle
dge
&E
xper
ienc
e
Baseline(s) &
Final Actuals
To-date
Actuals
Measure&
Analyze
Monitor&
Control
Baseline
Estimate&
Plan
A Foundation of Risk Management
Quantitative Project Management
15
16
10 Step Software Estimation Process:Consistent Processes = Reliable Estimates
1. Establish Estimate Scope
2. Establish Technical Baseline, Ground Rules, Assumptions
3. Collect Data
4. Estimate and Validate Software Size
5. Prepare Baseline Estimates
7. Quantify Risks and Risk Analysis
6. Review, Verify and Validate Estimate
8. Generate a Project Plan
9. Document Estimate and Lessons Learned
10. Track Project Throughout Development
17
This chapter discusses a 1972 GAO report on cost estimating• We reported that cost estimates were understated and causing unexpected cost growth
• Many of the factors causing this problem are still relevant today We also discuss a 12 step process for producing high quality cost estimates
GAO Publication: Characteristics of credible cost estimates and a reliable process for creating them
18
GAO Publication: Why cost estimates are required for government programs and challenges developing credible results
• Introduces why cost estimates are required for government programs
• Developing annual budgets, supporting management decisions about which program to fund, and evaluating resource requirements at key decision points
• Discusses various challenges associated with developing credible results
© 2010 Galorath Incorporated
Contractors At Least Level 3 Would Be Acceptable
Why should we care? Maturity is related to estimate viability… Better estimation process more likely to
be successful in execution
Balancing Resources & Schedule Is A Science
For a given Size, Complexity and Technology
Minimum Time
Optimal Effort(Lower Effort for Longer Schedule)
Calendar Time
Eff
ort
Mon
ths
Effort Increasedue to Longer Schedule
0 4 8 12 16 20
Schedule ProbabilityExample Application 1
Probability
Time (calendar months)
1%10%20%30%40%50%60%70%80%90%99%
Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs)
0 1800 3600 5400 7200 9000
Effort (person-hours)
1%10%20%30%40%50%60%70%80%90%99%
Effort ProbabilityExample Application 1
Probability
0 12 24 36 48 60
Defects ProbabilityExample Application 1Probability
Defects (count)
1%10%20%30%40%50%60%70%80%90%99%
21
Understanding & Tracking Defects, Growth And Other Metrics
Heath and Status Indicator shows status
and trends from the previous snapshot
•Including Size Growth and Defect Discovery/Removal
Rate
•User defined control limits to control the transition
between red-yellowyellow-green
Track defect
discovery and
removal rates
against expected
rates
Increased defect
reporting rate shows a
worsening trend
Track software
size growth 22
Considering Maintenance During Planning Can Yield More Successful Long Term Results
Staff Vs Maintenance Rigor
0500
100015002000250030003500
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85
Time
staf
f hou
rs p
er m
onth
develop
Rigor vhi+
Rigor nom
Rigor vlo
23
Software & IT Systems Are About Business Value
Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable
• 80 Calories per serving
• 2.5 Servings per can
• 4 Ounces, Condensed, 8 Ounces With Water
You have an estimate …Now what?
26
• Correlation does not imply causation• Just because two data points may sit side by side
doesn’t mean they are the same or will have the same outcome
• Casual analysis is a recognized error in medicine
Perhaps ???
The Error of Causal AnalysisCreating a False Association
Tumor Can Cause Headache
Headache doesn’t mean a tumor
27
Use Historical Measurement to evaluate your estimate!
It’s easy to dig deeper and deeper to justify an estimate!28
29
NormalizationAdjustments for homogeneity
• Time phased data can be particularly difficult to handle
• As spent dollars are influenced by inflation• Requires knowledge of time period involved
• May require converting to “fixed dollar” units
• Process data• Need to know specific sub-phase of the process
• May require allocation to sub-phase
30
Normalization: What To Look For In An Actual
Phase : All activities may not be included.
System Concept missing
Integration missing
Phases
System Concepts System Req & Design
System Req Analysis Preliminary Design
Detailed Design Code / Unit Testing
Software Test System Integration / OT&E
31
Normalization: What To Look For In An Actual: Labor
Labor : All categories may not be included.
Quality Assurance missing
Configuration Mgt missing
Labor Categories
Management System Engineering
Design Coding
Data Preparation Test
Configuration Mgmt. Quality Assurance
Estimation Best Practices
• Decide Why You Want An Estimate
• Map Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The Maturity
• Have A Documented, Repeatable Estimation Process
• Make The Estimating Process As Simple As Possible; But No Simpler
• Be Proactive: The Process Is Important, The Tools Go Along With The Process
• Get Buy-in From Program Managers
• Hold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own Them
• Tie The Estimate To The Plan
© 2009 Copyright Galorath Incorporated 32
Estimation Best Practices 2
• Evaluate Total Ownership Cost; Not Just Development
• Estimate A Range And Pick A Point For The Plan
• Re-estimate The Program When It Changes
• Avoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain Morale
• Keep A History: Start An Enterprise Database NOW…
• Business Case: Evaluate ROI In Addition To Costs
• Convert Expert Spreadsheets Into A Common Language
© 2009 Copyright Galorath Incorporated 33
Estimation Best Practices 3
• Track Progress Vs. Estimate Throughout The Life Cycle
• Estimate Schedule As Well As Effort (Cost) For Complete Picture
• Tie The Business Case Into The Estimating Process
• Attack Non-productive Rework As Part Of The Process
© 2009 Copyright Galorath Incorporated 34
Estimation Best Practices 4
• Have clear definitions: What does “complete” mean. What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.) Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.)
• Measure what you care about
• Estimating & tracking rework can help control costs
• Don’t ignore IT infrastructure and IT services costs
• Tracking defect sources can go along with the process
© 2009 Copyright Galorath Incorporated 35
Additional Information
• www.galorath.com
• Dan on estimating BLOG: www.galorath.com/wp
• Email: [email protected]
© 2009 Copyright Galorath Incorporated 36
Economics Is a Key Driver In Software (Source: Royce)
© 2009 Copyright Galorath Incorporated 37