cost risk – an overview dr. christian smart [email protected] president, greater alabama...
TRANSCRIPT
Cost Risk – An Overview
Dr. Christian [email protected]
President, Greater Alabama SCEA ChapterFebruary 17th, 2005
Agenda
• Introduction to Society of Cost Estimating and Analysis (SCEA)
• Cost Risk Overview• Upcoming Cost Risk Seminar
Society of Cost Estimating and Analysis (SCEA)
• Who we are– Organization for cost analysis professionals, a highly
diverse community that includes people who work in areas such as:
• Budget analysis• Earned value management• Cost estimation• Statistical analysis• Operations research• Accounting• Etc.
– Overlap with other professional organizations such as INCOSE and PMI
– Huntsville chapter is one of the largest and most active in the nation (over 150 members)
Society of Cost Estimating and Analysis (SCEA)
• What we offer– National Organization
• Hosts an annual conference and educational workshop– 2005 conference will be held in Denver, Colorado,
June 13th-17th, jointly with the International Society for Parametric Analysts (ISPA)
(http://www.ispa-cost.org/conf.htm)• Publishes National Estimator magazine and Journal of
Cost Analysis and Management, a refereed technical publication
• Provides the Certified Cost Estimator/Analyst professional credential
• Web site (http://www.sceaonline.net)
Society of Cost Estimating and Analysis (SCEA)
• What we offer– Local Chapter
• Monthly luncheons with presentations on topics of interest to the cost analysis community
– Some recent presentation topics include:» Decision Making Using Cost Risk Analysis» Nonparametric Regression in Cost Analysis » Schedule Risk Assessment » Earned Value Management Systems Concepts
• Chapter web site– http://www.scea-alabama.org/– includes full presentations dating back to 2003 and
other links of interest• Free training and materials for certification
preparation• Seminars and other forms of training
SCEA Certification
• Certified Cost Estimator/Analyst (CCE/A) professional credential– Recognized credential throughout the profession– Government procurements often require CCE/As– Certification by examination
• Two years of professional experience in cost analysis/estimating required to take the exam
– Recertification (every 5 years) by • Retaking the exam• Combination of experience, education, and service to
the profession
SCEA Certification
• SCEA certification exam will be given in Huntsville in mid-April and also at the national conference in June
Society of Cost Estimating and Analysis (SCEA)
Greater Alabama Chapter
Dr. H. Samuel CookeCertified Cost Estimator/Analyst, Society of Cost Estimating and Analysis (SCEA)
Professional Designation in Cost and Price Analysis, Air Force Institute of Technology (AFIT)Honor Graduate, Cost Analysis for Decision Making, Army Logistics Management Center, Ft. Lee, VA
SCEA Points of Contact
E-mail Sam Cooke at [email protected] if you are interested in attending the training Sessions.
E-mail Linda Adams at [email protected]
to be added to the SCEA distribution list to receive
the local chapter’s free monthly newsletter.
Cost Risk
““The only certainty is uncertainty”The only certainty is uncertainty” Pliny the
Elder (Gaius Plinius Secundus), AD 23-79,
Roman Senator, Imperial Fleet Commander,
Historian; died at the Mt. Vesuvius Eruption
Basic Terminology• Risk is the chance of uncertainty or loss. In a situation
that includes potentially favorable and unfavorable events, risk is the probability that an unfavorable event occurs.
• Uncertainty is the indefiniteness about the outcome of a situation. Uncertainty includes both favorable and unfavorable events.
• Cost Risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded.
• Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical, performance and programmatic input variables.
• Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events.
Probability
• Probability is the branch of mathematics used for the quantification of cost risk
• Basic terms– Probability Density Function (PDF) : describes a
range of values and their associated probabilities
– Cumulative Distribution Function (CDF) : describes a range of value and their associated cumulative probabilities; also called an “S-curve”
PDF CDF
PercentilesPercentiles
3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000 15,000
10th
20th30th
40th50th
60th
70th
80th
90th
Percentiles for an example Lognormal distribution:
Why Risk Analysis?
• There is uncertainty about each cost element, and it is usually not symmetric
• Cost elements are correlated• For these and other reasons, a point
estimate is likely to be much less than the median risk estimate and thus underestimate cost by a large amount– Point estimate is often less than the 30th
percentile, which means that the probability that the actual cost will exceed that estimate is at least 70%
Point Estimates Vs. Risk Estimates – Example*
Estimated Total Cost, in Millions (2004$)
DDT&E Flight Unit Production TotalPoint Estimate 6380.7 517.9 669.7 7050.5CER Risk Only - Mean 7117.6 480.5 642.6 7801.8Full Risk - Mean 7893.7 649.0 809.1 8743.0Point Esimtate + 30% 8294.9 673.3 870.6 9165.7Full Risk - 70th Percentile 8793.9 798.0 960.8 9701.1
Sources of Cost Risk Security Risks
Critical failure modes
Energy / Environmental Risks
Schedule problems and delays
Inadequate cost estimates
Process (need to assess contractor’s assumptions)
Models
“New Ways of Doing Business”
Inflation
Systems Engineering
Cost Improvement Curve Assumptions
State-of-the-Art-Advance (Technology Readiness)
Technical Risk Sources
Physical properties
Material Properties
Radiation Properties (emission and reception)
Material Availability Risks
Testing / Modeling Risks
Integration / Interface Risks
Program Personnel
Safety Risks
Software Design Risks
Historical cost data available
Amount of cost risk depends on the Basis of the Estimate
Inflation RiskInflation Bounds with Percentiles
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
Fiscal Year
Infl
atio
n M
ult
iplie
r
DOD 3600
80%
70%
60%
40%
30%
20%
Modeling Sources of Cost Risk
• Two common sources of uncertainty explicitly addressed in cost risk estimates are technical risk and estimation risk– Technical risk is associated with uncertainty in
model inputs• Weight, Technical and Management Parameters, etc.
– Estimation risk is associated with uncertainty in the estimation tools
• Cost estimating relationship standard errors, for example
Estimation Risk• Estimating methods usually involve
some uncertainty– For example, cost-estimating
relationships (CERs) that are based on historical data involve a high degree of uncertainty• CERs are equations that relate a variable or
a set of variables that drive the cost (or define the scope of a project) to cost
• Classic example: baXY where Y represents cost and X represents weight
Estimation Risk
• Since CERs are typically based on historical data, there is uncertainty associated with the equation’s goodness-of-fit to the historical data
• This can be measured by the standard deviation of the actuals versus the estimates for the data used in developing the CER
CER Risk
$
Cost Driver (Weight)
Cost = a + bXcCost = a + bXc
Inputvariable
CostEstimate
Historical data point
Cost estimating relationship
Standard percent error bounds
Technical Risk
• Define a triangular distribution about each estimate with the minimum, most likely, and maximum values
OptimisticCost
Best-Estimate Cost (Mode)
Cost Implication of Technical, Programmatic Assessment
DE
NS
ITY
L M H$
OptimisticCost
Best-Estimate Cost (Mode)
Cost Implication of Technical, Programmatic Assessment
DE
NS
ITY
L M H$
DE
NS
ITY
L M H$
Technical Risk – at the WBS Level
For each WBS element (D&D and flight unit cost):
For each CER input, define a triangular distribution using minimum value, most likely value, and maximum value
WL WM WH
Weight
DL = DM DH
New Design
Correlation Among Cost Elements
• There is correlation between hardware elements – A problem that results in an increase in
structures costs may cause an increase in thermal control costs
• There is correlation between hardware elements and systems level costs– Systems level costs are often a function of
hardware cost
Importance of Correlation
• Cost risks are often measured by the standard deviation of the total risk
• Not accounting for correlation will underestimate the total cost standard deviation– For a 10-element WBS, not accounting
for correlation will underestimate risk by as much as 70%
The Importance of Correlation - Illustration
Percent that Total-Cost Sigma is underestimated when
correlation assumed to be 0 instead of r given n WBS
elements
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Perc
en
t U
nd
ere
sti
mate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Perc
en
t U
nd
ere
sti
mate
d
n = 10
n = 30
n = 100n = 1000
Aggregation
+
+.
WBS Element 1
WBS Element 2
Total Hardware Cost
At Summation Levels
..
Risk – The Big Picture
X
CER Risk
Technical Risk
X
CER Risk
Technical Risk
X
CER Risk
Technical Risk
.
.
.
Subsystem 1
Subsystem 2
Subsystem N
.
.
.
222 ,
2NN ,
1
1
1
321
23212
13121
...
...............
...
...
,,,
,,,
,,,
NNN
N
N
211 ,
Correlation Matrix
N
j
j
ijiji
N
ii
N
ii
VarianceCostTotal
MeanCostTotal
2
1
11
2
1
2
,
Methods for Aggregation
• Monte Carlo – Uses simulation– Computationally complex
• Must be careful to calculate correlations correctly
– Method used in ACE-IT
• Analytic Approximation– Uses method of moments– Computationally simple– Accuracy similar to Monte Carlo– Method used in the NASA/Air Force Cost Model
(NAFCOM)
NAFCOM Example - Inputs
• The capability to define triangular distributions for all cost driver inputs and for cost thruputs is made available when Risk is turned On.
NAFCOM Example - Outputs• The final result is
uncertainty distributions for DDT&E, Flight Unit, Production and Total Cost.
• Result data includes summary statistics, probability densities, and cumulative distributions for each major estimating element (i.e. stage, bus)
Risk Management
Cost Risk Seminar
• Location– Teledyne Brown Engineering, Huntsville, AL
• Date– March 15, 2005, 8:00 AM to 5:00 PM
• Price is $150 for SCEA and INCOSE members, and Teledyne Brown employees, $200 for others– Payment due February 28th
• Lunch and refreshments will be provided at no additional charge
• CEU credit will be awarded for attending the seminar
Cost Risk Seminar
• Seminar is self-contained, presupposes no special knowledge of cost analysis or risk
• All necessary background material is covered in the course
• What you will gain from attending the seminar– Understanding of the basics of cost risk analysis– Some advanced topics covered also treated,
material is broad in scope– All attendees receive a hard-copy of the training
materials, 200+ pages of Powerpoint charts
Cost Risk Seminar
• Presenter is Paul Garvey– Chief Scientist, Mitre Corp.– Internationally recognized expert in cost risk – Author of Probability Methods for Cost
Uncertainty Analysis: A Systems Engineering Perspective(http://www.dekker.com/servlet/product/productid/8966-0)
– Winner of several best paper awards at the annual Dept. of Defense Cost Analysis Symposium
Cost Risk Seminar - Outline
Section A. Introduction • Part I. Introduction and Historical
Perspective• Part II. The Problem Space• Part III. Presenting Cost as a Probability
Distribution• Part IV. Developing Cost as a Probability
Distribution• Part V. Using the Cost Probability Distribution• Part VI. Issues and Considerations• Part VII. Benefits of Cost Uncertainty Analysis
Cost Risk Seminar – Outline (continued)
Section B. Theory Fundamentals• Part I. Concepts of Probability Theory• Part II. Random Variables, Distributions, and the
Theory of Expectation • Part III. Special Distributions for Cost Uncertainty
AnalysisSpecial Technical Topics• Part IV. The Central Limit Theorem and a Cost
Analysis Perspective• Part V. Correlation in Cost Uncertainty Analysis• Part VI. Distribution Function of a System’s Total
Cost
Cost Risk Seminar – Outline (continued)
Section C. ApplicationsSystem Cost Uncertainty Analysis• Part I. Work Breakdown Structures• Part II. An Analytical Framework and Monte
Carlo Simulation• Part III. Case StudyModeling Cost and Schedule Uncertainties• Part IV. Introduction• Part V. Joint Probability Models for Cost-
Schedule• Part VI. Case Study
Cost Risk Seminar – Sample Chart
Treating Cost as a Random Variable
• The cost of a future system can be significantly affected by uncertainty. The existence of uncertainty implies the existence of a range of possible costs. How can a decision-maker be shown the chance a particular cost in the range of possible costs will be realized?
• The probability distribution is a recommended approach for providing this insight. Probability distributions result when independent variables (e.g., weight, power-output, staff-level) used to derive a system’s cost randomly assume values across ranges of possible values. For instance, the cost of a satellite might be derived on the basis of a range of possible weight values, with each value randomly occurring
• This approach treats cost as a random variable. It is a recognition that values for these variables (such as weight) are not typically known with sufficient precision to perfectly predict cost, at a time when such predictions are needed
Cost Risk Seminar – Sample Chart 2
Part VI. Case Study
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 10 20 30 40 50 60
xi1or2
X1
X2
Probability
Figure 4. Simulated versus Theoretical Marginal Distributions
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 5 10 15 20 25 30
Probability
P(X1 a x2 24mos) P(X1 a x2 36mos)P(X1 a)
x1
Left-Most CDF: Mean cost is 7.7 ($M) for a 24 mos. scheduleMiddle CDF: Mean cost is 9.8 ($M) unconditioned scheduleRight-Most CDF: Mean cost is 10.7 ($M) for a 36 mos. schedule
Right-Most CDF: A 24 month schedule has an 80% chance of overrun; 36 month schedule has a 30% chance of overrun