cost risk – an overview dr. christian smart christian.b.smart@saic.com president, greater alabama...

Cost Risk – An Overview

Dr. Christian Smartchristian.b.smart@saic.com

President, Greater Alabama SCEA ChapterFebruary 17th, 2005

Agenda

• Introduction to Society of Cost Estimating and Analysis (SCEA)

• Cost Risk Overview• Upcoming Cost Risk Seminar

Society of Cost Estimating and Analysis (SCEA)

• Who we are– Organization for cost analysis professionals, a highly

diverse community that includes people who work in areas such as:

• Budget analysis• Earned value management• Cost estimation• Statistical analysis• Operations research• Accounting• Etc.

– Overlap with other professional organizations such as INCOSE and PMI

– Huntsville chapter is one of the largest and most active in the nation (over 150 members)

Society of Cost Estimating and Analysis (SCEA)

• What we offer– National Organization

• Hosts an annual conference and educational workshop– 2005 conference will be held in Denver, Colorado,

June 13th-17th, jointly with the International Society for Parametric Analysts (ISPA)

(http://www.ispa-cost.org/conf.htm)• Publishes National Estimator magazine and Journal of

Cost Analysis and Management, a refereed technical publication

• Provides the Certified Cost Estimator/Analyst professional credential

• Web site (http://www.sceaonline.net)

Society of Cost Estimating and Analysis (SCEA)

• What we offer– Local Chapter

• Monthly luncheons with presentations on topics of interest to the cost analysis community

– Some recent presentation topics include:» Decision Making Using Cost Risk Analysis» Nonparametric Regression in Cost Analysis » Schedule Risk Assessment » Earned Value Management Systems Concepts

• Chapter web site– http://www.scea-alabama.org/– includes full presentations dating back to 2003 and

other links of interest• Free training and materials for certification

preparation• Seminars and other forms of training

SCEA Certification

• Certified Cost Estimator/Analyst (CCE/A) professional credential– Recognized credential throughout the profession– Government procurements often require CCE/As– Certification by examination

• Two years of professional experience in cost analysis/estimating required to take the exam

– Recertification (every 5 years) by • Retaking the exam• Combination of experience, education, and service to

the profession

SCEA Certification

• SCEA certification exam will be given in Huntsville in mid-April and also at the national conference in June

Society of Cost Estimating and Analysis (SCEA)

Greater Alabama Chapter

Dr. H. Samuel CookeCertified Cost Estimator/Analyst, Society of Cost Estimating and Analysis (SCEA)

Professional Designation in Cost and Price Analysis, Air Force Institute of Technology (AFIT)Honor Graduate, Cost Analysis for Decision Making, Army Logistics Management Center, Ft. Lee, VA

SCEA Points of Contact

E-mail Sam Cooke at cooke@westar.com if you are interested in attending the training Sessions.

E-mail Linda Adams at linda.d.adams@saic.com

to be added to the SCEA distribution list to receive

the local chapter’s free monthly newsletter.

Cost Risk

““The only certainty is uncertainty”The only certainty is uncertainty” Pliny the

Elder (Gaius Plinius Secundus), AD 23-79,

Roman Senator, Imperial Fleet Commander,

Historian; died at the Mt. Vesuvius Eruption

Basic Terminology• Risk is the chance of uncertainty or loss. In a situation

that includes potentially favorable and unfavorable events, risk is the probability that an unfavorable event occurs.

• Uncertainty is the indefiniteness about the outcome of a situation. Uncertainty includes both favorable and unfavorable events.

• Cost Risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded.

• Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical, performance and programmatic input variables.

• Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events.

Probability

• Probability is the branch of mathematics used for the quantification of cost risk

• Basic terms– Probability Density Function (PDF) : describes a

range of values and their associated probabilities

– Cumulative Distribution Function (CDF) : describes a range of value and their associated cumulative probabilities; also called an “S-curve”

PDF CDF

PercentilesPercentiles

3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000 15,000

10th

20th30th

40th50th

60th

70th

80th

90th

Percentiles for an example Lognormal distribution:

Why Risk Analysis?

• There is uncertainty about each cost element, and it is usually not symmetric

• Cost elements are correlated• For these and other reasons, a point

estimate is likely to be much less than the median risk estimate and thus underestimate cost by a large amount– Point estimate is often less than the 30th

percentile, which means that the probability that the actual cost will exceed that estimate is at least 70%

Point Estimates Vs. Risk Estimates – Example*

Estimated Total Cost, in Millions (2004$)

DDT&E Flight Unit Production TotalPoint Estimate 6380.7 517.9 669.7 7050.5CER Risk Only - Mean 7117.6 480.5 642.6 7801.8Full Risk - Mean 7893.7 649.0 809.1 8743.0Point Esimtate + 30% 8294.9 673.3 870.6 9165.7Full Risk - 70th Percentile 8793.9 798.0 960.8 9701.1

Sources of Cost Risk Security Risks

Critical failure modes

Energy / Environmental Risks

Schedule problems and delays

Inadequate cost estimates

Process (need to assess contractor’s assumptions)

Models

“New Ways of Doing Business”

Inflation

Systems Engineering

Cost Improvement Curve Assumptions

State-of-the-Art-Advance (Technology Readiness)

Technical Risk Sources

Physical properties

Material Properties

Radiation Properties (emission and reception)

Material Availability Risks

Testing / Modeling Risks

Integration / Interface Risks

Program Personnel

Safety Risks

Software Design Risks

Historical cost data available

Amount of cost risk depends on the Basis of the Estimate

Inflation RiskInflation Bounds with Percentiles

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Fiscal Year

Infl

atio

n M

ult

iplie

r

DOD 3600

80%

70%

60%

40%

30%

20%

Modeling Sources of Cost Risk

• Two common sources of uncertainty explicitly addressed in cost risk estimates are technical risk and estimation risk– Technical risk is associated with uncertainty in

model inputs• Weight, Technical and Management Parameters, etc.

– Estimation risk is associated with uncertainty in the estimation tools

• Cost estimating relationship standard errors, for example

Estimation Risk• Estimating methods usually involve

some uncertainty– For example, cost-estimating

relationships (CERs) that are based on historical data involve a high degree of uncertainty• CERs are equations that relate a variable or

a set of variables that drive the cost (or define the scope of a project) to cost

• Classic example: baXY where Y represents cost and X represents weight

Estimation Risk

• Since CERs are typically based on historical data, there is uncertainty associated with the equation’s goodness-of-fit to the historical data

• This can be measured by the standard deviation of the actuals versus the estimates for the data used in developing the CER

CER Risk

$

Cost Driver (Weight)

Cost = a + bXcCost = a + bXc

Inputvariable

CostEstimate

Historical data point

Cost estimating relationship

Standard percent error bounds

Technical Risk

• Define a triangular distribution about each estimate with the minimum, most likely, and maximum values

OptimisticCost

Best-Estimate Cost (Mode)

Cost Implication of Technical, Programmatic Assessment

DE

NS

ITY

L M H$

OptimisticCost

Best-Estimate Cost (Mode)

Cost Implication of Technical, Programmatic Assessment

DE

NS

ITY

L M H$

DE

NS

ITY

L M H$

Technical Risk – at the WBS Level

For each WBS element (D&D and flight unit cost):

For each CER input, define a triangular distribution using minimum value, most likely value, and maximum value

WL WM WH

Weight

DL = DM DH

New Design

Correlation Among Cost Elements

• There is correlation between hardware elements – A problem that results in an increase in

structures costs may cause an increase in thermal control costs

• There is correlation between hardware elements and systems level costs– Systems level costs are often a function of

hardware cost

Importance of Correlation

• Cost risks are often measured by the standard deviation of the total risk

• Not accounting for correlation will underestimate the total cost standard deviation– For a 10-element WBS, not accounting

for correlation will underestimate risk by as much as 70%

The Importance of Correlation - Illustration

Percent that Total-Cost Sigma is underestimated when

correlation assumed to be 0 instead of r given n WBS

elements

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Perc

en

t U

nd

ere

sti

mate

d

n = 10

n = 30

n = 100n = 1000

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Actual Correlation

Perc

en

t U

nd

ere

sti

mate

d

n = 10

n = 30

n = 100n = 1000

Aggregation

+

+.

WBS Element 1

WBS Element 2

Total Hardware Cost

At Summation Levels

..

Risk – The Big Picture

X

CER Risk

Technical Risk

X

CER Risk

Technical Risk

X

CER Risk

Technical Risk

.

.

.

Subsystem 1

Subsystem 2

Subsystem N

.

.

.

222 ,

2NN ,

1

1

1

321

23212

13121

...

...............

...

...

,,,

,,,

,,,

NNN

N

N

211 ,

Correlation Matrix

N

j

j

ijiji

N

ii

N

ii

VarianceCostTotal

MeanCostTotal

2

1

11

2

1

2

,

Methods for Aggregation

• Monte Carlo – Uses simulation– Computationally complex

• Must be careful to calculate correlations correctly

– Method used in ACE-IT

• Analytic Approximation– Uses method of moments– Computationally simple– Accuracy similar to Monte Carlo– Method used in the NASA/Air Force Cost Model

(NAFCOM)

NAFCOM Example - Inputs

• The capability to define triangular distributions for all cost driver inputs and for cost thruputs is made available when Risk is turned On.

NAFCOM Example - Outputs• The final result is

uncertainty distributions for DDT&E, Flight Unit, Production and Total Cost.

• Result data includes summary statistics, probability densities, and cumulative distributions for each major estimating element (i.e. stage, bus)

Risk Management

Cost Risk Seminar

• Location– Teledyne Brown Engineering, Huntsville, AL

• Date– March 15, 2005, 8:00 AM to 5:00 PM

• Price is $150 for SCEA and INCOSE members, and Teledyne Brown employees, $200 for others– Payment due February 28th

• Lunch and refreshments will be provided at no additional charge

• CEU credit will be awarded for attending the seminar

Cost Risk Seminar

• Seminar is self-contained, presupposes no special knowledge of cost analysis or risk

• All necessary background material is covered in the course

• What you will gain from attending the seminar– Understanding of the basics of cost risk analysis– Some advanced topics covered also treated,

material is broad in scope– All attendees receive a hard-copy of the training

materials, 200+ pages of Powerpoint charts

Cost Risk Seminar

• Presenter is Paul Garvey– Chief Scientist, Mitre Corp.– Internationally recognized expert in cost risk – Author of Probability Methods for Cost

Uncertainty Analysis: A Systems Engineering Perspective(http://www.dekker.com/servlet/product/productid/8966-0)

– Winner of several best paper awards at the annual Dept. of Defense Cost Analysis Symposium

Cost Risk Seminar - Outline

Section A. Introduction • Part I. Introduction and Historical

Perspective• Part II. The Problem Space• Part III. Presenting Cost as a Probability

Distribution• Part IV. Developing Cost as a Probability

Distribution• Part V. Using the Cost Probability Distribution• Part VI. Issues and Considerations• Part VII. Benefits of Cost Uncertainty Analysis

Cost Risk Seminar – Outline (continued)

Section B. Theory Fundamentals• Part I. Concepts of Probability Theory• Part II. Random Variables, Distributions, and the

Theory of Expectation • Part III. Special Distributions for Cost Uncertainty

AnalysisSpecial Technical Topics• Part IV. The Central Limit Theorem and a Cost

Analysis Perspective• Part V. Correlation in Cost Uncertainty Analysis• Part VI. Distribution Function of a System’s Total

Cost

Cost Risk Seminar – Outline (continued)

Section C. ApplicationsSystem Cost Uncertainty Analysis• Part I. Work Breakdown Structures• Part II. An Analytical Framework and Monte

Carlo Simulation• Part III. Case StudyModeling Cost and Schedule Uncertainties• Part IV. Introduction• Part V. Joint Probability Models for Cost-

Schedule• Part VI. Case Study

Cost Risk Seminar – Sample Chart

Treating Cost as a Random Variable

• The cost of a future system can be significantly affected by uncertainty. The existence of uncertainty implies the existence of a range of possible costs. How can a decision-maker be shown the chance a particular cost in the range of possible costs will be realized?

• The probability distribution is a recommended approach for providing this insight. Probability distributions result when independent variables (e.g., weight, power-output, staff-level) used to derive a system’s cost randomly assume values across ranges of possible values. For instance, the cost of a satellite might be derived on the basis of a range of possible weight values, with each value randomly occurring

• This approach treats cost as a random variable. It is a recognition that values for these variables (such as weight) are not typically known with sufficient precision to perfectly predict cost, at a time when such predictions are needed

Cost Risk Seminar – Sample Chart 2

Part VI. Case Study

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40 50 60

xi1or2

X1

X2

Probability

Figure 4. Simulated versus Theoretical Marginal Distributions

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30

Probability

P(X1 a x2 24mos) P(X1 a x2 36mos)P(X1 a)

x1

Left-Most CDF: Mean cost is 7.7 ($M) for a 24 mos. scheduleMiddle CDF: Mean cost is 9.8 ($M) unconditioned scheduleRight-Most CDF: Mean cost is 10.7 ($M) for a 36 mos. schedule

Right-Most CDF: A 24 month schedule has an 80% chance of overrun; 36 month schedule has a 30% chance of overrun