proxy estimation costing for systems (pecs)

Example © 2012 Lockheed Martin Corporation. All Rights Reserved.

October 2012

Proxy Estimation Costing for Systems (PECS)

Reggie ColeLockheed Martin Senior Fellow

2


Discussion Topics

Why Do We Need Yet Another Cost Model?– The gap in early-stage system cost modeling

Systems Engineering Effort as a Proxy Estimator for System Cost

– And the role of COSYSMO is arriving at this proxy estimate

Proxy Estimation Costing for Systems (PECS)– Derivation of the PECS Model– The PECS modeling approach

Case Study for Affordability Analysis Using the PECS Model– The real power of the PECS model

3


Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy – So Does Solution Information

Problem-Space Description

Cost Estimate ± 25%

High-Level Solution Description


Detailed Solution Description

Cost Estimate ± 5%

High-Level Solution Assumptions


Increasing Effort and Cost-Modeling Expertise

Increasingly Refined

Information About the

Solution

Increasingly Refined Cost Estimate

Incr

easin

gly R

efin

ed S

olut

ion

We Have a Good Selection of Tools for Late-Stage Cost Modeling

We Have Gaps in Early-Stage Cost Modeling

4


Systems Engineering Effort as a Proxy Measure of Overall System Size and Complexity

Proxy Measures– Proxy measures are used when you cannot directly measure what you

want to measure – and when an indirect measure provides sufficient insight– Proxy measures are often used in clinical studies since direct measurement

is often infeasible or can even alter the outcome– It is not always possible to directly measure what you want to measure – or

directly estimate what you want to estimate

System Engineering Effort is a Proxy Measure for System Cost– There is strong evidence for the link between systems engineering effort

and program cost – dating back to a NASA study in the 1980s– The optimal relationship between systems engineering effort and overall

program cost is 10% - 15%– Industry has long used a parametric relationship between software cost

and systems engineering cost for software-intensive systems– Systems engineering effort can be an effective proxy measure for overall

system cost

H. Dickinson, S. Hrisos, M. Eccles, J. Francis, M. Johnston, Statistical Considerations in a Systematic Review of Proxy Measures of Clinical Behaviour, Implementation Science, 2010E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004

5


COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Size, Complexity and Reuse

Number of System RequirementsNumber of Major System Interfaces

Number of Critical AlgorithmsNumber of Operational Scenarios

Size Drivers

Requirements Understanding Architecture Understanding

Level of Service Requirements

Migration Complexity

Technology Risk

Level of Documentation Required

Diversity of Installed Platforms

Level of Design RecursionStakeholder Team Cohesion

Personnel / Team Capability

Personnel Experience / Continuity

Process Capability

Multisite Coordination

Level of Tool Support

Cost Drivers

Managed ElementsAdopted ElementsDeleted ElementsModified Elements

New Elements

Reuse FactorsInitial Estimate of System Size

Scaled Estimate of System Size

Consolidated Cost Driver Factor

Estimate of Systems Engineering Effort…Also a Biased Proxy Estimator for System Scope…And System Cost

6


An Approach for De-Biasing the Proxy Estimator –Relationship Between SE Effort and Total Effort

Total Program Overrun32 NASA Programs

R2 = 0.5206

0

20

40

60

80

100

120

140

160

180

200

0 5 10 15 20

Definition Percent of Total Estimate

Prog

ram

Ove

rrun

Definition $Definition Percent = ---------------------------------- Target + Definition$

Actual + Definition$Program Overrun = ---------------------------------- Target + Definition$

0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24% 28%

SE Effort = SE Quality * SE Cost/Actual Cost

Act

ual/P

lann

ed C

ost

NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort

W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,” Internal Presentation, NASA Comptroller’s Office, 1992

E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004

INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort

7


The PECS Cost Function

c)(Stochasti Costs Travel : c)(Stochasti Costs Material :

c)(Stochasti RateLabor : c)(Stochasti COSYSMO UsingComputedEffort SE :

c)(StochastiEffort Program Total Effort to SE Convertingfor Factor : stic)(DeterminiFactor n Calibratio COSYSMO :

:Where

Travel

Materials

Labor

SE

Conv

Cal

TravelMaterialsConv

LaborCalSESystem

CostCostRateEffortFF

CostCostFRateFEffortCost

Variable Type DescriptionCOSYSMO Calibration Factor Deterministic Scalar Value Organization-specific calibration factor

Effort Conversion Factor Triangular Distributed Random Variable Three-point estimate of factor to convert SE effort to total program effort (nominally 0.08, 0.12 and 0.16)

SE Effort Triangular Distributed Random Variable Three-point estimate for SE effort, generated using COSYSMO

Labor Rate Triangular Distributed Random Variable Three-point estimate for composite labor rate

Material Costs Triangular Distributed Random Variable Three-point estimate for material costs

Travel Costs Triangular Distributed Random Variable Three-point estimate for travel costs

This Model is Well Positioned for Monte Carlo Analysis

8


The PECS Model – Putting It All Together

Size Drivers (Problem Space) Customer Requirements System Interfaces Major Algorithms Operational Scenarios

Complexity Drivers (Problem/Solution) Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Documentation Needs Installations/Platform Diversity Levels of Recursion in the Design Stakeholder Team Cohesion Personnel/Team Capability Personnel Experience/Continuity Process Capability Multisite Coordination Tool Support

Reuse Factors (Solution Space) New Modified Deleted Adopted Managed

0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24% 28%

SE Effort = SE Quality * SE Cost/Actual Cost

Act

ual/P

lann

ed C

ost

SE Effort is an estimator for total system cost…but it is a biased estimator

Estimator Bias Function is Based on the Well-Established Relationship Between SE Effort and Overall Program Effort

Proxy Estimation Costing for Systems (PECS)

Estimator De-Biasing

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100

Cost

Estia

te ($

M)

Confidence

Monte Carlo Analysis of System Cost

Three different COSYMO scenarios – optimistic, expected & pessimistic – provide the basis for the Monte Carlo analysis of system cost

TravelMaterialsConv

LaborCalSESystem CostCost

FRateFEffortCost

9


Case Study – The COSYSMO Scenarios

The case study is based on a large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers.The case study looks at three cost scenarios: Case 1 – The original unprecedented system (for calibration purposes) Case 2 – Replacement system (as a new development) Case 3 – Replacement system (as a largely COTS/GOTS approach)

0.00

2.00

4.00

6.00

8.00

10.00

0

500

1000

1500

2000

Pessimistic Expected Optimistic

Cost

Driv

er F

acto

r

Size

(Effe

ctive

Req

uire

men

ts)

Case 1 - Large Unprecedented System

Requirements System I/F

Algorithms Scenarios

Cost Driver Factor

0.00

0.20

0.40

0.60

0.80

0

200

400

600

800

1000

1200


Cost

Driv

er F

acto

r

Size

(Effe

ctive

Req

uire

men

ts)

Case 2 - Replacement System (Developed)



Cost Driver Factor

0

0.1

0.2

0.3

0.4

0.5

0.6

0

100

200

300

400

500

600


Cost

Driv

er F

acto

r

Size

(Effe

ctive

Req

uire

men

ts)

Case 3 - Replacement System (COTS/GOTS)



Cost Driver Factor

COSYSMO Scenarios for PECS – Three Scenarios for Each Case

10


Case Study – The Monte Carlo Analysis

0.00

500.00

1000.00

1500.00

2000.00

2500.00

3000.00

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100

Cost

Estia

te ($

M)

Confidence

Case 1 - Large Unprecedented System

0.00

20.00

40.00

60.00

80.00

100.00

120.00

140.00

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100

Cost

Estia

te ($

M)

Confidence

Case 2 - Replacement System (Development)

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

45.00

50.00

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100

Cost

Estia

te ($

M)

Confidence

Case 3 - Repacement System (COTS/GOTS)

Case 1Average 80/20 Cost = $1.9BUsed as a calibration point for the model

Case 2Average 80/20 Cost = $77MInitial Solution for Replacement System

Case 3Average 80/20 Cost = $30M More Affordable Solution, Based on COTS/GOTS Solution

The PECS Model Enables Rapid Turn-Around Analysis of Alternatives and “Should Cost” Analysis

11


Conclusion

The PECS Model is Based on Well-Established Approaches– COSYSMO provides the basis for estimation of systems engineering effort

– and a biased proxy estimator for overall system cost– There is a well-established relationship between systems engineering effort

and overall effort used to de-bias the COSYSMO-modeled effort– Monte Carlo analysis is a well-established technique for cost modeling

The PECS Model Can Improve System Cost Modeling– The PECS Model occupies an important niche – fully parametric system

cost modeling in the early stages of system definition– The PECS Model can serve as a powerful affordability analysis tool –

supporting rapid-turnaround analysis of alternatives– But…the PECS Model is not a replacement for existing models

Next Steps– Broader validation of the model– Cross-industry review of the model

12


proxy estimation costing for systems (pecs)

Documents