Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 1

Dr. Ricardo ValerdiMassachusetts Institute of Technologyrvalerdi@mit.edu

Workshop on Estimating and Managing Through-Life-CostsBath UniversityNovember 12, 2008

Parametric Cost Modeling for Systems Engineering

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘082

Roadmap(1) Systems Engineering fundamentals;

(2) Explanation of COSYSMO size and cost drivers;

(3) Limitations;

(4) Recent developments;

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

The Delphic Sybil

Michelangelo Buonarroti

Capella Sistina, Il Vaticano (1508-1512)

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

4

Feasibility Plans/Rqts. Design Develop and Test

Phases and Milestones

Relative Size

Range

OperationalConcept

Life Cycle Objectives

Life Cycle Architecture

Initial Operating Capability

x

0.5x

0.25x

4x

2x

Cone of Uncertainty

Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

Cost Commitment on Projects

Detail Designand

Development

100

25

50

75

Conceptual-Preliminary

Design

Constructionand/or

Production

System Use, Phaseout,and Disposal

NEED

% Commitment to Technology,Configuration, Performance, Cost, etc.

Cost Incurred

System-Specific Knowledge

Ease of Change

Blanchard, B., Fabrycky, W., Systems Engineering & Analysis, Prentice Hall, 1998.

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 6

Why measure systems engineering?Cost Overrun as a Function of SE Effort

NASA Data

Honour, E.C., “Understanding the Value of Systems Engineering,” Proceedings of the INCOSE International Symposium, Toulouse, France, 2004.

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

Contract Engineering WBS Based On Standards

1.0 – System/Project1.1 – Integrated Project Management (IPM)

1.2 – Systems Engineering

1.3 – Prime Mission Product (PMP)1.3.1 – Subsystem / Configuration Item (CI) 1…n (Specify Names)

1.3.2 – PMP Application Software

1.3.3 – PMP System Software

1.3.4 – PMP Integration, Assembly, Test & Checkout (IATC)

1.3.5 – Operations/Production Support

1.4 – Platform Integration

1.5 – System Test & Evaluation (ST&E)

1.6 – Training

1.7 – Data Management

1.8 – Peculiar Support Equipment

1.9 – Common Support Equipment

1.10 – Operational / Site Activation

1.11 – Industrial Facilities

Product-oriented construct, by tailoring MIL-

HDBK 881A and ANSI/EIA 632

Six Functions:

1. Systems Engineering

2. Software Engineering

3. Electrical Engineering

4. Mechanical Engineering

5. Support Engineering

6. Project Engineering Management

Six Functions:

1. Systems Engineering

2. Software Engineering

3. Electrical Engineering

4. Mechanical Engineering

5. Support Engineering

6. Project Engineering Management

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 8

COCOMO II• COCOMO is the most widely used, thoroughly

documented and calibrated software cost model• COCOMO - the “COnstructive COst MOdel”

– COCOMO II is the update to COCOMO 1981– ongoing research with annual calibrations made

available • Originally developed by Dr. Barry Boehm and

published in 1981 book Software Engineering Economics

• COCOMO II described in Software Cost Estimation with COCOMO II (Prentice Hall 2000)

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘089

COSYSMO Scope• Addresses first four phases of the

system engineering lifecycle (per ISO/IEC 15288)

• Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632)

Conceptualize DevelopOper Test & Eval

Transition to Operation

Operate, Maintain, or Enhance

Replace orDismantle

Valerdi, R., The Constructive Systems Engineering Cost Model: Quantifying the Costs of Systems Engineering Effort in Complex Systems, VDM Verlag, 2008

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 10

How is Systems Engineering Defined?• Acquisition and Supply

– Supply Process– Acquisition Process

• Technical Management– Planning Process– Assessment Process– Control Process

• System Design– Requirements Definition Process– Solution Definition Process

• Product Realization– Implementation Process– Transition to Use Process

• Technical Evaluation

– Systems Analysis Process

– Requirements Validation Process

– System Verification Process

– End Products Validation Process

EIA/ANSI 632, Processes for Engineering a System, 1999.

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 11

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

+3 Volatility Factors

- Application factors-8 factors

- Team factors-6 factors

- Schedule driver WBS guided by EIA/ANSI 632

COSYSMO Operational Concept

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 12

Where: PMNS = effort in Person Months (Nominal Schedule)

A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN}wx = weight for “easy”, “nominal”, or “difficult” size driver

= quantity of “k” size driverE = represents diseconomy of scaleEM = effort multiplier for the jth cost driver. The geometric product results in an

overall effort adjustment factor to the nominal effort.

x

Model Form

14

1,,,,,, )(

jj

E

kkdkdknknkekeNS EMwwwAPM

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

COSYSMO Data SourcesBoeing Integrated Defense Systems (Seal Beach, CA)

Raytheon Intelligence & Information Systems (Garland, TX)

Northrop Grumman Mission Systems (Redondo Beach, CA)

Lockheed Martin Transportation & Security Solutions (Rockville, MD)

Integrated Systems & Solutions (Valley Forge, PA)

Systems Integration (Owego, NY)

Aeronautics (Marietta, GA)

Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY)

General Dynamics Maritime Digital Systems/AIS (Pittsfield, MA)Surveillance & Reconnaissance Systems/AIS (Bloomington, MN)

BAE Systems National Security Solutions/ISS (San Diego, CA)

Information & Electronic Warfare Systems (Nashua, NH)

SAIC Army Transformation (Orlando, FL)

Integrated Data Solutions & Analysis (McLean, VA)

L-3 Communications Greenville, TX

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 14

Academic prototype

Commercial Implementations

Proprietary Implementations

COSYSMO-R

SECOST

SEEMaP

ImpactAcademic Curricula

Intelligence CommunitySheppard Mullin, LLC

Policy & Contracts

Model

14

1,,,,,, )(

jj

E

kkdkdknknkekeNS EMwwwAPM

10 theses

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 15

4 Size Drivers1. Number of System Requirements

2. Number of System Interfaces

3. Number of System Specific Algorithms

4. Number of Operational Scenarios

Weighted by complexity, volatility, and degree of reuse

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 16

Counting Rules ExampleCOSYSMO example for sky, kite, sea,

and underwater levels where:

Sky level: Build an SE cost model

Kite level: Adopt EIA 632 as the WBS and ISO 15288 as the life cycle standard

Sea level: Utilize size and cost drivers, definitions, and counting rules

Underwater level: Perform statistical analysis of data with software tools and implement model in Excel

Source: Cockburn 2001

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 17

14 Cost Drivers

1. Requirements understanding

2. Architecture understanding

3. Level of service requirements

4. Migration complexity

5. Technology Risk

6. Documentation Match to Life Cycle Needs

7. # and Diversity of Installations/Platforms

8. # of Recursive Levels in the Design

Application Factors (8)

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 18

14 Cost Drivers (cont.)

1. Stakeholder team cohesion

2. Personnel/team capability

3. Personnel experience/continuity

4. Process capability

5. Multisite coordination

6. Tool support

Team Factors (6)

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 19

Cost Driver Rating ScalesVery Low Low Nominal High Very High

Extra High EMR

Requirements Understanding 1.87 1.37 1.00 0.77 0.60   3.12

Architecture Understanding 1.64 1.28 1.00 0.81 0.65   2.52

Level of Service Requirements 0.62 0.79 1.00 1.36 1.85   2.98

Migration Complexity     1.00 1.25 1.55 1.93 1.93

Technology Risk 0.67 0.82 1.00 1.32 1.75   2.61

Documentation 0.78 0.88 1.00 1.13 1.28   1.64

# and diversity of installations/platforms     1.00 1.23 1.52 1.87 1.87

# of recursive levels in the design 0.76 0.87 1.00 1.21 1.47   1.93

Stakeholder team cohesion 1.50 1.22 1.00 0.81 0.65   2.31

Personnel/team capability 1.50 1.22 1.00 0.81 0.65   2.31

Personnel experience/continuity 1.48 1.22 1.00 0.82 0.67   2.21

Process capability 1.47 1.21 1.00 0.88 0.77 0.68 2.16

Multisite coordination 1.39 1.18 1.00 0.90 0.80 0.72 1.93

Tool support 1.39 1.18 1.00 0.85 0.72   1.93

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 20

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

ISO/IEC 15288

Conceptualize DevelopTransition to

Operation

Acquisition & Supply

Technical Management

System Design

Product Realization

Technical Evaluation

Operational Test &

Evaluation

AN

SI/E

IA 6

32

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 22

Limitations of the model1. Mostly qualitative drivers2. Variance of Delphi responses3. Small sample size4. Aerospace-heavy5. Calibration is biased by successful

projects because successful projects share data, bad ones don’t

6. Model will not work outside of calibrated range

7. A fool with a tool is still a fool

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 23

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

Reuse Continuum

Modified vs. New Threshold

Modified

Adopted

New 1.0

0

Deleted

Managed

Reu

se w

eigh

t0.65

0.510.43

0.15

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

Lessons Learned from Development

Lesson #1: Scope of the model

Lesson #2: Types of projects needed for data collection effort

Lesson #3: Size drivers

Lesson #4: Effort Multiplier

Lesson #5: Systems Engineering hours across life cycle stages

Lesson #6: Data collection form

Lesson #7: Definition

Lesson #8: Significance vs. data availability

Lesson #9: Influence of data on the drivers and statistical significance

Lesson #10: Data safeguarding procedure

Lesson #11: Buy-in from constituents

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

Lessons Learned from ValidationLesson #1: Skills Needed to use COSYSMO

Provide a list of assumptions/prerequisites for model use as well as the appropriate training/resources for COSYSMO understanding

Lesson #2: Model Usability

Understanding usability will lead to more reliable inputs to the model especially at the early phases of the life cycle where there is little project information available

Lesson #3: Model Adoption

Providing organizations with a sequential process driven by implementation experience will facilitate the adoption of COSYSMO

Lesson #4: Accounting for Reuse

Providing a way to account for reuse in systems engineering is essential for improving the accuracy of the model

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

Rechtin’sSystems

ArchitectingHeuristics

COSYSMOModel

COSYSMOTool

Model Development Heuristics

Model CalibrationHeuristics

Model Usage

Heuristics

Cost Estimation Heuristics

inspired implemented in

experiencelead to

confirmed

Experiential Closed Loop

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08

Estimating-related HeuristicHeuristic #30: Models are optimistic.

Heuristic #31: People are generally optimistic.

Koehler, D. J., Harvey, N. (1997). Confidence judgments by actors and observers. Journal of Behavioral Decision Making. 10, 221-

242.

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08% time added for architecture and risk resolution

% t

ime

add

ed t

o o

vera

ll sc

hed

ule

% of project schedule devoted to initial architecture and risk resolution

Added schedule devoted to rework (COCOMO II RESL factor)

Total % added schedule

Sweet Spot

0

20

40

60

80

100

0 10 20 30 40 50

10,000KSLOC

100KSLOC

10KSLOC

Bath University Workshop on Estimating and Managing Through-Life-Costs Nov ‘08 30

ContactRicardo Valerdi

MIT Lean Advancement Initiative

rvalerdi@mit.edu

(617) 253-8583

www.valerdi.com/cosysmo