1 secost is a tool developed by raytheon and used with permission 10/21/06 copyright © 2006...

1SECOST is a tool developed by Raytheon and used with permission 10/21/06

Copyright © 2006 Raytheon Company

21st International Forum on Systems, Software, and COCOMO Cost Modeling

Systems Engineering Cost Estimation: System-of-Systems

Jon K. Ilseng

November 8, 2006



• Introduction

• Systems Engineering Definitions

• COSYSMO

• SECOST History

• SECOST Capabilities and Functionality

• SECOST Cost Estimation Mode: System-of-Interest

• System-of-Systems Definition

• System-of-Systems Examples

• SECOST Cost Estimation Mode: System-of-Systems

• SECOST Recommended Modifications

• Summary and Conclusions

Agenda


Copyright © 2006 Raytheon CompanyIntroduction

• Systems Engineering Cost Estimates

- Our customers require believable and accurate estimates

- OSD AT&L encouraging defense contractors to find “most accurate and consistent systems engineering cost estimation method”

- Critical that all 5 Raytheon Business Units (IIS, NCS, SAS, IDS, RMS) submit accurate, consistent, and believable cost estimates



• Systems Engineering Cost Estimates- Various cost estimates used past 20 years for a “system-of-interest”

> Heuristic and rule of thumb – Systems Engineer use knowledge & experience to prepare cost estimates; no documented written evidence

> Expert Opinion – From SE Domain Expert; no scientific or historical basis> Case Studies – Provide vital information; no scientific basis for how cost

estimates actually prepared> Top down and design-to-cost – Top-down approach starts at defined system

level; tries to capture SE Tasks but not scientific or historical basis; DTC is designing a system to meet cost targets

> Bottoms-up – Common approach beginning with lowest level cost component and rolls up to highest level for total estimate; resource intensive effort which is time-consuming and many times no actual historical data to justify estimate

> Parametric – Employs cost estimating relationships (CERs); most accurate cost estimation method; provides repeatable and most credible estimation framework; less time-consuming than bottoms-up



• Systems Engineering Cost Estimates- Motivation for improving SE Cost Estimates

> Improved estimation process consistency> Improved accuracy, stakeholder confidence due to historical basis

– Establish cost directly correlated with customer supplied requirements & sizing artifacts

– Parametric Cost Initiative software & hardware study performed a few years back

• Parametric estimates 25-35% less expensive to develop than “bottoms-up”

• Parametric estimates every bit as accurate as “bottoms-up”> Improved cost realism, more quantifiable risks

– Leaner, more supportable cost bids> Increased Trade Space/Reduced Cycle Time

– Some cost estimates require short turn-around time> Better CMMI Support

– CMMI model reveals several requirements for cost estimation model as organization moves from Level 2 through Levels 4 & 5



Formal SE Cost Estimation Supports CMMI

Level 2•Project Planning•Project Monitoring and Control•Consistent WBS•Size and Complexity Drivers•Attribute-Based Estimates

Level 3•Integrated Project Monitoring•Measurement Repository•Monitor Attributes

•Actuals vs Plan

Level 4•Organizational Process Performance•Quantitative Project Management•Data Collection•Parametric Cost Estimation Model•Local Model Calibration

Level 5•Organizational Innovation and Deployment• SE Parametric Modeling Represents an Innovative Approach

Introduction



• Parametric Cost Estimation Method- October 2003 OSD SE Summit

> OSD’s position is parametric-based estimates are recommended technique for preparing SE Cost Estimates

- Why Parametric Cost Estimation Method> Provides a credible source> Shortens cost estimating cycle times> Creates more easily defended negotiation position with customer> Reduces customer-approval cycle times> Uses historical data to improve quality of cost estimates> Establishes greater consistency in cost estimating process> DCAA/DCMA Government agencies very supportive of parametrics


Copyright © 2006 Raytheon CompanySystems Engineering Definitions

• Systems Engineering- Various definitions across SE Domain

> Applying scientific & engineering efforts to integrate related technical parameters

> System solution which satisfies customers’ expectations> Interdisciplinary approach which involves integrating various engineering

disciplines (i.e., electrical design, software design, hardware design

- Notice something missing from these various definitions!> No mention of cost estimating, however, does not diminish importance of it

- Pro-active aggressive SE process during program life-cycle> Lower life cycle costs> High system quality & enhanced technical solution> Minimizes cost & schedule overruns


Copyright © 2006 Raytheon CompanySystems Engineering Definitions

• System-of-Interest- Defined by ISO/IEC 15288 architectural structures- Comprised of interacting system elements- Each system element independent of each other; can operate on their

own- Only provides real value when connected together to provide system

functions >System

>A system

>System Element

>System Element

>System Element

>is completely composed of >a set of interacting

>system elements

System Structure


Copyright © 2006 Raytheon CompanyCOSYSMO

• Consortium developed COnstructive SYStems Engineering Cost MOdel (COSYSMO) parametric model

> University of Southern California Center for Software Engineering (USC/CSE)

> USC/CSE Industrial Affiliates (including Raytheon)• COSYSMO Cost Model

- Accurately estimates time & effort for SE tasks- Parametric-based cost model- Successfully defended August 2005 doctoral dissertation- Open public domain model- COSYSMO led by Dr. Barry Boehm, USC/CSE, developer of COCOMO family

of software cost models- COSYSMO drawn significant interest & support from INCOSE



SECOST - History

2. Raytheon Affiliate builds initial version of USC’s COSYSMO - Early 2001- “MyCOSYSMO” leveraged off

Garland’s SWCOST > COCOMO II based> Proven tool - used at Garland

and other IIS sites for 8+ years

3. SECOST developed as Raytheon’s Proprietary version of MyCOSYSMO- Early 2004

- Added back Company Proprietary functionality

4. SECOST deployed at several Raytheon business units as a “second opinion” for proposals

1. Beginning development of COSYSMO Initiated by paper walked-on in 2000

by a Raytheon Affiliate to USC/CSE- Joint effort by USC/CSE and

INCOSE


Copyright © 2006 Raytheon CompanySECOST Capabilities and Functionality

• Suite of MS Excel spreadsheets• Generates, documents, and archives SE cost estimates within

single process-focused framework• Supports ROMs, budgetary estimates, formal proposal bids• Supports multiple levels of estimate formality & complexity• Consists of SECOST Framework• USC COSYSMO is embedded engine • Interfaces with standard Raytheon Pricing Systems• Supports Cost Volume & generates Basis-of-Estimates• In-Process & Historical Data Collection

- Results used for local COSYSMO model calibrations- Local COSYSMO model calibrations feeds USC COSYSMO



SECOST Cost Estimation Mode: System-of-Interest

• Cost estimation mode prepare SE cost estimates for future pursuits

• Data collection mode collects SE labor hours expended during program execution

• Current Raytheon NCS Systems Engineering Cost Estimation Enabler

- Using SECOST Size Drivers (Requirements, Interfaces, Algorithms, Operational Scenarios)

- Using SECOST Size Drivers Complexity Criteria- Using SECOST EREQ Conversion and Reuse Factors



SECOST Cost Estimation Mode: System-of-Interest

• Focus on Cost Estimation Mode• Cost Estimation Mode 15 steps

- 1) Initialize Project Parameters (e.g. project name, period of performance, type of estimate)

- 2) Enter SE Contractor Work Breakdown Structure (e.g. Technical Management, IV&V, Requirements Definition & Validation)

- 3) Document Project Assumptions> Assumptions always associated with SE cost estimates

- 4) Document and Register Project Risks> Program risks always associated with SE cost estimates



SECOST Cost Estimation Mode: System-of-Interest (continued)

• Cost Estimation Mode 15 steps (continued)- 5) Set COSYSMO Effort Multipliers (continued)

> Application Effort Multiplier – evaluates specific COSYSMO application factors on scale from Very Low to Extremely High

– Requirements Understanding– Architecture Understanding– Level of Service Requirements– Migration Complexity– Number & Diversity of Installations/Platforms– Number of Recursive Levels in Design– Documentation to match lifecycle needs– Technology Risk




• Cost Estimation Mode 15 steps (continued)- 5) Set COSYSMO Effort Multipliers (continued)

> Team Effort Multiplier – evaluates specific COSYSMO team factors on scale from Very Low to Extremely High

– Stakeholder Team Cohesion– Personnel/Team Capability– Personnel Experience/Continuity– Process Capability– Multisite Coordination– Tool Support

- 6) Determine Labor Distributions among Raytheon Salary Labor Grades- 7) Estimate Four SE Size Drivers

> System-Level Requirements– Decompose system-of-interest objectives & capabilities into requirements that

can be tested, verified, or designed– Count number of requirements (“shalls”) in system specification; only

requirements managed by SE – not HW or SW




• Cost Estimation Mode 15 steps (continued)- 7) Estimate Four SE Size Drivers (continued)

> System-Level External & Internal Interfaces– Functional interfaces (e.g. protocols or timing requirements) not physical

interfaces (e.g. number of wires)– Interfaces that involve SE for your defined system-of-interest– Only count number of unique interface types – not every interface

> System-Level Algorithms– Algorithm sources are functional block diagram, mode description document,

system specification, etc.

> System-Level Operational Scenarios– Typically quantified by number of system test thread packages, unique end-to-

end tests, number of use cases

- 8) Determine Effort Hours> Outputs total SE hours and equivalent requirements (EREQs)




• Cost Estimation Mode 15 steps (continued)- 9) Time Phase SE Estimate

> Spread total SE hours among CWBS

- 10) Submit to Pricing Group> Pricing analyst processes SECOST file (e.g. adds appropriate Raytheon

Business Unit, correct CLIN, other pricing variables)

- 11) Process Pricing Group Data> After pricing analyst processes SECOST file, sent back to SE estimator to

copy and paste process SECOST file into SECOST Worksheet

- 12) Conduct Internal Estimate Review> Internal review among SE Estimator, Lead SE, Program Manager

- 13) Determine and Signoff Final Bid> After internal SE Review completed & approved by SE Center Director,

final cost estimate presented to Raytheon Senior Management




• Cost Estimation Mode 15 steps (continued)- 14) Finalize Management Bid Review Charts

> SECOST provides four management review package charts– SE Labor Cost Summary Chart– Past Program SE Sizing and Unit Cost– Monte Carlo Output Sample Distribution– Monte Carlo Output Cost “Probability of Success”

- 15) Archive the Estimate> Most important step; provides rationale and data if questions or issues are

raised during cost estimation phase


Copyright © 2006 Raytheon CompanySystem-of-Systems Definition

• System-of-Systems (SoS) is not the same as a Family-of-Systems (FoS)

- FoS do not create capability beyond additive sum of member systems’ individual capabilities

- FoS belong to domain or product lines (e.g. family of missiles, family of aircraft)

- FoS lacks synergy of a SoS- FoS do not acquire qualitatively new properties as result of its grouping

• U.S Department of Defense (DoD) SoS Definition- “A SoS is a set or arrangement of interdependent systems that are

related or connected to provide a given capability. The loss or any part of the system will significantly degrade the performance or capabilities of the whole. The development of a system of systems solution will involve trade space between the systems as well within an individual system’s performance.”


Copyright © 2006 Raytheon CompanySystem-of-Systems Definition

• My thesis used the U.S. DoD SoS definition• SoS Characteristics

- Researched five main sources which truly defined SoS> Addressing the System of Systems Challenge Paper> Purdue University School of Aeronautics & Astronautics> “Systems of System Approaches in U.S. Department of Defense” presentation

at 1st Annual SoS Engineering Conference> SoS Engineering Center of Excellence> “System of Systems Engineering” presentation at 1st Annual SoS Engineering

Conference

- Four SoS common characteristics shared by five SoS definitions> Emergence – Whole is greater than the sum of its parts; SoS behave as

collective whole, interacting with its environment to adapt and respond> Independence – Each system within SoS can operate on their own> Lack of Ownership – SoS does not have an identified owner at SoS level> Evolutionary – SoS is never completely formed; continues to be a living

system


Copyright © 2006 Raytheon CompanySystem-of-Systems Examples

• U.S. DoD Programs• SoS examples in commercial world (e.g. internet)• Focus on U.S. DoD Programs

- Raytheon’s primary customer is U.S. DoD- DoD driving towards mandating “jointness for services (i.e. Air Force,

Army, Navy, Marine Corps)- Current GWOT & OIF campaigns have Airmen, Sailors, Soldiers,

Marines fighting together as joint units- Emphasis on jointness forced services to develop warfighting strategies

to support joint warfighting- Current C2ISR, communications & computers, COE capabilities need

integration to support joint warfighting- Integration of these capabilities provides integrated capability-centric

jointness system; in other words a SoS



• Future Combat Systems SoS- SoSCOE

> Software that allows various systems to operate seamlessly> Approximately 35 million lines of code

- Battle Command Software> Consists of four software packages

– Mission Planning & Preparation, Situation Understanding, Battle Command & Mission Execution, Warfighter-Machine Interface

- Communications & Computers> FCS SoS connected to C4ISR network by multilayered Communications &

Computer network> Network provides secure access to information sources over extended

distances & complex terrain

- ISR> Distributed & networked array of ISR Sensors

- Networked Logistics Systems> Integrates logistics into C4ISR network



Networked SoS which develops combat power, sustainability, agility, and versatility for full spectrum military operations

Networked SoS manned by soldiers & fighting team of teams



• DoD Distributed Common Ground System SoS- Combination of U.S. Air Force, Army, Navy, and Marine Corps ground

and surface systems- Each service’s DCGS consists of elements and processes, exploits, and

posts ISR sensor data- Each service’s DCGS consists of legacy systems- DoD currently preparing migration plans to integrate all service DCGS

elements> Achieves a net-centric DCGS> Integration of services’ DCGS is referred to as DoD DCGS SoS

- DoD mandated DCGS SoS migrate to net-centric warfare & net-centric DCGS Enterprise



Improves accuracy and timeliness of intelligence provided to warfighter

Promotes standards-based ISR infrastructure



• Land Warrior SoS- High-tech SoS which provides U.S. Army soldier enhanced capabilities- Integrated fighting system which helps increase soldier’s

> Lethality> Battle Command Compatibility> Survivability> Mobility> Awareness> Situational Awareness> Combat effectiveness



• Land Warrior SoS- Consists of following subsystems

> Weapon Subsystem– Integrates weapon-mounted sensors (multifunction laser, daylight video sight,

thermal weapon sight)

> Soldier Control Unit– Provides primary user interface to system functions

> Personal Area Network Cables– Distributes power & data through the system

> Personal Clothing & Individual Equipment– Consists of utility belt & subsystem pouches

> Computer/Master Hub Subsystem– Provides control of system functions



• Land Warrior SoS- Consists of following subsystems

> Power Source Subsystem– Provides centralized power from dual disposable or rechargeable batteries

> CommsNet Radio Subsystem– Provides transmit/receive voice & data capability

> Navigation Subsystem– Provides position location data to the soldier & time reference to system

> Helmet Subsystem– Provides full-color display for computer interface



• NASA Exploration SoS- Represents U.S. President’s vision for U.S. space exploration- New capabilities & systems enabling safe & successful human & robotic

missions



• NASA Exploration SoS- Consists of following subsystems

> Crew Transportation System– Flight elements which deliver human crew from Earth to mission destination &

return crew safely to Earth

> Cargo Delivery System– Delivers all non-crew exploration vehicle flight elements to accomplish human

exploration objectives

> Ground Support System– Provides all common ground-based capabilities needed to execute exploration

missions

> Robotic Precursor System– Provides measurements, technology, & demonstrations in advance of human missions

> In-Space Support System– Encompasses capabilities provided by space-based infrastructure elements (e.g.

communications, navigation, surveillance)

> Destination Surface System– Encompasses all elements necessary to enable long-duration human exploration

mission



SECOST Cost Estimation Mode: System-of-Systems

• Can SECOST be used for SoS- Yes, with recommended modifications & additions to COSYSMO Size

Drivers & Application and Team Effort Multipliers- SECOST does not account for SoS characteristics

> Emergence, Independence, Lack of Ownership, Evolutionary> Complex Integration Efforts

– Many domains involved

> Design Optimization– Approach not feasible for SoS; have to evaluate each system within SoS to

determine best optimal design

> Complex interface design & management– Management of SoS interfaces more difficult than for system-of-interest

> Decomposing FoS functional requirements & allocating to SoS– No SoS hierarchy; results in stovepipe solutions

> SoS IV&V– More complex than system-of-interest IV&V



SECOST Cost Recommended Modifications

REQ Complexity

Criteria Specification Traceability Overlap

Use

Cases Functional

Requirements Nonfunctional Requirements

Easy Simple to Implement

Traceable to source

Little requirements

overlap

Less than 50

Well-defined software technical details

Little constraints on

software design

Nominal Familiar

Can be traced to

source with some effort

Some overlap

Greater than 50 but less

than 100

Moderately defined software technical details

Moderate constraints on

software design

Difficult Complex to

implement or engineer

Hard to trace to source

High degree of requirements

overlap

Greater than 100

Little defined software technical details

High constraints on

software design

• Add Software Requirements Size Driver- Previously mentioned SoS programs have extensive software-level

requirements> DoD DCGS SoS has over 2,000 unique software requirement

- Significant costs associated with numerous software requirements & software integration




• Add Software Modules Size Driver- Software module defined as CSCI

> Newly developed, COTS, GFE

- Previous mentioned SoS Programs contained numerous CSCIs> DoD DCGS SoS has over 3,000 CSCIs

REQ Complexity

Criteria New SLOC Reused SLOC Testability

Easy Less than 100,000

Less than 200,000

Little concern

Nominal

Greater than 100,00 but less than 500,000

Greater than 200,00 but less than 700,000

Moderate concern

Difficult Greater than

500,000 Greater than

700,000 High concern




• Modify Interfaces Complexity Criteria- Managing SoS interfaces complex (e.g. FCS SoS has over 2,000

unique interfaces)

I/ F Complexity

Criteria Complexity Coupling Consensus Behavior

Independence

Management

Easy Simple

message Uncoupled

Strong Consensus

Well Behaved

Low

Low number of interfaces (less

than 20)

Nominal Moderate

Complexity Loosely coupled

Moderate Consensus

Predictable Behavior

Moderate

Moderate number of interfaces

(greater than 20 but less than 75)

Difficult Complex

protocol(s) Highly

coupled Low

Consensus Poorly

Behaved

High

High number of interfaces

(greater than 75)




• Modify COSYSMO Application Effort Multiplier- Requirements Understanding

> Rates the level of understanding of the SoS system and software requirements

- Migration Complexity, Number of Diversity of Installations/Platforms, Number of Recursive Levels in the Design, Documentation to Match Lifecycle Needs

> Add word “SoS” to each of these COSYSMO Application Factors

- Add IV&V COSYSMO Application Factor> Rates maturity & experience of performing SoS IV&V tasks

- Add SoS Integration COSYSMO Application Factor> Rates maturity & experience of performing as a SoS integrator on

previous SoS programs




• Modify COSYSMO Team Effort Multiplier- Stakeholder Team Cohesion

> Add the following viewpoints “What is the number of stakeholders involved?” and “Is there a defined and agreed to list of stakeholder responsibilities?”

- Add word “SoS” to Personnel Experience/Continuity COSYSMO Team Factors

> Add words “and experience of systems engineers who have worked on SoS programs”



Summary and Conclusions

• October 2003 OSD SE Summit> OSD’s position is parametric-based estimates are recommended technique

for preparing SE Cost Estimates Current SE Cost Estimation Method

• Raytheon parametric-based SECOST Tool- Provides SE cost estimates for system-of-interest

• DoD SoS Programs are increasing in importance and come with challenges

• No credible method for performing SoS SE cost estimates• Parametric-based SECOST tool can predict SoS SE cost

estimates- With additions & modifications to SECOST

• Parametric-based SECOST tool can provide accurate, credible & believable SoS SE cost estimates

1 secost is a tool developed by raytheon and used with permission 10/21/06 copyright © 2006...

Documents