dod parametric cost estimating handbook 2nd ed

SPONSORED BY DOD

PARAMETRIC

COST

ESTIMATING

HANDBOOK

Joint

Government/Industry

Initiative

Fall 1995

Parametric Cost Estimating Handbook

PREFACE

This Handbook is intended to be used as a general guide for implementing and evaluating

parametric based estimating systems, and as the text material for a basic course in parametric

estimating techniques. The information contained in this Handbook complements and enhances

the guidance provided by DCAA Cost Audit Manual (CAM), FARs/DFARs, and other

government regulations or public laws relating to cost estimating. The Handbook is structured so

each chapter can be used as a stand alone guide and reference. If the reader is interested in only

one subject, then one chapter can be read without reading any other. Expected users of this

Handbook include cost model developers, analysts, auditors, and all levels of cost management.

Every phase of the government’s procurement process is being reviewed and changed to

reflect the realities of today’s cost conscious environment. Among the many aspects of the

procurement process under scrutiny is the use of traditional cost estimating tools and processes.

These tools and processes, at times, have proven expensive to maintain and operate and did not

always provide realistic or timely results. Today’s Acquisition Reform mind-set dictates that the

cost estimating community consider a different set of tools and processes which are less labor

intensive to operate and result in credible cost estimates.

It is partly from the impetus of Acquisition Reform that this document has been created.

All of us in the cost estimating community need to find ways to perform our work more cost

effectively. Pushed by the common goal of reducing proposal preparation cost, contractor and

oversight management are pursuing a cooperative approach to problem solving. This Handbook

is an example of such a cooperative effort. If the Handbook succeeds to help produce better

estimates, analyses, or audits, then the process has been successful.

This Handbook is just one of several by-products of a joint Industry/Government

Parametric Cost Estimating Initiative. The Initiative Steering Committee and implementation

team includes representation from contractors, buying activities, Defense Contract Management

Command and Defense Contract Audit Agency. The Initiative’s far reaching action items

include working with pilot contractor sites to test the expanded use of parametric cost estimating

techniques and tools, developing parametric cost estimating training, and preparing and

distributing this Handbook.

This is the first edition of the Handbook. It will be used and tested at pilot contractor

sites. The Initiative Steering Committee will be responsible for receiving and incorporating any

comments for improving the Handbook from the pilot sites and others, and publishing the second

edition of the Handbook. The second edition will be issued in the summer of 1996. A user reply

sheet is provided in this Preface to facilitate submission of your suggestions. An Order Form for

this Handbook is included at the end of the Preface.

The preparation of this Handbook included the review of nearly four thousand pages of

documentation. Some of that total is included here. The material has come from many sources,

both individual and organizational contributors. There is no way all of the contributors can be

acknowledged, since most of them are anonymous. There are, however two who should be


mentioned at this time, besides those acknowledged later in the test. First, the Parametric

Estimating Initiative Steering Committee for authorizing this Handbook and specifically,

NAVSEA for funding the Handbook’s development.

Thanks to the rest of the Committee for their inputs, reviews and critiques of the draft.

And, especially, the Space Systems Cost Analysis Group (SSCAG) who supported the initial

concept of the Handbook, and the may individuals within that organization who conceptualized,

critiqued, and developed much of the material.

Bill Brundick,

General Editor


PARAMETRIC ESTIMATING INITIATIVE STEERING COMMITTEE

Executive Chairman Bob Scott Bob Spiker

Executive Director of Contract Management Controller, Financial and Government Accounting

Defense Contract Management Command - AQC Westinghouse Electric Corporation - ESG

Cameron Station P.O. Box 17319, Mail Stop A585

Alexandria, Virginia, 22304-6190 Baltimore, Maryland 21203-7319

703-274-0821 phone 410-765-2913 phone

703-274-0823 fax 410-765-6038 fax

Michael Thibault

Deputy Director

DCAA Headquarters

Cameron Station

Alexandria, Virginia 22304-6178

703-274-7281 phone

703-617-7450 fax

Co-Chairman

Jim Collins David Eck

Parametric Estimating Specialist Chief, Policy Formulation Division

Westinghouse Electric Corporation D61-ESG DCAA Headquarters

P.O. Box 1693, Mail Stop 1293 Cameron Station, Room 4A250

Baltimore, Maryland 21203 Alexandria, Virginia 22304-6178

410-765-8033 phone 703-274-7314 phone

410-765-5289 fax 703-617-7452

Members

Jim Balinskas Dean Boyle

Contract Pricing & Finance Division Deputy Technical Assessment Manager

NASA Headquarters, Code HC DPRO - Northrup Grumman

300 E Street SW Defense Logistics Agency, DCMC

Washington, DC 20546 Bethpage, New York 11714-3593

202-358-0445 phone 516-575-9742 phone

202-358-3082 fax 516-575-6527

Gary Constantine Marty Deutsch

Manager, Parametric Estimating Chief, Estimating Policy Review & Compliance

E-Systems - Greenville Martin-Marietta Astronautics

P.O. Box 6056, CBN93 P.O. Box 179, Mail Stop DC 2503

Greenville, TX 75403-6056 Denver, Colorado 80201-0179

903-457-5666 phone 303-971-6060 phone

903-457-4619 fax 303-971-5143 fax

Mel Eisman Jim Gleason

Senior Research Associate Procurement Analyst

Rand Corporation US Army Material Command, AMCAQ-E

P.O. Box 2138, Mail Stop HMRP/5 5001 Eisenhower Avenue, Room 9N15

Santa Monica, California 90407-2138 Alexandria, Virginia 22333-0001

310-393-0411 x6704 phone 703-274-4437 phone

310-451-7032 fax 703-274-3198 fax

Virgil Hertling Paul Lubell


Directorate of Pricing and Finance Senior Engineer, Technical Estimating

US Air Force Headquarters, AFMC/PKF Westinghouse Electric Corporation - ESG

Building 262, 4375 Chidlaw Road, Suite 6 P.O. Box 1693, Mail Stip 1293

Wright Patterson AFB, Ohio 45433-5006 Baltimore, Maryland 21203

513-257-6861 phone 410-765-6163 phone

513-476-2435 fax 410-765-5289 fax

Bernard Rudwick Marcia Rutledge

Professor - Financial Management Contracting Officer, ASW & Mine Systems Branch

Defense Systems Management College US Navy, Naval Sea Systems Command

Fort Belvoir, Virginia 22060-5426 2531 Jefferson Davis Highway, NC-3, Room 5E08

703-805-3783 phone Arlington, Virginia 22242-5160

703-805-3184 fax 703-602-0951 x639 phone

703-602-7023 fax

George Salantai Amir Tarmohamed

Manager, Estimating Proposal Anaysis & Definitization Team

McDonnell Douglas Aircraft Company Defense Logistics Agency, DCMC-AQCOD

Building 305, Post 2E, Room 232, Mail Code 306-5485 Cameron Station, Room 8A454

St. Louis, Missouri 63166 Alexandria, Virginia 22304-6190

314-233-8461 phone 703-274-4130 phone

314-232-7171 fax 703-617-0944 fax


USER REPLY SHEET PARAMETRIC COST ESTIMATING HANDBOOK

Please send comments to:

Marcia Rutledge

Contracting Officer, ASW & Mine Systems Branch

US Navy, Naval Sea Systems Command

2531 Jefferson Davis Highway, NC-3, Room 5E08

Arlington, Virginia 22242-5160

703-602-0951 x639 phone

703-602-7023 fax

We would appreciate specific comments such as the suggested alternative language and rationale

to support the language, additional information that would add value to the Handbook, or

existing information that can be deleted. Specific examples that enhance the Handbook are

welcome.

Comment # _______ Chapter _______ Page # _______


ORDER FORM

for the

PARAMETRIC COST ESTIMATING HANDBOOK

YES, please send me _____ copies of the Parametric Cost Estimating Handbook.

There is no charge for the Handbook. Requestors are encouraged to limit requests

to one copy per organization.

Please send the Handbook to:

Company/Organization (Please type or print)

Attention/Individual

Street Address

City, State, Zip Code

Daytime phone number including area code

All orders should be sent to:

Naval Sea Systems Command

2351 Jefferson Davis Highway

NC-3, Room 5E08

Arlington, VA 22242-5160

Attn: M. Rutledge, SEA 0263

Contracting Officer, ASW & Mine Systems Branch

14 September 1995

FOREWORD

Early in 1994, a joint Government/Industry Committee was formed to study ways to

enhance the use of parametric cost estimating techniques. The Committee found that the lack of

training was one of the largest barriers to the use of parametrics. The Committee sponsored this

Handbook to provide training and background information on the use and evaluation of

parametric tools. The Committee is also working with the Defense Acquisition University to

develop classroom training that would be available to both government and industry.

The Committee is also sponsoring a pilot program to test the use of parametric tools. As

of September 1995, eleven companies are participating in the pilot program. The pilot program

is expected to last until the Summer of 1996. This Handbook will be tested at the pilot program

sites. The Committee will update the Handbook to incorporate comments, best practices, and

additional examples developed at the pilot sites. The Committee also invites your comments on

the Handbook and any examples you may have. A User Reply Sheet is provided in the Preface

to facilitate your input. The Committee expects to publish the second edition of the Handbook

by the Summer of 1996.

The Committee hopes, with your help, to make the Handbook the guide people turn to

when using and evaluating parametric tools.

Robert Scott Executive Director of Contract Management

Defense Contract Management Command

Robert Spiker Controller, Financial & Government Accounting

Westinghouse Electric Corporation - ESG

Michael Thibault Deputy Director

Defense Contract Audit Agency

Executive Chairmen

Joint Government/Industry

Parametric Cost Estimating Initiative

Steering Committee

Parametric Cost Analysis Handbook

i

PARAMETRIC COST ESTIMATING HANDBOOK

TABLE OF CONTENTS

FOREWORD

PREFACE

CHAPTER I -- INTRODUCTION, BACKGROUND AND DEFINITIONS

Introduction .............................................................................................................. 1

Background .............................................................................................................. 4

Definitions and Terms .............................................................................................. 9

Cost Realism ............................................................................................................ 10

CHAPTER II -- COLLECTION AND NORMALIZATION OF PARAMETRIC DATA

Generalizations ......................................................................................................... 12

Significant Adjustments to Parametric Data ............................................................ 14

Consistent Scope .......................................................................................... 14

Anomalies .................................................................................................... 14

Improved Technology .................................................................................. 15

Other Types of Adjustments and Data Normalization ............................................. 15

Inflation ........................................................................................................ 15

Learning Curve ............................................................................................ 16

Production Rate ............................................................................................ 16

Calibration and Validation of Cost Models ............................................................. 17

Some Review and Audit Considerations ................................................................. 18

Data Normalization Process ..................................................................................... 19

Pitfalls to the Use of a Parametric Model ................................................................ 25

Two Illustrations of Typical Data Normalization Problems .................................... 27

Summary .................................................................................................................. 29

CHAPTER III -- ELEMENTARY STATISTICAL TECHNIQUES AND CER

DEVELOPMENT

CER and Model Development - Uncertainty and Risk Reduction .......................... 32

Developing Cost Estimating Relationships (CERs) ................................................. 34

Hypothesis Testing of a Logical CER ......................................................... 36

The CER Model ........................................................................................... 36

When to Use a CER ................................................................................................. 36

Strengths and Weaknesses of CERs ......................................................................... 37

Strengths ...................................................................................................... 38

Weaknesses .................................................................................................. 38

Regression Analysis ................................................................................................. 38

Curve Fitting ............................................................................................................ 41

Graphical Method ........................................................................................ 41

LSBF Method .............................................................................................. 41


ii

Multiple Regression ..................................................................................... 44

Curvilinear Regression ................................................................................ 45

“Goodness” of Fit, R and R2

.................................................................................... 45

Correlation Analysis .................................................................................... 45

Coefficient of Determination ....................................................................... 46

Coefficient of Correlation ............................................................................ 47

The Learning Curve ................................................................................................. 47

Limitations, Errors and Caveats of LSBF Techniques ............................................. 49

Extrapolation Beyond The Range of the Observed Data ............................. 49

Cause and Effect .......................................................................................... 50

Using Past Trends to Estimate Future Trends ............................................. 50

Misinterpreting the Coefficients of Correlation and Determination ............ 50

Summary ...................................................................................................... 50

Examples of CER Use .............................................................................................. 51

Construction ................................................................................................. 51

Weapons Procurement ................................................................................. 51

Electronics ................................................................................................... 51

Cost and Price Analysis ............................................................................... 52

Information and Techniques ........................................................................ 52

Estimate Reliability ...................................................................................... 53

Two Examples of CER Use ......................................................................... 54

Common CERs ............................................................................................ 60

Acceptance Criteria for a Cost Estimating Relationship .......................................... 61

Auditing and Analyzing a CER ............................................................................... 63

CER Analysis ............................................................................................... 63

General Features .......................................................................................... 63

Evaluating and Estimate .............................................................................. 64

Summary of CER Audit and Analysis ......................................................... 65

CHAPTER IV -- HARDWARE COST MODELING

Introduction .............................................................................................................. 68

Overview of Hardware Cost Modeling .................................................................... 70

Micro-Circuit and Electronic Module Modeling ..................................................... 77

Hardware Operations and Support (O&S) of Life Cycle Models ............................ 79

Deployment and Employment ..................................................................... 81

Hardware Parameters ................................................................................... 82

Program Globals .......................................................................................... 82

Commercial Models ................................................................................................. 82

MicroFASTE ............................................................................................... 83

Price Parametric Models .............................................................................. 87

SEER ............................................................................................................ 90

Regression Based Product Specific Cost Models .................................................... 91

Non-Commercial Cost Models ................................................................................ 93

Non-Statistical Cost Models .................................................................................... 94

Model Calibration .................................................................................................... 95


iii

Cost Model Audit and Analysis ............................................................................... 96

Analyzing a Product Specific Cost Model ................................................... 96

Summary of Cost Model Audit and Analysis .............................................. 99

CHAPTER V -- SOFTWARE PARAMETRIC COST ESTIMATING

Software Definition .................................................................................................. 101

The Importance of Software Today ......................................................................... 102

The Software Development Process ........................................................................ 107

The Waterfall Model .................................................................................... 107

The Software Cost Estimating Process .................................................................... 109

Define Project Objectives and Requirements .............................................. 111

Plan the Activities ........................................................................................ 111

Software Estimation Risks ........................................................................... 112

Estimation Methodologies ........................................................................... 113

Software Cost Estimating Standards ............................................................ 116

Benefits ........................................................................................................ 116

Examples of Parametric Software Cost Estimating ................................................. 116

Parametric Software Cost Estimating Tools ............................................................ 119

Desired Functional Capabilities of Parametric Tools .................................. 120

Input Data Collection ................................................................................... 122

Some Commercial Tools ............................................................................. 122

Software Sizing Tools .................................................................................. 127

Glossary of Terms .................................................................................................... 130

Model Calibration .................................................................................................... 130

Trends and Conclusions ........................................................................................... 131

Trends .......................................................................................................... 131

Conclusions .................................................................................................. 132

CHAPTER VI -- AUDITING PARAMETRIC ESTIMATES - A MANAGEMENT

VIEWPOINT

Cost Estimating Relationships: A DCAA Perspective ........................................... 133

Introduction .................................................................................................. 133

Background .................................................................................................. 134

Parametric Criteria ....................................................................................... 134

Logical Relationships .................................................................................. 135

Significant Statistical Relationships ............................................................ 135

Verifiable Data ............................................................................................. 135

Reasonably Accurate Predictions ................................................................ 135

Proper System Monitoring ........................................................................... 136

Audit Planning and Requirements ............................................................... 136

Observations and Suggestions ..................................................................... 140

Summary ...................................................................................................... 142

Estimating System Reviews ..................................................................................... 143

Forward Pricing Rate Agreement (FPRAs) ............................................................. 145

What to Look For in a Parametric Estimate ............................................................. 148


iv

Rules of Thumb ........................................................................................... 150

An Example ................................................................................................. 151

CHAPTER VII -- BUSINESS APPLICATIONS OF PARAMETRIC ESTIMATING

Introduction .............................................................................................................. 153

Parametric Estimating in New Business Development ............................................ 153

Estimating Production Buys Using Parametrics ...................................................... 154

Example: Estimating Production Buys Using Parametrics ......................... 155

Estimating Spares and Change Orders ..................................................................... 158

Appendix A Definitions of Estimating Terminology .................................................... 160

Appendix B Work Breakdown Structure ....................................................................... 177

Appendix C DCAA CAM 9-1000 Section 10, Review of Parametric Cost Estimates . 198

Appendix D More About Statistics ................................................................................ 205

Appendix E Examples of Other Hardware Estimating Models .................................... 209

Appendix F Some Currently Available Software Estimation Products ........................ 218

Appendix G Parametric Estimating System Checklist .................................................. 236


1

CHAPTER 1

INTRODUCTION, BACKGROUND,

AND

DEFINITIONS


2

CHAPTER I

INTRODUCTION, BACKGROUND, AND DEFINITIONS

INTRODUCTION

This Handbook is intended to be a living document. As advances are made in parametric

estimating methodology, they will be introduced into the body of this material. Changes

suggested from experienced users are solicited, as well as recommendations from other experts in

the field. However, the Handbook is primarily intended for the beginning parametrics

practitioner and to be used to enhance parametric training in the field. When using the

Handbook, however, we assume that the reader has a basic understanding of algebra and

statistics.

Defined, a parametric cost estimate is one that uses Cost Estimating Relationships

(CER’s) and associated mathematical algorithms (or logic) to establish cost estimates. For

example, detailed cost estimates for manufacturing and test of an end item (for instance, a

hardware assembly) can be developed using very precise Industrial Engineering standards and

analysis. Performed in this manner, the cost estimating process is laborious and time consuming.

However, if history has demonstrated that test (as the dependent variance) has normally been

valued at about 25% of the manufacturing value (the independent variable), then a detailed test

estimate need not be performed and can simply be computed at the 25% (CER) level. It is

important, though, that any CER’s used be carefully tested for validity using standard statistical

approaches. An exploration of certain statistical approaches relevant to CER development is

included later in this Handbook.

The need to reengineer business processes and reduce cost in the Department of Defense

(DoD) has led to a parametric cost estimating initiative. In every corner of every aspect of

defense contracting, Business Process Reengineering (BPR) has become a nineties buzzword.

The cumbersome techniques that evolved into the development of the “normal” cost estimating

processes of today are beginning to yield to more efficient and less costly approaches to achieve


3

the same, or superior results. Parametric estimating approaches fit very well into overall BPR

methods.

The importance of Business Process Reengineering was recently underscored by Lloyd K.

Mosemann, II, Deputy Assistant Secretary of the Air Force, in his closing Keynote Address

entitled “Predictability,” to the Software Technology Conference in Salt Lake City, Utah, on

Thursday, April 14, 1994. Although addressing the software process, Mr. Mosemann’s

comments are relevant to the cost estimating process in general. In summary, he said, in part:

“There seems to be an inability within the software community, in general, to predict how

much a software system will cost, when it will become operational, and whether or not it will

satisfy user requirements. We need to deliver on our promises.

“We have a poor track record regarding predictions. A 1979 GAO report concluded that only

two percent of software contracted for was useable exactly as delivered. Twenty different studies

have come to the same conclusion. Therefore, we in the DoD are focusing our attention on

process improvement: These include: specific metrics usage plans, reuse plans, peer

inspections, process controls, proposed architectures in executable code, and government access

to contractor on-line development environments.

“This emphasis on process will give all of us in the software community a greater confidence

that the prospective contractor will deliver the promised product on time and on budget.”

Mr. Mosemann’s emphasis on process improvements to improve the quality of predictability

of cost and schedule fits nicely with the concept of expanding the use of parametric tools in the

cost estimating workplace. Parametrics can play a role in the BPR process as was underscored

by Anthony A. DeMarco in his article, CAPE (Computer Aided Parametric Estimating for

Business process Re-Engineering, in the PRICE Newsletter, October 1994. In his article, in

summary, Mr. DeMarco states that:

“Business Processing Reengineering (BPR) is the reengineering of an organization by

examining existing processes and then revamping and revising them for incremental

improvement. It is doing more with less and sometimes entails “starting over.”


4

There are five phases to BPR. They are:

1. Create an organization for improvement,

2. Develop an understanding of the process,

3. Streamline the process,

4. Model, implement, measure, and control, and,

5. Design and implement continuous improvement.

“Parametric tools can assist BPR. On one level, they can improve and streamline the BPR

phases. On another level, parametric technology is the ‘best practice’ for estimating. Parametric

tools bring speed, accuracy and flexibility to estimating processes, processes that are often

bogged down in bureaucracy and unnecessary detail.”

The need to reengineer the DoD cost estimating process (Acquisition Reform initiatives)

became self evident to certain people from both government and industry. A Steering Committee

was chartered by government and industry executives to explore the role played by parametrics in

the cost estimating process. One goal was to determine what barriers, if any, exist to expanding

the role of parametrics, and to develop the action plans to overcome those barriers. The

committee consists of representatives from all of the Armed Services, the oversite community,

selected contractors. This Handbook has been authorized by that Steering Committee.

The Handbook is intended to be used by both model developers and model reviewers,

their management or oversite, either technical or financial. Government and industry cost

analysts and auditors who utilize CER’s and/or parametric cost models to develop or evaluate an

estimate generated with these parametric tools should find the Handbook useful. It is also

intended to be utilized as a source document by trainees within a generic parametric cost

estimating training program.

This Handbook includes basic information concerning data collection, Cost Estimating

Relationship (CER) development, parametric cost models, and statistical techniques.

Parametric techniques are a credible cost estimating methodology that can provide

accurate and supportable contractor estimates, lower cost proposal processes, and more cost-

effective estimating systems.


5

An estimating workbench context model is shown in Figure I-1. The model indicates the

tools required within the estimating community of contractors, customers and government

agencies. Figure I-2 is a graphical representation of the complete parametric cost estimating

process. The figure indicates the process from inputs through modeling and into a post processor

phase. The post processor allows for the conversion of parametric output into a cost proposal.

BACKGROUND

The origins of parametric cost estimating date back to World War II. The war caused a

demand for military aircraft in numbers and models that far exceeded anything the aircraft

industry had manufactured before. While there had been some rudimentary work from time to

time to develop parametric techniques for predicting cost, there was no widespread use of any

cost estimating technique beyond a laborious buildup of labor-hours and materials. A type of

statistical estimating had been suggested in 1936 by T. P. Wright in the Journal of Aeronautical

Science. Wright provided equations which could be used to predict the cost of airplanes over

long production runs, a theory which came to be called the learning curve. By the time the

demand for airplanes had exploded in the early years of World War II, industrial engineers were

using Wright’s learning curve to predict the unit cost of airplanes.

In the late 1940’s, the DoD, and, especially, the United States Air Force began a study of

multiple scenarios concerning how the country should proceed into the age of jet aircraft,

missiles and rockets. The Military saw a need for a stable, highly skilled cadre of analysts to

help with the evaluation of such alternatives. Around 1950, the military established the Rand

Corporation in Santa Monica, California, as a civil “think-tank” for independent analysis. Over

the years, Rand’s work represents some of the earliest and most systematic studies of cost

estimating in the airplane industry.

The first assignments given to Rand concerned studies of first and second generation

ICBM’s, jet fighters and jet bombers. While the learning curve technique still proved useful for

predicting the behavior of recurring cost, there were still no techniques other than detailed labor-

hour and material estimating for projecting what the first unit cost might be (a key input to the


6

ESTIMATING WORKBENCH

CONTEXT MODEL

PARAMETRIC

MODELS/TOOLS

REQUEST FOR

SUPPORTING DATA

SOLICITATION RESPONSE

ESTIMATING

WORKBENCH

• PROPOSAL/COST DATA,

• HISTORICAL DATA,

• CONTRACTOR DATA,

• COST ELEMENT ANALYSIS

• CERs

• OTHER PARAMETRIC DATA

• OTHER NONCOST DATA

• EXTERNAL DATA -

COST COMPARISONS

• BACKGROUND DATA -

HISTORICAL DATA USERS

FIGURE I-1

BUYERS/REQUESTORS,

CONTRACTING OFFICERS

COST ANALYSTS/

PRICING TEAM/

DPROS

PROGRAM

MANAGERS

AUDITORS

ACADEMIA

CUSTOMERS

OTHER GOV’T

AGENCIES/SOURCES


7

learning curve equation). Worse still, no methods were available for quickly estimating the non-

recurring costs associated with research, development, testing and evaluation (RDT&E). In the

defense business in the early to mid 1950’s, RDT&E had suddenly become a much more

important consideration. There were two reasons for that fact. First, a shrinking defense budget

(between World War II and the Korean War) had cut the number of production units for most

military programs, and second, the cost of new technology had greatly magnified the cost of

development. The inability to quickly, and accurately, estimate RDT&E and first unit production

costs had become a distinct problem.


8

Fortunately, within Rand, a cost analysis department had been started in 1950. This

group proved to be prolific contributors to the art and science of cost analysis -- so much so that

the literature of aerospace cost estimating of the 1950’s and 1960’s is dominated by the scores of

Rand cost studies that were published during that time. In the mid 1950’s, Rand developed the

most basic tool of the cost estimating discipline, the Cost Estimating Relationship (CER), and

merged the CER with the learning curve to form the foundation of parametric aerospace

estimating. This estimating approach is still used today.

By 1951, Rand derived CER’s for aircraft cost as a function of such variables as speed,

range, and altitude. Acceptable statistical correlations were observed. When the data was

segregated by aircraft types (e.g., fighters, bombers, cargo aircraft, etc.), families of curves were

discovered. Each curve corresponded to different levels of product or program complexity. This

Parametric stratification especially helped clarify development cost trends. Eventually, a useable

set of predictive equations were derived which were quickly put to use in Air Force planning

activities.

The use of CER’s and data stratification were basic breakthroughs in cost estimating,

especially for RDT&E and first unit costs. For the first time, cost analysts saw the promise of

being able to estimate relatively quickly and accurately the cost of proposed new systems. Rand

extended the methods throughout the 1950’s, and by the early 1960’s, the techniques were being

applied to all phases of aerospace systems.

Since these rather humble beginnings, the state-of-the-art in parametric estimating has

been steadily improving by an explosive growth in the number of practitioners, important

methodological improvements, and greatly expanded databases. All of the major aerospace

contractors and government aerospace organizations have dedicated staffs of parametricians

who maintain and expand databases, develop parametric cost models, and utilize the tools of

parametrics to make estimates of new and ongoing programs. NASA and the DoD routinely use

parametric estimates to form the basis of new project cost commitments to Congress. The

contractor community also routinely uses parametric cost models, especially during product

concept definition. These estimates are used for decision making regarding bid strategies and

are used as submittals to the government. It is only at the production and full scale development

phase that parametrics are not commonly utilized for official proposal submissions (although


9

contractors still frequently use parametrics to generate target costs and cross-checks on the

labor-material/buildup estimates).

Over the past several years industry and professional estimating associations (e.g.,

International Society of Parametric Analyst (ISPA), Society of Cost Estimating and Analysis

(SCEA), and the Space Systems Cost Analysis Group (SSCAG)) have been actively working

with both Defense Contract Management Command (DCMC) and Defense Contract Audit

Agency (DCAA) to explore the expanded opportunities for the use of parametric cost estimating

techniques in firm business proposals. ISPA was formed in 1978 when a parametric estimating

user’s group evolved into a more generic Society. The Space Systems Cost Analysis Group

formed in 1977 under the sponsorship of the U.S. Air Force Space Division, with a mission to:

1. Promote Cost Analysis Research

2. Develop new tools to improve cost estimating techniques

3. Promote sound practices, and

4. Provide a forum for government and industry cost analysts concerned with the

development and production of space-design hardware and software.

Then, in April 1994, a joint Industry and Government workshop on Parametric Cost

Estimating occurred at the Defense Contract Audit Institute in Memphis, TN. Under the

initiative and leadership of the DCMC, the DCAA, and industry proponents, a group of

knowledgeable government and industry executives, policy formulators, and parametric

practitioners were assembled to evaluate why there is not greater use of parametric cost

estimating in DoD and NASA business proposals; identification of the barriers to expanded use

of parametrics; and, action planning to take advantage of identified opportunities.

At the conclusion of the workshop, it became clear to the participants that there were no

barriers which precluded further implementation and use of parametric cost estimating by

contractors in DoD or NASA business proposals. Rather, barrier analysis and actions

recommended focused on the need for industry leaders to demonstrate that parametric

estimating systems can be relied upon by the Government customers, and the need for the

Government to train employees so that there exists a clear message that valid parametric

estimates are a useful and often cost effective estimating approach.


10

DEFINITIONS AND TERMS

A complete glossary of parametric terminology, taken from numerous sources, is included

in this Handbook as Appendix A. A few of the more important definitions are noted in this

chapter.

There are several definitions of parametric estimating, but for the purpose of this

Handbook, the formal one adopted is as follows: A technique employing one or more CER’S

and associated mathematical relationships and logic. The technique is used to measure and/or

estimate the cost associated with the development, manufacture, or modification of a specified

end item. The measurement is based on the technical, physical, or other end item

characteristics.

This definition establishes the clear linkage between cost and a product’s (or end item)

technical parameters. Without this linkage, a product cost cannot be effectively defined. Non-

parametric estimating systems generally do not connect technical (parametric) and cost elements

with any substantial precision.

And, a Parametric Cost Model is defined as: A parametric cost model is a group of cost

estimating relationships used together to estimate entire cost proposals or significant portions

thereof. These models are often computerized and may include many inter-related CER’s, both

cost-to-cost and cost-to-noncost. Some models use a very limited number of independently

estimated values and a series of Parametric inter-related cost-to-cost and cost-to-noncost

estimating relationships to predict complex proposal cost structures.

Parametric cost estimating is a technique used by both contractors and the Government in

planning, budgeting, and performance stages of the acquisition process. The technique is used

by contractors to expedite the development of cost estimates when discrete estimating

techniques would require inordinate amounts of time and resources and would produce similar

results. Reliance on properly developed and carefully evaluated CER’s and parametric cost

models to produce realistic cost estimates can save both Industry and the Government time and

resources in the evaluation and definitization cycle of the proposal or contract.

The concept includes the use of cost-to-cost CER’s such as engineering labor overhead

rates and material overhead rates which when reviewed using traditional evaluation criteria, are


11

considered valid estimators by the government. However, the technique also uses cost-to-

noncost CER’s which require additional analysis to determine their validity and acceptability as

estimating tools.

Parametric techniques focus on the cost drivers, not the miscellaneous details. The

drivers are the controllable system design or planning characteristics and have a predominant

effect on system cost. Parametrics uses the few important parameters that have the most

significant cost impact on the product(s), hardware or software, being estimated.

COST REALISM

A widely used term today is “cost realism.” Now, no one expects a cost estimate to

precisely predict what a hardware or software product or a time and material service will cost.

So, cost realism is not about the exact cost estimate. It’s about the system of logic, the

assumptions about the future, and the reasonableness of the historical basis of the estimate. That

is, it’s about the things that make up the foundation of the estimate.

Cost realism analysis answers questions such as:

* Are the assumptions used in the estimating process reasonable?

* Has the historical data base used been normalized to account for environmental

parameters such as inflation?

* Is the cost estimate logical? Does it make sense in the context of the hardware or

software product or service being estimated?

* Does the estimate display a bias toward being too low or too high? If so, how is this

bias displayed in the estimate?

* Is the cost estimating organization motivated to produce an inordinately high or low

estimate in order to serve their own purposes?

* If the product is fixed price sole source, has the historical basis data been “cherry

picked” to insure the cost estimate obtained is unreasonably high (contractor) or

unreasonably low (auditor or government customer)?


12

* If the program is competitive, has the contractor or government program office created

program expectations that are far too optimistic?

The cost estimator or analyst must ensure that they are working toward the goal of cost

realism. It doesn’t matter whether or not they are employed by private industry, or the customer

as a cost analyst or an auditor. If a contractor chooses to accept a management challenge in a

competitive procurement, that’s certainly acceptable. However, the basis for the challenge

should be clearly identified.

There is no easy answer to the cost realism dilemma we all face. Unreasonable biases

and expectations from contractor and customer have driven the cost estimating process in the

past, and personal and programmatic motivations may continue to drive it in the future. But one

thing is certain: the cost estimating process will continue to confront future unknowns. These

unknowns are what make the cost estimating job one of the most difficult there is. But sound

assumptions, high quality historical data, and unbiased analysts and estimators will improve the

process for all.


13

CHAPTER II

COLLECTION AND NORMALIZATION

OF PARAMETRIC DATA


14

CHAPTER II

COLLECTION AND NORMALIZATION OF PARAMETRIC DATA

This chapter provides guidelines concerning data types, data sources, data normalization

and adjustment techniques. This chapter also includes utilization recommendations about data to

support the development and use of auditable CERs and cost models.

GENERALIZATIONS

A universal format for collecting technical and cost information is the Work Breakdown

Structure (WBS). (See Appendix B) The WBS provides for uniform definitions and collection of

cost and technical information. MIL-STD-881B provides guidelines for establishing the WBS at

DoD, Service and contractor levels.

Historical cost and labor hours data are required as a basis for cost estimating, and

parametric estimating is no exception. Data should be collected and maintained in a manner that

provides a complete audit trail, and expenditure dates should be recorded so that dollar valued costs

can be adjusted for inflation.

Also required is Technical Non-Cost Data that describes the physical, performance and

engineering characteristics of a system, sub-system or individual item. For instance, weight is a

common non-cost variable used in CER’s and parametric estimating models. (Other typical

examples of cost driver variables include horsepower, materials of construction, watts, thrust,

length, etc.)

A fundamental requirement for the inclusion of a non-cost variable in a CER is that it be a

statistically significant predictor of cost (that is, it should be a cost driver).

Relevant program data including development and production schedules, quantities

produced, production rates, equivalent units, breaks in production, significant design changes, and

anomalies such as strikes, explosions, and other natural disasters are also necessary to fully explain

any perturbations in historical data. Such perturbations may exhibit themselves in a profile of


15

monthly cost accounting data as the labor hour charging may show an unusual "spike" or

"depression" in the level of charged hours. Such historical information comes from knowledgeable

program personnel or program records (also known as program "memory").

The collecting point for cost and labor hours data is, in most instances, called the general

ledger or a company accounting system. All cost and labor hours data, used in parametric CER’s or

cost models, must be consistent with, and traceable back to, the original collecting point (the

source). The data should also be consistent with accounting procedures and cost accounting

standards.

Technical non-cost data comes from engineering drawings, engineering specifications,

certification documents, or direct experience (i.e., weighing an item). Schedule, quantity and

equivalent units, and similar information comes from Industrial Engineering, Operations

Departments, program files or other program intelligence.

Inflation indices normally combine external and internal information. Examples of external

information used in these determinations include the Consumer Price Index (CPI), Producer Price

Index (PPI), Commodity Price Indices and other forecasts of inflation from various econometric

models.

There are other external sources of data including databases containing pooled and

normalized information from various places (other companies or public record information).

Although such information can often be useful, weaknesses of these sources include:

(1) The inability of the user to have knowledge of the procedures (i.e., accounting) used

by the other contributors.

(2) The treatment of anomalies (how they were handled) in the original data.

(3) Knowledge of the manufacturing processes used and how they compare to the current

scenario being estimated.

(4) The inability to accurately forecast future indices.

Internal contractor information includes analyses such as private corporate inflation studies,

or "market basket" analyses. Such interval information provides data specific to a company's

product line(s) (i.e., radar products) that could be relevant to a generic segment of the economy as a

whole (i.e., electronics); etc. Such specific analyses would normally be prepared as part of an


16

exercise to benchmark government provided indices (the CPI), and to compare corporate

performance to broader standards.

It is important to realize that sources of data can be almost unlimited, and all relevant

information should be considered in a parametric analysis, if practical. Although major sources are

described above, data sources should not be constrained to a specific list.

Any data included in calculating parametric parameters will vary between model

developers. However, the way in which parametric models are calculated from historical data and

the way they are applied in the estimating process should be consistent within individual estimating

systems.

SIGNIFICANT ADJUSTMENTS TO PARAMETRIC DATA

What follows below are some of the more significant adjustments that may have to be made

to historical parametric cost data.

Consistent Scope

Adjustments are appropriate for differences in program or product scope between the

historical data and the estimate being made. For example, if the systems engineering department

made a comparison of five similar programs and then realized that only two of the five had design

to cost (DTC) requirements. To normalize the data, the DTC hours were deleted from the two

programs to create a consistent systems scope and definition for CER development.

Anomalies

Historical cost data should be adjusted for anomalies (unusual events), prior to CER

analysis, when it is not reasonable to expect these unusual costs to be present in the new projects.

The adjustments and judgments used in preparing the historical data for analysis, should be fully

documented. For example, a comparison has been made to compare the development test program

from five similar programs and then certain observations are made (from history and interviews)

that one of the programs experienced a major test failure (e.g., qualification, ground test, flight test).


17

A considerable amount of labor resources were required to fact find and then determine the root

cause of and develop an action plan for a solution. Should the hours be left in or deleted?

Improved Technology

Cost changes, due to changes in technology, are a matter of judgment and analysis. All

bases for such adjustments should be documented and disclosed. For example, electronic circuitry

was originally designed with discreet components, but now the electronics are ASIC technology. A

hardware enclosure once was made from aluminum and now is made, for weight constraints, of

magnesium. What is the impact on the hours? Perfect historical data may not exist, but judgment

and analysis should supply reasonable results.

For example, suppose there are four production (manufacturing hours) lots of data that look

like this:

Lot 1 = 256,000 = 853 hours/unit

Lot 2 = 332,000 = 738 hours/unit

Lot 3 = 361,760 = 952 hours/unit

Lot 4 = 207,000 = 690 hours/unit

Clearly, Lot 3's history should be investigated. It is not acceptable to merely "throw out"

Lot 3 and work with the other three lots. A careful analysis should be performed on the data to

determine why it behaved the way it did. There may have been a strike, or possibly an unusual and

serious engineering problem impacted production costs. In any event, careful analysis is important.

OTHER TYPES OF ADJUSTMENTS AND DATA NORMALIZATION

Inflation

There are no fixed ways to establish universal inflation indices (past, present or future) that

fit all possible situations. Inflation indices are influenced by internal considerations as well as

external inflation rates. Therefore, while generalized inflation indices may be used, it may also be

possible to tailor and negotiate indices used on an individual basis to specific labor rate agreements

(e.g., FPRAs) and the actual materials used on the project. Inflation indices should be based on the


18

cost of materials and labor on a unit basis (piece, pounds, hour) and should not include other

considerations like changes in manpower loading or the amount of materials used per unit of

production. The key to inflation adjustments is consistency. If cost is adjusted to a fixed reference

date for calibration purposes, the same type of inflation index must be used in escalating the cost

forward or backwards, from the reference date, and then to the date of the estimate.

Learning Curve

The learning curve, as originally conceived, analyses labor hours over successive production

units of a manufactured item. The curve is defined by the following equation:

Hours/Unit = First Unit Hours *Ub

or

Fixed Year Cost/Unit = First Unit Cost *Ub

Where: U = Unit number

b = Slope of the curve

In parametric models, the learning curve is often used to analyze the direct cost of

successively manufactured units. Direct Cost equals the cost of both touch labor and direct

materials - in fixed year dollars. Sometimes this may be called an improvement curve. The slope is

calculated using hours or constant year dollars. A more detailed explanation of learning curve

theory is presented in Chapter III.

Production Rate

Production rate effects (changes in production rate, i.e., units/months) can be calculated in

various ways. For example, by adding another term to the learning or improvement curve equation

we would obtain:

Hours/Unit = Ub * Rr

or,

Fixed Yr $/Unit First Unit $ * Ub * Rr

Where: U = Unit number

b = Learning curve slope


19

R = Production rate

r = Production rate curve slope

The net effect of adding the production rate effect equation (Rr) is to adjust First Unit $ for

rate. The equation will also yield a different "b" value.

Rate effect may be ignored or can be treated in different ways in different models. If

possible, rate effects should be derived from historical data program behavior patterns observed as

production rates change while holding the learning curve constant.

The rate effect can vary considerably depending on what was required to effect the change.

For example, were new facilities required or did the change involve only a change in manpower or

overtime?

CALIBRATION AND VALIDATION OF COST MODELS

Once data has been collected and normalized, cost models can be developed. Although we

will discuss cost models in much more depth in Chapters IV and V, a few comments are relevant

here.

There are two general types of cost models: internal (contractor developed) and

commercially available. Internal, contractor developed models are derived from unique contractor

data and generally do not require calibration since they have been calibrated in a defacto manner.

On the other hand, commercial models are based on more universal data, and almost always need

some form of calibration to be useful.

The cost driver equation(s) utilized in a commercial cost model are based on a database

external to the specific data being used to support the current estimate. Calibration, then, is the

process of computing a multiplier(s), to be used with the general purpose equation(s), such that the

combined equation(s) will predict the cost as reflected by the technical and programmatic data

being used to support the estimate.

Specialized (Internal) cost models are based directly on the data being used to support the

estimate. Since the CER’s are derived directly from the supporting data, the model is, by definition,

calibrated.


20

The result of calibrating an item, in a commercial model, is a calibration factor which is

used in the commercial model's equations, such that the equations are then made to calculate the

value of the item.

Cost models need to be calibrated and validated for acceptance. The validation of a cost

model is a process which usually includes the following steps:

(1) Calibrate the model to historical cost data.

(2) Estimate the cost of past completed projects.

(3) Compare the estimates with actual costs to demonstrate acceptable accuracy.

It is the combined use of the model with the estimating process that must achieve acceptable

results to provide a basis for the validation of the model. It may also involve disclosure of how the

model works so that the effects of scaling and heuristic analysis can be evaluated by management,

customers or auditors.

Validation implies that interested parties have agreed that the model is a valid and

acceptable estimating tool, and predicts costs within a reasonable range of accuracy.

SOME REVIEW AND AUDIT CONSIDERATIONS

Almost certainly, any data utilized will have to undergo a review and audit, so proper

documentation is a must. Documentation should include:

(1) A record of all mathematical formulas and "goodness of fit" and other statistics.

(2) A record of adjustments to the original cost data and why those adjustments were made.

(3) A statement of work for the historical data; judgment factors should be identified with

rationale.

(4) An audit trail for the data used for validation that will allow rederivation of the adjusted

data and/or CER from the original source(s). This information must be identified and

available upon request.

Any CER’s and data used in a cost model will need to be updated periodically and/or as

agreed to with the PCO. The updating procedure should satisfy the Truth in Negotiation Act


21

(TINA) requirements by the time of agreement on price or another time as agreed upon by the

parties.

DATA NORMALIZATION PROCESS

Specifying an estimating methodology is an important early step in the estimating process.

The basic estimating methodologies (analogy, catalog prices, extrapolation, factors/ratios,

grassroots and parametric) are all data-driven. To use any of these methodologies, credible and

timely data inputs are required. If data required for a specific approach is not available, then that

methodology cannot be used.

Given that all methodologies are data-driven, it is critical that the estimator know the best

data sources. Here are nine basic sources of data and a description of what specific data can be

obtained from each source. Definitions of the differences between primary and secondary sources

of data are provided. Finally, there is a review of the type of information that should be available

from an accounting system, and a description of how to collect and analyze data is also given.

The information presented in this Handbook will help the collection and analysis of the two

data types (primary and secondary) required to specify, and apply a parametric estimating

methodology. Remember - any data needs to be available, reliable and convincing before an

estimating methodology can be chosen that utilizes the foundation data. The two types of data are:

1. Primary data is obtained from the original source. Primary data is considered the best in

quality, and ultimately the most useful.

2. Secondary data is derived (possibly "sanitized") from primary data. It is not obtained

directly from the source. Since it was derived (actually changed) from the original data,

it may be of lower overall quality and usefulness.

When preparing a cost estimate, look for all credible data sources. If at all possible, use

primary sources of data.

There are nine main sources of data and they are listed in the chart below.


22

NINE SOURCES OF DATA

1. Basic Accounting Records Primary

2. Cost Reports Either (Primary or Secondary)

3. Historical Databases Either

4. Functional Specialist Either

5. Other Organizations Either

6. Technical Databases Either

7. Other Information Systems Either

8. Contracts Secondary

9. Cost Proposals Secondary

The following normalization process description is not intended to be all inclusive.

Normalizing Cost Data

Making Units/Elements of Cost Consistent

Making Year of Economics Consistent

Normalizing The Size Data

Weight and Density Comparisons

Weight Contingency Application (weight reduction programs?)

Percent Electronics

Normalizing Products By Mission Application

Grouping Vehicles by Complexity

Calibrating Like Vehicles

Normalizing End Terms For Homogeneity

Account for Absent Cost Items

Removing Inapplicable Cost Items


23

Normalizing Recurring/Non-Recurring Cost

Prime Contractors' Estimates

Time Phased Costs

Flight-Article Equivalent Units

Normalizing State-Of-Development Variables

Mission Uniqueness

Product Uniqueness

Normalizing Environments (Platform):

Manned Space

Unmanned Space

Aerospace

Shipboard

Commercial

Collecting the data to produce an estimate, and evaluating the data for reasonableness, is a

very critical and time-consuming step of the estimating process.

When collecting the data needed to integrate cost, schedule, and technical information for

an estimate, it is important to obtain cost information, and also the technical and schedule

information. The technical and schedule characteristics of programs are important because they

drive cost. They provide the basis for the final cost.

For example, assume the cost of another program is available and a program engineer has

been asked to relate the cost of the program to that of some other program. If the engineer is not

provided with specific technical and schedule information that defines the similar program, the

engineer is not going to be able to accurately compare the programs, nor is he or she going to be

able to respond to questions a cost estimator may have regarding the product being estimated vis-à-

vis the historical data.


24

The bottom line is that the cost analysts and estimators are not solely concerned with cost

data. They need to have technical and schedule data available in order to adjust, interpret, and lend

credence to the cost data being used for estimating purposes.

A cost estimator has to know the standard sources where historical cost data exists. This

knowledge comes from experience and from those people, the so-called local experts, that are

available to answer key questions.

A cost analyst or estimator should be constantly searching out new sources of data. A new

source might keep cost and technical data on some item of importance to the current estimate. Do

not hesitate to ask anyone who might know or be able to help, since it is critical to have relevant

cost, schedule and technical information at all times.

The chart below summarizes important points about data collection and evaluation.

DATA COLLECTION, EVALUATION AND NORMALIZATION

∗ Very Critical, Time Consuming Step

∗ Need Actual Historical Cost, Schedule, and Technical Information

∗ Know Standard Sources

∗ Search Out New Sources

∗ Capture Historical Data

In order to develop a parametric model, a necessary requirement is to possess historical cost,

schedule and technical data on a set of data points. The idea here is that generally more data is

better than less. It is necessary to know what trends exist, and to understand why the trends are as

they are. Some models have been found to be based on the opinions of experts instead of historical

data. Although the opinions of experts may be germane, sound historical data is preferable for

model development, audit and analysis.

In addition to the historical data points, information on the cost, technical and quantity

drivers needs to be examined to determine which does the best job of predicting cost. A statistical

analysis on the data is accomplished to determine the strongest predictor(s) or driver(s) of cost, that

is, the independent variable(s). (See further explanations in Chapter III).


25

It is very important to note that when performing a statistical analysis, be sure that

functional specialists can provide realistic and reliable parameters for independent variables, given

the stage of the program being estimated. Illustrating this point, suppose a statistical relationship is

developed that has very strong correlation, and a potential cost driver has been discovered.

However, data for the same independent variable for the estimate is not available. The parametric

model would not then help with the estimate.

Finally, knowledge of basic statistics, modeling skills and an understanding of analytical

techniques is necessary to develop parametric estimating relationships. (See also Chapter III).

The above information is summarized on the chart below.

TYPE OF INFORMATION NEEDED TO DEVELOP A PARAMETRIC MODEL

∗ Reliable Historical Cost, Schedule, and Technical Data on a Set of Data Points

* WBS, WBS Dictionary & Product Tree

* Analysis to Determine Significant Cost Drivers

* Knowledge of Basic Statistics, Modeling Skills and CER Development

* Analysis Techniques

To use a parametric model, the model needs to be well-documented. The documentation of

a parametric model should include the source of data used to derive the parameters, and the size and

range of the database. Additional information that should be included in the documentation of a

parametric model are: how the parameters were derived, what the model's limitations are, the time

frame of the database and, how well the parametric tool/model estimates its own database. All of

this information should be located in the source document of a parametric model and should be

read before deciding to use it in an estimate. By reading the source document, the strengths and

weakness of the parametric model can be assessed and a determination can be made about any

appropriateness for use.

A statistic called the Mean Absolute Deviation (MAD) can be developed for cost models. It

is a simple statistic that evaluates and assesses how well a parametric model estimates its own

database. For example, if the MAD is 20%, then it means that the parametric equation(s) estimates

its own database within plus or minus 20%. This is an important statistic to know, because if a


26

CER does not estimate its own database well, then its credibility with other data points outside its

database would be questionable.

To successfully use parametric model methodology, a requirement is to obtain realistic,

most-likely range estimates for the independent variable values. Sometimes functional experts are

not sure what the real performance requirements are for a new program. In such cases, a most-likely

range will provide values that reflect an assessment of the associated uncertainties or unknowns (a

Risk Analysis).

Again, the top functional experts who know the program should identify and estimate the

range of cost driving characteristics. They should also confirm the applicability of the specific

parametric cost model from a technical perspective.

The information needed to use a parametric data model is listed on the chart that follows.

TYPE OF INFORMATION NEEDED TO USE A PARAMETRIC MODEL

Well-Documented Parameters Identifying:

Source of data used to derive the parametric model

Size of database

Time frame of database

Range of database

How parameters were derived

Limitations spelled out

How well the parametric model estimates its own database

Consistent and well defined WBS dictionary

Realistic Estimates of Most Likely Range for Independent Variable Values

Top Functional Experts Knowledgeable about The Program You are Estimating

To identify most-likely range for cost drivers

To confirm applicability of parametric from technical perspective

A parametric estimating methodology can be used at any stage of a program's life cycle.

For example, a general parametric model may be utilized in the early, conceptual phase of a

program, although the same parametric model could be inappropriate to use in the follow-on


27

production phase of a program. However, a detailed parametric model used in production

estimating that is based on the experience and actual historical data of two or three previous

production lots, could yield excellent validity.

Hence, a parametric methodology can be used at any stage of a program's life cycle as long

as the parametric model is based on the level and type of information available at that stage.

The methodology can be used for any WBS element, not just hardware and software.

Parametrics can be successfully applied to Systems Engineering/Program Management, Test,

Training and Data, etc., provided that historical data points are available to develop solid, statistical

relationships that provide reliable estimates of independent variables.

PITFALLS TO THE USE OF A PARAMETRIC MODEL

When a parametric model is applied to values outside its database range, the credibility of

the resulting estimate becomes questionable. In cost estimating, one rarely finds large, directly

applicable databases, and the source document has to be evaluated to determine if the parametric

can be applied to the current estimate. However, it is possible to develop parametric tools that

relate cost based on generic complexity values or tables. Such generalized parameters, can be

related to the task at hand by an experienced modeler that results in a good cost model, but a

parametric model always needs to make sense for the present estimate.

Additionally and before using, one should validate models based on expert opinion. This is

accomplished first by obtaining some actual, historical data points (technical, schedule, and cost) on

completed programs similar to the current program. With this data in hand, apply the model to the

actual technical and schedule information and see how well the parametric model predicts the

known cost. If the model estimated the actual costs with an acceptable margin of error, validate the

model for programs that are similar to the historical data point. Careful validation will help insure

that cost models are appropriately used.

Many times a parametric model needs to be adjusted if the new system has cost drivers

and/or requirements that are not reflected in the parametric's database. In some of these cases a

combination of parametric methodology with an approach taken from the analogy methodology can


28

be used to develop an estimate. This is accomplished by adjusting the results of the parametric

approach with scaling or complexity factors that reflect any unique requirements.

For example, parametrics and analogy approaches could be effectively combined to

estimate the costs of a new program for which technology advancement is anticipated. First, either

develop or use an existing parametric model, based on similar data points, to estimate the cost of

the program, without technology advancement.

Second, address the technology advancement by consulting with functional experts to

obtain a most-likely range for a relative scaling factor that will reflect any advancements in

technology. The relative scaling or complexity factor is applied to the result of the parametric

estimate, and adjusts it for the impact of technology advancement. This is a solid and valid

estimating approach for assessing the cost impacts of technology advancement, or other

"complexity" differences.

In such cases, the parametric model has to be adjusted so that it makes sense vis-à-vis the

current estimate.

If there exist no realistic estimates for the independent variable values for the product or

effort being estimated, then parametric models should not be used. The level of uncertainty in the

estimate will be considerable, and the validity of the estimate questionable. In cases such as this,

parametrics can only be used as a benchmark.

As stated previously, it is very difficult for functional specialists to provide a single point

estimate. Moreover, a single point estimate does not reflect the uncertainty inherent in any

functional expert's opinions. Consider requesting most likely range values rather than point

estimates, if possible.

Even after a parametric analysis has been completed it is prudent to follow it with a risk

analysis evaluation of the high risk areas and key cost drivers.

The chart below displays these points.

PITFALLS TO AVOID IN THE USE OF A PARAMETRIC ESTIMATE

* Using the parametric model outside its database range

* Using a parametric model not researched or validated


29

* Using a parametric model without adjustment when new system requirements are not

reflected in the parametric's database

* Using a parametric model without access to realistic estimates of the independent

variables' values for product/effort being estimated

* Requesting point estimates for independent variable value values versus a most likely

range, if possible and practical

TWO ILLUSTRATIONS OF TYPICAL DATA NORMALIZATION PROBLEMS

Illustration 1

You plan to do a parametric estimate of a system using some company history. The system

under investigation is similar to a system built several years ago. The two systems compare as

follows:

Parameter Historical System Prospective System

Date of Fabrication Jul 89-Jun 91 Jul 95-Dec 95

Production Quantity 500 750

Size- Weight 22 lb. external case 20 lb. external case

5 lb. int. chassis 5 lb. int. chassis

8lb. of elec parts 10 lb. elec parts

Volume 1 cu ft-roughly cubical .75 cu ft-rec. solid

12.l x ll.5 x l2.5 8 x lO x l6.2

Other Prog Features Manual of oper incl. Minor chgs to manual

5% Elec parts as spares

Normalization: In this instance, we would require adjusting for inflation factors, the

quantity difference, the rate of production effect and the added elements in the original program (the

spare parts and manual). The analyst should be careful normalizing these data. General inflation

factors are almost certainly not appropriate to most situations. Ideally, the analyst will have a good

index of costs which is specific to the industry and will use labor cost adjustments specific to

his/her company. The quantity and rate adjustments will have to consider the quantity effects on


30

the company's vendors and the ratio of overhead and setup to the total production cost. Likewise,

with rate factors each labor element will have to be examined to determine how strongly the rate

affects labor costs. On the other hand, the physical parameters do not suggest that significant

adjustments or normalizations are required.

The first order normalization of the historic data would consist of:

1) Material escalation using industry or company material cost history.

2) Labor escalation using company history.

3) Material quantity price breaks using company history.

4) Possible production rate effects on touch labor (if any) and unit overhead costs.

Because both cases are single lot batches, and are within a factor of two in quantity, only a

small learning curve or production rate adjustment likely is required.

Illustration 2

You are considering building some equipment for the first time. Relevant labor effort

history relates to equipment built some time ago but reliable data is only available from the third

lot, beginning at unit 250. The available history indicates a cost improvement curve on this history

of 95% from the fourth lot on. This history is considered the best available.

Normalization here requires two steps. Unless you believe the early lot cost improvement

curve is the same as the historical lot improvement curve of the later lots (unlikely), you will need

to identify cost improvement curves which may be applicable. These data points will have to be

normalized to some common condition and the resulting improvement curve applied to the case at

hand. Suppose the relevant history is as follows:

Case Date Quan. Lots Rate of Prod. Improvement Curve

Case A 1985-87 1000 6 5 per day 90%

Case B 1990 400 2 2 per day 83%

Case C 1991 350 3 1.9 per day 91%


31

Case D 1993-95 5000 10 7 per day 93%

Clearly, the selection of an appropriate learning curve will be a judgment call that will

depend upon the case at hand and how it relates to the historical data. For instance, there is some

suggestion that the fewer the number of lots, the steeper the learning curve. How does the

production rate and quantity match the data? In this situation, the analyst could break up the

production history and examine the relationship of the learning curve to the lot sequence, the

quantity, and the production rate. The date may reflect the technology involved and any historical

comparisons will have to include the impact of relevant technology.

SUMMARY

In summary, let’s explore why there might be a database problem. Given the large amount

data available on a hardware system, one might wonder why building a database presents such a

problem. Let's consider a few of the more important reasons:

(1) Information in the wrong format. While we do indeed possess a great deal of data, in

many cases this data is not in an appropriate format to be very useful in developing CER’s. The

reason is primarily due to the fact that the information systems which generate required data were

originally established to meet organizational objectives that are no longer relevant or are not

concerned about cost estimating or analysis. In short, the orientation of a large number of past and

existing information systems is toward the input side with little or no provision for making

meaningful translations reflecting output data useful in CER development or similar types of

analysis.

(2) Differences in definitions of categories and inconsistent WBS's and WBS categories.

We also have a problem in building a database when the definition of the content of cost categories

fails to correspond to the definition of analogous categories in existing databases. For example,

some analysts put engineering drawings into the data category while others put engineering

drawings into the engineering category. A properly defined WBS product tree and dictionary can

avoid or minimize these inconsistencies.


32

(3) The Influence of Temporal Factors. It goes without saying that historical data is

generated over time. This means that numerous dynamic factors will influence data being collected

in certain areas. For example, the definition of the content of various cost categories being used to

accumulate the historical data may change as a system evolves. Similarly, inflation changes will

occur and be reflected in the cost data being collected over time. In addition, as the DoD deals with

a rapidly changing technical environment, cost data generated for a given era or class of technology

is necessarily limited. In fact, we would consider ourselves especially lucky if we can obtain five to

ten good data points for certain types of hardware. Too few data points can lead to problems in our

attempts to develop statistically sound CER’s for forecasting costs into the near and distant future.

(4) Comparability Problems. Such problems include, but are not limited to, changes to a

company's department numbers, accounting systems and disclosure statements, changes from

indirect to direct change personnel for a given function, among others.

To summarize, when we develop a database, we must normalize that database to be sure we

have comparable data. For example, if we are building a database with cost data, we must first

remove the effects of inflation so that all costs are displayed in constant dollars. We must also

normalize the data for consistency in content; that is, we must ensure that a particular cost category

has the same definition in terms of content. Normalizing cost data is a challenging problem, but it

is a problem which must be resolved if a good database is to be constructed.

Resolving database problems so that an information system exists to meet our needs is not

easy. For example cost analysis methodologies typically vary considerably from one analysis or

estimate to another. The requirements for CER’s -- hence the data and information requirements --

are not constant over time. In short, the analyst who is working in support of long range forecasting

needs cannot specify information needs once and for all. These needs must be reviewed

periodically.

The maintenance and update of any database must also be considered. An outdated

database may be of very little use in forecasting future acquisition costs. However, we must also

consider the costs of maintaining and updating a database. We may find that we simply cannot

afford to provide for many current databases. Finally, since today we deal with a set of rapidly

changing technology, we may find ourselves limited to developing a CER with a relatively small

database.


33

Resolving the database problem must, at a minimum, focus on being provided with an

explicit SOW within the RFP and standardized cost category definitions. Then we must consider

how to manage and utilize the database we have within our individual organizations.


34

CHAPTER III

ELEMENTARY STATISTICAL TECHNIQUES

AND

CER DEVELOPMENT


35

CHAPTER III

ELEMENTARY STATISTICAL TECHNIQUES AND CER DEVELOPMENT

Proper CER development and application depends heavily upon understanding of certain

mathematical and statistical techniques. Some of the simpler and more widely used techniques

are explained in this chapter. Many other techniques are available in standard statistical text

books.

CER AND MODEL DEVELOPMENT - UNCERTAINTY AND RISK REDUCTION

CER development and cost modeling are forms of forecasting real world cost events.

How much effort will it take to build a bomber or a space sensor? An estimate has a probability

of being within a given percentage of the “correct” answer. The better the project cost data and

cost model, the closer will be the predicted cost to the final actual cost at project completion.

Since a model of the real world involves simplifications, the final actual cost will rarely equal the

estimated cost.

These modeling uncertainties translate into “risk”” of producing a realistic cost estimate

within a given percentage of the final actual project cost. Such uncertainties can be grouped into

two major categories:

1. Uncertainty of any organization to perform as planned due to unexpected resource

or schedulings in the scope of effort to produce the design, prototype or product, and,

2. Uncertainty associated with the development (and, hence, usefulness) of any cost

model. This item includes:

a. Uncertainty associated with omission of a key cost driver,

b. Mis-specification of the form of the model equation,

c. Modeling limitations associated with a lack of data, and,

d. Data consistency across multiple project databases.


36

The first type of risk, (1) can be addressed through improved specifications in the scope

of work and an improvement in the clarity of understanding between the customer and

contractor. This would result in a decrease in the amount of contract and engineering change

proposals (CCE/ECPs) and unnecessary rework.

One of the second types of risk, (2a), uncertainty associated with omission of key cost

drivers, is addressed through development of historical cost data by product line, defined Work

Breakdown Structure (WBS), and systematic understanding of the types of cost associated with

development, prototype, and production programs.

In risk (2b), the avoidance of the mis-specification of the form of the model equation,

careful review of in-house and available industry data is required to create the basis of the model.

Affirmative answers are required to the following four questions. Does the model make basis

economic sense? Is the cost estimate an extrapolation within the scope of the cost and database?

Is this type of cost estimate no higher a risk estimate than an interpolation between existing

historical data in size and complexity? How well do the outputs compare to history? If poor

patterns exist between the model and history, it is likely that an important parameter has been left

out. The form of the cost equation should then be re-examined.

Risk (2c) deals with the question about the sufficiency of data points that are available to

validate the cost model. A lack of data points will increase the uncertainty of the cost model

equation. Can more data points be included? The use of additional relevant data can increase the

validity of any model. In the absence of substantial data, have experts critique the reasonableness

of the model outputs.

Usually there is a lack of data in a relatively new technological or programmatic area of

cost modeling. How can we model the costs associated with an improvement in processes or

teaming arrangements that improve producibility and lower costs relative to the old ways of

program management and control? Metric data is now being gathered within many companies,

so this data issue should become much less severe in years to come. In the meantime,

organizational process improvement curves, a variation of the quantity of production (repetition)

based on historical cost improvement curves, can be used to evaluate the benefits of process

improvements.


37

Risk (2d), data consistency, can be mitigated by using Work Breakdown Structure (WBS)

cost elements which are standardized across projects. (At the end of this Handbook, Appendix B

contains a treatise about developing Work Breakdown Structures). If definitions for system

engineering, project management, etc., are not consistent, then the resulting cost model and its

cost estimating capability for each of the WBS elements may be flawed. Cost analysts need to

have at their disposal a correct WBS cost estimating tool, complete with a WBS dictionary. if a

cost model is developed for similar hardware items (objects) across programs, then the definition

of these items should be standardized as much as possible. Differences in definitions from

program to program or between product lines are captured by the complexity of labor, or material

cost differences.

DEVELOPING COST ESTIMATING RELATIONS (CERs)

For CERs to be valid, they must be developed based on sound statistical concepts. Once

valid CER’s have been developed, then parametric cost modeling can proceed. The following

discussion will review some of the more commonly used statistical techniques for CER

development.

CERs are the key tool used in estimating by the cost analyst. CERs may be used at any

time in the estimating process. For example, CERs may be used in the concept or validation

phase of estimating program costs because there is insufficient system definition to use anything

else. CERs may also be used in later phases of estimating program costs as a cross check of

another estimating procedure, or as the primary basis for an estimate.

The value of a CER is dependent upon the soundness of the database from which the

CER is developed and the appropriateness of that CER to what is to be estimated. Determination

of the “goodness” of a CER and its applicability to the system being estimated requires a

thorough analysis by the cost analyst. The process of validating and selecting a CER is the

subject of this chapter. We will begin by defining CERs and then look at how a CER may be

developed. Next, we will look at when to use a CER and note the strengths and weaknesses of

CERs. Finally, we will consider the techniques for developing CERs; the linear regression model


38

of the Least Squares Best Fit (LSBF) model, along with a few other statistical techniques.

Appendix D describes additional statistical techniques.

We can make a few general observations about CER development. We know that CERs

are analytical equations which relate various categories of cost (either in dollars or physical units)

to cost drivers or explanatory variables. We also know that CERs can take numerous

forms,ranging from informal rules of thumb or simple analogies to formal mathematical

functions derived from statistical analysis of empirical data. If we are going to develop a CER,

however, then we need to focus on assembling and refining the data that constitute the empirical

basis for that CER.

In deriving a CER, assembling the database is especially important and, generally, a very

time-consuming activity (see Chapter II). Deriving valid CERs is a difficult task and the number

of really good, valid CERs is significantly fewer than one might expect. While there are many

reasons for the lack of valid CERs, the number one reason is the lack of an appropriate database.

When developing a CER, the analyst must first hypothesize a logical estimating

relationship. For example, does it make sense to expect that costs will increase as aircraft engine

thrusts increase? Given that it does make sense that cost and engine thrust has some direct

relationship, the analyst will need to refine that hypothesis to determine whether the relationship

is linear or curvilinear. After developing a hypothetical relationship, the analyst needs to

assemble a database.

Sometimes, when assembling a database, the analyst discovers that the raw data is at least

partially in the wrong format for analytical purposes and displays irregularities and

inconsistencies. Adjustments to the raw data, therefore, almost always need to be made to ensure

a reasonably consistent and comparable database. It is important to note that no degree of

sophistication in the use of advanced mathematical statistics can compensate for a seriously

deficient database.

Since the data problem is fundamental, typically a considerable amount of time is devoted

to collecting data, adjusting that data to help ensure consistency and comparability, and providing

DEFINITION: Cost Estimating Relationships (CER’s) are mathematical expressions relating

cost as the dependent variable to one or more independent cost driving variables.


39

for proper storage or information so that it can be rapidly retrieved when it is needed. In fact,

more effort is typically devoted to assembling a quality database than to anything else. Given the

appropriate information, however, the analytical task of deriving CER equations is often

relatively easy.

Hypothesis Testing Of A Logical CER

Complementing the issues of deriving a good database is the need to hypothesize what

the CER should be. Some analysts believe the hypothesis comes first, then the data search to

build a good database. Other analysts believe the data search comes first, and given the

availability of data, the subsequent determination of a logical relationship or hypothesis occurs.

Regardless of the position taken, the analyst must determine a logical estimating relationship.

The analyst must structure the forecasting model and formulate the hypothesis to be tested. The

work may take several forms depending upon forecasting needs. It involves discussions with

engineers to identify potential cost driving variables, scrutiny of the technical and cost proposals,

and identification of cost relationships. Only with an understanding of hardware requirements

can an analyst attempt to hypothesize a forecasting model necessary to develop a CER.

The CER Model

Once the database is developed and a hypothesis determined, the analyst is ready to

mathematically model the CER. While this analysis can take several forms, both linear and

curvilinear, we will initially consider one simple model -- the LSBF model. A number of

statistical PC packages are available to generate the LSBF equation, but we will manually

develop the LSBF equation in the latter part of this chapter. Once established, the database and

the hypothesis testing complete the modeling activity and the equations are then relatively easy to

derive.

WHEN TO USE A CER

When a CER has been build from an assembled database based on a hypothesized logical

statistical relationship, and within an accepted or revised hypothesis, one is ready to apply the


40

CER. It may be used to forecast future costs or it may be used as a cross check of an estimate

done with another estimating technique. For example, one may have generated an estimate using

a grassroots approach -- a detailed build up by hours and rates -- then “benchmark” the grassroots

estimate with a CER.

A CER built for a specific forecast may be used with far more confidence than a generic

CER. One must be especially careful in using a generic CER when the characteristics of the

forecast universe are, or are likely to be, different from those reflected in the CER. One may

need to qualify a generic CER by reviewing the database and the assumptions made for its use.

A need to update the database with data appropriate for forecasting may be necessary. One may

also find that the generic CER can be used only as a starting point.

When using a generic CER as a point-of-departure, enhance or modify the forecast in

light of any other available supplementary information. This most likely will involve several

iterations before the final forecast is determined. It is important to carefully document the

iterations so that an audit trail exists explaining how the generic CER evolved to become the

final forecast.

In using any CER -- specially built or generic -- the main theme is invariably: Be careful;

Use good judgment! CERs are a fundamental estimating tool used by cost analysts. However,

we must us them while applying a good deal of judgment. For example, we need to be concerned

about the uncertainty of a future system in terms of chance elements in the “real” world; that is,

uncertainty about the state of the world in terms of technological, strategic, and political factors.

Also, we must carefully document as we go, so that an all important audit trail is maintained.

STRENGTHS AND WEAKNESSES OF CERs

In applying good judgment in the use of CERs, we need to remain mindful of the

strengths and weaknesses of CERs. Some of the more common ones are presented below:

Strengths


41

1. One of the principle strengths of CERs is that they are quick and easy to use. Given a

CER equation and the required input data, one can generally turn out an estimate

quickly.

2. A CER can be used with limited system information. Consequently, CERs are

especially useful in the RDT&E phase of a program.

3. A CER is an excellent (statistically sound) predictor if derived from a sound database,

and can be relied upon to produce quality estimates.

Weaknesses

1. CERs are sometimes too simplistic to forecast costs. Generally, if one has detailed

information, the detail may be reliably used for estimates. If available, another

estimating approach may be selected rather than a CER.

2. Problems with the database may mean that a particular CER should not be used.

While the analyst developing a CER should validate that CER, it is the responsibility

of any user to validate the CER by reviewing the source documentation. Read what

the CER is supposed to estimate, what data were used to build that CER, how old the

data are, how they were normalized, etc. Never use a cost model without reviewing

its source documentation.

Now that we know what a CER is, how to develop a CER, when to use a CER, and some

of a CER’s strengths and weaknesses, we can develop techniques for building CER’s. The LSBF

technique is only one mathematical transformation of the database - the linear regression model.

Other sophisticated curvilinear models can also be developed, but will not be explored in this

Handbook. An analyst should be mindful that little in the estimating world is linear.

REGRESSION ANALYSIS

The purpose of regression analysis is to improve our ability to predict the next “real

world” occurrence of our dependent variable. Regression analysis may be defined as the

mathematical nature of the association between two variables. The association is determined in


42

the form of a mathematical equation. Such an equation provides the ability to predict one

variable on the basis of the knowledge of the other variable. The variable whose value is to be

predicted is called the dependent variable. The variable about which knowledge is available or

can be obtained is called the independent variable. In other words, the dependent variable is

dependent upon he value of independent variables.

The relationships between variables may be linear or curvilinear. By linear, we mean that

the functional relationship can be described graphically (on a common X-Y coordinate system)

by a straight line and mathematically by the common form:

y = a + bx

where y = (represents) the calculated value of y - the dependent variable

x = the independent variable

b = the slope of the line, the change in y divided by the corresponding

change in x

a and b are constants for any value of x and y

Looking at the bi-variate regression equation -- the linear relationship of two variables --

we find that regression analysis can be described by an equation. The equation consists of two

distinctive parts, the functional part and the random part. The equation for a bi-variate regression

population is:

Y = A + BX + E

where A + BX is the functional part (a straight line) and E is the random part. A and B

are parameters of the population that exactly describe the intercept and slope of the

relationship.

E represents the ran or “error” part of the equation. The random part of the equation is

always there because the errors of assigning values, the errors of measurement, and errors

of observation. These types of errors are always with us because of our human

limitations, and the limitations associated with real world events.

Since it is practically impossible to capture data for an entire population, we normally

work with a sample from that population. We denote that we are working with a sample by

adjusting our equation to the form:


43

y = a + bx + e,

where a + bx represents the functional part of the equation and e represents the random part. Our

estimate of A and B in the population are represented by a and b, respectively, in the sample

equation. In this sense then, a and b are statistics. That is, they are estimates of population

parameters. As statistics, they are subject to sampling errors. As such, a good random sampling

plan is important.

1. The values of the dependent variable are distributed by a normal distribution function

about the regression line.

2. The mean value of each distribution lies on the regression line.

3. The variance of each array of the independent variable is constant.

4. The error term in any observation is independent of the error term in all other

observations. When this assumption is violated, data is said autocorrelated. This

assumption fixes the error term to be a truly random variable.

5. There are no errors in the values of the independent variables. The regression model

specifies that the independent variable be a fixed number -- not a random variable.

For example, you wish to estimate the cost of a new bomber aircraft at mach 2, then

mach 2 is the value of the independent variable. Mach 2 is a fixed number. The

regression model will not handle errors in the independent variables.

6. All causation in the model is one way. This simply means that if causation is built

into the model, the causation must go from he independent variable to the dependent

variable. Causation, though neither statistical nor a mathematical requirements, is a

highly desirable attribute when using the regression model for forecasting. Causation,

of course, is what we, as cost analysts, are expected to determine. We do this in our

hypothesis of a logical mathematical relationship in either building or reviewing a

CER equation.

CURVE FITTING


44

There are two standard methods of curve fitting. One method has the analyst plot the data

and fit a smooth curve to the data. This is known as the graphical method. The other method

uses formulas or a “best-fit” approach where an appropriate theoretical curve is assumed and

mathematical procedures are used to provide the one “best-fit” curve; this is known as the Least

Squares Best Fit (LSBF) method.

We are going to work the simplest model to handle, the straight line, which is expressed

as:

Y = a + bx

Graphical Method

To apply the graphical method, the data must first be plotted on graph paper. No attempt

should be made to make the smooth curve actually pass through the data points which have been

plotted; rather, the curve should pass between the data points leaving approximately an equal

number of points on either side of the line. For linear data, a clear ruler or other straightedge may

be used to fit the curve. The objective in fitting the curve is to “best-fit” the curve to the data

points plotted; that is, each data point plotted is equally important and the curve you fit must

consider each and every data point.

Although considered a rather outdated technique today, plotting the data is still always a

good idea. By plotting the data, we get a picture of the relationship and can easily focus on those

points which may require further investigation. Hence, as a first step, we shold plot the data and

note any data points which may require further investigations before developing a forecasting

graphical curve or mathematical equation.

LSBF Method

The LSBF method specifies the one line which best fits the data set we are working with.

The method does this by minimizing the sum of the squared deviations of the observed values of

Y and calculated values of Y. For example, if the distances: (Y1 - YC1,), (Y2 - YC2), (Y3 - YC3),

(Y4 - YC4), etc., parallel to the Y-axis, are measured from the observed data points to the curve,

then the LSBF line is the one that minimizes the following equation (see Figure III-1):

(Y1 - YC1)2 + (Y2 - YC2)

2 + (Y3 - YC3)2 + ... + (Yn - YCn)

2


45

y • yc4

y3 • • y4

• yc3

y1 • • yc2

• yc1 • y2

x

Figure III-1. The LSBF Line.

In other words, the sum of the deviations from the observed value of Y, and the

calculated value of Y - Yc squared, is a minimum; i.e., (Y - YC)2 is a minimum. This same

distance, (Y - YC) is the error term or residual. Therefore, the LSBF line is one that can be

defined as:

ΣE2 is a minimum because Σ (Y - YC)2 = ΣE2

For a straight line,

Y = a + bx

and, with N points, we have

(X1, Y1,), (X2, Y2), (X3, Y3), ... (Yn , Yn)

The sum of the squares of the distances is a minimum if,

ΣY = AN + BΣX and

ΣXY = AΣX + BΣX2

These two equations are called the normal equations of the LSBF line. Reference to any

comprehensive statistical textbook will illustrate that these two equations do meet the

requirements of the properties of ordinary LSBF regression. These properties are:

1. The technique considers all points.

2. The sum of the deviations between the line and observed points is zero, that is,

Σ(Y - Yc)2 = ΣE2 = a minimum.

Similarities between these two properties and the arithmetic mean should also be

observed. The arithmetic mean is the use of the values of the independent variable divided by


46

the number of observations or ΣX/n = x and the sum of the “Ys” divided by the number of

observations or Σy/n = y . Now, instead of considering the mean as a point when dealing with

only one variable, we are now using a line -- the LSBF regression line. Note that:

The parameters, a and b, define a unique line with a Y-intercept of a and a slope of b.

To calculate the values needed to solve for a and b, we need a spreadsheet (See Table III-

1). For example, use the values in Table III-2.

Table III-1. Table Needed to Get Sums, Squares and Cross Products

X Y X*Y X2 Y2

X1 Y1 X1 * Y1 X12 Y12

X2 Y2 X2 * Y2 X22 Y22

X3 Y3 X3 * Y3 X32 Y32

- - - - -

- - - - -

ΣXn ΣYn Σ(Xn * Yn) ΣXn2 ΣYn2

Table III-2

X Y X Y X2 Y2

4 10 40 16 100

11 24 264 121 576

3 8 24 9 64

9 12 108 81 144

7 9 63 49 81

2 3 6 4 9

36 66 505 280 974

We can substitute the table values into the equations for b and a:

where x x n= ∑ / and, y y n= ∑ /

Solving for b, we get:

bxy nx y

x nx=

−

−

∑

∑2 2

b = 505 - 6(6)11


47

280 - 6(6)2 b = 109 64 b = 1.703125 Solving for a yields:

a y bx= −

a = −11 1703125 6( . )

a =.78128

Therefore, the regression equation (calculated y) is

Y xc = +. .78128 170312

Multiple Regression

In simple regression analysis, a single independent variable (X) is used to estimate the

dependent variable (Y), and the relationship is assumed to be linear (a straight line). This is the

most common form of regression analysis used in contract pricing. However, there are more

complex versions of the regression equation that can be used to consider the effects of more than

one independent variable on Y. That is, multiple regression analysis assumes that the change in

Y can be better explained by using more than one independent variable. For example,

automobile gasoline consumption may be largely explained by the number of miles driven.

However, it might be better explained if we also considered factors such as the weight of the

automobile. In this case, the value of Y would be explained by two independent variables.

Yc = A + B1X1 + B2X2

where: Yc = the calculated or estimated value for the dependent variable

A = the Y intercept, the value of Y when X = 0

X1 = the first independent (explanatory) variable

B1 = the slope of the line related to the change in X1, the value by which change

when X1 changes by one

X2 = the second independent variable


48

B2 = the slope of the line related to the change in X2, the value by which change

when X2 changes by one

Multiple regression will not be considered in depth in this Handbook. Consult a statistics

text to learn more about multiple regression.

Curvilinear Regression

In some cases, the relationship between the independent variable(s) may not be linear.

Instead, a graph of the relationship on ordinary graph paper would depict a curve. For example,

improvement curve analysis uses a special form of curvilinear regression. As with multiple

regression, consult a statistics text to learn more about curvilinear regression.

“GOODNESS” OF FIT, R AND R2

Now that we have developed the LSBF regression equations, we need to determine how

good the equation is. That is, we would like to know how good a forecast we will get by using

our equation. In order to answer this question, we must consider a check for the “goodness” of

fit, the coefficient of correlation (R) and the related coefficient of determination (R2). There are

a number of other statistics we could check which would expand upon our knowledge of the

regression equation and our assurance of its forecasting capability. Some of these will be

discussed late.

Correlation Analysis

One indicator of the “goodness” of fit of a LSBF regression equation is correlation

analysis. Correlation analysis considers how closely the observed points fall to the LSBF

regression equation we develop. The assumption is that the more closely the observed values are

to the regression equation, the better the fit; hence, the more confidence we can expect to have in

the forecasting capability of our equation. It is important to note that correlation analysis refers

only to the “goodness” of fit or how closely the observed values are to the regression equation.

Correlation analysis tells us nothing about cause and effect, however.


49

Coefficient Of Determination

The coefficient of determination (R2) represents the proportion of variation in the

dependent variable that has been explained or accounted for by the regression line. The value of

the coefficient of determination may vary from zero to one. A coefficient of determination of

zero indicates that none of the variation in Y is explained by the regression equation; whereas a

coefficient of determination of one indicates that 100 percent of the variation of Y has been

explained by the regression equation. Graphically, when the R2 is zero, the observed values

appear as in Figure III-2 (bottom) and when the R2 is one, the observed values all fall right on the

regression line as in Figure III-2 (top).

y y

Perfect Positive x Perfect Negative x R2 = 1.0 R2 = 1.0

y

No Correlation x

R2 = 0

FIGURE III-2

In order to calculate R2 we need to use the equation:

Rxy nx y

x x x y y y

2

2

2 2=

−

− • −

∑

∑∑∑∑

( )

( ) ( )

R2 tells us the proportion of total variation that is explained by the regression line. Thus

R2 is a relative measure of the “goodness” of fit of the observed data points to the regression line.

For example, if we calculate R2 using the formula above and find that R2 = 0.70, this means that

70% of the total variation in the observed values of Y is explained by the observed values of X.

Similarly, if R2 = 0.50, then 50% of the variation in Y is explained by X. If the regression line

perfectly fits all the observed data points, then all residuals will be zero, which means that R2 =


50

1.00. In other words, a perfect straight-line fit will always yield R2 = 1. As the level of fit

becomes less accurate, less and less of the variation in Y is explained by Y’s relation with X,

which means that R2 must decrease. The lowest value of R2 is 0, which means that none of the

variation in Y is explained by the observed values of X. Some applications require R2 of at least

0.7 or 0.8. An R2 < 0.25, which corresponds to an R < 0.5, would never be acceptable.

Coefficient Of Correlation

The coefficient of correlation (R) measures both the strength and direction of the

relationship between X and Y. The meaning of the coefficient of correlation is not as explicit as

that of the coefficient of determination.

We can determine whether R is positive or negative by noting the sign of the scope of the

line, or b. In other words, R takes the same sign as the slope; if b is positive, use the positive root

of R2 and vice versa. For example, if R2 = 0.81; then R = + 0.9 and we determine whether R

takes the positive root (+) or the negative root (-) by noting the sign of b. If b is negative, then we

use the negative root of R2 to determine R. So to calculate R we need to know the sign of the

slope of the line.

It is most important to note that R does not tell us how much of the variation in Y is

explained by the regression line. R is only valuable in telling us whether we have a direct or an

inverse relationship and as a general indicator of the strength of the association.

THE LEARNING CURVE

Basic form of the “learning curve” equation is,

y = a*xb or, Log y = Log a + b Log x

where,

y = Cost of Unit #x (or average for x units)

a = Cost of first unit

b = Learning curve coefficient

Note that the equation Log y = Log a + b Log x is of precisely the same form as the

linear equation y = a + bx. This means that the equation Log y = Log a + b Log x can be


51

graphed as a straight line on log-log graph paper and all the regression formulae apply to this

equation just as they do to the equation y = a + bx. In order to derive a learning curve from cost

data (units or lots) the regression equations need to be used, whether or not the calculations are

performed manually or using a statistical package for your personal computer. In this sense, the

learning curve equation is a special case of the LSBF technique.

Since in learning curve methodologies cost is assumed to decrease by a fixed amount

each time quantity doubles, then this constant is called the learning curve “slope” or percentage

(i.e., 90%). For example,

For unit #1

Y1 = A(1)b = A (First Unit Cost) and

For unit #2

Y2 = A(2)b = Second Unit Cost

So,

Y2 = A*(2)b = 2b = a Constant, or “Slope” Y1 A

Slope = 2b, and, Log Slope = bLog2

Therefore, b = Log Slope

Log2

For a 90% “Slope,”

b = Log .9 = -0.152

Log 2

If we assume that A = 1.0, then the relative cost between any units can be computed.

Y3 = (3)-0.152 = 0.8462

Y6 = (6)-0.152 = 0.7616

Note that:

Y6 = .7616 = 0.9 Y3 .8462


52

Any good statistical package (for instance StatView) can perform all the calculations (and

many others) shown. A quality package will let you customize your results (create presentations)

save your work, and calculate all these statistics: frequency distributions, percentiles, t-tests,

variance tests, Pearson correlation and covariance, regression, ANOVA, factor analysis and

more. Graphics and tables such as scattergrams, line charts, pie charts, bar charts, histograms,

percentiles, factors, etc., should be available to the user. Statistical analysis is greatly simplified

using these tools.

LIMITATIONS, ERRORS AND CAVEATS OF LSBF TECHNIQUES.

When working with the LSBF technique, there are a number of limitations, errors and

caveats of which we need to be aware. The following are some of the more obvious.

Extrapolation Beyond The Range Of The Observed Data

A LSBF equation is truly valid only over the same range as the one from which the

sample data was initially taken. In forecasting into the future, we do not know the shape of the

curve; what we do know is that there is more estimating risk involved. We will give less

credence to forecasts which exceed the range of the original data. However, this does not mean

that we should never extrapolate beyond the relevant range; it may well be that forecasting

beyond the relevant range is the only suitable alternative we have. What we must keep in mind is

that extrapolation assigns values using a relationship that has been measured for circumstances

which may differ from those for the forecast. That is, we are assuming that the past will perfectly

predict the future -- and that will not always be true.

Cause And Effect

Regression and correlation analysis can in no way determine cause and effect. It is up to

the analyst to do a logic check, determine an appropriate hypothesis and analyze the data base

such that an assessment can be made regarding cause and effect. For example, an R2 = .95

relates the number of public telephones in a city to liquor sales. Clearly, there is no cause and

effect involved here. A deeper variable, population, is the truly independent variable that drives


53

both the number of public telephones and liquor sales. As analysts, we must ensure that we have

chosen approximately related data sets and CERS, and that real cause and effect is at work. Be

careful not to find relationships when they do not exist.

Using Past Trends To Estimate Future Trends

Conditions can change: if the underlying population changes due to changes in

technology, for example, then the LSBF equation would not be suitably used as a forecasting

tool. In using a CER, we need to make sure the factors in the forecast still apply to the original

historical LSBF equation.

Misinterpreting The Coefficients Of Correlation And Determination

R will always be greater than R2. If R2 is 0.64, then R is 0.80. Don’t confuse the two.

Summary

In this chapter we have so far discussed the development and use of Cost Estimating

Relationships (CERs). We noted that in using or developing a CER, a high quality database is

most critical (see Chapter II). Specifically, we highlighted the difficulties of assembling a good

database for CER development and indicated several reasons why there could be a problem. We

next considered the strengths and weaknesses of a CER. Finally, we developed one simple

model for generating a CER -- the LSBF model. We completed our discussion of CER’s by

identifying several limitations, errors and caveats when using CERs. Next, we’ll consider the use

of CERs.

EXAMPLES OF CER USE

Basically, CER’s reflect changes in prices or costs (in constant dollars) as some physical,

performance or other cost-driving parameter(s) is changed. The same parameter(s) for a new

item or service can be input to the CER model and a new price or cost can be estimated. Such

relationships may be practically applied to a wide variety of items and services.


54

Construction

Many construction contractors use a rule of thumb which relates floor space to building

cost. Once a general structural design is determined, the contractor or buyer can use this

relationship to estimate total building price or cost - excluding the cost of land. For example, if

we were building a brick two-story house with a basement, we may use $60/square foot (or

whatever value is currently reasonable for the application) to estimate the price of the house.

$60/sq ft x 2200 sq ft = $132,000 house price

Weapons Procurement

In the purchase of an airplane, CERs are often used to estimate the cost of the various

parts of the aircraft. One item may be the price for a wing of a certain type of airplane, a

supersonic fighter for example. History may enable the analyst to develop a CER relating wing

surface area to cost. You may find that there is an estimated $40,000 of wing cost (for instance

NRE) not related to surface area and another $1000/square foot that is related to surface area to

build one wing. For a wing with 200 square feet of surface area we could estimate a price as:

estimated price = $40,000 + 200 sq ft x $1,000 per sq ft

= $40,000 + 200,000

= $240,000

Electronics

Manufacturers of certain electronic items have discovered that the cost of completed

items varies directly with the number of total electronic parts in the item. Thus, the sum of the

number of resistors, capacitors, inductors, and transistors in a specific circuit design may serve as

an independent variable (cost driver) in a predictive CER. Assume a CER analysis indicates that

$0.57 a unit is not related to the number of components with another $.11 added per part. If

evaluation of the drawing revealed that an item was designed to contain 11 capacitors, 12

resistors, 5 transistors, and 2 inductors, the total part count becomes 30. Substituting the 30 parts

into the CER:

estimated cost = $0.57 + $.11 per part * number of parts

= $0.57 + $.11 (30)

= $0.57 + $3.3


55

= $3.87

Cost And Price Analysis

CERs can be used for price analysis in a variety of ways, including aircraft engines, ships,

trucks, and passenger autos. Generally, under the Truth in Negotation Act, price analysis is

required even when cost analysis is used. CERs provide an extremely useful tool for price

analysis after the completion of cost analysis.

Like other techniques of price analysis, the CER approach requires the evaluator to

determine whether the techniques used for comparison are fair and reasonable. To do this, the

evaluator must determine the basis for the estimate (CERs) and its reliability. Some questions

for this evaluation relating to how the estimate was made include:

1. What was the source of information?

2. What information and techniques were used?

3. How reliable were the earlier estimates?

The same parametric techniques used for developing estimates can be used in cost and

price analysis. Often the estimator uses known CER values as the basis for preparing an

estimate. Likewise, estimates can be based on CERs from “similar to” past history.

Information And Techniques

Detailed analysis of a requirement requires an in-depth study of specifications and

drawings, a physical inspection or “tear down” of the item or a similar item, or an analysis of

similar work. From such an analysis, the estimator can get a clear picture of the item or the

service to be performed, and develop appropriate information and cost relationships. Once the

tasks have been delineated, the user can estimate costs using personal experience, published time

standards, component purchase prices, CERs and other data.

In using this data for making an estimate, the estimator must consider the effects of such

factors as dollar value changes and changes in technology. Quantitative tools, such as use of

index numbers, can be used to allow for changes in the value of the dollar. Changes in

technology are harder to contend with in cost analysis. Changes in materials and changes in

manufacturing procedures occur today at an ever increasing rate. It is estimated that technology


56

“turns over” today every two years. Technology is an area requiring user expertise and

knowledge for evaluating price effects of the application of new technology, to the current

requirement. The evaluation of the cost impact of technology surges is a difficult task to

anticipate and quantify.

Estimate Reliability

In considering whether or not to use a specific estimate, we need to examine the “track

record” of the cost estimator or the organization providing the estimate. If, in the past, the

estimates have been close to actuals, greater reliance may be placed on the estimate. If estimates

have been significantly above or below known actuals, then lower reliability should be placed on

current estimates. Knowing the reliability of past estimates does not free the cost or price analyst

from the obligation to review the estimate and the estimating methodology as they relate to

proposal accuracy.

Cost and price analysis should be temperred with product value. Knowledge of the

product, its functions and its use is essential for sound contract pricing. Value analysis is a

systematic and objective evaluation of the function of a product and its related costs. Its purpose

is to ensure optimum value. Questions that help in the evaluation are:

1. What must the product do?

2. What does it cost now? What does it cost to operate and maintain?

3. In what other ways can the function be performed?

4. What will these alternatives cost?

Two Examples Of CER Use

One can utilize Consumer Price Index numbers to perform simple value or price analysis.

Index numbers are, quite simply, CER’s, and predict, from history the effects of inflation. For

example, assume an item of equipment in 1980 cost $28,000 when an appropriate Consumer

Price Index (CPI) was 140. If the current index is now 190 and an offer to sell the equipment for

$40,000 has been suggested, how much of the price increase is due to inflation? How much of

the price increase is due to other factors?

Solution: Px = Ix Py Iy


57

$28,000 = 1.40 Py 1.90 1.4 Py = 1.9 ($28,000) 1.4 Py = $53,2000 Py = $53,200 = $38,000 1.4

$38,000 now is roughly the equivalent of $28,000 in 1980. Hence the price difference due to

inflation is $38,000 - $28,000 = $10,000. The difference due to other causes is $2,000

($40,000 - $38,000).

The above example would illustrate the use of CPI numbers for a material cost

analysis. The steps were:

1. If we know what the price of an item was in the past, and we know the index

numbers for both that time period and today, we can then predict what the price of

that item should be now based on inflation alone.

2. If we have the same information as above, and we have a proposed price, we can

compare that price to what it should be based on inflation alone. If the proposed

price is higher or lower than we expect with inflation, then we must investigate

further to determine why a price or cost is higher or lower.

Consider the purchase of a house as another example that uses the LSBF technique.

Historical data for other homes purchased, may be examined during an analysis of proposed

prices for a newly designed house. Using this data, we can demonstrate a procedure for

developing a CER.

Step 1: Designate and define the dependent variable. In this case we will attempt to

directly estimate the cost of a new house.

Step 2: Select item characteristics to be tested for estimating the dependent variable. A

variety of home characteristics could be used to estimate cost. These include

such characteristics as: square feet of living area, exterior wall surface area,

number of baths, and others.


58

Step 3: Collect data concerning the relationship between the dependent and

independent variable. The example, Table III-3, shows data collected on five

house plans so that we can determine a fair and reasonable price for a house of

2100 square feet and 2.5 baths.

Table III-3.

House Model

Unit Cost

Baths

Sq. Feet Living Area

Sq. Feet Exterior Wall Surface

Burger 166,500 2.5 2,800 2,170

Metro 165,000 2.0 2,700 2,250

Suburban 168,000 3.0 2,860 2,190

Executive 160,000 2.0 2,440 1,990

Ambassador 157,000 2.0 1,600 1,400

New House Unknown 2.5 2,600 2,100

Step 4. Explore the relationship between the independent and dependent variables. As

stated earlier, analysis of the relationship between the item characteristics and the dependent

variable may be performed using a variety of techniques.

Step 5. Determine the relationship that best predicts the dependent variable. Figure III-4

graphically depicts the relationship between the number of baths in the house and the price of the

house. From the graph, there appears to be a relationship between the number of baths and house

price. The relationship, however, may not be a good estimating tool, since three houses with a

nearly $8,000 price difference (12 percent of the most expensive house) have the same number of

baths.

$170

$165

$160

$155

$150

1 2 3 4


59

FIGURE III-4. Number of Bathrooms

Figure III-5 graphically relates square feet of living area to price. In this graph, there

appears to be a strong linear relationship between house price and living area.

$170

Figure III-5. Cost vs Square Feet

Figure III-6 graphically depicts the relationship between price and exterior wall surface

area. Again, there appears to be a linear relationship between house price and this independent

variable.

$170

Figure III-6. Cost vs Exterior Wall Surface (Sq. Ft.)

$165

$160

$155

$150

1400 1500 1600 1700 1800 1900 2000 2100 2200 2300

$165

$160

$155

$150

1400 1600 1800 2000 2200 2400 2600 2800 3000


60

Based on this graphic analysis, it appears that square feet of living area and exterior wall

surface have the most potential for development of a cost estimating relationship. We may now

develop a “line-of-best-fit” graphic relationship by drawing a line through the average of the x

values and the average of the y values and minimizing the vertical distance between the data

points and the line (see Figure III-7 and Figure III-8).

Figure III-7. Linear Trend of Cost to Living Area (Sq. Ft.)

Viewing both these relationships, we might questions whether the Ambassador model

data should be included in developing our CER. In developing a CER, you need not use all

available data if all data is not comparable. However, you should not eliminate data just to get a

better looking relationship. In this case, we find that the Ambassador’s size is substantially

different from the other houses for which we have data and the house for which we are

estimating the price. This substantial difference in size might logically affect the relative

construction cost. The trend relationship in Figure III-8 and Figure III-9, using the data for the

four other houses, would be substantially different than relationships using the Ambassador data.

Based on this information, you might decided not to consider the Ambassador data in CER

development.


61

Figure III-8. Linear Trend of Cost to Exterior Wall Surface (Sq. Ft.)

If you eliminate the Ambassador data, you find that the fit of a straight line relationship of

price to the exterior wall surface is improved. For the relationship of price to square feet of

living area, you find a close relationship, i.e., almost a straight line. (See Figure III-9)

$170

FIGURE III-9. Square Feet (Thousands)

If you had to choose one relationship, you would probably select the one shown in Table

III-3 (square feet of living area) over the relationship involving exterior wall surface because

$160

2400 2600 2800 3000

$168

$166

$164

$162

(2700: $165,000)


62

there is so little variance shown about the trend line. If the analysis of these characteristics did

not reveal a useful predictive relationship, you might consider combining two or more of the

characteristics already discussed, or exploring new characteristics. However, since the

relationship between living area and price is so close, we may reasonably use it for our CER.

In documenting our findings, we can relate the process involved in selecting the living

area for price estimation. We can use the graph developed as an estimating tool. The cost of the

house could be calculated by using the same regression analysis formula discussed herein:

For square feet of living area: Y = $117,750 + $17.50 (2600)

Y = $117,750 + $45,500

Y = $163,250 estimated price

CERs, like most other tools of cost or price analysis, must be used with judgment. Judgment is

required to evaluate the historical relationships in the light of new technology, new design, and

other similar factors. Therefore, a knowledge of the factors involved in CER development is

essential to proper application of the CER. Blind use of any tool can lead to disaster.

COMMON CERs

Table III-4 lists some common CERs used to predict prices or costs of certain items. In

addition to CERs used for estimating total cost and prices, others may be used to estimate and

evaluate individual elements of cost. CERs, for example, are frequently used to estimate labor

hours. Tooling costs may be related to production labor hours, or some other facet of production.

Other direct costs may be directly related to the labor effort involved in a program.

Table III-4. CER Types

Product Independent Variable

Construction

Floor space, roof surface area, wall surface area


63

Product Independent Variable

Gears Net weight, percent of scrap, inches of teeth cut, harness, envelope

Trucks Empty weight, gross weight, horsepower, number of driving axles, loaded cruising speed

Passenger car Curb weight, wheel base, passenger space, horsepower

Turbine engine

Dry weight, maximum thrust, cruise thrust, specific fuel consumption, by-pass ratio, inlet temperature

Reciprocating engine

Dry weight, piston displacement, compression ratio, horsepower

Sheetmetal Net weight, percent of scrap, number of holes drilled, number of rivets placed, inches of welding, volume of envelope

Aircraft Empty weight, speed, useful load, wing area, power, landing speed

Diesel locomotive

Horsepower, weight, cruising speed, maximum load on standard grade at standard speed

ACCEPTANCE CRITERIA FOR A COST ESTIMATING RELATIONSHIP

How good is a CER equation and how good is the CER likely to be for estimating the

cost of new projects? What is the confidence level of answers at +/- x% from the number

estimated, i.e., how likely is the estimated cost to fall within a specified range of cost outcomes?

First, certain necessary conditions for a statistical analysis of a CER need to be stated:

1. There are more data points than coefficients to be estimated.

2. Error terms do not form a systematic pattern.

3. The independent variables are not highly correlated.

4. The form of the equation to be estimated is linear or has been translated into a

linear form using logarithms.

5. The model makes sense from an economics and technical point of view.


64

An R2 of equal or greater than 0.80 is desirable in curve fitting. An R2 of 0.50 associated

with the CER is as good as tossing a balanced coin. The CER explained 1/2 of the observed cost

outcomes. In general, the higher the R2 the better the “explanatory” capability of the cost

equation. However, an R2 of 1.0 can indicate an “identity” of the cost variable and explanatory

variable. The data and explanatory variable being used should then be reexamined for

redundancy.

The “F” statistic measures the ratio of “explanation” of the explanatory variables (cost

drivers) and the “residual” (error) term. The F statistic should have a value greater than 4.0 or 5.0

to indicate that a good cost driver has been selected for the cost model and that the form of the

equation is acceptable. (A value of 1.0 indicates that the cost driver explains only 1/2 of the

variation in the cost. This would not be a particularly good cost driver variable). The higher the F

value the better the prediction capability of the cost drivers. Also, the higher the “F” statistic, the

higher will be the R2 value.

“Partial” F statistics can be used to examine the contribution of a single cost driver term.

The higher the value, as in the “F” statistic, the better the additional contribution of the particular

cost driver. See a statistics text or Appendix D for a more detailed explanation of the “F”

statistics.

The “t” statistic can be used to test the validity of adding a particular cost driver

variable. First, a “null” hypothesis is made that the cost parameter adds no predictive value to

the model (cost equation). That is, the value of the parameter for the cost driver term being

reviewed has a value of 0 (zero). The “t” test is used to make a decision to accept or reject the

“null” hypothesis for a given cost driver term. Generally, a “t” value greater than five leads to

the conclusion to reject the null hypothesis. A “t” value less than five leads to the acceptance of

the null hypothesis, that the cost term does not add predictive value to the CER. Each case to

accept or reject the null hypothesis depends upon the difference between the hypothesized and

estimated coefficients, the confidence interval desired, and the degrees of freedom of the data

(number of data observations minus the numbers of parameters being estimated).


65

The “t” statistic is also used for other applications. For example, in order to determine if

two groups of data are from the same population or from two different populations.

When then degrees of freedom in a data set approach 30, the statistics of the “t”

distribution approach the normal distribution. If it is not known whether a normal distribution is

justified, the “law of large numbers” can be invoked that states that for a large enough sample

(large enough cost database), the error term involved in estimating cost will approach a normal

distribution. That is, the normal distribution can be used instead of the “t” distribution to test the

null hypothesis. The “t” is of similar shape to the normal distribution, except that there is a

larger probability of lower cost or higher cost (extreme outcomes) associated with the “t”

distribution rather than the normal distribution.

Also, an analysis of the plot residual values can be useful. If a pattern exists, then

correlation may be explained by other factors. If the plot of residuals is a scatter plot with no

patterns, then the CER equation may be good if other factors are favorable.

Another important statistical measure is the bandwidth or confidence interval associated

with the application of the CER cost estimates. The bandwidth of the cost estimate depends

upon the confidence interval required or desired, the parameter value, and the degrees of freedom

of the data.

See Appendix D for a more detailed explanation of the “t” distribution and confidence

intervals.

AUDITING AND ANALYZING A CER

CER Analysis

Cost estimating relationships (CERs) relate cost to some other program element in a

definite way. Examples of CERs are per diem rates, “shop supplies”, sales tax estimates, etc.

CERs supposedly relate one cost to another or with a well defined parameter. When rolled into

an interlocking algorithm, analysts have to probe both the estimate and the underlying data used

to develop a CER. What distinguishes a CER from a conventional estimating approach is that

CERs define a general relationship based on a set of data rather than a specific relationship based


66

on a direct precedent. A CER may be less precise than a convential estimating method but the

cost savings resulting from the CER approach may be worth the potential loss of precision.

Within detailed cost estimates, CER’s may be used for estimating small or derivative cost

elements. CER’s are also commonly used for budgetary estimates, “rough order of magnitude”

estimates, and simple cost-benefit calculations when the preliminary or uncertain nature of the

project discourages a costly estimating effort. However, well built cost models in the hands of a

professional can be better cost predictors than detail methods because certain judgement and

other biases are more controlled.

General Features

A CER can be a functional relationship between one variable and another and may

represent a statistical relationship between some well defined program element and some specific

cost. Since most company overhead rates are percentages of direct labor expense, these are

CERs. Computer and travel costs often show statistical relationships to engineering costs, design

is frequently closely correlated to drafting costs and these, in turn, to the number of drawings,

parts, size, weight, etc. Many costs can be related to other costs or non-cost variables in some

fashion but not all such relationships can be turned into a CER.

A CER must have two characteristics. First, it should link in some rational way the

underlying variable - the independent variable, and the cost being developed. Second, the CER

should have a strong statistical fit, R2, and confidence interval, with the basis element.

Evaluating an Estimate

CERs are used in lieu of a direct substantive link between a cost element and some basis

of estimate (BOE). Since CERs are developed from collections of cases and represent average

values, CERs have uncertainty built into them. (A direct BOE to cost estimate extrapolation is

preferred if the cost element is significant, if a good BOE can be found, and if some well defined

extrapolation can be postulated between the BOE and the cost element.) The first step in an

analysis of a CER influenced estimate is to identify how much of the total estimate is CER

driven.


67

The second step in analyzing a proposal using CERs is to evaluate the CERs themselves.

Since CERs supposedly relate an element of cost to some variable or factor, the analyst must

determine whether the relationship is truly functional and statistical. If the CER is a factor

implied by a functional relationship, the analyst needs an explanation of the function and support

for the assertion of a relationship. Both deterministic and statistical support are required. In

other words, does the relationship make logical sense and is the pattern of influence regular

enough to be helpful? Base data must be available for examination, preferably original,

“unsmoothed” data. Again, the purpose of a CER is to save time and effort. If the amount of the

proposal affected by CERs is not great, the evaluation effort applied to the CER should be an

appropriate amount.

In a “worse case” situation, the analyst may have to back track to the original data set

used to develop a CER. In that case the analyst should attempt to see if all the relevant cases have

been included and no “cherry picking” has occurred. In other words, what “risk” is involved by

using the CER?

Assuming the original data set is available and complete, the developer of the CER must

explain the theory of the relationship and the data processing performed. If “outliers” were

excluded, the estimator needs to explain why. If the explanation of the exclusion is

unsatisfactory, the analyst may want to develop a set of CER’s with the outliers included.

Ordinarily, outliers affect the deviation of the estimate rather than the value of the CER, but it is

useful to check. If several data points have been excluded and if these influence the CER mean

and standard deviation significantly, the CER may not be operationally useful even if theoreticaly

valid. To illustrate, suppose a relationship is identified between variable K and cost variable C.

Suppose the CER is 7 x K = C. If the value 7 is developed from an arithmetic average of a dozen

values but three “outliers” have been excluded, then inclusion of the outliers may spread out the

sample standard deviation to the point that confidence in the relationship may become suspect.

The CER estimator should be able to supply the original data set and his/her analysis. The cost

analyst may need to replicate the estimate to verify the calculation. If this is done, the CER

statistics should be examined for:

1. The sample size. Confidence in the estimate will increase as the sample size

increases.


68

2. The standard error of the mean (or the point estimate) should be shown along with

the standard deviation of the calculated mean.

3. The standard deviation of the sample set. What is the range of the majority of the

data points? Confidence in the estimate increases as more and more data points

fall within a specified range.

4. If the CER is developed from a correlation calculation, the cost analyst can

examine the coefficient of correlation. Correlation infers a link between the two

factors but the relationship may be accidental. Standard statistical tests exist for

checking the likelihood that a given correlation coefficient is accidental and

should be used if the sample set is small, or if “R” is less than .8 (R2 = .64).

The last step in evaluation of a CER is calculating the effect which reasonable variations

on the CER value can have on the estimate. If reasonable variations on the CER impact the

estimate greatly, the analyst has to be skeptical of the explanatory power of the CER. The effect

of this is to widen the actual range of an estimate.

Summary Of CER Audit And Analysis

CER analysis requires addressing the questions:

1. What is the proportion of the estimate directly affected by CERs?

2. How much precision is appropriate to the estimate in total and to the part affected

by the CERs?

3. Is there a rational relationship between the individual CER affected variables and

the underlying variables?

4. Is the pattern of relationship functional or purely statistical?

5. If functional, what is the functional relationship? And why?

6. If statistical, is the history of the relationship extensive enough to give a

confidence that it operates in the given case?

7. Is the pattern of relationship statistically significant? And at what level of

confidence?

8. What is the impact on the estimate of using reasonable variations of the CERs?


69

If the CERs represent a cost-effective response to an estimating problem and if they are

rationally developed and solidly based, the CERs are valid and accurate tools for an estimate.

Assuming no “show stopper” problems are uncovered, the analyst can accept the concept of the

CERs and apply such margins of variance as seem reasonable.

This chapter has presented the concept of Cost Estimating Relationships (CERs) and the

statistical underpinnings of CER development and application. The basic mathematical

relationships were described and examples showing the use of CERs were also presented. A flow

chart of the CER development process is shown in Figure III-10.


70

TO COST MODELS

FIGURE III-10. Typical CER Development Process

CER Database

Approval

Validation

Select CERs

C = aX + by

When Necessary

Revalidation

Selection of Variables

Weight # of Dwg Thrust Materials Range MIPs Impulse SLOC

Data Evaluation and Normalization

Unit Cost/Quantity Constant Year $ Escalation

Data Collection

Company Databank Library Contractors DoD/NASA

Regression and Curvefit

C = aX

C = aXb

C = aX + b

Test Relationships

Est

Actual

Data Analysis and Correlation

Correlation Matrix Data Plot Data Subsets Dimensional Analysis


71

CHAPTER IV

HARDWARE COST MODELING


72

CHAPTER IV

HARDWARE COST MODELING

INTRODUCTION

Hardware cost is defined as all program cost, excluding software development, for all

components, assemblies, subsystems, and systems for sea, ground, airborne or space applications.

Hardware cost includes: mechanical, electromechanical, electrical/electronic, structural or

pneumatic equipment, and may also include system test, or system operations. It is defined as the

cost to develop, design, build, test, launch, and operate a system.

Hardware cost estimating approaches are categorized into five basic methods: (a)

parametric, (b) analogy, (c) grassroots, (d) other (methods such as, standard time or expert

consensus approaches), and (e) combinations of the other form. The first three are well recognized

and clearly defined. The last two "methods" cover everything else. This section will focus on

parametric hardware cost modeling techniques.

Parametric estimating is the mathematical procedure or process in which product or service

descriptors directly yield consistent cost information. A parametric cost model is an estimating

system comprised of Cost Estimating Relationships (CERs) and other parametric estimating

functions, e.g., cost quantity relationships, inflation factors, staff skills, schedules, etc. Parametric

cost models yield product or service costs at designated Cost or Work Breakdown Structure

(CBS/WBS) levels and may provide departmentalized breakdowns of generic cost elements. A

parametric cost estimate provides a logical and repeatable relationship between input variables and

the resultant costs.

Hardware parametric cost estimating techniques are widely utilized by industry and

government. The use of these techniques expedites the development of price proposals by the

contractor, evaluation by the government, and subsequent negotiation.

The simplest form of parametric estimating tool is a CER. In this form a relationship

between cost and non-cost variables is investigated to determine any possible cause and effect

linkages. A statistical analysis is performed on representative samples of a historical database to


73

validate these linkages. The analysis is usually based on the investigation of the relationship

between two or more variables. The known variable (physical parameter, schedule, etc.) is called

the independent variable; the variable we are trying to predict, the cost of the item, is the dependent

variable. The resultant CER is an expression of the mathematical relationship between parameters

characterizing the item to be estimated and its cost. The strength of CER's lie in their relative

simplicity and the convenience they afford in predicting costs.

The primary limitations of CER's is that they will yield reliable cost estimates only within

the limits of the spread of independent variables that were used when the CER was developed. In

high technology areas, the limits of input parameters (independent variables) might go outside the

boundaries of the historical database. Therefore, either the CER's must be re-examined and updated

as the historical database increases, or the use of the CER must be limited, or used appropriately by

properly trained professionals.

A more sophisticated form of parametric estimating is the computerized Parametric

Estimating Model. These models estimate costs on a broader scale and are more versatile and

much less restrictive than the acceptable boundaries of CERs.

With parametric estimating models, system acquisition costs can be derived that are based

upon parameters such as quantity, size, weight, power consumption, environmental specification,

complexity and type of electronic packaging, etc. Some models can also provide schedule

information, or evaluate specified schedules in terms of costs. Parametric estimating models can

provide early cost measurement of system concepts with a limited description of physical or other

parameters. They are also frequently utilized for independent assessment of conventionally

prepared cost estimates.

Numerous "home grown" parametric cost models exist throughout industry and government

to cover a specific range or type of products or systems. There are also universal parametric

estimating systems that generate, internal to the model, appropriate expressions of CER's for a

broad range of products or systems. These models perform a mathematical extrapolation of past

experience in order to predict cost.

Computerized parametric cost estimating models often contain many mathematical

equations relating the input variables to cost. Each set of input parameters uniquely defines the

item of interest for cost modeling.


74

Universal parametric (i.e, generic) cost estimating models usually require calibration.

Considering the variety of products, the differences in accounting methods, etc., the calibration

process "fine tunes" these models to emulate the producer's organizational and product

characteristics. The calibration process simulates past product history and organizational traits.

Parametric estimating systems (models) exist to rapidly compute hardware development

and design costs, manufacturing costs of prototypes and production units along with all the

manufacturing support costs. Models are also available for computing operation and support costs

of hardware in the field. Also, models used for the development and maintenance costs of

computer software are available.

A model is a representation of a real-life system. In the case of cost estimating, a parametric

model is a system of data, equations, and inferences presented in mathematical form to represent or

describe the cost of an item. It relates knowns (system descriptions or parameters) to unknowns

(cost of systems). A model can be as simple as a single equation (CER), or as complex as an

inter-relation of many equations and functions. It can also be designed to estimate items as small as

modules or components, or as large as subsystems, systems, total programs, or large development

activities.

OVERVIEW OF HARDWARE COST MODELING

A Hardware Cost Model provides estimates of system acquisition costs based upon

quantitative parameters such as complexity, quantity, weight, and size; and qualitative parameters

such as environmental specification, type of packaging, and level of integration; and schedule

parameters such as months to first prototype, manufacturing rate, and amount of new design.

Hardware cost parametrics bring speed, accuracy, and flexibility to the cost estimating process.

Early cost measurement of concepts is crucial to a new venture since there is little opportunity to

change program costs significantly once a design has been detailed. Parametric cost models have

been developed to operate with limited concept description so that many alternatives can be costed

before designs and bills of material are finalized. Parametrics can also be used extensively as the

basis of a cost estimate in preparation of proposals as well as for the independent assessment of

conventionally prepared cost estimates.


75

Hardware cost models may be formulated as a universal system to generate appropriate

regressions or CERs for a range of products or systems. In essence, they permit extrapolation of

past experience to predict costs. Inputs to hardware parametrics cover a wide range of systems.

Since all assemblies must have weight and size, these are often used by the models as principal

descriptors. Electronic hardware items are characterized by the electronic application and type of

componentry. Mechanical and structural elements can be described in terms of their construction,

type of material, functionality, machinability, and manufacturing process.

Some uses of Parametric Hardware cost models are:

* Proposal preparation and submittal

* Estimates of cost to complete

* Estimates of modifications

* Should cost analysis

* Most Probable Cost estimates

* Evaluations of bids and proposals ("Sanity" Checks)

* Vendor negotiations

* Life cycle cost estimates

* Basis of estimates

* Independent cost estimates

* "What ifs"

* Design cost trade-off analysis

* Bid/No Bid decisions

* Estimates of spare parts costs

* Design to Cost

* Rough Order of Magnitude (ROM) cost estimates of new or advanced hardware

systems

Parametrics are applicable to all aspects of hardware acquisition - development, production,

purchased and furnished hardware, hardware modifications, subcontractor liaison, hardware/

software integration, multiple lot production, and hardware integration and test. A normal

modeling capability should exist at all levels of the Work Breakdown Structure (WBS). Estimates


76

of integration and test costs which are attributed to each level of integration should also be

estimated.

When a parametric model calculates a cost for manufacturing, it does not use a parts list and

labor resource build up, but rather a parametric representation of the parts and labor costs.

Fundamental Input Parameters To Typical Parametric Models Are Listed Below:

* Functional design parameters.

* Quantities of equipment to be developed, produced, modified, purchased furnished and

integrated and tested.

* Applications (technology of materials and processes) of structural and electronic

portions of the hardware.

* Hardware geometry consisting of size, weight of electronic and structural elements, and

electronic packaging density.

* Amount of new design required and complexity of the development engineering task.

* Operational environment and specification requirements of the hardware.

* Schedules for development, production, procurement, modification, and integration and

testing.

* Fabrication process to be used for production.

* Yield considerations for hardware development.

* Pertinent escalation rates and mark-ups for General and Administration charges, profit,

cost of money, and purchased item handling.

The fundamental characteristic of parametric inputs is inter-relationship. A change in any

one parameter is usually not localized to one cost element, but rather, may have a ripple effect on

several cost elements. Consider the impact of a change in quantity. Certainly this change would

cause a change in the manufacturing cost. It may also effect the fabrication process, and, hence, the

cost of tooling and test equipment. In addition, a change in quantity could impact the production

schedule, and thus, impact the costs associated with escalation. An impact on integration and test,

sustaining engineering, and project management would almost certainly result from a change in

quantity. This dynamic effect is characteristic of most input variables, and most parametric models.


77

The model’s input parameters uniquely define the hardware for cost estimating and

modeling. The resultant cost output is determined from the model's mathematical equations alone.

Cost may be estimated with a minimal amount of hardware information. This feature

makes models a legitimate tool for cost estimation of programs in the conceptual stage of

development. Of course, it is always preferable to provide as much information to the models as

possible. In this way, statistical uncertainty associated with the input variables is reduced.

The modeling activity is basically a conversation between the model and the cost analyst.

Parametric data representing the hardware to be costed is formulated, and the analyst calls upon

embedded text, tables, and utilities of the model to help create and store the data used to drive the

model. In the course of using a parametric model, the analyst controls the output formats, the

ability to perform sensitivity and risk analysis, and of course the data that are used to estimate the

cost of the hardware. The data elements created during a session usually represent systems or

sub-systems composed of many separate sub-assemblies. For example, an aircraft might be

represented as a system composed of an airframe, propulsion units, avionics, air to ground controls,

flight controls, and ordinance. In turn, the flight controls might be a sub-system composed of

instrumentation, radar, and various processors. At an even lower level, we might find a digital

processor composed of input/output circuits, data storage memory, program memory, logic circuits,

power supply, and chassis and housing. The number of hardware elements and relative detail of the

parametric information in each is determined by the analyst. There is no limit to the level of

detailed data used for an analysis as long as the cost model contains historically verifiable

parametric cost relationships at the desired level. Nor is there a requirement that precludes a

parametric user from treating the entire aircraft as a basic assembly of the lowest order. Thus, an

analysis might be accomplished with one data set representing the aircraft as an assembly, or it

might be attained with many elements representing the aircraft as a system of sub-systems,

assemblies, and sub-assemblies. The amount of detail provided to the model is purely at the

descretion of the analyst.

Parametrics can estimate costs for the development and production phases of the acquisition

process. Outputs are categorized by such cost elements as: drafting, design, project management,

system engineering, data, manufacturing, and tooling and test equipment, etc.


78

Some parametric models contain the effects of phased interactions between engineering and

manufacturing. In addition to considering a normal performance period, estimates of the cost

impacts due to accelerated or protracted engineering schedules or due to an operation plan that

requires stops and restarts of production effort can also be computed.

A comprehensive parametric model should allow an analyst to:

* Calibrate (Automated is best)

* Estimate the cost of multiple lot production

* Calculate manufacturing costs of non-homogenous assemblies

* Determine the cost impact of compressing or extending development or production

schedules.

* Estimate the cost impact of the development schedule (concurrency or lapse) on

production.

* Perform Cost Risk Analysis

A graphic depiction of hardware modeling is shown in Figure IV-1.

HARDWARE MODELING

Hardware model applies common parameters in the comparative evaluation of new requirements in light of analogous histories.

FIGURE IV - 1

Hardware modeling normally entails stringing together numerous CERs using an

appropriate mathematical logic. The math logic is derived during the calibration process, and

allows the math model to emulate the "real world" of program history or to other supportable data

or estimates. When the "real world" model is developed, then the new requirements can be

Input Parameters

• Magnitude (quantity)

• Operating Environment (MIL-SPEC, Airborne, etc.)

• Amount of new design & design reuse

• Engineering complexity

• Manufacturing complexity

• Schedulde

• HW & SW integration

• Weight/Volume

Output Parameters

• Cost

• Development

• Production

• Engineering

• Manufacturing

• Schedule risks

• Unit/System integration cost

⇒⇒⇒⇒ ⇒⇒⇒⇒


79

evaluated based on the calibrated model. Modeling offers significant advantages to the estimator in

that once CER development and the calibration process is complete, cost estimating is quick,

repeatable, and cost effective.

For instance, the first unit production missile cost (T1) may be modeled as

T1 = [(Structure CER) + (Propulsion CER) + (G&A CER)]*1.15 (integration factor)

Then,

Total Missile H/W Cost = T1 * Qb, where,

Q = quantity, and b is a cost improvement exponent.

Also, for systems engineering and program management we may get

S/E + P/M = .10*(Missile H/W cost)

Models can also be used to estimate non-recurring development costs. These costs would

be estimated using parameters such as the number of prototypes, number of tests, the amount of

new design and design repeat, design complexity factors, and others.

A simplified parametric cost estimating process is shown in Figure IV-2. The process

connects the parametric model to a post processor that in turn converts the parametric model

outputs into typical cost proposal reports. Ultimately, the customer needs proposal reports in

standardized formats so that competing contractors can be conveniently compared.

Just as the value of a house, when it is evaluated for a mortgage or a refinancing, is

calculated using parametric approaches (square feet, # rooms, zip code, # bathrooms, etc.), the

complex products we normally deal with can be evaluated in exactly the same manner. Parametric

modeling can greatly simplify the cost estimating and cost analysis processes.


80

PARAMETRIC COST ESTIMATING CONCEPT

PROGRAM HISTORY PROGRAM HISTORY Cost Functional Distribution Technical

New Program Baseline Forward Pricing Rates

WBS Lines of Code Make/Buy Weight Power Material Schedules Vendor Quotes

FIGURE IV-2

COST MODELS

COST ESTIMATING

RELATIONSHIPS

POST PROCESSOR

COST AND MANPOWER

REPORTS

BY WBS

BY LABOR TYPE TIME SPREADS

MAKE/BUY

COST BY WBS

MANPOWER BY WBS

$ BY HRS BY TIME

SUBCONTRACT SUMMARY

MANPOWER BY FUNCTION

OTHER REPORTS

Programmatic

s


81

MICRO-CIRCUIT AND ELECTRONIC MODULE MODELING

Parametric cost models exist that are designed to provide quick and reliable estimates for

development and production costs of individual custom micro-circuit chips and/or electronic

modules. These models provide the capability to estimate costs and schedules associated with the

development and production of modular electronics, custom Application Specific Integrated

Circuits (ASICs), hybrid or MMIC packages, common components, and system modules. In view

of the significant cumulative expenses for custom chips, these models are invaluable for

procurement or product planning. Models can assist analysts in future product planning and

incorporate state of the art technology in the cost estimating process.

Using a WBS structure, analysts can use models to form cost estimates for individual

micro-circuits (chips), the integration of these chips, and additional electronic components on

electronic modules. If the situation exists, these models may be integrated onto electronic modules

with additional components. The final electronic module may then be integrated into the WBS

used by the hardware models.

Micro-circuit model input parameters are determined by physically describing the

micro-circuit, other electronic components, and the electronic module. A wide range of custom

chips can be processed through use of generic input parameters which include:

* Number of pins

* Number of transistors and/or gates

* Die size

* Type of packaging

* CAD sophistication

* Amount of new design

* Manufacturing yield

* Development/Production schedules

Inputs for electronic modules include:

* Module size

* Number of sides used for component mounting

* Technology

* Function (Memory, Analog, etc.)

* Quantity produced and/or developed


82

* Packaging or interconnect method

* Board type

* Development/Production schedules

* Quantity and types of components

Components as defined for use within these models may be custom chips as well as

off-the-shelf items. The amount of detail about the components may be very limited or as detailed

as available information will permit. A component database file may be used to catalog standard

components that are used on several different electronic modules. This file may contain

information about the standard components including the cost, if known. Whenever a component

from a database file populates a module, it can be referred to by name with only the number of

components per module required as additional information. The analyst should be allowed to

provide as much cost information as is available. This information may contain component cost,

board cost, and/or assembly and test costs. Any costs not provided will then be estimated by the

model.

Output formats will depend on the item modeled. For a custom micro-circuit, costs might

be estimated for:

Development engineering costs such as:

* Chip specification,

* Chip design,

* Systems engineering,

* Project management,

* Data,

* Prototype manufacturing

For electronic modules, some of the costs that are estimated are:

* Development engineering:

* Drafting

* Design

* Systems engineering

* Project management

* Data


83

* Prototype manufacturing

* Tooling/Test Equipment

Production costs would be:

* Drafting

* Design

* Project management

* Data

* Manufacturing

* Tooling/Test Equipment.

As with the hardware models, micro-circuit models have the capability to be "fine tuned" to,

directly reflect a particular company and/or product through the process of calibration. In this

sequence of operation, the model uses actual cost data as inputs and provides development and

manufacturing indices directly related to the historical costs and specifications. These calibrated

indices can then be used as the basis for estimates of proposed custom micro-circuits and/or

modules.

HARDWARE OPERATIONS AND SUPPORT (O&S) OR LIFE CYCLE MODELS

Life cycle cost is defined as the cost associated with all the phases of a program: design,

development, prototype, production, and maintenance and operations (M&O or O&S). Models are

available to estimate life cycle costs of a variety of hardware programs. With appropriate

information, the life cycle costs of hardware systems can be evaluated. Through integration with

the WBS structure, cost factors for different equipments are calculated in order to estimate life

cycle costs. Moreover, the models provide for tailoring our analyses to fit a variety of maintenance

concepts and supply systems in order to customize specific programs and user organizations. After

the inputs of a set of descriptors, the models produce categories of life cycle cost in each of three

phases: development, production, and support.

In a typical application, analysts can instruct the model to select the most cost effective of

numerous built-in maintenance concepts. Operating features allow the analyst to specify a subset of


84

the maintenance concepts for model evaluation. In addition, the analyst can tailor a variety of

maintenance concepts by mixing the preestablished concepts. Some of these concepts could be:

* Discard CRU at Failure.

* Replace parts at depot.

* Replace parts at manufacturer location.

* Replace mods at equipment level. Scrap bad mods.

The strength of life cycle models are derived from three factors: 1) use of hardware

models to rapidly generate all hardware related input parameters -- particularly useful in the

decision making stages of early concept development before hardware design details are known;

2) availability of preset maintenance concepts - among which the models can be instructed to

determine the most cost effective approach; and, 3) ability to rapidly tailor program parameters to

reflect specific support conditions. The interactive feature of parametrics permit immediate

processing of sensitivity analyses to determine the effect of uncertainties of input data.

Some of the sensitivity analyses allow the determination of the effects of changes in:

* Number of equipment level maintenance and/or supply locations.

* Number of organization level maintenance and/or supply locations.

* Number of intermediate level maintenance and/or supply locations.

* Number of depot level maintenance and/or supply locations.

* Duration of the support period.

* Cost, size, and weight of units, modules, and parts.

* Cost of contractor repair.

* Test equipment costs and loading factor.

* Shipping costs.

* Crew size and labor rates.

* Dedicated vs. non-dedicated crews.

* Attrition.

Determination of the effects of the following may be addressed using risk analysis

features built into some models:

* Equipment operating time.

* Unit Mean Time Between Failure (MTBF).


85

* Unit and Module Mean Time To Repair (MTTR).

* Authorized stock levels at all supply levels.

* Safety stock coefficients.

* Scrap and repair fractions.

Another feature is the capability to model different theaters of deployment and multi year

specification of equipment deployment and employment (called gradual force structuring). This

capability permits more accurate modeling of remote depots sending work back to a central depot

or the manufacturing facility (contractor facility). Force structure data permits variation in force

levels for each year, as well as the planned levels of equipment operation for each year.

There are three principal categories of data required to estimate costs with Life Cycle

models: deployment and employment; hardware parameters; and program globals.

Deployment And Employment

This category of inputs pertain to the number of locations at which equipments are installed,

the number of locations at which they can be maintained, and the number of locations authorized to

stock spares; how often the equipments are operating; and the length of time of the support period.

The analyst defines the size of the program in terms of the number of locations at which equipments

are installed; the number of organization maintenance locations (for maintenance concepts that

involve organization maintenance); the number of Intermediate maintenance locations (for

concepts that use Intermediate maintenance); and the number of depot maintenance locations

(Government or contractor depots). With other inputs, the analyst can specify whether these

locations are maintenance only, maintenance and supply, or supply only. To further define the size

of the program, the analyst specifies the duration of the support period in years, and the "on-line" of

the installed equipments either as hours per month or as a fraction of real time.

Hardware Parameters

Hardware dependent parameters should be generated by the hardware parametric models for

each element modeled in the WBS structure, and then input into the life cycle model.

Program Globals


86

This is the largest group of inputs, and they are used to describe the maintenance and supply

system to the model. All have preset values that may be changed by the analyst. Unique sets may

be created for use in subsequent applications of the model.

Some categories of Program Globals are:

Spread Globals: Used to control distribution of resources and/or expenditures across

schedules.

Supply Administration: To uniquely define the administration as U.S. Air Force or Navy.

Discretionary Supply Times: Frequency of unit, module, and part ordering.

Input Data Multipliers: Coefficients to permit uniform variation of hardware input

parameters for cost effect measurement.

Miscellaneous: Includes controls for stock administration, programmable test equipment

software, test equipment maintenance, attrition, average number of parts consumed per

repair, false failure detection scheduled maintenance, and others.

Provisioning and Labor: Stock controls for meeting expected demand of all units, modules,

and parts at each pertinent supply location, number of hours in a work week, and labor costs

at each maintenance location.

By Theater: Theater specific parameters dealing with pipeline times at each

maintenance/supply location, fractions of repairs resulting in scrapping, and supply stocking

(inventory) cost factors.

COMMERCIAL MODELS

There are several commercially available hardware cost estimating models. Three typical

models (FASTE, PRICE, SEER) that estimate one or more of the hardware types are discussed

below. Although there are many others (See Appendix E) this chapter will only discribe these three

as representative. The models will allow an analyst to:

* Model an entire project through Engineering Development, Production, Integration,

Installation and Life Cycle Phases.

* Calibrate and Perform Forward Costing.

* Project Input Files and Output Files.


87

* An Economics File.

* Input variables to the model.

* Get needed information through the on-line Help system.

MicroFASTE

The MicroFASTE model is used to help the analyst to develop a parametric model to be

used to estimate the costs associated with implementing a project.

The goal of a particular project may be the production and installation of a Hardware

system, a Software system (or a combination of both). It may be the construction of nuclear power

stations, radar systems or manned space stations.

Even performing cost estimates for the implementation of a financial funding program, the

construction and operation of an underground coal or uranium mine, or the design and production

of a highly complex module containing very advanced micro-circuitry technology are possible

through the techniques of parametric systems analysis.

However, the MicroFASTE model (where the "E" on the end of "MicroFASTE" means

Equipment), is exclusively for use in performing parametric analyses for Hardware systems.

Hardware development projects involve several major phases of implementation. These

phases and activities have much in common whether the project involves production and

integration of a small hardware device, or a large, complex multi-assembly hardware system. It is

these phases in which costing studies are performed. MicroFASTE classifies these common

implementation phases into the following categories and sub-categories:

Equipment Acquisition Phases and Life Cycle (O&S)

1. Engineering

Design/Drafting

Involves the detail design engineering and drafting effort that implements the governing

specification.

Systems Engineering


88

Establishes the equipment's design, performance and test specifications, predicated on the

controlling requirements.

Documentation

The recording of engineering activities, preparation of equipment manuals and required

management reports.

Prototype and Testing

Covers all charges connected with the manufacture and testing of engineering prototypes,

and includes all brass and bread board models.

Special Engineering Tooling

Embodies the special tooling charges affiliated with the prototype efforts. It does not

include capital or amortized equipment that may be related to the tooling. When there are

no prototypes, there will be no tooling charges.

Project Management

Takes in the overall management of all areas connected with the engineering efforts such

as planning, budgeting, operations and financial controls.

2. Production

Manufacturing

Involves the direct production charges. This is the same cost value as calculated when

Total Production is specified without the detail breakdowns.

Engineering Support

Embodies the engineering effort that is realted to the manufacturing activity such as

material design reviews, corrections of defects, etc.

Documentation

The recording of production events as well as changes to equipment manuals as

necessitated by design modifications caused by production problems.


89

Production Tooling

Covers special required tooling. It does not include the cost of capital equipment, or tools

that are amortized in overhead accounts.

Project Management

Takes in the management of all areas associated with production such as planning,

budgeting, operations and financial control.

3. Life Cycle (O&S)

Life Cycle costs are the charges involved in the maintenance and support of the

deployed equipments. The Life Cycle period runs from the first day of the Starting Year

until the last day of the Finish Year specified in the Life Cycle procedure.

The maintenance concept varies in relation to the equipment's technology and size.

The deployed quantities, along with the indicated Mean-Time-Between-Failure and

On-Time-Fraction may significantly affect the equipment availability.

Although Life Cycle cost studies are performed in association with an Item's

acquisition studies, Life Cycle costs may not be accumulated into the project's total costing

structure. They must be kept track of separately; the reason for this is that the time frames

over which Life Cycle costs would be realized always span a much greater period of time

than other facets of the project.

System Integration

Integration is a study, as an independent analysis that covers the costs of consolidating

various Items into a higher order of assembly. In an Item's Acquisition study, there are variables

used to indicate the character of the Item's integration requirements. When an integration study has

been completed, the calculated integration costs may be accumulated into the project's total costs.

Equipment Installation

Equipment Installation studies are applicable to large energy producing systems such as

steam or nuclear power plants, and large hardware systems such as those installed in chemical

manufacturing plants. When a significant percentage of the project's total cost will be involved in

putting the equipment in place and making necessary mechanical and electrical connections to

perform its intended task, that is when it becomes necessary to perform Installation cost analyses.


90

Installation studies are not generally applicable to relatively small hardware systems, even

though the hardware system may indeed have to undergo an actual installation procedure to have it

perform its intended function. The costs associated with these activities are accounted for during

the System Integration phase of cost analysis.

Installation is a study that covers the costs of placing this type of equipment at its work site.

Installation studies are performed on an Item-by-Item basis to determine each pertinent Item's

installation costs. Installation costs, therefore, are not computed on a total system or project basis.

However, after completing an Item's Installation study, the calculated costs may be accumulated

into the project's total costs.

Normally, the Installation study should follow an Item's Acquisition study. Installation

efforts may include production and/or engineering phases, and their costs may be processed in

either a total or a detail manner.

Calibration

Typically, when a costing study is performed on some new unit of equipment, there are

various factors which may not be known about the item, such as how much one unit will weigh

after it is produced. The model provides the capability of performing a type of study called a

Calibration, which is a costing calculation in reverse, used to generate reference factors. When

weight and its complexity factor, traditional key factors in parametric cost calculations, are absent,

the model will generate a calculated or "fiducial" weight along with a corresponding weight factor

if input costs are given.

In performing calibrations for an Item, the analyst supplies as much information in

parametric form as is known about the item, and enter the previously known costs of a reasonable,

similar item. Then, after the model performs its calculations for the set of calibration data, a

calculated weight as well as other calculated reference factor values, including the complexity

factor, will be displayed. Those calculated values can be retained, and used as input values when

deriving the estimated costs of any new unit of equipment.

It may be desirable to perform an iterative series of Calibration and Forward Costing

exercises for a particular unit of equipment until reference factor values are derived that seem most

reasonable for a given project.

The processes of Calibration and Forward Costing can be applied to any or all of program

phases: Acquisition, Life Cycle, System Integration and Installation.


91

Price Parametric Models

PRICE models use the parametric approach to cost estimating. Parametric cost modeling is

based on cost estimating relationships (CERs) that make use of product characteristics (such as

hardware weight and complexity) to estimate costs and schedules.

The CERs in the PRICE models have been determined by statistically analyzing thousands

of completed projects where the product characteristics and project costs and schedules are known.

These projects encompass a wide range of products and companies. The PRICE models contain

many hundreds of CERs that work together to compute the costs and schedules for the development

and production of hardware and software products.

An example of parametric modeling is the technique used for estimating the cost of a house.

The actual cost of building a house is, of course, the total cost of the materials and labor. However,

defining the required materials and labor for developing this cost estimate are time consuming and

expensive. So, a parametric model that considers the characteristics of the house is used to estimate

the cost quickly. The characteristics are defined quantitatively (floor area, number of rooms, etc.)

and qualitatively (style of architecture, location, etc.).

This parametric technique for estimating the cost of building a house is widely used in the

insurance business and for taxation assessment.

Just as with the building example, determining and costing the material and labor

requirements for hardware and software projects are also time-consuming and labor-intensive, and

since the product characteristics can be defined long before the material and labor requirements,

parametric models permit estimates to be obtained earlier than with other techniques. Early

management and engineering decisions can be based on these "up front" estimates. Many business

decisions must be made before there is sufficient product definition to develop a conventional

estimate. Using parametrics makes it practical to explore the cost risks associated with

uncertainties in the project, and to get answers to "what if" questions such as: "What if we shorten

the development cycle?" "What if we build fewer prototypes?"

Predevelopment Cost and Schedule Estimates -- Cost and schedule estimates can be

obtained to develop and manufacture a product that is still only a concept. The PRICE models

accomplish this by using product characteristics to develop the estimate. PRICE does not require

labor hours and a bill of material.


92

This early-on estimating capability makes PRICE a tool for design engineers, since much of

a product's cost is locked in by early engineering decisions. PRICE can provide engineers with cost

information needed to develop minimum-cost designs.

Descriptions Of The PRICE Models

PRICE H

PRICE H is used to estimate the cost of developing and producing hardware. Most

manufactured items and assemblies can be estimated using PRICE H. PRICE H uses product

characteristics to develop the cost estimate. This makes the model a good tool to use at the product

concept stage, when there is insufficient definition to quantify the product labor and material

required for a conventional estimate. Key inputs to the PRICE H Model are:

* Weight -- tells the model the size of the product being estimated.

* Manufacturing Complexity -- a coded value that characterizes product and process

technologies and (optionally) the past performance of the organization.

* Platform -- a coded value that characterizes the quality, specification level, and reliability

requirements of the product application.

* Quantities -- the number of prototypes and production items to be estimated.

* Schedule -- the dates for the start and completion of the development and production phases

may be specified. The model will compute any dates that are not specified. Only the date for

the start of development is required.

* Development Costs -- effort associated with drafting, design engineering, systems engineering,

project management, data, prototype manufacturing, prototype tooling, and test equipment.

* Production Costs -- effort associated with drafting, design engineering, project management,

data, production tooling, manufacturing, and test equipment.

* All costs are reported at the material, labor, overhead, and dollar level.

It is generally acknowledged that most of the cost of a product is locked in by the early

engineering design decisions. This means that the early decision process should consider cost as

well as technical alternatives. The lack of product definition early in the concept stage precludes a

credible conventional cost estimate. With PRICE H, however, engineers and managers are able to

develop cost estimates for each alternative to select minimum cost-product designs. Before any


93

investment has been made in design engineering, PRICE H can compute the unit production cost of

a product.

PRICE H can be quickly iterated to obtain estimates under varying assumptions until the

value selected for the estimate represents acceptable risk. With PRICE H, cost sensitivity to project

uncertainties can be explored, and management's "what if" questions can be answered.

PRICE HL

The PRICE HL (Hardware Life Cycle) Model calculates costs, cost effectiveness, and

operational availability at the system, subsystem, assembly, and sub-assembly levels. It is used in

all phases of the acquisition process and is especially well suited when linked with PRICE H for

high-level analyses in the concept and system design phases. This coupling provides a system level

trade-off capability to effectively determine hardware life cycle costs during early stages of

hardware designs. PRICE HL provides cost outputs for all phases of the hardware life cycle.

PRICE M

The PRICE M (Electronic Module and Microcircuit) Model produces estimates of

development and production costs for electronic modules and application specific integrated

circuits (ASICs). It uses cost estimating relationships based on the number of transistors and/or

gates, percentage of new circuit cells and design repeat, specification level, degree of

computer-aided design, product familiarity, engineering experience, and calibration factors to

estimate ASIC specification and design costs. Additional relationships provide systems

engineering, project management, and data costs. PRICE M contains a database for frequently used

components that can contain predefined costs or that can be calculated from the component input

specification.

SEER

The SEER family of models include the basic hardware cost estimation model (SEER-H)

and a hardware life cycle cost model. The hardware cost model estimates hardware cost and

schedules and includes a tool for risk analysis. The hardware model is sensitive to differences in

hardware technologies ASIC, MCMS, exotic materials, miniaturization, etc., and to different

acquisition scenarios; make, modify, customer-furnished, purchased, off-the-shelf, etc. It is also


94

sensitive to differences in electronic versus mechanical parameters and makes estimates based on

each hardware item's unique design characteristics.

The SEER hardware life cycle model (SEER-HLC) evaluates the life cycle cost effects due

to variations in: reliability, mean time to repair and repair turnaround times. SEER-HLC

complements SEER-H, and both models will run on a personal computer. The models are based on

actual data, utilize user-friendly graphical interfaces, and possess built-in knowledge bases and

databases that allow for estimates from minimal inputs.

SEER-IC is an expert system for planning integrated circuit development and production.

SEER-IC provides a systematic, expert approach to cost and resource estimation and a capability

for Design to Cost (DTC) tradeoffs and risk analysis. SEER-IC includes an industry-wide

knowledge base.

SEER-DFM (Design for Manufacturability) is a tool designed to assist the engineer produce

and assemble products with efficiency, in a manner designed to exploit the best practices of an

organization. Diverse team members have an idea of what their true interests are and of how they

can work together and input to the model accordingly. The fundamental goal of DFM is to make

engineering decisions that optimize the manufacturing process.

Two fundamental analysis steps are taken in a DFM regime: the gross and detailed tradeoff

analysis.

* Gross analysis involves product design decisions, and also fundamental process and tooling

decisions. Factors that influence gross analysis include the quantity of the planned product,

the rate at which it will need to be produced, and the investment required. There are also

machinery, assembly and setup costs to contend with.

* Detailed analysis takes place once many of the primary production parameters, such as

design and basic processes, have been fixed. Factors that can be adjusted for and balanced

at the detailed level include tolerances, the proportion of surface finishes, secondary

machining options (drilling, reaming, polishing), and the age and degree of process

intervention (how soon is a machine recalibrated or rebalanced?).

SEER-DFM explores the DFM effort by developing engineering tradeoffs that are involved

in a DFM analysis and presents the results of each as costs. SEER-DFM models the categories of

materials and processes used in modern production lines. SEER-DFM integrates the following

models:


95

* machining (turning, boring milling, shaping, chemical-milling, grinding... ): The model

explores the tradeoffs from starting with raw stock vs. sand or investment-casting, etc.

The material may also be varied. Tooling, setup and other costs hinge on these choices.

* sheet metal fabrication (presses, shears, die press).

* mechanical assembly (spot welding, bolting, bracing).

* electrical assembly (PC board assembly, parts preparation, soldering & wave

soldering, fasteners).

* injection molding: parameters include weight, cycle time, and cavities in the mold.

* SEER-DFM performs the analysis from standard work measurements (standard

times).

REGRESSION-BASED PRODUCT SPECIFIC COST MODELS

In many cases regression-analysis based cost models are generated by government agencies

(or program offices), government support contractors, or hardware/software contractors. In such

cases, a sufficient quantity of cost data is usually available to build statistical, i.e., regression based

CERS. The flow diagram in Figure IV-3 shows the most commonly used process.


96

The point of departure is a Work Breakdown Structure (WBS) or a Cost Breakdown

Structure (CBS). There are many WBSs in existence, such as the standard WBS by DoD

(MIL-STD-881B) or contract specific WBS's (See Appendix B). Often, a contract specific WBS is


97

broken into work packages which are not very suitable for determining component or subsystem

costs. Considerable effort is usually required to extract cost drivers from the cost of WBS elements

which are meaningful and "uncontaminated" by activities which are applicable to more than one

WBS element.

Preliminary CERs are conceptualized, and both cost and technical data are collected

according to the WBS elements and CER format. The cost data are then normalized with respect to

base year constant dollars, quantity of hardware, completeness of cost inclusions, etc., and statistical

regressions are performed to correlate cost (dollars or hours) to cost drivers. Mathematical

equations for the CERs resulting from the regression analyses are first checked for their statistical

validity, and next for technical credibility. An extensive review cycle is then entered, complete

with Beta testing, checking by technical and cost experts within the government and contractor

communities, and modification of the CERs based on revised information or additional data

sources. The final step of this process consists of a significant amount of documentation of the

CER equations, description of included and excluded costs, data sources, ground rules, rationale,

limits, uncertainties and applicable comments.

This cost model process leads to parametric CERs which can be used by the government

and by contractors.

NON-COMMERCIAL COST MODELS

The Government often sponsors the Development of Parametric Cost Models (e.g.,

Unmanned Space Craft Model Version 7 by Tecolote Research. And the Missile Analysis and

Display Model by EER Systems Corporation which was sponsored by the USA Missile Command).

These Cost Models are usually based on historical data from a number of systems and are used by

Government and Contractor Cost Analysts alike to develop cost estimates. They are often used as

secondary estimating techniques and as crosschecks to the primary estimating methodology.

NON-STATISTICAL COST MODELS

Non-statistical cost models are those which are based on other types of mathematical

analysis, and are generally engineering models. In many cases, insufficient data exist to construct a

meaningful, regression based CER or model. For example, if only two cost data points are present


98

for an item (a hardware item like a transmitter, for instance) an approximation of a CER would

consist of fitting a power curve with an exceptable exponent through the two data points. The

exponent should make sense in light of the analyst's experience.

Non-statistical cost models are generally more judgmental than those based on regression

and additional actual data. Uncertainties associated with a small number of actual data points could

easily lead to large inaccuracies of the cost model. However, a larger amount of data on which a

regression-based cost model is based does not necessarily improve its accuracy. It depends on the

quality of the majority of the data points (quantity does not always equate to quality).

The use of non-statistical cost models puts additional emphasis on the competence of the

model originator and will generally increase the skepticism of an auditor. Non-statistical cost

models for proposals should generally be used under the following conditions:

(1) An insufficient (to produce statistically based CERs and models) amount of historical data

exists;

(2) The existing database is really not applicable (i.e., new technology or new products are

being estimated);

(3) The cost modeler has sufficient expertise, knowledge and competence to construct the

model (e.g., with respect to choice of variables);

(4) The model can be validated for at least one, preferably more than one, case; and,

(5) Excellent cost and technical visibility exists for the few data points which form the basis of

the model.

The auditor or reviewer needs to scrutinize the cost model with respect to the above

mentioned five criteria, and request sufficiently reasonable explanations with respect to the

derivation of the cost model. If necessary, the auditor may also require an explanation why the

particular set of variables are being used in the model.

Non-statistical cost models may not always have weight and complexity as independent

variables. Size may be implied by amount of energy, thrust level, tank volume, or parts count.

Complexity may be implied by temperature, efficiency, thermodynamic cycle type, or

manufacturing processes and packaging density. In many cases, non-statistical cost models can be

constructed by equating cost to these directly measurable parameters. Care has to be taken to

equate cost to independent technical parameters.


99

In general, non-statistical cost models are used at a top level, i.e., are often not divisible into

the typical WBS labor and material cost components. However, they can also cover simple

cost-to-cost ratios such as quality control labor to manufacturing touch labor hours with only one or

two data points. In this case, it is up to the contractor to convince the auditor that the historically

developed relationship (e.g., labor ratio) for one product will also hold for another (hopefully

similar) product.

MODEL CALIBRATION

The act of calibration standardizes a model. Many models are developed for specific

situations and are, by definition, calibrated to that situation. Such models usually are not useful

outside of their particular environment. However, general cost estimating models including

commercially available ones such as the FAST, PRICE and SEER Models (described earlier) are

designed to be useful as estimating tools for a wide range of situations. The act of calibration is

needed to increase the accuracy of one of these general models by making it temporarily a specific

model for whatever product it has been calibrated for. Calibration is in a sense customizing a

generic model.

Items which can be calibrated in a model are: product types, operating environments, labor

rates and factors, various relationships between functional cost items, and even the method of

accounting used by a contractor. All general models should be standardized (i.e. calibrated), unless

used by an experienced modeler with the appropriate education, skills and tools, and experience in

the technology being modeled.

Calibration is the process of determining if there is any deviation from a standard in order to

compute correction factors. For cost estimating models, the standard is considered historical actual

costs. The calibration procedure is theoretically very simple. It is simply running the model with

normal inputs against items for which the actual costs are known. These estimates are then

compared with the actual costs and the average deviation becomes the correction factor for the

model. The actual data used for the calibration runs determines what type of calibration is done. In

essence, the calibration factor obtained is really good only for the type of inputs that were used in

the calibration runs. For a general total model calibration, a wide range of components with actual

costs need to be used. Better yet, numerous calibrations should be performed with different types

of components in order to obtain a set of calibration factors for the various possible expected


100

estimating situations. For instance, the PRICE Hardware model addresses this situation using

internal complexity factors. These can be modified by calibration to the component level. Board

fabrication, parts costs, packaging costs, and the overall design and manufacturing costs have

factors that can be tuned to a specific product, product line, or business.

COST MODEL AUDIT AND ANALYSIS

Cost models are significantly more complex subjects than simple CERs. Typically a cost

model uses both CERs and mathematical relationships between the CERs. Both cost models and

CERs are based on historical data. The analyst reviewing a cost model must review the model, the

basis history and the specific application. In addition, models must be evaluated heuristically by test

cases and sensitivity analyses. If a product specific model is used in an estimate, the analyst needs

access to more information and be able to explore the model's internal relationships and data base.

The disadvantage of generic proprietary cost models is that the cost analyst sometimes does not

have access to the data base and the CERs embedded in the model's "black boxes."

Analyzing A Product Specific Cost Model

A non-standard cost model is usually built by the estimator for the specific purpose of

calculating specific product costs. As such, it will have specific company factors, CERs, and

relationships. It is not uncommon to have a pricing model as a back end to a proprietary model for

purposes of adding pricing factors and to do output formatting. In addition, it is common to have a

front end to the product specific to deal with purchased items, ancillary costs and sub-models. The

firm submitting an estimate may use a spread sheet which are utilized to adjust one model output or

format. See Figure IV-4.

Company

Specific Factors

Heuristics

Software Model

Hardware & Project Model


101

Pricing Factors

and Formats

FIGURE IV - 4

When considering calibration, from an auditor's standpoint, basically three things need to be

known about an estimate that is prepared using a model. There are:

(1) The skill level of the model builder.

(2) If the model was calibrated, what kind of data was used to do the calibration? Was it an

overall general calibration using several data points or a more specific calibration with

actual cost data that represents the project being modeled? A calibration will have more

credibility if it is performed with data similar to the project being estimated.

(3) The third consideration has to do with the quality of the actual cost and non-cost data

that went into the calibration. Historical actual costs are not necessarily "real" costs as

far as a calibration activity is concerned (See Chapter II: Normalization of Data). Other

sources such as actuals from a database, vendor firm quotes, vendor ROMS,

management estimates to completion, industry published costs, and even estimates from

other products or other organizations sometimes can be used as inputs for calibration.

The appropriateness of such inputs need to be considered. Knowing what went into the

calibration (for a specific model) and how it relates to the model's current estimate

needs to be taken into consideration when evaluating the appropriateness and quality of

the calibration.

When a product specific cost model is used to perform the actual cost estimate, the analyst

has to evaluate the model much the same as for single CERs. In addition, however, the relationship

between the cost elements has to be explored. Are the cost drivers generating only one set of costs

or are the various drivers producing overlapping costs - for example, if a factor is calculating both

program management and data, does the program management number exclude data costs? If

product specific models are used, the basis cost history is important for checking the calibration of

the model. It also has to be examined to see if the source information is clean and well defined.


102

Regardless of the model type, it is desirable for the model to be calibrated for the specific

situation at hand. For proprietary product specific models, the calibration has to include company

specific calibration in order to account for company peculiarities, accounting methods and rates.

Once the general features of the company are incorporated into the model, the auditor has to check

the basis of estimate (sometimes called complexity) factors. Since models usually broaden their

effective databases by incorporating functional relationships within parameters, the historical basis

data utilized by the model has to be examined for both cost validity and the mathematical

relationships between the technical parameters and cost. This step is important since the system to

be estimated rarely is exactly like (technically or functionally) the historical case and some

normalization is usually required in the value of the independent variables. The analyst must see

the documentation associated with the basis case both for the costs and the technical features. In

other words, the basis case must show the source data for the estimated engineering, data, prototype

costs, etc. The basis case documentation must also show the technical data - weight, size

complexity etc. Again, if the cost elements are significant, considerable detail may be necessary to

verify the basis calibration. For instance, if Printed Wiring Assembly (PWA) costs are a significant

driver for the "black box" being estimated, the basis costs and technical descriptions must be

available in some detail perhaps down to the board and chip count level.

Once the auditor has verified the model validity both for the corporation and for the specific

basis case, he or she can then address the application of the model to the case at hand. This

involves examining the tasks being estimated and the technical features of those tasks. In

development cost estimates, the technical detail may be thin, consisting of simply performance

requirements. In this case, the basis system performance characteristics become significant. For

production estimates, hardware item drawings should exist and can be compared directly to the

drawings of the basis case - which might even be a previous version of the item being estimated. In

all cases, any extrapolation must be supported by quality information. Vague references of

"engineering judgment" are not satisfactory; this judgment has to be translated into concrete

hardware definitions.

Summary Of Cost Model Audit And Analysis

In summary, the analyst reviewing a cost model has to work through several steps in order

to evaluate the model-based estimate. These are:


103

* Is the model proprietary or non-standard? Does the overall estimate contain interface

models or algorithms which transfer data into and between other models?

* If non-standard, are the model interrelationships well described and well defined, and do

they make logical sense?

* Do the model elements deal with all the costs in the estimate? If not, is there any

overlap between the costs derived through the model and the costs developed outside

the model?

* Is the model used without adjustment? If adjusted, what is the justification and source

data for the adjustments?

* Is the basis case used for reference in the estimate a close analog to the case being

estimated? If not, how far is the basis case, functionally and physically, from the case to

be estimated? What are the functional relationships within the model parameters, and

how sensitive is cost to error in specifications? What is the source documentation for

the base case?

* What are the values of the elements in the case to be estimated? How do they compare

to the values in the basis case? What are the source data for the case to be estimated?

Given that not all these questions have unequivocal answers, the auditor can examine how

variations in the parameter values affect the estimate. If the model is computerized, the auditor can

even do "what if" tests to see if some particular cost relationship is especially important. For those

elements which show significance and which are unsatisfactorily supported, the auditor can request

further information and data.

The objective of the analysis is to see whether the estimate is fair and reasonable. In

complicated logical structures, this issue of overall reasonableness may evolve as a result of

examination of the elements of the estimate. In addition, softness in data, vagueness in

specification and bias in projection can elevate uncertainty in the cost estimate to the point that this

uncertainty becomes a reasonableness issue. In short, the cost analyst evaluating a cost model has

to assure him/herself that the model is a good simulation of the true cost influences, that the basis of

the estimate is appropriate and well founded, and that the extension of the basis through the model

is a true picture of what is proposed in the technical description of the system. If the answer to all

these questions is “yes” and if the uncertainty associated with the estimate is reasonable, the analyst

is looking at a good cost model and estimate.


104

Appendix E lists other examples of hardware models and describes in some detail a few of

them.


105

CHAPTER V

SOFTWARE PARAMETRIC

COST ESTIMATING


106

CHAPTER V

SOFTWARE PARAMETRIC COST ESTIMATING

SOFTWARE DEFINITION

Software, in general, is a set of programs and accompanying documentation that direct

computers to perform desired functions. In simple terms, a software program is a set of

instructions for a computer. Software is critical, not only in the control of space based systems,

but in virtually all current military systems. For example, software controls aircraft engines,

drivers and simulators; it directs surveillance systems, handles space shuttle operations, and

controls account, inventory and Management Information Systems (MIS).

There are three basic types of software. These are:

* System software (also known as the operating system) is a collection of programs

that manages all the concurrent tasks being performed by a computer, including the

execution of application software programs.

* Utility software is a set of programs that perform routine day-to-day tasks, such as

listing or compressing data, copying files, etc.

* Application software performs specialized functions like Space Shuttle control and

the MIS functions mentioned above, or other useful work not related directly to the

operation of the computer itself.

Most analysts are familiar with all three types of software as all are used in personal

computers. MS-DOS (Micro Soft Disk Operating System), for example, is a trade name for the

operating system for Intel X86 Personal Computers. LOTUS 123 and Micro Soft Excel are

examples of application based software. Listing a directory of user files is a utility software

function. The system software procured by the US Government is usually a complex

combination of all three types of software.

THE IMPORTANCE OF SOFTWARE TODAY


107

As discussed earlier, maintenance problems are a driving force behind re-engineering,

but a new business strategy now challenges software organizations. It is called business process

re-engineering or just business re-engineering. The reader should never confuse business re-

engineering with software re-engineering. We have included this section to discuss how software

re-engineering is being used to support business re-engineering.

Business re-engineering is the fundamental rethinking and radical redesign of business

processes to achieve dramatic improvements in measures of performance, such as cost, quality,

service, and speed.

Traditionally, organizations tend to be structured around divisions such as manufacturing,

marketing, administrative support, etc. But this structure is often a barrier to quick

responsiveness to a rapidly changing environment. When confronted with a new objective,

client, strategy, etc., every division must be mobilized from the top down.

So how can software re-engineering help?

Software was originally developed to support business functions within the traditional

organizational structure. Thus, we are left with legacy systems that are marketing-oriented, or

manufacturing-oriented, etc. Software re-engineering captures the design behind the software.

Using new tools and techniques currently available, this design-information can be broken up

into “chucks” that are functionally integrated. These “chunks” are then analyzed and regrouped

around the newly identified business core processes.

In many ways, this seems a lot like the process of translating from structured analysis to

object-oriented analysis. Some advocates of business re-engineering admit this is the case. Both

involve changing the basic approach to software and business. Software (and organizations)

should correspond to a meta-model of the real-world. In the past, we attempted to force our

software designs and organizations to conform to a structure that was basically incompatible with

the real-world. Users expressed their real-world requirements in terms of familiar objects (e.g.

order forms , personnel records, etc.) which were translated into process oriented programs.

Maintenance also required a similar translation from the user’s modified needs to the process-

oriented representation (i.e. program). Now, with object-oriented analysis and business re-


108

engineering, we are beginning to realign our software and organizational structure so that they

correspond to real world objectives and needs.

Another area where software re-engineering can aid business re-engineering is the

identification of business rules from legacy course code. Embedded within legacy systems are

the implied rules that govern how the organization was run at the time the system was developed.

By extracting these business rules, an organization can better understand and, later, codify the

rules by which they conduct business.

Improving software quality and productivity is a major challenge faced by the Department

of Defense (DoD) and the non-military community (DOE, NASA) as well. In addition,

providing affordable weapon system flexibility through software is a specific challenge for the

DoD.

Improving software quality is basic to how well software meets the requirements and

expectations of the users. It also means ensuring that software is adequate, reliable, and efficient.

Improving productivity means favorably increasing the ratio between the resources required to

develop software and the size and complexity of the developed software.

The growth in computer use and computer hardware capabilities has placed demands of

increasing magnitude and complexity on computer software. Software development processes,

along with attendant methodologies, which may have worked well in the past, often break down

when applied to the development of today’s software. For example, studies show that every five

years the sizes of software projects, as measured by Source Lines Of Code (SLOC), increase by

an order of magnitude, and that the scaling of the development effort, demanded by the order-of-

magnitude increases now require fundamental changes in the development process. As the sizes

of software projects have increased, software development processes based on individual

programmers have given way to processes based on small teams and, in turn, small teams have

given way to larger teams and so on. Higher scaling of software development processes by

merely increasing team sizes reaches limits on effective project management and resource

availability.

Today’s users of software demand software applications of greater size and complexity

than before. Advances in computer hardware capabilities are more than adequate to match the

demands of users; however, software as it is developed using prevailing processes and


109

methodologies is not. The challenge is finding software development processes with attendant

methodologies and technologies that meet user demands and that improve software quality and

productivity.

As a percent of total cost, software cost has grown disproportionally more than hardware

cost (see Figure V-1).

FIGURE V-1 HARDWARE/SOFTWARE COST TRENDS

The military, like the business world today, sees software providing the versatility and

leverage to achieve performance goals. For example, software demonstrated its flexibility to

quickly change weapon system capabilities in Desert Storm, the most newsworthy being the

rapid development of a new software package for the Patriot Missile system to counter the

SCUD. Because of the versatility and leverage provided by software, the DoD’s appetite for

software in the future has been described as “virtually insatiable.”

Software is an increasingly important element in military systems of all kinds. The

capabilities of current and future military systems are dependent on the performance of the

systems’ software. As a system is upgraded or improved, much of the additional capability is


110

achieved through new software. In fact, many of the functions essential to mission or

organizational success are partly or completely accomplished through the use of software.

Unfortunately, software development and maintenance is an error-prone, time-consuming

and complex activity. Experience has revealed that many software development efforts falter

because the management of these projects fall into several common traps. These problems are:

* Lack of adequate requirements/specifications.

* Lack of attention to user needs and constraints.

* Lack of visibility into the development process.

* Lack of control at key points during development.

* Lack of standardization.

* Lack of attention to cost of ownership considerations.

* Lack of adequate documentation.

* Lack of adequate training and skills in estimating effort and schedule.

Falling into these traps leads directly to increased development and support costs,

schedule slips, and reduced systems capability.

DOD-STD-2167A/498, Defense System Software Development, established uniform

requirements for software development and is widely applied in all software application areas by

all DOD components. DOD-STD-7935A/498, DOD Automated Information System (AIS)

Documentation Standards provides guidelines for the development and revision of

documentation for information systems. For embedded systems, DOD-STD-2167A/498 is

appropriate. For information systems, both DOD-STD-2167A/498 and DOD-STD-7935A are

appropriate. The documentation requirements of DOD-STD-7935A take precedence over those

of DOD-STD-2167A/498. In both application areas, tailoring of these standards must be done to

reflect the unique needs and constraints of each project.

The actual software development tasks will be accomplished by a development

organization that is an element of, or subcontractor to, the system development contractor. On

occasion, this “contractor” may be a DOD agency. The development contractor has the

responsibility for delivering a software product that meets all contractual requirements.

Unfortunately, at the beginning of a contract’s development efforts, it us usually not possible to


111

specify precisely and completely all of the characteristics of the final software products and their

development processes. Experience has shown that the difference between successful and

unsuccessful development efforts is the vigor and timeliness of the direction given to the

contractor by the Program Manager, supported by the Project Office.

Figure V-2 indicates the relative impact of the penalty (cost) for delayed error correction

during a software project’s life cycle, from almost no cost during preliminary design to high cost

impacts during validation and operative stages. BPR focuses on the later stages (maintenance)

and the reduction of errors at those times by re-engineering the early phases.

In today’s world of shrinking budgets, providing affordable, flexible software systems

requires cost control and predictability that are not found in the traditional software development

processes. Increasingly, the DoD demands that software be developed within predictable costs

and schedule. Enter, therefore, parametric cost modeling.

FIGURE V-2 RELATIVE PENALTY-ERROR CORRECTION


112

THE SOFTWARE DEVELOPMENT PROCESS

This section defines the basic process of software development as currently practiced by

the majority of government contractors. Major phases and software development activities are

defined, as well as key milestones for the measurement of progress.

The Waterfall Model

DOD-STD-2167A/498, the current prevailing standard guiding software development,

has been interpreted as mandating as specific process for use on all military acquisitions. This

process is represented by the “Waterfall” Model, which serves as the conceptual guideline for

almost all Air Force and NASA software development. The process described by the model

involves development through specific, sequential stages. There are specific objectives to be

accomplished in each stage; each activity must be deemed successful for work to proceed to the

subsequent phase. The process is usually considered non-iterative. Each phase requires the

delivery of particular documentation products (Contract Data Requirements List - CDRL Items).

Many of the phases require successful completion of a government review process. Critics of the

“Waterfall” Model, in fact, find that the model is geared to recognize documents as a measure of

progress rather than actual results.

The eight major activities described in 2167A/498 are as follows:

* Systems Concept/System Requirements Analysis

* Software Requirements Analysis

* Software Parametric Cost Estimating

* Preliminary Design

* Detailed Design

* Coding and Computer Software unit (CSU) Testing

* Computer Software Component (CSC) Integration and Testing

* Computer Software Configuration Item (CSCI) Testing

* System Integration and Operational Testing


113

A schematic overview of the Waterfall Model, representing concurrent hardware and

software development, is presented in 2167A/498, and reproduced below as Figure V-3.

FIGURE V-3 WATERFALL MODEL


114

An alternative approach to software development involves the use of incremental builds.

With this approach, software development begins with the design of certain core functions to

meet critical requirements, and each successive software “build” provides additional functions or

enhances performance. Once system requirements are defined and preliminary system design is

complete, each build may follow the waterfall pattern for subsequent development phases. Each

successive build will usually have to be integrated with prior builds.

THE SOFTWARE COST ESTIMATING PROCESS

The overall process of developing a cost estimate for software is not different from the

process for estimating any other element of cost. There are, however, aspects of the process that

are peculiar to software estimating. Some of the unique aspects of software estimating are driven

by the nature of software as a product. Other problems are created by the nature of the estimating

methodologies. Brooks, in his 1982 collection of essays, referred to large system software

programming as a “tar pit.” His description of one such project is typical of Space and Missiles

Center and NASA experience with software development. He states:

“The product was late, it took more memory than planned, the costs were several times

the estimate, and it did not perform very well until several releases after the first.”

Why is it so difficult to estimate the cost of software development? Many of the

problems that plague the development effort itself are responsible for the difficulty encountered

in estimating that effort. One of the first steps in any estimate is to understand and define the

system to be estimated. Software, however, is intangible, invisible, and intractable. It is

inherently more difficult to understand and estimate a product or process that cannot be seen and

touched. Software grows and changes as it is written. When hardware design has been

inadequate, or when hardware fails to perform as expected, the “solution” is often attempted

through changes to the software. This change may occur late in the development process, and

sometimes results in unanticipated software growth. In this case it is most important to create a

picture, since the product can be highly ambiguous at this time.


115

The software WBS (see Appendix B) is an excellent tool for visualizing the software

product. The WBS need not be complex, nor does it need to be highly detailed. A simple

product tree line drawing is often adequate for initial software estimates. The hardware WBS

can be a useful tool in developing the initial WBS for software. There is usually a software

Computer Software Configuration Item (CSCI) or similar module associated with each hardware

Line Replaceable Unit (LRU). As the program evolves, the initial or draft WBS should include

all software associated with the program regardless of whether it is developed, furnished, or

purchased. If furnished or purchased software were omitted, it would not be possible to capture

the cost of integrating preexisting or purchased software with the development software.

The WBS should depict only major software functions, and major subdivisions. It should

not attempt to relate the software to the hardware it controls. Each of the major software

functional units can be modeled as a Computer Software Configuration Item (CSCI). Lower

level WBS elements can be modeled as a component. Once the WBS is established the next step

is to determine which estimating technique should be used for deriving the estimate.

One of the most challenging tasks in project management is accurately estimating needed

resources and required schedules for software development projects. The software estimation

process includes estimating the size of the software product to be produced, determining which

functions the software product must perform, estimating the effort required, developing

preliminary project schedules, and finally, estimating overall cost of the project.

Size and number of functions performed are considered major productivity

(“complexity”) factors during the software development process. Effort is divided into labor

categories and multiplied by labor rates to determine overall costs. Therefore, software

estimation is sometimes referred to as software cost estimation.

Software life cycle models identify various phases and associated activities required to

develop and maintain software, and provide excellent input into the software estimation process.

Some of the more common and accepted life cycle models include: (1) Waterfall Model; (2)

Rapid Prototyping; (3) Incremental Development Model; (4) Spiral Development Model; (5)

Reusable Software Model; and (6) the Transform Model [Boehm and Davis]. These models

form a baseline from which to begin the software estimation process and should be reviewed and

tailored to the proposed project.


116

Software identified as mission-critical and developed for the United States government

usually must be developed in accordance with DOD-STD-2167A/498. This standard establishes

uniform requirements for software development, and does not specify or discuss any particular

method of software development. However, it requires the inclusion and documentation of the

major software development life cycle activities. The standard also requires that reviews and

audits be held in accordance with MIL-STD-1521B, “Technical Reviews and Audits for Systems,

Equipment, and Computer Programs.”

Additionally, structured approaches to sub-task identification are extremely beneficial in

determining tasks and the required effort for each task. The WBS is a method which strongly

support this process.

The software estimation activity should be approached as a major task and therefore

should be well planned, reviewed and continually updated. The basic steps required to

accomplish software estimation are described in the following paragraphs.

Define Project Objectives and Requirements

The objectives and requirements of a software project are usually established by upper

management directive or by a contract Statement Of Work (SOW). A clear set of objectives

must be established in order to accurately identify project requirements. Project requirements

must also include specifications that must be met and applicable standards that must be followed.

Project objectives and requirements must be defined as clearly and precisely as possible in order

to accomplish the project correctly, as well as identify tasks and ultimately estimate costs as

accurately and early as possible.

Plan The Activities

As previously mentioned, the software estimation activity should be planned as a major

task. The plan should detail the purpose, products, schedules, responsibilities, procedures,

required resources, and assumptions made. The plan should include which estimation

methodologies, techniques, and tools will be used.


117

The project should be organized into a hierarchical set of tasks to the lowest level of

detail that available information will allow. Additionally, a breakdown of documentation

requirements and associated tasks should be defined (the detailed WBS).

The WBS helps establish a hierarchical view and organization of the project. The top

level is the software system or final software product, and subsequent levels help identify tasks

and associated sub-tasks and are used to define and encapsulate system functions. Each of these

tasks are divided into software development phases such as design, code and test, and integration.

All activities associated with each level must be defined including: project planning and control,

configuration management, product assurance and documentation.

In addition to early development of detailed knowledge about the project, the WBS

provides an excellent methodology for project data collection, tracking, and reporting. During

development of the project, each of the WBS tasks can be given a project budget, and a job

number which is used for reporting time spent on each project phase or activity. This provides

an excellent project tracking and history data collection method. Most government contracts

require that such a Cost/Schedule Control System Criteria (C/SCSC) be established. When the

data are collected to an established WBS, the information can be placed in a database to be used

in refining, tailoring, and customizing the software estimation process. This information

becomes an invaluable input to the software estimation process for future projects.

Software project tasks/subtasks should be defined to the smallest component possible.

All technical aspects of the project should be understood as fully as possible since the more

details known about the project the more accurate the estimates will be. Newer methodologies

are evolving which aid software developers in identifying various functions needed to support the

project, such as Object-Oriented Analysis and Design (OOA, OOD). Therefore, organizations

should actively keep abreast of evolving technologies.

Software Estimation Risks

The effects of inaccurate software estimation and schedule overruns are well known. The

problem stems from an inability to accurately assess risks associated with various software

development projects. Software estimation errors generally result from four major risk areas,

which are:


118

(1) The inability to accurately size the software project. This results in poor

implementations, emergency staffing, and cost overruns caused by underestimating

project needs.

(2) The inability to accurately specify a development environment which reflects

reality. This results in defining cost drivers which may be inappropriate,

underestimated or overestimated.

(3) The improper assessment of staff skills. This results in misalignment of skills to

tasks and ultimately miscalculations of schedules and level of effort required, as

well as either underestimating or overestimating project staffing requirements.

(4) The lack of well defined objectives, requirements, and specifications, or

unconstrained requirements growth during the software development life cycle.

This results in forever changing project goals, frustration, customer dissatisfaction,

and ultimately, cost overruns.

All potential risks associated with the proposed software development project should be

defined and weighed, and impacts to project cost should be determined. This information should

always be included in the software estimation process.

Estimation Methodologies

Several methods (if possible) should be used during the software estimation process. No

one methodology is necessarily better than the other, in fact, their strengths and weaknesses are

often complimentary to each other. It is recommended that more than one software estimation

methodology be used for comparison and verification purposes. One method may overlook

system level activities such as integration, while another method may have included this, but

overlooked some key post-processing components. Five of the methods discussed in Dr.

Boehm’s book Software Engineering Economics are: analogy, bottom-up, top-down, expert

judgment, and algorithms (parametrics).

These methods are often used in conjunction with each other and have been used for

many years by managers of software projects without the use of any formal software estimation


119

tools. Software estimation tools have only recently been developed which incorporate these

methods, and many incorporate multiple methodologies.

Analogy Method

Estimating by analogy means comparing the proposed project to previously completed

similar projects where project development information is known. Actual data from the completed

projects are extrapolated to estimate the proposed project. Estimating by analogy can be done

either at the system level or the component level.

The main strength of this method is that the estimates are based on actual project data and

past experience. Differences between completed projects and the proposed project can be identified

and impacts estimated. One problem with this method is in identifying those differences. This

method is also limited because similar projects may not exist, or the accuracy of available historical

data is suspect. Also, many projects for DOD weapon systems may not have historical precedents.

The analogy or comparative technique uses parametric approaches including the use of CER's.

Bottom-Up Method

Bottom-up estimation involves identifying and estimating each individual component

separately, then combining the results to produce an estimate of the entire project.

It is often difficult to perform a bottom-up estimate early in the life cycle process because

the necessary information may not be available. This method also tends to be more time consuming

and may not be feasible when either time or personnel are limited.

Top-Down Method

The top-down method of estimation is based on overall characteristics of the software

project. The project is partitioned into lower-level components and life cycle phases beginning at

the highest level. This method is more applicable to early cost estimates when only global

properties are known.

Advantages include consideration of system-level activities (integration, documentation,

project control, configuration management, etc.), many of which may be ignored in other estimating

methods. The top-down method is usually faster, easier to implement and requires minimal project


120

detail. However, disadvantages are that it can be less accurate and tends to overlook lower-level

components and possible technical problems. It also provides very little detail for justifying

decisions or estimates.

Expert Judgment Method

Expert judgment involves consulting with human experts to use their experience and

understanding of a proposed project to provide an estimate for the cost of the project.

The obvious advantage of this method is the expert can factor in differences between past

project experiences and requirements of the proposed project. The expert can also factor in project

impacts caused by new technologies, applications, and languages. Expert judgment always

compliments other estimation methodologies. One disadvantage is that the estimates can be no

better than the expertise and judgment of the expert. It is also hard to document the factors used by

the expert who contributed to the estimate.

Parametric Or Algorithmic Method

The algorithmic method involves the use of equations to perform software estimates. The

equations are based on research and historical data and use such inputs as Source Lines Of Code

(SLOC), number of functions to perform, and other cost drivers such as language, design

methodology, skill-levels, risk assessments, etc.

Advantages of this method include being able to generate repeatable results, easily

modifying input data, easily refining and customizing formulas, and better understanding of the

overall estimating methods since the formulas can be analyzed. However, the results are

questionable when estimating future projects which use new technologies, end equations are

generally unable to deal with exceptional conditions such as exceptional personnel in any software

cost estimating exercises, exceptional teamwork, and an exceptional match between skill-levels and

tasks. However, any estimating approach can be impacted by these drawbacks, and care should be

exercised when accounting for such concerns. Additionally, sometimes algorithms are developed

within companies for internal use and many be proprietary as well as more reflective of a specific

company's performance characteristics.


121

Software Cost Estimating Standards

As stated earlier, very often the government requires software development to follow DOD-

STD-2167A/498, which is the Department of Defense standard that specifies the overall process for

the development and documentation of mission critical software systems. This standard also

requires technical reviews and audits to be conducted in accordance with DOD-STD-1521B.

Other standards that may affect the estimating process are: MIL-STD-499A, MIL-STD-

498, Engineering Management; MIL-STD-490A, Specification Practices; MIL-STD-480B,

Configuration Control-Engineering Changes, Deviations and Waivers; DOD-STD-1703, Software

Products Standards. Software developed in accordance with these standards generally requires

more resources and is more time consuming. Therefore, the software estimation process must

include and adjust for these requirements where applicable.

Benefits

When the software estimation process is performed correctly, the benefits realized far

outweigh the cost of doing the estimating. Some of the major benefits include lowering the cost of

doing business, increasing the probability of winning new contracts, increasing and broadening the

skill-level of key staff members, acquiring a deeper knowledge of the proposed project prior to

beginning the software development effort, and understanding, refining, and applying the proper

software life cycle mode.

As these software estimating components are enhanced, refined, and continually applied,

the software estimating process, associated tools, and resulting products attain higher levels of

quality and ultimately benefit all.

EXAMPLES OF PARAMETRIC SOFTWARE COST ESTIMATING

Accurately estimating the resources and time needed for a software development project is

essential even for the survival of the project. In the great majority of cases, the resources and time

actually expended are much more than the initial planning estimates. An approach for estimating

the resources and schedule needed for software development is the use of a software cost and


122

schedule model that calculates the resources and time needed as a function of some other software

parameters (such as the size of the program to be developed).

The more times an organization has developed software of the same size and complexity for

the same type of application, the more accurate the estimates for a new project will be.

Unfortunately, when the Program Manager attempts to extrapolate information from small and less

complex software development efforts to larger, more complex software in different application

areas, the results are often unreliable. The problem results from and exponential relationship

between software size and development effort.

For example, one very widely used parametric software cost model is the Constructive Cost

Model (COCOMO). The basic COCOMO model has a very simple form:

MAN-MONTHS = K1 (Thousands of Delivered Source Instructions) *K2

Where K1 and K2 are two parameters dependent on the application and development environment.

Estimates from the basic COCOMO model can be made more accurate by taking into

account other factors concerning the required characteristics of the software to be developed, the

qualification and experience of the development team, and the software development environment.

Some of these factors are:

* Complexity of the software

* Required reliability

* Size of data base

* Required efficiency (memory and execution time)

* Analyst and programmer capability

* Experience of team in the application area

* Experience of team with the programming language and computer

* Use of tools and software engineering practices

* Required schedule


123

Many of these factors affect the person months required by an order of magnitude or more.

COCOMO assumes that the system and software requirements have already been defined, and that

these requirements are stable. This is often not the case.

Another popular software cost model is the Putnam model. The form of this model is:

PERSON YEARS= (Lines of Delivered source Instructions)

Ck (Development Time in Years)

Where: CK is a parameter dependent on the development environment.

CK has a range from-6,500 (poor) to 12,500 (excellent).

The Putnam model is very sensitive to the development time: decreasing the development

time can greatly increase the person-months needed for development.

One significant problem with the COCOMO, PUTNAM and similar models is that they are

based on knowing, or being able to estimate accurately, the size (in lines of code) of the software to

be developed. There is often great uncertainty in the software size. Computer programs, several of

which are available for PCs, have been developed to implement variations of the COCOMO and

other cost models.

Another estimating approach, called Function Point Analysis (FPA), is used to estimate

both the size of the software to be developed and the effort required for the development. In FPA,

you determine the number and complexity of inputs, outputs, user queries, files, and external

interfaces of the software to be developed. The sum of these numbers, weighted according to the

complexity of each, is the number of function points in the system.

This function point number is directly related to the number of end-user business functions

performed by the system. Using data from past projects, it is possible to estimate the size of the

software needed to implement these function points (typically about 100 source language

statements are needed for each function point) and the labor needed to develop the software

(typically about 1 to 5 function points per person month). FPA has been developed and applied

almost exclusively in Information System applications. A variation, called feature point analysis,

has been defined for other application areas. The chief difference between feature point analysis


124

and FPA is that the number and complexity of the algorithms to be implemented are considered in

calculating the number of feature points.

Do not depend on a single cost or schedule estimate. Use several estimating techniques or

cost models, compare the results, and determine the reasons for any large variations. Document the

assumptions made when making the estimates. Monitor the project to detect when assumptions

that turn out to be wrong jeopardize the accuracy of the estimate.

PARAMETRIC SOFTWARE COST ESTIMATING TOOLS

As mentioned earlier, one of the critical problems facing software development project

managers is determining accurate software estimations for level of effort, schedules, SLOC, and

overall costs. Since trends indicate that the cost of producing software products is escalating and

consuming an ever increasing percentage of budgets, the need to quickly generate more reliable

estimates is becoming even more important.

For many years, project managers have relied on software development teams to estimate

the cost of producing software products. This has always been a subjective and intuitive process

influenced by such factors as personality, opinions, and pressure to win contracts. The process has

encouraged low estimates and short schedules, the results of which have been devastating to

companies and projects, including projects of national importance.

For these reasons, software parametric cost estimating tools have been developed since the

late 1970's to provide a better defined and more consistent software estimating process. These tools

have been developed from historical data collected from thousands of software projects, as well as

research performed to identify key productivity factors. Early tools were hampered by the scarcity

of reliable data; however, as more data became available, estimating tools were improved and

continued to evolve. Most parametric software estimation tools use algorithms, and some of the

more advanced tools are rule-based or knowledge-based as well as interactive.

Good software estimating tools do not always guarantee reliable software estimates. If

inaccurate software size estimates and attribute ratings are input, then inaccurate estimates will

result. Additionally, organizations need to customize the software estimation tools to their own

development environment (the calibration process was discussed previously in Chapter IV).


125

This customization requires collecting and maintaining historical data from current and past

projects to provide the necessary inputs required for the software estimating tools. The software

development organization should establish a staff that is thoroughly trained in the software

estimating process and available estimating tools, and they should perform all the software

estimating activities. Experience and existing tools dictate what project development information

needs to be maintained.

Desired Functional Capabilities Of Parametric Tools

The following section provides an overview of some of the more popular and available

software estimating tools, including the major functional capabilities which software estimating

tools should perform, and the input data required to support the use of those tools.

Major functional capabilities that should be considered when selecting a software

estimating tool are listed below. Depending on the organization's needs, the level of significance of

these capabilities may differ, and should be considered accordingly. In addition, the organization

should analyze their own needs and identify additional desired capabilities specific to them. The

organization should then match available tools with overall needs.

In general, the tool should:

(1) Allow easy adaptation to an organization's development environment - This means the

tool needs to be capable of being customized to fit the using organization's

development environment. Customization includes allowing the developer to define

applicable inputs, as well as to modify coefficients and exponents of the equations

used by the tool. This feature will allow continuous improvement to the estimation

capability of the tool since the organization's historical data and current project data

will be included in the software estimate generated.

(2) Be relatively easy to learn and use - The tool should be well documented including

methodologies and equations used. Documentation should be at a level that is

understandable. The tool should include help menus and examples sufficient to assist

the support staff in answering questions and providing training. The amount of formal

training required to use the tool should be relatively minimal, required inputs should

be well defined, and visibility into internal equations and theories should be provided.


126

(3) Provide early estimates - The tool should be capable of generating estimates early and

quickly in the life cycle process when requirements and development environments

are not fully defined. The tool should also allow task detail to be added incrementally

as functions, activities, and other information becomes more completely defined.

Since there are many unknowns early in the estimating process, the tool should reflect

degrees of uncertainty based on the level of detail input (risk analysis). In general, the

tool should provide sufficient information to allow initial project resource planning as

well as reasonably early "go/no go" decisions.

(4) Be based on software life cycle phases and activities - The tool should be capable of

providing estimates for all phases and activities of the most commonly used software

life cycle models. It should allow the organization to decompose and map software

development tasks into those phases and activities, as well as support a program

WBS. In addition, it should allow for "what if" situations and include factors for

design trade-off studies.

(5) Allow for variations in application languages and application function - It is very

important that the tool provide estimates specific to the application of the software

project since the associated estimating equations, cost drivers, and life cycle phases

should be unique to each application are. General application categories include

Information Systems (IS), simulation and modeling systems, real-time systems,

accounting systems, and systems based on higher-order languages.

(6) Provide accurate size estimates - The size of a software development project is a

major cost driver in most estimating tools, yet size is one of the most difficult input

parameters to estimate accurately. The tool should include the capability to help

estimate the size of the software development project, or at least help define a method

for estimating the size.

(7) Provide accurate schedule estimates - As previously mentioned, schedule overruns are

common and can be extremely costly. The software estimating tool should be able to

provide schedule estimates accurately. The purpose of scheduling is not only to

predict task completion given task sequence and available resources, but also to


127

establish starting and ending dates for the associated work packages and life cycle

phases.

(8) Provide maintenance estimates separately - The software estimating tool should be

able to provide software maintenance estimates as a separate item. Software

maintenance includes such activities as correcting errors, modifying the software to

accommodate changes in requirements, or extending and enhancing software

performance.

Input Data Collection

A very important aspect of software estimating is data collection. Data must be collected

for inputs to the parametric tool and tool verification, validation, customization, and calibration.

Estimates generated by the tools are only as good as the input data used. Careful analysis of

all tool inputs are essential since small changes in input values can result in large variations in

overall cost and schedules.

Inputs vary between tools. Before using a tool, review input requirements and information

collected, documentation, and examples provided with the tool. When possible, discuss these with

individuals familiar with the tool.

Using historical data as a basis for customizing or calibrating a tool is essential. Insure that

information for current project development efforts are saved for future reference.

At a very minimum, use software life cycle model phases and activities as a basis for

collecting and maintaining project development information for future tool use.

Some Commercial Tools

There are many software estimating tools on the market today that provide software

estimation support. Some tools estimate the size of the software project, while others use size as an

input and provide estimates of effort, schedule and cost.

The tools presented in this handbook are not being recommended over other tools as each

one has unique capabilities and limitations. Therefore, organizations considering using estimating

tools should review as many available tools as possible, analyze their software estimating needs,

and then determine which tools are most appropriate to their application and development


128

environment. The list that follows is not all inclusive, and should not be considered complete. The

tools discussed are representative only, since there are many others that could be used.

REVIC

The Revised Enhanced Version of Intermediate COCOMO (REVIC) model was developed

by Mr. Raymond L. Kile formerly of Hughes Aerospace. The Air Force Contract Management

Division, Air Force System Command, Kirtland Air Force Base, New Mexico, sponsored the

development for use by its contract administrator.

The main difference between REVIC and COCOMO is the coefficients used in the effort

equations. REVIC changed the coefficients based on a database of recently completed DOD

projects. It also uses a different method of distributing effort and schedule to each phase of product

development, and applies an automatic calculation of standard deviation for risk assessment.

REVIC provides a single "weighted average" distribution for effort and schedule along with

the ability to let the user vary the percentages in the system engineering and development test and

evaluation phases. REVIC employs a different Ada model than Ada COCOMO. The REVIC

model has also been enhanced by using a Program Evaluation and Review Technique (PERT)

statistical method for determining the lines of code to be developed.

In addition to providing estimates for cost, manpower, and schedule, the program creates

estimates for typical DOD-STD-2167A/498 documentation sizing and long term software

maintenance. REVIC's schedule estimates are often considered lengthy because it assumes that a

project's documentation and reviews comply with the full requirements of DOD-STD-2167A/498.

REVIC operates on PC compatible systems.

PRICES

The PRICES tool is distributed by Lockheed - Martin PRICE Systems. This tool was first

developed in 1977, and is considered one of the first complex commercially available tools used for

software estimation. The equations used by this tool are proprietary. However, descriptions of the

methodology algorithms used can be found in papers published by PRICE Systems. A model

describing the PRICES parametric tool is shown in Figure V-4.


129

FIGURE V - 4

The PRICES tool is based on Cost Estimation Relationships (CERs) that make use of

product characteristics in order to generate estimates. CERs were determined by statistically

analyzing completed projects where product characteristics and project information were known, or

developed with expert judgment.

A major input to PRICES is Source Lines of Code (SLOC). Software size may be input

directly, or automatically calculated from quantitative descriptions (function point sizing). Other

inputs include software function, operating environment, software reuse, complexity factors,

productivity factors, and risk analysis factors. Successful use of the PRICES tool depends on the

ability of the user to define inputs correctly. It can be customized and calibrated to the needs of the

user. It is now available for Windows and UNIX/Motif.


130

SASET

The Software Architecture, Sizing and Estimating Tool (SASET) was developed for DOD

by the Martin Marietta Corporation. SASET is a forward-chaining, rule-based expert system

utilizing a hierarchiacally structured knowledge database. The database is composed of projects

with a wide range of applications.

SASET provides functional software sizing values, development schedules, and associated

man-loading outputs. It provides estimates for all types of programs and all phases of the

development cycle. It also provides estimates for maintenance support and performs a risk

assessment on sizing, scheduling, and budget data.

SASET uses a five-tiered approach for estimating including class of software, source lines

of code, software complexity, maintenance staff loading, and risk assessment. The user can either

input the program size directly or allow SASET to compute size, based on function-related inputs.

The tool also has an extensive customization file in which the user can adjust many parameters. It

operates on PC compatible systems.

SEER-SEM

System Evaluation and Estimation of Resources - Software Estimation Model (SEER-SEM)

is distributed by Galorath Associates and is currently under a five year Air Force wide license

agreement. It provides software estimates with knowledge bases developed from many years of

completed projects.

The knowledge base allows estimates with only minimal high level inputs. The user need

only select the platform (i.e. ground, unmanned space, etc.), application (i.e. command and control,

diagnostic), development methods (i.e. prototype, incremental), and development standards (i.e.

2167A/498). SEER-SEM is applicable to all types of software projects and considers all phases of

software development.

SEER-SEM is designed to run on PC compatible systems running Microsoft Windows

3.0/3.1 (Air Force license includes MS-DOS version). It is also available for the Apple MacIntosh

running system 6.0.3 and above and the UNIX/SUN work station.


131

A companion tool called the SEER-Software Sizing Model (SSM) is also distributed by

Galorath Associates and is used to estimate the size of the software product.

SLIM

The Software Life Cycle Model (SLIM) is marketed by Quantitative Software (QSM).

SLIM was developed in 1979 by Mr. Larry Putnam. Originally developed from analyses of ground-

based radar programs, the SLIM tool has been expanded to include other types of programs. It can

be customized for the user's development environment.

SLIM supports all phases of software development, except requirements analysis, as well as

all sizes of software projects, but was especially designed to support large projects.

Success in using SLIM depends on the user's ability to customize the tool to fit the software

development environment and to estimate both a Productivity Index (a measure of the software

developer's efficiency) and a Manpower Buildup Index (a measure of the software developer's

staffing capability). SLIM also provides a life cycle option which extrapolates development costs

into the maintenance phase.

A companion tool named SIZE Planner is also distributed by QSM and is used to estimate

the size of the software product.

QSM provides a training course and leases the tool via a time sharing service. There is also

a PC compatible version of SLIM that can be leased for a yearly fee.

SOFTCOST-R

SOFTCOST-R is a software estimating tool developed by Reifer Consultants, Inc. (RCI).

SOFTCOST-R is based upon the modeling work done by Dr. Robert Tausworthe of the Jet

Propulsion Laboratory. It contains a database of over 1500 data processing, scientific and real-time

programs. A key input is SLOC, which can be input directly or computed from Function Points.

SOFTCOST-R is applicable to all types of programs, however, it was specifically configured to

estimate real-time and scientific software systems, and considers all phases of the software

development cycle.

The tool's primary input is SLOC, however, it also uses the same inputs and provides the

same outputs as COCOMO which allows direct comparisons to be made.


132

SOFTCOST-R has some unique inputs such as user and peer reviews, customer experience,

and degree of standardization. It also supports a standard WBS for task planning and scheduling.

RCI provides SOFTCOST-Ada, which is a tool to estimate Ada and C++ development

costs. Softcost-Ada is a cost estimation tool specifically developed to estimate systems using

object-oriented techniques.

RCI also has a separate estimating tool called ASSET-R to estimate the size of the software

product. SOFTCOST-R, SOFTCOST-Ada, and ASSET-R are leased on an annual license basis,

and require a PC compatible running DOS 2.3 or higher.

SYSTEM-4

SYSTEM-4 is marketed by Computer Economics, Inc. (CEI). It contains a proprietary

model that is based on the work of Jensen, Boehm, Putnam, and other noted software experts.

SYSTEM-4 is applicable to all types of programs and all phases of the software life cycle.

Inputs consist of size (SLOC), twenty environmental factors, seven development factors, software

type, and constraints. This tool comes with 23 predefined default parameter files. The default sets

provide typical values for all parameters except size. There are also seven parameter subset files for

various implementations of DOD-STD-1703, DOD-STD-2167A/498, and varying degrees of Ada

experience.

The user must select one of the default sets and input the SLOC estimate to perform a quick

estimate. SYSTEM-4 can accommodate multiple CSCIs or tasks, and each task can be broken

down into elements and units. There is a limit of 64 tasks, 64 elements, and 64 units. SYSTEM-4

can be customized to reflect the user's software development environment.

CEI has a companion software size estimating tool called Computer Economics

Incorporated Sizing (CEIS) System. These tools operate on PC compatible systems.

Software Sizing Tools

As discussed previously, a very important factor in estimating software development

projects is the ability to estimate the size of the product. Many software estimating tools use size in

SLOC or functions performed as the major input. Size is also considered by software development


133

project managers as a major technical performance or productivity indicator that allows them to

track the project during software development.

The most commonly used method to estimate the size of a software product is by using both

expert judgment and the analogy method. The experts determine the functions the software will

perform and estimate the size by comparing the new system to completed projects with similar

characteristics. Often, parametric methods are the used to precisely size the software.

Estimating tools using analogy methods compare the new program to similar programs of

known size. Because past projects are not always exactly like the new project, the estimate is

adjusted by a factor determined from experience. These tools accept characteristics of new

programs as inputs, then search a database for similar programs. The tools either list the similar

programs or provide an estimate of size based on an average of the size of the similar programs

selected from the database.

Expert judgment tools use the opinion of one or more experts to estimate the size of the

program, or use structured questions designed to extract judgment from the experts. These are the

rule-based or expert system tools.

Many tools use the algorithmic method by applying equations to determine size estimates.

A technique that is becoming very widely used is Function Point Analysis (FPA). One problem

with FPA is that it was developed for IS-oriented programs and does not take into consideration the

number or complexity of algorithms in scientific and real-time programs.

Many software estimating tools such as REVIC and SLIM use extensions of the Program

Evaluation and Review Technique (PERT). PERT is based on a beta distribution of estimates

provided by the user and calculates expected size according to the equation:

Expected Size = (S + 4(M) + L)/6

where S, M, and L are estimates of the smallest size, most likely size, and the largest size,

respectively.

ASSET-R

ASSET-R is a function point sizing tool developed to estimate the size of data processing,

real-time, and scientific software systems, and is marketed by Reifer Consultants, Inc. It utilizes a

knowledge-based system which extends the theory of function points into scientific and real-time


134

systems by considering issues like concurrence, synchronization, and reuse in its mathematical

formulation. The formulas use as many as nine parameters to develop function point counts. It also

couples function point and operand/operator counts with architectural, language expansion, and

technology factors to generate the size estimate. ASSET-R works with RCI's SOFTCOST-R and

SOFTCOST-Ada software estimation tools. It operates on PC compatible systems.

CA-FPXpert

CA-FPXpert is distributed by Computer Associates International, Inc. It uses FPA for size

estimation of IS type software projects, and conforms to accepted IFPUG standard counting

practices. It includes an on-line tutor to help the function point counting process. CA-FPXpert

works in conjunction with CA-ESTIMACS to provide software size estimation input, and operates

on PC compatible systems.

CEIS

CEIS is marketed by Computer Economics, Inc. Estimates are generated by comparing the

attributes of the new project to the attributes of three reference projects of known size. The user

determines any six attributes that contribute to the number of lines of code and ranks them in order

of importance, then selects three reference projects of known size. Separate algorithms are used to

produce four independent estimates and to determine a level of confidence. CEIS works in

conjunction with SYSTEM-4, and operates on PC compatible systems.

SIZEEXPERT

SIZEEXPERT was developed by the Institute for Systems Analysis and is marketed by

Technology Application/Engineering Corporation. This tool is an expert judgment tool that

produces estimates of liens of code based on questions asked by COSTEXPERT. Both tools are

packaged and distributed together, and operate on PC compatible systems.

SEER-M

SEER-M is marketed by Galorath Associates and is available to government personnel

under and Air Force-wide contract. It produces software size estimates in lines of code or function


135

points. It also provides its own historical database in producing the size estimate. SEER-M works

with SEER-SEM software estimating tool, and operates on PC compatible systems.

Appendix F includes other currently available cost models, and discusses in more detail the

most popular software estimating tools.

GLOSSARY OF TERMS

Appendix A contains definitions of terms commonly used in software estimation

technology.

MODEL CALIBRATION

The act of calibration standardizes a model. Many models are developed for specific

situations and are, by definition, calibrated to that situation. Such models usually are not useful

outside of their particular environment. However, general cost estimating models including

commercially available ones such as the FAST, PRICE and SEER models (described earlier) are

designed to be useful as estimating tools for a wide range of situations. The act of calibration is

needed to increase the accuracy of one of these general models by making it temporarily a specific

model for whatever product it has been calibrated for. Calibration is in a sense customizing a

generic model.

Items which can be calibrated in a model are: product types, operating environments, labor

rates and factors, various relationships between functional cost items, and even the method of

accounting used by a contractor. All general models should be standardized (i.e. calibrated), unless

used by an experienced modeler with the appropriate education, skills and tools, and experience in

the technology being modeled.

Calibration is the process of determining the deviation from a standard in order to compute

the correction factors. For cost estimating models, the standard is considered historical actual costs.

The calibration procedure is theoretically very simple. It is simply running the model with normal

inputs (known parameters such as software lines of code) against items for which the actual cost are

known. These estimates are then compared with the actual costs and the average deviation


136

becomes a correction factor for the model. In essence, the calibration factor obtained is really good

only for the type of inputs that were used in the calibration runs. For a general total model

calibration, a wide range of components with actual costs need to be used. Better yet, numerous

calibrations should be performed with different types of components in order to obtain a set of

calibration factors for the various possible expected estimating situations. For instance, the PRICE

Software Model addresses this situation using internal productivity factors. These can be modified

by the calibration process.

TRENDS AND CONCLUSIONS

Trends

Advances in languages, development methodologies, and other areas will have to be

addressed by future software cost estimating models and associated methodologies. As software

technology matures, changes in development and support concepts occur which will impact

software cost estimating. Concepts such as prototyping and spiral development present a challenge

to cost estimation since normal software development cycles are altered.

Artificial Intelligence (AI) represents a growing area of modern technology. Since AI is

software-intensive, proper management of AI software, including cost estimating, will be a

challenge for software managers. The development of software for expert system and other AI

applications will probably require a different development process.

The trend for the future will include better and more accurate ways of developing software

estimating methodologies for:

* Software size estimates.

* Resource Estimates for maintenance and support.

* Incorporating the effects of Ada and new paradigms such as rapid prototyping and fourth

generation languages.

* Modeling the dynamic interaction of variables that affect productivity, cost, and quality.

* Object-Oriented Design.


137

The trend in software estimating tools is to provide a whole family of models which not

only estimate cost and effort of software development, but hardware as well. The tools are being

upgraded to support higher-order languages such as Ada and C++. The most significant

improvement is the use of data collected from past software projects to customize the tool to an

organization's environment. This is especially true within agencies of DOD and NASA.

Conclusions

This chapter has discussed the software estimating process and the various methodologies

used in software estimation. The basic software estimating functional capabilities were also

discussed. A few different software estimating tools were examined.

A review of product literature and user manuals indicate that many tools will perform most

of the functional capabilities outlined in this chapter. The users generally agree that the tools they

are using satisfy their requirements.

The software estimating organization must be able to customize the software estimating tool

to their own software development environment. This requires collecting historical data from past

completed projects to accurately provide the inputs that the software tool requires. The software

development organization should establish a staff that is thoroughly trained in the use of the tools.

This staff should do all the software estimating activities and determine what data should be

collected to provide a historical database for future reference.

The use of two or more software estimating tools using different methodologies is

recommended. The software development organization should select a primary tool for software

estimating and an alternate tool for comparison and validation. These tools should be used

throughout the software development process. Parametric tools are considered to be the best for

software estimating for the following reasons:

* Equations are based on previous historical development projects.

* Outputs are repeatable and formulas can be analyzed.

* They can be customized to fit the user's environment.

* They require minimal time and effort to use.

* They are particularly useful in earlier phases of software development.

* They are most frequently used by DOD agencies.


138

CHAPTER VI

AUDITING PARAMETRIC ESTIMATES

A MANAGEMENT VIEWPOINT


139

CHAPTER VI

AUDITING PARAMETRIC ESTIMATES

The following paper written by Michael Thibault, Deputy Director, DCAA, provides

background from an auditor's point of view on auditing parametric cost estimating techniques.

COST ESTIMATING RELATIONSHIPS: A DCAA PERSPECTIVE

Introduction

The U.S. Defense Contract Audit Agency (DCAA) was established in 1965. DCAA's

mission is to perform all necessary contract audits for the Department of Defense and, when

requested, to perform contract audit services on a reimbursable basis for other government

agencies. In essence, DCAA provides accounting and financial advisory services for procurement

and contract administration activities. Contract audit activities include providing professional

advice on accounting and financial matters to assist in the negotiation, award, administration,

repricing, and settlement of contracts. DCAA audits are conducted in accordance with generally

accepted government auditing standards established by the General Accounting Office.

This paper discusses the government's expectations when defense contractors use

parametric cost-estimating relationships for estimating government contract costs. The paper also

emphasizes that companies must meet all the adequacy criteria set out in the Federal Acquisition

Regulations (FAR) and applicable supplements to obtain approval for their estimating systems.

Companies must apply the same criteria to their parametric cost-estimating relationships to ensure

they are acceptable for use in estimating systems.

DCAA believes now, as it has always believed, that parametric estimating techniques using

cost-estimating relationships are acceptable in the appropriate circumstances for proposing costs on

government contracts. DCAA is ready and willing to work with industry in the evolution of

parametrics. Operation Desert Shield and Operation Desert Storm dramatically demonstrated that

our government must be capable of responding quickly to changing procurement requirements.


140

Parametric systems can help us do just that. Future estimating systems must be responsive,

accurate, and cost effective.

Background

DCAA was issuing official guidance on parametric systems as early as 1978. Parametrics

was broadly defined as a technique that employs one or most cost-estimating relationships to

estimate costs associated with developing, manufacturing, or modifying an end item. In the 1980s,

DCAA auditors reported an increase in the number of contractors using parametric cost estimating.

DCAA developed and issued audit guidance to assist the field auditors in this new area. Studies

conducted by DCAA; the Office of the Secretary of Defense Cost Analysis Improvement Group;

Headquarters, Air Force Contract Management Division; Headquarters, Aeronautical Systems

Division; and the Space Systems Cost Analysis Group provided the basis for DCAA audit guidance

issued in 1982.

Parametric Criteria

This guidance was also the subject of an article written for the Spring 1982 issue of Journal

of Parametrics published by the International Society of Parametric Analysts. Charles 0. Starret, Jr.,

then-Director of the Defense Contract Audit Agency, wrote the article entitled "Parametric Cost

Estimating -- An Audit Perspective." The guidance contained in that article is essentially the same

as the guidance given to DCAA auditors today. It reiterates DCAA's long-held view that

parametrics is an acceptable estimating technique. The 1982 article included the criteria a

contractor should apply before submitting a contract price proposal using parametrics. The criteria

are still on point today, and they are:

1) Logical relationships

2) Significant statistical relationships

3) Verifiable data

4) Reasonably accurate predictions

5) Proper system monitoring

Logical Relationships


141

Contractors are expected to demonstrate that cost-to-noncost-estimating relationships are

logical. "Logical relationship" is often difficult to determine in a finite sense, yet is very important.

DCAA's primary concern in this area is that a contractor consider all reasonable logical estimating

alternatives and not use only the first apparent set of variables. Contractor analysis may disclose

multiple alternatives that appear logical. Statistical testing should be used to help identify the best

alternative.

Significant Statistical Relationships

Contractors are also expected to demonstrate that a significant statistical relationship exists

among the variables used in a parametric cost-estimating relationship. There are several statistical

methods such as regression analysis that can be used to validate a cost-estimating relationship;

however, no single uniform test can be specified. Statistical testing may vary depending on an

overall risk assessment and the unique nature of a contractor's parametric data base and the related

estimating system. Proposal documentation should describe the statistical analysis performed,

including the contractor's explanation of why the cost-estimating relationship is statistically valid.

Verifiable Data

There must be a system in place for verifying data used for parametric cost-estimating

relationships. In many instances, the auditor will not have previously evaluated the accuracy of

noncost data used in parametric estimates. For monitoring and documenting noncost variables,

contractors may have to modify existing information systems or develop new ones. Information

that is adequate for day-to-day management needs may not be reliable enough for contract pricing.

Data used in parametric estimates must be accurately and consistently available over a period of

time, and easily traced to or reconciled with source documentation.

Reasonably Accurate Predictions

The contractor's demonstration that the parametric cost-estimating relationships predict

costs with a reasonable degree of accuracy is also important. The key is that if the contractor's

analysis of historical estimating and cost performance data shows that the parametric estimating


142

system is as accurate as a discrete estimating system, then the government has increased assurance

of receiving a fair and reasonable price.

As with any estimating relationship derived from prior history, it is essential for the

contractor to document that the work being estimated using parametric cost-estimating relationships

is comparable to the prior work from which the parametric data base was developed.

Proper System Monitoring

The contractor should also ensure that cost-to-noncost parametric rates and factors will be

monitored periodically in the same manner as is expected for cost-to-cost rates and factors.

Because of improved technology, production changes, or better pricing alternatives, cost-estimating

relationships can and do change. The contractor should be prepared to revalidate any parametric

cost-estimating relationship whenever system monitoring discloses that the relationship has

changed.

Audit Planning and Requirements

The old expression, "The more things change, the more things stay the same," seems to

apply. Government procurement procedures may change to accommodate changes in the services

and products it buys, but the basic procurement goal of getting products and services for fair and

reasonable prices remains the same. And so it goes with auditing. Whether DCAA audits proposed

costs for space stations or for missiles, DCAA's basic aim of ensuring that the proposed costs are

allowable, allocable, and reasonable and therefore acceptable for government reimbursement is still

the same. How DCAA accomplishes its audit objective varies with the sophistication of contractor

accounting and estimating systems.

Auditors begin the audit by ensuring they have the requisite familiarity with DCAA

guidance on estimating systems and techniques. This guidance is contained in DCAA's Contract

Audit Manual (CAM), which is available to the general public. The auditor then proceeds to do the

following:

1) Ensure they are familiar with the company's estimating policies and procedures.

2) Identify the estimating methods used to develop the proposal.


143

3) Determine that the supporting cost and pricing data for the individual proposal was derived

in accordance with the contractor's estimating system and is in compliance with applicable

regulations.

The auditor plans the audit scope using what is known about the contractor. For example,

the audit scope will vary depending upon the estimating methods the contractor uses. In addition,

the auditor will consider the following types of questions:

* What is the dollar amount and type of contract contemplated?

* Has the contractor established strong internal controls and sound accounting and

estimating systems?

* What kinds of testing does the contractor do to ensure compliance with these systems?

* What does our prior audit experience tell us about the contractor's internal controls or

estimating practices?

Audit planning requires the auditor to answer all of these questions and to make a

determination regarding the government's risk. Judgment is then exercised in deciding the degree

of risk that the estimate could be materially misstated. This assessment of risk will be used to

decide what audit procedures to employ.

The auditor identifies the method of estimating the contractor uses to determine the kind of

support that should be available. A contractor could be using any or all of the following methods:

1) Detailed -- also known as the bottoms-up approach. This method divides proposals into

their smallest component tasks and are normally supported by detailed bills of material.

2) Comparative -- develops proposed costs using like items produced in the past as a

baseline. Allowances are made for product dissimilarities and changes in such things as

complexity, scale, design, and materials.

3) Judgmental -- subjective method of estimating costs using estimates of prior experience,

judgment, memory, informal notes, and other data. It is typically used during the

research and development phase when drawings have not yet been developed.


144

Parametric estimating techniques may be used in conjunction with any of these methods.

Whatever the method selected by the contractor, it must comply with applicable laws and

regulations. The laws and regulations most often encountered in dealing with parametrics are:

1) CAS 401, "Consistency in Estimating, Accumulating, and Reporting Costs"

2) Truth in Negotiations Act (TINA)

3) FAR 15.800, Price Negotiation

4) DFARS 215.811, "Estimating Systems"

Cost Accounting Standards (CAS) provides guidance in accounting for contract costs at

larger contractors. CAS 401 requires that a contractor's estimating practices be consistent with

those governing the accumulation and reporting of costs during contract performance. Some

contractors see parametrics as being inconsistent with CAS 401. Contractors must ensure both cost

and noncost information used in estimating is separately accumulated and reported as required by

CAS 401.

The purpose of the Truth in Negotiations Act, 10 U.S.C. 2306(f), is to provide the

government with all facts available to the contractor at the time it certified the cost or pricing data

was accurate, complete, and current. Parametric estimates must meet the same basic disclosure

requirements under the act as discrete estimates. Although the principles are no different, proposals

supported in whole or in part with parametric estimating will have different types of cost or pricing

data than traditional discrete cost estimates.

Fundamental to the definition of cost or pricing data are "all facts ... which prudent buyers

and sellers would reasonably expect to have a significant effect on price negotiations" (FAR

15.801). Reasonable parallels may be drawn between the data examples provided in FAR for

discrete estimating approaches and the type of data pertinent to parametric estimating approaches.

The contractor is also expected to provide all factual data for the parametric cost estimates. This

data must be accurate, complete, and current.

Many contractors use parametric cost estimating for supplementary support or validation of

estimates developed using other methods. This requires judgment in selecting which data will be

used in developing the total cost estimate relied upon for the price proposal. In distinguishing

between fact and judgment, FAR states the certificate of cost or pricing data "does not constitute a


145

representation as to the accuracy of the contractor's judgment on the estimate of future costs or

projections. It does apply to the data upon which the contractor's judgment or estimate was based"

(FAR 15.804-4b). Thus, if a contractor develops a proposal using both parametric data and discrete

estimates, it would be prudent to disclose all pertinent facts to avoid later questions about

completeness of the submission.

Auditors are also required to evaluate estimating systems of major Department of Defense

(DoD) contractors. This includes ensuring that, if parametric estimating procedures are part of the

estimating system, they are properly disclosed. Of key concern to the auditor in evaluating the

estimating systems' disclosure of parametric procedures are the following:

1) Do the procedures clearly establish guidelines for when parametric techniques would be

appropriate?

2) Are there guidelines to ensure the consistent application of estimating techniques?

3) Is there proper identification of sources of data and the estimating methods and the

rationale used in developing cost estimates?

4) Do the procedures ensure that relevant personnel have sufficient training, experience,

and guidance to perform estimating tasks in accordance with the contractor's established

procedures?

5) Is there internal review of and accountability for the adequacy of the parametric

estimating techniques, including the comparison of projected results to actual results

and an analysis of any differences?

DCAA believes parametric estimating approaches are acceptable when they are properly

implemented. Auditors encounter it most often as a technique used in conjunction with other

estimating methods. For example, parametrics are often used for estimating costs of scrap and other

such factors. The majority of the proposals audited are not developed solely based on parametric

estimating techniques. The use of parametric estimating is most appropriate in such circumstances

where historical data is not available as when the program is at the engineering concept stage, or

when no bill of materials exists and the program definition is unclear. Contractors with good

parametric cost estimating systems analyze their perspective proposal to determine the appropriate

estimating technique for each part of the work breakdown structure.


146

Observations and Suggestions

The contractor can consider some observations made by DCAA auditors as to the pitfalls

contractors fall victim to when employing parametric techniques. The first is when a contractor fails

to do a cost-benefit analysis before implementing an elaborate parametric estimating model. Key

questions for any contractor considering implementing a complex parametric model are:

1) How often can we reasonably expect to use it?

2) How much time can we expect to save?

3) What are the costs of maintaining the model?

4) Will the model produce the necessary precision?

Contractors should be satisfied that implementation and monitoring costs do not outweigh the

benefit of reduced estimating costs. Moreover, it is critical that the environment is appropriate for

the use of parametrics. It would not be prudent to rely exclusively on parametric techniques to

estimate costs when directly applicable historical cost data are available. Such is the case of

follow-on production for the same or similar hardware. Contractors manufacturing mature weapon

systems already have a record of the actual costs. The use of parametric estimating may be

appropriate for certain aspects of follow-on production, however, the contractor should disclose any

data that may have a significant impact on cost. The exclusive use of parametrics is generally not

appropriate for economic forecasting of such elements as labor and indirect cost rates. For

parametric estimates, the contractor must ensure that any changes in accounting practices are

accounted for in the estimate and that labor and indirect cost rates are appropriately applied.

Another problem encountered is contractors failing to properly disclose their parametric

estimating practices. Auditors have experienced instances where the first time a parametric model is

disclosed is during the evaluation of the proposal. This is often too late. As mentioned earlier, larger

DoD contractors have an obligation to disclose in writing their estimating procedures. Making

proper and timely disclosure will minimize problems and expedite the negotiation process.

Contractors can take the lead in helping to streamline the oversight process. In a few words,

they should practice self-governance! DCAA has been a leading proponent of the self-governance

program. Self-governance is intended to encourage contractors to establish and maintain good

systems of internal control in key areas, including estimating systems. It requires contractors to


147

provide their own oversight -- to detect system weaknesses and take corrective action as necessary.

This initiative recognizes that prudent contractors already have the means in place to ensure their

operations are efficient and cost effective.

DCAA has another initiative that contractors should consider as a part of streamlining the

oversight process. It is called "coordinated audit planning." DCAA defines coordinated audit

planning as a voluntary process wherein the DCAA auditor and the contractor's internal and external

auditors consider each other's work in determining the nature, timing, and extent of his or her own

auditing procedures. Coordinated audit planning considers the extent to which reliance can be placed

upon work performed by the other auditor to minimize duplication of audit effort. In addition, this

process strengthens the evaluation of internal control systems.

Understanding estimating systems controls, assessing risk, and transaction testing are

common objectives of DCAA, and the internal and external auditors associated with contractors.

This often results in duplicative audit procedures. In the coordinated audit planning conducted to

date, DCAA found that the instances of duplication are significant. DCAA auditors are more than

willing to rely on the work done by contractor internal and external auditors, providing DCAA has

the opportunity to evaluate and test their work.

Contractors are expected to establish and maintain reliable estimating systems. Departmental

procurement officials, Members of Congress, and the average American citizen hold Defense

contractors to high and exacting standards. The funds involved can be enormous.

The expectations are equally high for the auditor whose job it is to protect the taxpayer's

interest. The DCAA auditor must comply with the American Institute of Certified Public

Accountants' Statements on Auditing Standards and the GAO's Government Auditing Standards.

These standards require that the auditor be independent in fact and in appearance. They also require

the auditor exercise a healthy degree of professional skepticism. These requirements, however,

should not render the DCAA auditor and the contractor enemies. Such polarized and adversarial

relationships are dysfunctional and not in either party's best interest.

Both parties are taking significant actions to improve relationships. These changes are

producing a culture change that is very positive: positive because contractors are beginning to more

fully accept their responsibilities; positive because auditors are more effectively communicating audit

plans and objectives.


148

Defense procurement is taking on a new, streamlined look in the 1990s. Government and

industry are both concerned with quicker, less costly means of procuring goods and services. This is

one reason why parametric methods continue to stir up so much interest. Parametric techniques,

properly applied, can assist contractors and the government alike in streamlining the acquisition

process. In addition, the ability of the government to place greater reliance on contractor oversight of

contractor systems will also result in meeting procurement needs more timely.

Summary

In today's and tomorrow's procurement environment, a great challenge facing us all is the

development of a cooperative work climate conducive to quickly acquiring quality products and

services at fair and reasonable prices. New estimating techniques such as parametrics can cut

estimating costs.

Adequate estimating systems, fully supported and self-governed by industry, can cut audit

costs. Quick estimating and audit turnaround times can cut procurement costs. All of this, however,

requires communication and teamwork. Whether our buying effort is for innovative space equipment

or for recurring maintenance, we must all work to meet the challenge.

Mike Thibault's message to the parametric community is clear: Parametric techniques are

acceptable cost estimating methodologies given that the following five criteria exist: logical

relationships, verifiable data, a significant statistical relationship (correlation) exists, techniques

produce accurate predictions, and they are easy to monitor and support.

He goes on to say that the auditor needs to consider the adequacy of the parametrics cost

estimating system and related internal controls, which includes: the audit trail, sufficiency of

documentation, currency and sources of data, procedures for calibration and validation, and the

appropriateness of parametrics use. Since historical data is normally used as the basis for all

estimating, the auditor will use basic auditing techniques to verify that costs are current, accurate, and

complete. Mr. Thibault completely dispels the myth that the audit community would not accept the

use of appropriate parametric cost estimating techniques for firm business proposals.

DCAA's complete audit guidance is included in its Contract Audit Manual, Chapter 9-1000.

This chapter is included in its entirety in Appendix C.


149

ESTIMATING SYSTEM REVIEWS

As part of a regulatory oversight requirement, DCAA will periodically perform contractor

estimating system reviews. The intent is to review the requirements delineated in DFARS 215.811

and 252.215-7003. DFARS 215.811 requires all DoD contractors to have adequate estimating

systems, requires certain large businesses to disclose their estimating systems in writing, provides

guidelines concerning the characteristics of an adequate estimating system, and provides guidance for

team estimating system reviews. If a contractor is required to disclose their system, all significant

parametric cost estimating techniques need to be disclosed.

It is DoD policy that contractors have estimating systems that consistently produce well

supported proposals acceptable as a basis for negotiating fair and reasonable prices. Estimating

systems should be consistent and integrated with a contractor's related management systems, and be

subject to applicable financial control systems. To be considered adequate, an estimating system

must be established, maintained, reliable, and consistently applied. It must also produce verifiable,

supportable and documented cost estimates.

FAR 15.8 provides the criteria for submission of cost or pricing data. Although FAR 15.8

does not mention parametric cost estimating techniques, there are no restrictions that preclude the use

of these techniques for firm business proposal submissions. FAR 15.804-6, Table 15-2, does require

the offeror to submit any information reasonably required to explain the estimating process. This

means that the contractor should clearly describe all parametric cost estimating techniques and

provide support for the data used in those techniques.

The FAR 15.804-6, Table 15-2, also states that for material cost estimates, the contractor

shall provide a consolidated summary of individual material quantities (e.g., bill of material) included

in the various tasks being proposed. In some cases it is not feasible for a contractor to provide a

consolidated bill of material or parametrics results in a better and more supportable estimate. For

example, in the research and development phase of a program, the material requirements often times

have not yet been defined, and, as a result, the preparation of a consolidated priced bill of material is

often not possible. In these circumstances, parametric cost estimating techniques may provide for a

reasonable basis to estimate material costs in lieu of a consolidated priced bill of material.


150

In any case, when a offeror uses estimating techniques other than a consolidated priced bill of

material to estimate material costs, the offeror must adequately describe the techniques being used,

provide sufficient support to allow for an independent evaluation, and explain why the techniques

used are the best in the circumstances in order to comply with the material cost criteria included in

Table 15-2.

DFARS 215.811-70 delineates attributes of an adequate estimating system. These are:

(1) Establishes clear responsibility for the preparation, review, and approval of cost

estimates.

(2) Provides a written description of the organization and duties of personnel responsible

for ... contributing to the estimating process ...

(3) Ensures that relevant personnel have sufficient training, experience and guidance ...

(4) Identifies sources of data and the estimating methods and rationale used in developing

cost estimates.

(5) Provides for appropriate supervision ...

(6) Provides for consistent application of estimating techniques.

(7) Provides for detection and timely correction of errors.

(8) Protects against cost duplication and omissions.

(9) Provides for the use of historical experience, including vendor pricing information

where appropriate.

(10) Requires use of appropriate analytical methods.

(11) Integrates information available from other management systems as appropriate.

(12) Requires management review [of the estimating system]

(13) Provides for internal review of and accountability for the adequacy of the estimating

system, including the comparison of projected results to actual results and an analysis of

any differences.

(14) Provides procedures to update cost estimates in a timely manner.

(15) Addresses responsibility for review and analysis ... of subcontract prices.

No well supported and documented parametric estimating system should be threatened by

any audit requirements, FAR or DFAR characteristics, or any other compliance (Truth in Negotiation


151

Act) issue. A well calibrated and validated parametric estimating system will be compliant in all

respects. The DFARS goes on to list indicators of or conditions that may cause significant estimating

deficiencies (from 215.811-77). Three in particular stand out:

(1) Failure to ensure that relevant historical experience is available to and utilized by cost

estimators, as appropriate.

(2) Consistent absence of analytical support for significant proposed cost amounts.

(3) Excessive reliance on individual personal judgment where historical experience or

commonly used standards are available.

The DFARS emphasis on historical experience is particularly satisfied by well supported

parametric systems. After effective calibration activities, parametric estimates are developed that are

auditable, compliant with regulations, and suitable for the negotiation of fair and reasonable prices

between government and contractor.

FORWARD PRICING RATE AGREEMENTS (FPRAs)

An FPRA, as defined in FAR 15.801, is a written agreement negotiated between a contractor

and the government to make certain rates and factors available during a specified period for use in

pricing contracts or contract modifications. Such rates and factors represent reasonable projections

of specific costs that are not easily estimated for, identified with, or generated by a specific contract,

contract end item or task. On the other hand, a Forward Pricing Agreement (FPA) is a written

agreement between a DoD contracting office and a large volume contractor which sets forth a

methodology that the contractor agrees to follow when pricing items covered by the FPA. It differs

from an FPRA in that once established, the FPA may be used to determine the complete final price of

individual orders. A typical FPA, for example, may be established to cover and expedite the

acquisition of spares.

It is clear that any CER or parametric methodology could fall under the umbrella of an FPRA

or an FPA. A forward pricing factor is generally represented as a percentage or ratio that is applied

to an existing cost determination or estimate in order to arrive at another, usually related, cost

determination or estimate. A costing rate used to convert labor hours to dollars is just such a factor


152

and is truly a CER. Parametric costing is done all the time. An FPRA for a parametric model is a

natural next step.

Therefore, when repetitive use of single CERs or parametric cost models is envisioned by the

contractor, an FPRA should be established. Procedures for establishing such a FPRA should follow

the basic guidelines listed in FAR Part 15.809-3 and should include the following:

a. Timely submission by the contractor, at least yearly, of an adequately supported FPRA

proposal.

b. Requirement for clear definitions of all elements of cost (dependent variables) which

will be developed by the CERs included in the FPRA and the applicable bases

(independent variables). Additionally, the proposal must indicate the period of

applicability of the proposed CERs and the conditions under which the CERs will be

used in price proposals.

c. The proposed CER's or parametric cost models should be evaluated in terms of the ten

criteria listed in the Table VI-1. The answer to the majority of these questions should

be affirmative - to support the need for a CER or Cost Model.

d. Tracking of the negotiated CERs and parametric cost models included in the FPRA is

required to test their validity as estimating tools. Guidelines for monitoring FPRAs are

included in Part 15.809-5 and the DCAA Contract Audit Manual (Vol. 1, Chapter

9-200).

It is the contractor's responsibility to identify the rate(s) to be included under the umbrella of

the FPRA or FPA and to specify the latest data already submitted in accordance with the agreement.

All data submitted in connection with the agreement, updated as necessary, form a part of the total

data that the offeror certifies to be current, accurate, and complete at the time of agreement on price

for an initial contract or for a contract modification.


153

TABLE VI-I

CRITERIA QUESTIONS TO ANSWER

Cost Benefit Is it more cost beneficial to have a CER for the cost

element than to generate a discreet estimate?

Predicts Well Do you anticipate the CER or parametric cost model to be a

good predictor of actual and reasonable cost?

Supportability Is the data that supports the development of the CER

auditable and/or traceable? (This can be from official

accounting data or other supporting data (e.g.,

subcontractor files)

Usability Can the CER be applied within the bounds of the

Estimating System?

Customer Acceptance Is the CER accepted by the PCO, DPRO, the CER owner,

etc.?

Management Acceptance Does the CER satisfy the contractor's requirements?

Ownership Can a group/individual be identified who would provide a

discrete input if the element of cost was not estimated

within the CER?

Applicability What is the intent for application of this CER or parametric

cost model (e.g., changes, restructures, etc.)?

Effective Implementation Are the persons doing the modeling/ analysis sufficiently

trained and experienced to use the tools correctly?


154

WHAT TO LOOK FOR IN A PARAMETRIC ESTIMATE

With the proliferation of parametric cost estimating models and tools, both commercial

models and “home grown” versions, it is impossible to describe what to look for in every model

and cost parameter. However, some generalizations can be made. The following is a summary,

since most of this has already been covered in the Handbook.

As a cost estimator or analyst, knowing and understanding the audit guidance described in

this chapter is a good start. If you follow the steps delineated earlier in this chapter, the cost

analysis or audit battle is more than half won.

If, as an analyst, you are confronted with a parametric cost estimate, you should take a

few steps to ensure a fair review. These are :

♦ Understand the cost model used. Do not hesitate to ask questions and to consult with the

experts. Most commercial model builders welcome calls from users, analysts and

auditors. You do not have to have a user’s understanding of the model, just a general

knowledge about how it works. Remember, there’s no such thing as a stupid question.

♦ Review the program inputs to the model. Is the schedule correct? Does the WBS

adequately describe the product being estimated? Is anything missing? Are there

duplications? Does the WBS follow the statement of work?

♦ Review the technical inputs to the model with your own engineers or technical

department, and your program office. Check them for reasonableness and benchmark

them against your own experience and the experience of the resident experts.

♦ Understand the model’s cost drivers. Generally, there are a few select parameters that

are the predominate cost estimating factors that drive cost. Many of the others are just

“tweaks” to the model. Concentrate on the cost drivers when performing your analysis.

Your may wish to combine the other minor “environmental” factors together as one

factor.

♦ Be aware of the assumptions the cost estimator made when he/she built the model. Are

they reasonable? Would you have made the same assumptions? If not, why not?

♦ Be knowledgeable of the historical cost basis for the model, if any. Be sure to review the

source documentation. Be wary of any model used that has no basis in historical cost.


155

How was the data “tuned” or normalized? Were data outliers disregarded? If so, why?

Was the model calibrated? How was the calibration performed? Were any universal or

“standard” cost factors used? Would they be applicable in this case?

♦ Question how the future environment might be different from the historical case. Have

these differences been accounted for? Would you account for them differently? If so,

how?

♦ Is the estimated program expected to push the state of the technology (state of the art)?

How is this planned to be accomplished? How is it accounted for in the estimate? Is it

seriously considered, or glossed over? Advances in the state of the art normally have

significant impacts on the cost drivers. Does this show up in the parametric estimate, or

are the historical factors used without adjustment? It is important to use technical experts

on this item. For instance, if a contractor proposes using a technology that has yet to be

proven in the laboratory, program trouble may lie ahead and result in inadequate

estimates. Consult with your technical people on this and related technical concerns.

♦ Review the track record of the estimator or the estimating organization. What is their

past performance? Have their estimates been reliable?

♦ Understand the economics involved with the model. What are the effective costing rates

used? Are they reasonable? Do they reflect the organization and skill levels being

estimated?

♦ Identify what cost factors have been “tweaked” and why. Focus on the “big ticket” items.

Using expert opinion and “rules of thumb,” are any significant cost factors outside the

range of reasonableness? For instance, it is very easy to calibrate a software cost model’s

cost (hours or dollars) per line of software code. Is the CER reasonable for the estimate?

However, different models may define a software line of code differently. It is therefore

vital to understand the definitions used in the model you are evaluating. If the model

violates your concept of reasonableness, then question it.

♦ Last, don’t be intimidated. Most (especially commercial) models are quite user friendly

and the model builders are helpful. Other people have learned how to use them, and you

are as smart as they are. The most difficult part is learning the language and gaining the

experience over time to develop your skills.


156

Rules Of Thumb

Rules of thumb are general rules, CER’s or other cost factors that are widely used, but do

not apply precisely to and specific case. Each estimating organization will have its own peculiar

set. Most long time estimators can develop a rapid estimate (a ROM) based on their own

personal rules of thumb. These rules come from their own experience. Examples of rules of

thumb are the “default” factors that populate many parametric models including the commercial

ones. If no factor input is available, the model uses a universally derived factor - one taken from

a universal data base or expert opinion.

Although there is nothing wrong with the use of these rules per se, the danger is evident.

The rules are only norms or benchmarks, and they will never apply to a specific estimate. Their

value lies in the comparison of a factor in a model to the universal case. Too much deviation

should be investigated. Why is the actual case so different from the rule? Or, stated differently,

why doesn’t the rule apply here?

For instance, if you believe that a line of software code (for a particular type of software)

should take about one-half hour to write (on average), and the parametric model indicates two

hours, then some investigation is in order. Although either CER could be the correct one, there is

far too much deviation from the norm, and suspicions are aroused. However, if the model

indicates six tenths of an hour that compares to your one-half hour, the rule had performed a

“sanity” check for you. But the use of the rule does not preclude a thorough cost analysis if you

have identified a cost driver.

Rules of thumb are important, but they must be used wisely. Develop your own list. This

can be accomplished through experience, or by consulting with experts in the field. Many rules

exist embedded within commercial parametric models, and are available to the users and

modelers. In any event, it is important to have a list of benchmark factors. You’ll probably need

a different list for each organization you deal with. Organizational and product differences will

demand it. But remember that there are no shortcuts or magic formulas, so use your rules as a

quality check, or for quick, rough order of magnitude estimates.

An Example


157

What follows is an example of some of the above discussion. The example here uses the

software cost estimating model, REVIC.

The fundamental REVIC equation is: Man Months = A*(KDSI)B*ΠFi, where,

A and B are coefficients of several Software Development Modes (for instance,

embedded , organic, Ada);

KDSI is Thousand Delivered Source Instructions; and;

ΠFi is the product of various environmental factors the model uses to adjust the

cost estimates. These environmental factors include, for example: analyst and

programmer capabilities, applications experience, programming language, storage

constraints, requirements volatility, reliability requirements, data base size,

product complexity, the use of modern programming practices and software tools,

platform (airborne, space, etc.), schedule compression, etc. Each factor is given a

factor value from very low through nominal to high.

Suppose you were given the following cost estimate:

REVIC = IMS EP 1 = 3.312(DSI/1000)1.2

*ΠFi

IMS EP 1 = (3.312)(1337/1000)1.2

*.874 = 4.1MM

What would you do with it? The basis of estimate (BOE) tells you this: “The Integrated

Management System Evaluation Package for Release 1 (IMS EP 1) requires 1337 new lines of

embedded source code of new algorithms and uses complex interfaces. Our superior

programming staff using state-of-the-art software tools, modern programming practices, and

possessing significant application experience allows the product of the environmental factors to

be less (13%) than nominal.” What do you notice about this estimate?

First, this particular estimate is relatively small (4.1MM). You may wish to move on to

the more important cost issues. However, you decide to “spot check” this one. You know very

little about REVIC, so you turn to the resident expert for advice. He/She confirms that the

technical description in the BOE is correct, so the model’s values for A and B are the correct


158

ones. You make a quick calculation to compare the estimate with a rule of thumb supplied by

your resident expert.

4.1MM * 155hrs./MM ÷ 1337DSI = .475hrs/DSI

Your resident expert informs you that he believes this factor to be quite low, possibly two

times. He also indicates that the DSI number needs to be verified for accuracy, since it is a major

cost driver for this estimate. He is also suspicious of the “....less (13%) than nominal”

environmental factors product. (Nominal here he believes should equal at least 1.5 due to a

partial space platform involved.) Therefore 13% appears to be a mistake of some kind, but it will

have to be investigated. He also argues that since this is a competitive proposal that’s almost

80% software, all software estimates may be “aggressive,” and should be carefully reviewed.

The model does not appear to have been calibrated except for the vague references “....superior

programming staff....” in the BOE. The product of the environmental factors seems to be based

upon judgment, although likely an educated one.

You decide to carefully review the other REVIC cost estimates in the proposal, so you

borrow a REVIC User’s Guide in order to become more proficient.

Although this is a relatively easy example, it demonstrates the general idea of what to

look for in a parametric estimate.


159

CHAPTER VII

BUSINESS APPLICATIONS OF

PARAMETRIC ESTIMATING


160

CHAPTER VII

BUSINESS APPLICATIONS OF PARAMETRIC ESTIMATING

INTRODUCTION

Recent initiatives affecting government procurement practices have further stressed an

already fragile world-wide estimating capability. Competition, accuracy, flexibility, and

method-proven credibility dictate reformation in the estimating process, especially for new business

pursuits. Parametric estimating methods for planning and strategizing new business cost issues is a

critical element in the reform of traditional bottoms-up approaches.

A paper written by Roy Summers and Bruce Fad summarized here discusses the need for

parametric modeling in the new business development process. The arguments are based on

experience gained through use of parametrics in this type environment. The role of parametric

modeling is examined along with the presentation of a process in which parametrics and

bottoms-up estimating are complementary. Organizational conditions leading to effective use of

parametrics are also considered. The questions of job responsibilities, management interface, and

functional structures are treated from the practitioners' viewpoint. Finally, the issue of credibility

includes perspectives from both the using and auditing organizations, with emphasis on calibration

and the use of testable databases.

PARAMETRIC ESTIMATING IN NEW BUSINESS DEVELOPMENT

Why do programs die? Why do CEO's, Boards of Directors, DoD, or Congress kill or

cancel technically superior projects? There are many examples, and, generally, it is not due to

technical viability. Even though millions or even billions are lost, the program graveyard claims

many projects. Sommers and Fad suggest that these events happen partly because of an inattention

to future cost issues during program development. There is a need to recognize and react to help

solve the cost problems. Technical success can often mask future cost issues.


161

The authors propose that an organization responsible for costing use parametric techniques

early on, and that cost and technical objectives be treated together. Parametrics are perfect for cost

estimating during a program's conceptual phase, and the tools connect technical and cost parameters

together in a very effective way.

New business development actions can attack the problem. These actions include:

1) Parallel and coordinated development of technical and programmatic (cost, schedule and

quantity) concepts from requirements.

2) Iterative and continuous cost improvement

3) Develop cost targets (bogeys) as the result of 1 & 2, above.

4) A review/reconciliation process that focuses on cost/risk analysis.

The authors go on to suggest that organizations not put off the cost estimating activity, and

that cost analysts immediately begin to explore technical/cost alternatives. They also suggest that

the parametric function belong to an independent department and not reside within another function

outside the organization that owns new business development.

To effectively integrate parametrics into an organization, upper management needs to be

involved. Upper management involvement will insure the creation of a cooperative team spirit, and

more effective management and customer acceptance including estimate credibility.

ESTIMATING PRODUCTION BUYS USING PARAMETRICS

A parametric approach can be used to calibrate production cost and then estimate future

production buy(s) and options using a commercial model. The benefit here is a simpler cost

proposal with no bill of material and no labor hours roll-up. The model can be calibrated and

estimates derived at any level of the WBS, hence useful for spare parts pricing. In either case, the

CERs or cost models can be used as the basis of estimate, and delivered to a proposal pricing

system via a post processor, or a spread sheet. In either event, the cost estimating process is greatly

simplified, and resource economies of scale can be realized.

Example: Estimating Production Buys Using Parametrics


162

What follows is an example of the comparative or analogous estimating approach.

RFQ And Historical Data

PDQ Inc., has received an RFQ for 84 XYZ-L Systems to be built and delivered in 1995.

A 120-lot of this same system was delivered in 1993, so recent cost history exists. The cost

history yields the following data:

Manufacturing labor (Hands-on) of 42,000 hours divided into:

Plant A - 18,000 hours

Plant B - 12,000 hours

Plant C - 12,000 hours

Purchased parts: $9,000,000

Manufacturing Support Labor: 17,280 hours which was generated from 1 Mechanical Engineer, 1

Quality Engineer, and 1 Test Engineer at each of the three plant locations.

Engineering Support Labor: 7,680 hours generated by 2 Engineers at Plant A, and 1 Engineer

each at Plants B and C.

Tooling Costs: $15,000

Test Equipment Costs: $25,000

The Interview Process

How will the proposed 84-lot be different from the historical (baseline) 120-lot?

VECP - AB123 has altered the system configuration. Mechanical Engineering has

estimated the VECP will reduce the assembly time at Plant A by 5 hours per system.

Plant B is over-capacity so five of the parts in the system will be resourced to Plant C.

These parts are similar in assembly and take 3 hours each to build.

Part number XYZ456, previously assembled at Plant A, is being purchased from an

outside vendor. Assembly time for this part at Plant A was 4 hours. It is currently being

purchased for $200.


163

Preparation Of The Estimate

Manufacturing Labor Calculations:

Plant A: 150 Hours Historical Baseline unitized

-5 Hours Adjustment for VECP-AB123

-4 Hours Make to Buy Change for P/N XYZ456

141 Hours Proposed Unit Hours Estimate for 84-lot

Plant B: 100 Hours Historical Baseline unitized

-15 Hours* Five parts (@ 3 hours per part) resourced to Plant C

85 Hours Proposed Unit Hours Estimate for 84-lot

Plant C: 100 Hours Historical Baseline unitized

+17.5 Hours Five parts resourced from Plant B

117.5 Hours Proposed Unit Hours Estimate for 84-lot

* The 15 hours for the 5 parts were adjusted to account for performance differences between Plants B and C.

Performance factor (inverse of efficiency) at Plant B was 1.5. The current performance factor at Plant C is 1.75. Adjustment: 15 hours divided by 1.5 times 1.75 equals 17.5.

Purchased Parts Calculations:

$75,000 Baseline Costs unitized (purchased 50% in 1992 and 50% in 1993)

Material for 84-lot will be purchased in 1994. Escalation is 3% per year.

$39,784 Baseline (50%) escalated from 1992 to 1994

$38,625 Baseline (50%) escalated from 1993 to 1994

$200 Make to Buy Change for P/N XYZ456 (not in baseline)

$78,609 Proposed Unit Base Estimate for 84-lot

Manufacturing Support Labor Calculations:

Support labor has historically followed a semi-variable pattern; i.e., when the production

quantity is halved, support can be reduced by 25%, or when production is doubled, support cost

is increased by 50%.


164

The proposed 84-lot is 60% of a halving of the baseline 120-lot. Therefore, the support

labor adjustment is -15% (60% X 25% = 15%) .

9.00 People Baseline

7.65 People Baseline Adjusted (-15%)

14,688 Hours Proposed support labor @ 1920 hours per person/year

Engineering Support Labor Calculations:

4 People Baseline

3 People Adjusted Baseline (Management Challenge)

5,760 Hours Proposed engineering labor @ 1920 hours per person/year

Tooling And Test Equipment Calculations:

$40,000 Baseline costs

$ 1,200 Adjusted for escalation from 1994 to 1995 (3% per year)

$ 4,120 Adjusted for age of equipment

$45,320 Proposed tooling and test equipment costs

Summary Of Direct Costs

Manufacturing Labor:

Plant A 141.0 Hours X 84 Systems X $75.80 per Hr. = $ 897,775

Plant B 85.0 Hours X 84 Systems X $81.50 per Hr. = $ 581,910

Plant C 117.5 Hours X 84 Systems X $77.10 per Hr. = $ 760,977

Subtotal Manufacturing Labor = $2,240,662

Purchased Parts:

Base $78,609 X 84 Systems = $6,603,156

Normal Material Production Allowance (NMPA) 5.0% = $330,158


165

Raw Material 2.5% = $165,079

Material Overhead (Applied to Purchase Parts Only) 8.0% = $528,252

Sub-Total Purchased Parts: = $7,626,645

Manufacturing Support: 14,688 Hours X $74.75 per Hr = $1,097,928

Engineering Support: 5,760 Hours X $83.00 per Hr. = $478,080

Tooling & Test Equipment: = $45,320

TOTAL DIRECT COSTS: $11,488,635

ESTIMATING SPARES AND CHANGE ORDERS

Parametric techniques can sometimes be used for estimating spares and change order

proposals. Using a calibrated system model for spares, select the WBS element that represents the

spare part(s) that need to be estimated. Then exercise the model at that point, either a commercial

version or one developed for the specific program. The estimate can be exercised very quickly, and

the calibrated model will yield accurate results.

Engineering Change Proposals (VECPs) or other change proposals have to be accounted for

by adjusting the actuals or a delta to the current estimate. The comparative approach is normally

well supported by history, as accurate as any other technique. It is much easier to do "what if"

analysis and compute option prices, and the proposals are less complex and smaller in volume and

more "user friendly", with significantly reduced preparation, review, audit and negotiation time.

Table VII-1 indicates the potential parametric applicability for various firm business

proposals. Appendix G contains a parametric estimating system checklist.

PARAMETRIC ESTIMATING APPLICATIONS FOR FIRM - BUSINESS PROPOSALS


166

Type of Proposal Engineering Manufacturing Test Support

1. New Business Development VA VA VA VA

2. Production VA VA VA VA

3. Follow-on Production VA VA VA VA

4. Development Contract Change-Order NVA NVA NVA VA

5. Production Contract Change-Order NVA NVA NVA VA

6. Spares MBA MBA MBA VA

NOTE:

Use of parametric estimating may be very applicable for new business development proposals/RFPs; their application may not be as good for unique

development contract change orders with little, if any, relevant history to use for CER development or Cost Model calibration. In addition, change order

data is collected at too low a level in the WBS for meaningful utilization.

KEY:

VA = Very Applicable; NVA = Not Very Applicable; MBA = May Be Applicable

TABLE VII - 1


167

APPENDIX A

DEFINITIONS OF ESTIMATING TERMINOLOGY


168

APPENDIX A

DEFINITIONS OF ESTIMATING TERMINOLOGY

Algorithmic Models - (also known as Parametric models) produce a cost estimate using one or

more mathematical algorithms using a number of variables considered to be the major cost

drivers. These models estimate effort or cost based primarily on the Hardware/Software size, and

other productivity factors known as cost driver attributes.

Analogy - A method of estimating developed on the basis of similarities between two or more

programs, systems, items, etc.

Analogy Models - use a method of estimating that involves comparing a proposed project with

one or more similar and completed projects where costs and schedules are known. Then,

extrapolating from the actual costs of completed projects, the model(s) estimates the cost of a

proposed project.

Analometric - A recent term meaning a method of combining the analogy and parametric

estimating methods to form a cost estimate when only 2 relevant data points are available. It is

usually combined with the "D" Factor to adjust (up or down) complexity from the 2 point CER

regression line.

Analysis - decision making context involving time horizons extending into the future. A

concrete set of specifications are not usually available. There could be many major uncertainties,

and a wide range of alternatives, each having several configurations. Analysis is usually

concerned with new equipment proposals and new methods of operation of systems never

produced before. Analysis Objectives are to find significant differences in resource requirements

among alternatives; and, how will resource requirements for any alternative change as key


169

configuration characteristics vary over their relevant ranges. It is often a "sensitivity" type of

investigation.

Analysis (NES Dictionary) - a systematic approach to problem solving. Complex problems are

simplified by separating them into more understandable elements.

Annual Change Traffic (ACT) - the fraction of a software product's source instructions which

undergoes change during a year, either through addition or modification. The ACT is the

quantity used to determine the product size for software maintenance effort estimation.

Anomalies - variances in cost related data caused by an unusual event(s) not expected to recur in

the future.

As Spent Dollars - the cost, in real year dollars, of a project recorded as the dollars were spent

without normalization for inflation.

Audit - the systematic examination of records and documents and the securing of evidence by

confirmation, physical inspection, or examination.

Audit Trail - information allowing the data being used in an estimate to be tracked back to the

original source for verification.

Benefit (NES Dictionary) - Result(s) attained in terms of the goal or objective rather than in

terms of output.

Benefit Cost Analysis (NES Dictionary) - an analytical approach to solving problems of choice.

It requires: (a) the definition of objectives; (b) identification of alternative ways of achieving

each objective; and (c) the identification for each objective or alternative which yields the

required level of benefits at the lowest cost. It is often referred to as cost-effectiveness analysis

when the benefits of the alternatives cannot be quantified in terms of dollars.


170

Benefits (NES Dictionary) - advantages which may be quantifiable or non-quantifiable. For

example, lowest cost is a benefit, so is the best accuracy and longest range.

Bottom-Up Models - use a method of estimation that estimates each component of the project

separately, and the results are combined ("Rolled Up") to produce an estimate of the entire

project.

Calibration - in terms of a cost model, a technique used to allow application of a general model

to a specific set of data. This is accomplished by calculating adjustment factor(s) to compensate

for differences between the referenced historical costs and the costs predicted by the cost model

using default values.

Computer-Aided Software Engineering: CASE - identifies a sector of the computer software

industry concerned with producing software development environments and tools. The main

components of a CASE product are individual tools which aid the software developer or project

manager during one or more phases of software development (or maintenance). Other features

are a common user interface; interoperability of tools; and a repository or encyclopedia to

provide a common tool base and central project database. CASE may also provide for code

generation.

Configuration Item - hardware or software, or an aggregate of both, which is designated by the

project configuration manager (or contracting agency) for configuration management.

Configuration Management - a discipline applying technical and administrative controls to (1)

identification and documentation of physical and functional characteristics of configuration

items; (2) any changes to characteristics of those configuration items; and (3) recording and

reporting of change processing and implementation of the system.


171

Constant Dollars - Computed values which remove the effect of price changes over time

(inflation). An estimate is said to be in constant dollars if costs for all work are adjusted to

reflect the level of prices of a base year.

Contract Work Breakdown Structure - Contract work breakdown structure is defined as the

complete work breakdown structure for a contract, i.e., the DoD approved work breakdown

structure for reporting purposes and its discretionary extension to the lower levels by the

contractor, in accordance with this standard and the contract work statement.

Cost (Fisher) - "Economic costs" are benefits lost. It is for this reason that economic costs are

often referred to as "alternative costs" or "opportunity costs." It is in alternatives, it is in

foregone opportunities, that the real meaning of "cost" must always be found. The only reason

you hesitate to spend a dollar, incidentally, is because of the alternative things that it could buy.

Some use the word "cost" when referring to resources. Cost of something is measured by the

resources used to attain it. Cost of attaining an objective at some point in time is measured by

the resources not available for use in attaining alternative objectives. Costs are a measure of

other defense capabilities foregone. Money cost is not necessarily the same as economic cost.

"Economic cost" implies the use of resources - manpower, raw materials, etc. Dollars are used

merely as a convenient common denominator for aggregating numerous heterogeneous physical

quantities into meaningful "packages" for purposes of analysis and decision mating.

Cost (NES Dictionary) - the amount paid or payable for the acquisition of materials, property, or

services. In contract and proposal usage, denotes dollars and amounts exclusive of fee or profit.

Also used with descriptive adjectives such as “acquisition cost," or "product cost," etc. Although

dollars normally are used as the unit of measure, the broad definition of cost equates to economic

resources, i.e., manpower, equipment, real facilities, supplies, and all other resources necessary

for weapon, project, program, or agency support systems and activities.

Cost Analysis (NES Dictionary) - the accumulation and analysis of actual costs, statistical data,

and other information on current and completed contracts or groups of contracts (programs).


172

Cost analysis also includes the extrapolation of these cost data to completion, comparisons and

analyses of these data, and cost extrapolations on a current contract with the cost data in the

contract value for reports to customers, program and functional managers, and price estimators.

In the procurement organizations of the Department of Defense, cost analysis is the review and

evaluation of a contractor's cost or pricing data and of the judgmental factors applied projecting

from the data to the estimated costs, in order to form an opinion on the degree to which the

contractor's proposed costs represent what performance of the contract should cost, assuming

reasonable economy and efficiency.

Cost Analysis (Large) - the primary purpose of cost analysis is comparison - to provide estimates

of the comparative or relative costs of competing systems, not to forecast precisely accurate costs

suitable for budget administration. In this context consistency of method is just as important,

perhaps more so, as accuracy in some absolute sense. In comparing the costs of military systems,

we prefer to speak of "cost analysis" rather than "cost estimation," because the identification of

the appropriate elements of cost -- the analytical breakdown of many complex interrelated

activities and equipment -- is so important a part of the method. Weapon system cost analysis is

much more than an estimate of the cost of the weapon itself. Weapon procurement costs may be

relatively small compared to other necessary costs, such as base facilities, training of personnel,

and operating expenses; and these other costs may vary greatly from system to system.

Cost Benefit Analysis (NES Dictionary) - a technique for assessing the range of costs and

benefits associated with a given option, usually to determine feasibility. Costs are generally in

monetary terms.

Cost Driver Attributes - productivity factors in the software product development process that

include software product attributes, computer attributes, personnel attributes, and project

attributes.


173

Cost Drivers - The controllable system design or planning characteristics that have a

predominant effect on the system's costs. Those few items, using Pareto's law, that have the most

significant cost impact.

Cost Effectiveness (NES Dictionary) - the measure of the benefits to be derived from a system

with cost as a primary or one of the primary measures.

Cost Effectiveness Analysis (NES Dictionary) - a method for examining alternative means of

accomplishing a desired military objective/mission for the purpose of selecting weapons and

forces which will provide the greatest military effectiveness for the cost.

Cost Estimating (Fisher) - "making an estimate" of the cost of something implies taking a rather

detailed set of specifications and "pricing them out."

Cost Estimating (NES Dictionary) - cost and price estimating is defined as the art of

predetermining the lowest realistic cost and price of an item or activity which assure a normal

profit.

Cost Estimating Relationship - An algorithm relating the cost of an element to physical or

functional characteristics of that cost element or a separate cost element; or relating the cost of

one cost element to the cost of another element.

Cost Estimating Relationships: CER - a mathematical expression which describes, for

predicative purposes, the cost of an item or activity as a function of one or more independent

variables.

Cost Factor - a brief arithmetic expression wherein cost is determined by the application of a

factor as a proportion.


174

Cost Model - an estimating tool consisting of one or more cost estimating relationships,

estimating methodologies, or estimating techniques used to predict the cost of a system or one of

its lower level elements.

Cost/Schedule Control System Criteria: C/SCSC - a set of criteria specified by the Federal

Government for reporting project schedule and financial information.

Current Dollars - Level of costs in the year actual cost will be incurred. When prior costs are

stated in current dollars, the figures given are the actual amounts paid. When future costs are

stated in current dollars, the figures given are the actual amounts expected to be paid including

any amount due to future price changes.

Deflators - The de-escalation factors used to adjust current cost/price to an earlier base year for

comparison purposes. A deflator is the inverse of an escalator.

Delivered Source Instructions: DSIs - the number of source lines of code developed by the

project. The number of DSIs is the primary input to many software cost estimating tools. The

term DELIVERED is generally meant to exclude non-delivered support software such as test

drivers. The term SOURCE INSTRUCTIONS includes all program instructions created by

project personnel and proceed into machine code by some combination of preprocessors,

compilers, and assemblers. It excludes comments and unmodified utility software. It includes

job control language, format statements, and data declarations.

Delphi Technique - a group forecasting technique, generally used for future events such as

technological developments, that uses estimates from experts and feedback summaries of these

estimates for additional estimates by these experts until a reasonable consensus occurs. It has

been used in various software cost-estimating activities, including estimation of factors

influencing software costs.


175

Detail Estimating: Grass Roots, Bottom-Up - The logical buildup of estimated hours and

material by use of blue-prints, production planning tickets, or other data whereby each operation

is assigned a time value.

Design to Cost: DTC - An acquisition management technique to achieve defense system designs

that meet stated cost requirements. Cost is addressed on a continuing basis as part of a system's

development and production process. The technique embodies early establishment of realistic

but rigorous cost objectives, goals, and olds and thresholds and a determined effort to achieve

them. Cost, as a key design parameter, is addressed on a continuing basis and as an inherent part

of the development and production process. Cost goals are established early in the development

phase, which reflect the best balance among life cycle cost, acceptable performance and schedule.

DOD-STD-2167A/498 - a US Department of Defense standard that specifies the overall process

for the development and documentation of mission-critical software systems. DOD-STD-

2167A/498 has direct connections to five other standards that are not software specific. These

five standards are as follows:

MIL-STD-1521 Technical Reviews and Audit for Systems, Equipments, and

Computer Software

MIL-STD-480 Configuration Control - Engineering Changes, Deviations, and

Waivers

MIL-STD-481 Configuration Control - Engineering Changes, Deviations, and

Waivers (Short Form)

MIL-STD-490 Specification Practices

MIL-STD-499 Engineering Management

MIL-STD-480 and 481 have been superseded by MIL-STD-973. MIL-STD-973

consolidates several earlier standards for configuration management and resolves inconsistencies

and ambiguities that existed between them. When DOD-STD-2167A/498 is applied to contract,

appropriate tailoring instructions should be included to indicate that the contract is to comply

with the more recent MIL-STD-973.


176

Domain - a specific phase or area of the software life cycle in which a developer works.

Domains define developers and users areas of responsibility and the scope of possible

relationships between products. The work can be organized by domains such as Software

Engineering Environments, Documentation, Project Management, etc.

DTC Goals - A cost bogey, in constant dollars, based upon a specified production quantity and

rate, established early during system development as a management objective and design

parameter for subsequent phases of the acquisition cycle. (see DTC Target).

DTC Parameter - Approved measurable values for selected cost elements established during

system development as design considerations and management objectives for subsequent life

cycle phases. A DTC parameter may be an objective, goal, or threshold. Values will be

expressed it constant dollars, resources required, or other measurable factor(s) that influences

costs.

DTC Targets - Approved DTC parameters divided into smaller. identifiable tasks or areas of

responsibility that serve as requisites for contractors or government activities.

DTC Threshold - Costs or values that, if exceeded, will cause a program review.

Economics (Hitch and McRean) - economics is concerned with allocating resources.

Economizing always means trying to make the most efficient use of the resources available.

Economics Analysis (Hitch and McRean) - economic analysis - ranging from just straight

thinking about alternative courses of action to systematic quantitative comparisons, to identify or

achieve near "optimal" solutions.

Economic Analysis (DoDI 7041.3) - a systematic approach to the problem of choosing how to

employ scarce resources and an investigation of the full implications of achieving a given

objective in the most efficient and effective manner.


177

Economic Analysis (NES Dictionary) - a systematic approach to a given problem, designed to

assist the manager in solving a problem of choice. The full problem is investigated; objectives

and alternatives are searched out and compared in light of their benefits and costs through the use

of an appropriate analytical framework. Often used to determine the best use of scarce resources.

Effectiveness (DoDI 7041.3) - the performance or output received from an approach or program.

Effort Adjustment Factor: EAF - a term used in COCOMO to calculate the cost driver

attribute's effect on the project. It is the product of the effort multipliers corresponding to each of

the cost drivers for the project.

Embedded Node - a term used by COCOMO to describe a project development that is

characterized by tight, inflexible constraints and interface requirements. The product must

operate within (is embedded in) a strongly coupled complex of hardware, software, regulations

and operational procedures. An embedded mode project will require a great deal of innovation.

An example would be a real-time system with timing constraints and customized hardware.

Equivalent Units - the number, or fraction, of completed units at a given time.

Estimating (NES Dictionary) - to predict costs. Generation of detailed and realistic forecasts of

hours, material costs, or other requirements for a task, subtask, operation, or a part or groups

thereof - generally in response to a request for proposal or specific statement of work. The art of

approximating the probable worth (or cost), extent, quantity, quality, or character of something

based on information available at the time. It also covers the generation of forecasted costing

rates and factors with which estimate numbers may be converted to costs, and the application of

these costing rates and factors to "estimate numbers" to establish forecasted costs.

Expert Judgment Models - use a method of software estimation that is based on consultation

with one or more experts that have experience with similar projects. An expert-consensus

mechanism such as the DELPHI TECHNIQUE may be used to produce the estimate.


178

Fixed Year Dollars - dollars that have been adjusted for inflation to a specific year.

Historical Data - a term used to describe a set of data reflecting actual cost or past experience of

a product or process.

Improvement Curve - a graphical representation of improvement curve theory that projects

resource requirements versus the number of units produced. Examples include labor hours, labor

cost, direct cost including labor and materials. When reflecting only labor hours the curve is

usually called a learning curve. When depicting cost, it is normally referred to as a Cost

Improvement Curve.

Inflation - an increase in the level of prices for the same item(s). Examples of indices that

reflect inflation include Consumer Price Index (CPI), Wholesale Price Index, and Producer Price

Index. When prices decline it is called deflation.

Knowledge Base - the repository of knowledge in a computer system or organization. The

collection of data, rules, and processes that are used to control a system, especially one using

artificial intelligence or expert system methods.

Life Cycle - the stages and process through which hardware or software passes during its

development and operational use. The useful life of a system. Its length depends on the nature

and volatility of the business, as well as the software development tools used to generate the

databases and applications.

Management Information Systems - a computer based system of processing and organizing

information that provides different levels of management within an organization with accurate

and timely information needed for supervising activities, tracking progress, making decisions,

and isolating and solving problems.


179

Metric - Quantitative analysis values calculated according to a precise definition and used to

establish comparative aspects of development progress, quality assessment or choice of options.

Normalize - to adjust data (normally cost data) for effects like inflation, anomalies, seasonal

patterns, technology changes, accounting system changes, reorganizations, etc.

Operation (Philip Morse: Notes on Operations Research, MIT Press 1962) - an operation is a

pattern of activity of men or of men and machines, engaged in carrying out a cooperative and

usually repetitive task, with pre-set goals and according to specified rules of operation.

Operational Research (Operational Research Quarterly, Vol. 13, No. 3, 282 (Sept. 1962)) -

operational research is the attack of modern science on complex problems arising in the direction

and management of large systems of men, machines, materials, and money in industry, business,

government, and defense. The distinctive approach is to develop a scientific model of the

system, incorporating measurements of factors such as chance and risk, with which to predict and

compare the outcomes of alternative decisions, strategies, or controls. The purpose is to help

management determine its policies and actions scientifically.

Operations Research (Norse, same as above) - the scientific study of operations.

Organic Mode - a term used by COCOMO to describe a project that is developed in a familiar,

stable environment. The product is similar to previously developed products. Most people

connected with the project have extensive experience in working with related systems and have a

thorough understanding of the project. The project contains a minimum of innovative data

processing architectures or algorithms. The product requires little innovation and is relatively

small, rarely greater than 50,000 DSIs.

Paradigm - a model, example, or pattern. A generally accepted way of thinking.


180

Parametric Cost Estimate - Estimate derived from statistical correlation of historic system costs

with performance and/or physical attributes of the system.

Parametric Cost Estimating Methods - Estimating methods based on physical or performance

characteristics and schedules of the end items.

Parametric Cost Model - A mathematical representation of parametric cost estimating

relationships that provides a logical and predictable correlation between the physical or

functional characteristics of a system, and the resultant cost of the system. A parametric cost

model is an estimating system comprising cost estimating relationships (CERs) and other

parametric estimating functions, e.g., cost quantity relationships, inflation factors, staff skills,

schedules, etc. Parametric cost models yield product or service costs at designated levels and

may provide departmentalized breakdown of generic cost elements. A parametric cost model

provides a logical and repeatable relationship between input variables and resultant costs.

Parametric Estimating - a mathematical procedure where product or service descriptors

(parameters) and cost algorithms directly yield consistent cost information.

Platform - hardware or software architecture of a particular model or family of computers. The

term sometimes refers to the hardware and its operating system.

Procedures - manual procedures are human tasks. Machine procedures are lists of routines or

programs to be executed, such as described by the Job Control Language (JCL) in a mini or

mainframe, or the batch processing language in a personal computer.

Process - the sequence of activities (in software development) described in terms of the user

roles, user tasks, rules, events, work products, resource use, and the relationships between them.

It may include the specific design methodology, language, documentation standards, etc.

Production Rate - the number of items produced in a given time period (e.g., items/month).


181

Program Evaluation (DoDI 7041.3) - is the economic analysis of on-going actions to determine

how best to improve an approved program/project based on actual performance. Program

evaluation studies entail a comparison of actual performance with the approved program/project.

Program Evaluation and Review Technique: PERT - a method used to size a software or

hardware product and calculate the Standard Deviation (SD) for risk assessment. For example,

the PERT equation (beta distribution) estimates the Equivalent Delivered Source Instructions

(EDBIs) and the SD based on the analyst's estimates of the lowest possible size, the most likely

size, and the highest possible size of each Computer Program Component (CPC).

Program Work Breakdown Structure - A program work breakdown structure covers the entire

acquisition cycle of a program, consists of at least the first three levels of a work breakdown

structure, as prescribed by MIL-STD-881B, and its extension by the DoD Component and/or

contractor(s). The program work breakdown structure has uniform element terminology,

definition, and placement in the family-tree structure.

Rapid Prototyping - the creation of a working model of a software module to demonstrate the

feasibility of the function. The prototype is later refined for inclusion in a final product.

Rayleigh Distribution - a curve that yields a good approximation to the actual labor curves on

software projects.

Real-Time - (1) Immediate response. The term may refer to fast transaction processing systems

in business; however, it is normally used to refer to process control applications. For example, in

avionics and space flight, real-time computers must respond instantly to signals sent to them.

(2) Any electronic operation that is performed in the same time frame as its real-world

counterpart. For example, it takes a fast computer to simulate complex, solid models moving on

screen at the same rate they move in the real world. Real-time video transmission produces a live

broadcast.


182

Re-engineering - process of restructuring and redesigning an operational (or coded) hardware or

software system or process in order to make it meet certain style, structure, or performance

standards.

Resource Analysis (Fisher) - systematically determining the economic resource impact of

alternative proposals for future courses of action.

Resources (Fisher) - raw materials, skilled personnel, scarce materials or chemicals, etc.

Resources (NES Dictionary) - consists of facilities, equipment, management, personnel,

laboratories, and scientific, technical, and manufacturing capability.

Reusability - ability to use all or the greater part of the same programming code or system design

in another application.

Reuse - software development technique that allows the design and construction of reusable

modules, objects, or units, that are stored in a library or database for future use in new

applications. Reuse can be applied to any methodology in the construction phase, but is most

effective when object oriented design methodologies are used.

Schedule - a time display of the milestone events and activities of a program or project.

Scope - a definition of how, when, where, and what a project is expected to include and

accomplish.

Security - the protection from accidental or malicious access, use, modification, destruction, or

disclosure. There are two aspects to security, confidentiality and integrity.

Semi-detached Mode - a term used by COCOMO to describe a project that is developed

somewhere between organic and embedded. The team members have a mixture of experienced


183

and inexperienced personnel. The software to be developed has some characteristics of both

organic and embedded modes. Semi-detached software can be as large as 300K DSIs.

Should Cost Estimate - An estimate of contract price which reflects reasonably achievable

economy and efficiency. It is generally accomplished by a team of procurement, contract

administration, cost analysis, audit and engineering representatives performing an in-depth

analysis of cost and cost effects. Its purpose is to develop realistic cost objectives.

Software Development Life Cycle - the stages and process through which software passes

during its development. This includes requirements definition, analysis, design, coding, testing,

and maintenance.

Software Development Life Cycle Methodology - application of methods, rules, and postulates

to the software development process to establish completeness criteria, assure an efficient

process, and develop a high quality product.

Software Method - (or Software Methodology) - focuses on how to navigate through each phase

of the software process model (determining data, control, or uses hierarchies; partitioning

functions; and allocating requirements) and how to represent phase products (structure charts;

stimulus-response threads; and state transition diagrams).

Software Tool - program that aids in the development of other software programs. It may assist

the programmer in the design, code, compile, link, edit, or debug phases.

Systems Analysis (Fisher) - Systems analysis may be defined as inquiry to assist decisionmakers

in choosing preferred future courses of action by (1) systematically examining and reexamining

the relevant objectives and the alternative policies or strategies for achieving them; and (2)

comparing quantitatively where possible the economic costs, effectiveness {benefits), and risks

of alternatives.


184

Technical Non-Cost Data - engineering variables that describe the item, subsystem or system.

The source of this data includes engineering documents, engineering drawings and works of a

technical nature which are specified to be delivered pursuant to the contract.

Top-Down Models - use a method of estimation that estimates the overall cost and effort of the

proposed project derived from global properties of the project. The total cost and schedule is

partitioned into components for planning purposes.

Update - to update an estimate or CER means to utilize the most recent data to make it current,

accurate and complete.

Validation - in terms of a cost model, a process used to determine whether the model selected

for a particular estimate is a reliable predictor of costs for the type of system being estimated.

Work Breakdown Structure - A work breakdown structure is a product-oriented family tree,

composed of hardware, software, services, data and facilities which results from system

engineering efforts during the development and production of a defense material item, and which

completely defines the program. A work breakdown structure displays and defines the product(s)

to be developed or produced and relates the elements of work to be accomplished to each other

and to the end product. MIL-STD 881B is the modern standard for developing a WBS. (See

Appendix B for more information.)

Workstation - high-performance, single user microcomputer or minicomputer that has been

specialized for graphics, CAD, CAE, or scientific application.


185

APPENDIX B

WORK BREAKDOWN STRUCTURE

BY:

NEIL F. ALBERT


186

DEVELOPING A USEABLE WORK BREAKDOWN STRUCTURE

Neil F. Albert

INTRODUCTION

This paper presents a guide for preparing, understanding and presenting a Work Breakdown

Structure (WBS). It discusses what determines a good work Breakdown Structure, provides a

general understanding for developing a program work breakdown structure, and shows how to

develop and implement a contract work breakdown structure. The primary objective of the paper is

to achieve consistent application of work breakdown structures using MIL-STD-881 as a

background source.

This paper is directed primarily at preparation of a defense program work breakdown structure.

However, the guidance is also appropriate for use with any work breakdown structure developed at

any phase during the acquisition process including concept exploration and definition,

demonstration and validation, engineering and manufacturing developments and production phases.

The guidelines are directed at both contractors and Government Activities in the development of

work breakdown structures for acquisition of defense material items.

DEFINITIONS

Several definitions are critical to the discussion in the following sections. These definitions are:

Program World Breakdown Structure.

The Program Work Breakdown Structure is the structure that encompasses an entire program

at a summary level. The program work breakdown structure consists of at least three levels of the

program with associated definitions and is used by the Government Activity and contractors to

develop and extend a Contract Work Breakdown Structure. The upper three levels of the program

work breakdown structure would typically be defined as follows:

Level 1: Level 1 is the entire defense material item; for example, electronic system refers to an

electronics capability such as a command and control system, radar system, communications

system, information system, sensor system, navigation/guidance system, electronic warfare

system, etc. Level 1 is usually directly identified as a major program or as a project or

subprogram within an aggregated program.

Level 2: Level 2 elements are major elements of the defense material item; for example, the

prime mission product which includes all hardware and software elements, aggregations of

system level services (e.g., system test and evaluation, and system engineering/program

management) and of data.

Level 3: Level 3 elements are elements subordinate to level 7 major elements; for example,

radar data processor, signal processor, antenna, type of service (e.g. development test and


187

evaluation, contractor technical support, training services), or types of data (e.g., technical

publications). Lower levels follow the same process.

Contract Work Breakdown Structure.

Contract work breakdown structure is the Government approved work breakdown structure for

reporting purposes and its discretionary extension to the lower levels by the contractor, in

accordance with Government direction and the contract work statement. It Includes all the

elements for the products (hardware, software, data, or services) which are the responsibility of the

contractor. The contract work statement will provide the reporting requirements for each contract

world breakdown structure element for which contract status is to be reported to the Government.

Normally, the contractor extends each level of the work breakdown structure to the point where

manageable size increments of work are defined. The contract work breakdown structure should

provide a consistent visible framework that facilitates uniform planning, assignment of

responsibility, and reporting of status.

Common WBS elements.

Common WBS elements are applicable to all types of systems. Common WBS elements

include:

- Integration, Assembly, Test and Checkout. Integration, assembly, test and checkout element

includes all effort required to assemble the level 3 equipment (hardware/software) elements

into a level 2 mission equipment (hardware software) as a whole and not directly part of any

other individual level 3 element.

- System Engineering/Program Management,

- System Test and Evaluation,

- Training,

- Data,

- Peculiar Support Equipment,

- Operational/Site Activation,

- Industrial Facilities, and

- Initial Spares and Repair Parts.

BACKGROUND

When the decision is made to develop and acquire a new or updated system, several factors are

considered when planning or monitoring efforts. One of these factors is determining the work

breakdown structure to use for the systems. A work breakdown structure is a product-oriented

family tree, composed of hardware, software, services, data and facilities, which results from

system engineering efforts during the development and production of a defense material item, and

which completely defines the program. A work breakdown structure displays and defines the

product(s) to be developed or produced and relates the elements of work to be accomplished to each

other and to the end product. Therefore, the work breakdown structure plays a significant role in

planning and assigning management and technical responsibilities; and monitoring and controlling

the progress and status of (a) engineering efforts, (b) resource allocations, (c) cost estimates, (d)

expenditures, and (e) Cost and technical performance. Seven defense material items are identified.


188

These are: Aircraft Systems Electronic/Automated Software Systems Missile Systems, Ordnance

Systems Ship Systems, Space Systems, and Surface Vehicle Systems.

Work Breakdown Structure Application

The work breakdown structure provides a framework for specifying the technical objectives of

the program by first defining the program in terms of hierarchically related product oriented

element and the work processes required for their completion. Each element of the work

breakdown structure provides logical summary points for assessing technical accomplishments, and

for measuring the cost and schedule performance accomplished in attaining the specified technical

objectives.

For each work breakdown structure element, the detailed technical objectives are defined as

well as the specific work tasks assigned to each contractor organization element and the resources,

materials and processes required to attain the objectives. As resources are employed and work

progresses on the task, current technical, schedule, and cost data are reported. The task data may

then be summarized to provide successive levels of management with the appropriate report on

planned, actual, and current projected status of the elements for which they are responsible.

Management will, thus, be better able to maintain visibility of status and to apply their efforts to

assure desired performance.

a. Technical Management

The work breakdown structure provides a framework for defining the technical objectives of the

program. Together with the contract statement of work and product specification, the work

breakdown structure aids in establishing a specification tree, defining configuration items, and

planning support tasks.

b. Contract Statement of Work

The statement of work (SOW) is the document which describes in clear understandable term

what products and services are to be delivered or services to be performed by the contractor.

Preparation of an effective SOW requires an understanding of both the products and services

that are needed to satisfy a particular requirement. A SOW prepared in explicit terms will

facilitate effective contractor evaluation after contract award. The SOW becomes the standard

for measuring contractor performance. Therefore, the SOW must clearly define the work to be

performed. In preparing the SOW for a system acquisition, the use of a standardized work

breakdown structure as a template for constructing the SOW will help streamline the process.

Use of the work breakdown structure will also provide the framework and facilitate a logical

arrangement of the SOW elements, provide a convenient checklist to ensure all necessary

elements of the program are addressed, and direct the contractor to meet specific contract

reporting needs.

c. Specification Tree

A specification tree, developed by system engineering, structures the Performance parameters

for the system or systems being developed. It subdivides the system(s) and its elements. The

performance characteristics are explicitly identified and quantified. Completed, the

specification tree represents a hierarchy of performance requirements for each component

element of the system for which design responsibility is assigned. Because specifications may


189

not be written for each product on the work breakdown structures the specification tree may not

match the work breakdown structure completely.

d. Configuration Management

Configuration management is the process of managing the technical configuration of items

being developed. In establishing, the requirement for configuration management on a program,

the Government Activity needs to designate which contract deliverable designated for

configuration management is called a Configuration Item. For software, this item is called

Computer Software Configuration Item (CSCI). Configuration management involves defining

the baseline configuration for the configuration items controlling the changes to that baseline,

and accounting for all approved changes. The frameworks for designating the configuration

items on a program is the work breakdown structure which needs to be extended sufficiently to

clearly define all elements subject to configuration management.

e. Financial Management

The work breakdown structure assists management in measuring cost and schedule

performance. By breaking the total product into successively smaller entities, management can

ensure that all required products are identified in terms of cost and schedule performance goals.

The planning of work by work breakdown structure elements serves as the basis for estimating

and scheduling resource requirements. The assignment of performance budgets to scheduled

segments of contract work identified to responsible organization units produces a time phased

plan against which actual performance can be compared and appropriate corrective action taken

when deviations from the plan are identified. This integrated approach to work planning also

simplifies the identification of potential cost and schedule impacts of proposed technical

changes.

f. Contract Budgeting

Funds management involves periodic comparison of actual costs with time phased budgets,

analysis of performance variances, and follow-up corrective actions as required. When work

breakdown structure product elements and the supporting work; are scheduled, a solid base for

time phases budgets is made. Assignment of planned resource cost estimates to scheduled

activities (tasks) and summarization by work breakdown structure element by time period

results in a time phased program/contract budget, which becomes the performance

measurement baseline.

g. Cost Estimating

Use of the work breakdown structure for cost estimating facilitates program and contract

management. The work breakdown structure helps the Government program office to plan,

coordinate, control and estimate the various program activities that the Government and the

contractors are conducting. It provides a common framework for tracking the estimated and

actual costs during the performance of each contract. The data from the various program

contracts support the Government program manager in evaluating contractor performance,

preparing budgets, and preparing program life-cycle costs e.g., as programs move through the

various phases of the acquisition process (conceptual design, development, and production), the


190

actual experience to date and the estimates for the remaining phases provides the basis for

reassessment of the total program costs.

h. Data Bases

Cost information accounted for by work breakdown structure element can be used for pricing

and negotiating contracts and contract changes, and for follow-on procurement. Over time, the

Government is accumulating a growing cost data base of similar work breakdown structure

elements from different programs. Such historical cost can be used to develop learning curves,

regression analyses and other techniques to estimate the cost requirements for like elements of

new programs. Actual cost data collected by the Government on each program can be

compared to the original estimates to identify trends, and establish validity of estimating

techniques. Contractors will similarly benefit from such data bases. Since contractors tend to

provide similar products on similar programs, the cost history accumulated on their programs

can assist them in bidding future contracts and budgeting new work.

Relationship To Other Contract Requirements

The work breakdown structure is the basis for communication throughout the acquisition

process. It provides the common link unifying the planning, scheduling, cost estimating, budgeting,

contracting, configuration management, and performance reporting disciplines. It is also the basis

used for contracts requiring cost/schedule reports placed on contract. This capability permits the

contractor to evaluate progress in terms of contract performance. Contract performance

assessments are made at the cost account level since the work breakdown structure facilitates the

summarization of data for successively higher levels of management.

WORK BREAKDOWN STRUCTURE DEVELOPMENT

Work is effort performed by people to transform or create products to solve identified problems

in order to verifiably meet specified objectives. Just as the organization hierarchically structures the

people who perform work, so the work breakdown structure hierarchically structures the products

to be produced and on which the people work. Examples of these products include equipment

(hardware/software), data, services and facilities for such systems as missile systems, helicopter

systems, automated systems, etc.

In order to use the work breakdown structure as a framework for structuring the technical

objectives of a program, in addition to its use as a management tool for cost and schedule control, it

is important that the work breakdown structure be product-oriented. It’s elements should represent

identifiable work products whether they be equipment (hardware, software) data or relatable service

products. Because any work breakdown structure is a product structure, not an organization

structure, complete definition of the effort encompasses the work to be performed by all

participants. Figure l provides an overview of the work breakdown structure development process.

Preparing a Program Work Breakdown Structure

The Government Activity is responsible for developing and maintaining the program work

breakdown structure. The Government Activity will structure a program work breakdown structure

for defense material items prior to program initiation. By selecting appropriate elements, the

Government Activity will be able to initially map its program work breakdown structure.


191

Figure 1


192

The program work breakdown structure should be developed in the conceptual stages of the

program. The program work breakdown structure evolves during conceptual design from an

iterative analysis of the program objective, functional design criteria, program scope, technical

performance requirements, proposed methods of performance, including procurement strategy, as

well as drawings, process flow charts, and other technical documentation. Before developing the

program work breakdown structure, the Government Activity must know the contractual structure

of the contract (e.g., the prime/subcontract relationship, make vs. buy plan, etc.). It is, therefore,

important that the development documentation detail the Government plan to build, integrate, and

field the system. Through this process the levels of reporting and elements for appropriate Request

for Proposal (RFP) selection are determined.

a. Program Work Breakdown Structure Element Selection Requirements

The program work breakdown structure elements must be selected by the Government Activity

and be structured in such a way that products and services may be readily summarized into the

program work breakdown structure. The program work breakdown structure and contract work

breakdown structure extensions will be used as a framework for technical and management

activities. The Government Activity will employ the program work; breakdown structure and

its contract work breakdown structure extensions as a coordinating medium in planning for

further system engineering, resource allocation, cost estimates contract actions, and work

execution. The reporting of progress, performance, and engineering evaluations, as well as

financial data, will be based on the program work breakdown structure. Figure II provides an

example of a top level program work breakdown structure.

b. Levels of Program Work Breakdown Structure

The program work breakdown structure contains at least the top three levels expanded to

identify elements with a significant degree of technical or cost risk. When program work

breakdown structure levels are stipulated to an excessively low level of a program, the

contractor's normal method of operation may be hampered; or excessive reporting requirements

may result. The SOW is the place to clearly communicate all program requirements. Figure III

provides an expanded program work breakdown structure which incorporates elements

necessary for contract visibility and control.

c. Considerations in Constructing a WBS

The following should be kept in mend when constructing a work breakdown:

(1) Many elements of a program are not products. A signal processor for example, is clearly a

product, as are mockups, and Computer Software Configuration Items (CSCIs). Design

engineering, requirements analysis, test engineering labor, aluminum, and direct costs, etc.,

are not products. Design engineering, test engineering, and requirements analysis are all

engineering functional efforts; aluminum is a material resource; and direct cost is an

accounting classification. As such, none of these elements are appropriate as work

breakdown structure elements.

(2) Program-phases (e.g., production), and types of funds (e.g., Research Development, Test

and Evaluation) are inappropriate elements of a work breakdown structure.


193

Figure 2


194

Figure 3


195

(3) Rework, retesting, and refurbishing should be treated as work on the appropriate work

breakdown structure element affected, not as separate elements of a work breakdown

structure.

(4) Non-recurring and recurring classifications are not work breakdown structure elements.

The government reporting requirements should require segregation of each work breakdown

structure element into its non-recurring and recurring parts.

(5) Cost saving efforts such as total quality management initiatives, should cost, warranty, etc.,

are not work breakdown structure elements. These efforts should be included in the cost of

the item they affect and not captured separately.

(6) The organization structure of the program office or the contractor should not be the basis for

development of a work breakdown structure. The work breakdown structure should always

retain its product orientation.

(7) Costs for meetings, travel, computer support, etc., arc to be included in the work breakdown

structure elements for which they are associated. They are not to be treated as separate

work breakdown structure elements.

(8) The use of generic term in a work breakdown structure is improper. The system(s) name

and/or nomenclature is appropriate. The work breakdown structure elements should be

clearly named to indicate the character of the product to avoid semantic confusion. For

example, if the level 1 system is Fire Control, then the Level 2 item (prime mission product)

is Fire Control Radar.

(9) Tooling (e.g., special test equipment, and factory support such as: assembly tools, dies, jigs,

fixtures, master forms, handling equipment, etc. should be included in the cost of the

equipment being produced. It is a functional cost, not a work breakdown structure element.

If the tooling cannot be assigned to an identified subsystem or component, it should be

included in the cost of integration, assembly, test, and checkout. Any additional quantities

produced for equipment support or maintenance in the field should be included and reported

under Peculiar Support Equipment. This same philosophy applies to software. For

example, when a software development facility/environment is created to support the

development of software, the cost associated with this element is considered part of the

CSCI it supports, or if more than one CSCI is involved, it should be included in integration,

assembly, test and checkout.

(10) The definition of integration, assembly, test, and checkout includes production acceptance

testing of R&D (including first article test) and production units, but excludes all system

engineering/program management, and system test and evaluation which are associated

with the overall system.

Figure IV provides and example of both a correct and an incorrect work breakdown structure.

e. Software in the Work Breakdown Structure

The importance of software in today's Government acquisition environment is growing. As a

result, software is identified in two ways for development of a work breakdown structure:

The first type of software is that which operates or runs on a specific piece of equipment, and

the second type of software is that which may be contracted for separately from the operating

equipment or is a stand alone (software intensive system). Software that is being developed

to reside on specific equipment must be identified as a subset of that equipment.


196

Figure 4


197

Multifunction software will be identified as a subset of the equipment work breakdown

structure element which either includes the software in the element specification or exercises

the most critical performance constraint. Figure V provides an example of how software

should be addressed as part of a specific equipment. In cases where the application of this

rule results in a conflict in the selection of the proper elements, the specification relationship

will take precedence. For example, an aircraft's electronic equipment typically has software

included in each of the subsystem elements. Software that resides and interfaces with more

than one equipment, i.e., application software, and overall system software which facilitates

the operation and maintenance of the computer systems and associated programs (e.g.,

operating systems, compilers, and utilities) will be called out at the appropriate level of the

program. (Ref. ANSI/IEEE STD 6 10. 19. 1990 for definitions of applications and system

software.)

It is incorrect to summarize all software on a program or contract in a work breakdown structure

(ref. Figure IV). By separating these elements from the hardware they support, performance

measurement and management control over each equipment is difficult to obtain. The true cost of

each equipment is not readily available for decisions concerning that equipment. Rather than

separately summarizing software, it is important to identify software with the hardware it supports.

(When needed, contractor's management systems can use an identifier for each software element to

produce internal summaries for software management purposes.) A separately contracted or stand

alone software will include the software, data, services, and facilities required to develop and

produce a software product for a command and control system, radar system, information system,

etc. Where software is considered stand alone (i.e., does not reside or support a specific equipment,

or is considered a pure software upgrade, etc.), the Government Activity should use the same

product-oriented work breakdown structure format. Figure VI provides an example of a work

breakdown structure for a stand alone software system.

f. WBS Dictionary - When developing a program work breakdown structure, the Government

Activity should also develop a WBS Dictionary. The program work breakdown structure

dictionary lists and defines the work breakdown structure elements. Although initially prepared

for the program work breakdown structure, it can be expanded in greater detail at lower levels

by contractors as they develop their contract work breakdown structure.

The dictionary lists elements to show their hierarchical relationship to each other, describes

each work breakdown structure element and the resources and processes required to produce it, and

provides a link to the detailed technical definition documents. The work breakdown structure

dictionary should be revised to reflect changes and should be maintained in a current status

throughout the life of the program.

Preparing a Contract Work Breakdown Structure

The individual work breakdown structure elements should be chosen from the program work

breakdown structure by the Government Activity for inclusion in a Request for Proposal (RFP).

This will be accomplished by selecting the appropriate program work breakdown structure elements

for the products that will be required by each contract. Contracts for WBS elements that are at level

3 or below in the Program Work Breakdown Structure will be moved to Level 2 and all


198

Figure 5


199

Figure 6


200

other applicable Level 2 Common WBS elements will be selected. The result is the contract work

breakdown structure. Figure VII depicts the development and relationship of the Program Work

Breakdown Structure with the Contract Work Breakdown Structure. Each RFP should include the

contract work breakdown structure and the initial WBS Dictionary prepared by the Government

Activity. The RFP should instruct potential contractors to extend the selected contract work

breakdown structure elements to define the complete contract scope.

a. RFP Solicitation Requirements

The contract line items, configuration item, contract work statement tasks, contract

specifications, and contractor responses will be expressed in term of the work; breakdown

structure to enhance its effectiveness in satisfying the objectives of the particular acquisition.

The Statement of Work; (SOW) tasks in the solicitation and final contract should be structured

to align with the contract work breakdown structure. It is important to develop the program

work breakdown structure with the development of the SOW so as to form consistency in

document structure. When aggregated with the program work breakdown structure, the

extended contract work breakdown structure will form a complete work breakdown structure

for that program which will be used throughout the acquisition cycle.

b. Extended Contract Work Breakdown Structure

The contractor extends the contract work breakdown structure in the RFP and submits the

complete contract work breakdown structure with its proposal. The proposal submitted should

be based on the work breakdown structure in the RFP. Contractors may suggest changes to the

selected contract work breakdown structure elements when a change is needed to meet an

essential requirement of the RFP or to enhance the effectiveness of the contract work

breakdown structure in satisfying the program objective. The contractor should extend the

contract work breakdown structure to the appropriate level which satisfies the critical visibility

requirements and does not overburden the contractor management system.

Implementation of Contract Work Breakdown Structure

When the Government selects the contractor, they negotiate the contract and the associated

contract work breakdown structure. The proposed contract work breakdown structure included in

the successful proposal serves as the basis for negotiating an approved contract work breakdown

structure with the Government. The contractor may have proposed alternate approaches to better

accomplish the contract objectives. The alternatives, if accepted by the Government Activity, may

impact the proposed program work breakdown structures. Revisions will be required to the

program work breakdown structure and the contract work breakdown structure to reflect these

changes. After adjustments and contract negotiations, the elements selected for the contract will

become the basis for contractor extension during the contracted effort. All extensions must sum to

the contract work breakdown structure reporting level in the contract.

a. Contract Work Breakdown Structure Approval and Contract Award

Following Government approval of the negotiated contract, including the contract work

breakdown structure, the Government awards the contract. The contract identifies the

requirement for providing the WBS Dictionary through the contract data requirements list

(CDRL). While strong, efforts should be placed on early and accurate work breakdown


201

Figure 7


202

structure planning, work breakdown structure revisions may result from expansion or

contraction of program/ contract scope and the movement of a program through its various

stages. Normally, changes to the work breakdown structure should not be made after contracts

are awarded and work is underway unless major rescoping of the program occurs. NOTE: The

sequence shown in the preceding paragraphs may be iterative as the program evolves, contracts

are awarded, and the work effort progresses through major program phases. Whenever the

work breakdown structure is revised, the ability to crosswalk and track back to the previous

work breakdown structure must be maintained.

b. Implementation with Subcontractors

Contractors may require the use of the work breakdown structure by subcontractors to permit

fulfillment of contractual requirements and provide adequate control of the subcontractor. Such

subcontractors, whose work accounts for a major segment of the subcontracted portion of the

prime contract, will be delineated in contracts at the time of award. It will be the prime or

associate contractor's responsibility to incorporate into the contract, with the affected

subcontractors, the work breakdown structure requirements.

c. Maintain Contract Work Breakdown Structure

The contract will indicate the levels of contract work breakdown structure at which costs will be

reported to the Government. Traceability of cost accumulations will be required to those

extended contract work breakdown structure levels which are used by the contractor for cost

control purposes. In the extended contract work breakdown structure, consideration will be

given to the specific contractual, technical, and managerial requirements of the defense material

item. The contractor has complete flexibility in extending the contract work breakdown

structure below the reporting requirement to reflect how work is to be accomplished, assuming

lower elements to be meaningful product-oriented lower indentures of a higher-level element.

Particular attention will be given to ensure the correlation of lower levels of the contract work

breakdown structure to the specification tree, contract line items, data items, and statement of

work tasks.

Relationship with Contractor Management System

As the end product is subdivided into smaller subproducts at lower work breakdown structure

levels, the work effort required by each element can be identified with functional organization units.

At some point within the work breakdown structure, the contractor will assign management

responsibility for technical, schedule, and cost performance. The responsible manager is called cost

account manager. The cost management system should provide the necessary visibility of the lower

levels of the work breakdown structure as it interfaces with the organization. At the juncture of the

work breakdown structure element and organization units cost accounts are usually established and

performance is planned, measured, recorded and controlled. To do so, the technical requirements

for the work and work product must be specified; the work scheduled, budgeted, and performed;

and product attainment of specified technical requirements verified.

a. Contractor Organizational Structure People performing work are organized to facilitate

effective management. Whether the organization is designed along program, function, natural

work teams or matrix lines, the organization structure reflects the way the people who will


203

accomplish the work have been organized. To assign specific work tasks, the organizational

structure must be linked effectively with the work breakdown structure. This linkage can occur

at any level of the work breakdown structure. Figure VIII depicts the linkage between the work

break-down structure and the contractor's organizational structure.

b. Cost Account Level -- To provide the responsible (cost account) manager with the technical,

schedule, and cost information needed to manage the organization’s work on the work

breakdown structure element for which he is responsible, the management control system must

be keyed to the same work breakdown structure element and organization unit. The appropriate

work breakdown structure level at which a cost account is established is purely a function of the

magnitude of the program and the type of product. The responsible organization level is a

function of the management span of control and upper management’s desire to delegate

technical, schedule, and cost responsibility for product/contract work breakdown structure

elements to lower management levels. In identifying cost accounts, the contractor must be

allowed to establish organizational responsibilities at meaningful and appropriate levels,

otherwise, the contractor's existing management control systems and responsibility assignments

may be affected adversely. For example, when software is a major component of cost and the

Government wants it identified separately, care must be taken to not unnecessarily complicate

the contractor work breakdown structure and contractor management system. To meet these

needs, special reporting requirements are specified in the SOW. In this example, Figure IX

shows how the cost management system with job coding (SW) and the work breakdown

structure can provide needed detail and visibility without extending the work breakdown

structure to extremely low levels.

Virtually all aspects of the contractor's management control system, including technical

definition, budgets, estimates, schedules, work assignments, accounting, progress assessments,

problem identification, and corrective actions, come together at the cost account. Performance

visibility is directly relatable to the level and content of the cost account.

SUMMARY

The development of any work breakdown structure is intended to achieve a clear understanding

and statement of the technical objectives and the end item(s) (or end product(s)) of the work to be

performed. The process of identifying these objectives assists in structuring the product elements

during the work breakdown structure development. Objectives derived from the overall program

objective are identified in such a way that identifiable products support economically and

technically identifiable subsystems of the program objectives. This process may be repeated until

the component level is reached. In this matter, subsystems support a total system capability.

When properly implemented, a good work breakdown structure can effectively satisfy the needs

of both the Government and contractor program managers. By coordinating the development of the

work breakdown structure, the statement of work the contract line item structure and the product

specification, cost and technical requirements and performance can be better translated, managed

and maintained. Without this generic framework, the acquisition process becomes difficult and

complicated.


204

Figure 8


205

Figure 9


206

Neil F. Albert is Director of the Cost Analysis Division at Management Consulting & Research,

Inc. (MCR), Falls Church, Virginia. He has over sixteen years of experience in directing and

performing cost estimating/analysis tasks. His experience includes engineering and Parametric

estimating, cost trade-off analysis, design-to-cost, and lifecycle cost analysis. He has held various

corporate positions as a consultant, senior manager and cost analyst including Manager of Cost

Analysis/Estimating for Textron Defense Systems. He holds an M.B.A. in Financial Management

and a B.A. in Finance. Mr. Albert is the Administrator for MIL-STD-881B Working Group. In this

capacity he was responsible for supporting the revision of MIL-STD-881A and for organizing and

generating MIL-STD-881B. Mr. Albert was a principal author of the work breakdown structure

User Guide which can be found in Appendix I of MIL-STD-881B.

REFERENCES

1. MIL-STD-881B (DRAFT), Work Breakdown Structures for Defense Material Items, 18

February 1992.

2. MIL-STD-881A, Work Breakdown Structures for Defense Material Items, 25 April l975.

3. DOE/MA-0295, Work Breakdown Structure Guide, February 6, 1987.

4. Contractor Cost Data Reporting (CCDR) System, 5 November 1973.

5. Cost/Schedule Control Systems Criteria Joint Implementation Guide , October 1987.

6. DI-A-3023, Contract Work Breakdown Structure, 21 May 1971.

7. MIL-STD-196D. Joint Electronics Type Designation System (JFTDS), l9 January 1985.

8. DOD-STD-2167A, Defense System Software Development, 29 February 1988.

9. ANSI/IEEE STD 610-1990, Software Engineering Terminology, 1990.

10. MIL-HDBK-171 (DRAFT), Work Breakdown Structures for Software Elements, 8 December

1992.


207

APPENDIX C

DCAA CAM 9-1000 SECTION 10

REVIEW OF PARAMETRIC COST ESTIMATES

NOTE: This is an excerpt from the Defense Contract Audit Agency Contract Audit Manual. The last page of this appendix includes an order

form to obtain the entire manual.


208

APPENDIX C

9-1000 SECTION 10 - REVIEW OF PARAMETRIC COST ESTIMATES

9-1001 INTRODUCTION

This section contains an overview and general guidance on reviewing cost-to-noncost estimating relationships, primarily in the context of contractor price proposals. This section also contains guidance on the use of estimating standards in price proposals. This supplementary guidance contains criteria contractors should meet before submitting proposals based on parametric cost estimates.

9-1002 PARAMETRIC COST ESTIMATING TERMINOLOGY

9-1002.1 Definition of Parametric Cost Estimating a. Parametric cost estimating ("parametrics") has been defined as a technique employing one or more cost estimating relationships (CERs) to estimate costs associated with the development manufacture, or modification of an end item. A CER expresses a quantifiable correlation between certain system costs and other system variables either of a cost or technical nature. CERs are said to represent the use of one or more independent variables to predict or estimate a dependent variable (cost). b. Parametrics encompasses even the simplest traditional arithmetic relationships among historical data such as simple factors or ratios used in estimating scrap costs. However, for audit purposes our guidance will limit special consideration of parametrics to more advanced or complex applications. These may involve extensive use of cost-to-noncost CERs, multiple independent variables related to a single cost effect, or independent variables defined in terms of weapon system performance or design characteristics rather than more discrete material requirements or production processes. EDP data bases and/or computer modeling may be used in these types of parametric cost estimating systems. c. Parametric estimating techniques may be used in conjunction with any of the following estimating methods: (1) Detailed—also known as the bottoms-up approach. This method divides proposals

into their smallest component tasks and are normally supported by detailed bills of material.

(2) Comparative—develops proposed costs using like items produced in the past as a baseline. Allowances are made for product dissimilarities and changes in such things as complexity, scale, design, and materials.

(3) Judgmental—subjective method of estimating costs using estimates of prior experience, judgment, memory, informal notes, and other data. It is typically used during the research and development phase when drawings have not yet been developed.

9-1002.2 Distinction Between Cost and Noncost Independent Variables


209

a. Although the basic criteria for cost-to-cost and cost-to-noncost CERs are generally comparable, the supplementary criteria in this section pertain to cost-to-noncost CERs. Audits of traditional cost-to-cost estimating rates and factors are covered in other sections of this chapter and in referenced appendixes. b. Cost-to-noncost CERs are CERs which use something other than cost or labor hours as the independent variable. Examples of noncost independent variables include end-item weight, performance requirements, density of electronic packaging, number or complexity of engineering drawings, production rates or constraints, and number of tools produced or retooled. CERs involving such variables, when significant, require that the accuracy and currency of the noncost variable data be audited. Special audit considerations are described in 9-1003 and 9-1004.

9-1002.3 Uses of Parametric Cost Estimates a. Parametric cost estimating is used by both contractors and government in planning, budgeting, and executing the acquisition process. Parametric cost models are generally made up of several CERs and can be used to estimate the costs for part of a proposal or the entire proposal. The cost models are often computerized and may be made up of both cost-to-cost and cost-to-noncost interrelated CERs. The guidance contained in this chapter is intended to assist in the review of parametric estimates, CERs, and/or cost models used in developing price proposals for negotiation of government contracts. b. Parametric cost estimates are often used to cross-check the reasonableness of estimates developed using other estimating methods. Generally, it would not be prudent to rely on parametric techniques based on a broad range of data points to estimate costs when directly applicable program or contract specific historical cost data is available, as in the case of follow-on production for the same hardware in the same plant. Nor would parametric techniques be appropriate for contract pricing of specific elements such as labor and indirect cost rates which require separate forecasting considerations such as time and place of contract performance. The use of a parametric estimating method is considered appropriate, for example, when the program is at the engineering concept stage and the program definition is unclear, or when no bill of materials exists. In such cases, the audit evaluation should determine that: (1) the parametric cost model was based on historical cost data and/or was calibrated to

that data, and (2) the contractor has demonstrated that the CER or cost model actually reflects or

replicates that data to a reasonable degree of accuracy.

9-1003 PARAMETRIC ESTIMATING CRITERIA FOR PRICE PROPOSALS

When a contractor uses parametric cost estimating techniques in a price proposal, the auditor will apply all pertinent criteria applicable to any proposal along with the supplemental criteria provided in 9-1004.

9-1003.1 Disclosure of Parametric Estimating Data a. The purpose of the Truth in Negotiations Act, 10 U.S.C. 2306(a), is to provide the government with all facts available to the contractor at the time of certification and that the cost or pricing data was current, complete, and accurate. Parametric estimates must meet the same basic disclosure requirements under the act as detailed estimates.


210

b. Although the principles are no different, proposals supported in whole or in part with parametric estimating will present new fact situations concerning cost or pricing data which is required to be submitted. A fundamental part of the definition of cost or pricing data is "all facts ... which prudent buyers and sellers would reasonably expect to have a significant effect on price negotiations" (FAR 15.801). Reasonable parallels may be drawn between the data examples provided in FAR for discrete estimating approaches and the type of data pertinent to parametric estimating approaches. For example, if a contractor uses a cost-to-noncost CER in developing an estimate, the data for the CER should be current, accurate, and complete. c. Many contractors use parametric cost estimating for supplementary support or for cross-checking estimates developed using other methods. Judgment is necessary in selecting the data to be used in developing the total cost estimate relied upon for the price proposal. In distinguishing between fact and judgment, FAR states the certificate of cost or pricing data "does not make representations as to the accuracy of the contractor's judgment on the estimated portion of future costs or projections. It does, however, apply to the data upon which the contractor's judgment is based" (FAR 15.804-4[b]). Therefore, if a contractor develops a proposal using both parametric data and discrete estimates, it would be prudent to disclose all pertinent facts to avoid later questions about full disclosure. d. The auditor should address the following questions during the evaluation of Parametric cost estimates: * Do the procedures clearly establish guidelines for when parametric techniques would be

appropriate? * Are there guidelines for the consistent application of estimating techniques? * Is there proper identification of sources of data and the estimating methods and

rationale used in developing cost estimates? * Do the procedures ensure that relevant personnel have sufficient training, experience,

and guidance to perform estimating tasks in accordance with the contractor's established procedures?

* Is there an internal review of and accountability for the adequacy of the estimating system, including the comparison of projected results to actual results and an analysis of any differences?

9-1004 SUPPLEMENTAL ESTIMATING CRITERIA

The auditor should also consider the following supplemental criteria when evaluating parametric cost estimates.

9-1004.1 Logical Relationships The contractor should demonstrate that the cost-to-noncost estimating relationships used are the most logical. A contractor should consider all reasonably logical estimating alternatives and not limit the analysis to the first apparent set of variables. When a contractor's analysis discloses multiple alternatives that appear logical, statistical testing (9-1004.3) of selected logical relationships may be used to provide the basis for choosing the best alternative.

9-1004.2 Verifiable Data


211

The contractor should demonstrate that data used for parametric cost estimating relationships can be verified. In many instances the auditor will not have previously evaluated the accuracy of noncost data used in parametric estimates. For monitoring and documenting noncost variables, contractors may have to modify existing information systems or develop new ones. Information that is adequate for day-to-day management needs may not be reliable enough for contract pricing. Data used in parametric estimates must be accurately and consistently available over a period of time and easily traced to or reconciled with source documentation.

9-1004.3 Statistical Validity The contractor should demonstrate that a significant statistical relationship exists among the variables used in a parametric cost estimating relationship. There are several statistical methods such as regression analysis that can be used to validate a cost estimating relationship, however, no single uniform test can be specified. Statistical testing may vary depending on an overall risk assessment and the unique nature of a contractor's parametric data base and the related estimating system. Proposal documentation should describe the statistical analysis performed and include the contractor's explanation of the CER's statistical validity.

9-1004.4 Cost Prediction Results The contractor should demonstrate that the parametric cost estimating relationships used can predict costs with a reasonable degree of accuracy. As with the use of any estimating relationship derived from prior history, it is essential in the use of parametric CERs for the contractor to document that work being estimated is comparable to the prior work from which the parametric data base was developed.

9-1004.5 System Monitoring The contractor should ensure that cost-to-noncost parametric rates are periodically monitored in the same manner as cost-to-cost rates and factors. If a CER is validated and will only be used in a one-time major new pricing application, rate monitoring capability is not essential (see 9-1004.2). However, if it is expected that the rates should be considered as an ongoing estimating technique, CER monitoring is critical. The contractor should revalidate any CER whenever system monitoring discloses that the relationship has changed.

9-1005 AREAS FOR SPECIAL CONSIDERATION IN PARAMETRIC COST

ESTIMATING

9-1005.1 Parametric Estimating for Change Orders Change order pricing using parametric cost estimating relationships may need to be considered in a different light than initial contract pricing actions. The contractor may use cost estimating relationships which are unique to change order proposals. In general, contractors do not segregate costs separately for individual change orders Therefore, it is important that the contractor have a system in place to validate, verify, and monitor CERs unique to change orders. However, if the CER was applicable to the basic contract and change orders, the CER could be validated without cost segregation.

9-1005.2 Forward Pricing Rate Agreements


212

a. Contractors may submit proposals for forward pricing rate agreements (FPRAs) or formula pricing agreements (FPAs) for parametric cost estimating relationships to reduce proposal documentation efforts and enhance government understanding and acceptance of the contractor's system. Government and contractor time can be saved by including the contractor's most commonly used CERs in FPRAs or FPAs. (See FAR 15.809 for basic criteria.) However, such an agreement is not a substitute for contractor compliance at the time of submitting a specific price proposal. FAR requires that the contractor describe any FPRAs in each specific pricing proposal to which the rates apply and identify the latest cost or pricing data already submitted in accordance with the agreement. All data submitted in connection with the agreement is certified as being accurate, complete, and current at the time of agreement on price on each pricing action the rates are used on, not at the time of negotiation of the FPA or FPRA (FAR 15.809 [d]). b. Key considerations in auditing FPRA/FPA proposals for parametric CERs follow: (1) FPRAs/FPAs do not appear practicable for CERs that are intended for use on only

one or few proposals. (2) Comparability of the work being estimated to the parametric data base is critical.

FPRA proposals for CERs must include documentation clearly describing circumstances when the rates should be used and the data used to estimate the rates must be clearly related to the circumstances.

(3) Validation of all the parametric criteria (9-1003) is especially important if a single CER or family of CERs is to be used repetitively on a large number of proposals.

9-1005.3 Subcontract Pricing Considerations a. FAR 15.806-2(a) requires that when a contractor is required to submit certified cost or pricing data, the contractor will also submit to the government accurate, complete, and current cost or pricing data from prospective subcontractors in support of each subcontract cost estimate that is: (1) $1,000,000 or more, (2) both more than $100,000 and more than 10 percent of the prime contractor's proposed

price, or (3) considered to be necessary to adequately price the prime contract. Use of parametric CERs does not relieve a contractor of its responsibility to disclose planned subcontract procurements and the related subcontractor cost or pricing data. b. When proposed material costs are based on parametric estimates the contractor must demonstrate that the type of materials required for the proposal are the same as included in the CER data base. The auditor should perform audit procedures to determine if: (1) materials included in the CER data base are not estimated separately in the proposal,

and (2) adjustments have been made to the CER data base for those items which were

previously manufactured in-house and now are being purchased. If the CER data base has not been adjusted, the contractor should provide a detailed cost estimate for purchased materials.

c. The contractor should explain any major differences between parametric estimates of subcontract costs and the subcontractor's quoted price and provide the rationale for using the parametric estimate instead of the quote.


213

d. Consistency in subcontract cost estimating must be maintained within the contractor's estimating system. Any significant deviations from normal practices in the proposal must be identified and justified by the contractor.

9-1005.4 Parametric Estimating Efficiency a. A primary justification for using parametrics is reduced estimating and negotiation costs. Contractors should perform a cost-benefit analysis before implementing an elaborate parametric estimating model. Their analysis should show that implementation and monitoring costs do not outweigh the benefit of reduced estimating costs. In many instances, new reporting systems may have to be developed to provide reliable noncost independent variables. In addition, the costs of CER validation and monitoring may be substantial. b. When the contractor's cost-benefit analysis indicates that the parametric system implementation costs might outweigh the benefits of reduced estimating costs and/or increased estimating accuracy, the matter should be pursued for potential cost avoidance recommendations.

9-1005.5 Data Base Adjustment Considerations a. One basic criterion (9-1004.4) is that the parametric data base be comparable to work being estimated. However, a contractor may have to adapt a partially comparable data base to its cost history using a "calibration" factor An example would be an adjustment to the data base to estimate the savings as a result of continuous improvement initiatives such as TQM. The utilization of complexity factors and/or adjustments to modify contractor developed in-house CERs is a valid technique. However, the use of such factors or adjustments should be fully documented and disclosed. In addition, this approach increases the contractor's burden to document compliance with the other criteria. b. If a contractor does not support the adjustment factors, the contracting officer should be promptly notified (see 9-1005.7). In addition, the auditor should determine if a qualified or adverse opinion is required. The audit report should disclose the costs associated with the unsupported factors.

9-1005.6 Contract Administration Interface a. Upon receipt of a request to review a price proposal, the auditor will coordinate with the Plant Representative/ACO to make arrangements for any needed technical reviews of the proposal. Because of the special nature of cost-to-noncost estimating relationships, and the possibility of limited cost history and added audit testing, complete coordination is especially important when parametric estimates are involved. b. While the auditor will address special areas of concern as requested by the PCO and/or the Plant Representative/ACO, the audit scope will be established by the auditor in accordance with the auditing standards, unless the PCO only requests a review of part of a price proposal. c. Auditors should be available, on request, to explain applicable price proposal criteria and identify any prospective audit concerns to both government and contractor personnel. An example of such audit advice would be to identify operating reports or records that have not been previously used to forecast costs and would therefore require added contractor support and audit testing. Such advance coordination will help avoid unnecessary contractor system development costs.


214

9-1005.7 Reporting of Estimating Deficiencies All proposal and estimating deficiencies found during the review of the parametric estimating technique should be immediately reported to the Plant Representative/ACO. These may include incorrect, incomplete, or non-current data and use of inappropriate estimating techniques. When a proposal evaluation discloses estimating system deficiencies, a separate report entitled "Estimating System Deficiency Disclosed during Evaluation of Proposal No. XXX" will be issued immediately after the proposed audit report.

9-1006 ESTIMATING STANDARDS

9-1006.1 Distinction Between Estimating Standards and Parametric Cost Estimating a. In terms of historical evolution and sophistication, the terminology of estimating standards as covered in this paragraph might be viewed as falling between traditional cost-to-cost estimating rates and factors and the more advanced types of parametric estimating systems (see 9-1002). However, a contractor may elect to use any combination of these evaluating methods, perhaps in the same proposal. b. Estimating standards are normally developed through the use of motion time-measurement studies performed by industrial engineers. Parametrics, on the other hand, are developed by relating historical costs to one or more noncost drivers. While estimating standards usually represent cost-to-noncost relationships they have traditionally been limited to narrower or more discrete elements of estimated cost than may be the case in more complex parametric CERs. Also, the logic of the estimating relationship and the appropriateness of the mathematics in estimating standards will usually be readily apparent. c. Estimating standards will not necessarily require validation under the criteria for parametric cost estimating relationships contained in 9-1003 and 9-1004. Especially when such standards (e.g., hours/pound, hours/drawing, hours/page) have been in place and accepted by government personnel, the evaluation guidance in this paragraph will likely be sufficient.

9-1006.2 Use of Estimating Standards a. Estimating standards may be established by relating engineering and/or production costs (efforts time, and/or materials) to specific characteristics of a product such as composition, weight, size or duration. This approach is designed to save estimating effort and has been used frequently in estimating construction costs and costs of recurring job orders such as printing. Many contractors use the technique in shop-order budgeting and production control. b. Estimating standards may be used to estimate the cost of a single material item required for the work, or the cost of a single labor operation, for example, welding electrodes per ton of structural steel, press operations time per page, or guard service costs per week. More complex, composite standards may be used to estimate costs of groups of components or broader classes of labor operations. c. Use of estimating standards may be appropriate in contract cost estimating situations when there is a close correlation between an amount of production cost and the related product or process characteristic. The data sets being correlated must have been measured in a uniform manner. The cost data used should be verifiable by reasonable means. The units of measure used for base characteristics should be uniform and readily identifiable; the quantity or value of a characteristic should be readily determinable. Standards may be derived from industry-wide


215

statistics but should be relevant and verifiable to the experience of the particular contractor using them.

9-1006.3 Applicability to Price Proposals Traditionally, estimating standards have been used to estimate costs in lump sums, often including supervision, indirect costs, and occasionally general and administrative expense. To comply with the SF 1411 and cost accounting standards the contractor will normally have to factor the estimate to identify the costs by cost element or function. Alternatively a proposed cost based on an estimating standard might qualify for submission as an "other" cost element if the cost can be tracked as such and is a relatively minor pan of the total proposal.

9-1006.4 Audit Procedures a. Depending on materiality and risk of the costs estimated, the auditor should examine the development and application of estimating standards to determine whether their use is proper in the circumstances. Evaluate all cost and noncost data applicable to each significant estimating standard and determine whether the data has been properly used in the computations. Assure that the measurements and correlation are adequate for the purpose. Determine whether the basis for the standard (for example, the product mix, production rates, and production methods) is sufficiently similar or comparable to that contemplated in the estimate at hand. b. When changes are contemplated in the design or production of an end item or the rate or method of production, the contractor's adjustments of the estimating standards require special scrutiny. Review by government technical specialists may be necessary in this situation. c. During audits of historical costs, sufficient information may be readily available from which the auditor could develop estimating standards to use as one means of appraising recurring contractor estimates. However, this will not substitute for audit review of cost estimates as submitted by the contractor.


216

Superintendent of Documents Order Form

The total cost of my order is $________. Price includes regular shipping and handling and is subject to change. International customers please add 25%. _______________________________________________________ Company or personal name (Please type or print)

________________________________________________________________________

Additional address/attention line

________________________________________________________________________

Street Address

________________________________________________________________________

City, State, Zip Code

________________________________________________________________________

Daytime phone including area code

________________________________________________________________________

Purchase order number (optional)

Check method of payment:

__________________________________________________________________

Authorizing signature

Mail to: Superintendent of Documents

P.O. Box 371954, Pittsburgh, PA 15250-7954

YES, please send me the following publications:

subscriptions to DCAA CONTRACT AUDIT MANUAL (DCAAM) for $32.00

per year ($40.00 foreign)

Charge

your

order.

It’s

easy!

To fax

your orders

(202)512-2250

Check payable to Superintendent of Documents GPO Deposit Account

VISA MasterCard

To phone

your orders

(202) 512-1800

Thank you for

your order! (expiration date)


217

APPENDIX D

MORE ABOUT STATISTICS


218

APPENDIX D

MORE ABOUT STATISTICS

In this Appendix we will discuss, in additional detail, some statistical topics including

error, sample size, the Student's "t" distribution, the "f" distribution, and confidence intervals.

STATISTICAL INFERENCE

In statistical inference we make generalizations based on samples, and, traditionally, such

inferences have been divided into problems of estimation and hypothesis testing. In estimation

we assign a numerical value to a population parameter on the basis of sample data. In any CER,

we are attempting to predict the population behavior(s) from a sample. We need to ask ourselves

the question, "How good is our estimation?" When we test a hypothesis, we accept or reject

assumptions concerning the parameters or the form of a population. When we use a CER, we

need to be confident of the CERs ability to predict the future - without this confidence, the CER

should not be used.

Z-scores

In general, if X is a measurement belonging to a set of data having the mean x (or µ for

a population) and the standard deviation s (or σ for a population), then its value in standard

units, denoted by Z, is

Zx x

S=

− or, Z =

−Χ µ

σ

depending on whether the data constitute a sample or a population. In these units, Z tells us how

many standard deviations a value lies about or below the mean of the set of data to which it

belongs.

Error

An estimate is generally called a point estimate, since it consists of a single number, or a

single point on the real number scale. Although this is the most common way in which estimates

are expressed, it leaves room for many questions. For instance, it does not tell us on how much

information the estimate is based, and it does not tell us anything about the possible size of the

error. And, of course, we must expect an error. An estimate's reliability depends upon two

things - the size of the sample and the size of the population standard deviation, σ . Any

statistics textbook will show that the error term is,

E Z= ∗( )ασ

2n

where,


219

Z(α /2) represents the number of standard deviations from the mean that we are willing to allow

our estimate to be "off" either way by probability of 1− α . This result applies when n is large

and the population is infinite. The two values which are most commonly used for 1− α are

0.95 and 0.99, with corresponding Z scores, Z ( 0.025) = 1.96 (standard deviations) for 1− α =

0.95 and Z (0.005) = 2.575 for 1− α = 0.99, respectively.

There is one complication with this result. To be able to judge the size of the error we

might make when we use x as an estimate of µ , we must know the value of the population

standard deviation, σ . Since this is not the case in most practical situations, we have no choice

but to replace σ with an estimate, usually the sample standard deviation, s. In general, this is

considered to be reasonable provided the sample is sufficiently large (n ≥ 30).

Sample Size

The formula for E can also be used to determine the sample size that is needed to attain a

desired degree of precision. Suppose that we want to use the mean of a large random sample to

estimate the mean of a population, and we want to be able to assert with probability 1− α that

the error of this estimate will be less than some prescribed quantity E. Solving the the previous

equation for n, we get,

nZ

=∗

( / )α σ2

2

Ε

Confidence Intervals

For large random samples from infinite populations, the sampling distribution of the

mean is approximately normal with the mean µ and the standard deviation σ / n , namely,

that,

Zx

n=

− µ

σ

is a random variable having approximately the standard normal distribution. Since the

probability is 1− α that a random variable having the standard normal distribution will take on

a value between -Z(α / 2 ) and Z(α / 2 ), namely, that −Z ( / )α 2 < Z < Z ( / )α 2 , we can

substitute into this inequality the foregoing expression for z and it yeilds,

− <−

<Zx

nZ( / )

/( / )α

µ

σα2 2

Using some algebraic manipulation, we get

x Z n x z n− ∗ < < + ∗( / ) / ( / ) /α σ µ α σ2 2


220

and we can assert with probability 1− α that it will be satisfied for any given sample. In other

words, we can assert with (1− α )% confidence that the interval, above, determined on the basis

of a large random sample, contains the population mean we are trying to estimate. When σ is

unknown and n is at least 30, we replace σ by the sample standard deviation, s.

An interval such as this is called a confidence interval, its endpoints are called confidence

limits, and the probability 1− α is called the degree of confidence. Again, the values most

commonly used for 1− α are 0.95 and 0.99, the corresponding values of Z ( / )α 2 are 1.96 and

2.575, and the resulting confidence intervals are referred to as 95% and 99% confidence intervals

for µ.

Confidence Intervals for Means (Small Samples)

To develop corresponding theory which applies also to small samples, it will be necessary

to assume that the population we are sampling has roughly the shape of a normal distribution.

We can then base our methods on the statistic tx

s n=

− µ

/, whose sampling distribution is a

continuous distribution called the t distribution. This distribution is symetrical and bell-shaped

with zero mean. The exact shape of the t distribution depends as a parameter called the number

of degrees of freedom, given by n-1, the sample size less one. For the t distribution we define

t(α/2) in the same way in which Z(α/2) was defined. However, t(α/2) depends on n-1 (degrees of

freedom) and its value must be looked up in a table of values. In the same way as before we can

arrive at the following small sample confidence interval for µ:

x t s n x t s n− ∗ < < + ∗( / ) / ( / ) /α µ α2 2

The degree of confidence is 1− α and the only difference between this formula and the

large sample formula is the t(α/2) takes the place of Z(α/2).

Analysis of Variance and the F Statistic

The F statistic is a statistic for a test concerning the differences among means. It is

defined as:

F = estimate of σ2 based on the variation among the x 's

estimate of σ2 based on the variation within the samples

and is called a variance ratio. The F distribution is a theoretical distribution which depends on

two parameters called the numerator and denominator degrees of freedom. When the F statistic

is used to compare the means of k samples of size n, the numerator and denominator degrees of

freedom are, respectively, k-1 and k(n-1).

This is a simple form of an analysis of variance. The basic idea of an analysis of variance

is to express a measure of the total variation of a set of data as a sum of terms, which can be

attributed to specific sources, or causes of variation. Two such sources of variation could be 1)

actual differences, and, 2) chance differences. As a measure of the total variation of an

observation consisting of k samples of size n, we use the total sum of squares,


221

SST xi

k

ijj

n

X= −= =

∑ ∑1

2

1

( ...) ,

where Xij is the jth observation of the ith sample, i = 1, 2, ... , k, and j = 1, 2, ... , n, and x , the

mean of all the k measurements or observations is called the grand mean. If we divide the total

sum of squares by kn-1, we get the variance of the combined data.

Letting X i devote the mean of the ith sample, i = 1, 2, ..., k, we can write the following

identity:

SST n X X X Xii

i

k

j

n

i

k

ij= ∗ − + −= ==

∑ ∑∑ ( ...) ( )2

1 11

2

Looking closely at the two terms into which the total sum of squares SST has been

partitioned, we find that the first term is a measure of the variation among the means. Similarly,

the second term is a measure of the variation within the individual samples, or chance variation.

Dividing the first term by k-1 and the second by k (n-1), we get the numerator and the

denominator of the F Statistic as defined, above. The first term is often referred to as the

treatment sum of squares, SST and the second term as the error sum of squares, SSE,

experimental error, or chance.

Refer to a statistics textbook for a further explanation of these and other statistical

subjects.


222

APPENDIX E

EXAMPLES OF OTHER HARDWARE

ESTIMATING MODELS

FROM THE SPACE SYSTEMS COST ANALYSIS GROUP


223

APPENDIX E

EXAMPLES OF OTHER HARDWARE ESTIMATING MODELS

OTHER HARDWARE MODELS

Advance Missions Cost Model NASA JSC

Aerospace Small Satellite Cost Model The Aerospace Corp.

ALS/ADP Cost Model USAF Phillips Lab

EHF/SHF Communications Cost Model USAF SMC/FMC

Hypercost NASA JPL

Launch Vehicle Cost Model TECOLOTE

NASCOM NASA MARSHALL

Non Nuclear Power NASA LeRC

Nuclear Space Power NASA LeRC

Parametric Cost Model Parametric Consultants/England

Passive Sensor Cost Model USAF SMC/FMC

SCEEDOS USAF SMC

Sensat Owl Wise Laboratory

Solid Rocket Motor Cost Model TECOLOTE

STACEM AF Astronautics Lab

TRANSCOST Deutsche Aerospace

Unmanned Spacecraft Cost Model USAF SMC

Some examples follow.


224

COST MODEL DESCRIPTION

Title: Advance Missions Cost Model (AMCM) Purpose: To predict development, recurring and mission operations costs of

future space systems, including launch vehicles, spacecraft and human explorations missions.

Applicability: The AMCM is a system level cost model, therefore it is more

appropriate for large scale programs requiring many different systems that will be integrated to perform a complex mission. The methodology is most useful in the pre-conceptual and conceptual design phases of a program when the actual design of the systems is not known and many factors are being traded off.

Model Description: The AMCM is a two equation, multi-variable cost estimating

relationship. The first equation predicts the development and production cost of the system based of various technical and programmatic factors. The second equation predicts the basic mission operations cost of the system. Both equations are fitted to a historical database for AMCM includes ground vehicles, ships, aircraft, helicopters, and missiles as well as launch vehicles and spacecraft. The database spans 50 years of systems development. Most of the systems are US Government developed, but some commercial and European systems are included.

Status/Availability: The AMCM is complete and is periodically updated. The latest

release of the development model is Version 4.1 dated July 1992. The mission operations segment of the model was last updated in November 1986. Papers describing the model are available from the author; Kelley Cyr of NASA JSC. A personal computer version of the model that runs with Microsoft Excel is also being developed.

Input Variables: - Development Quantity - Production Quantity - Unit Dry Weight - Specification value (a numeric value representing the operating

type and mission environment) - Initial Operating Capability (IOC) year - Block Number - Difficulty Factor - Number of Years of Operations Output: Development, production and operations costs in millions of

dollars. For space systems, the cost estimates are for prime


225

contractor cost only and do not include fee, program support, or other government costs. For non-space systems, the costs are all inclusive.

Data Source: A large historical 50-year database of land, water, air and space

systems. The data includes 54 spacecrafts, 22 space transportation systems, 61 aircraft, 86 missiles, 29 ships, and 18 ground vehicles. All of the data points are from programs that were completed through IOC.

Point of Contact: Kelley Cyr, National Aeronautics and Space Administration,

Johnson Space Center, Mail Code BZ, Houston, Texas 77058 User Community: Cost Analysts, Project Managers, Systems Engineers Principle Ground Rules/ The AMCM development model is primarily based on raw Assumption/Limitations: data from contractors, government reports and published sources.

The original data was adjusted to constant year 1991 dollars using various inflation factors. The basic model does not distinguish between non-recurring and recurring cost because of the difficulties associated with separating the costs in development programs with very small production runs. The recurring cost of the system can be computed from the CER’s by calculating the marginal cost. The mean absolute percentage error (MAPE) for the model is approximately 45%.

Software: The CER’s can be easily programmed in any language or

spreadsheet. A Microsoft Excel add-in application is being developed.

Equipment: The Microsoft Excel add-in application will be able to run on any

platform that supports Microsoft Excel. CER Format: Cost = a • Qb • Wc • dS • e(1/(IOC-1900)) • Bf •gD Q is the quantity W is the weight s is the Specification IOC is the IOC year B is the Block Number D is the Difficulty Factor a-g are model parameters


226

Title: Aerospace Small Satellite Cost Model (SSCM) Purpose: To evaluate cost of designing, building, and testing a modern small

satellite. Model Description: SSCM estimates the first-unit development and production cost by

computing a weighted average of parametric CER’s derived from actual Small-satellite cost and technical databases. Included are production, integration, assembly, and testing. Excluded are any payload development or production costs.

Status/Availability: SSCM is currently in a test-and-evaluation configuration; a more

elaborate program with more user features is forthcoming. The current version is available to government agencies and participating small-satellite contractors.

Input Variables: Mission type (communications, remote sensing, or space

experiments), Procurement environment (military, commercial or NASA), Subsystem masses, performance measures, hardware legacy and production schedule.

Output: Development cost range and cost sensitivities. Data Source: Proprietary data from variety of small-satellite programs such as

REX, ALEXIS, STEP, APEX, RADCAL, and LOSAT-X. Point of Contact: Erik Burgess, Resource and Requirements Analysis Department,

the Aerospace Corporation, (310) 336-4148. User Community: Project Managers and Cost Analysts. Principle Ground Rules/ SSCM estimates but costs only; payload development/ Assumptions/Limitations production costs must be separately determined. Software: SSCM is currently hosted in a stand-alone compiled program that

can operate on any IBM-compatible computer. It requires only a 360K-byte driver and DOS 2.0 or higher.

Equipment: See above. CER Format: Cost as explicit functions of input variables. Test Results/Evaluation: Validated against several small-satellite development programs.


227

Title: Aerospace Launch Vehicle Cost Model (LVCM) Purpose: To evaluate cost of existing, modified, new launch vehicle designs. Model Description: LVCM determines RDT&E cost by subsystem, average unit

operations cost by subsystem for each stage, total vehicle life-cycle cost, fiscal year funding requirements for overall vehicle program.

Status/Availability: Internal Aerospace Corporation use only. Input Variables: Launch site characteristics (site location, relative number of

launchers from each site); propellant type; weight, quantity pre vehicle, design status and previous quantity produced of each of the following subsystems: Structure, Thermal Control, Reentry Protection, Landing System, Electrical Power, Electrical Wiring, Guidance and Control, Data Handling, Instrumentation, Communication, Propulsion, Engines, Reaction Control Propulsion Hardware, Interstage Adapter and Payload Fairing.

Output: LVCM produces both numerical and graphical results. Major

output is life-cycle cost estimate for complete vehicle broken down by principal categories of RDT&E, Investment and Operations, which are further broken down by each stage, fairing, launch operations, flight operations, recovery (and refurbishment) operations, spares, software, facilities, system engineering and technical directions and government support.

Data Source: Proprietary data from variety of DoD launch vehicle programs. Point of Contact: Ronald Hovden, Resource and Requirements Analysis Department,

the Aerospace Corporation, (310) 336--5832.. User Community: Aerospace Corporation Project Managers and Cost Analysts. Principle Ground Rules/ Required detailed knowledge of input parameters. Assumptions/Limitations Software/Equipment: Written in FORTRAN 77, currently hosted on IBM mainframe.

Inputs are selected via an interactive, panel-driven front-end. Lahey PC version is under development, but not yet fully functional.

CER Format: Cost as explicit functions of input variables. Test Results/Evaluation: Validated against a number of existing DoD launch vehicles.


228

Title: Unmanned Spacecraft Cost Model, Sixth Edition, Nov 1988

(USCM6) Purpose: To estimate total Space Segment Cost including non-recurring and

recurring cost of components as well as subsystems for earth orbiting unmanned spacecraft.

Publisher: USAF, Space Division, Los Angeles AFB Model Description: (see attached schematic) By using physical descriptions of components, the model’s CER’s

develop component NR and REC cost. Other CER’s develop support costs such as Program Management, Systems Engineering, Data, etc.

Input Variables: component weights number of sensors volume, design life, BOL power number of solar cells, battery capacity power output, analog electronics weight total spacecraft dry weight Output: NR and T1 cost for components NR and T1 cost for integration and assembly NR and T1 cost for ferrite devices NR AGE; launch and orbital ops support NR and T1 for Program Management, Systems Engineering,

Systems Test and Data. Data Source: The database consists of 9 military, 6 NASA, and 3 commercial

satellite programs with start dates between 1964 and 1979 Point of Contact: Publisher User Community: From managers interested in top-level estimates to analysts

needing detailed information on CER development. Principle Ground Rules/ Estimates are in 1986 dollars excluding fee. Costs cover all NR Assumptions/Limitations and REC Spacecraft cost including launch operations (relating to

S/C only) and orbital support. Equipment: No equipment other than a calculator is needed. Macros are

provided for uses on PCs running LOTUS 1-2-3.


229

CER Format: Both single and multivariate using linear and log/log relationships. An interesting not is the introduction of the “PING” factor with the log/log relationship.

Test Results/Evaluation: TBD


230

UNMANNED SPACE VEHICLE COST MODEL 6TH EDITION MODEL NONRECURRING COST FLOW

INPUTS Weight Sensor Types/Wt Torque Methods/Wt /Vol Wt/BOL Power Wt*BOL Pwr/# cells Wt*BOL Pwr Wt*BOL Pwr Wt*BOL Pwr/Batt Cap S/C $ Pwr Output/TWTA/Subsys Weight/Design Life Space Weight Vehicle + Weight/Design Life AGE $ Weight/Comm Mission Wt/Orbit/Subsys Dry Wt/Stabilization Type S/C + Comm + I&A $ Pwr Output/TWTA/Subsys Weight/Design Life Weight Comm $ Weight/Design Life Weight/Comm Mission Wt/Orbit/Subsys

S/C •Structure •Attitude Control Subsystem •Attitude Determination •RCS •Thermal Control •Electrical Power Subsystem •Generation •Storage •Distribution •Conditioning Total $ •Telemetry, Tracking & Command Microwave •Transmitter/Amplifer Ferrite •Receiver/Exciter Devices •Transponder •Digital Electronics •Analog Electronics •Antenna Propulsion (AKM)

I N T E G R A T I O N

&

A S S E M B L Y

A E R O S P A C E

G R O U N D

E Q U I P M E N T

•Communications Payload Total $ Microwave •Transmitter/Amplifier Ferrite •Receiver/Exciter Devices •Transponder •Digital Electronics •Analog Electronics •Antennas

Program Level Costs

• Prog Mgmt

• Sys Engr

• Sys Test

• Data


231

UNMANNED SPACE VEHICLE COST MODEL 6TH EDITION MODEL RECURRING COST FLOW

INPUTS Space Veh Dry Wt/Mission Weight/Manner Sensor Types, Wt Vol/Design Life/Wt Wt/Manned Area/BOL Pwr/# cells $ Wt*BOL Pwr Wt*BOL Pwr Wt*BOL Pwr Weight Weight/Sync Orbit Weight/Output Power Suite Weight Weight Total Impulse + Quantity Weight Weight/Sync Orbit $ Weight/Output Power Component Power Suite Weight Weight Space Veh Dry Wt $

S/C •Integration & Assembly •Structure •Attitude Control Subsystem •Attitude Determination •RCS •Thermal Control •Electrical Power Subsystem •Generation •Storage •Distribution •Conditioning •Telemetry, Tracking & Command Microwave •Transmitter/Amplifier Ferrite •Receiver/Exciter Devices •Transponder •Digital Electronics •Analog Electronics •Antenna •Propulsion (AKM)

•Communications Payload Total $ Microwave •Transmitter/Amplifier Ferrite •Receiver/Exciter Devices •Transponder •Digital Electronics •Analog Electronics •Antennas

Launch & Orbital Operations Support (LOOS)

Program Level

Costs

• Prog Mgmt

• Sys Engr

• Sys Test

• Data


232

APPENDIX F

SOME CURRENTLY AVAILABLE

SOFTWARE ESTIMATION PRODUCTS

PRODUCED BY THE SOFTWARE

SUBGROUP OF THE

SPACE SYSTEMS COST ANALYSIS GROUP


233

APPENDIX F

SOME CURRENTLY AVAILABLE SOFTWARE ESTIMATION PRODUCTS

This section incorporates inputs from a variety of sources, primarily from model

developers. No attempt has been made to substantiate the information at this point. Comments

and updates are welcome, as are full descriptions of other models not discussed here.

Contents:

PRICE-S

REVIC

SASET

SEER-SEM

SLIM

SoftCost-R & SoftCost-Ada

Other Models


234

PRICE-S

This model was developed originally by Martin Marietta Price Systems (initially RCA

Price, then GE Price) as one of a family of models for hardware and software cost estimation.

Developed in 1977 by F. Freiman and Dr. R. Park, it was the first commercially available

detailed parametric software cost model to be extensively marketed and used. In 1987, the model

was modified and re-validated for modern software development practices. The PRICE-S model

is proprietary, it can be leased for yearly use on IBM or compatible PC, and operates within

Microsoft windows. It is also available for use on a UNIX workstation. The model is applicable

to all types of software projects, and considers all DoD-STD-2167A development phases.

Inputs

One of the primary inputs for the PRICE-S model is source lines of code (SLOC). This may be

input by the user or computed using either object-oriented or function point sizing models. Both

sizing models are included in the PRICE-S package. Other key inputs include:

1. Application: a measure of the type (or types) of software, described by one of seven

categories (mathematical, string manipulation, data storage and retrieval, on-line, real-

time, interactive, or operating system).

2. Productivity Factor: A calibratable parameter which relates the software program to

the productivity, efficiency/inefficiencies, software development practices and

management practices of the development organization.

3. Complexities: Three complexity parameters which relate the project to the expected

completion time, based on organizational experience, personnel, development tools,

hardware characteristics, and other complicating factors.

4. Platform: the operating environment, in terms of specification, structure and

reliability requirements.

5. Utilization: Percentage of hardware memory or processing speed utilized by the

software.

6. New Design/New Code: Percentage of new design and new code.

7. Integration (Internal): Effort to integrate various software components together to

form an integrated and tested CSCI.

8. Integration (Extenal): Effort to integrate various software CSCI’s together to form an

integrated and tested software system.

9. Schedule: Software project start and/or end dates.

10. Optional Input Parameters: Financial factors, escalation, risk simulation.

Processing

The PRICE-S algorithms are published in the paper entitled "Central Equations of

PRICE S" which is available from PRICE Systems. It states that PRICE-S computes a "weight"

of software based on the product of instructions and application inputs. The productivity factor

and complexity inputs are very sensitive parameters which affect effort and schedule

calculations. Platform is known to be an exponential input; hence, it can be very sensitive. A

new weighted design and code value are calculated by the model based on the type or category of

instructions. Both new design and code affect schedule and cost calculations. Internal


235

integration input parameters affect the CSCI cost and schedule for integrating and testing the

CSCI. The external integration input parameter is used to calculate software to software

integration cost and schedule.

Outputs

PRICE-S computes an estimate in person effort (person hours or months). Effort can be

converted to cost in dollars or other currency units using financial factors parameters. Software

development schedules are calculated for nine DoD-STD-2167A phases: System Concept

through Operational Test and Evaluation. Six elements of costs are calculated and reported for

each schedule phase: Design Engineering, Programming, Data, Systems Engineering Project

Management, Quality Assurance, and Configuration Management. The PRICE-S model also

contains several optional outputs including over thirty graphs, Gantt charts, sensitivity matrices,

resource expenditure profiles, schedule reports. In addition, Microsoft Project files, spreadsheet

files, and risk analysis reports can be generated. The risk analysis report is a Cumulative

Probability Distribution and is generated using either Monte Carlo or Latin Hypercube

simulation.

Calibration

The PRICE-S model can be run in ECIRP (PRICE backwards) mode to calibrate selected

parameters. The most common calibration is that of the productivity factor, which, according to

the PRICE-S manual, tends to remain constant for a given organization. It is also possible to

calibrate platform, application, and selected internal factors.

Life Cycle Considerations

The PRICE-S life cycle model, included in the PRICE-S package, is a detailed model

which computes software support costs. The primary inputs include PRICE-S development

inputs, support descriptors which include software support life, number of installations, expected

growth, and support productivity factors. The model also has a modification mode which allows

up to four modifications per software CSCI. The PRICE-S life cycle model calculates support

effort and outputs the cost in three support phases: maintenance, enhancements, and growth. The

model allocates effort or cost across six elements of costs for each support phase.

Risk Analysis

The PRICE-S model contains a robust Monte Carlo simulation utility, which facilitates

rigorous risk analysis. Uncertainty can be characterized using probability distributions to define

input parameters. Normal, Beta, Triangular and Uniform distributions are among those

available. Simulation results are consolidated and reported as a probabilistic estimate.

Contact

Lockheed-Martin PRICE Systems

700 East Gate Drive, Suite 200

Mt. Laurel, NJ 08054

(800) 437-7423 a.k.a (800) 43PRICE

REVIC


236

The Revised Intermediate COCOMO (REVIC) model was developed by Ray Kile and the

U.S. Air Force Cost Analysis Agency. It is a copyrighted program that runs under DOS on an

IBM PC or compatible computer. The model predicts the development costs for software

development from requirements analysis through completion of the software acceptance testing

and maintenance costs for fifteen years. REVIC uses the intermediate COCOMO set of

equations for calculating the effort (man-power in staff-months and staff-hours) and schedule

(elapsed time in calendar months) to complete software development projects based on an

estimate of the lines of code to be developed and a description of the development environment.

The forms of the basic equations are:

(I) MM = AB(KDSI)P(Fi)

(2) TDEV = CD(MM)

Equation (1) predicts the manpower in man-months (MM) based on the estimated lines of

code to be developed (KDSI = Delivered Source Instructions in thousands) and the product of

a group of environmental factors (Fi). The coefficients (A,C), exponents (B,D) and the factor

(Fi) are determined by statistical analysis from a database of completed projects. These

variables attempt to account for the variations in the total development environment (such as

programmer's capabilities or experience with the hardware or software) that tend to increase

or decrease the total effort and schedule. The results from equation (1) are input to equation

(2) to determine the schedule (TDEV = Development Time) in months needed to complete

the development.

REVIC enhancement of the intermediate COCOMO includes:

The addition of a fourth mode - Ada. Intermediate COCOMO has three modes of software

development: organic, semi-detached, and embedded. These modes describe the overall software

development in terms of size, number of interfaces, and complexity. REVIC adds a fourth mode,

Ada development, to the model. This mode describes programs developed using an object-

oriented analysis methodology or use of the Ada language (with separately cornpilable

specifications and body code parts). Each mode provides a different set of coefficients for the

basic equations.

Addition of the first and last development phases. COCOMO provides a set of tables

distributing the effort and schedule to the phases of development (system engineering,

preliminary design, critical design, etc.) and activities (system analysis, coding, test plarming,

etc.) as a percentage of the total. COCOMO covers four development phases (preliminary design,

critical design, code and unit test, and integration and test) in the estimate. REVIC adds two

more development phases: software requirements engineering, and integration & test after FQT.

REVIC predicts the effort and schedule in the software requirements engineering phase

by taking a percentage of the development phases. It provides a default value (12% for effort,

30% for schedule) for this percentage based on average programs, but allows the user to change

the percentage.


237

COCOMO development phase ends at completion of the integration & test phase (after

successful FQT). This phase covers the integration of software CSC's into CSCI's and testing of

the CSCI's against the test criteria developed during the program. It does not include the system-

level integration (commonly called software builds) of CSCI's, and the system-level testing to

ensure that system-level specification requirements are met. The software to software and

software to hardware integration and testing is accounted for in the Development Test and

Evaluation (DT&E) phase. REVIC predicts the effort and schedule for this phase by taking a

percentage of the development effort. REVIC provides a default percentage of 22% for effort

and 26% for schedule based on average programs. It allows the user to modify these percentages

if desired.

Complete interface between the model and the user. Users are not required to have extensive

knowledge about the model or detailed knowledge of algorithms. REVIC contains extensive

prompting and help screens. REVIC also removes the typical intimidation factor that prevents

analysts from successfully using models.

Provide the capability to interactively constrain the schedules and staffing levels. Schedules

can be constrained either in the aggregate or by phase of the development effort. Staffing can be

constrained by phase. Using these features, the analyst can estimate cost overruns, underruns,

and schedule slips at any major milestone by entering the actuals-to-date at any milestone, and

letting the program calculate the remaining effort and schedule.

Inputs: Same as COCOMO inputs.

Processing

While REVIC processing is mostly the same as Intermediate COCOMO, it provides a

single weighted "average" distribution for effort and schedule, along with the ability to allow the

user to vary the percentages in the system engineering and DT&E phases. On the other hand,

COCOMO provides a table for distributing the effort and schedule over the development phases,

depending on the size of the code being developed. REVIC has been enhanced by using

statistical methods for determining the lines of code to be developed. Low, high, and most

probable estimates for each CSC are used to calculate the effective lines of code and standard

deviation. The effective lines of code and standard deviation are then used in the equations,

rather than the linear sum of the estimates. This quantifies, and to some extent, reduces the

existing uncertainties regarding software size. [see Boehm's Software Engineering Economics for

a discussion of effective lines of code]. REVIC automatically performs sensitivity analysis

showing the plus and minus three sigma values for effort, and the approximate resultant

schedule.

Outputs

The user is presented with a menu allowing full exercise of the analytical features and displays of

the program. All inputs are easily recalled and changed to facilitate analyses and the user can

constrain the solution in a variety of ways.

Calibration and Accuracy


238

REVIC's coefficients have been calibrated using recently completed DoD projects

(development phase only) by using the techniques in Dr. Boehm's book. On the average, the

values predicted by the effort and schedule equations in REVIC are higher than in COCOMO. A

study validating REVIC equations using a database different from that used for initial calibration

was published by the Air Force's HQ AFCMD/EPR. In terms of accuracy, the model compares

favorably with expensive commercial models.

The level of accuracy provided by the model is directly proportional to the user's

confidence in the lines-of-code (LOC) estimates and a description of the development

environment. When little is known about the project or environment, the model can be run

leaving all environment parameters at their default (nominal) settings. The only required input is

LOC. As the details of the project become known, the data file can be reloaded into the program,

and the nominal values can be changed to reflect the new knowledge permitting continual

improvement of the accuracy of the model.

Maintenance Estimate

REVIC provides an estimate for software maintenance over a fifteen year period by using

the Boehm's COCOMO equation:

(3) MMam = MMnom * ACT P (MFi), where MMnom is the result of equation (1) without

multiplying by the environmental factor (Fi); ACT is annual change traffic as a percentage,

and Mfi is the environmental factors for maintenance.

REVIC provides a default percentage of ACT and allows it to be changed. REVIC also

assumes a transition period after delivery of the software, during which residual errors are found

before reaching a steady state condition. This provides a declining, positive delta to the ACT

during the first three years. Beginning the fourth year, REVIC assumes the maintenance activity

consists of both error correction and new software enhancements.

Other Reference Materials

There is context-sensitive help available on-line while running REVIC, and the

information is enough to input data and obtain results in all cases. However, the developers

recommend that Boehm's Software Engineering Economics be read to fully understand the

implications of parametric modeling.

Contact

The Air Force Cost Analysis Agency has assumed responsibility for REVIC upkeep and

distribution. For suggestions for upgrades or problem reporting, contact:

Air Force Analysis Agency

AFCSTC/IS (REVIC)

1111 Jefferson Davis Highway, Suite 403

Arlington, VA 22202

[information current as of 5/92]

(703) 604-0412


239

REVIC is available on the Air Force cost bulletin board at (800) 344-3602 or from Washington,

D.C. at . (703) 604-0412. Connection should be 1200 or 2400 BAUD with 8 bits, no parity, and

one stop bit. [information current as of 5/92]

REVIC and its user manual are also available on the Air Force Software Technology

Support Center (STSC) bulletin board at autovon 458-7653 or (801) 777-7653. Connection

at 2400 BAUD with 8 bits, no parity, and one stop bit. For access problems call Vern

Phipps or Rick Dahl at (801) 777-7703. [information current as of 5/92]


240

SASET

SASET is a sophisticated software cost estimating model built by Martin Marietta

Astronautics during 1986-1990 under contract with the Naval Center for Cost Analysis under

auspices of the Office of Naval research. Enhancements to the model were accomplished under

contract with both the Navy and Air Force cost centers. The model combines database

information from over 500 successfully completed software projects with "expert" factors to

determine software development and maintenance costs, schedules, and sizing numbers used by

engineers, estimators, and top-level management.

Cost estimating relationships were created, project complexity factors were established,

and independent variables were quantified. The result was database-derived software estimating

equations for assembly and high-order language software. These equations and resulting

software parametric models have been validated by comparing project sizing, labor actuals, and

schedules with SASET outputs.

During data collection, analysis, and model requirements generation activities, it was

decided that the parametric model would include the entire software development life cycle from

systems requirements through systems test, with an option to add maintenance, and provide

budget and schedule outputs for the engineering development organizations. The database

collection approach consists of decomposing software actuals by CSCI class, type and language.

Inputs

SASET starts from an initial state of no information about the software project, and then

proceeds to prompt the user in an orderly fashion for the information necessary to gain insight

and emulation of project environment, complexities, and sizing for up to 50 CSCI's, to produce

schedule and effort outputs.

Processing

SASET is a database-derived expert system. The calibrated equations for effort budget

and schedule are computed for the system environment, and software complexity, brought

together for a common set and then multiplied against the equivalent new HOL SLOC.

Outputs

The computerized model has the capability of sizing, costing, scheduling and integrating

up to 50 multiple CSCI's simultaneously. The model determines a labor estimate expressed in

staff months and spread over the phases and subphases of software development life cycle for

each individual CSCI and the total project and helps to establish a logical development sequence.

The model also identifies an optimal schedule and effort.

Calibration

All input variables and calibration constants of SASET are open and capable of being

changed. The calibration process requires a set of software development records as input. The

required data from each record includes staff hours, source lines of code, schedule, and project

complexity. Records are sorted by class (manned flight, avionics, ground, etc.) and type of

software (system, application, support), then run through multiple regression analysis for

productivity constants for each type of software.


241


SASET performs detailed environment effort allocation across subphases of the

development life cycle. It will also spread calculated maintenance effort across time. The effort

included is for engineering organizations only; any other efforts needing coverage need to be

added manually.

Contact

Air Force Analysis Agency

AFCSTC/IS (REVIC)

1111 Jefferson Davis Highway, Suite 403

Arlington, VA 22202

[information current as of 5/92]

(703) 604-0412


242

SEER-SEM

SEER-SEM is part of a family of software and hardware cost, schedule and risk

estimation tools. SEER models run on IBM, Macintosh, and Sun/UNIX platforms with no

special hardware requirements. SEER-SEM is used throughout the aerospace and defense

industry on two continents. All issues found in today's software environments are addressed.

Inputs

SEER-SEM accepts source lines of code (SLOC) or function points or both. When

selecting function points, the user may use IFPUG standard function points or SEER function-

based inputs which include internal functions. Users follow a Work Breakdown Structure

(WBS) describing each CSCI, CSC, and CSU (module or element) to be estimated. Knowledge

bases are used to provide fast and consistent inputs describing complexity, personnel capabilities

and experience, development support environment, product development requirements, product

reusability requirements, development environment complexity, target environment, schedule,

staffing and probability. Users can modify all inputs to their specifications at any time.

There are four sets of knowledge bases that automatically input environment factors.

These knowledge bases cover a wide variety of scenarios and help users produce fast and reliable

estimates. Knowledge bases are easily calibrated to user environments to give quick and accurate

estimates for the entire life cycle. Users can also change and modify each input at any time.

Knowledge bases include the following:

Platform describes the primary operating platform. Platform knowledge bases include avionics,

business, ground-based, manned space, unmanned space, shipboard, and more.

Application describes the overall function of the software under estimation. Application

knowledge bases include computer-aided design, command & control, database, MIS, office

automation, radar, simulation, and more.

Development Method describes the development methods to be used during the development.

These knowledge bases include Ada, spiral, prototyping, object oriented design, evolving,

traditional waterfall, and more.

Development Standard describes the development documentation, quality, test standards and

practices which will be followed during the development. These knowledge bases include

commercial, ISO-9000, 2167A, 1703, 1679, 7935A and more.

Processing

SEER-SEM uses proprietary algorithms which are found in the back of the User's

Manual. Parameter (input) sensitivities and other insights into the model are also found in the

user's documentation. Knowledge bases can be printed out by users. SEER-SEM utilizes a

unique process that simulates a 10,000 iteration Monte Carlo for risk analysis.

Outputs


243

SEER-SEM has almost 30 informative reports and charts covering all aspects of software

costs, schedules and risk. The Quick Estimate Report is easily tailored to instantly give the user

specific details for trade-off analyses and decision support information. A Detailed Staffing

Profile follows SEI suggested staffing categories. Risk reports and graphs based on person

months, costs, and schedule are standard features. SEER-SEM gives a minimum schedule

output. However, schedules, personnel, and other factors can be changed to give effort and cost

tradeoffs.

Calibration

Calibration of SEER-SEM involves the effort to customize input values to more closely

reflect particular program development characteristics. SEER-SEM's Design To Technology and

Design To Size functions provide the tools for calibration activities. The SEER knowledge bases

are flexible and easy to create and modify, providing the user with a mechanism for building

custom calibrated knowledge bases.


SEER-SEM estimates all elements of the life-cycle, beginning with the preliminary

design phase and ending with software maintenance. SEER-SEM has many features which

support Life Cycle Cost Analysis. Total life cycle cost is reported in the Basic Estimate Report,

Activity Report, and the Labor Allocation Reports. The Set Reference feature allows for quick

analysis of what happens to both development and maintenance costs with the change of any

parameter.

Maintenance

SEER-SEM baseline maintenance includes all adaptive, perfective and corrective

maintenance. Additionally, you may add annual change rate and growth percents to anticipate

any functional growth or enhancements over the software maintenance period. Enhancements

and block upgrades can also be estimated.

Contact

Galorath Associates, Inc.,

SEER Technologies Division

P.O. Box 90579

Los Angeles, CA 90009 (3 1 0)670-3404.


244

SLIM

SLIM for Windows 3.1 was developed by Quantitative Software Management, Inc.

(QSM). All of the theory behind the model has been published by Prentice Hall in 1992 in the

book, Measures for Excellence: Reliable Software, on Time, Within Budget by Lawrence H.

Putnam and Ware Meyers. SLIM is based on QSM's Software Equation. This was derived from

the Rayleigh-Norden model and has been validated over a 15 year time period with thousands of

real, completed projects. The equation takes the conceptual form:

(1) Quantity of Function = Process Productivity * Effort * Schedule

This means that the product of the time and effort coupled with the process productivity

of the development organization determines how much function can be delivered. Extensive

empirical study of software data has shown that very strong linearities exist in software behavior.

This is taken into account by the calculational form of the software equation and discloses how

these non-linearities can be exploited by management. The calculation form of the equation is

expressed:

(2) Size = Process Productivity Parameter * (Effort/B)(1/3) * Time(4/3)

where,

1. The process Productivity Parameter is the development process proficiency of the

organization. It is determined from historical data.

2. Size is the quantity of function created in source lines of code written, function points,

objects, or other measures of function.

3. Effort is the development effort required. It includes all categories of labor used on

the project. Effort is consistent with the time specified below.

4. B is a complexity adjustment factor. It provides for specialized skills for integration

testing, documentation, and management as the size of the system increases.

5. Time is the elapsed calendar development time from the start of detailed design until

the product is ready to enter into operational service (frequently this is a 95%

reliability level).

SLIM is applicable to all types and sizes of projects. It computes schedule, effort, cost,

staffing for all software development phases and reliability for the main construction phase.

Because the software equation effectively models design intensive processes and is not

methodology dependent, SLIM works well with waterfall, spiral, incremental, and prototyping

development methodologies. It works with all languages, and function points as well as other

sizing metrics. It is specifically designed to address the concerns of senior management, such as:

1. What options are available if the schedule is accelerated by four months to meet a

tight market window?

2. How many people must be added to get two months schedule compression and how

much will it cost?


245

3. When will the defect rate be low enough so I can ship a reliable product and have

satisfied customers?

4. If the requirements grow or substantially change, what will be the impact on schedule,

cost, and reliability?

5. How can I quantify the value of my process improvement program?

Computing Platforms

SLIM is available for use on IBM PC or compatible machines running Windows 3.1.

SLIM is licensed on an annual basis with full support and free upgrades.

Inputs

The primary input for SLIM is SLOC, function points, objects, CSCI, or any valid

measure of function to be created. The model uses size ranges for input: minimum, most likely,

and maximum. Other important inputs include:

1. Language: Multiple choices and mixes.

2. System Type: One of nine (business, scientific, command & control, real time, etc.).

3. Environmental Information: Tools, methods, practices, database usage; standards in

place and adherence and usage of those standards.

4. Experience: Personnel skill and qualifications.

5. Process Productivity Parameter: a macroscopic factor determined by calibration from

historical data. It is a reliable tuning factor that accurately reflects application

complexity and the efficiency of the organization in building software. This is a

secsitive parameter that is capable of measuring real productivity and process

improvement. SLIM contains and expert system to determine the Process

Productivity Parameter when the user has no historical data. This [non-linear]

parameter is dealt with in terms of a linear scale ranging from 0 to 40.

6. Management Constraints: Maximum allowavle schedule, manimum cost, maximum

and minimum staff size, required reliability at the time the software goes into service

as well as the desired probabilities for each of these constraints.

7. Accounting: Labor rates, inflation rates, and other economic factors.

8. Flexibility: Extensive tailoring for milestones, phase definitions, and fraction of time

and effort applied to each phase based on the organization’s own history.

Processing

There are three primary modes of operation: building and using an historical database,

performing estimating and analysis, and creating presentations and reports.

For estimation, SLIM uses the software equation in conjunction with management

constraints for schedule, cost, staffing and required reliability to determine an optimal solution

with the highest probability of successful completion. Through Monte Carlo simulation

techniques, the size range estimates are mapped through the software equation to provide

estimates of the uncertainty in schedule, cost staffing and reliability. The solution obtained can

be compared with the user's historical data and QSM's historical data to test its reasonableness.

This discloses impossible or highly improbable solutions so that expensive mistakes are avoided.


246

Outputs

The primary output of SLIM is the optimal solution, which provides development time,

cost, effort and reliability expected at delivery. It also provides comprehensive sensitivity and

risk profiles for all key input and output variables, and a consistency check with similar projects.

SLIM's graphical interactive user interface makes it easy to explore quickly extensive tradeoff

and "what if" scenarios including design to cost, schedule, effort and risk. It has 181 different

output tables and graphs from which the user can choose. These outputs constitute a

comprehensive set of development plans to measure and control the project while it is underway.

Calibration

The process productivity parameter for SLIM can (and should) be obtained by calibration

using historical data. All that is required are project size, development time and effort. These

numbers are input into the software equation to solve for the process productivity. The historical

data can also be used to compare with any current solution to compare for reasonableness.


The user can customize his life cycle in terms of phases, sub-phases, staffing profile

shapes and milestones. Explicit shapes are provided after delivery to deal with maintenance,

ongoing support and enhancement releases.

Contact

Quantitative Software Management, Inc.

1057 Waverly Way

McLean, VA 22102

(703) 790-0055


247

SOFTCOST-R & SOFTCOST-ADA

The SoftCost-R model was developed by Dr. Don Reifer based on the work of Dr. Robert

Tausworthe of the NASA Jet Propulsion Laboratory. SoftCost is now marketed by Resource

Calculations, Inc. of Englewood, Colorado. It contains a database of over 1500 data processing,

scientific and real-time programs. SoftCost-R is applicable to all types of prograrns and

considers all phases of the software development cycle. The model is available for lease on IBM

PC's. A separate model SoftCost-Ada is available to model Ada language and other object-

oriented environments.

SoftCost-Ada has been developed to match the new object-oriented and reuse paradigm

which are emerging not only in Ada, but also C++ and other object-oriented techniques. It

contains a database of over 150 completed projects, primarily Ada.

SoftCost-R Inputs

A key input of SoftCost-R is size, which can either be directly input in SLOC or

computed from function points. SoftCost-R uses a more sophisticated sizing model than

COCOMO; besides reused code, sizes of modules added or deleted may be included. The other

inputs are in four categories like COCOMO. Some SoftCost-R inputs are similar to COCOMO,

but many of the more than thirty inputs are unique. Examples of unique inputs are use of peer

reviews, customer experience, and degree of standardization. Each input except size requires a

rating ranging from "very low" to "extra high", with "nominal" ratings having no effect on effort

calculations. SoftCost-R also uses COCOMO inputs to compare the results of SoftCost-R with

those of an updated version of COCOMO.

SoftCost-Ada Inputs

In the main, the inputs are the same as SoftCost-R, with some changes to reflect the new

paradigm. There is no COCOMO comparison.

Processing

SoftCost-R is not a simple regression model. It uses powerful multivariable differential

calculus to develop solutions relying on the T-W probability distribution. This provides the

ability for the user to perform "what-if" analysis and look at what would happen to schedule if

effort were constrained. Such a capability is not present in COCOMO. SoftCost-R is one of the

few models for which the mathematical algorithms are completely described in the user's manual.

The SoftCost-R equation is:

(1) PM = P0 * Al * A2 * (SLOC)B

where,

1. PM = number of person-months,

2. P0 is a constant factor that may be calibrated,

3. A 1 is the "Reifer cost factor" which is an exponential product of nine inputs,

4. A2 is a productivity factor computed from 22 inputs,

5. B is an exponent which may be calibrated.


248

The user's manual illustrates values assigned to ratings for all model inputs to help the

user understand the effect of each input on effort and schedule.

Outputs

SoftCost-R computes an estimate in person-months of effort and schedule for each

project, plus a productivity value. Other outputs include a side-by-side comparison with a recent

version of COCOMO, several "what if" analysis options, a resource allocation summary for any

of three development methods (traditional waterfall, incremental development, or Ada object-

oriented), and schedule outputs for Gantt and PERT charts. SoftCost-Ada output formats are

similar, and can interface with project planning tools in the same way.

Calibration

The model contains a calibration file which contains values for multiple calibration

constants and cost drivers. The user may change these values to better describe the user's unique

environment, and store alternative calibration and WBS files for different jobs. SoftCost-Ada

and SoftCost-R are similar.


SoftCost-R contains a separate life cycle model for support costs. In addition to

SoftCost-R development inputs, life cycle inputs include annual change traffic, length of support

period, a sustaining engineering factor, and economic factors. In addition to annual and total

support costs, the life cycle model has optional reports for various staffing options, fixed levels

of maintenance, and fixed work force levels. Both SoftCost versions are similar, and use the

same staff limited approach to life cycle resource allocation.

Contact

Mr. A.J. (Tony) Collins

Resource Calculations, Inc.

7853 East Arapahoe Court, Suite 2500

Englewood, CO 80112-1361

Telephone: (303) 267-0379

Facsimile: (303) 220-5620


249

OTHER MODELS

CHECKPOINT

Software Productivity Research, Inc.

I New England Park

Burlington, MA 01803

(617) 273-0140

COSTAR

SoftStar Systems

28 Ponemah Road

Arnherst,NH 03031

(603) 672-0987

SOFTST��R

Softstar Systems

28 Ponemah Road

Arnherst,NH 03031

(603) 672-0987

Sells COSTAR, a COCOMO variant; supports Ada COCOMO and function points

IBM PC, VAX/VMS

Dan Ligett (personal letter - Jan 16, 1987) [info checked brochure 10/92]

SWAN

IIT Research Institute

201 Mill Street

Rome, NY 13440

(315)336-2359

Fax: (315) 339-7002

Personto Contact: Anthony H. Williarns (315)339-7105 [ok 12/13/93]

A COCOMO variant developed for US Arrny, PM Training Devices, 12350 Research

Parkway, Orlando, FL. 32826

Written in Ada; runs on IBM AT, MS-DOS 3.0 or higher

Includes Ada COCOMO & Function Point Analysis

Target Software

Dr. George Bozoki

552 Marine Parkway, #1202

Redwood City, CA 94065

(415)592-2560

SSM, the Software Sizing Model


250

Computer Economics

4560 Admiralty Way, Suite 109

Marina Del Rey, CA 90292

(213) 827-7300

Jensen Model ("System 4"), CEIS (CEI Sizer), training

[ref. 1987 IITRI Software Sizing model paper]

Mainstay Software Corporation

1099 18th Street, Suite 2920

Denver, COLO 80202

(303) 298-8961

Dan Walkovitz, President

Product: Mainstay - an analytical database system

[ok 8/92, Setzer]

Howard Rubin Associates, Inc.

1666 Massachusetts Avenue, Suite 7

Lexington, MA. 02173

(617)861-7171

Products:

FPXpert - an expert system for function point counting, analysis and management;

RA-METRICS - a measurement tool and metrics respository; primarily oriented toward

information systems, promotes concept of "measurement dashboard"

COSTMODL

Developed at the Software Technology Branch at NASA/JSC to be used for Space Station

Freedom Software Support Environment and will be available from COSMIC.


251

APPENDIX G

PARAMETRIC ESTIMATING SYSTEM

CHECKLIST


252

APPENDIX G

PARAMETRIC ESTIMATING SYSTEM CHECKLIST

Elements of Good Estimating Practice

The list that follows identifies elements of good practice that support reliable estimating

processes.

Questions:

Are there other elements of good estimating practice that should be listed?

Are there other items of evidence that indicate good practice that should be listed?

Should any of the elements or items of evidence be reworded or eliminated?

Note: The objective here is to identify elements of good practice that are distinguished, if

possible, from elements of good estimating processes. This means, among other things, that the

elements in the list should be independent of the estimating methods or models used.

The element in this list should not tell people how to estimate. They should simply provide

guidelines and testable criteria for doing it well.


253

Why Do We Estimate Size, Cost, and Schedule?

To scope proposed tasks

To explore alternative system concepts

design to cost/budget

To explore alternative design concepts

To explore alternative proposals for enhancements and upgrades

To identify key design elements

To identify key process parameters

prioritize needs vs wants

To identify key assumptions

To identify and quantify uncertainties

To identify tasks and their relationships

To assess schedule feasibility

To identify, allocate and schedule resources

To assess an organization’s ability to perform within targeted costs

To evaluate the consequences of internal and external constraints

To establish achievable objectives

To establish a basis for quality service

To establish commitments

To bound the risk against customer needs

To balance levels of risk against customer needs

To provide a basis of successful risk management

build vs buy analysis

To prepare successful proposals

To evaluate proposals from competing bidders

To establish baselines for project tracking

enhance/reuse vs redesign analysis To predict life-cycle costs

To predict returns on investments

To provide information for establishing business and investment strategies


254

ELEMENTS OF GOOD ESTIMATING PRACTICE

Element

1. The objective (purpose) of the

estimate is stated in writing

2. The product to be produced is clearly

described

3. The tasks to be estimated are clearly

identified

4. People from related but different

projects or disciplines are involved

in the estimating process

5. The validity of the estimate is

supported by relating it demonstrated

performance on completed projects

6. More than one cost model or

estimating approach is used

7. Potential cost and schedules impacts

are estimated for all identified tasks

Evidence of Maturity

Information objectively captured in historic

database with a standard structure for ease of

analysis

Estimates for product size and content are

supported by systematic engineering

analyses

The terms and parameters that describe the

product enable comparisons to be made with

other products

Checklists are used to identify the elements

and activities included (and excluded from)

the estimate

Estimating and analysis team members

found in historical database

Values assigned to cost model parameters

are supported by comparisons to values used

to describe completed projects

The reasons for values assigned each cost

model parameter are documented in writing

Differences in results are analyzed and

accounted for

A structured process is used to identify and

scope technical risks

Uncertainties in values of descriptive

parameters are identified and quantified

The effects of uncertainties in descriptive

parameters are evaluated


255

8. Dictated schedules are analyzed for

impacts on cost

9. Estimates (or estimates to complete)

are updated whenever...

- changes to requirements affect cost

or schedule

- constraints change

- actual values for product or process

parameters are found to be

significantly different from those

on which the plan is based

- tracking measures indicate that

critical path tasks cannot be

completed as planned

10. The estimating origination has a

method (such as a database) for

organizing and retaining information

on completed projects

Managers (and customers, where

appropriate) are informed of potential cost

savings associated with alternative schedules

Captured in historical database

Program manager uses the process to

manage the program and associated risks

Process flags management when risk

exceeds defined limits

Elements in the database are recorded and

described in ways that avoid

misinterpretation

Schedule milestones (start and finish dates)

are described in terms of criteria for

initiation or completion

Effort measures account for unpaid overtime

The database is populated with a useful set

of completed projects

Cost models are used to provide consistent

frameworks (standard terms, WBS,

parameters, etc.) for recording historical data

Inconsistencies in historical data are sought

out and explained (this is perhaps best done

with the same cost models used for

estimating)


256

11. Postmortems are held at the

completion of each project

- to ensure that events that affected

cost or schedule are described and

recorded while still fresh in

people’s minds

- to ensure that recorded data are

valid

12. The emphasis throughout is on

developing consistency in describing

completed projects and in relating

new work to demonstrated

performance on those projects

Technical seminars and training programs

incorporate this information in a timely

manner

The consistency achieved when fitting cost

models to completed projects is measured

and tracked

The term “model accuracy” (as in “the

model made the estimate”) is never used


257

INDICATORS OF ESTIMATING CAPABILITY

What organizational support is needed to successfully establish and sustain a reliable estimating

capability?

The following checklist identifies seven indicators of good support. For each indicator, it lists

items of evidence one can look for to help evaluate the quality of this support.

Questions:

Are there other indicators of organizations support that should be listed?

Are there other items of evidence that should be listed?

Should any of the indicators or items of evidence be reworded or eliminated?

Note: The objective of this checklist is to provide guidance to organizations that are seeking to

establish and sustain a successful estimating capability.


258

Indicators of Estimating Capability

- A Checklist for Successful Estimating Environments -

Indicator

1. Management

acknowledges its

responsibility for

developing and sustaining

an estimating capability

2. The estimating function is

funded

3. Estimators have been

equipped with the tools

and training needed for

reliable estimating

4. The people assigned as

estimators are experienced

and capable

Evidence of Maturity

One or more managers have been assigned

responsibility for developing and sustaining the

organization’s estimating capability

The responsibility has been in place for sufficient time

to be effective

At least one person in the organization has a standing

assignment as an estimator

The funding provides for:

- establishing and sustaining a corporate memory

- preparing estimates

- capturing data from ongoing and completed projects

- acquiring and using estimating models and tools

- educating and training estimators

The funding is adequate (past, present, and future)

The funding is stable

Estimators have up-to-date desktop computing

facilities (hardware and software)

Cost models and software such as spreadsheets,

databases, and statistical programs have been acquired

or developed

Estimators have received training in the cost, schedule,

and size models they use

Estimators have received training in estimating and

model building

Independence/objectivity

The estimators have professional experience/

knowledge with the processes whose costs are

estimated


259

5. The estimating capability

of the organization is

quantified, tracked, and

evaluated

6. Recognition and career

paths exists such that

qualified people want to

become estimators

7. Estimators help quantify

and track progress in

software process

improvement

Estimators have educational backgrounds that support

quantitative analysis as well as qualitative

The average years of estimating experience among

people assigned as estimators is computed and tracked

Consistent methods are used for capturing and

recording information from completed projects

Cost model calibrations are up-to-date

Estimators participate in professional activities or

societies related to estimating

The people assigned as estimators are viewed as a

repository of estimating expertise

Management tracks and reviews the effectiveness of its

estimating capability

- what is working well

- what’s not working well

- opportunities for improvement

Estimating is seen by both employees and managers as

a career-broadening assignment

Previous estimators have moved on to positions of

equal or higher responsibility

People volunteer to be estimators

Cost models are used to account for factors that make

projects different, so that process improvements can be

meaningfully measured and compared

Trend analyzed derived form cost model calibrations

are used to track changes in the organization’s

processing and performance parameters

Cost models are used as on-going tools in managing

program execution


260

A Manager’s Checklist for Validating Cost and Schedule Estimates

When you (or your boss) receive a cost or schedule estimate, what should you look for to

determine your willingness to rely on that estimate?

The checklist on the following pages identifies key issues that have been identified as important

to estimates we can trust. Each issue is associated with elements of evidence that, if presents

support the credibility of the estimate.

Question:

Are there other issues that you (or your boss) look at?

Are there other issues that you should be looking at?

What evidence relative to any of these issues would support you willingness to rely on an

estimate?

Should any parts of the present checklist be reworded or eliminated?

Note: The objective of this checklist is to provide operational guidelines that help people

evaluate the credibility of cost and schedule estimates for software projects.

A secondary objective is to motivate and guide organizations toward improving estimating

processes and practices.