honourcode, inc. toward ontology for se-roi 1 toward an ontology for measuring systems engineering...
Post on 18-Dec-2015
216 views
TRANSCRIPT
Toward Ontology for SE-ROI 1
Honourcode, Inc.
Toward an Ontology for Measuring Systems Engineering
Return on Investment
Advancing the knowledge of systems engineering
Eric Honour+1 (850) [email protected] Dr. Ricardo Valerdi+1 (617) [email protected]
Version
Toward Ontology for SE-ROI 2Honourcode, Inc.
Topics
SE-ROI Project “Ontology” concept COSYSMO work toward ontology Categorization from current standards Future directions
Toward Ontology for SE-ROI 3
Honourcode, Inc.
Systems Engineering Return on Investment
Summary of the SE-ROI Project
Toward Ontology for SE-ROI 4Honourcode, Inc.
Heuristic Claim of SE
Better systems engineering leads to Better system quality/value Lower cost Shorter schedule
SYSTEMDESIGN
DETAILDESIGN
PRODUCTIONINTEGRATION TEST
Traditional Design
Time
Risk
SavedTime/Cost
“System Thinking” Design Time
Risk
Toward Ontology for SE-ROI 5Honourcode, Inc.
0.4
0.6
0.8
1.0
1.2
0% 4% 8% 12% 16% 20% 24%
SE Effort
Dev
elo
pm
ent
Qu
alit
y(C
ost
/Sch
edu
le B
ased
)
Value of SE – 2004 Results
Problems/challenges: Quantitative data on SE not
available in program databases All data points were subjective
Detailed structure not availableSource: SECOE 01-03INCOSE 2003
Value = 1.0 if program met cost/schedule goals
Greater SE Effort led to better cost/schedule compliance and better predictability
Each dot is one program, with sizes between $1M and $6.5B
Toward Ontology for SE-ROI 6Honourcode, Inc.
SE-ROI Project
Interviews• Just-completed programs•Key PM/SE/Admin•Translate program data
into project structure
•Program characterization•Program success data•SE data (hours, quality,
methods)
Statistical correlation
Desired Results1. Statistical correlation
of SE methods with program success.
2. Leading SE indicators that can be used during a program.
3. Identification of good SE practices under different conditions.
0.6
1.0
1.4
1.8
2.2
2.6
3.0
0% 4% 8% 12% 16% 20% 24%
SE Effort = SE Quality * SE Cost/Actual Cost
Act
ual
/Pla
nn
ed S
ched
ule
Toward Ontology for SE-ROI 7
Honourcode, Inc.
“Ontology” Concept
What is this word and how does it relate to systems engineering?
Toward Ontology for SE-ROI 8Honourcode, Inc.
Current State of SE Definition
Fragmented by domain opinions
•Military – DOD/MOD•Space - NASA/ESA•Commercial products•Aircraft•Automobiles•Nuclear waste•Process engineering•Tool vendors•Etc. Etc. Etc.
Fragmented by discipline opinions
•Technical leaders•System architects•System analysts•Requirements
engineers•Operations analysts•Design engineers
Fragmented by standards
•ANSI/EIA-632• IEEE-1220• ISO-15288•CMMI•MIL-STD-499C
Toward Ontology for SE-ROI 9Honourcode, Inc.
Ontology
“…“…a branch of metaphysics concerned with a branch of metaphysics concerned with the nature and relations of being”the nature and relations of being”
functions
inputs
outputs
structure
understanding
aesthetics
components
interfaces
• POSIWID – The Purpose of a Systems Is What It Does - Jack Ring
• The purpose of systems engineering is different in the eyes of different people, because they perceive different actions/results from SE
methods
categories
Toward Ontology for SE-ROI 10Honourcode, Inc.
Purpose of this Paper
Explore the variety of what people see in SE
Formulate some general categories Interpret historical SE effort data Provide a structure for the data-gathering
in the SE-ROI project.
Toward Ontology for SE-ROI 11
Honourcode, Inc.
COSYSMO work toward ontology
An exploration of ontology as performed in the COSYSMO project
Toward Ontology for SE-ROI 12Honourcode, Inc.
COSYSMO Effort Profile
How is Systems Engineering effort distributed over time?
Phase Conceptualize DevelopOperational
Test & Eval
Transition to
Operation
%Effort(STDEV)
23 (12) 36 (16) 27 (13) 14 (9)
Toward Ontology for SE-ROI 13Honourcode, Inc.
Effort Distribution Across ANSI/EIA 632 Fundamental Processes
ANSI/EIA 632 Fundamental Process
AverageStandard Deviation
Acquisition & Supply 7% 3.5
Technical Management 17% 4.5
System Design 30% 6.1
Product Realization 15% 8.7
Technical Evaluation 31% 8.7
Toward Ontology for SE-ROI 14Honourcode, Inc.
Systems Engineering Effort Profile
Toward Ontology for SE-ROI 15
Honourcode, Inc.
Categorization from Current Standards
The start of an ontology, by identifying the widely-accepted categories.
Toward Ontology for SE-ROI 16Honourcode, Inc.
Categories in the Standards
Mission/Purpose Definition Requirements Management System Architecting System Implementation Technical Analysis Technical Management/Leadership Verification & Validation
ANSI/EIA-632
IEEE-1220
ISO-15288
CMMI
MIL-STD-499C
Colored boxes on following slides show the terminology used by each standard
Toward Ontology for SE-ROI 17Honourcode, Inc.
Mission/Purpose Definition
Define the mission or purpose of the new/changed system. Typically described in the language of the system
users rather than in technical language
ANSI/EIA-632•Not included
IEEE-1220•Define customer
expectations ISO-15288•Stakeholder
needs definition
CMMI•Develop customer
requirements
MIL-STD-499C•Not included
Toward Ontology for SE-ROI 18Honourcode, Inc.
Requirements Management
Creation and management of requirements Efforts to define, analyze, validate, and manage
the requirements
ANSI/EIA-632•System design•Requirements
definition
IEEE-1220•Requirements
analysis ISO-15288•Requirements
analysis
CMMI•Requirements
development•Requirements mgmt
MIL-STD-499C•System requirements
analysis and validation
Toward Ontology for SE-ROI 19Honourcode, Inc.
System Architecting
Define the system in terms of its component elements and their relationships Diagrams that depict the system, its
environment, components, and relationships
IEEE-1220•Synthesis
ISO-15288•Architectural design•System life cycle mgmt
CMMI•Technical solution ANSI/EIA-632
•System design•Solution definition MIL-STD-499C
•System product technical requirements analysis and validation
•Design or physical solution representation
Toward Ontology for SE-ROI 20Honourcode, Inc.
System Implementation
Development/completion of the system Specific system-level efforts in the standards are
system integration and transition to use
IEEE-1220•Not included
ISO-15288• Implementation• Integration •Transition
CMMI•Product integration
ANSI/EIA-632•Product realization• Implementation•Transition to use
MIL-STD-499C•Not included
Toward Ontology for SE-ROI 21Honourcode, Inc.
Technical Analysis
System-level technical analysis Assessment of system performance Trade-off analysis of alternatives
IEEE-1220•Functional analysis•Requirements trade
studies and assessments•Functional trade studies
and assessments •Design trade studies and
assessmentsISO-15288•Requirements analysis
CMMI•Measurement and analysis
ANSI/EIA-632•Technical evaluation•System analysis
MIL-STD-499C•Functional analysis,
allocations and validation•Assessments of system
effectiveness, cost, schedule, and risk
•Tradeoff analyses
Toward Ontology for SE-ROI 22Honourcode, Inc.
Technical Management/Leadership
Guiding the engineering teams involved in system design programs Size/complexity of teams demands leadership
IEEE-1220• Technical mgmt• Track analysis data• Track requirements
and design changes• Track performance• Track product metrics• Update specifications• Update architectures• Update plans• Maintain database
ISO-15288• Planning, Assessment,
Control• Decision mgmt• Config mgmt• Acquisition, Supply• Resource mgmt • Risk mgmt
CMMI• Project planning• Project monitoring & control• Supplier agreement mgmt• Process/product quality assur.• Configuration mgmt• Integrated project mgmt• Decision analysis/resolution• Quantitative project mgmt • Risk mgmt
ANSI/EIA-632• Technical Mgmt• Planning• Assessment• Control MIL-STD-499C
• Planning• Monitoring• Decision making, control,
and baseline maintenance• Risk mgmt• Baseline change control• Interface mgmt• Data mgmt• Subcontract mgmt• Technical reviews/audits
Toward Ontology for SE-ROI 23Honourcode, Inc.
Verification & Validation
Verification: comparison of the system with its requirements through objective evidence.
Validation: comparison of the system or requirements with the intended mission
IEEE-1220•Requirement verification•Functional verification•Design verification
ISO-15288•Verification•Validation•Quality mgmt
CMMI•Verification•Validation
ANSI/EIA-632•Technical Evaluation•Requirements validation•System verification•End products validation MIL-STD-499C
•Design or physical solution verification and validation
Toward Ontology for SE-ROI 24
Honourcode, Inc.
Future Directions
Where is SE-ROI going?
Toward Ontology for SE-ROI 25Honourcode, Inc.
Project Advisory Group
Group of interested people/organizations Communicate via web, telecon, meetings Help define the data organization Build public interest in the project Provide access to real programs View interim (protected) data as it develops
AF Institute of TechnologyNorthrop GrummanDOD Office of Secy of DefenseDRSJohns Hopkins UnivMITThe Mitre Corp
NAVAIRRaytheonRand CorporationRafaelSystems & Software ConsortiumUniv of South AustraliaUSN Chief Engineer
Current members come from:
For information, see http://www.hcode.com/seroi/
Toward Ontology for SE-ROI 26Honourcode, Inc.
Summary
Systems engineering current state of knowledge is fragmented
Broadly-accepted ontology is needed SE-ROI project needs categorization now
Structure the data to be correlated Discover leading SE indicators Identify SE best practices and methods
Categorization across standards helps develop the needed ontology
Toward Ontology for SE-ROI 27
Honourcode, Inc.
Questions?
Eric Honour+1 (850) [email protected] Dr. Ricardo Valerdi+1 (617) 253-8583
For information, see http://www.hcode.com/seroi/For information, see http://www.hcode.com/seroi/
Toward Ontology for SE-ROI 28Honourcode, Inc.
Survey of SE-ROI Knowledge
1992 Gruhl (NASA)
Project Definition – NASA
Program definition 10-15% reduces cost overruns
1990 Ancona/ Caldwell
Boundary Management Study
Boundary management averages 14%; more is better
2000 Miller (MIT)
Large Engineering Projects Study
Programs value cost over schedule over tech Leadership important
1995 Franz (Boeing)
Impact of Systems Engineering on Quality and Schedule
Better SE led to significant cost reduction
2003 Barker (IBM)
Systems Engineering Effectiveness
Better SE reduced parametric costs by 30%
2004 Kludze (NASA)
Impact of Systems Engineering on Complex Systems
General belief that SE improves program cost
2004 Honour (SECOE)
Value of Systems Engineering
Greater SE 10-15% reduces cost/schedule overruns
Toward Ontology for SE-ROI 29Honourcode, Inc.
Next Steps
Define interview data sheets Use this categorization
Identify and interview trial projects Obtain initial data Evaluate the interview data sheets
Identify and interview projects Ongoing effort for 2-3 years
Perform statistical correlation work Ongoing effort for duration of project Interim reports to participating organizations
Final report expected 2009