risk management in system of systems development – are you ready? dr. peter hantos senior...

Risk Management in System of Systems Development – Are You Ready?

Dr. Peter Hantos

Senior Engineering Specialist

Software Acquisition and Process Department

© 2005. The Aerospace Corporation. All Rights Reserved.

20th International Forum on COCOMO and Cost Modeling

COCOMO 2005 – Peter Hantos Slide 2

Use of Trademarks, Service Marks and Trade Names

Use of any trademarks in this material is not intended in any way to infringe on the rights of the trademark

holder. All trademarks, service marks, and trade names are the property of their respective owners.


Acknowledgements

• This work would not have been possible without the following: Reviewers

– Suellen Eslinger, Software Engineering Subdivision

– Dr. Leslie J. Holloway, Software Engineering Subdivision

Sponsor– Michael Zambrana, USAF Space and Missile Systems

Center, Directorate of Systems Engineering

Funding source– Mission-Oriented Investigation and Experimentation (MOIE)

Research Program (Software Acquisition Task)

Inspiration– Dr. Barry W. Boehm, University of Southern California


Agenda

1. Objectives

2. Are System of Systems (SoS) Different from Software Intensive Systems (SIS)?

3. The Importance of Risk Management in SoS

4. Overview of Risk Management Risk Classes

5. CMMI® (Capability Maturity Model Integration) Concerns

6. Summary

__________________® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University


Objectives

• Emphasize the importance of Risk Management in System of Systems development

• Introduce a top-level taxonomy of Risk Management risk classes

• Present examples that demonstrate the applicability of these risk classes to System of Systems development


Are SoS’ Different from SIS’?

• The Jury is Still Out Note different presentations at this conference Note the cost estimation community’s difficulties in

clarifying the differences between COSYSMO and COSOSIMO cost drivers

• Nevertheless, in my opinion: By-and-large there are no substantial differences

– Most of the “new” difficulties are consequences of the scale and diversity, factors that were always relevant for SIS

– Suggested Metaphor: SoS’ are SIS’ on steroids – Steroids help to acquire bulk and enhance performance

– The single, key discriminator is the presence of emergent characteristics of SoS


The Importance of Risk Management in SoS

• Emergence: Emergence refers to unexpected global system properties*

– In Evolution Theory, emergence is the rise of a system that cannot be predicted or explained from antecedent conditions

• SoS developments pursue new, desirable capabilities via the integration of systems The concern is the emergence of undesirable unexpected system

properties This presents the classic Risk Management challenge:

– Balancing opportunities with risks

• Prudent Risk Management practices are still effective In reality, most of the undesirable side-effects of System of Systems

development are predictable– Using again the steroid metaphor:

– Steroids help to acquire bulk and enhance performance, but they also bring in undesirable side-effects

– The side-effects of steroids are well known, but their users either ignore them or they act surprised because of lack of education.

________* Source: Encyclopedia Britannica


Overview of Risk Management Risk Classes

Hypothesis:The following 5 risk classes provide a systematic way to

organizing risks related to the Risk Management process, and

represent a consistent framework to identify, analyze risks and

to develop mitigation solutions.

1. Organizational culture

2. Risk Management practices

3. Organizational structure

4. Risk Management structure

5. Contractual structurePlease note that while the risk classes are meant to be orthogonal

by design, actual risks have complex interdependencies.Please note that while the risk classes are meant to be orthogonal

by design, actual risks have complex interdependencies.


Organizational Culture

• It has one of the most far-reaching consequences on the efficacy of Risk Management and overall program success Intricate relationship with many risk factors

• Risk Management is an evolutionary asset The organization’s culture has to embrace investments that only

yield rewards in the long haul

• Selected examples of problematic cultural behavior [Lister02] Authoritarian management

– “Don’t bring me problems, bring me only solutions!”

Superstition/fatalism– “Risk Management is a charade, and one cannot control fate”

Maintaining a risk list but doing nothing– “… do nothing till the problem is on the prowl”

High-speed, frantic world– “winner-take-all business mentality”


Organizational Culture (Cont.)

• Risk Management Philosophy True buy-in or lip service to Risk Management? The healthy approach is opportunity-driven Risk Management

– Typical aberrations:– Risk avoidance at all cost– Equating Risk Management with Crisis Management

What is the common belief of the executive management with respect to who knows what the risks are?

– They themselves (… and they don’t need anybody to tell them )– Outsiders - RAB (Risk Advisory Board), or – The people on the program - SEI/SRE (Software Engineering

Institute/Software Risk Evaluation) Approach Frequency of risk-related activities

– Infrequent – RAB and SRE solutions only– Infrequent – activities only when they can not be avoided

– E.g., mandated by policy at Key Decision Points of the Life Cycle [NSSAP03]

– Frequent – Continuous Risk Management


Risk Management Practices

• Existing contractor Risk Management practices The contractors’ existing Risk Management practices will have

a major impact on the overall effectiveness of the SoS Risk Management program

There is an extensive literature on Risk Management Practice related concerns [See Conrow02, Conrow04, and Shepherd03]

– Nevertheless, most of these concerns are not scaling or diversity related (e.g., improper rating of risks, misuse of quantitative risk analysis methods, etc.); hence we will not cover them here

• Due to time limitations, we present only the following two concerns that are relevant in the SoS context:

– Risk list is established at the beginning of the project (For example an SRE is conducted) but the risks are not monitored and the list is never updated

– This is a critical issue due to the dynamic nature of SoS

– Top 10 Risk List is established, but there are various deficiencies in the way risks are handled

– Specific issues are discussed later


Organizational Structure

• The prevailing organizational structure in large-scale acquisitions is IPT (Integrated Product Team) Selected, noble objectives in creating IPTs [Kaminski95]

– Focus on program success not on functional area performance

– Continuous insight instead of oversight

– Stakeholder collaboration providing workable suggestions and practical solutions

Nevertheless, some IPT caveats exists, particularly from software perspective– The typical IPT structure

– Fragments the technical effort– Does not provide a software-trained management structure– Does not place software at a level commensurate with its

risk


Example Top-10 Risk Lists for Satellite System Development

Legend for Risks: High (H) Medium (M) Low (L)

Legend for BD/MS (Burn-Down/Milestone Status): >1 Month late (V) <1 Month late (L) On Plan (O)

GROUND SEGMENT

# RISK BD/MS DESCRIPTION AREA

1 H L Long-term planning of Ground Segment Lab resources Management Processes

2 H L No Multi-mission XXXX Processor Sustainment Lab is planned Management Processes

3 H L Performance impacts on Ground Station during Phase 1 to Phase 2 transition Management Processes

4 H O Multi-mission XXXX Processor obsolescence upon delivery COTS

5 H O Inadequate test resources for interim deliverable XXXX module Management Processes

6 H O Lack of proactive Ground Segment staffing process Management Processes

7 H O No plan for the Special Test Equipment upgrade COTS

8 H O No external interfaces into the Contractor’s Development Facility Management Processes

9 H O XXXX frequency pre-correction accuracy Performance

10 M V Inadequate schedule margin for Interim XXXX deliverable Management Processes

SYSTEM

# RISK BD/MS DESCRIPTION AREA

1 H V No scope for Ground Segment external deliverables Management Processes

2 H L Algorithm development of XXXX Performance

3 H O Operability/usability objectives for Human Systems Integration in Increment 2 Miscellaneous Other

4 M V Space-to-Ground Interface Control Definition revision is Work In Process Architecture

5 M V Interim XXXX Interdependency Uncertainty Management Processes

6 M V Process deficiency in managing Increment 2 Schedule Interdependencies Management Processes

7 M L XXXX combined waveform not in XXXX Test Plan Miscellaneous Other

8 M L XXXX translator certification to avoid data errors Miscellaneous Other

9 M O XXXX regulatory compliance Miscellaneous Other

10 M O Operation of a high-level scenario to meet special operational expectations Performance


Space Segment IPT

Program Level IPT

Ground Segment IPT

Systems Engineering and Integration Team

Typical Contractor Space System IPT Structure


Space Segment IPT

Program Level IPT

Ground Segment IPT


Ground Element 1 IPT

Hardware IPTs

Software IPTs

Ground Integration Team

...

Ground Segment IPT Structure


Space Segment IPT

Program Level IPT

Ground Segment IPT



Hardware IPTs

Software IPTs


Hardware IPTs

Processing Subsystem 1 IPT

Software IPTs

SpacecraftIPT

...

...

Space Segment/Spacecraft IPT Structure


Space Segment IPT

Ground Segment IPT



Hardware IPTs

Software IPTs


Hardware IPTs


Software IPTs


Hardware IPTs

Software IPTs

SpacecraftIPT

PayloadIPT

Space Integration Team

...

... ...

Program - Level IPT

Space Segment/Payload IPT Structure


Software IPTs (L4)

Software is Buried Deeply in the IPT Structure

Space Segment IPT (L2)

L1

Ground Segment IPT (L2)

- Software ProcessGround Element 1

IPT (L3)

Hardware IPTs (L4)

Ground Integration Team (L3)

Hardware IPTs (L5)

Processing Subsystem 1

IPT (L4)

Software IPTs (L5)


IPT (L4)

Hardware IPTs (L5)

Software IPTs (L5)

SpacecraftIPT (L3)

PayloadIPT (L3)


...

... ...

Program Level IPT (L1)


(L2)


Risk Management Structure

• Risk management must be applied at every level of the program’s organizational structure Risks must be managed at the level with the necessary

responsibility and authority to make decisions Software risk management must be defined and practiced at all

levels of the IPT hierarchy, by software personnel

• All components of the Risk Management structure have to be fully integrated Key caveats:

– “Risk Management is everybody’s responsibility”– Unfortunately, when something is everybody’s responsibility then

chances are that nobody will do anything

– Major disconnect might exist amongst– Risk Advisory Board

– Risk Management Boards

– Program Management/Risk-driven Spiral Planning


Risk Management Structure (Cont.)

• Acquirer vs. contractor risks Acquirer and contractor risks are not the same

– Even in case of total transparency, it has to be clear that the Acquirer Organization’s risks are not the same as the contractors’ risks; hence an independent, Acquirer Risk Management structure must be maintained as well

• FFRDC (Federally Founded Research & Development Center) risks Acquirer and FFRDC supporter risks are not the same

– In case of FFRDC support to the Acquisition Organization, it must be noted that the FFRDC’s risks are also different from the supported Acquisition Organization’s risks, and they have to be managed independently


Contractual Structure

• It is a key element of the overall acquisition context Contracts can calibrate and balance the necessary

insight/oversight– Contracts are the proper vehicles to enable the visibility into the

Contractors’ risks and Risk Management practices on a continuing basis

– Unless the contract specifies it, contractors will not necessarily volunteer early information on their risks

– Example follows to demonstrate the complexity of the space team

• CMMI The so-called “New CMM Math” is a special concern

– Specific issues are discussed later


Prime Contractor

Program OfficeContractor

InternalFormal

Agreement

Ground Processing

Development

Contracts

Commercial Customers

Commercial Sector

Government Sector

Commercial Organization

Contractincluding IMP (Integrated Master

Plan)

Government

Contractor InternalFormal

Agreement

(1) Government contracts with the Prime Contractor’s Government Program Office,

(2) Program Office contracts internally for developing the Ground Segment and some augmenting work with a Department in the Company’s Commercial Sector,

but, still might not have enough capacity to develop the complete Ground Segment and no capability to develop the Space Segment…

Example – The Complexity of the Space Team Structure


Space Example – Cont.


Software IPTs (L4)

Contractor Relationships Superimposed on the IPT Structure

Company CCompany C

Space Segment IPT (L2)

L1

Ground Segment IPT (L2)

- Software ProcessGround Element 1

IPT (L3)

Hardware IPTs (L4)

Ground Integration Team (L3)

Hardware IPTs (L5)


IPT (L4)

Software IPTs (L5)


IPT (L4)

Hardware IPTs (L5)

Software IPTs (L5)

SpacecraftIPT (L3)

PayloadIPT (L3)


...

... ...

Company BCompany B

Company ACompany A

Prime ContractorPrime Contractor

Program Level IPT (L1)


(L2)


Contracts in SoS Development with Lead System Integrator

Subcontractor Development

Contractor


InternalFormal

Agreement

Development

Contracts


Commercial Sector

Government Sector


Division A Division B

Program Officeand

Development

Subcontractor

External Subcontract


Agreement


Agreement

Subcontractor Development

Contractor


InternalFormal

Agreement

Development

Contracts


Commercial Sector

Government Sector


Division A Division B

Program Officeand

Development

Subcontractor

External Subcontract


Agreement


Agreement

Contract

Government

Subcontract Subcontract


CMMI® Concerns

• Good News: The CMMI provides effective coverage for many of the

critical concerns via selected Process Areas– RM (Risk Management)

– SAM (Supplier Agreement Management)

– ISM (Integrated Supplier Management),

– IT (Integrated Teaming)

– OEI (Organizational Environment for Integration)


CMMI® Concerns

• Bad News: Even in a truly mature organization the integration of these

Process Areas is a serious challenge

– “New CMM Math” [Ferguson02] 4+4=2 – Contractors’ processes might not be compatible even though

separately and individually they have received high maturity ratings– Particularly in the contracting area legal issues could hinder the contractors’

responsiveness to process change requests

– Contractors are not very eager to change their proven processes anyway

Reality Check– Primarily in Risk Management, in the various measurements-related

areas, and in defect tracking the existence of different but proven local standards and tools is highly probable

– The expectation for their standardization and unification in a dynamically operating SoS environment is not feasible

– Don’t keep identifying lack of process/tool standardization as a root cause

– Acknowledge continuous process harmonization as a key management objective


Summary

• A taxonomy of risk classes was presented as a systematic way to organizing risks related to the Risk Management process

• This taxonomy can also provide a consistent framework to identify, analyze risks, and develop mitigation solutions

• On the basis of the discussed examples we can conclude that System of Systems development risks are not different

from the already well-known Software-Intensive System development risks

Prudent Risk Management practices are not omnipotent “Silver Bullets” but still, highly effective in System of Systems development


Acronyms

CMM Capability Maturity Model CMMI Capability Maturity Model Integration

COSOSIMO Constructive System of Systems Integration Cost Model COSYSMO Constructive Systems Engineering Cost Model

FFRDC Federally Founded Research & Development Center IPT Integrated Product Team ISM Integrated Supplier Management

IT Integrated Teaming OEI Organizational Environment for Integration

RAB Risk Advisory Board RM Risk Management

SAM Supplier Agreement Management SIS Software-Intensive System

SoS System of Systems SRE Software Risk Evaluation


References

[Conrow02] Conrow, E. H., “Achieving Effective Risk Management by Overcoming Some Common Pitfalls”, Cutter IT Journal, February 2002, Vol. 15, No. 2

[Conrow04] Conrow, E. H., “The Top 10 Risk Analysis Issues and Why They Should Be Important to You”, Risk Management Intelligence Network Risk Rant, June 2, 2004

[Ferguson02] Ferguson, J., and Penn, M. L., “New CMM Math”, CMMI Conference, Denver, CO, November 14, 2002

[Kaminski95] Kaminski, P. G., “Integrated Teams: One Important Step Forward in Military Acquisition Affairs”, DoD IPT Conference, Woodbridge, VA, June 20, 1995

[Lister02] Lister, T., “Risk Management for Software and System Projects: Utterly Doomed”, Cutter IT Journal, February 2002, Vol. 15, No. 2

[NSSAP03] National Security Space Acquisition Policy 03-01, December 27 2004

[Shepherd03] Shepherd, B., “Managing Risk in a Program Office Environment”, Acquisition Review Quarterly, Spring 2003


Contact Information

Peter HantosThe Aerospace Corporation

P.O. Box 92957-M1/112Los Angeles, CA 90009-2957

Phone: (310) 336-1802Email: [email protected]

risk management in system of systems development – are you ready? dr. peter hantos senior...

Documents

peter hantos slide

importance of risk management

cost modeling slide

sos emergence

software intensive systems

missile systems center

steroids steroids

use of trademarks