assuring mission success in complex settings

© 2006 Carnegie Mellon University

Assuring Mission Success inComplex Settings

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Christopher Alberts and Audrey DorofeeMarch 2007

2Assuring Mission Success in Complex SettingsAlberts and Dorofee, March 2007© 2006 Carnegie Mellon University

Contents / Agenda

Background

SEI MOSAIC: Managing for Success

SEI MOSAIC Project

• SEI MOSAIC Toolkit

• Mission Diagnostic

• MAAP

Conclusion


Managing Complexity

Managers are responsible for overseeing increasingly complex projects,programs, and operational processes.

• Multiple points of management control

• Complex tasks

• Complex, distributed supporttechnologies

• Multiple, detailed status reports

• A variety of managementtechniques (project, security,financial, technology, etc.)

• Requirements of multiple stakeholders


Need for a New Approach

Traditional analysis and managementapproaches not designed for complexenvironments

• Cannot handle organizational andtechnological complexity

• Do not easily scale to distributedenvironments

Need new methods, tools, andtechniques to

• Position projects, programs, andprocesses for success

• Establish and maintain confidence inachieving objectives


Managing for Mission Success

Managing for mission success requires establishing and maintaining areasonable degree of confidence that a mission’s objectives will besuccessfully achieved.

Declare, Respond to , and Close Incidents

QA

/Tec

h W

rite

rsC

ontra

cto

r M

ana

ger

Fie

ldO

ther

G

rou

psIT

Gov

ernm

ent

M

ana

ger

Inci

dent

Re

spon

se

Te

amH

elp

Des

k

Sh

ift L

ead

s

Se

nio

r W

atch

O

ffice

rs

Wat

ch

Off

icer

s H

elp

De

sk

Sec

urity

Tie

r 2

Hel

p D

esk

Sec

urity

Tie

r 1

Check document

quality

Approve release to

field

Approve release to

field OR

Recommend incident

XOR

Monitor status

Approve recommendation

Declare incident

XOR

XOR

Respond to incident

Respond to incident

Respond to incident

Respond to incident

Respond to incident

Close incident

Communicate with IM Team

XOR

Respond to incident

Respond to incident

Respond to incident

Respond to incident

Conduct data call

Implement actions and report status

To Incident Response Team: Respond to event (on Detect,

Triage, and Respond to Events)

This indicates a coordinated response to an incident led by

Incident Response Team.

Obtain approval to

release

From Incident Response Team: Respond to event (on Detect,

Triage, and Respond to Events)

If approved

If not approved

If declared

If not declared

If communication or data call

If not approved

If approved

If not approved

If data call approved

C DIf IM Team resolved incident and directs field to implement actions

If communication approved

Send communication

If IM Team resolved incident and implemented actions

Note: Communications to the field include bulletins , advisories, and alerts, etc.

Note: This workflow is a snapshot, based on available information

Respond to incident

Implement actions

Software and SystemDevelopment

Detect, Triage, and Respond to Events

IRC

M

anag

erQ

A/T

ech

Writ

ers

Fiel

dO

ther

G

roup

sIT

Gro

upG

over

nmen

t M

anag

erIn

cide

nt R

espo

nse

Tea

mC

all C

ente

r S

hift

Lead

sW

atch

O

ffice

rs

Sen

ior.

W

atch

O

ffice

rs

Cal

l Cen

ter T

ier 2

Call

Cen

ter

Tier

1

XOR

Reassigned outs ide IRC

Receive call

Monitor queue

Monitor queue

Receive email

Triage and respond t o

eventXOR

Report events to CC

Triage and respond to

eventXOR


eventXOR

Respond to event

Respond to event

Respond to event

Send communication

Conduct data c all

Check doc ument

quality

Approve release to

field

Approve releas e to

field

XOR

XOR

Monitor events and indicators

Triage event

XOR


Monitor data c all status

XOR

Monitor open source

information

From Gov ernment Manager or Center (on Declare, Respond to,

and C lose Incidents)

From Call Center Tier 1 , Tier 2, or Shift Leads

Implement actions

Reas signed outside IRC

Resolved event

Resolved event

Implement actions and close ticket

Obtain approval to

releas e

To F ield: Implement actions and close t icket

To F ield: Implement actions and close t icket

Reassigned outside I RC

Resolved ev ent

if reassigned outs ide of IM process

If resolved and no action required

If escalatedIf assigned to Tier 1 queue

If assigned to T ier 2 queue

If handled

If escalated to Shift Leads

If res olved and no action required

If escalated to service groups (IR Team, IT

Group)


If not approved

If not approved

If approved

If potent ial incident

If comminication or dat a call

If esc alated to IR Team

If assigned to Call Center T ier 2

If resolved and field action required


If res olved and field ac tion required

If res olved and field act ion required

To Field: Implement actions and close ticket

To IR Team: Recommend Incident (on Declare, Respond to, and Close Inc idents)

Res olved ev ent


If resolved and f ield action required

If communicat ion approved

A

A

A

D B

If escalated to IR Team

A

Note: This work flow is a snapshot, bas ed on available informat ion

Note: Communications to the field include bul letins , advis ories, and alerts, etc.

If es calated to other groups

From IR Team, Center mgmt, or government mgmt. This triggers other groups to respond t o the event

(e. g., IG, law enforcement)

C

To Other Groups : Respond to event (e.g., IG, law

enforcement, other center or government groups )

If escalated to other groups

B

if reassigned outside of IM process

if reassigned outs ide of IM process

Produces Systems andInfrastructures

Operational Mission

Su

pp

ort

s

`

Data

Internet


SEI MOSAIC:Managing for Success


Overview

SEI Mission-Oriented Success Analysis and Improvement Criteria(MOSAIC) is a structured decision-making approach that

• Establishes a reasonable degreeof confidence in the potential fora successful mission

• Helps ensure mission successin projects, programs,processes, and systems


Strategic Allocation of Resources

People need a way to make appropriate tradeoffs among a broadrange of factors.

Broad Tradeoff Space

Performance

Dependability

Interoperability Safety

Security Quality


SEI MOSAIC: A Lifecycle Approach

Perform during any lifecycle phase

Supports most system lifecyclemodels

RequirementsAnalysis

DesignPlanning

DevelopmentActivities

Testing/Integration

Release/Production

ConceptExploration

StrategyEvaluation

Operations/Maintenance


Managing the Outcome

An outcome is the result achieved whenexecuting a mission.

• A range of potential outcomes is possible

• Some outcomes are acceptable—success

• Some outcomes are unacceptable—failure

SEI MOSAIC defines an approach formanaging the expected outcome in relationto the desired outcome.

• What is the mission likely to achieve?

• What do I want the mission to achieve?


Range of Potential Outcomes

MissionActivities

Potential Events

Current Conditions

Range of Potential Outcomes

1

2

3

4

5


Positioning for Success

A range of outcomes is possible for any givenmission.

Conditions and potential events

• affect mission execution and influence amission’s eventual outcome

• must be appropriately managed to positiona mission for success

The objective is to drive the expectedoutcome toward acceptable states.

1

2

3

4

5

AcceptableOutcomes

UnacceptableOutcomes


Unique Features of SEI MOSAIC

Broad scope(distributed processes, systems of systems)

Interrelated view of risk

Outcome-driven

Opportunity seeking

“Playing to win”

Narrow scope(single project, system, or organization)

Linear view of risk(cause-effect pairs)

Threat-driven

Hazard avoidance

“Playing not to lose”

Traditional Risk Management SEI MOSAIC


SEI MOSAIC Project


Characteristics of Current Approaches

A prevalence of one-size-fits-all analysis and management methods

• Complex solutions that are not easily tailored (especially to smallorganizations)

• Tied to specific domains or problems

Locally optimized results

• Narrow tradeoff space

• Subset of the lifecycle

• Narrow scope (e.g., single project, system, or organization)


SEI MOSAIC Approach

Each SEI MOSAIC method is tailored to

• A given situation, problem space, or lifecycle phase

• The domain or application area

• The circumstances at hand

SEI MOSAIC is focused on global effectiveness and mission success.

• Broad tradeoff space

• Lifecycle focus (development and operations)

• Broad scope (e.g., distributed processes, supply chains, systemsof systems)


SEI MOSAIC Toolkit

Financial SectorMilitary and Defense Complex MissionsMedical Sector


DesignPlanning


Testing/Integration

Release/Production

ConceptExploration

StrategyEvaluation


SEI MOSAIC Toolkit


SEI MOSAIC Methods

Our current work is focused on developing a suite of analysis methods.

Two methods so far:

• Mission Diagnostic is a basic approachthat provides a quick, high-levelevaluation.

• Mission Assurance Analysis Protocol(MAAP) is a comprehensive approachthat provides an in-depth evaluation.


Mission Diagnostic

WhatA time-efficient means of assessingthe potential for success

WhyTo determine whether conditions arefavorable for a successful outcome

Key ResultsAn evaluation of key indicators and anestimate of the success potential


Key Indicators

Evaluate a set of indicatorsrepresenting key aspects ofmanagement, for example:

• Realistic goals

• Customer requirements

• Staffing requirements

• Technology feasibility

• Plans and schedules

“Are customer requirements and needs well understood?”


Evaluating Key Indicators

1. Are goals realistic andwell articulated?

Question Answer

YesNo Likelyyes

Equallylikely

Likelyno

q q q n q

Each indicator is evaluated based on the data that havebeen collected.

Uncertainty is incorporated into the range of answers foreach indicator.


Indicator Evaluation Criteria

Answer

Yes

Likely yes

Equally likely

Likely no

No

Definition

The answer is almost certainly “yes.” Very little uncertaintyexists.

The answer is most likely “yes.” However, a degree ofuncertainty exists.

The answer is just as likely to be “yes” or “no.” A highdegree of uncertainty exists.

The answer is most likely “no.” However, a degree ofuncertainty exists.

The answer is almost certainly “no.” Very little uncertaintyexists.


Indicator Analysis

A simple analysis provides insight intoa mission’s health.

Ind

icat

or

1

Ind

icat

or

2

Ind

icat

or

3

Ind

icat

or

4

Ind

icat

or

5

Ind

icat

or

6

Ind

icat

or

7

Ind

icat

or

8

Ind

icat

or

9

Ind

icat

or

10

Yes

Likely yes

Equally likely

Likely no

No

Uncertainty Line


Managing the Potential for Success

Excellent

Good

Borderline

Poor

Critical

Success Threshold

Current State Desired State

Mis

sio

n H

ealt

hThe goal is to improve a mission’s current state of health.


Indicators for Software Development Programs

• Are goals realistic and wellarticulated?

• Are communication andinformation sharing aboutmission activities effective?

• Are customer requirements andneeds well understood?

• Are stakeholder politics or otherexternal pressures minimal?

• Does the process designsupport efficient and effectiveexecution?

• Are process controlmechanisms are effective?

• Is task execution efficient andeffective?

• Are staffing and fundingsufficient to execute all missionactivities?

• Are the technological andphysical infrastructuresadequate to support all missionactivities?

• Are changing circumstancesand unpredictable eventseffectively managed?


Evaluating Indicators

The following data are recorded foreach indicator:

• Indicator score

• Rationale for indicator score

• Analysis approach(for example, intuition,qualitative analysis,quantitative analysis, other)

• Potential actions

• Evaluators

• Date


Mission Diagnostic Exercise and Handout


Tailoring Questions

The following questions can be used when tailoring or developing a setof indicators:

• What constitutes a successful result for the project or process?

• What constitutes an unsuccessful result, or failure, for the projector process?

• What circumstances or conditions tend to produce a successfuloutcome when conducting the project or process?

• What circumstances or conditions tend to produce anunsuccessful outcome, or failure, when conducting the project orprocess?


Mission Diagnostic Across the Lifecycle

How much uncertainty in these indicators can you tolerate at differentpoints in the lifecycle?


DesignPlanning


Testing/Integration

Release/Production

ConceptExploration

StrategyEvaluation



MAAP

WhatA systematic approach for thoroughlyanalyzing the potential for success

WhyTo characterize the full range of driversaffecting the success potential

To set management priorities to ensurethe success potential is maintainedwithin tolerance

Key ResultsAn operational model, customized analysis artifacts, a measure of thesuccess potential, and strategies for keeping the success potential withintolerance


Operational Model of Mission Activities

Detect, Triage, and Respond to Events

IRC

M

anag

erQ

A/T

ech

Writ

ers

Fiel

dO

ther

G

roup

sIT

Gro

upG

over

nmen

t M

anag

erIn

cide

nt R

espo

nse

Tea

mC

all C

ente

r S

hift

Lead

sW

atch

O

ffice

rs

Sen

ior.

W

atch

O

ffice

rs

Cal

l Cen

ter T

ier 2

Cal

l Cen

ter Ti

er 1

XOR

Reassigned outside IRC

Receive call

Monitor queue

Monitor queue

Receive email


eventXOR

Report events to CC


eventXOR


eventXOR

Respond to event

Respond to event

Respond to event

Send communication

Conduct data call

Check document

quality

Approve release to

field

Approve release to

field

XOR

XOR

Monitor events and indicators

Triage event

XOR


Monitor data call status

XOR

Monitor open source

information

From Government Manager or Center (on Declare, Respond to,

and Close Incidents)

From Call Center Tier 1, Tier 2, or Shift Leads

Implement actions


Resolved event

Resolved event

Implement actions and close ticket

Obtain approval to

release




Resolved event



If escalatedIf assigned to Tier 1 queue

If assigned to Tier 2 queue

If handled

If escalated to Shift Leads


If escalated to service groups (IR Team, IT

Group)


If not approved

If not approved

If approved

If potential incident

If comminication or data call


If assigned to Call Center Tier 2






To IR Team: Recommend Incident (on Declare, Respond to, and Close Incidents)

Resolved event



If communication approved

A

A

A

D B


A

Note: This workflow is a snapshot, based on available information

Note: Communications to the field include bulletins , advisories, and alerts, etc.


From IR Team, Center mgmt, or government mgmt. This triggers other groups to respond to the event

(e.g., IG, law enforcement)

C

To Other Groups: Respond to event (e.g., IG, law

enforcement, other center or government groups )


B




Drivers of Success and Failure

A broad range of driversmust be considered whenanalyzing the potential formission success.

Activity

Event

SuccessFailure

Environment

Mission

Design


Mission

A mission threat is a fundamental flaw, or weaknesses, inthe purpose and scope of a work process.


Process Design

A design threat is an inherent weakness in the layout of awork process.


Activity Management

An activity threat is a flaw, or weaknesses, arising from themanner in which activities are managed and performed.


Operational Environment

An environment threat is an inherent constraint, weakness,or flaw in the overarching operational environment in which aprocess is conducted.


Event Management

An event threat is a set of circumstances triggeredby an unpredictable occurrence that introducesunexpected change into a process.

Event

Event


Scenario-Based Analysis

Expectedoutcome

Scenario 1Expected operationalconditions

Scenario 2When stressed byEvent 1

Scenario 3When stressed byEvent 2

Event 1

Event 2

Outcomeresulting fromEvent 2

Outcome duringexpected operationalconditions

Outcomeresulting fromEvent 1


Complex Risks

Sites don’t notify CIRC when performing internal

scans

Limited backup capability for IDS

tuning

IDS tools inherently provide false positive

Reliance on pre -existing KSAs

Best person for job not always selected

Limited time and opportunity to stay

current

Inadequate equipment for online training

Heavily reliant on on-the-job training

Lack of comprehensive , cross,

and QA training

Training is informal and based on mentoring

IR team has too many tasks relative to

number of staffAll security events go

to IR team

Insufficient tools to support IR tasks

Difficult to find qualified staff

R8False positives could be

forwarded by Watch Office

Watch Office staff have uneven skills for recognizing false

positives

Inadequate training program

IR team is bottleneck

R4Events could be

escalated unnecessarily by Call Center

R8False positives could be

forwarded by Watch Office

R20Understaffing could lead to quality and response

time problems

Inadequate staffing

IDS tools provide false positives

Inadequate and inefficient tuning of IDS tools exacerbates false

positives


Outcome Analysis

The goal is to ensure that the expected outcome for each objective inall evaluated scenarios is acceptable to key stakeholders.

Best Outcome

Success threshold for cost

ProductObjective

Gap between current anddesired states for cost

Worst Outcome

ScheduleObjective

CostObjective …


Unique Features of SEI MOSAIC

• Manages the potential for success

• Can be applied to highly distributedprograms and operationalprocesses

• Provides a ‘global’ view of a mission

• Analyzes issues that are toocomplex for other techniques


Potential Application Areas

• Large, distributed software development programs

• Organizations in dynamic, rapidly changing business environments

• Organizations with strict reliability, security, and safety requirements

• Large, distributed supply chains

• Processes supporting critical infrastructures

• Distributed information-technology (IT) processes


Future Research and Development

Refine the current SEI MOSAIC analysisprotocols.

Define and pilot additional SEI MOSAICanalysis protocols.

Begin work on an approach for real-timemonitoring and management of missionoutcomes.


For Additional Information

Telephone 412 / 268-5800

Fax 412 / 268-5758

WWW http://www.sei.cmu.edu/msce/

U.S. mail Customer RelationsSoftware Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213-3890

assuring mission success in complex settings

Documents