perspectives on risk‐informed decision‐making · 2020. 10. 31. · ibex aura gems juno maven...

1

Perspectives on Risk‐Informed Decision‐MakingEnabling a Compelling Tomorrow

Michael A. JohnsonChief Technologist, Engineering and Technology DirectorateNASA Goddard Space Flight Center

[email protected]

2

NASA/SP-2010-576Version 1.0April 2010

Questions considered‐

• What is NASA’s view on risk and risk‐informed decision making?

• How does NASA incorporate risk insights into its day‐to‐day decision making?

• What are challenges and successes with integrating risk‐informed decision making at NASA?

3

National Aeronautics and Space Administration (NASA)capabilities are distributed across the Nation.

Background

4

Goddard Space Flight Center (GSFC) is distributed across five locations.

The main campus is in Greenbelt, MD.

GreenbeltGreenbelt

Goddard Institute for Space StudiesGoddard Institute for Space Studies

Independent Verification and Validation FacilityIndependent Verification and Validation Facility

Wallops Flight FacilityWallops Flight Facility

Goddard Space Flight Center

White Sands ComplexWhite Sands Complex

Background

5

TRACETRACE

ACEACE

SOHOSOHO

RHESSIRHESSI

WindWindVoyagerVoyager

GeotailGeotail

TIMEDTIMED

FASTFASTPolarPolar

StereoStereo

THEMISTHEMIS

IMAGEIMAGE

MMSMMSSolar‐BSolar‐B

QuikSCATQuikSCAT

ACRIMSATACRIMSAT

EO‐1EO‐1

COBECOBE

Landsat 7Landsat 7

TRMMTRMM

TDRSSTDRSS

AquaAqua

TerraTerra

CloudSatCloudSat

CALIPSOCALIPSO

GRACEGRACE

SORCESORCE

ICESatICESat

MessengerMessenger

CassiniCassini

New HorizonsNew Horizons

LROLRO

AquariusAquarius

RXTERXTE

ClusterClusterSDOSDO

NPPNPP

AIMAIM

LDCMLDCM

GPMGPM

TOMSTOMS

JWSTJWST

ComptonGRO

ComptonGRO

HSTHST

SpitzerSpitzer

SwiftSwiftFUSEFUSE

GALEXGALEX

FermiFermi

WMAPWMAP

Mars ScienceLaboratoryMars ScienceLaboratory

POESPOES

GOESGOES

WISEWISE

IBEXIBEX

AuraAura

GEMSGEMS

MAVENMAVENJunoJuno

LADEELADEE

RBSPRBSP

TWINS(Instrument)

TWINS(Instrument)

EUVEEUVE

SWASSWAS

NuSTARNuSTAR

IntegralIntegralIUEIUE

ERBSERBS

TOPEXTOPEX

Osiris‐Rex(Sample Return)


PioneerPioneer

GalileoGalileo

Astro‐HAstro‐H

5

GSFC has flown over 300 Heliophysics, Earth Science, Astrophysics, and Planetary missions

GSFC provides essential technical and scientific support to address national and world societal needs through our work with and for other Federal agencies.

We derive and share information, solutions, and technology for NASA, the Nation, and the world.

We engage with other NASA centers, universities, and industry in search of opportunities to blend our competencies with their specialized assets in order to obtain knowledge that can be applied to addressing some of the most profound and compelling questions of our time.

Background

6

GSFC has a history of mission success:

No spacecraft failures since 1975.

Success has been derived in part, by enforcing a mostly static set of Mission Assurance Requirements (MAR).

1

3

5

12

8

13

9 9

13

11

9

5

78

4

8 8

2

5

8

32

5

23

4

1 1 1

32

4 45

4

6

87

5 5 5

7

4

6

4

2 2

7

2

45

2

43

7

23

21

3

1

1

1

1

3

12

12

1959

1960

1961

1962

1963

1964

1965

1966

1967

1968

1969

1970

1971

1972

1973

1974

1975

1976

1977

1978

1979

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

Success Failure (due to LV) Failure (due to SC)

3

4

* GSFC does not develop or deliver launch vehicles

*

GSFC Mission Outcomes

301 SuccessesNumerous Launch Vehicle (LV) failures 0 Spacecraft (SC) failures since 1975

Background

7


• What is NASA’s GSFC’s view on risk and risk‐informed decision making?

• How does NASA GSFC incorporate risk insights into its day‐to‐day decision making?

• What are challenges and successes with integrating risk‐informed decision making at NASA GSFC?

8

It would be wrong to conclude that mission success derived from the MAR approach would obviate a need for change.

To effectively and efficiently meet today’s and tomorrow’s space exploration challenges, GSFC must not only make advancements in what we do; we must also advance how we do it.

1

3

5

12

8

13

9 9

13

11

9

5

78

4

8 8

2

5

8

32

5

234

1 1 1

32

4 454

6

87

5 5 5

7

4

6

4

2 2

7

2

45

2

43

7

232

1

3

1

1

1

1

3

12

12

Success Failure (due to LV) Failure (due to SC)

34

* GSFC does not develop or deliver launch vehicles

*

GSFC Mission Outcomes

What is GSFC’s view on risk and risk‐informed decision making?

9

Key Elements• Risk‐informed Framework• Risk‐informed Requirements Generation• Risk‐informed Decision Making• Risk‐informed Review and Audit

GSFC implements a Risk‐based Safety and Mission Assurance (SMA)‐ The process of applying limited resources to maximize the chance for safety and mission success by focusing on mitigating specific risks that are applicable to the project vs. simply enforcing a set of requirements because they have always worked.

Risk‐based Safety and Mission Assurance (SMA) began formal GSFC implementation in 2016

What is GSFC’s view on risk and risk‐informed decision making?

10

TRACETRACE

ACEACE

SOHOSOHO

RHESSIRHESSI

WindWindVoyagerVoyager

GeotailGeotail

TIMEDTIMED

FASTFASTPolarPolar

StereoStereo

THEMISTHEMIS

IMAGEIMAGE

MMSMMSSolar‐BSolar‐B

QuikSCATQuikSCAT

ACRIMSATACRIMSAT

EO‐1EO‐1

COBECOBE

Landsat 7Landsat 7

TRMMTRMM

TDRSSTDRSS

AquaAqua

TerraTerra

CloudSatCloudSat

CALIPSOCALIPSO

GRACEGRACE

SORCESORCE

ICESatICESat

MessengerMessenger

CassiniCassini

New HorizonsNew Horizons

LROLRO

AquariusAquarius

RXTERXTE

ClusterClusterSDOSDO

NPPNPP

AIMAIM

LDCMLDCM

GPMGPM

TOMSTOMS

JWSTJWST

ComptonGRO

ComptonGRO

HSTHST

SpitzerSpitzer

SwiftSwiftFUSEFUSE

GALEXGALEX

FermiFermi

WMAPWMAP

Mars ScienceLaboratory

Mars ScienceLaboratory

POESPOES

GOESGOES

WISEWISE

IBEXIBEX

AuraAura

GEMSGEMS

MAVENMAVENJunoJuno

LADEELADEE

RBSPRBSP

TWINS(Instrument)

TWINS(Instrument)

EUVEEUVE

SWASSWAS

NuSTARNuSTAR

IntegralIntegralIUEIUE

ERBSERBS

TOPEXTOPEX



PioneerPioneer

GalileoGalileo

Astro‐HAstro‐H

10

Developing space flight systems is GSFC’s “day‐to‐day”

• What is GSFC’s view on risk and risk‐informed decision making?

• How does GSFC incorporate risk insights into its day‐to‐day decision making?

• What are challenges and successes with integrating risk‐informed decision making at GSFC?

11

Ref: Jesse Leitner, Bhanu Sood, Eric Isaac, Jack Shue, Nancy Lindsey & JeannettPlante (2018) Risk‐based safety and mission assurance: Approach and experiences in practice, Quality Engineering, 30:4, 648‐662, DOI: 10.1080/08982112.2018.1473584

Printed circuit boards are a foundational component of most spaceflight electrical, electronic, and electromechanical systems

How do we assess the quality of the bare board and its manufacturing process?

How does GSFC incorporate risk insights into its day‐to‐day decision making?

Populated printed circuit board

Example: Printed Circuit Board Test Coupon Nonconformance Handling

12

Printed Circuit Board (PCB) PCB

PCB PCB

COUPON

COUPON

COUPON

COUPON

PCB PANEL

One PCB panel typically yields multiple bare PCBs. Test “coupons” inform the board/ fabrication process quality assessment.

Example: Printed Circuit Board Test Coupon Nonconformance HandlingHow does GSFC incorporate risk insights into its day‐to‐day decision making?

Populated printed circuit board

13

Traditional Approach: Evaluate test coupons to favored standard. Reject and refabricate all boards associated with nonconforming coupons.

Risk‐based Approach: Use requirements from one or more commonly‐used standards to define minimum requirements for a range of risk levels. Assess risk when nonconformance is found and determine whether risk is acceptable. Only refabricate if cause for nonconformance is understood and can be eliminated.

Risk‐based Results Summary: Out of 193 boards with nonconforming test coupons, over 85% had either no elevated risk or very low risk associated with the non‐conformance. The risk acceptance resulted in millions of dollars and hundreds of weeks of schedule saved relative to the traditional approach.

Copper wicking

Capped vias

Copper wicking and vias are two (of several) coupon elements evaluated during coupon analysis.


14

Traditional Approach: Evaluate test coupons to favored standard. Reject and refabricate all boards associated with nonconforming coupons.

Risk‐based Approach: Use requirements from one or more commonly‐used standards to define minimum requirements for a range of risk levels. Assess risk when nonconformance is found and determine whether risk is acceptable. Only refabricate if cause for nonconformance is understood and can be eliminated.

Risk‐based Results Summary: Out of 193 boards with nonconforming test coupons, over 85% had either no elevated risk or very low risk associated with the non‐conformance. The risk acceptance resulted in millions of dollars and hundreds of weeks of schedule saved relative to the traditional approach.

Copper wicking

Capped vias

Copper wicking and vias are two (of several) coupon elements evaluated during coupon analysis.


15


• What is GSFC’s view on risk and risk‐informed decision making?

• How does GSFC incorporate risk insights into its day‐to‐day decision making?

• What are challenges and successes with integrating risk‐informed decision making at GSFC?

16

What are challenges and successes with integrating risk‐informed decision making at NASA?

NASA is increasingly infusing small satellites (mass < 180 kg) into its missions and mission planning activities

• Increased cost and schedule efficiencies

• Greater spatial, temporal, and spectral coverage

• More frequent launch opportunities

• Greater potential for innovative mission implementation approaches

• Greater opportunities to leverage private sector innovation

Dellingr CubeSat deployment from the International Space Station, December 2017

Development and use of SmallSats raises multiple challenges and successes, many relevant to RIDM

17

Leveraging private sector innovation

The use of commercial off‐the‐shelf (COTS) components is increasing.

However, vendors may provide minimal mission assurance documentation, and components may have little flight heritage.

This presents an SMA challenge.

Dellingr 6U CubeSat‐ internal view (during development)

Dellingr CubeSat. 10 x 20 x 30 cm, ~11 kg

What are challenges and successes with integrating risk‐informed decision making at NASA?Example: The Dellingr CubeSat Mission

18

Consistent with this SMA challenge, the Dellingr spacecraft exhibited numerous component anomalies and failures during development and on‐orbit.

Yet, the mission is successful because the “mission system” was resilient to component failure.

Ref: Cutting Edge‐ Goddard’s Emerging Technologies, Fall 2018


19

Success Lesson:

RIDM should not be considered just a safety and mission assurance team activity.

The degree to which the approach is infused across the mission full‐lifecycle (formulation through operations) and throughout the mission team can provide mission resiliency and inform success.

Ground systems

Operations teamScience andengineering team

Spacecraft

The Dellingr “system” mitigated numerous on‐orbit component failures/anomalies and infrastructure challenges.


20

Other High‐level RIDM Integration Challenges

• The fear of failure: changing a “failure is not an option” dominant culture is not easy

• Organizational compartmentalization is a barrier to “mission system‐level thinking”

• Personal comfort and protection can be derived from strictly adhering to static SMA requirements

• Organizational inertia can be hard to overcome

• Organizations may need to re‐learn the “art of thinking”

An organization’s history of success can be its largest barrier to RIDM infusion

What are challenges and successes with integrating risk‐informed decision making at NASA?

21

In Closing

• GSFC had a history of success prior to implementing risk‐based SMA

• Yet, advancement essential and is not just informed by the “what”; it’s also the “how”

• GSFC has migrated to risk‐based SMA, yielding millions of dollars and hundreds of weeks of schedule saved relative to a static requirement‐based approach

• The degree to which a risk‐informed approach is infused across the mission full‐lifecycle (formulation through operations) and throughout the mission team can provide mission resiliency and inform success.

• An organization’s history of success can be its most significant barrier to RIDM infusion

Re‐learning the “art of thinking” can lead to a compelling tomorrow.

22

For further information:

1 Dezfuli, Homayoon & Stamatelatos, Michael & Maggio, Gaspare & Everett, Chris. (2010). NASA/SP‐2010‐576, NASA Risk‐Informed Decision Making Handbook.

23

2Jesse Leitner, Bhanu Sood, Eric Isaac, Jack Shue, Nancy Lindsey & Jeannett Plante (2018) Risk‐based safety and mission assurance: Approach and experiences in practice, Quality Engineering, 30:4, 648‐662, DOI: 10.1080/08982112.2018.1473584

For further information:

The paper Risk‐based safety and mission assurance: Approach and experiences in practice includes additional PCB case study details and additional risk‐based SMA examples and results.

url https://doi.org/10.1080/08982112.2018.1473584

24

[email protected]

Thank You

perspectives on risk‐informed decision‐making · 2020. 10. 31. · ibex aura gems juno maven...

Documents