problem reporting - taxonomy bob beil 6/12/07 standards working group, nasa std-0006 bob beil nesc...

Problem Reporting - TaxonomyBob Beil

6/12/07

Standards Working Group, NASA STD-0006

Bob Beil

NESC Systems Engineering Office

June 12, 2007

This briefing is for status only and does not represent complete engineering data analysis 2

Bob Beil

6/12/07Agenda

• Background

• Team Listing

• Status

• Current Activities

• Backup


Bob Beil

6/12/07Information Overload!

SSP, ISSPRACAdatabasesExploration….

Robotic Missiondatabases

GSFCSOARSdatabase

JPL, ARC,Stennis etc

We are drowning in data, but starving for knowledge!


Bob Beil

6/12/07

NESC led Data Mining & Trending Working Group (DMTWG)

• NESC tasked to form DMTWG to assist the Agency in the formulation and implementation of a plan to strengthen trending of NASA programs and projects, and to ensure appropriate visibility of data mining and trending within the Agency

– Function as an advisory group to the NASA Office of the Chief Engineer and the EMB.

– Assist the Office of the Chief Engineer in the development, coordination, and implementation of best practices for data mining and trending of technical data.


Bob Beil

6/12/07NESC Data Mining/Trending Activities To Date

• NESC received data mining/trending task

• Conducted initial DMTWG workshop– Recommendations and lessons learned (including identification of

need for taxonomy/data dictionary)

• Initiated Recurring Anomalies effort– Completed SSP recurring anomalies effort

– ISS report to be presented to NRB June ‘07

• NESC funded taxonomy working group produced a proposed taxonomy standard

• Proposed Standard formed backbone of CxP requirements document (CxP 70068)

• Response team formed to perform requested data mining activities within the Agency


Bob Beil

6/12/07Team Listing

Team ListingBob Beil Systems Engineering Office (SEO) - Co-LeadVickie Parsons NESC SEO – Co-LeadTina Panontin Ames Research CenterRoxana Wales Ames Research CenterMichael Rackley Goddard Space Flight CenterJames Milne Goddard Space Flight CenterTim Barth Kennedy Space CenterJohn McPherson Marshall Space Flight CenterJayne Dutra Jet Propulsion LaboratoryLarry Shaw Johnson Space Center


Bob Beil

6/12/07Background cont., why taxonomy

• There is a frequently quoted statistic that more than 80% of information housed in corporate repositories is unstructured data, for NASA it is at least equivalent

• The reality is that until computers can consistently and accurately recognize concepts, using a taxonomy as a framework for categorizing documents will aid in navigation and retrieval.

– Natural language searching and keyword searching yield high retrieval but can miss essential pieces of content that do not contain the specific terms that are being searched.

– A taxonomy can be counted on to improve search precision and facilitate discovery when drilling down into a subject hierarchy


Bob Beil

6/12/07Background, Taxonomy

• The Office of Chief Engineer requested the NESC lead a small group to develop a proposal for a common taxonomy to be used by all NASA projects in the classifying of nonconformance's, anomalies and problems

– The intent was to determine what information is required to be collected and maintained in order to facilitate future trending and root cause analysis

• Taxonomy:

The science of classification

Today, our electronic technology allows us the opportunity to present information from multiple viewpoints maximizing the probability of discovery of relevant information by the end user

•A taxonomy provides an important piece of the puzzle to performing efficient data mining and trending


Bob Beil

6/12/07Background cont.

• NASA personnel with experience in both human spaceflight and robotic missions were recruited

– Expertise in knowledge management systems, anomaly resolution, trending, current problem reporting systems and taxonomy development

• Developed use cases across various missions

• Using use cases, developed taxonomy(ies) for a Problem Reporting System

• Product provides suggestions for inclusion in a problem reporting system and may serve as a partial guide to system developers; the intent was not to design a problem reporting system


Bob Beil

6/12/07Status

• In Review

ACTION STAGE DUE DATE OPEN ACTION

COMPLETED/DATE

NASA-STD-0006Common NASA Taxonomy for Problem Reporting, Analysis, and Resolution Standard

Agency Review OVERDUE

(Note: Preparing for EMB Concurrence)

ARC

JSC

LaRC

NESC

DFRC/08-23-06

GRC/10-17-06

GSFC/c/11-08-06

JPL/10-02-06

KSC/c/09-01-06

MSFC/c/No Date

SSC/11-08-06

WSTF/08-17-06

OSMA/08-09-06


Bob Beil

6/12/07Resolution Matrix Summary

73 comments

• Concur

– 47

• Reject

– 24, have discussed all but 3 with authors

• Require further consideration

– 2, forward work


Bob Beil

6/12/07Current Activities

• CxP PRACA, CxP 70068 (Constellation Program Problem Reporting, Analysis, and Corrective Action (PRACA) Requirements)

– Proposed Standard provide backbone for this document

– Noted weakness of Standard was software inclusion

• Working to incorporate lessons learned into proposed Standard

• NESC funded effort planned to benchmark the data dictionary/taxonomy


Bob Beil

6/12/07

Back-upBack-up


Bob Beil

6/12/07Report Summary

• Data elements marked with an asterisk in the share column represent the minimum set of data elements that all projects should make available through a common user interface to organizations such as the NESC, tasked with performing trending across projects

– This set of data elements should provide enough information to facilitate the root cause and trend analysis required of the individual projects by NPR 7120.5C

– Given differences between human spaceflight and robotic missions life cycles and post launch activities, some data elements may not be applicable for both.


Bob Beil

6/12/07

Recommended Data Elements and Taxonomies for Problem Reporting

Field Share Source Definition EXPECTED PROBLEM REPORT PROCESSING PHASE(S) FOR DATA FIELD POPULATION: 1=Initiation, 2=Analysis, 3=CloseOut

Criticality Code * Pick list Assessment of the severity of the problem based on FMEA and CIL for the assembly level … this is the functional criticality level

1-2

Problem Description

* Large text string

Detailed description of the problem - "prescription" for what would be information to be included is provided

1

Mishap Report? Yes/No Did this problem result in a formal mishap report due to damage of equipment or personal injury?

1-2-3

Last Update Field * Formatted String

Automatically filled by software when record saved

1-2-3 (auto)

Program Taxonomy Program name (program attributes defined in NPR 7120.5C)

1

Defect Characteristics

* Example list

A fault/flaw/discrepancy/nonconformance in a component or process that causes discrepant performance of the component or assembly involved

2


Bob Beil

6/12/07Pick List

Field Potential Values Definitions of Values

Criticality Code 1 Single failure that could result in death or loss of vehicle

1R Redundant hardware items that could cause a criticality 1 event if all items fail

1S Safety or hazard monitoring hardware items that could cause the system to fail to detect, combat, or operate when needed during a hazardous condition, potentially resulting in a criticality 1 event

2 Single failure that could result in severe and/or permanent injury, major property damage, or a loss of mission

2R Redundant hardware items that could cause a criticality 2 event if all items fail

3 Single failure that could result in minor injury, minor property damage, a significant mission delay, or a mission degradation in which some mission goals not achieved

4 All other failures that result in unscheduled maintenance or repair

Where pick lists or taxonomies are provided for individual data elements, these are meant to be a starting point and not all inclusive


Bob Beil

6/12/07Example List

Field Potential Values Definitions of Values

Defect Characteristics

MiswiredAbradedDingedPart omittedWornShort-circuitedDelaminated

These are definitely not an exhaustive list.

NOTE: These are examples, individual projects should create pick lists or links wherever possible to represent their situation. These data fields were included here because the team did not believe that even a high level set of

values would be consistent between projects.


Bob Beil

6/12/07Taxonomy

Field Taxonomy Comments

Program NASA taxonomy http://lurch.hq.nasa.gov/2005/11/21/pops.owl

These fields should be linked into NASA's formal taxonomies to auto-fill where possible.


Bob Beil

6/12/07Report Summary cont.

• Critical need for projects to capture information on aberrant events in order to determine the causes and prevent future occurrences

• Relevant data needs to be translated into shared data elements and provided to a common source so trending across projects may be accomplished by independent organizations

• Report includes suggested definitions for key terms

• Report also includes a white paper on the characteristics of a good taxonomy

• Additional recommendations were provided for consideration when developing an actual system


Bob Beil

6/12/07Background, Why Data Mining?

• Data mining to find patterns

• Analysis to identify correlations or clusters

• Search for explanations

A B C A D C D B A C B B B D D D A D C B B D B A C B B D A D C B B D D

B B B B B B B B B

D D D D D D

AC AC AC AC AC AC

B B B

D D D D D

A B C A D C D B A C B B B D D D A D C B B D B A C B B D A D C B B D D

“The nontrivial extraction of implicit, previously unknown, and potentially useful information from data.”

Trending: the search for patterns and subtle relationships in data to infer relative data behavior.


Bob Beil

6/12/07NESC Data Mining Activities

• Data Mining Trending Working Group (DMTWG) meeting monthly

– Comprised of representatives from all centers as well as NTSB, FAA, DHS, & INPO

– Workshop scheduled for June 20/21 to discuss successes in data mining

• Purchased and Installing SAS (statistical analysis software) on a LaRC server to make available for DMTWG

• ARC developing custom data mining tool (clustering based) to aid search for recurring anomalies

• Successful data mining and trending involves a variety of methods and tools as well as subject matter expertise

problem reporting - taxonomy bob beil 6/12/07 standards working group, nasa std-0006 bob beil nesc...

Documents