learning from mistakes; ula’s error prevention program · in order to ensure all employees...

of 10

Learning From Mistakes; ULA’s Error Prevention Program

James E. Allison; United Launch Alliance; Denver, Colorado, USA

Keywords: error prevention, error reporting, lessons learned, root cause analysis

Abstract The ability to learn from one’s mistakes or errors has long been recognized as a positive trait and effective tool. Dale Carnegie stated “The successful man will profit from his mistakes and try again in a different way” (Quotations Book 3). This paper describes how United Launch Alliance (ULA) has applied this valuable concept in an aerospace environment. ULA has implemented an Error Prevention Program that recognizes errors as learning opportunities and develops and maintains a workforce culture that rewards error reporting instead of punishing employees when errors occur. ULA’s Error Prevention Program rigorously processes errors regarding both rocket production, test operations and launch operations. Each reported error is tracked through resolution as follows: 1) root cause analysis, 2) corrective action review via formal Corrective Action review boards, 3) executive management review (called an Error Prevention Council) that extracts lessons learned and issues action items to reduce error occurrence company wide. Lessons Learned are formally documented and released company-wide via one or more of the following forums: 1) Safety Bulletins, 2) weekly Perfect Product Delivery emails, 3) Error Prevention Success Stories. These publications promote enterprise learning from past errors and encourage proactive elimination / mitigation of hazards that may lead to future errors. In addition, ULA’s Error Prevention Program collects a variety of data and has developed metrics to measure program progress and effectiveness.

Introduction: The ability to learn from one’s mistakes and errors has long been recognized as a positive trait and effective tool. Dale Carnegie stated “The successful man will profit from his mistakes and try again in a different way” (ref. 3). This paper describes how United Launch Alliance (ULA) has applied this valuable concept in an aerospace environment. ULA Background: A full description of ULA’s role in the aerospace business can be found on the ULA Website “http://ulalaunch.com/index_about.html”. The following paragraphs offer a brief glimpse into ULA’s origins: “On May 2, 2005, The Boeing Company and the Lockheed Martin Corporation announced their intention to form a joint venture called the United Launch Alliance (ULA).” “ULA brings together two of the launch industry’s most experienced and successful teams—the Lockheed Martin Atlas and Boeing Delta teams—that have supported America’s presence in space for more than 50 years. Atlas and Delta expendable launch vehicles have carried nearly 850 combined payloads to space ranging from weather, telecommunications and national security satellites that protect and improve life on Earth, to deep space and interplanetary exploration missions that further our knowledge of the universe.” “Under ULA, Delta and Atlas rockets will provide safe, cost-efficient, readily available and reliable access to space of U.S. government missions, continuing the tradition of supporting strategic U.S. space initiatives with advanced, robust launch solutions.” “ULA is structured as a 50-50 joint venture between Lockheed Martin and Boeing. The United Launch Alliance team consists of approximately 3,900 employees working at sites across the country. ULA is headquartered in Denver, Colo. Major assembly and integration operations will be located at the manufacturing and assembly facility in Decatur, Ala. Atlas V mechanical structures, payload fairing and adapter fabrication and assembly are performed in Harlingen, Tex.” (ref. 1)

of 10

ULA launch facilities are located at Cape Canaveral Air Force Station, Florida, and Vandenberg Air Force Base, California.

ULA And Error Prevention

Error Prevention is an important element of ULA’s culture. ULA’s Quality Assurance and System Safety organization promotes the “Perfect Product Delivery”1 ethic and applies it to all ULA endeavors. The Perfect Product Delivery ethic is: “Delivering all work products in a manner that completely satisfies the needs of the downstream customer.” As seen in Figure 1, ULA’s Error Prevention Process anchors the Perfect Product Delivery ethic by addressing its four principal elements. Error Prevention Process Principal #1: “Achieve Excellence In Everything We Do”, is nicely demonstrated by Vince Lombardi’s often quoted speech to the Green Bay Packers on his first day as their head coach. “Gentlemen, we are going to relentlessly chase perfection, knowing full well we will not catch it, because nothing is perfect. But we are going to relentlessly chase it, because in the process we will catch excellence.” (ref. 2) The ULA Error Prevention Process “chases perfection” by identifying the “imperfections”, i.e. the mistakes and errors in ULA’s processes and tools. ULA “catches” or “achieves” excellence by systematically correcting these imperfections. Error Prevention Process Principal #2: “Continuously Improve Every Process And Product” is achieved by “continuously” repeating the imperfection identification and correction process to yield ever improving processes and products. Error Prevention Process Principal #3: “Develop A World-class Work Environment” is an easily achieved Error Prevention Process byproduct. The continuous identification and elimination of errors leads to a work environment that is error free, safer, more efficient and less stressful – which are arguably world-class characteristics. Error Prevention Process Principal #4: “Deliver Program Success” is also an easily achieved byproduct. The capacity for a company’s “success” increases dramatically when errors are eliminated from the underlying processes that produce their products.

Figure 1—ULA’s Error Prevention Process Addresses The Key Principals Of ULA’s Perfect Product Delivery Ethic.

Figure 1—ULA’s Error Prevention Process Addresses The Key Principals Of ULA’s Perfect Product Delivery Ethic.

Perfect Product Delivery Ethic:• Relentless pursuit of perfection to achieve excellence in everything we do;

• Applies our passion for Mission Success to continuously improve every process and product, to completely meet the needs of every customer; and

• Inspires and empowers all employees to dedicate our innovative talents to deliver program success and develop a world-class work environment.

Error PreventionProcess PrincipalsError Prevention

Process Principals• Achieve excellence in

everything we do

• Continuously improveevery process andproduct

• Develop a world-classwork environment

• Deliver program success



everything we do




™

1 Perfect Product Delivery is a registered trademark of United Launch Alliance, LLC.

of 10

ULA’s Error Prevention Process

ULA’s Error Prevention Process can be divided into the five tiered flow demonstrated in Figure 2. The remaining sections of this paper describe the detailed operations associated with each tier. As seen in Figure 2, the Error Prevention Process is fed by ongoing voluntary mistake and error disclosures originated by ULA employees across the United States. Encouraging Voluntary Event Disclosures: Nobody wants to make a mistake or voluntarily accept responsibility when errors occur. ULA recognizes every mistake/error as a potentially valuable learning opportunity and encourages all ULA employees to participate in voluntary Event disclosure. To eliminate the negative stigma typically associated with disclosure, ULA uses the word “Event” to replace the less palatable terms “mistake” and “error”. It is far easier for employees to tell management they experienced “an Event” than to say they made a mistake or error. When Events are reported, ULA management does not immediately attribute their cause to employee actions. Formal analysis techniques are applied to determine and address each event’s true cause or causes. ULA management accepts the responsibility to provide the elements the workforce requires to perform their jobs successfully, and understands Events usually occur when one or more of the following elements is lacking:

• tools and tooling • clear, well written procedures • appropriate and safe working conditions • adequate time, schedule and budget

In order to ensure all employees participate in voluntary Event disclosure and pursue all common Error Prevention goals, ULA gives each employee the opportunity to participate in a four hour Error Prevention seminar addressing the following objectives:

• realize Event potential is real (share real-life ULA Event scenarios) • realize Events can be costly (may cause personnel injury, increased costs, loss of product and schedule

delays) • recognize the importance and value of voluntary Event disclosure (Event disclosure / reporting feeds the

Error Prevention Process) • recognize the importance and value of Event elimination / mitigation (Event elimination / mitigation

prevents personnel injury, increased costs, loss of product and schedule delays) • recognize potential Event situations before they occur • learn how to communicate concern about potential Event situations before they occur • learn the skills and techniques necessary to prevent specific Events • learn how to motivate fellow employees to understand and embrace these objectives

These goals are continuously reinforced through the following data sharing activities:

• Company Wide Event Reporting – (details regarding each Event are shared company wide via “Flash Notices” within 24 hours of Event occurrence)

• Weekly Perfect Product Delivery Attention To Detail Topic Emails – (weekly distributed Email addressing ULA specific Error Prevention topics – these Emails are designed to promote Error Prevention related discussions)

• Periodic Safety Bulletin Releases – (bulletins designed to share data, cautions and warnings regarding hazards recently identified via the Error Prevention Process across the company are distributed as needed)

• Company Wide Access To The Error Prevention Web Site – (this Web Site offers all employees unlimited access to Error Prevention topics and historical ULA Event data and related metrics)

of 10

Event ReportingProcess

Event ReportingProcess

Cause Analysis& Corrective Action

Process

Cause Analysis& Corrective Action

Process

Corrective Action Board

Review Process

Corrective Action Board

Review Process

Error Prevention Council

Review Process

Error Prevention Council

Review Process

Share LessonsLearned ProcessShare Lessons

Learned Process

Event Event

Figure 2—The ULA Error Prevention Process Flow is fed by ongoing voluntary Event disclosure.

Figure 2—The ULA Error Prevention Process Flow is fed by ongoing voluntary Event disclosure.

Error Prevention Process Flow

Ongoing VoluntaryEvent Disclosures

Via Workforce



everything we do






everything we do




Continuous Feedback Encourages Ongoing Disclosure:

• Company Wide Event Reporting• Weekly Attention To Detail Emails• Periodic Safety Bulletin Releases• Error Prevention Web Site Access

of 10

Event Reporting Process

The “Event Reporting Process” demonstrated in Figure 3 is triggered each time an Event occurs. The Goals of the Event Reporting Process are listed below:

• collect and document Event related information • communicate Event occurrence across the ULA enterprise • communicate Event occurrence with customers as appropriate (the requirement to report Events to the

customer is defined by specific agreements between ULA and each customer)

The first step in the Event Reporting Process is to collect and document Event related data. An Error Prevention Team Point Of Contact (POC) is assigned to each of ULA’s geographic sites, and is available to guide and assist with Event reporting, investigation and processing. Error Prevention Team POCs work together with the Person In Charge (PIC) of the procedure or operation that was being executed when the Event occurred to collect the data necessary to populate a “Flash Report”. Figure 3 demonstrates the content of a Flash Report. Flash Reports and related data are stored in the ULA Error Prevention Data Base for future reference. ULA’s goal is to distribute Flash Reports via Email within 24 hours of occurrence – hence the term “Flash” Report. Flash Report recipients include ULA executive management and employees that request membership in the Flash Report distribution list. Flash Reports are also sent to the customer community when Events meet criteria agreed to by ULA and the customer. NOTE: Customers may gain visibility to all ULA Events by attending monthly Quality Assurance And System Safety Reviews and visiting the ULA Error Prevention website.

Photos of both the Event area and affected hardware elementsPhoto Record

Reference related non-conformance documentation

Related Report Identification

Did this event cause any of the following:Death ?Injury ?Flight Hardware Damage ?Support Hardware Damage ?

Preliminary Loss Assessment

WHO gathered this data ?Submitter Identification

Identifies the procedure in effect when this Event occurred

Procedure In Use Identification

WHAT happened ?Event Description

A brief statement explaining this Event’s significance. (Example: Widget was damaged and procedure was stopped)

Event Significance

Identify which program the Event affects – Note: production Events may impact more than one program.

Program (Atlas, Delta II or Delta IV)

WHERE did the Event occur ?Event Location

WHEN did the Event occur ?Event Date and Time

Uniquely identifies each EventFlash Report Identifier

DescriptionFlash Report Data Element

Flash Report Content

Photos of both the Event area and affected hardware elementsPhoto Record

Reference related non-conformance documentation

Related Report Identification

Did this event cause any of the following:Death ?Injury ?Flight Hardware Damage ?Support Hardware Damage ?

Preliminary Loss Assessment

WHO gathered this data ?Submitter Identification

Identifies the procedure in effect when this Event occurred

Procedure In Use Identification

WHAT happened ?Event Description

A brief statement explaining this Event’s significance. (Example: Widget was damaged and procedure was stopped)

Event Significance

Identify which program the Event affects – Note: production Events may impact more than one program.

Program (Atlas, Delta II or Delta IV)

WHERE did the Event occur ?Event Location

WHEN did the Event occur ?Event Date and Time

Uniquely identifies each EventFlash Report Identifier

DescriptionFlash Report Data Element

Flash Report Content

Error PreventionDatabase

Collect & DocumentEvent Data


Populate FlashReport


Send Flash NoticeTo Customer


Send Flash NoticeTo Enterprise


SendTo

Customer?

SendTo

Customer?

Event Reporting

YES

NO

Flash Reports

Event Pictures

Email

Email

WEB

EventReporting

EventReporting

CAC / ACA

C / A

CABReviewCAB

Review

EPCReviewEPC

Review

Share LessonsLearned


Event Event











SendTo

Customer?

SendTo

Customer?

Event Reporting

YES

NO

Flash Reports

Event Pictures

EmailEmail

EmailEmail

WEBWEB

EventReporting

EventReporting

CAC / ACA

C / A

CABReviewCAB

Review

EPCReviewEPC

Review



Event Event

Figure 3—The Event Reporting Process is triggered each time an Event occurs.Figure 3—The Event Reporting Process is triggered each time an Event occurs.

of 10

Cause Analysis & Corrective Action Process

The “Cause Analysis & Corrective Action Process” demonstrated in Figure 4 is executed to determine the cause or cause(s) that led to an Event, and develop Corrective Action Plans necessary to preclude Event reoccurrence. The Cause Analysis & Corrective Action Process’s goal is to determine the following causes for each Event:

• Direct Cause – The mechanism directly responsible for the Event. • Root Cause(s)– Process or knowledge related factors that result in the existence of the direct cause

mechanism. • Systemic Cause(s) – Cultural and organizational factors that allowed the root causes to occur.

The first step in the Cause Analysis & Corrective Action Process is to gather the data necessary to perform a cause analysis. This data may be taken from Event related data stored in the ULA Error Prevention Data Base, or be “new” data discovered / developed by specific cause analysis related activities. As new data is obtained, it is also stored in the Error Prevention Data Base for future reference. Typically, the site resident Error Prevention Team POC hosts and runs Cause Analysis meetings. The individuals invited to participate in these meeting typically include the PIC, the Responsible Engineer assigned to the procedure that experienced the Event, a Quality Engineer, individuals present when the Event occurred and others as deemed necessary by the POC, PIC and other participants. ULA employs a variety of Cause Analysis techniques. The most commonly used Cause Analysis techniques are shown in Figure 4. Once the group agrees upon an Event’s causes, Corrective Action Plans are developed to eliminate reoccurrence. Action items are developed, assigned and tracked to ensure Corrective Action Plans are executed in both a timely and effective manner.

Error PreventionDB

GatherEvent DataGather

Event Data

Cause Analysis &Corrective Action

Session


Session

Document Direct,Root &

Systemic Cause(s)


Systemic Cause(s)

Document Corrective Action Plan(s)


Cause Analysis & Corrective Action

Cause Analysis &

Corrective Action Plans

WEB

EventReporting

EventReporting

CAC / ACA

C / A

CABReviewCAB

Review

EPCReviewEPC

Review



Event Event

Error PreventionDB

GatherEvent DataGather

Event Data


Session


Session


Systemic Cause(s)


Systemic Cause(s)



Cause Analysis & Corrective Action

Cause Analysis &

Corrective Action Plans

WEBWEB

EventReporting

EventReporting

CAC / ACA

C / A

CABReviewCAB

Review

EPCReviewEPC

Review



Event Event

40

United Launch Alliance 3-Legged 5-Why Form

Technicians rolled 2nd stage without all roll rings installed causing the stage to shift.

Technicians were trying to position tank for x-ray of orbital tube welds (OTW)

Roll ring segments were not in place to accommodate 2nd stage

Email was sent requesting x-ray of OTWs

The corrosion issue not communicated and identified as the area of interest to the 3rd shift x-ray technicians

GSE removed to facilitate x-ray of suspect corrosion area.

Inadequate procedures in place to preclude hardware damage from unintended tool use

3rd shift activity not taken into consideration

Why?

Why?

Why?

Why?

Why did we continue?

Why?

Why?

A

C

B

Root CausesCorrective Actions Date

A. Work instruction to be

drafted to standardize communication between sending and receiving parties for NDT requests. Part of GB project # 1848……NDT CALL BOARD

B. Need procedure for “lock-

out” of production tooling/equipment when required. Needs to be technician friendly

5/10/2008 5/10/2008

C. Develop roll checklist for

applicable areas in factory with emphasis on securing h/w.

D. Work instruction to be

drafted to standardize communication between sending and receiving parties for NDT requests. Part of GB project #1848……NDT CALL BOARD

5/10/2008 5/10/2008

Look Across / Within Plant: NDT Call Board to be rolled out shop wide.

Define Problem

Use this path for the specific nonconformance being investigated

A. Use this path to investigate why thecondition was not detected

Use this path to investigate thesystemic root cause

ORB Requested Root Cause/ Corrective Action for World View 2 Second Stage Incident Issue Date: 2/26/2009

Why?

“Standard”communication via the NDT focal was not completed

NDT Technicians did not detect missing roll ring segments

B. Use this path to investigate why thecondition was not detected

Techs performed pre-task briefing but only focused on tubes and harnesses related to OTW X-ray

Tank center Mgr accommodated the 2nd stage Mgr’s request by arranging for x-ray of corroded area on 1/16/2009 1st shift exclusively

D

Why?

NDT Technicians

anticipated X-ray of OTW only

Standard planning flow and protocol for sending and receiving technicians (except for no NDT focal “to do” list)

Standard work available & email notification to perform X-rays of OTW

Why?

44

Apollo Root Cause Analysis

32

AHU LEAK INCOMPOSITES CLEAN ROOM

AHU LeakAHU LeakCredible (contributing)- C

Probable (Root) - R

Not Credible - NC

1.0 Environmental

CauseCauseCauseCause

2.0 Process

5.0 Documentation 6.0 Tools & Equip

2.1 Planning2.2 Technical Assignments

2.3 Team Communications2.4 Preventative Maintenance

2.5 BCMS (web based energyenergy mgmt sys)

1.1 Access1.2 Lighting

1.3 Noise1.4 Wind

4.1 System Knowledge4.2 Assignments

4.3 Training & Certifications4.4 Human Error

4.5 Process Discipline

5.1 Task Instructions

5.2 Engineering Drawings

5.3 Design Intent5.4 MP2/7I

6.1 HVAC Design6.2 Facility Design/Air Handler Control Logic

6.3 Flow Rates/Coils

1.5 Outside Temperature

6.4 Fans/Logic

EffectEffect

3.0 Flight HardwareN/A

4.6 Distractions

2.0 Process1.1 Access

1.3 Noise1.4

1.5

3.0 Flight Hardware

4.0 Personnel

4.7 Experience Level

NC

NC

NC

NC

NC

NC

NCNC

NC

NC

NC

NC

NC

NC

C

NC

C

C

NC

R

C

NCNC

C

C

Root Cause Fishbone

Apollo Root Cause

5 Why Analysis

Cause Analysis Techniques Employed By ULA

40

United Launch Alliance 3-Legged 5-Why Form

Technicians rolled 2nd stage without all roll rings installed causing the stage to shift.

Technicians were trying to position tank for x-ray of orbital tube welds (OTW)

Roll ring segments were not in place to accommodate 2nd stage

Email was sent requesting x-ray of OTWs

The corrosion issue not communicated and identified as the area of interest to the 3rd shift x-ray technicians

GSE removed to facilitate x-ray of suspect corrosion area.

Inadequate procedures in place to preclude hardware damage from unintended tool use

3rd shift activity not taken into consideration

Why?

Why?

Why?

Why?

Why did we continue?

Why?

Why?

A

C

B

Root CausesCorrective Actions Date

A. Work instruction to be

drafted to standardize communication between sending and receiving parties for NDT requests. Part of GB project # 1848……NDT CALL BOARD

B. Need procedure for “lock-

out” of production tooling/equipment when required. Needs to be technician friendly

5/10/2008 5/10/2008

C. Develop roll checklist for

applicable areas in factory with emphasis on securing h/w.

D. Work instruction to be

drafted to standardize communication between sending and receiving parties for NDT requests. Part of GB project #1848……NDT CALL BOARD

5/10/2008 5/10/2008

Look Across / Within Plant: NDT Call Board to be rolled out shop wide.

Define Problem

Use this path for the specific nonconformance being investigated

A. Use this path to investigate why thecondition was not detected

Use this path to investigate thesystemic root cause

ORB Requested Root Cause/ Corrective Action for World View 2 Second Stage Incident Issue Date: 2/26/2009

Why?

“Standard”communication via the NDT focal was not completed

NDT Technicians did not detect missing roll ring segments

B. Use this path to investigate why thecondition was not detected

Techs performed pre-task briefing but only focused on tubes and harnesses related to OTW X-ray

Tank center Mgr accommodated the 2nd stage Mgr’s request by arranging for x-ray of corroded area on 1/16/2009 1st shift exclusively

D

Why?

NDT Technicians

anticipated X-ray of OTW only

Standard planning flow and protocol for sending and receiving technicians (except for no NDT focal “to do” list)

Standard work available & email notification to perform X-rays of OTW

Why?

44

Apollo Root Cause Analysis

32

AHU LEAK INCOMPOSITES CLEAN ROOM


Probable (Root) - R

Not Credible - NC

1.0 Environmental


2.0 Process






1.3 Noise1.4 Wind










6.4 Fans/Logic

EffectEffect


4.6 Distractions


1.3 Noise1.4

1.5

3.0 Flight Hardware

4.0 Personnel


NC

NC

NC

NC

NC

NC

NCNC

NC

NC

NC

NC

NC

NC

C

NC

C

C

NC

R

C

NCNC

C

C

Root Cause Fishbone

Apollo Root Cause

5 Why Analysis

Cause Analysis Techniques Employed By ULA

Figure 4—The Cause Analysis And Corrective Action Process is executed to determine the cause or cause(s) that led to an Event, and develop Corrective Action Plans necessary to preclude Event reoccurrence.

Figure 4—The Cause Analysis And Corrective Action Process is executed to determine the cause or cause(s) that led to an Event, and develop Corrective Action Plans necessary to preclude Event reoccurrence.

of 10

Corrective Action Board (CAB)

The “Corrective Action Board Process” demonstrated in Figure 5 is executed to verify the causes and corrective actions determined in the Cause Analysis And Corrective Action Process are both accurate and appropriate. If the CAB determines that either the Cause Analysis or Corrective Action Plans for a given Event are lacking in any way, the Event is rejected and returned to the Cause Analysis and Corrective Action Process for reprocessing. Rejected Events iterate between the Cause Analysis And Corrective Action Process and CAB until approved. CAB presentations follow a set format featuring the following data elements:

• Event Description • Cause Analysis Process employed and associated results • Corrective Action Plans • Lessons Learned

An example CAB presentation is included in Figure 5. CABs are held on a regularly scheduled basis across the ULA enterprise to address Events, as well as other deficiencies, anomalies and improvement opportunities that arise through the course of business. Event related presentations are typically presented to the CAB via the PIC responsible for a given Event with assistance from the local Error Prevention Team POC. CAB presentation material is stored in the Error Prevention Data Base.

Corrective Action Board (CAB)

Error PreventionDB

Review Cause(s)Review Cause(s)

ReviewCorrective Action

Plan(s)

ReviewCorrective Action

Plan(s)

CABApproval

?

CABApproval

?

CABApproval

?

CABApproval

?

CAC/A

YES

NO

YES

NO

CAC/A

CAB PresentationMaterial

Prepare CABPresentation

Material

Prepare CABPresentation

Material

WEBWEB

EventReporting

EventReporting

CAC / ACA

C / A

CABReviewCAB

Review

EPCReviewEPC

Review



Event Event

CAC/A

Figure 5—The Corrective Action Board is executed to verify an Events causes and corrective actions are both accurate and appropriate.

Figure 5—The Corrective Action Board is executed to verify an Events causes and corrective actions are both accurate and appropriate.

Typical CAB Presentation Format

Lower Ring Clamp (Not Tethered

Upper Ring Clamp (Tethered)

EVENT08-082 Friction Stir Weld Holding Ring Clamp Falls

Lessons Learned:We learned to communicate incident with teams in an all team meeting, to Identify any hand tools/wrenches that do not have tether capability and Immediately stop use of any item that can not be secured unless there is drop mitigation (safety nets, pads, etc).

Corrective Actions: Facilities Maintenance has been assigned to tether

all clamps to eliminate falling. All clamps completed by 1-05-2009

Evaluate Decatur Drop Prevention against Core ULA Drop Prevention Directives.

Enlist launch site and Atlas personnel to help define best practices.

Incident:

Date: October 31, 2008Program: Delta IVLocation: Delta IV WeldCenter

Scope: Near Miss

Documentation: GOAL- 4887

A technician was removing a clamp from the Friction Stir Weld (FSW) #1 lower holding ring, when it slipped from his hands and fell approx 10ft. to the floor. There was no hardware in the FSW fixture at the time of the incident. The clamp weighing ~ 1 pound presented no potential damage to hardware.

Event Significance: Potential Injury to Personnel.

YWas the Huddle More Effective ?

NAESOP Huddle Held ?

Flight HW Damage ? N

Personnel Injury ? N

Root Cause:Clamp system on the top section of the FSW is tethered to keep from any clamps falling. Clamps on lower ring were never tethered at the time upper ring was completed.

Lower Ring Clamp (Not Tethered

Upper Ring Clamp (Tethered)

EVENT08-082 Friction Stir Weld Holding Ring Clamp Falls

Lessons Learned:We learned to communicate incident with teams in an all team meeting, to Identify any hand tools/wrenches that do not have tether capability and Immediately stop use of any item that can not be secured unless there is drop mitigation (safety nets, pads, etc).

Corrective Actions: Facilities Maintenance has been assigned to tether

all clamps to eliminate falling. All clamps completed by 1-05-2009

Evaluate Decatur Drop Prevention against Core ULA Drop Prevention Directives.

Enlist launch site and Atlas personnel to help define best practices.

Incident:

Date: October 31, 2008Program: Delta IVLocation: Delta IV WeldCenter

Scope: Near Miss

Documentation: GOAL- 4887

A technician was removing a clamp from the Friction Stir Weld (FSW) #1 lower holding ring, when it slipped from his hands and fell approx 10ft. to the floor. There was no hardware in the FSW fixture at the time of the incident. The clamp weighing ~ 1 pound presented no potential damage to hardware.

Event Significance: Potential Injury to Personnel.

YWas the Huddle More Effective ?

NAESOP Huddle Held ?

Flight HW Damage ? N

Personnel Injury ? N

Root Cause:Clamp system on the top section of the FSW is tethered to keep from any clamps falling. Clamps on lower ring were never tethered at the time upper ring was completed.

of 10

Error Prevention Council (EPC) The “Error Prevention Council Process” demonstrated in Figure 6 is executed monthly to ensure sound corrective actions are taken in response to incidents and to ensure ULA-wide lessons learned and improved practices are applied across the ULA Enterprise where applicable. The goals of the EPC are as follows:

• Learn from our Events, ULA wide • Leverage lessons learned to prevent future Events • Look for common causes, trends and take necessary preventive actions to reduce future risk.

EPC implementation was driven by the following rational:

EPC Rational: Events represent a significant cost to ULA and introduce risk. By identifying and sharing the Lessons Learned associated with each Event, the EPC will help prevent future Events. Ideally, the money saved via this process will far exceed the costs associated with the original incidents.

The EPC is chaired by the Vice President of ULA’s Quality Assurance and System Safety organization and has representation from all major disciplines and sites. The basic elements of the EPC process are listed below:

• Review each ULA Event to Identify Lesson Learned that can be shared across the enterprise • Verify each Event’s Root Cause(s) and Corrective Action plans are understood and documented • Drive Lesson Learned and Corrective Action sharing via action items, bulletins, procedure updates, other

actions as necessary

While it is not the main goal of the EPC to verify Root Cause analysis and Corrective Action Plans, if either is found incomplete or unacceptable during review, an Event can be sent back to CAB for re-evaluation / clarification. EPC presentation material is stored in the Error Prevention Database for future reference.

4/15/2009 8

Review EventCause(s), C/A(s) & Lessons Learned

Review EventCause(s), C/A(s) & Lessons Learned

Error Prevention Council (EPC)

EnterpriseActions

?

EnterpriseActions

?

EPCApproval

?

EPCApproval

?CAB

YES

NO

YES

NO

Assign ProactiveError Prevention

Actions Items

Assign ProactiveError Prevention

Actions Items

Error PreventionDB

EPC Presentation Material

EPC Minutes

Prepare EPCPresentation

Material

Prepare EPCPresentation

Material

WEB

EventReporting

EventReporting

CAC / ACA

C / A

CABReviewCAB

Review

EPCReviewEPC

Review



Event Event

CAB

Figure 6—The EPC is executed monthly to ensure ULA-wide lessons learned and improved practices are applied where applicable.

Figure 6—The EPC is executed monthly to ensure ULA-wide lessons learned and improved practices are applied where applicable.

31

EVENT08-106 Roof Leak In Composites Clean Room

Corrective Actions:Repair/replace pre-heat coil on AHU 10-5 and replace

damaged ceiling tiles.

Set up meeting with The Comfort Group (Original Equipment Manufacturer of the Boeing Control Management System) to understand system design and operations.

Review other Decatur Factory AHU’s for similar issues.

Incident:A ceiling leak was discovered in the composites clean room at approximately 11:30 AM. The leak resulted in a large puddle of water on the floor (approx. 12 ft diameter), two ceiling tiles falling, and several other ceiling tiles damaged. Inside the clean room there was the Heavy Half bi-sector of the GPS-IIF fairing that was completely bagged and ready for shipment. No water leakage or any type of damaged came to this hardware. The team moved it out of the clean room before any major water leaks had travel to the area. Due to the urgent need to move the hardware out of harm's way it was moved without the use of EDRs.

Lessons Learned: The outside air temperature was 16°F, the coldest day in six years. Outside air dampener needed to be closed: This could not be achieved while maintaining positive pressure in the booth

Direct Cause: In order to achieve positive pressure in the booth, Outside air is required. The outside air dampener was 100% opened allowing below freezing air to blow across the coil. This constant air supply caused the pre-heat coil to freeze and eventually burst.

Root Cause: The current Air Handler Unit (unit 10-5) design does not contain complete control logic to properly control the operation of the system. This lack of control logic allowed the outside air dampeners to stay open and the supply fans to continue to run. This below freezing air flow caused the pre-heat coil to freeze.

NIf Huddle Not Held, Would it Have Helped?

N/NIf Huddle Held, Could it Be Improved

Flight HW Damage ? NPersonnel Injury ? N

NC46627-01/Decatur

GOAL 5130Documentation

Decatur Facility Clean RoomLocation

DecaturProgram

12/22/2008Date

Damage To Hardware and Potential Injury to

PersonnelScope

31

EVENT08-106 Roof Leak In Composites Clean Room

Corrective Actions:Repair/replace pre-heat coil on AHU 10-5 and replace

damaged ceiling tiles.

Set up meeting with The Comfort Group (Original Equipment Manufacturer of the Boeing Control Management System) to understand system design and operations.

Review other Decatur Factory AHU’s for similar issues.

Incident:A ceiling leak was discovered in the composites clean room at approximately 11:30 AM. The leak resulted in a large puddle of water on the floor (approx. 12 ft diameter), two ceiling tiles falling, and several other ceiling tiles damaged. Inside the clean room there was the Heavy Half bi-sector of the GPS-IIF fairing that was completely bagged and ready for shipment. No water leakage or any type of damaged came to this hardware. The team moved it out of the clean room before any major water leaks had travel to the area. Due to the urgent need to move the hardware out of harm's way it was moved without the use of EDRs.

Lessons Learned: The outside air temperature was 16°F, the coldest day in six years. Outside air dampener needed to be closed: This could not be achieved while maintaining positive pressure in the booth

Direct Cause: In order to achieve positive pressure in the booth, Outside air is required. The outside air dampener was 100% opened allowing below freezing air to blow across the coil. This constant air supply caused the pre-heat coil to freeze and eventually burst.

Root Cause: The current Air Handler Unit (unit 10-5) design does not contain complete control logic to properly control the operation of the system. This lack of control logic allowed the outside air dampeners to stay open and the supply fans to continue to run. This below freezing air flow caused the pre-heat coil to freeze.

NIf Huddle Not Held, Would it Have Helped?

N/NIf Huddle Held, Could it Be Improved

Flight HW Damage ? NPersonnel Injury ? N

NC46627-01/Decatur

GOAL 5130Documentation

Decatur Facility Clean RoomLocation

DecaturProgram

12/22/2008Date

Damage To Hardware and Potential Injury to

PersonnelScope

32

Root Cause Analysis:AHU LEAK IN COMPOSITES CLEAN ROOM


Probable (Root) - R

Not Credible - NC

1.0 Environmental


2.0 Process






1.3 Noise1.4 Wind










6.4 Fans/Logic

EffectEffect


4.6 Distractions


1.3 Noise1.4

1.5

3.0 Flight Hardware

4.0 Personnel


NC

NC

NC

NC

NC

NC

NCNC

NC

NC

NC

NC

NC

NC

C

NC

C

C

NC

R

C

NCNC

C

C

32

Root Cause Analysis:AHU LEAK IN COMPOSITES CLEAN ROOM


Probable (Root) - R

Not Credible - NC

1.0 Environmental


2.0 Process






1.3 Noise1.4 Wind










6.4 Fans/Logic

EffectEffect


4.6 Distractions


1.3 Noise1.4

1.5

3.0 Flight Hardware

4.0 Personnel


NC

NC

NC

NC

NC

NC

NCNC

NC

NC

NC

NC

NC

NC

C

NC

C

C

NC

R

C

NCNC

C

C

33

Enterprise Learning• Proposed Enterprise Prevention Actions:

Develop and implement a modified system redesign that will sufficient control of the outside air intake dampeners and modify the duct work, if necessary, in order to obtain the correct amount of positive pressure in the clean room without the use of outside air.

• Proposed story to share:- There we were …Investigating why water was leaking out of air handler AHU

(unit 10-5) when we noticed that the pre-heat coil had busted and was leaking. We noticed that the outside air dampener was 100% open allowing cold air to flow across the coils.

– We Learned …That there were no controls on the outside air intake dampeners. We also learned that some ill advised modifications were made to the booth five years earlier in a cost savings effort.

33

Enterprise Learning• Proposed Enterprise Prevention Actions:

Develop and implement a modified system redesign that will sufficient control of the outside air intake dampeners and modify the duct work, if necessary, in order to obtain the correct amount of positive pressure in the clean room without the use of outside air.

• Proposed story to share:- There we were …Investigating why water was leaking out of air handler AHU

(unit 10-5) when we noticed that the pre-heat coil had busted and was leaking. We noticed that the outside air dampener was 100% open allowing cold air to flow across the coils.

– We Learned …That there were no controls on the outside air intake dampeners. We also learned that some ill advised modifications were made to the booth five years earlier in a cost savings effort.

Typical EPC Presentation Format

of 10

Share Lessons Learned

The “Share Lessons Learned Process” generates and distributes a variety of error prevention data and publications as shown in Figure 7. The rational behind each publication is described below:

• Safety Bulletins identify specific hazards discovered by the Error Prevention process. Safety Bulletins are typically developed for hazards that can not be immediately resolved and pose a threat of reoccurrence if not quickly identified and communicated.

• Perfect Product Delivery Attention To Detail Topics are sent weekly via Email to all ULA management and members of the workforce. Management is asked to distribute and discuss these topics during staff meetings. The goal is to keep constant attention on Error Prevention, and share the findings of the Error Prevention Process.

• There We Were Stories offer Event descriptions in a first person narrative format. These stories are intended to help readers develop a “this could happen to me” attitude which will in turn increase interest in and adherence to the Error Prevention Process.

• Success Stories demonstrate the successful application of Error Prevention techniques in a single page format. Success Stories may demonstrate how an Event led to the elimination or mitigation of a hazard, or demonstrate how an Event was prevented by proactively identifying and eliminating or mitigating a hazard before an Event actually occurred.

The goals that drive the Share Lessons Learned Process are as follows:

• Increase awareness that Error Prevention is a necessary and worthwhile process by keeping the workforce continuously abreast of the Events experienced by ULA on a regular basis.

• Increase awareness that Error Prevention techniques can be and regularly are successful in preventing Events and Event reoccurrence.

The Value Of Story Telling: Error Prevention publications are intentionally documented in “story telling” formats. When Event details are distilled into short concise narratives that tell simple but complete and accurate stories they become easier to understand, remember and share. As a result, Event related stories are more likely to be the topic of discussion in and across the workplace … and when employees freely discuss, understand and share Events, an Error Prevention Culture is realized.

• Safety Bulletins• Perfect Product Delivery

Attention To Detail Weekly Emails

• There We Were Stories• Success Stories

ULA Error Prevention Publications:• Safety Bulletins• Perfect Product Delivery

Attention To Detail Weekly Emails

• There We Were Stories• Success Stories

ULA Error Prevention Publications:

Share Lessons Learned

Error PreventionDB

Often Attached To Perfect Product Delivery Attention To Detail Topics

Success Stories


There We Were Stories

Error Prevention Publications

All ULA Management & Interested ULA Employees

Perfect Product Delivery Attention To Detail Topics

Specific ULA Departments Based On Subject

Safety Bulletins

Distributed To:Publication


Success Stories


There We Were Stories

Error Prevention Publications

All ULA Management & Interested ULA Employees

Perfect Product Delivery Attention To Detail Topics

Specific ULA Departments Based On Subject

Safety Bulletins

Distributed To:Publication

Error Prevention Publications. . .

Figure 7 – The Share Lessons Learned Process Generates And Distributes A Variety Of Error Prevention Data And Publications

Figure 7 – The Share Lessons Learned Process Generates And Distributes A Variety Of Error Prevention Data And Publications

EventReporting

EventReporting

CAC / ACA

C / A

CABReviewCAB

Review

EPCReviewEPC

Review



Event Event

WEBWEB

EmailEmail

of 10

ULA Error Prevention Program Effectiveness The goal of this paper is to describe ULA’s Error Prevention Process – not to measure or judge this process’s effectiveness. ULA was newly formed in 2006, and has not been in operation long enough to accurately measure and judge effectiveness with respect to long term goals. ULA is, and has been, actively gathering data and developing metrics to measure effectiveness – and intends to share this data via future publications. The following list offers a high level overview of the positive trends ULA currently attributes to the Error Prevention Process:

• Increased Event Reporting: The number of events reported across ULA is growing. ULA believes increased reporting leads to increased awareness and ultimately a decrease in the overall cost of Events.

• Decrease in number of high dollar Events: As higher numbers of Events are reported, ULA has realized more opportunities to eliminate or mitigate their associated hazards. As hazards are eliminated or mitigated, the likelihood of high dollar Events decreases. Although ULA data covers only a short time span, Error Prevention data from the Lockheed Martin heritage Atlas program indicatess a positive trend.

• Cause Spectrum Narrowing: Causes related to equipment are less frequently observed as the Error Prevention Process progresses. It is believed that increased Event reporting has led to increased equipment related cause elimination and/or mitigation. In other words, as equipment related events are encountered, equipment issues / concerns are repaired or replaced yielding a long term reduction in equipment related issues. Unreported equipment problems would linger and cause more frequent reoccurrences that were potentially more costly and severe.

• Shift to EP culture: ULA has observed the following trends that indicate the Error Prevention Process is making a positive impact on ULA:

o Increased Error Prevention awareness across the workforce o Evidence that a “Tell All Share All” attitude / culture has emerged and is growing o Management respect for the workforce (management does not blame the workforce for Events)

References

1. United Launch Alliance (2009) About ULA. Retrieved, 4/1/2009, from About ULA website,

http://ulalaunch.com/index_about.html 2. Schaap, Jeremy “We Will Catch Excellence.” Parade Magazine 3 Feb 2003: 3. Kothari, A. (2007) Quotations Book, 4/17/2009, from Quotations Book website,

http://quotationsbook.com/quote/26691/

Biography

James E. Allison, ULA System Safety, Error Prevention Lead, P.O. Box 277005, Littleton, CO 80127-7005, USA, telephone – (303) 977-1006, e-mail – [email protected].

Mr. Allison is currently leading United Launch Alliance’s (ULA) effort to develop and maintain an error prevention program for Atlas and Delta rocket production and launch operations. The ULA Error Prevention Team’s goal is to develop a culture that recognizes errors as opportunities for learning and process improvement. Mr. Allison is a 1977 Graduate of Penn State University and has 30 + years experience in the Aerospace Industry. Previous affiliations include Hughes Aircraft, Martin Marietta and Lockheed Martin. Mr. Allison is the father of 17 year old triplets, and credits them for providing countless opportunities to gain experience in the areas of Error Prevention, Root Cause Analysis and Problem Solving.

learning from mistakes; ula’s error prevention program · in order to ensure all employees...

Documents