schneider electric - nuclear navy and reducing human error

The US Nuclear Navy: Eradicating Human Error

Domenic Alcaro, LT, USN, Ret

Schneider Electric

Schneider Electric 2- Division - Name – Date

Agenda

●Engineering Plant Overview●Picking the Right People●Training the Right People●Procedures-Zero Deviation Allowed●Policies●System Design●War Stories●Cool Pictures●Non-Military Applications

Schneider Electric 3- Division - Name – Date

The Nuclear Submarine

Schneider Electric 4- Division - Name – Date

Engineering Plant Overview(Each plant below made up of multiple, smaller, complex subsystems)

●Reactor Plant●Steam Plant●Electrical Plant●Hydraulic Plant●Cooling Plants●Fresh Water Make-Up Plants●Multiple Auxiliary Plants (I.e., High Pressure Air, Low Pressure Air, Refuse Removal, etc.)

●Etc…(and this is just the back half of the sub)

Schneider Electric 5- Division - Name – Date

Picking the Right People

●The Infamous Admiral Rickover, “Father of the Nuclear Navy”

●Technical Competency Screening●Leadership Screening●Psychological Screening (claustrophobia is rarely an issue)

●Reliability and “Quality of Character” Screening – The PRP, Personnel Reliability Program

●Applied to both Officer and Enlisted Ranks

Schneider Electric 6- Division - Name – Date

Training the Right People:Prior to Arriving On Board

●Nuclear Power School

●Nuclear Prototype Training

●General Submarine Training

●If time permits, other Navy Schools

Schneider Electric 7- Division - Name – Date

Training the Right People:After Arriving On Board

●Qualification Cards (Qual Cards)●General Submarine Qual Cards●Individual Watch Station Qual Cards

●Under Instruction (UI) Watches (the RIGHT way to do OJT)

●Extensive Testing (Written and Oral)●On Board Continuing Training Program●Off Ship Navy Schools●Drills, Drills, Drills (Drill Coordinator is an actual assigned position)

Schneider Electric 8- Division - Name – Date

Procedures

●Only with permission of CO can procedures be violated – this mentality is religion

●All procedures are always conducted with a checklist in front of you, even if you have memorized them

Schneider Electric 9- Division - Name – Date

Procedures - Emergency

●Emergency Procedures●To quickly bring plant to a stable condition●Immediate Actions (these actions for EVERY Emergency Procedure must be flawlessly memorized by every watch stander)

●Supplementary Actions

Schneider Electric 10- Division - Name – Date

Procedures - Operating

●Operating Procedures●To carry out the day to day functions of the boat●I.e., Reactor Startup, Shifting the Electric Plant, Warming up the Main Engines

Schneider Electric 11- Division - Name – Date

Maintenance Procedures

●The PMS (Planned Maintenance System)●Daily, weekly, bi-weekly, monthly, bi-monthly, quarterly, semi-

annually, annually, bi-annually, every 3, 5, 10 years, etc.●Approvals and Notifications●Status Boards●Complete “re-commissioning” required for some equipment after

every maintenance procedure●Audits

●Some General Maintenance Practices●The Tag-Out System●The Valve Positioning Checklist●Electrical Safety Procedures

Schneider Electric 12- Division - Name – Date

The Tag-Out System

●Tag-Out Form always filled out first●Filled out and signed by 2 individuals●Presented to Officer-In-Charge with schematic of system affected for 3rd individual verification

●Yellow and Red Tags●“Hang Tags, Position Valves”●Some rules apply, i.e., “high energy” systems require 2 valve isolation and an electrical system must be completely deenergized whenever feasible

Schneider Electric 13- Division - Name – Date

The Valve Positioning Checklist (some examples)

●Check for Tags●Check the VPI (Valve Positioning Indicator)●Always check in the shut direction first●Partially position the valve and check for appropriate

system response●Fully position the valve and check for appropriate system

response●When opening valve, go to fully open, then back off ¼ to ½

turn●When shutting (not “closing!”) valve, torque as appropriate

using calibrated wrench●And the list goes on

Schneider Electric 14- Division - Name – Date

Electrical Safety Procedures (some examples)

●Completely deenergize and tag-out as much as possible

●Electrical safety area with rubber mats●Completely remove all metal●Minimize skin exposure and don electrical safety gloves

●Use only tools made “electrically safe”●Test for voltage first●Rope around waist and CPR training

Schneider Electric 15- Division - Name – Date

Policies

●Use of Common Language●I.e., port and starboard, shut vs. close, etc.●Avoid acronyms (even though Navy is full of them)

●Repeat Backs●Mandatory – no exceptions

Schneider Electric 16- Division - Name – Date

Policies (continued)

●Status Boards●All major systems status manually updated via grease pencils and status boards

●Use of Logs and Trending

●Meticulous Record Keeping

●Tool and Meter Calibration Programs

Schneider Electric 17- Division - Name – Date

Policies (continued)

●Watches●Pre-watch plant tours●Turnover procedures

●Checks and Balances inserted into everything

●Self-Audits●Internal and External Inspections

●ORSE

Schneider Electric 18- Division - Name – Date

Policies (continued)

●Lessons Learned and Incident Reports●Formalized program

●Shared Knowledge●Mandatory

●Manuals Update System●Very meticulous system for ensuring manual revisions and changes were always made correctly and on time

Schneider Electric 19- Division - Name – Date

System Design●First and foremost, the entire sub is “pre-engineered” so that system compatibility is ensured

●Alarms and Indicators●Interlocks and Redundancy everywhere●The Importance of Standardization

●Every class (I.e., Los Angeles, Ohio, Seawolf, etc.) of sub is virtually identical

●Personnel from one sub can transfer to another sub with little to no additional training

●Labeling, color coding, general procedures all identical regardless of type of sub

Schneider Electric 20- Division - Name – Date

The Irony of Human Error●Goal is to remove the “error” in “human error,” not the “human”

●Reliance on People●This one is important●Although much needs to be put in place to take out

human error, having the human in the process is critical to success (“can’t live with ‘em, can’t live without ‘em”)

●No machine can reason like a human●Saving lives means making hard decisions

Schneider Electric 21- Division - Name – Date

War Stories (some relevant, some funny, some interesting)

●Hurried Maintenance and the Blackout

●The Bubbler Boy

●The USS Greenville

●A Friendly Encounter

●Fast Water Pick-Up

●The Pipe and the COB

Schneider Electric 22- Division - Name – Date

The Result

●The US Nuclear Navy has never had a nuclear incident in over 50 years of operating nuclear ships and submarines at sea

Schneider Electric 23- Division - Name – Date

The Nuclear Propulsion Plant Operator

Tough Minded, Skeptical, Sometimes Even Cantankerous, But Always Technically Competent,

ALWAYS THINKING – What If?

A SPECIAL BREED. He Makes the Difference Between Safe and Effective Operation and

Unacceptable Risk.

Schneider Electric 24- Division - Name – Date

Now for the Big Question:

What, if any, of these concepts can or should be applied to

commercial data center facilities?

Schneider Electric 25- Division - Name – Date

Commercial Reality

●Cannot train for too long without contributing to bottom line

●Cannot force individuals to “follow orders”

●Cannot discipline in the same way

●Some HR issues are handled differently

Schneider Electric 26- Division - Name – Date

But, we can…●Provide Leadership

● Instill desire to perform at one’s top potential● Hire the best

●Utilize Non-Deviation Procedural Compliance● SOP’s for normal operating procedures● EOP’s for emergency procedure (easily accessible and memorized)● MOP’s for maintenance and other procedures● Change control process for all procedures● Maintain status boards● Methodical documentation of maintenance● Etc.

●Train When Possible● Schools● OJT● Perform drills

●Share Knowledge●These are just some examples, challenge yourself as to what else

might apply● Idea is to apply risk mitigation whenever possible

Schneider Electric 27- Division - Name – Date

Cool Pictures…

Schneider Electric 28- Division - Name – Date

Schneider Electric 29- Division - Name – Date

Schneider Electric 30- Division - Name – Date

Schneider Electric 31- Division - Name – Date

Schneider Electric 32- Division - Name – Date

Schneider Electric 33- Division - Name – Date

Schneider Electric 34- Division - Name – Date

Schneider Electric 35- Division - Name – Date

Schneider Electric 36- Division - Name – Date

Schneider Electric 37- Division - Name – Date

Schneider Electric 38- Division - Name – Date

Schneider Electric 39- Division - Name – Date

Schneider Electric 40- Division - Name – Date

Schneider Electric 41- Division - Name – Date

Schneider Electric 42- Division - Name – Date

Schneider Electric 43- Division - Name – Date

Schneider Electric 44- Division - Name – Date

Schneider Electric 45- Division - Name – Date

Schneider Electric 46- Division - Name – Date

Schneider Electric 47- Division - Name – Date

Schneider Electric 48- Division - Name – Date

Schneider Electric 49- Division - Name – Date

Schneider Electric 50- Division - Name – Date

Schneider Electric 51- Division - Name – Date

Schneider Electric 52- Division - Name – Date

Schneider Electric 53- Division - Name – Date

Schneider Electric 54- Division - Name – Date

Schneider Electric 55- Division - Name – Date

Schneider Electric 56- Division - Name – Date

Schneider Electric 57- Division - Name – Date

Schneider Electric 58- Division - Name – Date

Schneider Electric 59- Division - Name – Date

Schneider Electric 60- Division - Name – Date

Schneider Electric 61- Division - Name – Date

Schneider Electric 62- Division - Name – Date

Schneider Electric 63- Division - Name – Date

QuestionsAnd

Thank You