pep-ii operational safety and management presentation by roger erickson b-factory operations review...

PEP-II Operational Safety and Management

Presentation by Roger Erickson

B-Factory Operations ReviewSLAC, April 26, 2006

04/26/06 B-Factory Operations Review

OutlineOutline

Accelerator Matrix Management Communications with Stakeholders MCC Control Room Staffing Operator Profile and Training Accelerator Operations Documentation Maintenance Management ROD and Down Time Planning Conclusions


Main Control CenterMain Control Center


Matrix ManagementMatrix Management

Accelerator programs are managed through a matrix organization structure with links to:

Program Coordinators for PEP-II and Linac Pulsed Programs. Liaison persons representing each experiment. System Physicists for each major section of the accelerator facility. Area Managers – one for each major section, who report to the

System Physicists. Accelerator physicists and engineers responsible for subsystems or

commissioning tasks. Technical support departments with engineering and technician

support for on-call maintenance. Accelerator Division Safety Office. ES&H Departments, including Radiation Physics and OHP.


Communications with StakeholdersCommunications with Stakeholders

8:00 Meeting every day, seven days a week, in MCC conference room.• Operations report by EOIC, including run time statistics, progress in carrying out

scheduled program, descriptions of problems, etc.• Reports by representatives of major experimental user groups.• Attended by PPA, Accel Div, and Operations management reps, Program Deputy,

System Physicists, Area Managers, and representatives of user groups and major technical support departments; typical attendance: 45 persons.

8:15 Maintenance Meeting, five days each week.• Run by Accelerator Operations Department maintenance managers.• Attended by supervisors of all maintenance groups, Area Managers, and interested

physicists and engineers.• Attendance: typically 45 persons.

Accelerator Plans Meeting, weekly.• Review of operational progress, problems, and plans.• Attended by Accel Systems Div and Dept Heads, Program Coordinators, Accelerator

Section Heads and ADSO.


Control Room StaffingControl Room Staffing

Program Deputy• Duty rotates among Accelerator Division physicists.• Oversees program for 3 or 4 day period.• Not always present in the control room.

Operations Engineer• 1 EOIC per shift (from a group of 7), on rotating 7-week cycle. • Responsible for safely carrying out the scheduled program.• Exempt position; directs the Accelerator Systems Operators.• Must be present if accelerator is capable of producing beam.• All have experience as accelerator systems operators.

Accelerator Systems OperatorsTypically 3 operators per shift on rotating 5-week cycle: • Linac and sources. • PEP-II rings.• Secondary program, PPS control, and backup.


Typical Operator ProfileTypical Operator Profile

Typical applicant is a recent graduate with a bachelor’s degree in physics.

• 13 have bachelor’s degrees.• 7 have master’s degrees.• 4 have a technical background and extensive SLAC

experience.

Classifications:• Accelerator Systems Operator 1

• 1-2 years of experience• Accelerator Systems Operator 2

• 2-5 years of experience• Accelerator Systems Operator 3

• 5+ years of experience• Operations Engineer

• 5+ years of experience


Operator Training ProgramOperator Training Program

Workbook and evaluation for each classification:• Accelerator Systems Operator 1• Accelerator Systems Operator 2• Accelerator Systems Operator 3• Operations Engineer

ES&H Training (including RWT-II and LRCA)

Accelerator Operations Safety Procedures Training

(including PPS operation, searching, interlock testing for each zone.)

US Particle Accelerator School

Downtime seminars and special projects


Accelerator Operations DocumentationAccelerator Operations Documentation

Accelerator Division Operations Directives

Safety DocumentsSearch Procedures

(45 titles)

Training DocumentsASO-2 Qualification Workbook

(17 titles)

Administrative Documentsexample: Administrative Guidelines

(11 titles)

Accelerator Operations DocumentsLinac Turn-on Checklist

(38 titles)

Maintenance DocumentsCall-in Lists

(4 titles)

Documentation DocumentsAD Documentation Archive Plan

(5 titles)

Archived Documents and LogsMCC Operations Log

(83 titles)


AD Safety Documents with Site-Wide ApplicabilityAD Safety Documents with Site-Wide Applicability

SLAC Guidelines for Operations Work Authorization for the Linac, PEP-II, FFTB, and Associated

Buildings Summary of Requirements for Work in SLAC Accelerator

Housings Restart Plan for the Linac, PEP-II, and BABAR Restart Plan for the FFTB and ESA Accelerator Programs as an

Extension of the Linac, PEP-II, and BABAR Restart Plan Linear Accelerator Facility Safety Assessment Document PEP-II Safety Assessment Document BaBar Safety Assessment Document Policy for Working with Electrical Hazards in SLAC Accelerator

Tunnels Lock and Tag Verification Procedures for Accelerator Magnets in

the Linac Complex and PEP-II


AcceleratorAccelerator Operations Safety DocumentsOperations Safety Documents MCC Key Safe Checkout Log CID PPS Log DRIP/DR PPS Log Linac PPS Log Positron PPS Log BSY PPS Log ESA PPS Log FFTB PPS Log PEP PPS Log PLS Keys Procedures and Log MCC Spare Keys Key Safe Checkout PPS Zone Maps Logging Requirement for Work on

BCS Devices VVS Remote Turn-on Procedure Search Procedures Electrical Hazard Test Procedures Entry and Exit Procedures

Safety Inspection Checklists PPS Interlock Checklists Beam Containment System

Procedures High-Voltage Test Facility Safety

Procedures BSOIC Certification Checklists Gun Safety Procedures Laser Safety Procedures Fire Alarm System Reference Manual PEP PPS Overview Safety Configuration Work Control Incident Response Procedures Alarm Response Procedures Warning Response Procedures Accelerator Operations BAS

Instructions Accelerator Operations BAS Key and

Lock Procedures

•163 controlled documents in the Accelerator Division (incl. non-safety related).•MCC Operations Log is now electronic, web-accessible, and archived.


Operator Safety ProceduresOperator Safety Procedures

Operators are trained to carry out certain well-defined tasks, following formally controlled procedures.

Examples:• Interlock Checklists• Search Procedures• Entry and Exit Procedures• Safety Inspection Checklists

Safety procedures:• undergo a formal review and approval process.• are updated as conditions change.• are available on the AD Documentation web site, although the signed

originals are preserved in paper form.

Completed checklists and log books are archived.


Maintenance ManagementMaintenance Management

Problems that interrupt the scheduled program trigger immediate repairs to restore operations.

24-hour on-site staff to repair:• Mechanical utilities• Power supplies• Modulators• Cryogenics

On-call staff for:• Electrical power• Magnets and other mechanical systems• RF systems• Vacuum systems• OHP

Extensive call-in lists identify first responders, alternates, and experts for all systems.


Opportunistic MaintenanceOpportunistic Maintenance

Opportunistic Approach• Whenever possible, identified maintenance, repair, and installation jobs are pre-

planned, but deferred until an opportunity arises. The accelerator systems are not turned off for maintenance on a routine, scheduled basis.

Coordination• Daily 8:15am meeting.• Coordinated by Operations Maintenance Office.• Area Managers are advocates for their areas.

Standby Maintenance List (SML)• Jobs that are planned and can be launched on short notice.• Provides options for EOIC to take advantage of unscheduled interruptions.

Repair Opportunity Day (ROD)• Launched only when required by a driving event.• Carefully planned, and typically involve 100 or more jobs.

ARTEMIS Electronic Database


ARTEMISARTEMIS

ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance Information System) is an Oracle database with a Remedy interface, accessible over the internet by everyone involved with SLAC accelerator maintenance.

Operators report problems as they occur by making entries in ARTEMIS.

Repair work is launched immediately if the problem is inhibiting the scheduled accelerator program.

New ARTEMIS entries are reviewed at the daily maintenance meeting. Open questions are addressed and jobs are assigned.

Jobs may be assigned to the Standby Maintenance List (SML), or to the Repair Opportunity Day (ROD) list, or marked for the next long down time.

Jobs are tracked using ARTEMIS until they are finally closed.


Daily Maintenance Meeting Daily Maintenance Meeting


Review of New ProblemsReview of New ProblemsSample from an actual maintenance meeting.


Standby Maintenance ListStandby Maintenance List


Repair Opportunity DaysRepair Opportunity Days

A ROD list of pending repair jobs is derived from the ARTEMIS database and maintained by the Operations Department. Installation and PM jobs are added as needed.

All ROD jobs are reviewed in the daily maintenance meetings.

The ROD list can be viewed over the internet.

When a “driving event” occurs, a detailed plan can be assembled and launched quickly, using the ROD list.

A typical ROD day includes about 100 distinct jobs.


Long Down Time PlanningLong Down Time Planning

Long Down Times usually occur once each year and typically last two to three months.

Area Managers collect job requests from system physicists, project engineers, maintenance groups, and the ARTEMIS database.

A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting about four months before the down time.

This schedule lists each job with an assigned shop, and lists the Project or Group Manager, Responsible Task Person, and Area Manager.

Schedule specifies time and duration of each job (with 1 day time resolution), and number of persons required.

Schedule specifies authorizations required (WAF, RSWCF, EWP, etc.).

Detailed (shift by shift) schedule of pre-run testing, safety certifications, and turn-on activities is prepared by the Operations Dept and revised as needed on a real-time basis.


Maintenance StatisticsMaintenance Statistics

ARTEMIS entries since Jan 1989: 77,328 (as of April 18, 2006)

Total identified problems not yet closed: 456Many of these involve non-critical problems that were observed, but went away without a clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.

ARTEMIS entries in one year (since April 18, 2005):• 3166 new problems (8.7 per day).• 450 problems resulted in lost PEP-II run time.• 3296 problems closed (9.0 per day).

Active jobs:• SML: 52• Next ROD: 32• Aug-Nov down time: 80


Operational Safety Reviews

Linac and PEP-II Re-Start Validation Review (March 2005)

FFTB Re-Start Review (June 2, 2005)

ESA Re-Start Review (July 19, 2005, secondary beams only)

Director’s Integrated Safety Management (ISMS) Review (Sept 2005)

DOE ISMS Review (October 2005)

Injector Test Facility Re-Start Validation Review (October 2005)

ESHAC Review (November 2005)

ESHAC Review (March 2006)


New Safety Processes Introduced in FY05

JHAMs (Job Hazard Analysis and Mitigation) individualized and documented for every employee.

Expanded training requirements for most workers.

AHAs (Area Hazard Analysis) posted for every building.

Arc-flash analysis and warning labels for circuit breakers and contactors (except Cat 0 light switches, etc.)

All hoisting and rigging equipment, including lifting fixtures, straps, etc. must be certified.

Fall protection equipment and training required for tall ladders, raised platforms, rooftop work, etc.


ConclusionsConclusions

PEP-II B-Factory operations are managed by the Accelerator Systems Division in the PPA Directorate.

Control room operations are carried out by trained operators in compliance with documented policies and procedures for all safety-related activities.

Maintenance and installation work is accomplished through a matrix organization, drawing on engineers, technicians, and ES&H and shop services from the Operations Directorate.

We have developed organizational structures, software tools, and documentation that enable us to manage an ambitious accelerator-based science program safely and effectively.

Enhanced policies, procedures, and training programs introduced in the past two years ensure that all our work is carried out with high standards for safety.