availability performance of lcls x-ray fel at slac william colocho for the lcls team

Availability Performance of LCLS X-Ray FEL at SLAC

William Colocho for the LCLS team.

2

Outline

• Quick LCLS Overview

• Availability and downtime vs. time

• Downtime per system and planned upgrades

• Tuning and Availability

• Operational Availability Goals

• Questions

ARW 2011/LCLS AvailabilityCape Town, April 2011

3

• Operated by Stanford University for the US Department of Energy

• Study the molecular world in ways never before possible

• 3 of 6 LCLS scientific instruments fully commissioned

• By February 2011 1,162 unique scientists from 28 countries have submitted 427 proposals to use LCLS • Typically two experiments are scheduled for a five day user run.

LCLS FEL at SLAC


4

A Tool for the Global Science Community


“By 2013 six instruments will give researchers unprecedented tools for a broad range of research in material science, medicine, chemistry, energy science, physics, biology and environmental science” (LCLS fact sheet:http://www-group.slac.stanford.edu/com/docs/lcls_fact_sheets.pdf )

5

LCLS Free Electron LASER

Parameter Hard x-rays Soft x-rays Short pulse soft

Short pulse hard

unit

Final e- energy 6.7 – 14.7 3.3 – 6.7 3.3 – 6.7 6.7-14.7 GeV

Bunch Charge 150,250 150,250 20,40,80 20,40,80 pC

Pulse Duration 70-250 70-400 <10 <10 fs (fwhm)

Repetition Rate 1,10,60,120 1,10,60,120 1,10,60,120 1,10,60,120 Hz

Photon Bandwidth

~0.2-0.5 ~0.2-0.5 ? ? %


For full list see: https://slacportal.slac.stanford.edu/sites/lclscore_public/Accelerator_Physics_Published_Documents/LCLS-parameters.pdf

• Each experiment selects a set of parameters to optimize their science goal.• Each configuration has different hardware and software requirements.

6

LCLS Major Components

• 197 Quadrupoles• 158 X, 159 Y Steering

Correctors• 175 Beam Position

Monitors• Klystrons/Modulators

– 4 Injector Stations– 31 “Linac 2” Stations– 48 “Linac 3” Stations

• 33 Undulators• 33x5 CAM Movers• ~1.8 Km of Vacuum• ~124 Vacuum Pumps• ~ 45 CAMAC crates• 2 Injector LASER

systems• 33 Beam Shut Off Ion

Chambers


7ARW 2011/LCLS Availability

Cape Town, April 2011

LCLS Availability: 94.8%

Program Delivered User OffConfig. Change

Tuning DownScheduled

OffPhotons Hardware

USER 87.6 3.1 4.2 1.9 3.3 0.6 94.8 96.7

COMM 85.8 6.1 3.2 1.9 2.8 0.3 95.2 97.2

ALL 87.1 3.8 4.0 1.9 3.2 0.5 94.9 96.8

Percent of Scheduled Hours (%)

Plots and tables with the latest LCLS Availability information are automatically updated every shift during user runs.

COMM: Hutch Commissioning Programs.

Run I: 92.8% Photon, 97.0% HardwareRun II: 92.5% Photon, 97.2% Hardware

Runs I and II Photon Availability combined“Tuning” and “Configuration Change” as one category.

8

LCLS Availability Terms

• Photon Availability: (Delivered + User Off + Configuration Change) / Scheduled On

• Hardware Availability: (Delivered + User Off + Configuration Change + Tuning)/ Scheduled On

• Scheduled On: Delivered + User Off + Configuration Change + Tuning + Down


9

Photon or Operational Availability

Short term performance as important as long term.

Users scheduled for machine time for five day intervals.


10

Trends• Tuning, Downtime and Configuration Change time improved during the user run.

• Tuning and Configuration Change time reduction likely due to increased operational efficiencies.

• Downtime reduction not fully explained. Further analysis (Weibull analysis) needed.

•Notice correlation between tuning time and down time. (Down time leads to tuning time).




Downtime Statistics and future upgrades

12

MTBM and MDT• Mean Time Between Maintenance Includes Corrective Maintenance and it does not include Preventive Maintenance.

• We accept that there will be preventive maintenance scheduled.

• This MTBM number should better reflect “Maintainability” when it most matters: During beam delivery to users.

•Only Selected Systems shown. See conference paper for full list.

• Data is 6 month snapshot (69.7 operating days).


SystemMTBM (Days)

MDT (Hours)

1.3 Power supplies 17.4 0.41.6 Linac Modulators 10 0.31.7 Subbooster supplies 23.2 0.93.1 S-band Klystrons 34.9 0.83.3 Hi-power Subboosters 34.8 1.13.6 sol. Stat subboosters, Low-Level RF 6.3 0.65.1 Electrical (on site) 23.2 0.35.3 Cooling water (mechanical) 69.7 1.46.2 LCLS gun laser system 7.7 0.97.1.1 Micros & Camac crates 11.6 1.47.1.3 Timing 23.2 0.57.1.4 Vacuum I&C 34.9 0.27.1.15 MPS 34.8 1.27.2.1 MCC:: (VMS System) 69.7 1.57.3.2 PPS 7.7 17.4.1 Controls Software 8.7 0.510.1 Procedural Error 23.2 0.410.2 Physicists Software 34.9 0.912.1 PCDS Hardware 13.9 0.2

13

Tuning Reduction Task Force

• Members from Physics, Operations and Software groups.

• Working on reducing tuning time during Machine Development and Repair Days recovery.

• Expect this effort to also reduce tuning time during User runs.

• Tuning reduced operational availability by ~2%


14

Tuning Task Force

• Identify how tuning time is spent; Mine data from electronic logbook. Talk to Operators.

• Measure execution time of software procedures.

• Understand Machine configuration reproducibility.

• Ongoing development of software to simplify and automate tuning procedures.


15

Tuning: Need Metrics Pie!

• Injector Emittance, LI21 Emittance X, LI28 Emittance X, LTU Emittance X, Injector Matching, LI28 Matching, LTU Matching, Injector Phase Scans, Charge Change, Energy Change, Mirror reconfiguration, DL2 Dispersion with Quads, DL2 Dispersion with CORs, Undulator Re-pointing, Cathode Alignment, BSY Steering, LTU Steering, Laser Heater Alignment, Laser Heater Timing, C-Iris steering, MPS Trips at some settings, MPS Trips with Slotted Foil, Taper reconfiguration, Gas Detector Calibration, E-loss scan, Klystron Phasing, RF Amplitude Calibration, Zeroing e+ correctors, BBA, LI21 Emittance Y, LI28 Emittance Y, LTU Emittance Y, XCAV Set-points for FEL, L3 Steering…


16

LCLS Availability Goal is 95%

• That is 95% Operational Availability to Users.

• Before run III, goals did not take Tuning into account.

• A tuning budget (~2%) will be included in the availability plan. This sets the hardware availability goal at 97%.


17

Realizable Goals


Sub-System

RUN III

Previous Goal New Goal

Power Supplies (See Session 9, Wednesday) 99.6 99.42 99.77 Magnets (See Session 14, Thursday) 100 99.96 99.95 RF

99.37 98.98 99.44 Vacuum

99.93 99.86 99.95 Utilities

99.84 99.38 99.84 Guns & lasers

99.52 99.42 99.64 Controls

98.36 98.23 98.66 Non-Radiation Safety

100 99.95 99.95 Alignment

100 99.82 99.95 Other

99.82 99.92 99.93 Unassigned

99.99 99.98 99.95 Photon Controls

99.93 99.98 99.95 Total

96.7 95.00 97.00

• Goals based on past performance

• Remove worst two weeks or events from historical data (Life of the project ~2.5 years).

• System owners in charge and accountable for given system’s availability performance.

18

Conclusion

• LCLS science program is ongoing.

• Two year hardware availability ~ 97%

• Photon availability of 95% realizable.

• Multiple upgrade projects underway.

• Further analysis needed to better understand long term time dependence of individual subsystem and component failure rate.


19

Thank You!

• Questions.


If we use failure rates as a metric for determining which systems (and therefore groups) will receive attention and/or resources; are we setting up a system where failure is rewarded?

In a climate of decreasing resources and budget constrains: Is there a model that correlates Availability and Reliability with Budget Allowance?

availability performance of lcls x-ray fel at slac william colocho for the lcls team

Documents

lcls availability cape

hardware availability

lcls fel

lcls team

lcls scientific instruments

lcls major components

configuration change

tuning time