streamlining cdf shift operations
DESCRIPTION
Streamlining CDF Shift Operations. Peter Wilson Mary Convery CDF Detector Operations All Experimenters Meeting February 26, 2007. Running CDF thru FY2009. Challenge: collect high quality data with high efficiency as the pool of physicists shrinks! LHC will turn on in FY2008 - PowerPoint PPT PresentationTRANSCRIPT
Streamlining CDF Shift Operations
Peter Wilson
Mary Convery
CDF Detector Operations
All Experimenters Meeting
February 26, 2007
Peter Wilson - All Expt Meet
February 26, 2007 slide - 2
Challenge: collect high quality data with high efficiency as the pool of physicists shrinks! LHC will turn on in FY2008 Many people working on CDF also on ATLAS or CMS Expect fewer Postdocs and students in 2008/2009
However, we don’t need as many people as in 2001: Experiment is mature Systems are well understood Procedures are well defined Upgrades/changes are minimal
Effort initiated by previous Ops heads (Willis Sakumoto and Pat Lukens) to re-organize shift operations to use fewer people while maintaining or increasing operational efficiency
Running CDF thru FY2009
Peter Wilson - All Expt Meet
February 26, 2007 slide - 3
CDF Shift Crew - Ops Man
Operations Manager (Ops Man) - On call 24/7 Makes daily operational plan Typically experienced postdoc - 1 wk on/2 wks off for 6 months
CDF Control Room
Peter Wilson - All Expt Meet
February 26, 2007 slide - 4
CDF Shift Crew - SciCoScientific Coordinator (SciCo) - 8/7 x 3shifts Leader of the shift crew Responsible for implementing daily plan Communicate with MCR Summon expert help Provide food for the crew! Faculty, lab scientific, staff or senior postdoc
CDF Control Room
Peter Wilson - All Expt Meet
February 26, 2007 slide - 5
CDF Shift Crew - DAQ Ace
DAQ ACE - 8/7 x 3 shifts “Runs” the experiment Controls the data acquisition system
Setup run configurations Start/stop DAQ runs
Monitor DAQ status, diagnose and fix DAQ problems Run calibrations
CDF Control Room
Peter Wilson - All Expt Meet
February 26, 2007 slide - 6
CDF Shift Crew - Mon AceMonitoring ACE - 8/7 x 3 shifts Detector HV monitoring and control. Diagnose/fix HV problems Monitor status of low voltage power supplies Responsible for safety of Si in the event of a cooling alarm Monitor beam conditions for detector safety Twelve Students and postdocs cover DAQ & Monitoring Shifts in 10 week rotation
CDF Control Room
Peter Wilson - All Expt Meet
February 26, 2007 slide - 7
CDF Shift Crew - COConsumer Operator - 8/7 x 3 shifts Data quality monitoring for all detector subsystems Ten “Consumers” each receiving a few Hz (of ~100Hz) of events from L3
Detector hit occupancies and average pulse-heights in minimum bias data Trigger rates and trigger algorithm validation (compare emulation to data) DAQ performance and errors
Compare results to references Check all routine detector calibrations Any CDF physicist
CDF Control Room
Peter Wilson - All Expt Meet
February 26, 2007 slide - 8
CDF Shift Crew - CryoProcess System (Cryo) Operator - 12/7 x 2 shifts Solenoid and Cryogenic System operation Gas systems Cooling systems for electronics and detectors Electronics rack protection Four PPD technicians on rotation, one other tech fills in for sick leave/vacations
CDF Cryo Control Room (1st Floor)
Peter Wilson - All Expt Meet
February 26, 2007 slide - 9
Streamlining the Shift Crew
Consumer Operator Shifts taken remotely Pilot operation at INFN-Pisa Reduce need for travel for overseas members of CDF “OWL” shift is 7:00-15:00 in Pisa, Italy or 15:00-23:00 in
Tsukuba, Japan
Reduce the number of Aces from 2 to 1 Consolidate responsibilities with the remaining ACE Transfer some monitoring tasks to Cryo Operator SciCo trained on some tasks of monitoring Silicon detector
- Safety of detector if Ace is not in control room (eg high beam losses)
Automate tasks to allow shift crew to focus on decision making
Peter Wilson - All Expt Meet
February 26, 2007 slide - 10
Remote CO Effort started early in 2006:
All monitoring plots available via Web Tools for remote starting and stopping of Consumer programs CO shift room in Pisa: monitoring workstations and video conferencing CDF Pisa Video conference for SciCo-CO communication Remote CO does all of the same functions as local CO Experience shared with and gained from US CMS remote operations Team from Fermilab, Pisa, Tsukuba, UCLA and Waseda led by Kaori Maeshima
and Fabrizio Scuri First official remote shift: week of November 10
Commissioning in October 2006: simultaneous Remote and local CO Through June 2007: schedule 1week/month of remote CO from Pisa Easy to extend to other locations: planning to start shifts in Japan starting July 2007 Goal: all OWL shifts covered remotely in 2008-2009
Peter Wilson - All Expt Meet
February 26, 2007 slide - 11
Single ACE Operation Planning started in Summer 2006 Implementation starting mid-October
Re-arranged control room computer consoles- Added ACNET and HV control next to DAQ consoles
- DAQ and Monitor functions from one work area Automated recovery of Silicon HV crate communication failures, Ace
previously left the control room (Hockerization Auto-Hockerization) Automated generation of Hourly ACNET plots of beam losses Updated monitoring pages and documentation for Silicon Cooling and
LV PS monitoring Cryo Operator Add audible alarm for Silicon detector PS current monitoring SciCo Improved SciCO training Team from Silicon detector group, CDF Ops, CDF DAQ and
Accelerator (Ron Moore and Brian Hedricks) led by operations heads First shift with single Ace Dec 29
Followed 10 days where 2nd Ace acted only as backup
Peter Wilson - All Expt Meet
February 26, 2007 slide - 12
CDF Owl Shift - Feb 15
SciCo - John Yoh
CO in Pisa (Simone Donati)
Ace (Michal Kreps)
Peter Wilson - All Expt Meet
February 26, 2007 slide - 13
Owl Shift in Pisa - Feb 15
Peter Wilson - All Expt Meet
February 26, 2007 slide - 14
Efficiency High with Reduced Crew
Monthly average good runs w/ Si
Peter Wilson - All Expt Meet
February 26, 2007 slide - 15
Experience with Reduced Shift Crew
No significant problems with shift crew changes, biggest challenges have been substitutions for shifters who are ill! New Years weekend needed to find substitute Ace Remote CO needed to leave after 6 hours of shift
Operational efficiency has not dropped With single Ace, crew is operating even more as team Experience from three weeks of remote CO: as effective as
local CO Automation is reducing downtime:
Recovery of Si HV Hockerization took 10-15min Auto-Hockerization now takes about 4 mintues Typically about once per store
Peter Wilson - All Expt Meet
February 26, 2007 slide - 16
Additional Improvements Coming
In case of dangerous beam conditions, detector HV must be set to Stand-By: CDF TEVMON tool monitors losses, abort gap beam, TevRF, etc
based on ACNET variables, warns shift crew of dangerous conditions If shift crew is occupied might be delayed in responding New TEVMON based on an Open Access Client (OAC) by Ron Moore
- ACNET variables indicate danger level and provide permit for operation of detectors
- Automatically set Silicon HV to standby in dangerous conditions Final testing underway
HV problems sometimes require Ace to leave control room: Plug calorimeter HV regularly requires procedure to re-establish
communications move control PC and interface hardware from 3rd floor to control room (2nd flr)
Other HV systems need to have programs restarted or PCs to be rebooted on 1st floor working on remote access to HV PCs from control room
Peter Wilson - All Expt Meet
February 26, 2007 slide - 17
Some Key Members of the Team
Peter Wilson - All Expt Meet
February 26, 2007 slide - 18
Members of the TeamThank you to the many people who helped make this happen: Fermilab: Bill Badgett, Kurt Biery, Ray Culbertson, Steve Hahn, Brian
Hendricks, Jonathan Lewis, Kaori Maeshima, Ron Moore, JJ Schmidt, Peter Wilson
CDF Technical support team: Dervin Allen, Del Allspach, Jim Loskot, Lew Morris, Bill Noe, George Wyatt
Duke: Dean Hidas INFN Pisa: Dario Fabiani, Luciano Ristori, Fabrizio Scuri, Jose Enrique-
Garcia Tsukuba: Taichi Kubo, Koji Nakamura Waseda: Junji Naganoma, Tetsuo Arisawa Rochester: Jennifer Gimmel, Willis Sakumoto Rockefeller: Mary Convery UCLA: Charles Plager Yale: Ulrich Husemann
Peter Wilson - All Expt Meet
February 26, 2007 slide - 19
Summary CDF has successfully reduced the shift crew from 4 physicists
to 3. 25% of Owl shifts have only 2 at B0
Process of reducing shift crew has improved the tools for shift takers
Operational efficiency has been maintained or improved during the transition Additional improvements are coming which should result in more
efficient shift operation We are working on other improvements to increase our
operational efficiency and prepare for a smaller physicist support For example, adapting Electronic Worklists from AD for scheduling
accesses and tests