dt shifter training infrastructure a. benvenuti, m.giunta 07/07/2010
Post on 06-Jan-2018
220 Views
Preview:
DESCRIPTION
TRANSCRIPT
DT Shifter Training
Infrastructure
A. Benvenuti, M.Giunta
07/07/2010
General Info• The Technical Shifter should call DT Infrastructure Expert (165507)
• 79930 given as a backup
• If you need an expert you can call:
163651 (Alberto)162355 (Marina or Daniele)
DT Safety Considerations• Overheating:
- Inside the detector (MiniCrates) due to Cooling Plant failure (MUON Circuit), affects all MC, or obstructed flow-limiter, affects two MC in adjacent sectors.- Electronic modules inside Tower or USC Racks, due to Cooling Plant failure
(Racks Circuit) that affects all racks, or a turbine failure that affects only the modules inside one rack. The most dangerous case is a turbine failure in a tower rack containing Sector Collector crates.
•Water Leaks:- Are detected by “leak detection” cables inside the detector and tower racks. Major leaks can also induce HV trips in one or more chambers.
DT expert (165507, backup: 79930) should be called for any error induced by Overheating or Water leaks •Gas Failure:
- Not critical, the DT chambers can be operated for ~ day without gas flow
DT States• ON:
- LV ON- HV ON: wires +3600V, strips +1800V, cathodes -1200V
• Standby:- LV ON- HV ON: wires +1900V, strips +1800V, cathodes -1200V
• Dormant:- LV ON- Sector Collector mezzanine boards (opto couplers) OFF- HV ON: wires +1900V, strips +1800V, cathodes -1200V
• OFF:- LV OFF- HV OFF
The DT should be in Standby mode when the Magnet is ramping.
http://cmsonline.cern.ch/portal/page/portal/CMS%20online%20system/DCS/Automation?_piref815_847251_815_847248_847248.strutsAction=%2FsubmitDetector.do%3Bjsessionid%3D898af9ec30d87aa1ed82970647a6b953ea8018662e18.e34LbhiLbx8Nay0LbxqRbN8Naxb0n6jAmljGr5XDqQLvpAe%3FSelectedDetector%3DDT
DT State vs Machine/Beam State
How to connect to PVSS from home (I)• RemoteDesktop connection to the Terminal Server:
- Connect to cerntscms01 as mudcs (usual pwd)- tunnel on lxplus → Source port: 8881, Destination: cerntscms01.cern.ch:3389- RemoteDesktop connects to→ localhost:8881
•For MAC Users: you can use CORD
How to connect to PVSS from home (II)
DT Detector Panel
1)
2)
7)
4)
5)
6)
3)
3)
Error conditions flagged by Red color on tabs and audio alarm (to be implemented):1) Cooling2) UX / USC Turbines, Smoke
alarms 3) MC
Temperatures/Communication Error
4) Water Leaks5) LV Temperatures6) Gas System7) More than 1 MC LV error8) HV Problem
6)1)
5)
8)
Yoke Circuit Flow Error• No action required
Minicrate (MUON) Flow Error• Cooling Plant Problem Call DT Expert (165507)• MC will switch OFF after 15’
Rack Circuit Flow Error• Cooling Plant Problem Call DT Expert (165507)• LV, HV switch OFF after 20’
Supply T > 18°C • Cooling Plant Problem Call DT Expert (165507)• DT-DCS shutoff in 30’
Time windows can be reset from “Reset Timeouts” button
Cooling Problem
Cooling Problem
• Tell the Technical Shifter to contact CCC and find out the time estimate to fix the problem
• Call immediately:163651 (Alberto) or162355 (Marina or Daniele)
UXC Rack Problem
Turbine is OFF (current ~ 0.4A) Call DT Expert (165507)
High Voltage USC:
• HV OFF in corresponding wheels within 300 seconds. Time window can be reset by apposite button
LV USC S4F02: No Action Required
USC Rack Problem
• Check that affected components have been switched OFF• Individual modules can be switched OFF by clicking the “ON” button• Entire Wheel can be switched ON/OFF from the “Wheel ON-OFF” panel
The LV Panel can be reached from the Detector Panel clicking the “MAO” button
Caution: Toggle Buttons if pushed will change state
MAO Panelactions described in
page 6
A drop of water in one of the leak detection cables will trigger an alarm and the corresponding buttons will turn red.
Call DT Expert (165507) No further action required from shifter
Water Leak
• A3100 Modules power the SC crates and have the largest load hence the high T.
• In Standby mode the SC Mezzanines will be off and the Temperatures will be lower.
• If the Temperature is above 40°C in several modules there is a cooling problem in the “Rack circuit”
Time Histories are available from the buttons
LV Boards Temperature Problem
Gas Alarm due to:• Stopped gas flowDT Chambers are safe without gas circulation for ~ 1day. Do NOT call expert during the night
• Communication Error Data are not stored in DataBase Call DT Expert (165507) from 08:00 to 23:00
Button becomes RED when Communication is lost
Gas Problem
HV Problem
During the day:Call DT Expert (165507)
During the night:Open ‘HV Layers Info’ panel: if you have>2 entries in the ‘Layers in STANDALONE’ columnOr>5 entries in the ‘Layers not ON’ column Call DT Expert (165507)
All Previous Problems
• Call immediately:163651 (Alberto) or162355 (Marina or Daniele)
YB+1 YB+2MiniCrates
Temperature
• If Temperature exceeds 40°C for 1 or 2 MC call Expert and it will help you to Switch OFF the MC• If high T appears in several MC it is a cooling problem
To Reach this panel
MiniCrate Temperature Problem
MiniCrate Temperature Problem
• In one MiniCrate- Switch OFF the MiniCrate (see next slide)
• In several MiniCrates- Probable Cooling Problem, call immediately:
163651 (Alberto) or162355 (Marina or Daniele)
How to switch off one Minicrate
You can choose if doing it yourself or asking the Technical Shifter to do itIf you want to do it:
- ask the Technical Shifter to Release the partition (DT+, DT- or DT0)- Take(*) the partition- Perform the action- When you’re done Release(*) the partition- Tell the Technical Shifter to Take(*) back the partition(*) see next slide to learn how to Take and Release
Give a SWITCH_OFF_LV command from the Chamber Panel
How to Take/Release a FsM node
Click the open lock and then ‘Take’• How to Take a node
• How to Release a nodeClick the lock and then ‘Release’
MiniCrates Communication Error
In case communication is lost between PVSS and the MC control program, the Wheel Banner will turn red and blinking and the chambers will turn orange.Click the “Reset” button on the wheel and send “MC_Reset” from the top node.
The LV will be switched OFF after 30’. Call DT Expert (165507)
You must “take” the “DT_Detector” to do any actionand “release” it once you are done
MiniCrate Communication Problem
• In one wheel • Restart the CCB Server for that wheel, as explained in the DT Expert
FAQ twiki: https://twiki.cern.ch/twiki/bin/view/CMS/DTexpertFAQ2009x07x27
• In ALL the wheels• Call DT Field Manager
MiniCrate with Boards Off
2.00
• Try to switch ON the MiniCrate (from DCS GUI -> Power All) once- If command fails or boards switch off again, call DT Field Manager
If the boxes are not green it means the LV channels actually tripped; in this case call the DT Field Manager
• HV and LV can be switched OFF with a fast but safe procedure from the “Emergency Stop” buttons
Automatic DCS/DSS ActionsFailure condition Agent Threshold Time Action Granularity How to suspend action
DT-DSS 200'' SMS to expertDT-DSS 10' SMS to expertDT-DSS 15' SC and MC power cut All
DCS smooth LV power off AllDT-DSS 170'' Kill LV All
DSS 180" LV power cut AllDCS smooth HV power off (+,0,-)DCS smooth LV power off one wheel
DT-DSS 6' Kill HV (+,0,-)DSS ~7' HV power cut (+,0,-)DCS smooth HV power off All
DT-DSS 6' Kill HV AllDSS ~7' HV power cut AllDSS HV power cut AllDCS 1" LV power off All
DT-DSS < 1'' Kill LV & HV AllDSS 3'' LV power cut All
MC high temperature MC 40°C ~10'' Auto power off Single MCTSC mezzanine 50°C Auto power off Single board
DT-DSS 29°C Top crate SMS to experts27°C Bottom crate SMS to experts
DT-DSS 30°C Top crate SC and MC power cut SC one wheel, MC half wheel28°C Bottom crate SC and MC power cut SC one wheel, MC half wheel
DCS smooth Rack power off Single rackDSS 1' Rack power cut Single rack
Smoke/Fire in one of S4F01<->S4F09 racks
DSS 1' Rack power cut All racks
Smoke in USC DSS Racks power cutGas flow error DCS SMS and Email
Wrong gas mixture Gas Group Turn gas offGas overpressure Release valves Restore normal pressure
MC cooling flow DCS10 l/min each wheel
or 1 m3/h for all15' LV power off All
Rack cooling flow DCS 10 m3/h 20' LV & HV power off AllRack cooling supply over temp DSS 18°C 30' LV & HV power off All
AUG Nothing can be done
PVSS - LV mainframe communication error
Power cut of LV racks
Power cut of HV racks
SC high temperature
Smoke/Fire in rack
Pause or reset the watchdog: https://twiki.cern.ch/twiki/bin/view/CMS/DTexp
ertFAQ2009x08x04Increase the latency time of the LV Kill signal:
https://twiki.cern.ch/twiki/bin/view/CMS/DTexpertFAQ2010x03x09
Increase the latency time of the HV Kill signal: https://twiki.cern.ch/twiki/bin/view/CMS/DTexp
ertFAQ2010x03x10
Same procedure as https://twiki.cern.ch/twiki/bin/view/CMS/DTexp
ertFAQ2010x03x09 but at the end unclick the "Enable" button in the Pt1000 frame
Smooth shutdown button (one per wheel)
Same procedure as https://twiki.cern.ch/twiki/bin/
view/CMS/DTexpertFAQ2010x03x10 but change "Button timeout"
Problem Granularity• HV
• LV
Failure Fraction of DT affected
1 channel 1 layer
1 A877 1 chamber
1 A876 1 sector
1 SY1527 4 or 8 sectors (in 1 or 2 wheels)
1 HV rack 1 or 2 wheels
Failure Fraction of DT affected
1 channel (A3050 or A3009) 1 chamber
1 A3050 1 or 2 chambers
1 A3009 or 1 Easy 3000S crate 1 sector
1 MAO (except S12) 1 or 2 sector (N.B. MAO S12 = 1 wheel/2 SC crates)
1 A3100 6 sectors/1 SC crate
1 LV Rack (except X2 Near) 1 quadrant (N.B. X2 Near = 1 wheel/2 SC crates)
1 Branch Controller (A1676A) half wheel
SY1527 in S4 whole detector
top related