mp3 report : w27 to w33 (22 slides, 10’ ?) j.ph tock for the mp3 team: te-mpe : n. catalan, r....

1

MP3 report : W27 to W33(22 slides, 10’ ?)

J.Ph Tock for the MP3 team:TE-MPE : N. Catalan, R. Denz, M. Koratzinos, A. Verweij, M. Zerlauth

TE-MSC : B. Auchmann, S. Le Naour, M. Modena, A. Musso, J.Ph. Tock+ K Dahlerup, E Nowak, A Rijlart, Z Charifouline

JPhT – MP3 18th August 2010

- Technical issues- Organisation update

JPhT – MP3 18th August 2010 2

• EICs have the right to and are doing the swaps / Also sending loggingMP3 has to be informed of QPS issues (written in MP3 logbook or email or …)

• Will run on board B before next technical stop to identify problematic channels ; only communication is affected and not the protection

• A new firmware is under test in the lab so this will become transparent Implementation is foreseen during the Christmas break

• Circuits concerned (from MP3 logbook) on > 5 weeks:RCBXV2.R2 (10), RCBXH2.R2(2), RCO.A56B1&2 (5), RCBXV&H1.R2 (3) , RQS.A12B2 (5), RQTL11.L5B2 (5), RQTF.A67B2 (2), RQTF.A81B1 (5), RQTF.A56B2 (2), RQTD.A56B2(1) + some more from OP logbook

Out of 2400 boards

Switches from board A to board B


• 15/7 : RB.A45 : Quench heater power supply of B17.R4 dischargesReplacement postponed to the day after with monitoring of the 3 other power supplies

• 17/7 : RB.A56 : Quench heater power supply of C34R5 discharges• 20/7 : RB.A12 : Quench heater power supply of C34L2 discharges• 20/7 : RB.A56 : Quench heater power supply of B11L6 discharges• 23/7 : RB.A45 : Quench heater power supply of C18.R4 discharges• 28/7 : RB.A45 : Quench heater power supply of B8.L5 discharges• 17/8 : RB.A12 : Quench heater power supply of B24.L2 discharges

Discharge of Quench Heater Power Supply

The procedures were reviewed and agreed. We should not start a fill without redundancy but we may allow to loose redundancy during the fill. So, if an HDS is lost when there is beam in the machine, nothing will happen. However the injection permit is lost and no new beam will be injected … except if QPS_OK masked

Was not applied yesterday … QPS_OK was masked (no mention in logbook) for injection so relying on the operator to check if no more QHPS discharges.

Software will check this automatically and initiate (after EIC validation?) a slow power abort if 2 dipoles or 1 quadrupole QHPS is discharging. Implementation planned for October TS

• 7 cases in > 5 weeks – MTBF is 106 hours - > 6000 QHPS in the LHC• Fault identified ; some replacement will take place. The availability should be

improved by a factor of at least 10


• Spurious trigger signal not related to any real event which is self curing (FILLING BUFFER --> FILLING BUFFER NOT OK --> FLT TIMEOUT --> LOGGING --> QPS-OK back)The origin of the temporary problem is related to the Global Busbar Detector, not filling the buffer, is related to a time-out and will last for 2 to 5 minutes. It has an automatic reset (6/8, 9/8,..)

In case of loss of QPS_OK, waiting 5 to 10 minutes can solve the problem

• 15/8: Checking QPS signal in a stable beams phase, it was noticed that QPS_OK was lost for a communication problem. Protection is still ensured. Intervention for a reset of the local main BB detector in the tunnel was planned that solved the issue

Personnal computer to be rebooted locally (access) ; will be solved during Chritmas break

Check allowed to plan the intervention and minimize its impact• There is a plan to remove it during the Christmas break as nQPS

is taking care of this. The information on the total voltage across the BB being a useful information, this will be kept but only in monitoring and not in the interlock

Global BB detector : mains


• Request functions have been smoothened– 8 RQTD.B2 tripped ; feedback functions (13/7)– Various RQTF/D; real time trims (15/7 & 5/8)– Various RQS; new ramping-up parameters (23/7)– Various RQTF/D due to tune feedback re-enabled by mistake (1/8)– RCBV16.R4B2 (3/8)– RSF2.A23B2 and RSF.A67B2; squeeze functions(12/8)– RCBXH1.R2: Trips due to mismatch (30 mA) of the current before the start of

squeeze

• Partial watch dog is implemented (OK for preloaded functions)• Limits increased:

– RQX.L/R2 & 8 : During setting for the 10A/s ramp, problem encountered with these 4 inner triplet circuits (slightly exceeded the initial commissioning value of 6A/s - needing something like 6.05 A/s). There is no limitation for a slight increase from the converter point of view so the limit was increased to 6.2A/s. Values have been changed in LSA and

synchronised into the FGC. Approved by MP3

Trips due to too fast request


• Status regularly checked to plan for a reset, minimizing operation disturbance

• No perturbation in the last 6 weeks• Autonomy > 10 days

Undulators

Triplets circuits• Many trips due to PC when ramping down• Triggers MP3 verifications of the heaters

firing• EPC issue under investigation to assess if

solution is possible


• B18R5 old QPS; Fip com lost; access required; incorreclty assembled connector (8/7)

• Segments 8R4 & 15 R4 of the mains: WorldFip experts accessed tunnel (10/7)

• RB.A12 : No communication with current leads; Access to tunnel to restart main BB detector (12/7) ; will not be used any more next year

• S56 global BB detector (26/7): reboot from tunnel ; will not be used any more next year

• DQAMGS.B11L5 : needed MPE piquet access in tunnel (10/8)

Communication problems with access


• Communication lost on DQAMC.21L7 S67: cured by resetting the gateway(8/7)

• RCBV10.R1B1 : PC communication (8/7)• RQT13.L6B1, RQT13.L6B2, RCO.A56B1 and RCO.A56B2 : (Fast) Power cycled

cured it (8/7 & 15/7 & 17/7)• RQTF/D in 78 : Power cycle cured it (10/7)• RCO/D.A56B1 : communication lost on crate; Power cycle cured it (12/7)• RQRF.A67B2 LD1 com pb (13/7); Power cycle cured it • Network in CCC; lost QPS_OK of 6kA&600A in34 not visible; masked as at

injection level ; came back < 15 minutes later (15/7)• S67 communication lost with gateway; cured by rebooting gateway (15/7) • RQD/F.A12 : Lost communication with MQ.16L2 (DQAMC.16L2) while

ramping up the current. By miracle, communication came back after 25 minutes (23/7)

• RQT13.L6B1&2, RCO.A56B1&2: Loss of com. with the crate ; cured by power cycle (4/8)

• S67 mains lost due to communication problem(8/8)• Loss of QPS signals in S12 ; server had to be rebooted (12/8)• Loss of Loss of communication with main BB detector of RQF/D.A67 cured

by sending logging (EIC) (15/8)

Communication problems without access


• 10/7 : S67&78 lost: origin RQ10(9).R6 & L8– No PM of RQ9, as on same acquisition system as RQ10, will be solved

after Christmas break– Long cables and specific settings (see further)

• 13/7 : RQ5.L2– Trip du to noise on U_RES ; QH discharge is OK ; During a ramp ;

Noisy board that was replaced

External? electrical perturbation (1/2)


• 23/7 : RQ9.R2– Trip ; heaters fired ; U_Res over threshold but origin of

perturbation not identified

External? electrical perturbation (2/2)

JPhT – MP3 18th August 2010

• 27/7 : RQ6.R1– Trip ; heaters fired ; U_Res over threshold but origin of

perturbation not identified ; – typical analysis with heater discharge and recharge checked


MP3 is 1st line for analysis

Some examples:• 16/7 : RQT12.R5B2 : Trip due to fan problem on PC• 18/7 : RQ6.R5 : PC trip (EPC issue)• 25/7 : RCS.A45B1 : Tripped 3 times; EPC changed a power module• 1/8 : RB.A56 : beam dumped by RB trip due to water fault issue in the PC.

EPC intervened• 7/8 : RQ8.R6 : Trip first thought to be due to noisy QPS board but finally was

due to a PC component that has been changed• 8/8 : RCO/D.A56B1 : Trip (<1A) during the night ; indication that possible

earth current fault ; EPC intervened• 16/8 : RB.A56 : Cannot be rearmed due to water fault so intervention by CV• 17/8 : RCBH30.L2B2 : Circuit tripped due to a VIN DC UNDER VOLTAGE

voltage• 17/8 : RQT13.L2B2 : Tripped while precycling due to the PC with an internal

voltage reference fault


• RSS.A34B1Self opening of the switch (#3) ; first event of this kind in the LHCThe faulty switch Z of the EE system has been replaced on 19/07Circuit not used in between

• RB.A81Trip du to nQPS board B21R8 (was changed- Broken resistor on the board)

8th of July


• RCBCV7.L2B2 and RCBCH7.L2B1 – Test following intervention on the PC side– Resistance measured during PNO.A1 is slightly larger than the tolerance. Already

observed in the hardware commissioning but very stable.– To be followed by MP3

22nd of July


Trip of RB.A78• Sector trip when switched ON at 08h18m18s. To get access in L8, the EiC has

put the RB on OFF without first requesting STANDBY. The result was that the current in the RB was SLOWLY decaying towards 0 (see timber plot). Access was then given with current in the RB (about 500 A) although the EiC thought there was 0 A. After access was finished, the EiC put the RB on standby. This created a huge voltage in the circuit because there was still a current of about 200 A in the circuit at the standby command. Then the main busbar detector tripped because U_RES went above the threshold and the switches were opened.

• Sequence was not followed• This is managed in another body

Not really MP3 but safety issue : 23rd of July : RB.A78

Access with 500 A in the RBSector trip with 200 A


• Beam dump triggered by QPS

• The trip was caused by the spontaneous openig of a circuit breaker powering the QPS detector in 18L3. Intervention was required

• 2nd occurrence this year

Monday 3th August


• RCBXH1.R2 U_res signal with a significant offset tripped during pre-cycle

• Required an access at 02:00 to be reset from the tunnel ; will be investigated at next TS

Tuesday 27 July


• Special configuration applied to these magnets to measure higher range voltage – Poor resolution on the low voltage

range– High sensitivity to noise of the

triggering signal

• Restoring the nominal settings will imply removing two resistors for each card. –MP3 recommended to test in the lab

and implement 1-2 examples during the next technical stop.

– Activity on-going

Q10-Q9 sensitivity (to perturbations)


• Measurement of bus-bar resistance for the 6kA circuits– Reference values for time evolution and

nQPS upgrade• Already 14 magnets measured in the

dispersion suppressor– All values around 5 nOhms for 4 to 5

splices. –Measurements require special

equipment displaced and connected during the technical stops.

IPQs bus-bar resistance measurements


• Hardware procedures and test plan for next shutdown/break under revision

• New tools on preparation– New interlocks that may lock the circuits after

automatic analysis of the data – 1st version to be released for September 2010

• BLM threshold adjustment tests– A dedicated MP3 meeting will be organized for

this subject. Location was already identified but needs to be confirmed.

Parallel activities


• IPQs: New system for Q10R2 (Maybe Q9 & 8 too)– Will require a full re-commissioning (Thursday

night?)

• Change of switch on QHPS :– Beginning of a campaign ; will improve

availability– Recharging will be monitored / no discharge test

planned– Then campaign to be completed during Christmas

break

• Snubber capacitors (S12 or 78)– Will be installed but not connected– Connection during the Christmas break– Intervention on critical safety devices– Dedicated tests have to be foreseen, with

powering

MPE interventions at next Tech Stop


• Two persons on call for LHC operation1. 7:00 to 19:00 on weekdays2. 19:00 to 7:00 on weekdays and 24/24 on

weekends

• Attendance to the 8:30 and 9:00 meetings

• Close follow-up of the LHC operation• Shifts will be organized after technical

stops• Same commitment and workload for

the MP3 members but resource optimization

New supervision mode from MP3 (Since 28th of July 2010)

x162266


Average: 2.1/day, stable (2.0 over the last 30 days)Technical Stops – Recovery from TSSwitch to supervision mode

MP3 Actions

mp3 report : w27 to w33 (22 slides, 10’ ?) j.ph tock for the mp3 team: te-mpe : n. catalan, r....

Documents