m&o review, fnal, july 22, 2005 : cms si tracker project: j. incandela 1 us cms silicon tracker...

M&O Review, FNAL, July 22, 2005 : CMS Si Tracker Project: J. Incandela 1

US CMS Silicon TrackerUS CMS Silicon Tracker

I&C and M&O planning

Joe Incandela

University of California Santa Barbara

US CMS Silicon Tracker Project Manager

July 22, 2005Fermilab


OverviewOverview

• There have many reviews:• US CMS PMG Review of M&O: Fermilab, April 11-12, 2001• US M&O Evaluation Group Review: Fermilab, April 8-10,

2003• US LHC M&O Evaluation Group Meeting: Brookhaven,

February 5-7, 2004 • US LHC M&O Evaluation Group Meeting: Fermilab, January

27-29, 2005

• Will show excerpts from a few• Main message is that Sitrk planning has not changed

substantially over the years

• Will show overview of current M&O project file• Some I&C is included in the project file

• Discuss current integration planning and a new initiative for remote effort


Tracker Outer Barrel SiliconTracker Outer Barrel Silicon

• Joe Incandela, University of California Santa Barbara

• US CMS Silicon Tracker Project Manager

• April 9, 2003

• Outline:

• Basic considerations and assumptions

• Project end and transition to M&O

• Maintenance and Operation Plan


CMS TrackerCMS Tracker

5.4 m

End Caps (TEC 1&2)

2,4

m

Inner Barrel & Disks

(TIB & TID)

PixelsOuter Barrel (TOB)

~100 m2

volume 24.4 m3

running temperature – 10 0C


Basic ConsiderationsBasic Considerations

•Basis of estimates • Experience considered:

• CDF Run 1b and Run 2a

• D0 Run 2a

• Scaling factor for resources ~ 1 despite large increase in channel and module counts

•Considerations• TOB = CERN+US CMS but…

• CERN has large general infrastructure responsibility

• US CMS group is an obvious choice to play a lead role in M&O team.

• Operational expertise lies with those having the most testing experience during construction

• Very useful Tevatron experience

•Resource driving assumptions:• US will assume a major portion of

responsibility for TOB M&O• Need continuous coverage of 6

main M&O areas• Detector should be ready for

physics as early as possible

CDF Run2a CMS TOBLayers 8 6Length 1.9 m 2.4 m

Inner Radius 1.35 cm 60 cmOuter Radius 28 cm 106 cm

Channels 722432 3096576Modules 704 5208

(US CMS will now also produce and test >3000 TEC modules…)


CDF/DO ExperienceCDF/DO Experience

• CDF and D0 installation & operation• A very big job which takes many months and many people • Current (~steady state) operations requires on the order of 7 high level

experts with backup of ~10 additional operators and monitors• We assume that some monitoring can be done remotely for TOB

CDF Silicon installation and commissioning group

CDF Silicon Cables exiting the bore of the outer tracker

“ Our current commissioning effort is taking so long partially because of inadequate resources (both physicist and non-physicist). So, I would advise that for commissioning especially, you do not underestimate the need for … resources - ask for enough dedicated engineers to complement as many physicists as you can get. ” -- co-leader of CDF silicon operations group in mid 2002.


Project end & Transition to M&OProject end & Transition to M&O

• US must anticipate the need to cover much of the effort for the installation and commissioning (I&C) and probably the lion’s share of the maintenance and operation (M&O) of the TOB.

• Some I&C resources thru FY05 (now FY06) are included in the construction project.

• Installation/testing of rods in wheels

• (Originally) scheduled for Jan 04 – April 05

• (Now scheduled for Sep. 05 – May 06)

• 2-4 FTE Sr. Physicists, 7 Post-docs, students.

• Testing modules on rods at CERN before installation

• Limited system tests.

• 1 FTE Engineer (base).

• Rod & Module repairs, cooling system, mechanics.

• Laboratory space and equipment for testing and repairs.

• In late 2006, large M&O support team must be in place.• We estimate ~ 12 FTE physicists at CERN together with continued

engineering and technical support & associated M&S and operating costs for maintaining a small laboratory space for testing and repairs.


M&O for Tracker Outer Barrel: 6 Main M&O areas

M&O for Tracker Outer Barrel: 6 Main M&O areas

1. Ops. 2-3 phys. 1/2 eng. 1/2 tech.• 24/7 on-call to avoid dead time.

• Leadership: organize US effort, executive decisions, interpret diagnostic results.

• Interface with CMS admin.

2. DAQ 1.5 phys. 1/2 eng. 1/2 tech.• Front end and higher level modules.

• FEDs and EB managers.

• Diagnostics, testing, (repairs, re-testing ?), swapping DAQ modules.

3. Radiation protection and cooling ~1.5 phys. 1/4+1/4 engineer, 1 tech.

• Monitoring, repair.

• System issues: cooling tech present or on call at all times to deal with problems as they arise.

4. Crates and power supplies. 1.5 phys, 1/2 eng. 1/2 tech.

• Testing supplies prior to installation, retest after installation.

• Safety interlocks. A serious ongoing maintenance issue.

5. Monitoring and calibration databases 4 phys. 1/2 tech. at CERN

• Additional remote effort (see below). • Need to catch problems in data and

in calibration techniques.• Very significant effort just to

maintain a good channel list.

6. Alignment 1.5 phys.• Laser alignment runs, hardware

maintenance.• Offline alignment and monitoring of

alignment constants.


Other areasOther areas

• Not included in our resource projections: Module Failures analysis and development of failure prevention methods:

• CDF/DZero have required considerable resources to

• Understand why some silicon modules have failed

• Example: Resonant wirebond breakage

• Develop failure prevention strategies

• For resonant wirebond breakage - developed completely new operating procedures to prevent repetitive readout sequences that led to exactly periodic current swings.

• “If these resources are forced to come from operations, as was the case at CDF, things will be strained and commissioning and/or operational efficiency will be affected.” – CDF silicon operation group co-leader

• Some of this can be done remotely (see discussion on FNAL remote monitoring/diagnostics station below)


US CMS Silicon TrackerUS CMS Silicon Tracker

US LHC DetectorMaintenance and Operations

Evaluation Group Meeting

Joe Incandela

University of California Santa Barbara

US CMS Silicon Tracker Project ManagerJanuary 28, 2005

Outline:Project overview and Status

Schedule and Project Completion

Installation and Commissioning, Maintenance and Operations

R&D for upgrades


Project end & Transition to M&OProject end & Transition to M&O

• We assume we will be needed to cover at least half of the effort of installation and commissioning (I&C) of the TOB.

• Installation/testing of rods in wheels (05-06).

• 2 FTE Sr. Physicists, 7 Post-docs & students.

• Testing modules on rods before installation.

• Limited system tests.

• 1 FTE Engineer (base).

• Rod & Module repairs, cooling system, mechanics.

• Laboratory space and equipment for testing and repairs.

• For maintenance and operation (M&O) of the TOB we need a team to provide support in 6 areas of operation.

• By late FY06 early FY07, US M&O team must be in place.

• We estimate 13 physicists at CERN together with continued engineering and technical support & associated M&S and operating costs for maintaining a small laboratory space for testing.

Only substantial changes are in the schedule dates…


Project I&CProject I&C

Project I&C now overlaps with M&O in FY06 but this is probably a good thing because more resources are going to be needed to recover schedule.


M&O Cost EstimateM&O Cost Estimate

Silicon Tracker M&O Cost Estimate

FY02-FY09 Total $2,861,512 AY$

0

200

400

600

800

1,000

FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09

AY

$k

Labor

M&S

Silicon Tracker M&O Commitment Profile

FY02-FY09 Total $2,861,512 AY$

0

200

400

600

800

1,000


AY

$k


US CMS SITRK M&O ResourcesUS CMS SITRK M&O Resources

•Manpower on Base• Post-docs, students, and one engineer..

• Post-docs all redirected from other programs or the production effort.

•Manpower on Project• All technicians and additional engineering.

U.S. CMS Silicon Tracker M&O Resources

0.0

5.0

10.0

15.0

20.0


FT

E's

Technicians

Engineers

Physicists


M&O Thru 2009 (as of Jan. 2005)M&O Thru 2009 (as of Jan. 2005)

1.1M$ COLA (using Jan. formula with 1.1 FCSH/USD)

1.2M$ Engineering + technical + ALL travel

350k$ R&D


Experience mattersExperience matters

First 2 years of operation (FY07-08) will be critical years• Difficult problems will have to be solved and will require the most

experienced people• Establish all steady state operations of the tracker

• Diagnosis and classification of many failure modes and problems• Detailed operating procedures & training guidelines

• Refine all monitoring, calibration and alignment methods• Current uncertainties make it difficult to argue strongly that we

absolutely need all of these resources. On the other hand, Tevatron experience tells us to prepare as strong a team as possible


Upgrade R&D IssuesUpgrade R&D Issues

• CMS silicon has limited lifetime.• 10 years of operation (we hope).

• Further, possibly higher luminosity running will require more rad-hard silicon than currently exists.

• R&D for current silicon took many years.

• We need to be involved in future efforts at the level of prototyping and testing silicon or silicon-alternative designs.

• 65k$ per year M&S

• FY04 spent 90k$ for NRE and new sensor production

• Demina (Rochester) Bortoletto (Purdue)

• FY06+

• Some ideas now being developed.

• Will visit HPK in October.


Summary & Conclusions (Jan. 2005)Summary & Conclusions (Jan. 2005)

• US Groups now responsible for building and testing roughly 65% of the CMS tracker modules

• We have lost a year due to component problems.• Expanded capacity to recover schedule

• Components now ok.

• Based on CDF/D0 experience, significant resources needed for I&C and M&O.

• We estimate 13 scientists plus engineering and technical support to allow US to have significant involvement in all aspects of operations

Nota Bene: The US tracker group now has played a more important role in the construction project than originally anticipated or panned (e.g. problems found and solved with sensors, hybrids, rods… and a major increase in fraction of tracker modules built in US). Nevertheless, we have not made any significant changes to our M&O resource plans… We believe we have a good plan!


TOB Project as of July 2005TOB Project as of July 2005

• US TOB rod integration effort stalled • Serious problem found in US at both FNAL and UCsB• A very robust solution has been found (Option 7)

• New interconnect cards are required (see next slides)• We pay 80k$ in order to expedite them (saves >1 month)

• Total schedule hit is going to be around 5 months • Module production to continue

• Need to purchase 400 sensors from HPK (250$) ASAP!!!• Need additional module storage costing ~15k$

• We are currently storing almost 2000 modules!

• TOB integration to proceed • Use rods modified (Option 6) and fully tested in the US• September installation of one full cooling segment of TOB6- at

CERN with substantial US involvement (we hope)• New Clean room (Tracker Integration Center – TIC) to be ready for

support tube in October, fully operational by December (my estimates).


Rods OverviewRods Overview

• Basic components procured

• Assembly of frames is DONE (modulo ICC)

• Integration of modules on rods is understood• Rate of 8/d in the US is not be a problem …≳

• Testing was problematic and rod production was halted• I2C communication errors seen on 30-50% of rods

• Recent and very substantial effort to understand and remedy

• Many thanks to : A. Marchioro, Wim Beaumont, M. Johnson, G. Maggazu, Slawek Tkaczyk…

• Simple solution (number 6) an improvement but not quite..

• A better solution (number 7) adopted and requires new ICC

• Secondary issues of header errors and failures in LT tests are also under study


Main causeMain cause

SCL on FE-Hybrid

SDAUnequal fall-time of SCL and SDA signals

Picture shows sequence of SCL/SDA signals with SDA pulled-down in AOH and measured on FEH


Option 6Option 6

AOH

FE-Hybrid

SCL

SDA

82

82

PSUControl

PSU FE

LLD

APV

Jumper creates a common time domain but some recent information from US indicates may still be marginal


Option 7Option 7

AOH

FE-Hybrid

SCL

SDA

100 pF

Parasitic on AOH CA

PSUControl

PSU FE

ICC

CRT245(Line driver)

22

pF

pF

330

LLD

DCU/APV


Effect of “damping” resistorEffect of “damping” resistor

SDA with“damping”

SDA without“damping”

SCL not show on this slide


Current (limited) productionCurrent (limited) production

• While I2C solution 7 is being prepared we are assembling and testing a limited number of rods with "jumpers" (I2C Solution 6)

• Test I2C solution 6

• To be shipped to CERN to populate 1st cooling segment, and possibly the cosmic rack

• Gain experience with (somewhat) large scale rod production

• Somewhat longer testing cycle to double-triple-quadruple-check I2C behavior

• Gain experience with testing protocol• Document/investigate non-I2C failures

• Refine cut requirements


Part Count as of July 13, 2005Part Count as of July 13, 2005

UCSBUCSB FNALFNAL

Rods jumpered with Rods jumpered with “solution 6”“solution 6”

1414 3434

Rods populated with Rods populated with modulesmodules

1414 3434

Rods tested at SRTRods tested at SRT 1010 3333

Rods tested with MRTRods tested with MRT 66 1818


SRT results – out of 43 tested rodsSRT results – out of 43 tested rods

• 5 with unconnected thermistors (as read out by DCU, i.e., these are not the "hardwired" thermistors)• 4 at FNAL, 1 at UCSB

• 1 fails ring redundancy test• FNAL

• 1 Laser failure• FNAL

• 1 with strange problem, open ring? • UCSB, under investigation

• All 4 SS4 rods (UCSB) have unusually high noise

Discussed in TOB m

eeting at July

Tracker Week: agreed upon

actions to be taken


But also…But also…

• Still low rate of I2C errors 230 that we thought would have been cured by jumper (Solution 6)

• Other not-yet-understood I2C errors

• A few other annoyances like header errors

HW or SW problems?

Experts are actively investigating


Rod PlansRod Plans

• After this "limited production" we will stop• Take stock of situation, firm up procedures

• Attack other (non I2C) problems

• Hold a mini-workshop to get the UCSB and FNAL-based people together in August• We will likely consider another fairly substantial

production exercise• Allows one to hunt more rare problems

• Module storage…


Thermal screen operational

Wheel ready for insertion in the tracker support tube and…

The tube with thermal screen are already in the assembly clean room.

Wheel on assembly chariot

Meanwhile …Wheel is ready!Meanwhile …Wheel is ready!


Preparations for TOB integration - where we are as of July 14, 2005

Preparations for TOB integration - where we are as of July 14, 2005

He leak test stand OKRod precabling stand OKCu power cables two weeksCable “bridge” one weekVacuum system for soldering to be commissionedService supports in productionCooling manifolds in productionReadout hardware FEDs/FEC available - firmware 1 week (?)Power supplies availableTest software to be commissioned on C-Rack Plumbing in 186 (cooling and He) Basically done

DB tools defined - only ribbon registration implemented

------ later ------

Alu cables need to update layout and start production (lengths OK)Cable test box in production in Lyon (mid-August) - need updated pinoutDOHMs finalizing tail lengths - start production


US involvement US involvement

• New key personnel• Jeff Spalding – will become deputy manager responsible for

overseeing initial US integration activities at CERN• Very experienced manager

• Assist the talented but a bit inexperienced CERN team

• Improve US-CERN communications

• Slawek Tkaczyk- will initially lead the US testing effort during integration at CERN.

• Also very experienced

• Will oversee experienced US physicists at CERN during integration

• September ~1 month integration Exercise• A strong US team at CERN under Spalding and Tkaczyk

• Two experienced US rod testing physicists

• Two experienced US Silicon technicians/engineers


Goals of September ExerciseGoals of September Exercise

• Establish detailed procedures• Establish rod acceptance tests (upon receipt from US) and rod

tests prior to installation on wheel• Full installation and commissioning of a rod on wheel

• Serial installation/commissioning of up to ~20 rods

• Removal sequence• Unsolder all installed rods and remove

• Batch installation • Methods/procedures for batch mode installation of rods

• Slot in place up to ~10 rods and test connectivity (front ends not powered)

• Solder cooling pipes all rods and test cooling ring• Cable and test all rods and test DAQ ring(s)

• Use this information to develop resource loaded schedule for full TOB integration

• We expect to be revising schedule more than once, as other problems develop in commissioning. Information from all integrated experience will be used together with resources/time available to optimize safe assembly…


Integration in 2006Integration in 2006

• After the September exercises… Plan to have US team start to appear in late ‘05 or early ’06

• Depends upon restart of rod production – If smooth, we can transfer some of the experts to CERN.

• Current management planning is directed toward evolution from sub-system integration teams to full tracker integration team.

• We are already working with the integration manager (Nicola Bacchetta) to develop TOB integration planning documents that will be reviewed in Autumn.

• Idea is to develop a uniform set of protocols followed by all subsystems.

• By the time we have completed the subsystems’ integration we would like the transition to commissioning of the full tracker to be as smooth as possible.

• This is also facilitated by the new tracker integration center.


Current Status of TOBCurrent Status of TOB

• Module production lines have completed 25% of TOB• >99% yield and >99.95% good channels

• Production of 40-50 modules/day can be achieved, provided that we have adequate parts flow and stable testing

• Rod frames production in Helsinki is complete

• Rod integration and testing capacity ≳8 rods/day

• I2C communication problem is understood• Rod integration and testing has restarted in the US with solution 6

• Solution 7 likely definitive – will learn more in coming weeks.

• TOB Wheel is ready and other preparations are in train

• Integration exercises will start soon• Very capable team at CERN

• US may send a team in September (headed by Jeff Spalding)


Tracker DAQ Test Facility at FNALTracker DAQ Test Facility at FNAL

Being organized by M.Johnson, J. Spalding and S. Tkaczyk.(S. Tkaczyk provided these slides – modified somewhat by J. I.)

• Motivation: Allows US CMS tracker community to: • Participate in online code development for monitoring data

quality and detector performance• Experience is crucial for the tracker to be usable early in running!

• Educate new people in using the CMS DAQ HW and SW tools• Much more effective to learn this before taking shifts at CERN!

• Study long term performance of strips and pixels• Could determine our upgrade strategy

• Troubleshoot tracker problems emerging during regular operation of the CMS tracker

• Pathological modules/hybrids found and fully categorized in US production efforts will be used as a “library of pathology” that could be useful for development of work-rounds for problems arising during data-taking.


TrackerDAQ@FNAL: Configuration

TrackerDAQ@FNAL: Configuration

• HW - a replica of the CMS Tracker DAQ slice to read out a set of RODs (and pixel structures?):

• Components: FEDs, FEC, TTC, PS, cables, fibers, DCS, interlocks.

• SW – Run Control, XDAQ shell with tracker-specific extensions for data acquisition, DCS, PS control, calibrations, online data analysis


TrackerDAQ@FNAL: Remote Data Quality Monitoring

TrackerDAQ@FNAL: Remote Data Quality Monitoring

• Allow CMS collaborators to participate in the operation of the tracker in the US.

• Develop expert level knowledge among US CMS tracker community by means of:

• Operation of the elements of the tracker (strips, pixels) at FNAL CMS DAQ test facility

• Development of online software in collaboration with other tracker teams

• Participation in daily operation and monitoring tasks at CERN through regular shifts at FNAL


TrackerDAQ@FNAL: ImplementationTrackerDAQ@FNAL: Implementation

• Dedicated consoles and workstations (costs included in the Remote Operation Center)

• Scheduling/oversight of regular tracker shifts at FNAL – 1/4 FTE

• DAQ test stand at SiDet• Will find some eqpt. at FNAL and get some on loan from

CMS. (To purchase all is > $100k).• HW support – ¼ FTE• PC, Software Support – ¼ FTE Comp. Prof.

• Total Physicists: 6 at FNAL and 3 at CERN• Would include post-docs/students from university groups

• Other items:• Dedicated audio/video equipment at CERN and FNAL• Travel Support


Preliminary (crude) estimatesPreliminary (crude) estimates

• FY06 • $20-50K -DAQ Hardware

• $10K -Audio and Video Equipment

• $25K -Analysis Workstations and Data Storage

• $25K-50K - travel support – (5-10 people)

• 1/2 FTE support – technician + comp. professional

• FY07-FY09• $50K/year – travel support

• $10K-$15K/year - maintenance


ConclusionsConclusions

• TOB module production is on schedule• Rod production halted to fix major problem• Integration planning is underway

• Schedule must be revisited (this will not be the last time…)• US beginning to structure a strong team at CERN

• Jeff Spalding the US tracker project deputy in residence• Slawek Tkaczyk will oversee US testing team• Critical integration exercise at CERN in September

• US I&C and M&O planning is stable and adequate• Key will be to see if we actually get the support we need to

fulfill our plan which has been stable for many years.

• TrackerDAQ@FNAL: Early stages of planning for an FNAL-based training/monitoring/online-development facility for the CMS tracker

• An excellent and necessary US tool!


Additional HPK Silicon for TOBAdditional HPK Silicon for TOB

Modules Needed Needed Ordered

3528 OB2 7056 7058

1680 OB1 3360 3180

Yields Central Optimistic Pessimistic

Assembly 99.4% 100.0% 98.8%

ARCS Tests 99.2% 99.6% 98.8%

LT Tests 99.9% 99.9% 99.9%

Rods 100.0% 100.0% 100.0%

Total 98.5% 99.5% 97.5%


Qty for various scenariosQty for various scenariosIf 1% loss on

rods

If 0.5% loss on

rods

Needed (No spares) OB2 OB1 OB2 OB1 OB2 OB1

Central 7167 3413 7239 3447 7203 3430

Optimistic 7095 3379 7167 3413 7131 3396

Pessimistic 7239 3447 7312 3482 7276 3465

Rod Loss 0% 0% 1% 1% 0.5% 0.5%

Remaining Order OB2 OB1 OB2 OB1 OB2 OB1

Central 109 233 181 267 145 250

Optimistic 37 199 109 233 73 216

Pessimistic 181 267 254 302 218 285


ProposalProposal

•Cost per sensor is 428 Euros• 171,200 euro @$1.22/Euro

• 207k$

• 20% for currency fluctuations & contingency etc…

250k$ Proposal OB2 OB1

150 250

m&o review, fnal, july 22, 2005 : cms si tracker project: j. incandela 1 us cms silicon tracker...

Documents