july 7, 2008slac annual program reviewpage 1 current atlas program charles young

July 7, 2008 SLAC Annual Program Review

Current ATLAS Program

Charles Young


Outline

* History and context. * Initial involvements.

– High Level Trigger (HLT). – Pixel.

* Additional activities. – Western Tier 2 (WT2). – Simulation. – 3D Sensors. – Remote Monitoring. – Production support.

* Personnel, roles and financial data. * Summary


History of SLAC’s ATLAS Involvement

* Initial investigations in 2005. – Internal discussions at SLAC in spring.

– Informal meetings with US ATLAS physicists.

– Met with ATLAS and US ATLAS management in June. • Identified high level trigger and pixel detector as two areas where ATLAS

needs help and where SLAC has corresponding interests and strengths.

– Started work on pixel module testing at LBNL.

* Formal steps to join ATLAS. – Submitted application in February 2006.

– ATLAS Collaboration Board vote in July 2006.

– Separate applications to join ATLAS HLT and ATLAS pixel.

– Competition for 2nd round of US ATLAS Tier 2.

* Further engagements in the last two years. – Aligned with SLAC’s core competencies and long-term interests.


Context of SLAC’s ATLAS Involvement

* Some smaller institutions are members of ATLAS by affiliation with larger groups like SLAC. – University of Iowa.

– California State University at Fresno.

* Informal arrangements to supervise/mentor non-Stanford students resident at SLAC and at CERN.

* Western Tier 2 provides computing resources and a fully supported ATLAS environment for users.

* Support of ATLAS-wide activities, e.g. – First ATLAS Physics Workshop of the Americas in 2007.

– Jet/Energy Scale workshop in 2008.

* Coordination with and support of university groups. – Office space.

– Common interests and joint activities.


Outline






High Level Trigger

* Details in talks by Rainer Bartoldus and Ignacio Aracena. * HLT configuration.

– Delivering total of O(0.1 – 1) TB of data to O(2000) nodes x 8 clients/cores per node not possible with single server.

– DbProxy solution is hierarchical and scalable. – Rainer Bartoldus, Sarah Demers, Amedeo Perazzo, Andy Salnikov, Su Dong.

* HLT algorithms: Jet, Missing ET, Tau and b-jet slices. * Partial event building.

– Calibration or alignment events require information from only part of ATLAS. – Can afford much greater rate. – Code implemented and validated, and now working on monitoring. – Ignacio Aracena, Anson Hook.

* Online beam spot measurement. – Critical for certain trigger algorithms such as b-tagging. – Useful to LHC. – Rainer Bartoldus, David Miller, Su Dong.

* Operational responsibilities.


Why the Pixel Detector?

* Historical and current interest in silicon sensors. – SLD. – SiD. – LSST. – ATLAS B-Layer Replacement. – ATLAS Tracker Upgrade.

* Data flow and read-out. * Interest in physics topics that rely on pixel detector.

– b-tagging.


Pixel Detector

* Pixel package lowered into the Pit on 6/25/07. – Two weeks after review last year. – Installed on 6/29/07.

* Connections started in February 2008. – SCT services and cooling must be installed first. – Connection finished on time in April 2008.

* Sign-off, i.e. certification that Inner Detector (ID) volume can be sealed. – Suspended on 5/1/08 after 6 days due to cooling plant failure.

• Three of out 6 compressors failed “simultaneously”. – Approximately 2/3 of sign-off activities completed. – ID volume sealed.

* Plan and schedule. – Cooling plant back in operation on 7/21/08. – First used for beam pipe bake-out (~10 days).

• Cooling is an absolute requirement! – Then resume pixel sign-off and start commissioning early August.

• LHC will start injections at about the same time. – Pixel HV interlocked to be off during injection. – Incompatible with sign-off and commissioning.

• One month time estimate will stretch out depending actual time available.


Pixel Activities

* Cooling system and monitoring. * Calibration. * Digital Signal Processor (DSP). * High voltage (HV) distribution. * Track based alignment.


Cooling System and Monitoring

* Evaporative cooling system mandatory for detector operations (and for beam pipe bake-out) but has presented serious problems. – Heater failures last year. – Some instability during sign-off in May 2008. – Cooling plant failure.

* Detailed study of operational parameters and temperature measurements from sign-off to understand the system better and to find a more stable operation point. – Bart Butler, Joe Dipirro (Fresno).

* Implement additional monitoring and alarms. – Lawrence Carlson (Fresno).

* Long term trends of operational and environmental conditions. – Prafulla Behera (Iowa), Gerald Rude (Fresno), Steve Wilburn (Fresno),

CY. * Use event display program to display non-event data in

geographically accurate format. – Jim Black, Tim Nelson, Ben Zastovnick (Fresno).


Calibration

* Enormous undertaking by many people over a very long time!* Detector noise studies.

– Essential to understand noise and mask out bad channels (out of 80 x 106 channels) to avoid overwhelming DAQ bandwidth or compromising track and vertex performance.

* Digital scan analysis. – Basic communications with pixel modules.

* Back of Crate card (BOC) tuning algorithm. – Optimize operational parameters for robust optical communications

between VME crate and pixel detector.

* Analysis Framework. – Track calibration results over time. – Compare calibration with reference or another calibration. – Combine multiple calibrations runs into one coherent set of parameters

for the entire pixel detector.

* Bart Butler, Claus Horn, Ariel Schwartzman.


Digital Signal Processor

* Pixel read-out system has 132 Read-out Drivers (ROD) in 9 ROD crates. * Each ROD has 5 DSP (one Master and four Slaves). * DSP’s functions include:

– Data taking related code is relatively stable. – Calibration tasks, e.g. threshold scan, noisy pixel scan and so on.

• Focus of much of activity and significant progress in the last two years. • Work still going on at a furious pace.

* Manpower is slim and declining. – Jed Biesiada (LBNL), Alex Schreiner (Iowa) and Matthias Wittgen.

* SLAC’s roles. – Improve the software environment to enhance productivity. – Maintain test stands and assure their coherence with CERN setup. – Effort hampered by recent developments at CERN introducing incompatibilities with test

stands. * Long-term sustainable manpower needed for pixel operations.

– Matthias Wittgen will increase his level of involvement. – Martin Kocian is joining. – New RA may get involved also.

* Data flow is an area SLAC has long participated in, and is an area we intend stay active and take a leading role for ATLAS and other projects.


Outline






Tier 2 Computing Center

* Three sites selected in 2005 by US ATLAS. * Second call for proposals in 2006.

– SLAC was one of two sites selected. * Structure of Western Tier 2 (WT2).

– Supported by 8 institutions. • Lawrence Berkeley National Lab. • SLAC. • University of Arizona. • University of California, Irvine. • University of California, Santa Cruz. • University of Oregon. • University of Washington. • University of Wisconsin, Madison.

– Advisory group with representative from each institution. • Current Chair is Gordon Watts (Washington).

* Functions. – Production use via centralized control. – User activities, particularly analysis.

* Leverages and continues SLAC’s capabilities and expertise in computing, especially in the area of data-intensive analysis applications.


Simulation

* Formation of “fast shower” working group by ATLAS ~2 years ago was due largely to our initiative.

– Calorimeter was the dominant consumer of CPU time. – Kick-off workshop at SLAC in late 2006. – SLAC participation: one rotation student, Bart Butler, Zach Marshall (Columbia), Dennis

Wright, CY. * ATLAS created Simulation Optimization Group to address other optimization issues.

– Code speed in other places, but also physics quality. – Work mostly delegated to subsystem experts with assistance and advice from SLAC Geant4

experts Makoto Asai and Dennis Wright. – Zach Marshall and CY are members.

* Recent engagement in muon system. – Geometry issues.

• Remove clashes of sensitive detectors as well as dead material. . • Implement cut-out chambers with complicated geometry.

– Code optimization. – Structural or fundamental improvements.

• “Stepper dispatcher” and field integration and AMDB database. • Transition radiation process and hadronic physics validation and improvements.

– Makoto Asai and Dennis Wright. • Core developers in Geant4.

– Mentoring of less experienced people. * Leveraging of SLAC’s team of Geant4 core developers and experience from BaBar.


Overlay

* Average of >20 minimum bias events per beam crossing at design luminosity. – Governed by bunch luminosity rather than total beam luminosity.

* Other background sources. – Beam halo and beam gas. – Cavern background. – Adjacent bunches (due to electronics integration time).

* Crucial to include all such effects in simulation, but difficult to achieve from pure simulation. – Cross sections not known a priori. – Dependency on parameters such as beam condition, (dynamic) beam

pipe vacuum, etc. * Most reliably method is to record “zero bias” data, and overlay on

simulated events. * Mike Kelsey and Peter Kim will join Bill Lockman (UCSC) on overlay

in muon system and in validation tools. * Leverages experience from BaBar.


* 3D pixel sensor is candidate for B-layer replacement and/or tracker upgrade. – Radiation hard due to electrode proximity. – Active edge allows for butt joints.

* Beam tests. – Three 3D sensors.

• Two non-irradiated sensors. • One irradiated sensor (1015 p / cm2). • ATLAS front-end chip.

– External tracking with three sets of (x,y) silicon strip detectors. – Just concluded a 3-week run.

• Bart Butler, David Miller, CY. – Analysis to determine efficiency and resolution near the sensor edges. – Next run scheduled for November.

* Leverages our connections to Stanford University (where 3D sensors were first fabricated and continue to be developed), the new silicon lab at SLAC, and close working relationship with LBNL.

3D Sensor


Remote Monitoring

* Routine shift role by remote collaborators. – Reduce travel time and costs. – Take on work load during normal waking hours. – Implementation must be affordable at remote sites.

* Experts cannot all be resident at CERN all the time. – May be on travel and have laptop only.

* Decided to use the ATLAS Control Room (ACR) tools and export the interface and display.

* NX implementation first tested by Eric Feng (Chicago). – Peter Kim tested it at SLAC during a recent weekend combined run.

• Performance is generally acceptable. • Start-up is a little slow. • Scatterplots can be very slow if not properly handled.

– Working with other US groups who want to test. * Plan to install a high-functionality station at SLAC.

– Available for use by not only SLAC physicists but also ATLAS collaborators, whether they are normally resident at SLAC or visiting.


Production Support

* ATLAS production currently runs (0.5 – 1) M jobs per month, and already has ~15M jobs in its DB.

* We estimate it will go up by (at least) 100x. * Production DB is needed to bridge requests and job executions. * Must be reliable and scalable to track all jobs.

– Which ones have finished successfully?

– Which ones need to be re-run?

– Which ones should be aborted?

– What are the worst production problems? • Cannot fix problems that we cannot identify!

* Douglas Smith will help re-structure the ATLAS Production DB. * Takes advantage of the experience from BaBar, and continues our

involvement in data management.


Outline


– High Level Trigger (HLT). – Pixel. – Western Tier 2 (WT2).

* Additional activities. – Simulation. – 3D Sensors. – Remote Monitoring. – Production support.



SLAC ATLAS Personnel

* RA: Ignacio Aracena, Sarah Demers, Claus Horn, Paul Jackson. [4]

* Students: Jim Black, Bart Butler, David Miller, Dan Silverstein, Anson Hook. [5]

* Engineers: David Nelson, Marco Oriunno. [2]* Staff: Makoto Asai, Tim Barklow, Rainer Bartoldus, Mike

Kelsey, Peter Kim, Martin Kocian, Tim Nelson, Amedeo Perazzo, Andy Salnikov, Douglas Smith, Matthias Wittgen, Dennis Wright, CY. [13]

* Faculty: Ariel Schwartzman, Su Dong. [2]


Roles in ATLAS

* B-Layer Replacement Task Force. * DAQ Steering Group Scientific Secretary. * Collaboration Council Chair Advisory Group.

– Also functioned as Spokesperson Nominating Committee.

* DAQ Partial Event Build. * ETmiss Validation coordinator of jet/ETmiss combined performance

group. * HLT DbProxy responsible. * HLT Release Validation Manager. * Pixel Monitoring Coordinator (till 6/08). * Pixel Run Coordinator. * Referees for CSC Notes:

– Soft Muons and b-tagging. – Jet Algorithms. – In-situ Calibration.


Evolution of ATLAS budget: FY07-FY10

FY 2007-2010 Total M$ by Cost TypeATLAS

0

2

4

6

8

10

12

FY07 FY08 FY09 FY10

(

M$

)

Labor M&S Allocation of PPA DPS

FY 2007-2010 Total M$ by ActivityATLAS

0

2

4

6

8

10

12

FY07 FY08 FY09 FY10

(

M$

)

Research Operat'nUpgrade ATLAS Tier 2Allocation of PPA DPS


ATLAS manpower and budget: FY08

FY 2008 FTE by Job CategoryATLAS

Engineer / Computing

Professional, 5.2

Permanent PhD, 5.5

Temporary PhD, 3.4

Graduate Students, 0.8

Administrative / Technician,

0.6

Other, 0.0

Total FTE: 15.6

FY 2008 Total M$ by ActivityATLAS

Upgrade, 0.0

Allocation of PPA DPS, 0.5ATLAS Tier 2,

0.6

Research, 3.9

Total M$ of ATLAS: 4.531


FY 2010 FTE by Job CategoryATLAS

Other, 0.0Administrative / Technician,

0.7

Graduate Students, 0.9

Temporary PhD, 11.0

Permanent PhD, 9.3

Engineer / Computing

Professional, 9.6

Total FTE: 31.5

ATLAS manpower and budget: FY10

FY 2010 Total M$ by ActivityATLAS

Operat'n, 2.7

Upgrade, 2.2

ATLAS Tier 2, 0.6

Allocation of PPA DPS, 0.8

Research, 3.2

Total M$ of ATLAS: 8.7


Summary

* SLAC joined ATLAS 2 years ago. – Started in HLT and pixels. – Involvement has grown since that time.

* Key contributions to a number of areas. – Also see Ariel’s talk on physics.

* Significant roles in the collaboration. * Support of US institutions.

– Western Tier 2. – Affiliation of smaller groups. – Support of people resident at/near SLAC.

* Expect to grow in size of group and in scope of work. * Focus on projects and areas consistent with SLAC’s

experience, expertise and interest.


Additional Material


Sign-Off Status

Q1 Q2 Q3 Q4

Modules (13 bistaves6 bisectors) 410

(14 bistaves6 bisectors)436



Config modules for cooling sign-off

410 436 206 (7bistaves and 4 bistaves missing)

462

Threshold Scan 356 316 -- 368

Results 242 191 -- 271

BOC tuning -- Done on failing -- Done on failing

Threshold Scan -- Not analyzed (by me)

-- All but four modules

87%

60%

40%


Vertical Heater in Pixel


High Voltage Distribution

* HV distribution and module-level voltage control at Patch Panel 4 (HVPP4). – David Miller.

* Significant redesign and rework needed on HVPP4. * Analyzed vulnerability to noise.

– Inject 1.25 V peak-to-peak signal at HVPP4. • 100 Hz to 40M Hz frequency range. .

• For comparison, depletion voltage ~100 V.

– Readout noise increased from ~180 to ~450 electrons equivalent for worst case frequency of 600K Hz.

• Typical operational threshold ~4000 electrons.

– Data collected by Konstantin Toms (New Mexico) and Michael Koehler (Seigen).


With noise injection

Without noise injection


Track Based Alignment

* Base approach is to use high-momentum tracks from p-p collisions during normal data taking.

– Adjust module position and orientation to improve tracking residuals. * Tracks all coming from a point source leads to degenerate solutions.

– Such as overall size scale or momentum bias. – Luminous region is small compared with ATLAS trackers.

* Non-collision tracks can provide additional constraints. – Cosmic ray – low rate and restricted angles. – Beam halo – poor to non-existent illumination of pixel detector. – Beam gas – hope to have very little! – Offset interaction point (IP).

* Proposed to offset interaction point (IP) by one RF bucket, i.e. +/- 37.5 cm. – LHC indicated this is not difficult. – Investigating what improvements are possible.

• Bart Butler, Heather Gray (Columbia), CY.

* Trigger algorithms and DAQ bandwidth issues. – Use partial event build to minimize data size. – Applicable to both normal IP and offset IP. – Ignacio Aracena, Anson Hook.


Offset IP for Alignment


Cutout Muon Chambers

* Special chambers with notches or holes removed to fit in tight spaces.

* Blue: Monitored Drift Tube (MDT).

* Red: Resistive Plate Chamber (RPC).

* Green: cut-away view of detector feet assembly.


Production Request(# events, configurations, etc) Bamboo

(create Panda commands)Panda (Job Execution)

Grid-Z

CPU CPU CPU CPU CPU

Grid SiteGrid Site

Grid Site

Production DB

Tasks &

jobs

Grid-A

Tasks & jobs

Results &

status


LHC Commissioning Plan

Parameter Phase A Phase B Phase C Nominal

k / no. bunches 43-156 936 2808 2808

Bunch spacing (ns) 2021-566 75 25 25

N (1011 protons) 0.4-0.9 0.4-0.9 0.5 1.15

Crossing angle (rad) 0 250 280 280

*/*nom) 2 1 1

* (m, IR1&5) 32 22 16 16

L (cm-2s-1) 6x1030-1032 1032-1033 (1-2)x1033 1034

july 7, 2008slac annual program reviewpage 1 current atlas program charles young

Documents

atlas pixel

atlas hlt

support of atlas

members of atlas

slac annual program

us atlas physicists

us atlas management

atlas physics workshop