worldwide lhc computing grid status and …...worldwide lhc computing grid status and progress...

Worldwide LHC Computing Grid

Status and Progress Report

April 2015 – September 2015

6 October 2015

Table of Contents

Services and Applications WLCG Services ......................................................................................... 3 Grid Deployment Board ............................................................................. 6 Applications Area ....................................................................................... 8

Experiments ALICE ...................................................................................................... 11 ATLAS ..................................................................................................... 12 CMS ......................................................................................................... 14 LHCb ....................................................................................................... 15

WLCG Tier-1 Sites Reliability .............................................................................. 17 Tier-2 Sites Reliability .............................................................................. 25

WLCG – Status and Progress Report 2015H2 (April 2015 – September 2015) 1

WLCG Service Report April 2015 – September 2015

Maria Alandes, Maria Dimou and Andrea Sciabà

October 2015

In April, a WLCG workshop was held in Okinawa. The focus of the discussions was on the current trends and the long-term future of the LHC computing, highlighting the challenges in view of HL-LHC, which to be met will require a substantial increase of resources, of software efficiency and of collaboration among experiments. Clear indications of an increasing importance of opportunistic resources and more mainstream technologies (in particular for data) emerged, together with the need for simpler, more sustainable site models. The results of the WLCG site survey were also presented, showing that the complexity of Grid site management is perceived as a concern, and indicating ways to improve the level of communication within WLCG operations. Addressing the previous point, a WLCG operations web portal was commissioned in July to consolidate all available information relevant for operations and to host articles of interest for the community.

At CERN, a pilot batch service based on HTCondor, using as front-end both ARC-CE and HTCondor CEs, has been set up to be tested by the experiments and is being successfully used. The old VOMS servers and their aliases have been finally removed from the infrastructure, and the same was done for the LFC service at CERN. The FTS3 service worldwide received several updates to address functionality and performance issues. Experiment workflows helped identifying network issues at both Meyrin and Wigner, largely solved now, and significant effort was devoted in understanding and improving the CPU efficiency of the batch resources at CERN.

The start of Run2 at the beginning of June was very successful for WLCG, with a smooth transition to data taking, as demonstrated by the very small number of alarm tickets and the experiments reporting no major issues. ATLAS and CMS run now a significant fraction of their data processing in multicore pilots and the WLCG multicore task force was active in the consolidation of the deployment and in improving the efficiency of the scheduling policies. Multicore accounting has been enabled at almost all WLCG sites and it is available from the preproduction accounting portal.

The WLCG middleware-readiness working group is regularly validating new versions of dCache, DPM, StoRM and ARC using realistic experiment workflows and developed a web application to keep track of the versions installed at the participating sites. The effort available for the ARGUS development and support has improved thanks to the Indigo-DataCloud project, and validation on CentOS7/SL7 platforms has already begun with DPM.

The Network and Transfer Metrics working group commissioned the OSG perfSONAR data store and dashboard, providing programmatic and visual access for all perfSONAR results, with the recent addition of IPv6 information for the sites that provide it. Work has started to automatically discover problematic patterns and a clear procedure was defined to report network incidents.

Two new task forces were formed: one to enable HTTP(S) access to WLCG storage and one to revisit the Information System in view of the actual needs of the experiments, taking into account the planned decommissioning by OSG of the publication of information to the central BDII.

The organisation of the next WLCG workshop is underway and it will be held in Lisbon, starting on February 1st 2016 for three days. The plan is to have an agenda largely driven by the community with ample time for discussion.

Summary of Main Service Incidents An overview of the main incidents for which a Service Incident Report (SIR) was produced is provided below. N.B. SIRs are reported by calendar quarter. For consistency and to facilitate the comparison with previous numbers, Figures 1 and 2 still provide the breakdown per quarter rather than for the full semester.


The full list of SIRs is available at https://twiki.cern.ch/twiki/bin/view/LCG/WLCGServiceIncidents

Site Date(s) Duration Service Area Summary

CERN 9/7/15 2 hours CVMFS Infrastructure Symbolic link vanished unexpectedly from every batch worker within /cvmfs/cms.cern.ch causing all CMS jobs in WLCG to fail

FNAL 15/4/15 15 days dCache Storage A misconfigured storage device caused data corruption affecting 672 CMS files

Table 1 - Summary of SIRs in Q2 and Q3 2015

Figure 1 - Service Incidents by Area


https://twiki.cern.ch/twiki/bin/view/LCG/WLCGServiceIncidents

Figure 2 - Time to Resolution

Summary and Conclusions The operation of the WLCG services continues to be successful as the LHC Run2 is ongoing, with very few major incidents and several improvements to the infrastructure. Decommissioning of old services and an increased effort to provide more efficient tools and services, as needed to meet the requirement of the next years of operations, characterize the activity of the last months.


GDB Report

April – September 2015 Michel Jouvin

October 2015

Introduction There were four GDB meetings held during this period: the one in April was canceled due to its proximity with the WLCG workshop. The agendas show the topics covered, the slides presented and associated papers and the meeting summaries. They can be viewed at http://indico.cern.ch/categoryDisplay.py?categId=3l181. GDB remained lively with many topical discussions, several of them with a dedicated pre-GDB meeting (two during this period)

Pre-GDBs covered several new technologies:

- Batch systems

- Cloud traceability

To improve discussion and action follow-up, meeting summaries are produced after each meeting and a consolidated list of actions decided is updated after each meeting. All this material is publically available at https://twiki.cern.ch/twiki/bin/view/LCG/WLCGGDBDocs.

During GDB, we got reports of several grid infrastructure providers for WLCG: EGI, OSG, and NORDUGRID. We also had a first presentation from BELLE2 experiment and started to discuss how BELLE2 could be associated with GDB to maximize the sharing of solutions for the common needs.

Main Topics Future benchmark needs: this has been a topic for GDB for quite some time, previously mainly on the next generation HEP-SPEC benchmark. This focus changed in the last period to address the new needs identified in the discussion with experiments: the need for a fast but precise benchmark that can be run as part of a workload to help in particular with the “job masonry”, i.e. identify the most suitable job to run in the resources (e.g. time left) available. Linked with a discussion at MB, this should be followed by a dedicated TF on this topic. About HEP-SPEC, the last discussion tended to show that there was no urgency to release a new one, allowing to wait for the availability of the new version of the SPEC suite used so far.

Batch systems: this has been another hot topic in the last 18 months. Since the beginning of the year, a growing number of sites moved to HTCondor for their batch system. A pre-GDB last spring allowed to tighten the links between all sites using this batch system and to ensure that sites planning to switch can benefit from the experience of early adopters. This was also an occasion to ensure that we have a good coordination with the HTCondor development team. The main outcomes of this work are a mailing list to facilitate the communication between sites and a new HTCondor European Workshop planned end of February 2016. We also discuss with middleware maintainers the gaps identified in the support of HTCondor by CE technologies used in WLCG.

Clouds: the main focus of the GDB-related work on this topic in the last six months have been about security traceability to ensure that WLCG cloud infrastructures are well integrated into the security operational procedures developed over the years when setting up the grid infrastructure. Ensuring traceability has been identified as the most crucial challenge, in particular as cloud resources are volatile. The outcome is a set of recommendation to cloud sites on how to configure/use a few standard services to ensure this traceability in the context of the WLCG way of using clouds (through experiment frameworks rather than by end users).


http://indico.cern.ch/categoryDisplay.py?categId=3l181

https://twiki.cern.ch/twiki/bin/view/LCG/WLCGGDBDocs

GDB future role: with the current GDB chair approaching the end of its mandate, it was a good time to discuss future role of GDB as a contributor to WLCG thinking about medium/long-term evolution. This will be coupled with the proposal of a WLCG Technical Forum (WTF), presented by the project leader to both MB and GDB.


Applications Area Report

April 2015 – September 2015 Benedikt Hegner

October 2015

CERNVM CernVM file system The main activity of the past months have been: the release of new production version 2.1.20, which occurred on 31 March 2015; the consolidation of recently introduced/reviewed features such as garbage collection, the S3 interface and the Parrot connector library; the improvements in test coverage and benchmarking; the integration of the build and testing system in Jenkins. It was also decided to align the versioning scheme to the usual practice to have the minor number increase indicating possible API changes. The new scheme has been used for the pre-release of the fixes and improvements, version 2.2.0-0, tagged on 26 August 2015. CernVM appliance As expected, the SL 6.6 based CernVM 3.4 was released on 4 June 2015, featuring support for running Docker containers and fixing the notification of cernvm-update. This was followed on 10 August 2015 by CernVM 3.5, based on SL 6.7 and providing fixes for contextualization on CloudStack and adding support for Microsoft Azure, therefore completing the support for the three large commercial infrastructure-as-a-service providers Amazon, Google, and Microsoft. Work for CernVM 4, based on SL 7 / CentOS 7, is well advanced and a public preview is expected before the end of the year. ROOT A third major release for ROOT 6, version 6.04, was made available at beginning of June with the major causes of increased memory consumption due to header parsing and slow startup time fixed. The memory footprint for version 6 is now comparable with version 5 within a few percent. Release 6.04 is the version recommended for the LHC experiments to use in production. Several additional bug fix releases for 5.34, 6.02 and 6.04 has been produced. Detailed release notes are available from the page http://root.cern.ch/releases. The next major release is scheduled for end of November 2015. The ROOT web site has been migrated to Drupal 7 and hosted by the CERN Web infrastructure. The organization, contents and the graphical style of the site have been reviewed thoroughly. The new site is available under the same URL: http://root.cern.ch. The code generated reference documentation has been migrated to the Doxygen tool. The current results are available at https://root.cern.ch/doc/master/index.html. Work is still needed for the full completion of the reference guide. The ROOT Users’ Workshop took place on 15-18 September 2015 in Saas-Fee, Switzerland. This was the perfect occasion to celebrate the ROOT's 20th anniversary. About 70 people participated to the workshop representing widely the user community. Presentations by users and developers were very well received and triggered very interesting discussions. The development team is analyzing the very useful feedback from users in order to propose a coherent development plan for future releases, in particular ROOT 7, which should be ready for the next long shutdown of LHC.


https://cern.ch/aidasoft/USolids

http://cern.ch/geant4/support/ReleaseNotes4.10.1.html

Over the period of reference, in total 459 new issues (bugs, feature requests) have been created and 291 issues have been resolved.

Simulation

A preview of the new Geant4 release 10.2 (scheduled for December), 10.2-beta, was announced on 26 June 2015. Major features include a new implementation of the Goudsmit-Saunderson model for multiple-scattering, being now used for e+e- below 100 MeV as default in the experimental physics-list FTFP_BERT_TRV; the original Urban multiple-scattering model has also been updated for step limitation and new correction for positrons. Also included a new Monash Compton model including polarization for low-energy electromagnetic physics. With this new release, models for PAI, MicroElec, and DNA can be enabled via UI commands on top of existing EM physics constructors. A major revision of the Quark-Gluon-String (QGS) hadronic model has been made, implementing Reggeon cascading and associated "Fermi motion"; also implemented a new algorithm for parton kinematical properties determination and calculation of residual nucleus properties. The production of evaporated neutrons in Fritiof (FTF) model has been improved, as well as fragmentation of fast residual nuclei in the Binary Cascade. Implemented also an alternative model for photon-evaporation, which can be optionally enabled. The GDML schema has been updated to support optional ‘userinfo’ field, allowing for 'global' auxiliary fields. The GDML parser has now defined the ability to optionally export and import geometrical regions associated to logical-volumes and dump production cuts. A dedicated thread is now adopted in multi-threaded mode for visualization, allowing for more efficient interactive experience. Also implemented support for n-tuple vector columns with Csv output. The detailed list of fixes can be found in the release notes:

http://cern.ch/geant4/support/Beta4.10.2-1.txt

New patch releases of Geant4 for the 10.1 series, 10.1.p01 and 10.1.p02, have been also released in March and June respectively. The patches include fixes in several areas, including porting on new compiler versions and compatibility with external software. The detailed list of fixes can be found in the release notes:

http://cern.ch/geant4/support/Patch4.10.1-1.txt http://cern.ch/geant4/support/Patch4.10.1-2.txt

The HepMC event record is a de-facto standard used by the theoretical community as well as the experiments to interface the Monte Carlo event generators to the detector simulation. Following the outcome of the HepMC workshop held in January 2015, the development of the new major version (3.0) of the HepMC event record was started in February this year. After a phase of the design and prototyping, a model based on the Plain Old Data (POD) structures and smart pointers duality has been selected. The smart pointers allow safe and efficient transient representation of the events, while the PODs are used for fast I/O operations. A dedicated module for the support of the Root I/O is now part of HepMC package. The HepMC-3 development has been completed. The new version has been extensively tested and benchmarked over the last few months. It is already used by the GeantV prototype as the input format of the events. Interfaces to the main generators (Pythia8, Tauola++ and Photos++) have been implemented and several others are under development. The first public release of HepMC-3 is planned to take place during the next couple of weeks.

CORAL and COOL

Two new release tags 3.0.3 and 3.0.4 of CORAL and COOL, using ROOT6 and CMT, were prepared for ATLAS and LHCb during Q2 2015. Several releases were built based on these tags, against different sets of external packages. In particular, COOL 3.0.4 was built both against successive patch releases of ROOT 6.02 (up to 6.02.12 in LCG78root6) and against the first production release of ROOT 6.04 (6.04.02 in LCG79). The upgrade to ROOT 6.04 using ORCJIT


http://cern.ch/geant4/support/Patch4.10.0-3.txt




(as opposed to ROOT 6.02 using JIT) finally solves a long-standing issue with Python crashes in PyCool when COOL C++ exceptions are thrown. The many workarounds that had been added to COOL for this issue in ROOT 6.02-based releases will be removed when all experiments have moved to ROOT 6.04 and support for ROOT 6.02 is dropped. Other notable changes in these software versions include: the first releases on CERN CentOS7; the code and configuration port to clang60 on MacOSX, to gcc51 and to Ubuntu; fixes for all residual defects found in Coverity scans and for other bugs; further progress in integrating CoralServer tests with more recent versions of the ATLAS HLT software; further progress in replacing Boost by c++11 in the internal implementation of CORAL and COOL. All changes to the code bases in 3.0.4 were also backported to two final sets of tags for the CORAL 2.4 / 2.3 and the COOL 2.9 / 2.8 branches, based on ROOT5 and CMT.

The main achievement for CORAL and COOL during Q3 2015 consisted in the move of the internal build and configuration system from CMT to cmake, and in its integration into the lcgcmake infrastructure for nightly and release builds. This was also accompanied by a major restructuring of the two SVN repositories, moving packages to allow direct checkouts into the desired directory structure, and freezing the obsolete development branches CORAL2 and COOL2 based on ROOT5 and CMT. The scripts to set up the runtime environment and run tests from cmake-based installations have also been improved and made fully relocatable, which considerably simplifies the procedure to reproduce and debug issues in the nightlies, particularly those coming from external packages such as ROOT. New candidate release tags 3.1.0 of CORAL and COOL, based for the first time on cmake and on the new SVN structure, have been prepared for an upcoming LCG80 configuration. The new tags also include improvements to PyCool and progress in the port to icc15 and clang35 on Linux.

Software Process & Infrastructure

In this period four new ROOT6 versions, 6.02.08, 6.02.10, 6.02.12 and 6.04.02, have been included in the LCG releases LCG_75root6, LCG_77root6, LCG_78root6, and LCG_79 respectively. Releases for ROOT5 are no longer maintained nor expected. Therefore, the suffix "root6" has been dropped from the release names starting with the LCG_79 tag. In addition, a minor LCG_76root6 has been released as a patch to the previous LCG_75root6 enabling ipv6 for Python2.7.9. In total, 17 new packages and 14 packages updates have been included in all these releases. From LCG_75root6, CentOs7 builds have been included as part of the LCG releases together with slc6. From LCG_78root6 (July 2015) CVMFS has become a regular end-system for LCG releases together with AFS. The publication of new releases details into the LCG documentation page has been automatized (lcgsoft.web.cern.ch). This has been extended to previous LCG releases not published yet in that page.

Regular nightly builds of ROOT 6.04 have been included in the LCG nightly build system. The dev2 branch, previously dedicated to ROOT5, has been modified to host the ROOT 6.04 builds. New compiler versions gcc4.8.4, gcc4.9.2, gcc4.9.3, gcc5.1.0, gcc5.2.0 and clang37 have been installed in AFS. The newest compilers have also been installed in CVMFS following the same tree file structure as in AFS. In addition, the CMake toolkit 3.2.3 version has been installed in both AFS and CVMFS end-systems. The migration of the lcgcmake repository from svn to GITLAB has been completed in this period.


ALICE Report April 2015 – September 2015

Predrag Buncic

October 2015

Finalization of Run1 processing and MC simulation, start of Run 2 data taking and data processing status During the reporting period ALICE has completed large portions of signal-embedding MC productions needed for the reprocessed 2010, 2011 and 2013 RAW data. This allows for full analysis of the latest reconstruction passes to proceed in all Physics Working Groups. In addition to that, we have started a large-scale MC production with Geant4, analogous to the G3 MC general-purpose production for the 2010 periods to compare precisely the detectors responses and, in particular, the dE/dx calibration of the TPC to the various data taking conditions throughout the year.

The O2 project framework development, as well as specific detector implementations, is proceeding according to plan and following the designs given in the Upgrade of the Online-Offline computing system Technical Design Report. Many of the results from O2 software demonstrators were presented at the CHEP2015 conference in Okinawa, Japan (April 2015).

The 13 TeV p-p data taking started smoothly with the full set of detectors, including the newly installed TRD modules, the AD and DCAL. In the period June-September, we have accumulated 2.8PB of RAW data. The data storage – full set of RAW CASTOR at T0 and replication to tape at the T1s – is running well, without any backlog. We have fully exercised the new 10 Gb/sec LHCOPN link to KISTI T1 at full speed and RAW data is being replicated there according to the center share and pledges. The processing of RAW data is ongoing with the same pattern as in Run1 – software calibration cycle on specific triggers, followed immediately by full (Pass1) reconstruction of the entire data set. With the presently available to ALICE batch capacities at T0 and T1s, we are able to follow the data taking, with Pass1 completed and ready for first-level analysis 2-3 days after the data taking. The calibration cycle itself allows for precise offline detector and running conditions QA 24 hours after data taking. The July data-taking period has already been re-calibrated after Pass1 and has been processed in Pass2. The corresponding MC has been produced as well.

The new HLT farm and software were tuned in the first 2 months of data taking. The new compression algorithms have been exercised in parasitic mode (Mode B), compared with the reconstruction of non-compressed data and validated. Presently HLT is running in compression mode (Mode C). The resulting compression of RAW is similar to the one achieved in Run1.

In the period July to September, ALICE was preparing for the 25th Quark Matter Conference (Sep 27 – Oct 3) in Kobe, Japan. The amount of physics analysis on the Grid increased, as usual in the preparation period, reaching up to 80% of all Grid resources available to ALICE. Despite the increased analysis activity, the overall job efficiency was stable, with a slight (3-4 percent) decrease from mid-August to the 3rd week of September. The prevalence of organized analysis is one of the reasons that the efficiency remained high.

The Grid utilization in the accounting period remained very good, with no major incidents. The CPU/Wall efficiency remained constant, at about 85% across all tiers.


ATLAS Report April 2015 – September 2015

Eric Lancon and Simone Campana

October 2015

ATLAS computing activities in the pre-data taking period of 2015 (from February to June) focused on the simulation and reconstruction of proton-proton Monte Carlo events at the LHC Run-2 conditions for 2015. In particular, ATLAS simulated more than 3 billion Geant4 events and reconstructed them under both the 50ns and 25ns data taking conditions. From June onwards the acquisition, processing and storing of LHC collision data became the main focus. LHC data have been processed in quasi real time at the T0 and reprocessed with improved conditions data on the Grid. Thanks to the software improvements carried out during Long Shutdown 1 (improvement by a factor 4 in the reconstruction time), ATLAS is capable to run the full first pass reconstruction on CERN resources, while spilling over part of the workload to T1 centers has also been commissioned. In periods of low data processing activity, the CERN infrastructure is transparently utilized for Monte Carlo production. The ATLAS HLT farm at P1 has been commissioned for the Run-2 online system during the first 6 months of 2015. It was fully utilized for Grid simulation production during two weeks of LHC technical stop and machine development in August.

The major LS1 developments in distributed computing services were completed on schedule in 2014 and commissioned in time for Run-2. The new production system (Prodsys2) and data management system (Rucio) are now fully relied on in production and deliver the expected quality of service, while new capabilities are being introduced regularly. The Event Index system has been deployed in production in early summer and replaces the previous TAG database for specific use cases. It utilizes the Hadoop technology as backend and together with a new platform for data analytics represents the main ATLAS use case thus far for non-relational databases. The Event Service, implemented for fine grained (event level) data processing and especially targeted towards opportunistic resources, is going through large scale testing and we plan to utilize it in production in the next quarter. The integration of further opportunistic resources available through non-Grid interfaces continues; in particular, generic solutions to integrate and leverage HPC resources are being studied. Volunteer computing through the BOINC system is today an important activity, including from the outreach perspective, and offers resources comparable with a large T2 site.

The ATLAS simulation software developments focused on the preparation for the 2016 production campaign, with improved geometry based on the learning from 2015 data. Geant4.10 has been integrated in the simulation framework and is being validated. Progress is being made in the area of fast simulation to be able to cope with expectations for 2016. The ATLAS reconstruction developed for 2015 data taking proved to be very stable and a very low level of patching is needed. The activity focuses in preparing for 2016 data taking with a new release including ROOT6 and considerable algorithmic improvements. The work to develop a multithreaded framework for Run3 started.

A new analysis model was put in place and is being exercised in Run-2: the new xAOD format (readable both via the ATLAS Athena offline software and via ROOT) fully replaced the previous one, with backward compatibility maintained. The derivation framework implementing the experiment train model was developed during the shutdown and is currently utilized through Prodsys-2: AODs from first pass reconstruction are skimmed, slimmed and thinned within 2 days from the time they are produced. They are made available to physics analysis in a variety of data formats tailored to specific analyses, currently numbering ~80. The full Monte Carlo sample can be processed through the derivation framework and results delivered within 15 days for the full 2015 sample. Distributed analysis on derived data as of today represents 20% of the CPU utilization on WLCG resources.

The data lifetime model introduced at the end of 2014 is now fully operational, enabling the deletion of unused data from both disk and tape. More than 20 PB of data have been removed from tape in the April to September period. The ATLAS data popularity system regularly replicates little-used


data to tape, freeing up disk. We have measured only a small number of tape recalls initiated by physics analysis user request.

Finally, ATLAS started considering the effort to produce samples for upgrade phase2 studies in preparation for the TDRs. Improvements and compromises are being evaluated to reduce the resource requests for such activity in the next two years.


CMS Report April 2015 – September 2015

Maria Girone and David Lange

October 2015

After a development, integration and validation period during Long Shutdown 1 (LS1), the software and computing groups have focused on operations for Run2. The activities during the shutdown concentrated on improving code performance and computing services, improving data accessibility and the efficiency of storage usage for all members of the collaboration, and achieving greater flexibility for where workflows could be executed. Given the size and number of changes made in LS1, the start-up has been impressively very smooth.

Since the last LHCC, the computing group has been reconstructing and serving data events and producing simulation in support of Run2 physics activities. Improvements were made in the workflow submission tools, the operations procedures and the monitoring of the progress of processing campaigns. More flexible and agile operations have been achieved though the deployment of improved data and workflow management tools, such as the Data Federation (AAA), Dynamic Data Management and the improved WMAgent.

The AAA data federation allows CMS to stream data over the wide area from nearly any computing facility, resulting in very significant improvements in accessing the data. The Dynamic Data Management Systems monitors the data samples that are accessed and ensures replicas are created and destroyed depending on the “popularity” of each sample. The WMAgent system is the common set of software services that form the basis of all the organized processing control systems from prompt-reconstruction at the Tier-0 to opportunistic processing at any available facility. CMS currently has very low latency for simulation production: less than 1% of the requests are in the system for more than 21 days, and there is routinely no backlog of requests for processing.

The Tier-0 facility has also gone through a major revision during LS1. The changes include the deployment of the CMS multithreaded reconstruction application, which has resulted in large memory savings and reduced latency enabling the handling of the large Run2 trigger rates. During LS1, the Tier-0 moved to a new Agile Infrastructure based on OpenStack, which is similar to what is used to exploit the capacity of the HLT farm during shutdown periods. OpenStack is a cloud-based system for dynamic resource provisioning. CMS was an early adopter of the system in production and worked in concert with IT to achieve a stable infrastructure. The average Tier-0 utilization has been low due to the lower than expected machine live time and event pileup conditions. The ability of the infrastructure to scale with higher load will be better determined in the fall, as the performance of the LHC continues to improve.

Analysis activity on the new data and newly simulated events has been robust. During LS1, CMS deployed the next generation of distributed analysis tools (CRAB3). By now, the majority of the user community has migrated to CRAB3, and operational support for the previous tool, CRAB2, has been dropped. In addition to the new analysis tool, CMS deployed a central global queue for the pilot submission system. All computing work is now centrally tracked and prioritized in one place, which allows the sorting of tasks between activities. Production can be prioritized above all analysis; a group of analysis users can be given more resources than others across all centers. Importantly, the changes resource provisioning based on priority can be done dynamically with response times measured in hours.

The CMS software group continues to produce improved software releases to support the development of the major fronts of CMS. The two areas of primary development are Run2 and the Phase2 upgrades, where the simulation and reconstruction software continues to evolve. We continue to improve the algorithmic use of the multithreaded framework in order to achieve a more widespread deployment in 2016. The software for the current run continues to perform well and a new release is expected for a final data reprocessing toward the end of the year.


LHCb Report April 2015 – September 2015

Marco Cattaneo

October 2015

Real data processing Activities related to data taking were tested at the end of May, and started at the beginning of the LHC physics run at the end of June. Several rounds of small productions were undertaken in June and July to commission the new online calibration and alignment that allows to run in HLT2 a reconstruction identical to the offline reconstruction, and removes the need for an end of year reconstruction reprocessing. The Turbo stream was also commissioned during this period – this stream takes advantage of the offline quality reconstruction available in the HLT to make physics selections directly in the HLT, avoiding the need for an offline reconstruction and stripping pass, thus allowing a much faster turnaround on larger statistics. The first analyses of 2015 data, using the Turbo stream, were presented in the 2015 summer conferences within days of data taking.

Due to the slower than expected startup of LHC, CPU resources used for real data processing represented only 3.5% of total resources used during the period; this has increased to approximately 15% during the last week of September.

Monte Carlo Simulation Simulation has been running at almost full speed using all available resources, being the dominant activity in terms of CPU work. Additional unpledged resources, as well as clouds, on-demand and volunteer computing resources, were also successfully used. The production of simulated events somewhat decreased during the early part of the summer as the collaboration concentrated on preparations for the new data and on commissioning a major update of the simulation package.

Data Storage Storage resources are no longer a concern. It is likely that the storage resources provided to LHCb will not be saturated by the end of the WLCG year in March 2016, due to the reduced LHC live time with respect to the initial expectations.

The usage of datasets produced for physics analysis is constantly monitored, the subsequent analysis of data popularity having allowed LHCb to free a significant amount of disk space.

We have followed the recommendation of the CRSG and changed our computing model to no longer require a second copy of all derived data for data preservation purposes. In case of tape losses, derived data would be regenerated. This results in a reduction of the tape requests for 2016 and 2017. Tape needs have also been reduced by no longer storing the content of the RAW banks in the reconstruction output (FULL.DST). This represents approximately a factor two saving on the size of the FULL.DST, and approximately a one-quarter reduction in future tape requirements.

Figures The first figure shows the number of running LHCb jobs by job type as a function of time.

The second figure shows the evolution of LHCb tape requirements until the end of 2017. The blue curve shows the requirements before the changes described above, the green curve the new requirements. The red curve shows tape pledged until 2015, and flat budget increments for 2016 and 2017. It can be seen that the evolution of tape needs until 2017 is no longer a concern.


WLCG Sites Reliability

ALICE September 2015

From Apr-2015

Target for each site is 97.0%

Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: CREAM-CE + ARC-CE

Average of 8 best sites

(not always the same sites)

Month Reliability

Apr-2015 100%

May-2015 100%

Jun-2015 100%

Jul-2015 100%

Aug-2015 100%

Sep-2015 100%

Average of ALL Tier-0 and

Tier-1 sites

Month Reliability

Apr-2015 100%

May-2015 100%

Jun-2015 99%

Jul-2015 99%

Aug-2015 100%

Sep-2015 100%

Detailed Monthly Site Reliability

Site Apr-2015 May-2015 Jun-2015 Jul-2015 Aug-2015 Sep-2015

CERN 100% 99% 100% 100% 100% 99%

CCIN2P3 100% 100% 100% 100% 100% 100%

CNAF 100% 100% 99% 98% 100% 100%

FZK 100% 100% 100% 100% 100% 100%

KISTI_GSDC 100% 100% 100% 100% 100% 100%

NIKHEF 100% 99% 97% 100% 100% 99%

RAL 100% 100% 100% 100% 100% 100%

RRC_KI_T1 100% 100% 100% 100% 100% 100%

SARA 99% 100% 99% 97% 100% 99%

Target 97% 97% 97% 97% 97% 97%



ATLAS September 2015

From Apr-2015


Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: (OSG-CE + CREAM-CE + ARC-CE) * (all SRMv2 + all OSG-SRMv2)



Month Reliability

Apr-2015 99%

May-2015 100%

Jun-2015 100%

Jul-2015 100%

Aug-2015 100%

Sep-2015 100%


Tier-1 sites

Month Reliability

Apr-2015 97%

May-2015 95%

Jun-2015 91%

Jul-2015 92%

Aug-2015 92%

Sep-2015 92%



CERN-PROD 99% 99% 100% 100% 99% 99%

BNL-ATLAS 100% 100% 100% 100% 100% 99%

FZK-LCG2 100% 99% 99% 100% 100% 99%

IN2P3-CC 100% 100% 100% 100% 100% 99%

INFN-T1 99% 99% 94% 100% 99% 100%

NDGF-T1 77% 97% 99% 100% 100% 100%

NIKHEF-ELPROD 99% 100% 97% 98% 98% 99%

RAL-LCG2 99% 100% 98% 99% 100% 99%

RRC-KI-T1 98% 99% 99% 99% 100% 100%

SARA-MATRIX 98% 94% 94% 95% 100% 99%

TRIUMF-LCG2 99% 96% 100% 100% 100% 100%

Taiwan-LCG2 98% 51% N/A N/A N/A 1%

pic 99% 99% 99% 100% 100% 100%

Target 97% 97% 97% 97% 97% 97%



CMS September 2015

From Apr-2015


Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: (OSG-CE + CREAM-CE + ARC-CE) * (all SRMv2 + all OSG-SRMv2)



Month Reliability

Apr-2015 98%

May-2015 98%

Jun-2015 99%

Jul-2015 99%

Aug-2015 99%

Sep-2015 96%


Tier-1 sites

Month Reliability

Apr-2015 98%

May-2015 98%

Jun-2015 99%

Jul-2015 99%

Aug-2015 99%

Sep-2015 96%



T0_CH_CERN 100% 100% 100% 96% 100% 82%

T1_DE_KIT 100% 98% 100% 100% 100% 85%

T1_ES_PIC 96% 98% 99% 100% 99% 99%

T1_FR_CCIN2P3 99% 99% 99% 100% 100% 100%

T1_IT_CNAF 100% 97% 98% 100% 98% 100%

T1_RU_JINR 93% 97% 98% 99% 100% 100%

T1_UK_RAL 98% 99% 99% 98% 98% 100%

T1_US_FNAL 100% 99% 99% 100% 100% 100%

Target 97% 97% 97% 97% 97% 97%



LHCB September 2015

From Apr-2015


Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: (CREAM-CE + ARC-CE) * all SRMv2



Month Reliability

Apr-2015 99%

May-2015 100%

Jun-2015 100%

Jul-2015 100%

Aug-2015 100%

Sep-2015 100%


Tier-1 sites

Month Reliability

Apr-2015 99%

May-2015 99%

Jun-2015 99%

Jul-2015 100%

Aug-2015 99%

Sep-2015 99%



LCG.CERN.ch 99% 100% 100% 100% 100% 100%

LCG.CNAF.it 100% 100% 99% 100% 89% 92%

LCG.GRIDKA.de 100% 100% 100% 100% 100% 98%

LCG.IN2P3.fr 97% 100% 100% 100% 100% 100%

LCG.NIKHEF.nl 100% 100% 98% 100% 100% 100%

LCG.PIC.es 99% 100% 99% 100% 100% 100%

LCG.RAL.uk 99% 99% 100% 99% 99% 100%

LCG.RRCKI.ru 100% 95% 100% 100% 100% 100%

LCG.SARA.nl 98% 98% 97% 98% 99% 100%

Target 97% 97% 97% 97% 97% 97%


Availability of WLCG Tier-0 + Tier-1 Sites

ALICE Apr-2015 - Sep-2015

Target Availability for each site is 97.0%. Target for 8 best sites is 98.0%

Availability Algorithm: CREAM-CE + ARC-CE

CERN Avail: 100% Unkn: 0% CCIN2P3 Avail: 99% Unkn: 0% CNAF Avail: 100% Unkn: 0% FZK Avail: 99% Unkn: 0%

KISTI_GSDC Avail: 99% Unkn: 1% NIKHEF Avail: 99% Unkn: 0% RAL Avail: 100% Unkn: 0% RRC_KI_T1 Avail: 100% Unkn: 0%

SARA Avail: 98% Unkn: 2%



ATLAS Apr-2015 - Sep-2015


Availability Algorithm: (OSG-CE + CREAM-CE + ARC-CE) * (all SRMv2 + all OSG-SRMv2)

CERN-PROD Avail: 99% Unkn: 0% BNL-ATLAS Avail: 100% Unkn: 0% FZK-LCG2 Avail: 98% Unkn: 0% IN2P3-CC Avail: 99% Unkn: 0%

INFN-T1 Avail: 97% Unkn: 0% NDGF-T1 Avail: 95% Unkn: 0% NIKHEF-ELPROD Avail: 98% Unkn: 0% RAL-LCG2 Avail: 99% Unkn: 0%

RRC-KI-T1 Avail: 99% Unkn: 0% SARA-MATRIX Avail: 96% Unkn: 2% TRIUMF-LCG2 Avail: 99% Unkn: 0% Taiwan-LCG2 Avail: 25% Unkn: 0%

pic Avail: 99% Unkn: 0%



CMS Apr-2015 - Sep-2015



T0_CH_CERN Avail: 96% Unkn: 0% T1_DE_KIT Avail: 96% Unkn: 1% T1_ES_PIC Avail: 98% Unkn: 0% T1_FR_CCIN2P3 Avail: 98% Unkn: 0%

T1_IT_CNAF Avail: 97% Unkn: 0% T1_RU_JINR Avail: 96% Unkn: 1% T1_UK_RAL Avail: 98% Unkn: 0% T1_US_FNAL Avail: 99% Unkn: 1%



LHCB Apr-2015 - Sep-2015


Availability Algorithm: (CREAM-CE + ARC-CE) * all SRMv2

LCG.CERN.ch Avail: 100% Unkn: 1% LCG.CNAF.it Avail: 95% Unkn: 1% LCG.GRIDKA.de Avail: 99% Unkn: 1% LCG.IN2P3.fr Avail: 98% Unkn: 3%

LCG.NIKHEF.nl Avail: 100% Unkn: 0% LCG.PIC.es Avail: 99% Unkn: 1% LCG.RAL.uk Avail: 99% Unkn: 1% LCG.RRCKI.ru Avail: 99% Unkn: 0%

LCG.SARA.nl Avail: 97% Unkn: 2%


September 2015 **Tier-2 Availability and Reliablity Report

ALICE

Federation Summary - Sorted by Name *

Color coding: N/A <30% <60% <90% >=90%

Availability Algorithm: CREAM-CE + ARC-CE

Federation Availability Reliability Federation Availability Reliability

CZ-Prague-T2 95% 95% PL-TIER2-WLCG 88% 88%

FR-GRIF 100% 100% RO-LCG 88% 88%

FR-IN2P3-IPHC 100% 100% RU-RDIG 99% 99%

FR-IN2P3-LPC 99% 99% SK-Tier2-Federation 100% 100%

FR-IN2P3-LPSC 100% 100% T2-LATINAMERICA 100% 100%

FR-IN2P3-SUBATECH 100% 100% TH-Tier2 100% 100%

HU-HGCC-T2 100% 100% UA-Tier2-Federation 67% 67%

IN-DAE-KOLKATA-TIER2 100% 100% UK-SouthGrid 100% 100%

IT-INFN-T2 92% 92%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201509/wlcg/WLCG_All_Sites_ALICE_Sep2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports



ATLAS


Color coding: N/A <30% <60% <90% >=90%



AT-HEPHY-VIENNA-UIBK 50% 50% JP-Tokyo-ATLAS-T2 88% 88%

AU-ATLAS 100% 100% PL-TIER2-WLCG 86% 86%

CA-EAST-T2 96% 97% PT-LIP-LCG-Tier2 50% 50%

CA-WEST-T2 97% 97% RO-LCG 99% 99%

CH-CHIPP-CSCS 99% 99% RU-RDIG 96% 96%

CN-IHEP 100% 100% SE-SNIC-T2 86% 86%

CZ-Prague-T2 97% 97% SI-SiGNET 99% 99%

DE-DESY-GOE-ATLAS-T2 98% 98% SK-Tier2-Federation 82% 82%

DE-DESY-RWTH-CMS-T2 100% 100% T2-LATINAMERICA 99% 99%

DE-FREIBURGWUPPERTAL 98% 98% TR-Tier2-federation N/A N/A

DE-MCAT 97% 97% TW-FTT-T2 N/A N/A

ES-ATLAS-T2 99% 100% UK-London-Tier2 96% 96%

FR-GRIF 99% 99% UK-NorthGrid 79% 85%

FR-IN2P3-CC-T2 97% 100% UK-ScotGrid 94% 95%

FR-IN2P3-CPPM 99% 99% UK-SouthGrid 99% 99%

FR-IN2P3-LAPP 98% 98% US-AGLT2 100% 100%

FR-IN2P3-LPC 96% 96% US-MWT2 100% 100%

FR-IN2P3-LPSC 99% 99% US-NET2 99% 99%

IL-HEPTier-2 85% 85% US-SWT2 98% 98%

IT-INFN-T2 96% 98% US-WT2 92% 92%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201509/wlcg/WLCG_All_Sites_ATLAS_Sep2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports



CMS


Color coding: N/A <30% <60% <90% >=90%



AT-HEPHY-VIENNA-UIBK 88% 97% PK-CMS-T2 85% 85%

BE-TIER2 92% 99% PL-TIER2-WLCG 86% 86%

BR-SP-SPRACE 95% 95% PT-LIP-LCG-Tier2 99% 99%

CERN-PROD 82% 82% RU-RDIG 82% 82%

CH-CHIPP-CSCS 99% 99% T2-LATINAMERICA 78% 84%

CN-IHEP 100% 100% T2_US_Caltech 98% 98%

DE-DESY-RWTH-CMS-T2 94% 96% T2_US_Florida 95% 95%

EE-NICPB 69% 69% T2_US_MIT 86% 86%

ES-CMS-T2 93% 94% T2_US_Nebraska 89% 89%

FI-HIP-T2 82% 82% T2_US_Purdue 98% 98%

FR-GRIF 99% 99% T2_US_UCSD 99% 99%

FR-IN2P3-CC-T2 97% 100% T2_US_Wisconsin 98% 98%

FR-IN2P3-IPHC 99% 99% TH-Tier2 83% 83%

GR-Ioannina-HEP 98% 98% TR-Tier2-federation 100% 100%

HU-HGCC-T2 99% 99% UA-Tier2-Federation 74% 89%

IN-INDIACMS-TIFR 90% 97% UK-London-Tier2 99% 99%

IT-INFN-T2 93% 95% UK-SouthGrid 100% 100%

KR-KNU-T2 100% 100%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201509/wlcg/WLCG_All_Sites_CMS_Sep2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports



LHCB


Color coding: N/A <30% <60% <90% >=90%

Availability Algorithm: (CREAM-CE + ARC-CE) * all SRMv2


CH-CHIPP-CSCS 99% 99% PL-TIER2-WLCG 94% 94%

DE-DESY-RWTH-CMS-T2 100% 100% RO-LCG 80% 80%

ES-LHCb-T2 98% 98% RU-RDIG 99% 99%

FR-GRIF 99% 99% T2-LATINAMERICA 97% 100%

FR-IN2P3-CPPM 98% 98% UK-London-Tier2 68% 68%

FR-IN2P3-LAPP 98% 98% UK-NorthGrid 51% 54%

FR-IN2P3-LPC 99% 99% UK-ScotGrid 95% 96%

IL-HEPTier-2 89% 89% UK-SouthGrid 71% 71%

IT-INFN-T2 93% 94%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201509/wlcg/WLCG_All_Sites_LHCB_Sep2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports


worldwide lhc computing grid status and …...worldwide lhc computing grid status and progress...

Documents