Slide 1

CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it CERN IT Department Overview of Activities Frdric Hemmer IT Department Head CERN 10 February 2010 Slide 2 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Outline General Services Collaborative Tools CDS/Invenio Indico Networking Internal Wireless External CIXP Computer Security CNIC/PLC Spam Intrusion Detection Grid Computing Tier-0 WLCG Data Storage EC Projects Openlab Competency Centres Workshops Summer Student program UNOSAT 10 February 2010 CERN IT Department - Frdric Hemmer 2 Slide 3 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Department Head Frdric Hemmer Communication Systems (CS) Jean-Michel Jouanigot Departmental Infrastructure (DI) Alan Silverman Database Services (DB) Tony Cass Experiment Support (ES) Jamie Shiers Computing Facilities (CF) Wayne Salter Operating Systems & Information Services (OIS) Christian Isnard EU Projects Bob Jones Deputy Head David Foster Planning Officer Alan Silverman WLCG Ian Bird Data & Storage Services (DSS) Alberto Pace User & Document Services (UDS) Tim Smith Platform & Engineering Services (PES) Helge Meinhard Grid Technology (GT) Markus Schulz IT Organization 2010 Director of Research and Computing Sergio Bertolucci CERN openlab CERN School of Computing Computer Security Department Heads Office (DHO) Slide 4 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it IT Staff Breakdown 4 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 5 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it General Services Data Centre Operations (Tier 0) 24x7 operator support and System Administration services to support 24x7 operation of all IT services. Hardware installation & retirement (~7,000 hardware movements/year) Management and Automation framework for large scale Linux clusters Installed Capacity 6300 systems, 39000 processing cores CPU servers, disk servers, infrastructure servers 13900 TB usable on 42600 disk drives 34000 TB on 45000 tape cartridges (56000 slots), 160 tape drives Tenders in progress or planned (estimates) 2400 systems, 16000 processing cores 19000 TB usable on 20000 disk drives 10 February 2010 CERN IT Department - Frdric Hemmer 5 Slide 6 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it General Services (II) E-Mail and Distribution Lists Up to 2 M incoming messages/day, 99% detected as spam 18000 mailboxes (~ 68% owned by physics community) Web Services 8725 Web sites (~45% owned by physics community, 30% AFS-based) Active Directory, CERN Certification Authority & CERN Authentication Central authentication service for Linux and Windows computers and applications Online X509 Certificate Authority Windows Services 60 TB of DFS workspaces (60 TB) ~ 6000 active PCs managed by CMF Windows Terminal Servers and Custom Servers 120 custom servers (not for public use) hosted for various departments including 62 Windows Terminal Servers 10 February 2010 CERN IT Department - Frdric Hemmer 6 Slide 7 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it OIS services significant numbers 15000 Linux systems (Quattor managed or updating from linuxsoft.cern.ch) 5700 In the Computer Center 3100 elsewhere at CERN 6200 outside CERN 6000 active NICE PCs, >1500 Macs. Infrastructure of 300+ servers, including 120 custom servers hosted for various dept. including 62 Windows Terminal Servers. 60 TB DFS workspace including 30 TB for Media Archive in collaboration with UDS, 15 TB Home Directories, 15 TB Project workspaces. 18000 mailboxes, ~ 8000 e-groups, 3.6TB of mail data, 40 production mail servers, ~ 2 M incoming messages/day, ~ 99% detected as spam Fax service, 3000 faxes/month, 1700 users. 8725 Web sites including 845 SharePoint sites, 35 production Web servers, 5.6 M hits/day, 2.2 TBytes/day transferred in June 2009 CERN Certification Authority: 5000 user certificates, 9000 host certificates issued. CERN Authentication usage increased 50000 authentications/day, 180 applications registered. 7 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 8 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it General Services (III) Database and Application Deployment Services Mainly based on Oracle software AIS DBs and Applications, EDMS, Accelerator DBs, IT DBs, CASTOR DBs, Physics databases (Calibration, Alignment, etc...), Public J2EE Service, etc... 120 General Purpose Databases, 240 TB of NAS storage 130 Web /Application Servers with 700 virtual hosts 50 Terabytes of worldwide replicated Physics databases Engineering and Software Development Services Mechanical and electronic CAE, field calculations, structural analysis, simulations, mathematics, etc 50 packages, 1000 users Twiki Service 6000 users, 36000 pages updated per month Version Control Services (CVS/SVN) 2000 users, 200 projects Audiovisual Service: support, record and archive official committees and events VideoConference Service: provide video conferencing in rooms across site Video Conferencing System (Indico) Distributed and used worldwide CDS-Invenio, a Digital Library Open Source Software produced, used and maintained at CERN free support via mailing lists commercial-like support via a maintenance contract 10 February 2010 CERN IT Department - Frdric Hemmer 8 Slide 9 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Invenio Comprehensive solution for the management of document repositories of moderate to large size Currently installed and in use by over a dozen scientific institutions worldwide. Some examples: MeIND - HBZ NRW, Cologne, Germany EPFL Infoscience - Lausanne, Switzerland Aristotle University of Thessaloniki Greece Dipsit de Documents, Universitat Aut. de Barcelona, Spain RomDoc - UPB-CTTPI, Bucharest, Romania Repozytorium Eny Politechnika - Wroclaw Univ. of Tech., Poland Pacific Rim Library - Hong Kong Academic Repository of Rwanda KIST, Kigali, Rwanda Being deployed: AstroParticle Data System, NASA/Smithsonian, ILO 10 February 2010 CERN IT Department - Frdric Hemmer 9 Slide 10 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Invenio in Africa A digital library workshop initiated by UNESCO was organized in Sep 2009 in Kigali to train librarians on how to use CDS-Invenio, attended by librarians from Rwanda, Cameroun, Ghana and Mozambique The Academic Repository of Rwanda was set up and additional training was provided by CERN A fruitful collaboration is on-going with the Kwame Nkrumah University of Science and Technology, Kumasi, Ghana Invenio solution is under consideration for the Documentation Centre on Genocide documents and for the Parliament of Ghana documentation 10 February 2010 CERN IT Department - Frdric Hemmer 10 Slide 11 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Networking Design, implementation and support of CERNs internal and external networking infrastructure in support of desktop, technical and scientific computing Several 10 Gbps backbones in a multi manufacturer environment: GPN, TN, ENs, LCG and External networks Switching capacity of the internal LCG network is 4.8Tbps Deployment and maintenance of network star points and wireless network services More than 400 star points and ~50 000 UTP sockets ~450 wireless base stations Management CERN Firewall and provision of Internet connectivity >10 Gbps Internet connectivity Internal and external firewalling Development and management of tools for network and telecom monitoring, user request provisioning and issue tracking Central database for automatic network equipment configuration About 19 000 network and telecom connectivity requests per year 11 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 12 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it WLCG LHCOPN (simplified) 12 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 13 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Telephony and CIXP Provision and support of telephony services Telephone exchange network of 10 000 lines IP telephony, Audio conferencing, Switchboard, Call centres GSM Mobile services Dedicated VPN of more than 4300 subscriptions Including LHC Tunnel coverage VHF network for the fire brigade Integrated operation services for both network and telecom services Several support contracts using same software tools Management of the CERN Internet Exchange (CIXP) Around 40 clients (telco operators, institutions) 13 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 14 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Computer Security 14 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 15 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it 15 Slide 16 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Frdric Hemmer 16 Tier 0 at CERN: Acquisition, First pass processing Storage & Distribution 26 January 2010 Slide 17 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it WLCG Grid Computing Frdric Hemmer 17 Tier-0 (CERN): Data recording Initial data reconstruction Data distribution Tier-1 (11 centres): Permanent storage Re-processing Analysis Tier-2 (~130 centres): Simulation End-user analysis 26 January 2010 Slide 18 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Data & Storage Services The data management challenges Storing 15000000 gigabytes of data every year Ensure that any file, including the smallest kilobyte is available, anywhere from the internet, within a short time (small latency) Cope with ever-changing storage technologies Long term data preservation All past data must be kept readable for the future Castor Software for data management at CERN (Castor) and for partner data centres 1000 servers, 2000 cores, 10000 disks AFS distributed filesystem operations (30 TB, 500 Million files) Backups (~2 PB, 1 Billion files) 18 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 19 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it 2009 Tier 0-1 Data Transfers Final readiness test (STEP09)Preparation for LHC startupLHC physics data More than 8 GB/s peak transfers from Castor fileservers at CERN Slide 20 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Grid Computing Now Slide 21 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Archeology Astronomy Astrophysics Civil Protection Comp. Chemistry Earth Sciences Finance Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences >250 sites 48 countries >50,000 CPUs >20 PetaBytes >10,000 users >150 VOs >150,000 jobs/day LCG has been the driving force for the European multi-science Grid EGEE (Enabling Grids for E-sciencE) EGEE is now a global effort, and the largest Grid infrastructure worldwide Co-funded by the European Commission (Cost: ~170 M over 6 years, funded by EU ~100M) EGEE already used for >100 applications, including Impact of the LHC Computing Grid in Europe 21 Frdric Hemmer Slide 22 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it 22Health-e-Child Similarity Search Temporal Modelling Visual Data Mining Genetics Profiling Treatment Response Inferring Outcome Biomechanical Models Tumor Growth Modelling Semantic Browsing Personalised Simulation Surgery Planning RV and LV Automatic Modelling Measurement of Pulmonary Trunk Slide 23 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Grid related collaborating projects and other IT-EC projects ~39 2-yr project-funded posts (~26 EGEE posts) on project and IT group-related activities Total requested EC contribution (except PARTNER project): 12.1 M EUR 20 January 2010 23 IT Department Meeting - Frdric Hemmer Slide 24 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Capacity Planning 24 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 25 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Dashboard widely used for STEP09 For many key activities of STEP09 the LHC experiments, in particular, ATLAS and CMS relied on Dashboard monitoring Results of STEP09 postmortem analysis (issues and missing functionality) will define the short-term plan for future Dashboard development. ATLAS data transfer and data processing activities had been monitored by ATLAS Dashboard 3GB/s ALICE used Dashboard for transfer troubleshooting Peaks of 5.5 GB/s 25 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 26 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Dashboard for monitoring of STEP09 for CMS Use of Dashboard in CMS is steadily growing. In June 2009 2713 unique visitors, about 30K pages viewed daily During STEP09 Dashboard was used for job processing, distributed analysis monitoring and monitoring of the site availability 26 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 27 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Grid Technology Coordinate the EGEE grid operation activity Used by LCG as the main production infrastructure Coordinate grid operations security Integrate and Certify the gLite Middleware To enable a high quality infrastructure for EGEE and LCG Provide software lifecycle management services for grid projects ( ETICS ) Provide software for data management at CERN and partner data centres, tier-1s and tier-2s (FTS, DPM, LFC, GFAL, LCG_utils, MSG) Support EGEE related grid projects Health-e-Child, Baltic-Grid, SEE Grid, D4Science 27 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 28 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Other activities CERN openlab Intel, Oracle, Siemens, HP ProCurve http://cern.ch/openlab CERN School of Computing http://cern.ch/csc UNOSAT http://cern.ch/unosat 28 CERN IT Department - Frdric Hemmer 10 February 2010 Slide 29 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Collaboration with Industry: CERN openlab A science industry partnership to drive R&D and innovation Started in 2002, now in phase 3 Motto: you make it we break it Evaluates state-of-the-art technologies in a very complex environment and improves them Test in a research environment today what will be used in industry tomorrow Training: CERN School of Computing openlab student programme Topical seminars January 2010 Slide 30 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it CERN openlab phase III Covers 2009-2011 Status Partners: HP, Intel, Oracle and Siemens Topics Global wireless coverage for CERN (HP Procurve) Power-efficient solutions (Intel) Performance Tuning (Oracle) Control systems and PLC security (Siemens) Advanced storage systems and/or global file system (partner to be identified) 100Gb/s networking (partner to be identified) 30 January 2010 Slide 31 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it openlab people: students in 2009 Summer 2009 31 Slide 32 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it Collaboration with Institutions: UNOSAT 10 February 2010 CERN IT Department - Frdric Hemmer 32 Slide 33 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/it 26 January 2010 Frdric Hemmer 33