IT Department IT Department Infrastructure and General Infrastructure and General

ServicesServices

DTI Global-Watch MissionDTI Global-Watch Mission

John M. FergusonJohn M. Ferguson28 June 200428 June 2004

22

IT Department StructureIT Department StructureInfrastructure and General Infrastructure and General

ServicesServices

Administrative Administrative Information Services Information Services (AIS)(AIS)

Communication Communication Systems (CS)Systems (CS)

Databases (DB)Databases (DB) Internet Services (IS)Internet Services (IS) Product Support (PS)Product Support (PS) User and Document User and Document

Services (UDS)Services (UDS)

Physics ServicesPhysics Services

Architecture and Data Architecture and Data Challenges (ADC)Challenges (ADC)

Controls (CO)Controls (CO) EGEE project (EGE)EGEE project (EGE) Fabric Infrastructure Fabric Infrastructure

and Operations (FIO)and Operations (FIO) Grid Deployment (GD)Grid Deployment (GD) Grid Middleware (GM)Grid Middleware (GM) LHC computing Grid LHC computing Grid

project (LCG)project (LCG)

33

Administrative Information ServicesAdministrative Information Services Speakers: Martens, Mathieson, de JongheSpeakers: Martens, Mathieson, de Jonghe

MandateMandate Provide CERN with a set of integrated and reliable Provide CERN with a set of integrated and reliable

Corporate Information SystemsCorporate Information Systems Ensure coherent, reliable, comprehensive and Ensure coherent, reliable, comprehensive and

accurate management information across CERNaccurate management information across CERN Provide optimal support for all business processes at Provide optimal support for all business processes at

CERNCERN

‘‘DifferencesDifferences’: ’: early move to Unix & RDBMS, early use of the web, in-house early move to Unix & RDBMS, early use of the web, in-house development of interface layer to corporate applications, development of interface layer to corporate applications, early use of workflow, early adoption of Java, J2EE, coding early use of workflow, early adoption of Java, J2EE, coding practicespractices

44

ApproachApproach Best of BreedBest of Breed

Buy best solution for each business unitBuy best solution for each business unit Manage the integration & customisationManage the integration & customisation Provide transparent layer for end-usersProvide transparent layer for end-users

E.g. Financials, HR, Payroll, e-recruitmentE.g. Financials, HR, Payroll, e-recruitment Be-spoke developmentBe-spoke development

Where market availability doesn’t meet CERN needs AND Where market availability doesn’t meet CERN needs AND there is a strong business case:there is a strong business case:E.g. Collaborative project management, Workflow, E.g. Collaborative project management, Workflow, Reporting, Earned Value Management, Contract Reporting, Earned Value Management, Contract ManagementManagement

55

Internet Services GroupInternet Services Group Speaker: PaceSpeaker: Pace

MandateMandate Provide core computing services in three specific areasProvide core computing services in three specific areas

Windows Desktop Services Windows Desktop Services (Desktops, Servers, Core applications)(Desktops, Servers, Core applications)

Collaborative Services Collaborative Services (Mail, Calendar, Mailing lists, Directories, Video Conferencing (Mail, Calendar, Mailing lists, Directories, Video Conferencing …)…)

Web Services Web Services (Server infrastructure, Authoring Tools)(Server infrastructure, Authoring Tools)

‘‘DifferencesDifferences’: ’: Scale of desktop user community, managed desktop Scale of desktop user community, managed desktop environment?, openness of network to large number of external environment?, openness of network to large number of external (on site) visitors, anti spam/ security, automation of web services(on site) visitors, anti spam/ security, automation of web services

66

IS - Windows ServicesIS - Windows Services Provide a consistent desktop environment to CERN usersProvide a consistent desktop environment to CERN users

Directories, file and home directories storageDirectories, file and home directories storage Standardized desktop computing environment Standardized desktop computing environment A core set of application software A core set of application software Technical support to all CERN usersTechnical support to all CERN users

Windows Server infrastructureWindows Server infrastructure ~ 130 servers~ 130 servers ~ 16 TB of disk space, 30 TB raw disks~ 16 TB of disk space, 30 TB raw disks Used also for Web and Mail servicesUsed also for Web and Mail services

Windows Desktop ManagementWindows Desktop Management ~ 10000 User Accounts, 23000 Contacts, 1700 Groups~ 10000 User Accounts, 23000 Contacts, 1700 Groups ~ 7800 Computer accounts in the domain~ 7800 Computer accounts in the domain ~ 5500 Managed desktops (2000 and XP)~ 5500 Managed desktops (2000 and XP)

24x7 Service operation24x7 Service operation

77

IS - Collaborative ToolsIS - Collaborative Tools Provide a consistent collaborative environment to Provide a consistent collaborative environment to

the CERN usersthe CERN users Including messaging, calendaring and videoconferencingIncluding messaging, calendaring and videoconferencing Consisting of both server and client-side services. Consisting of both server and client-side services.

Electronic MailElectronic Mail Around 20 serversAround 20 servers 14,000 accounts14,000 accounts More than 1 million mails / week, 80 % spamMore than 1 million mails / week, 80 % spam More than 3,000 mailing listsMore than 3,000 mailing lists

VideoconferencingVideoconferencing 7 VC rooms operational, 2 more in project7 VC rooms operational, 2 more in project 1,300 registered videoconferences in 20021,300 registered videoconferences in 2002

720 in first half 2003720 in first half 2003

88

IS - Web servicesIS - Web services

Provide a consistent web hosting and authoring Provide a consistent web hosting and authoring environment to the CERN users environment to the CERN users

Roughly 25 Servers:Roughly 25 Servers: ~ 5500 Sites~ 5500 Sites ~ 1.000.000 HTTP requests/day~ 1.000.000 HTTP requests/day

24x7 Service operation24x7 Service operation Intranet Search Engine Service (Inktomi)Intranet Search Engine Service (Inktomi)

99

1010

User and Document ServicesUser and Document Services Speakers: Draper, Le MeurSpeakers: Draper, Le Meur

MandateMandate Production, management and diffusion of information in the Production, management and diffusion of information in the

form of documents, images and multimedia, CERN form of documents, images and multimedia, CERN Document Server CDS and related tools including CDS Document Server CDS and related tools including CDS Agenda. Printing services, specialised graphics printingAgenda. Printing services, specialised graphics printing

HelpdeskHelpdesk PC and Macintosh: procurement /installation of desktop PC and Macintosh: procurement /installation of desktop

computers, installation of standard CERN softwarecomputers, installation of standard CERN software Audio-visual services to users of the main CERN meeting Audio-visual services to users of the main CERN meeting

rooms. Digital video recording, archiving and webcasting of rooms. Digital video recording, archiving and webcasting of events events

Provide support to the CERN Scientific Information ServiceProvide support to the CERN Scientific Information Service

‘‘DifferencesDifferences’:’: Early development of preprint server and subsequently CDS, Early development of preprint server and subsequently CDS,

IndicoIndico

1111

CERN Document ServerCERN Document Server

CDS is a electronic document archival and retrieval systemCDS is a electronic document archival and retrieval system Started in 1994 as CERN Preprint serverStarted in 1994 as CERN Preprint server Now,Now,

Packaged as GNU Public Library software (Packaged as GNU Public Library software (CDSwareCDSware)) licensed to > 20 external institutes and companieslicensed to > 20 external institutes and companies

- e.g. EPFL (Lausanne) and San Diego e.g. EPFL (Lausanne) and San Diego Supercomputing CenterSupercomputing Center

Contains about 650Contains about 650kk metadata records metadata records and and 3350k50k full full textstexts

Each month: 12,000 unique visitors Each month: 12,000 unique visitors -- 120,000 120,000 searchessearches

1212

InDiCoInDiCoA system for Integrated Digital Conference managementA system for Integrated Digital Conference management

Funded by EU as part of FP6Funded by EU as part of FP6 Partners were CERN, SISSA & Udine (IT) and univ Partners were CERN, SISSA & Udine (IT) and univ

Amsterdam(NL)Amsterdam(NL) CERN’s contribution will be a complete software CERN’s contribution will be a complete software

package for the planning,management and archival of package for the planning,management and archival of any type of conferenceany type of conference

Developed from CDS Agenda which is in use at CERNDeveloped from CDS Agenda which is in use at CERN More than 10,000 agenda createdMore than 10,000 agenda created More than 70,000 talks (including multimedia More than 70,000 talks (including multimedia

material) in the database material) in the database Year on year growth of about 200%Year on year growth of about 200%

Currently in prototype and will be used by CHEP’04 in Currently in prototype and will be used by CHEP’04 in SeptemberSeptember

1313

Computing Support to CERN Computing Support to CERN LibraryLibrary

Provide database & system adminisProvide database & system administration for the tration for the integrated library system (ALEPH500)integrated library system (ALEPH500)

Customize system for introduction of new collections:Customize system for introduction of new collections: There are > 400 collectionsThere are > 400 collections

Develop automation programmes for CERN librarians: Develop automation programmes for CERN librarians: automatically add preprints from the LANL archive into the automatically add preprints from the LANL archive into the

CERN databaseCERN database Extract authors (>2000 for LHC experiments) from prepintsExtract authors (>2000 for LHC experiments) from prepints Extract references and publishing informationExtract references and publishing information Etc.Etc.

1414

Database Group - DBDatabase Group - DB Speaker: ShiersSpeaker: Shiers

MandateMandate Provide support for services, based on the Oracle Database Provide support for services, based on the Oracle Database

and Application Server across the lab. and Application Server across the lab. Includes software installation and distribution, consultancy, Includes software installation and distribution, consultancy,

first line support and database management as well as first line support and database management as well as administration for all of the main databases at CERN. administration for all of the main databases at CERN.

Also covers maintaining relations with the supplier and Also covers maintaining relations with the supplier and license / contract management.license / contract management.

Specific areas include support for lab’s Administrative Specific areas include support for lab’s Administrative Information Services (AIS), Physics communityInformation Services (AIS), Physics community and Technical and Technical and Accelerator Sectors.and Accelerator Sectors.

‘DifferencesDifferences’Data volume now GB/TB growing significantly, distributed Data volume now GB/TB growing significantly, distributed

deployment.deployment.Long term relationship with OracleLong term relationship with Oracle

1515

DB- Service OverviewDB- Service Overview AIS servicesAIS services: infrastructure for the development and deployment : infrastructure for the development and deployment

of the core administrative applications of the lab, consisting of of the core administrative applications of the lab, consisting of applications developed in-house as well as third party software applications developed in-house as well as third party software (together with IT-AIS.)(together with IT-AIS.)

Technical / Accelerator servicesTechnical / Accelerator services: Provide the infrastructure : Provide the infrastructure services required for the database applications in these sectors services required for the database applications in these sectors as well as the infrastructure on which CERN's Engineering Data as well as the infrastructure on which CERN's Engineering Data Management Service (EDMS) is run (together with TS-CSE.)Management Service (EDMS) is run (together with TS-CSE.)

Physics servicesPhysics services: Provide database and data management : Provide database and data management services, including the LCG File Catalog, as well as the Oracle services, including the LCG File Catalog, as well as the Oracle Database and Application Server for physics / detector Database and Application Server for physics / detector applications.applications.

Persistency Framework ProjectPersistency Framework Project: includes both POOL (object : includes both POOL (object persistency) and detector conditions DB (together with PH-SFT / persistency) and detector conditions DB (together with PH-SFT / experiments)experiments)

1616

DB - Oracle at CERNDB - Oracle at CERN Originally chosen in LEP construction phase (1983)Originally chosen in LEP construction phase (1983) Services have expanded across laboratory:Services have expanded across laboratory:

Now used in all areas of the lab’s workNow used in all areas of the lab’s work Accelerator construction, operation, deconstruction (LEP), Accelerator construction, operation, deconstruction (LEP),

…… AIS applications (CET, EDH, HRT, …)AIS applications (CET, EDH, HRT, …) Physics related services: Detector construction, calibration, Physics related services: Detector construction, calibration,

LCG File Catalog, Event Metadata …LCG File Catalog, Event Metadata …

And from Oracle Database to DB + Application Server + Grid And from Oracle Database to DB + Application Server + Grid ControlControl

Total number of services ~100Total number of services ~100 Sun Cluster, “single instance” Suns, DiskServers running RHEL, …Sun Cluster, “single instance” Suns, DiskServers running RHEL, … Total non-physics data: few 100s of GB (400GB alone for AIS…)Total non-physics data: few 100s of GB (400GB alone for AIS…) Total physics data (COMPASS): few TBTotal physics data (COMPASS): few TB Expect to move to Oracle 10g prior to LHC operationExpect to move to Oracle 10g prior to LHC operation

Contains numerous features requested by CERN, but of Contains numerous features requested by CERN, but of general interestgeneral interest

Databases will increase in volume to 10-100TB rangeDatabases will increase in volume to 10-100TB range

1717

Communication Systems - CSCommunication Systems - CS Speakers: Martin, JouanigotSpeakers: Martin, Jouanigot

MandateMandate Provide infrastructure and support operations for the Provide infrastructure and support operations for the following:following:

Internal Networking (CN)Internal Networking (CN) Network Management and Operations (NSO)Network Management and Operations (NSO) External Networking (EN)External Networking (EN) Telephony/GSM/Audio-Visual services (TEL)Telephony/GSM/Audio-Visual services (TEL)

‘‘DifferencesDifferences’’External network capacity requirements, internal user network External network capacity requirements, internal user network dimensions, technical network, grid requirements, cluster dimensions, technical network, grid requirements, cluster communicationscommunications

1818

Communication SystemsCommunication Systems Internal NetworkingInternal Networking

33’000 internal network connections33’000 internal network connections 15’000 systems connected15’000 systems connected 220 star points 300 new network devices/year (routers, switches etc) – 220 star points 300 new network devices/year (routers, switches etc) –

2400 total2400 total 1000 work orders (add/move/remove) per month1000 work orders (add/move/remove) per month 1200 backbone access points1200 backbone access points 70 wireless base station installations70 wireless base station installations

TelephonyTelephony 9000 fixed line phones, 15 VoIP concentrators, 4 PABX exchanges, 600 9000 fixed line phones, 15 VoIP concentrators, 4 PABX exchanges, 600

operator assisted calls/dayoperator assisted calls/day 2800 active GSM mobiles, 42 base stations2800 active GSM mobiles, 42 base stations GSM and emergency phone installations for the LHC tunnelGSM and emergency phone installations for the LHC tunnel 16 VHF installations for the firemen16 VHF installations for the firemen 80 Intercom installations for access control80 Intercom installations for access control 150 phone installations for CERN lifts150 phone installations for CERN lifts

1919

Main Connections to Main Connections to CERNCERN

SWITCH

CIXP

WHO

Europe

USA

CERN

CERN

1Gbp

s

1Gbp

s

USLIC 10Gbps

Swiss National Research Network

Mission Oriented & World Health Org.

General purpose A&R and commodityInternet connections(Europe/USA/World)

Commercial

IN2P3

45Mbps

1Gbps

GEANT (2.5/10Gbps)

1/10Gbps

DataTAG 10Gbps

Network Research

NetherLight ATRIUM/VTHD

2.5Gbps10Gbps

From ~25G (2003)

To ~40G (2004)

2020

Moving towards LHCMoving towards LHC

Create the Grid Communications Infrastructure needed.Create the Grid Communications Infrastructure needed. Understand the external networking requirements in more Understand the external networking requirements in more

detail and work towards building and testing the detail and work towards building and testing the infrastructure.infrastructure.

Involves working with T1’s NREN’s GEANT and others.Involves working with T1’s NREN’s GEANT and others. Probably need 70-100 Gb/sec aggregate wide-area Probably need 70-100 Gb/sec aggregate wide-area

bandwidthbandwidth Build the high performance farm routing backboneBuild the high performance farm routing backbone

Will require state of the art routing technology to Will require state of the art routing technology to connect more than 8000 systems at 1Gb/secconnect more than 8000 systems at 1Gb/sec

2121

Product Support Product Support Software Licence Office – assist in negotiations of CERN’s Software Licence Office – assist in negotiations of CERN’s

software licencessoftware licences Licence Service – fail-safe service for FlexLM and other Licence Service – fail-safe service for FlexLM and other

industry licence tools hosting licences for a range of industry licence tools hosting licences for a range of software such as compilers, development tools, LSF, CATIA, software such as compilers, development tools, LSF, CATIA, etcetc

Installation and product support for a wide range of Installation and product support for a wide range of engineering software (Euclid, CATIA, Autocad, Cadence, and engineering software (Euclid, CATIA, Autocad, Cadence, and many more)many more)

Application support for electronics engineering tools such Application support for electronics engineering tools such as Cadence and PSpiceas Cadence and PSpice

Installation and product support for a range of Software Installation and product support for a range of Software Development tools such as Framemaker, Rational Rose, etcDevelopment tools such as Framemaker, Rational Rose, etc

2222

Product Support Product Support

Solaris SupportSolaris Support Support for the Solaris operating systemSupport for the Solaris operating system Support for the Solaris-based SUNDEV clusterSupport for the Solaris-based SUNDEV cluster

CVS service CVS service Print ServicePrint Service

Support for the fail-safe central print serversSupport for the fail-safe central print servers Support for the UNIX print clientSupport for the UNIX print client

Desktop Support Contract managementDesktop Support Contract management Local Organisation coordination of CHEP ’04 conferenceLocal Organisation coordination of CHEP ’04 conference

2323

Questions/Comments?Questions/Comments?