case study: developing and integrating a cloud sourcing strategy at cern
TRANSCRIPT
![Page 1: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/1.jpg)
![Page 2: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/2.jpg)
Cloud Sourcing for CERNDr Helge Meinhard / CERN-ITGartner Sourcing Summit London07 June 2016
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 2
![Page 3: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/3.jpg)
CERN
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 307-Jun-2016
![Page 4: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/4.jpg)
CERN
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 307-Jun-2016
![Page 5: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/5.jpg)
CERN• International organisation close
to Geneva, straddling Swiss-French border, founded 1954
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 3
1954: 12 Member States
07-Jun-2016
![Page 6: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/6.jpg)
CERN• International organisation close
to Geneva, straddling Swiss-French border, founded 1954
• Facilities for fundamental research in particle physics
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 3
1954: 12 Member States
07-Jun-2016
![Page 7: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/7.jpg)
CERN• International organisation close
to Geneva, straddling Swiss-French border, founded 1954
• Facilities for fundamental research in particle physics
• 21 member states,1 B CHF budget
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 3
1954: 12 Member States
Members: Austria, Belgium, Bulgaria, Czech republic, Denmark Finland, France, Germany, Greece, Hungary, Israel, Italy, Netherlands, Norway, Poland, Portugal, Slovak Republic, Spain, Sweden, Switzerland, United KingdomCandidate for membership: RomaniaAssociate member: SerbiaObservers: European Commission, India, Japan, Russia, Turkey, UNESCO, United States of AmericaNumerous non-member states with collaboration agreement
07-Jun-2016
![Page 8: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/8.jpg)
CERN• International organisation close
to Geneva, straddling Swiss-French border, founded 1954
• Facilities for fundamental research in particle physics
• 21 member states,1 B CHF budget
• 3’197 staff, fellows, apprentices, …
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 3
1954: 12 Member States
Members: Austria, Belgium, Bulgaria, Czech republic, Denmark Finland, France, Germany, Greece, Hungary, Israel, Italy, Netherlands, Norway, Poland, Portugal, Slovak Republic, Spain, Sweden, Switzerland, United KingdomCandidate for membership: RomaniaAssociate member: SerbiaObservers: European Commission, India, Japan, Russia, Turkey, UNESCO, United States of AmericaNumerous non-member states with collaboration agreement
2’531 staff members, 645 fellows, 21 apprentices
07-Jun-2016
![Page 9: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/9.jpg)
CERN• International organisation close
to Geneva, straddling Swiss-French border, founded 1954
• Facilities for fundamental research in particle physics
• 21 member states,1 B CHF budget
• 3’197 staff, fellows, apprentices, …
• 13’128 associates
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 3
1954: 12 Member States
Members: Austria, Belgium, Bulgaria, Czech republic, Denmark Finland, France, Germany, Greece, Hungary, Israel, Italy, Netherlands, Norway, Poland, Portugal, Slovak Republic, Spain, Sweden, Switzerland, United KingdomCandidate for membership: RomaniaAssociate member: SerbiaObservers: European Commission, India, Japan, Russia, Turkey, UNESCO, United States of AmericaNumerous non-member states with collaboration agreement
2’531 staff members, 645 fellows, 21 apprentices
7’000 member states, 1’800 USA, 900 Russia, 270 Japan, …
07-Jun-2016
![Page 10: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/10.jpg)
CERN• International organisation close
to Geneva, straddling Swiss-French border, founded 1954
• Facilities for fundamental research in particle physics
• 21 member states,1 B CHF budget
• 3’197 staff, fellows, apprentices, …
• 13’128 associates
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 3
“Science for peace”
1954: 12 Member States
Members: Austria, Belgium, Bulgaria, Czech republic, Denmark Finland, France, Germany, Greece, Hungary, Israel, Italy, Netherlands, Norway, Poland, Portugal, Slovak Republic, Spain, Sweden, Switzerland, United KingdomCandidate for membership: RomaniaAssociate member: SerbiaObservers: European Commission, India, Japan, Russia, Turkey, UNESCO, United States of AmericaNumerous non-member states with collaboration agreement
2’531 staff members, 645 fellows, 21 apprentices
7’000 member states, 1’800 USA, 900 Russia, 270 Japan, …
07-Jun-2016
![Page 11: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/11.jpg)
CERN – Where the Web was Born
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 407-Jun-2016
![Page 12: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/12.jpg)
CERN – Where the Web was Born
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 407-Jun-2016
![Page 13: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/13.jpg)
CERN – Where the Web was Born
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 407-Jun-2016
![Page 14: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/14.jpg)
Tools (1): LHC
Exploration of a new energy frontierin p-p and Pb-Pb collisions
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 507-Jun-2016
![Page 15: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/15.jpg)
Tools (1): LHC
Exploration of a new energy frontierin p-p and Pb-Pb collisions
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 507-Jun-2016
![Page 16: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/16.jpg)
Tools (1): LHC
Exploration of a new energy frontierin p-p and Pb-Pb collisions
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 507-Jun-2016
LHC ring:27 km circumference
Run 1 (2010-2013): 4+4 TeVRun 2 (2015-2018): 6.5 + 6.5 TeV
![Page 17: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/17.jpg)
Tools (2): Detectors
Exploration of a new energy frontierin p-p and Pb-Pb collisions
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 607-Jun-2016
![Page 18: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/18.jpg)
Tools (2): Detectors
Exploration of a new energy frontierin p-p and Pb-Pb collisions
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 607-Jun-2016
![Page 19: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/19.jpg)
Tools (2): Detectors
Exploration of a new energy frontierin p-p and Pb-Pb collisions
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 607-Jun-2016
ATLAS (A Toroidal Lhc ApparatuS)• 25 m diameter, 46 m length, 7’000 tons• 3’000 scientists (including 1’000 grad students)• 150 million channels• 40 MHz collision rate• Event rate after filtering: 300 Hz in Run 1; up to
1’000 Hz in Run 2
![Page 20: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/20.jpg)
Tools (2): Detectors
Exploration of a new energy frontierin p-p and Pb-Pb collisions
CMSATLAS
General Purpose,proton-proton, heavy ionsDiscovery of new physics:
Higgs, SuperSymmetry
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 607-Jun-2016
![Page 21: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/21.jpg)
Tools (2): Detectors
Exploration of a new energy frontierin p-p and Pb-Pb collisions
CMS
LHCb
ATLAS
General Purpose,proton-proton, heavy ionsDiscovery of new physics:
Higgs, SuperSymmetry
pp, B-Physics, CP Violation(matter-antimatter symmetry)
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 607-Jun-2016
![Page 22: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/22.jpg)
Tools (2): Detectors
Exploration of a new energy frontierin p-p and Pb-Pb collisions
CMS
ALICE
LHCb
ATLAS
General Purpose,proton-proton, heavy ionsDiscovery of new physics:
Higgs, SuperSymmetry
pp, B-Physics, CP Violation(matter-antimatter symmetry)
Heavy ions, pp(state of matter of early universe)
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 607-Jun-2016
![Page 23: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/23.jpg)
Results so far• Many… the most spectacular
one being
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 707-Jun-2016
![Page 24: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/24.jpg)
Results so far• Many… the most spectacular one
being04 July 2012: Discovery of a “Higgs-like particle”
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 707-Jun-2016
![Page 25: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/25.jpg)
Results so far• Many… the most spectacular one
being04 July 2012: Discovery of a “Higgs-like particle”
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 707-Jun-2016
![Page 26: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/26.jpg)
Results so far• Many… the most spectacular one
being04 July 2012: Discovery of a “Higgs-like particle”
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 707-Jun-2016
![Page 27: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/27.jpg)
Results so far• Many… the most spectacular one being
04 July 2012: Discovery of a “Higgs-like particle”
• March 2013: The particle is indeed a Higgs boson
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 707-Jun-2016
![Page 28: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/28.jpg)
Results so far• Many… the most spectacular one being
04 July 2012: Discovery of a “Higgs-like particle”
• March 2013: The particle is indeed a Higgs boson
• 08 Oct 2013 / 10 Dec 2013: Nobel price to Peter Higgs and François Englert• CERN, ATLAS and CMS explicitly mentioned
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 707-Jun-2016
![Page 29: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/29.jpg)
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 8
What is the data? 150 million sensors deliver data
…40 million times per second • Up to 6 GB/s to be permanently stored after filtering
• Almost 30 PB/y in Run 1
• Expect ~50 PB/y in Run 2
• 2023: 400 PB/y(?)
07-Jun-2016
![Page 30: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/30.jpg)
The Worldwide LHC Computing GridAn International collaboration to distribute and analyse LHC data
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 9
![Page 31: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/31.jpg)
Integrates computer centres worldwide that provide computing and storage resource into a single infrastructure accessible by all LHC physicists
The Worldwide LHC Computing GridAn International collaboration to distribute and analyse LHC data
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 9
![Page 32: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/32.jpg)
Integrates computer centres worldwide that provide computing and storage resource into a single infrastructure accessible by all LHC physicists
The Worldwide LHC Computing GridAn International collaboration to distribute and analyse LHC data
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 9
![Page 33: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/33.jpg)
Integrates computer centres worldwide that provide computing and storage resource into a single infrastructure accessible by all LHC physicists
The Worldwide LHC Computing Grid
Tier-0 (CERN):data recording, reconstruction and distribution
An International collaboration to distribute and analyse LHC data
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 9
![Page 34: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/34.jpg)
Integrates computer centres worldwide that provide computing and storage resource into a single infrastructure accessible by all LHC physicists
The Worldwide LHC Computing Grid
Tier-1: permanent storage, re-processing, analysis
Tier-0 (CERN):data recording, reconstruction and distribution
An International collaboration to distribute and analyse LHC data
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 9
![Page 35: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/35.jpg)
Integrates computer centres worldwide that provide computing and storage resource into a single infrastructure accessible by all LHC physicists
The Worldwide LHC Computing Grid
Tier-1: permanent storage, re-processing, analysis
Tier-0 (CERN):data recording, reconstruction and distribution
Tier-2: Simulation,end-user analysis
An International collaboration to distribute and analyse LHC data
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 9
![Page 36: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/36.jpg)
Integrates computer centres worldwide that provide computing and storage resource into a single infrastructure accessible by all LHC physicists
The Worldwide LHC Computing Grid
Tier-1: permanent storage, re-processing, analysis
Tier-0 (CERN):data recording, reconstruction and distribution
Tier-2: Simulation,end-user analysis
> 2 million jobs/day
~350’000 cores
500 PB of storage
more than 170 sites, 40 countries
10-100 Gb links
An International collaboration to distribute and analyse LHC data
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 9
![Page 37: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/37.jpg)
Challenges
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 10
![Page 38: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/38.jpg)
ChallengesRun 2:• Moore’s law helps, but not
sufficient
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 10
![Page 39: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/39.jpg)
ChallengesRun 2:• Moore’s law helps, but not
sufficient• Large effort spent to improve
software efficiency• Exploit multi-threading, new
instruction sets, …
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 10
![Page 40: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/40.jpg)
ChallengesRun 2:• Moore’s law helps, but not
sufficient• Large effort spent to improve
software efficiency• Exploit multi-threading, new
instruction sets, …• Still need factor 2 in terms of
cores, storage etc.
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 10
![Page 41: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/41.jpg)
Tier-0: 15% of WLCG
![Page 42: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/42.jpg)
Tier-0: 15% of WLCG
CERN data centre3.5 MW - full
![Page 43: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/43.jpg)
Tier-0: 15% of WLCG
CERN data centre3.5 MW - full
Tier-0 extension:Wigner Research Centre,Budapest/Hungary
![Page 44: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/44.jpg)
Tier-0: 15% of WLCG
CERN data centre3.5 MW - full
Tier-0 extension:Wigner Research Centre,Budapest/Hungary
Two dedicated100GE connections
![Page 45: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/45.jpg)
Transforming In-House Resources (1)Before Wigner deployment:• Physical servers only
• Inefficient resource usage• Strong coupling of services with HW life-cycle
• Vertical view• Service managers responsible for entire stack
• Home-made tools of 10 years ago• Successful at the time, but Increasingly brittle• Lack of support for dynamic host creation/deletion• Limited scalability
• Person-power: (at best) constant• … despite many more machines
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 12
![Page 46: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/46.jpg)
Transforming In-House Resources (2)Current situation:• Full support for physical and virtual servers• Full support for remote machines• Horizontal view
• Responsibilities by layers of service deployment• Large fraction of resources run as private cloud under OpenStack• Scaling to large numbers
(> 15’000 physical, several 100’000s virtual)• Support for dynamic host creation/deletion
• Deploy new services/servers in hours rather than weeks/months• Optimise operational and resource efficiency
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 13
![Page 47: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/47.jpg)
Scaling up Further: Public Clouds (1)• Additional resources, perhaps later replacing
on-premise capacity• Potential benefits:
• Economy of scale• More elastic, adapts to changing demands• Somebody else worries about machines and
infrastructure
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 14
![Page 48: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/48.jpg)
Scaling up Further: Public Clouds (2)• Potential issues:
• Cloud provider’s business models not well adapted to procurement rules and procedures of public organisations
• Lack of skills for and experience with procurements• Market largely not targeting compute-heavy tasks
• Performance metrics/benchmarks not established• Legal impediments• Not integrated with on-premise resources and/or publicly
funded e-infrastructures
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 15
![Page 49: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/49.jpg)
CERN Approach
1607-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch
Series of short procurement projects of increasing size and complexity
![Page 50: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/50.jpg)
CERN Approach
1607-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch
Series of short procurement projects of increasing size and complexity
![Page 51: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/51.jpg)
CERN Approach
1607-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch
Series of short procurement projects of increasing size and complexity
![Page 52: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/52.jpg)
CERN Approach
1607-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch
Series of short procurement projects of increasing size and complexity
![Page 53: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/53.jpg)
CERN Approach
1607-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch
Series of short procurement projects of increasing size and complexity
![Page 54: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/54.jpg)
CERN Approach
1607-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch
Series of short procurement projects of increasing size and complexity
![Page 55: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/55.jpg)
CERN Approach
16
Open Telekom Cloud
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch
Series of short procurement projects of increasing size and complexity
![Page 56: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/56.jpg)
CERN Procurements• HN-1 (ATOS): Detector simulation for ATLAS • (Microsoft Azure evaluation)• HN-2 (DBCE): Detector simulation for ATLAS, CMS, ALICE,
LHCb• (IBM SoftLayer evaluation)• HN-3 (T-Systems OTC): Detector simulation, reconstruction and
analysis for ATLAS, CMS, ALICE, LHCb• Setting up and ramping up now for 4’000 cores July – September• Includes storage for data-heavy workflows
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 17
![Page 57: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/57.jpg)
Some Lessons Learned• APIs not as stable and well-defined as you would expect
them to be• Consider requiring support for ecosystem tool such as
Terraform, libcloud, jclouds• Accounting is key – required on user side, too• Benchmarking/performance metrics required• Avoid brokers – they risk adding complexity, cost,
intransparency• YMMV – EMBL and ESA have reported rather positive
experience
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 18
![Page 58: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/58.jpg)
Future Requirements• Not only LHC, but a number of particle
physics projects with high data rates• Not only particle physics, but also other
physics fields (e.g. astronomy)• Not only physics, but also other sciences
(e.g. life sciences, material science)
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 19
![Page 59: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/59.jpg)
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 20
2010 2015 2020 2025 2030 2035
Physics
Construct Physics
Design Proto Construct Physics
Construct Physics
Construct Physics
Construct Physics
LHC
SuperKEKB
HL-LHC
FCC
LBNF
ILC Design
Integrated view between Europe (ESPP), USA (P5), Japan
ScienceScience
Science
SKALSSTCTA
Par
ticle
Phy
sics
Ast
ro-
phys
ics
![Page 60: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/60.jpg)
Not only Physics
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 21
Growth of EBI (European Bioinformatics Institute) repositories, lines are 12 month doubling
![Page 61: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/61.jpg)
Procurers: CERN, CNRS, DESY, EMBL-EBI, ESRF,IFAE, INFN, KIT, SURFSara, STFC
Experts: Trust-IT & EGI.eu
The group of procurers have committed• >1.6M€ of procurement funds• Manpower for testing/evaluation• Use-cases with applications & data• In-house IT resources
To procure innovative IaaS level cloud services integrated into a hybrid cloud model• Commercial cloud services• European e-Infrastructures
Services will be made available to end-users from many research communities
Co-funded via H2020 (Jan’16-Jun’18)• Grant Agreement 687614
HELIX NEBULA Science CloudJoint Pre-Commercial Procurement
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 22Total procurement commitment >5M€
07-Jun-2016
![Page 62: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/62.jpg)
Cloud sourcing for CERN - Helge Meinhard at CERN.ch 2307-Jun-2016
HNSciCloud PCP Project Phases
![Page 63: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/63.jpg)
• High Energy Physics• LHC experiments• Belle II• COMPASS
• Astronomy• CTA – Cherenkov Telescope Array• MAGIC• Pierre Auger Observatory
• Life Sciences• ELIXIR• Euro-BioImaging• Pan-Cancer• BBMRI• WeNMR
• Photon/Neutron science• PETRA III, European XFEL, 3DIX, OCEAN, OSIRIS
• Long tail of science
User groups to be supported
![Page 64: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/64.jpg)
Technical Challenges• Compute
• Integration of some HPC requirements• Storage
• Caching at provider’s site, if possible automatically (avoid managed storage)
• Network• Connection via GÉANT• Support of identity federation (eduGAIN) for IT managers
• Procurement• Match of cloud providers’ business model with public procurement
rules
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 25
![Page 65: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/65.jpg)
HNSciCloud – Current Status• Technical and administrative work well advanced
• Consortium agreement signed• Subcommittees established and working
• Tender announced in Jan 2016• Open Market consultation successfully held on 17
March 2016• Tender material in final phase of preparation
• To be published this summer
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 26
![Page 66: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/66.jpg)
Summary• Public clouds have a large potential of addressing the
requirements of public research organisations for ever more resources and of dealing with peak demands
• Using public clouds isn’t as easy as you would like it to be• A full integration of public clouds with on-premise
resources and public e-infrastructures is required• Commercial cloud services are expected to play an
increasing role in the computing models of scientific Research Infrastructures as part of a hybrid cloud platform
07-Jun-2016Cloud sourcing for CERN - Helge Meinhard at CERN.ch 27
![Page 67: Case Study: Developing and Integrating a Cloud Sourcing Strategy at CERN](https://reader035.vdocuments.net/reader035/viewer/2022070520/58f16dc31a28ab47508b456d/html5/thumbnails/67.jpg)
Accelerating Science and Innovation