20121205 open stack_accelerating_science_v3
TRANSCRIPT
Accelerating Sciencewith OpenStack
The CERN User Story
@noggin143
EMEA OpenStack Day, London5th December 2012
What is CERN ?
OpenStack London December 2012 Tim Bell, CERN 2
• Conseil Européen pour la Recherche Nucléaire – aka European Laboratory for Particle Physics
• Between Geneva and the Jura mountains, straddling the Swiss-French border
• Founded in 1954 with an international treaty
• Our business is fundamental physics , what is the universe made of and how does it work
OpenStack London December 2012 Tim Bell, CERN 3
Answering fundamental questions…• How to explain particles have mass?
We have theories and accumulating experimental evidence.. Getting close…
• What is 96% of the universe made of ?We can only see 4% of its estimated mass!
• Why isn’t there anti-matterin the universe?
Nature should be symmetric…
• What was the state of matter justafter the « Big Bang » ?
Travelling back to the earliest instants ofthe universe would help…
Community collaboration on an international scale
Tim Bell, CERN 4OpenStack London December 2012
Tim Bell, CERN 5
The Large Hadron Collider
OpenStack London December 2012
6
The Large Hadron Collider (LHC) tunnel
OpenStack London December 2012 Tim Bell, CERN
OpenStack London December 2012 Tim Bell, CERN 7
Accumulating events in 2009-2011
OpenStack London December 2012 Tim Bell, CERN 8
OpenStack London December 2012 Tim Bell, CERN 9
Heavy Ion Collisions
OpenStack London December 2012 Tim Bell, CERN 10
Data Acquisition and Trigger Farms
OpenStack London December 2012 Tim Bell, CERN 11
OpenStack London December 2012 Tim Bell, CERN 12
OpenStack London December 2012 Tim Bell, CERN 13
Tier-1 (11 centres):•Permanent storage•Re-processing•Analysis
Tier-0 (CERN):•Data recording•Initial data reconstruction•Data distribution
Tier-2 (~200 centres):• Simulation• End-user analysis
• Data is recorded at CERN and Tier-1s and analysed in the Worldwide LHC Computing Grid• In a normal day, the grid provides 100,000 CPU days executing over 2 million jobs
OpenStack London December 2012 Tim Bell, CERN 14
• Data Centre by Numbers– Hardware installation & retirement
• ~7,000 hardware movements/year; ~1,800 disk failures/year
Xeon 51502%
Xeon 516010%
Xeon E5335
7%Xeon
E534514%
Xeon E5405
6%
Xeon E541016%
Xeon L5420
8%
Xeon L552033%
Xeon 3GHz4%
Fujitsu3%
Hitachi23% HP
0% Maxtor
0% Seagate15%
Western Digital
59%
Other0%
High Speed Routers(640 Mbps → 2.4 Tbps) 24
Ethernet Switches 350
10 Gbps ports 2,000
Switching Capacity 4.8 Tbps
1 Gbps ports 16,939
10 Gbps ports 558
Racks 828
Servers 11,728
Processors 15,694
Cores 64,238
HEPSpec06 482,507
Disks 64,109
Raw disk capacity (TiB) 63,289
Memory modules 56,014
Memory capacity (TiB) 158
RAID controllers 3,749
Tape Drives 160
Tape Cartridges 45,000
Tape slots 56,000
Tape Capacity (TiB) 73,000
IT Power Consumption 2,456 KW
Total Power Consumption 3,890 KW
OpenStack London December 2012 Tim Bell, CERN 15
Our Challenges - Data storage
OpenStack London December 2012 Tim Bell, CERN 16
• >20 years retention• 6GB/s average• 25GB/s peaks• 35PB/year recorded
OpenStack London December 2012 Tim Bell, CERN 17
45,000 tapes holding 80PB of physics data
New data centre to expand capacity
OpenStack London December 2012 Tim Bell, CERN 18
• Data centre in Geneva at the limit of electrical capacity at 3.5MW
• New centre chosen in Budapest, Hungary
• Additional 2.7MW of usable power
• Hands off facility• Deploying from 2013
with 200Gbit/s network to CERN
Time to change strategy• Rationale
– Need to manage twice the servers as today– No increase in staff numbers– Tools becoming increasingly brittle and will not scale as-is
• Approach– CERN is no longer a special case for compute– Adopt an open source tool chain model– Our engineers rapidly iterate
• Evaluate solutions in the problem domain• Identify functional gaps and challenge them• Select first choice but be prepared to change in future
– Contribute new function back to the community
OpenStack London December 2012 Tim Bell, CERN 19
Building Blocks
OpenStack London December 2012 Tim Bell, CERN 20
Bamboo
Koji, Mock
AIMS/PXEForeman
Yum repoPulp
Puppet-DB
mcollective, yum
JIRA
Lemon /Hadoop
git
OpenStack Nova
Hardware database
Puppet
Active Directory /LDAP
Training and Support
• Buy the book rather than guru mentoring• Follow the mailing lists to learn• Newcomers are rapidly productive (and often know more than us)• Community and Enterprise support means we’re not on our own
OpenStack London December 2012 Tim Bell, CERN 21
Staff Motivation
• Skills valuable outside of CERN when an engineer’s contracts end
OpenStack London December 2012 Tim Bell, CERN 22
Prepare the move to the clouds• Improve operational efficiency
– Machine ordering, reception and testing– Hardware interventions with long running programs– Multiple operating system demand
• Improve resource efficiency– Exploit idle resources, especially waiting for disk and tape I/O– Highly variable load such as interactive or build machines
• Enable cloud architectures– Gradual migration to cloud interfaces and workflows– Autoscaling and scheduling
• Improve responsiveness– Self-Service with coffee break response time
OpenStack London December 2012 Tim Bell, CERN 23
Public Procurement Purchase ModelStep Time (Days) Elapsed (Days)
User expresses requirement 0
Market Survey prepared 15 15
Market Survey for possible vendors 30 45
Specifications prepared 15 60
Vendor responses 30 90
Test systems evaluated 30 120
Offers adjudicated 10 130
Finance committee 30 160
Hardware delivered 90 250
Burn in and acceptance 30 days typical 380 worst case
280
Total 280+ Days
OpenStack London December 2012 Tim Bell, CERN 24
Service Model
OpenStack London December 2012 Tim Bell, CERN 25
• Pets are given names like pussinboots.cern.ch
• They are unique, lovingly hand raised and cared for
• When they get ill, you nurse them back to health
• Cattle are given numbers like vm0042.cern.ch
• They are almost identical to other cattle• When they get ill, you get another one
• Future application architectures should use Cattle but Pets with strong configuration management are viable and still needed
Current Status of OpenStack at CERN• Working with Essex OpenStack release
– Excellent experience with Fedora/RedHat team using EPEL packages– Started Folsom test environment in November
• Focusing on the compute side to start with– Nova compute with KVM and Hyper-V– Keystone identity now integrated with Active Directory– Replaced network layer with CERN code for our legacy network
management system
• Current pre-production installation– 170 Hypervisors– 2,700 VMs– 3 DevOps part time running and enhancing the service– Running production ‘cattle’ workloads for stress testing
OpenStack London December 2012 Tim Bell, CERN 26
OpenStack London December 2012 Tim Bell, CERN 27
When communities combine…• OpenStack’s many components and options make
configuration complex out of the box• Puppet forge module from PuppetLabs does our configuration• The Foreman adds OpenStack provisioning for user kiosk to a
configured machine in 15 minutes
OpenStack London December 2012 Tim Bell, CERN 28
Foreman to manage Puppetized VM
OpenStack London December 2012 Tim Bell, CERN 29
Opportunistic Clouds in online experiment farms• The CERN experiments have farms of 1000s of Linux servers
close to the detectors to filter the 1PByte/s down to 6GByte/s to be recorded to tape
• When the accelerator is not running, these machines are currently idle
– Accelerator has regular maintenance slots of several days– Long Shutdown due from March 2013-November 2014
• One of the experiments have deployed OpenStack on their farm
– Simulation (low I/O, high CPU)– Analysis (high I/O, high CPU, high network)
OpenStack London December 2012 Tim Bell, CERN 30
Federated European Clouds
• Two significant European projects around Federated Clouds– European Grid Initiative Federated Cloud as a federation of grid sites providing
IaaS– HELiX Nebula European Union funded project to create a scientific cloud based
on commercial providers
OpenStack London December 2012 Tim Bell, CERN 31
EGI Federated Cloud Sites
CESGA CESNET INFN SARA
Cyfronet FZ Jülich SZTAKI IPHC
GRIF GRNET KTH Oxford
GWDG IGI TCD IN2P3
STFC
Ongoing Projects• Production Preparation
– Following High Availability recommendations from the community– Integrate monitoring with CERN frameworks
• Quota management with Boson– CERN does not have infinite compute resources or budget– Experiments are allocated a quota to split between their projects– Collaborating with the community to develop a distributed quota
manager
• Block Storage Evaluation– Investigating Gluster, NetApp and Ceph integration for EBS
functionality
OpenStack London December 2012 Tim Bell, CERN 32
Going further forward• Deployment to production
– Planned for Q1 2013 based on Folsom– No legacy tools in the second data centre
• Exploit new functionality– Ceilometer for metering– Bare metal for non-virtualised use cases such as high I/O servers– X.509 user certificate authentication– Load balancing as a service– Cells for scalability
Ramping to 15,000 hypervisors with 100,000 to 300,000 VMs by 2015 OpenStack London December 2012 Tim Bell, CERN 33
Final Thoughts
OpenStack London December 2012 Tim Bell, CERN 34
• A small project to share documents at CERN in the ‘90s created the massive phenomenon that is today’s world wide web
• Open Source• Vibrant community and eco-system
• Working with the Puppet and OpenStack communities has shown the power of collaboration
• We have built a toolchain in one year with part time resources
• Sharing with other organisations to achieve scale is the only economically feasible path
• CERN contributes and benefits from contributions of others
Questions ?
OpenStack London December 2012 Tim Bell, CERN 35
References
OpenStack London December 2012 Tim Bell, CERN 36
CERN http://public.web.cern.ch/public/Scientific Linux http://www.scientificlinux.org/Worldwide LHC Computing Grid http://wlcg.web.cern.ch/
Jobs http://cern.ch/jobsDetailed Report on Agile Infrastructure http://cern.ch/go/N8wpHELiX Nebula http://helix-nebula.eu/EGI Cloud Taskforce https://wiki.egi.eu/wiki/Fedcloud-tf
Backup Slides
OpenStack London December 2012 Tim Bell, CERN 37
OpenStack London December 2012 Tim Bell, CERN 38
CERN’s tools
• The world’s most powerful accelerator: LHC– A 27 km long tunnel filled with high-tech instruments– Equipped with thousands of superconducting magnets– Accelerates particles to energies never before obtained– Produces particle collisions creating microscopic “big bangs”
• Very large sophisticated detectors– Four experiments each the size of a cathedral– Hundred million measurement channels each– Data acquisition systems treating Petabytes per second
• Top level computing to distribute and analyse the data– A Computing Grid linking ~200 computer centres around the globe– Sufficient computing power and storage to handle 25 Petabytes per
year, making them available to thousands of physicists for analysis
OpenStack London December 2012 Tim Bell, CERN 39
Our Infrastructure
• Hardware is generally based on commodity, white-box servers– Open tendering process based on SpecInt/CHF, CHF/Watt and GB/CHF– Compute nodes typically dual processor, 2GB per core– Bulk storage on 24x2TB disk storage-in-a-box with a RAID card
• Vast majority of servers run Scientific Linux, developed by Fermilab and CERN, based on Redhat Enterprise
– Focus is on stability in view of the number of centres on the WLCG
OpenStack London December 2012 Tim Bell, CERN 40
New architecture data flows
OpenStack London December 2012 Tim Bell, CERN 41
Virtualisation on SCVMM/Hyper-V
OpenStack London December 2012 Tim Bell, CERN 42Mar-
10
Apr-10
May-10
Jun-10Jul-1
0
Aug-10
Sep-10
Oct-10
Nov-10
Dec-10
Jan-11
Feb-11
Mar-11
Apr-11
May-11
Jun-11Jul-1
1
Aug-11
Sep-11
Oct-11
Nov-11
Dec-11
Jan-12
Feb-12
Mar-12
Apr-12
May-12
Jun-12Jul-1
2
Aug-12
Sep-12
Oct-12
0
500
1000
1500
2000
2500
3000
3500
WindowsLinux
Scaling up with Puppet and OpenStack
• Use LHC@Home based on BOINC for simulating magnetics guiding particles around the LHC
• Naturally, there is a puppet module puppet-boinc• 1000 VMs spun up to stress test the hypervisors with Puppet,
Foreman and OpenStack
OpenStack London December 2012 Tim Bell, CERN 43
Federated Cloud Commonalities
• Basic building blocks– Each site gives an IaaS endpoint with an API and common security
policy• OCCI? CDMI ? Libcloud ? Jclouds ?
– Image stores available across the sites – Federated identity management based on X.509 certificates– Consolidation of accounting information to validate pledges and usage
• Multiple cloud technologies– OpenStack– OpenNebula– Proprietary
OpenStack London December 2012 Tim Bell, CERN 44
Supporting the Pets with OpenStack
• Network– CERN specific component to interface to our legacy network
environment
• Configuration– Using Puppet with Puppetlabs modules for rapid deployment
• External Block Storage– Currently using nova-volume with Gluster backing store
• Live migration to maximise availability– KVM live migration using Gluster– KVM and Hyper-V block migration
OpenStack London December 2012 Tim Bell, CERN 45
Active Directory Integration• CERN’s Active Directory
– Unified identity management across the site– 44,000 users– 29,000 groups– 200 arrivals/departures per month
• Full integration with Active Directory via LDAP– Uses the OpenLDAP backend with some particular configuration
settings– Aim for minimal changes to Active Directory– 7 patches submitted around hard coded values and additional filtering
• Now in use in our pre-production instance– Map project roles (admins, members) to groups– Documentation in the OpenStack wiki
OpenStack London December 2012 Tim Bell, CERN 46
Welcome Back Hyper-V!
• We currently use Hyper-V/System Centre for our server consolidation activities
– But need to scale to 100x current installation size
• Choice of hypervisors should be tactical– Performance– Compatibility/Support with integration components– Image migration from legacy environments
• CERN is working closely with the Hyper-V OpenStack team and Microsoft to arrive at parity
– Puppet to configure hypervisors on Windows– Most functions work well but further work on Console, Ceilometer, …
OpenStack London December 2012 Tim Bell, CERN 47
Active Directory Integration• CERN’s Active Directory
– Unified identity management across the site– 44,000 users– 29,000 groups– 200 arrivals/departures per month
• Full integration with Active Directory via LDAP– Uses the OpenLDAP backend with some particular configuration
settings– Aim for minimal changes to Active Directory– 7 patches submitted around hard coded values and additional filtering
• Now in use in our pre-production instance– Map project roles (admins, members) to groups– Documentation in the OpenStack wiki
OpenStack London December 2012 Tim Bell, CERN 48
OpenStack London December 2012 Tim Bell, CERN 49