![Page 1: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/1.jpg)
www.nimbusproject.org
Infrastructure Clouds for Science and
Education: Platform Tools
1 11/12/2012
Kate Keahey, John Bresnahan, Patrick Armstrong, Pierre Riteau
Argonne National Laboratory
University of Chicago
![Page 2: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/2.jpg)
www.nimbusproject.org
The Power of Infrastructure Clouds
Virtualization opens the flood gates
11/12/2012 2
• Outsourcing
• Virtual appliances
– Freeze your stack in time
– Run it anywhere
• Multi-cloud applications
– Run many copies all over the world!
• Elasticity
![Page 3: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/3.jpg)
www.nimbusproject.org
Harnessing The Power
• Organization tools and techniques
11/12/2012 3
![Page 4: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/4.jpg)
www.nimbusproject.org
What Needs To Be Harnessed
• VM (appliance) creation and development
– configuration management tools (chef, puppet)
• VM hypervisors
– Infrastructure-as-a-Service (IaaS)
• Cloud applications
– virtual clusters, cloudinit.d, CloudFormation
• Elasticity
– Auto-scaling tools, phantom
11/12/2012 4
![Page 5: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/5.jpg)
www.nimbusproject.org
What Needs To Be Organized?
• VM (appliance) creation and development
– configuration management tools (chef, puppet)
• VM hypervisors
– Infrastructure-as-a-Service (IaaS)
• Cloud applications
– virtual clusters, cloudinit.d, CloudFormation,
• Elasticity
– Auto-scaling tools, phantom
11/12/2012 5
![Page 6: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/6.jpg)
www.nimbusproject.org
VM Applications
• An entire system frozen in time
– Full software stacks (versions)
– Configuration files
– Important for science!
• A dedicated modular service
– Web service, database, AMQP node, etc
• Demos
• A binary single file (or set of files)
– Easy to freeze
11/12/2012 6
![Page 7: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/7.jpg)
www.nimbusproject.org
Developing Appliances
• A single binary image?
– Many developers?
– Version control?
– Merging conflicts?
• Base image with a description
– Ex: Ubuntu 11.04 base images plus a set of scripts
• Configuration Management Software
– Chef, Puppet, FG Rain, etc
11/12/2012 7
![Page 8: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/8.jpg)
www.nimbusproject.org
• Software stack description – ruby and json
• A library of cookbooks
• Cookbooks contain recipes – Ex: apache2 server with php4
• Attributes to customize each recipe – Ex: on what port will apache listen
• Templates for configuration files
• Appliance developers make recipes – Version control can be done with git/svn/cvs…
11/12/2012 8
Chef
![Page 9: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/9.jpg)
www.nimbusproject.org
Example Recipe
11/12/2012 9
app_dir = node[:appdir] ve_dir = node[:virtualenv][:path] git app_dir do repository node[:autoscale][:git_repo] reference node[:autoscale][:git_branch] action :sync user node[:username] group node[:groupname] end execute "run install" do cwd app_dir user node[:username] group node[:groupname] command "python setup.py install" end
![Page 10: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/10.jpg)
www.nimbusproject.org
Example Template
11/12/2012 10
phantom: system: type: epu rabbit: <%= node[:autoscale][:rabbit_host] %> rabbit_port: <%= node[:autoscale][:rabbit_port] %> rabbit_ssl: False rabbit_user: <%= node[:autoscale][:rabbit_username] %> rabbit_pw: <%= node[:autoscale][:rabbit_password] %> rabbit_exchange: <%= node[:autoscale][:rabbit_exchange] %> authz: type: sqldb dburl: <%= node[:autoscale][:dburl] %>
phantom: system: type: epu rabbit: vm-102.uc.futuregrid.org rabbit_port: 5672 rabbit_ssl: False rabbit_user: XXX rabbit_pw: PPPPPP rabbit_exchange: default_dashi_exchange authz: type: sqldb dburl: mysql://nimbus:[email protected]/testphantom
![Page 11: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/11.jpg)
www.nimbusproject.org
What Needs To Be Organized?
• VM (appliance) creation and development
– configuration management tools (chef, puppet)
• VM hypervisors
– Infrastructure-as-a-Service (IaaS)
• Cloud applications
– virtual clusters, cloudinit.d, CloudFormation,
• Elasticity
– Auto-scaling tools, phantom
11/12/2012 11
![Page 12: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/12.jpg)
www.nimbusproject.org
Cloud Applications Virtual Clusters
11/12/2012 12
![Page 13: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/13.jpg)
www.nimbusproject.org
Linux Clusters
• A cheap answer to Super Computers
– Many “commodity” machines interconnected to operate as single machine
• Load distributed across nodes
• Compute clusters
• A single head node with N worker nodes
11/12/2012 13
Head
Worker
Worker
Worker
Worker
Worker
Worker
![Page 14: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/14.jpg)
www.nimbusproject.org
Virtual Clusters
• The same thing
– composed of VMs in a cloud
• Advantages
– A stable, pre-fabricated workspace
– Elastic
– repeatable
• Disadvantages
– Unknown hardware conditions
• Network, noisy neighbors, etc.
11/12/2012 14
Head
Worker
Worker
Worker
Worker
Worker
Worker
![Page 15: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/15.jpg)
www.nimbusproject.org
Nimbus One-Click Clusters
• Virtual Clusters In Nimbus Clouds
• Turnkey, tightly-coupled cluster
– Shared trust/security context
– Shared configuration/context information
• Sample images
– NFS file systems, torque queues, GridFTP servers, etc…
• Easily repeatable and distributable
– An xml file and a VM image
11/12/2012 15
![Page 16: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/16.jpg)
www.nimbusproject.org
Nimbus One-Click Clusters Context
• Personalize a VM instance
– seed them with secrets, access policies, and just-in-time configurations
– Populate /etc/hosts with cluster member addresses
– Set up SSH host-based authentication across all accounts
• Must run a light weight script (context agent) on boot.
– Context broker/agent discussed in detail later
11/12/2012 16
![Page 17: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/17.jpg)
www.nimbusproject.org
Other Cloud Applications
11/12/2012 17
![Page 18: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/18.jpg)
www.nimbusproject.org
Cloud Applications
• More than 1 VM needed for the job
• Information exchange is needed
– Manual information exchange
• Multi-cloud
– Cloud independence required
11/12/2012 18
Web Server database
Web Web Web Server
nginx
Web Servers
![Page 19: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/19.jpg)
www.nimbusproject.org
A Simplified Deployment Scenario
11/12/2012 19
![Page 20: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/20.jpg)
www.nimbusproject.org
cloudinit.d
• Multicloud VM dependency management – Uses the libcloud abstraction library
• Integrated with chef solo
• ini file format descriptions – Coupled with any executable script
• Launch plan end-users/operators – Lightweight
– Copy launch plan and “one click” action
– Easily reconfigured for various clouds
• Launch plan/application developers: – Minimal software assumptions (ssh)
– “Stem cell” deployment approach
– Incremental launch plan development
11/12/2012 20
[svc-alamoHTTP]
iaas_key: XXXXXX
iaas_secret: XXXX
iaas_host: alamo.futuregrid.org
iaas_port: 8443
iaas: Nimbus
image: ubunut10.10
ssh_username: ubuntu
localsshkeypath: ~/.ssh/fg.pem
readypgm: http-test.py
bootpgm: http-boot.sh
![Page 21: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/21.jpg)
www.nimbusproject.org
cloudinit.d Overview
• Services
• Run Levels
– Collections of
services without
dependencies on
each other
• Launch Plan
– An ordered set of
run levels
11/12/2012 21
![Page 22: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/22.jpg)
www.nimbusproject.org
cloudinit.d example
11/12/2012 22
Wordpress Web server
MySQL database
• Wordpress example
• 2 virtual machine running in EC2 – MySQL
– Wordpress
• MySQL contact information injected into wordpress
![Page 23: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/23.jpg)
www.nimbusproject.org 11/12/2012 23
![Page 24: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/24.jpg)
www.nimbusproject.org
What Needs To Be Organized?
• VM (appliance) creation and development
– configuration management tools (chef, puppet)
• VM hypervisors
– Infrastructure-as-a-Service (IaaS)
• Cloud applications
– virtual clusters, cloudinit.d, CloudFormation,
• Elasticity
– Auto-scaling tools, phantom
11/12/2012 24
![Page 25: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/25.jpg)
www.nimbusproject.org
Scaling Considerations
• Reasons to scale – Business vs science
• Cost vs quota
• Lossy environment – VMs fail more often than bare metal
– N preserving
• Spot instances – If the price is right
• Backfill – If resources are idle
11/12/2012 25
![Page 26: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/26.jpg)
www.nimbusproject.org
Infrastructure Platform Goals • Multi-cloud
– Work across private, community and commercial clouds
• Any Scale – Scale in response to a diverse set of sensors/triggers
– Both system and application sensors
• High Availability – “Any VM can die”: system or user VMs
– Minimizing time to recovery (TTR)
• Your Polices, Our Enactment – User-defined sensors/triggers and policies
• Engineered from the ground up to work with infrastructure clouds
• Easy on the user
11/12/2012 26
![Page 27: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/27.jpg)
www.nimbusproject.org
Phantom Architecture
11/12/2012 27
MySQL
nginx
REST HTTPS
Web Application HTTPS REST Service
Web Application
FutureGrid Clouds
RabbitMQ EPUM
Provisioner
DTRS
Zookeeper Cluster
REST Service REST Service IaaS
Clouds
![Page 28: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/28.jpg)
www.nimbusproject.org
How Can Science Plug Into This Power
Example Embarrassingly Parallel
Scientific Application
Demonstration
11/12/2012 28
![Page 29: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/29.jpg)
www.nimbusproject.org
…
M subtask messages
Task Queue
Application Start the workers
Using Nimbus Domains
![Page 30: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/30.jpg)
www.nimbusproject.org
…
Preserve N worker VMs
M subtask messages
Cumulus/S3
Message Queue
“N preserving” policy
Infrastructure Compute Cloud
Get task
Results/Checkpoints
Application Start the workers
Using Nimbus Domains
![Page 31: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/31.jpg)
www.nimbusproject.org 11/12/2012 31
![Page 32: Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure Clouds for Science and Education: Platform Tools 11/12/2012 1 Kate Keahey, John Bresnahan,](https://reader034.vdocuments.net/reader034/viewer/2022042419/5f363e3e6ad2bc189100b6d9/html5/thumbnails/32.jpg)
www.nimbusproject.org
One Click Cluster Exercise
• Examine the one click cluster definition
• Launch a virtual cluster
• Inspect the virtual cluster
11/12/2012 32
<?xml version="1.0" encoding="UTF-8"?> <cluster> <workspace> <name>head-node</name> <image>base-cluster-cc14.gz</image> <quantity>1</quantity> <nic wantlogin="true">public</nic> <ctx> <provides> <identity /> <role>nfsserver</role> </provides>
32