cloud foundry bosh where the platform rubber meets the infrastructure road - chefconf
Post on 06-May-2015
11.794 Views
Preview:
DESCRIPTION
TRANSCRIPT
Cloud Foundry BOSHWhere the Platform rubber
meets the Infrastructure road
Patrick ChanezonSenior DirectorDeveloper Relationschanezonp@vmware.comhttp://twitter.com/chanezon ChefConf, San Francisco, May 2012
with slides from Martin Englund
Cloud Foundry Engineering
@pmenglund
Wednesday, May 16, 12
What is Cloud Computing?
2
Cloud According to my daughter Eliette
Wednesday, May 16, 12
Cloud Stack - Classic Pyramid
3
Platform As A Service
Infrastructure As A Service
SoftwareAs A Service
Wednesday, May 16, 12
Cloud Stack - Developer Perspective
4
InfrastructureAs A Service
SoftwareAs A Service
Platform As A Service
Wednesday, May 16, 12
5
Infrastructure
Wednesday, May 16, 12
IaaS/Virtualization getting mainstream
§ AWS, Joyent, Rackspace,...
§ Open Source projects: OpenStack, DeltaCloud, Eucalyptus
§ Automation: Chef, Juju
§ Standardization? DMTF
§ Inside the Firewall, Virtualization: VMware, Microsoft, Xen, KVM
§ 50% of workloads are virtualized
§ Easy to provision, manage instance...BUT
§ Still need to manage backups, software stacks, monitor, upgrades
6
Wednesday, May 16, 12
With Infrastructure, you still need to build your own platform
§ Need to build a distributed platform on top of you infrastructure
§ Story of the AWS meltdown from last summer• http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html
• http://www.readwriteweb.com/cloud/2010/12/chaos-monkey-how-netflix-uses.php
• http://news.ycombinator.com/item?id=2477296
• http://stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html
§ Twilio, Smugmug, SimpleGeo survived it because they built their own distributed platform on top of IaaS
§ Enterprise customers want to consider Infrastructure like CDNs• Multi Cloud usage
• Based on Open Source de facto standards, or full standards whenever that happens
7
Wednesday, May 16, 12
8
Platforms
Let the wild rumpus begin!
Wednesday, May 16, 12
Platforms
§ Raise the Unit of currency to be application & services instead of infrastructure
§ Google App Engine, Cloud Foundry, SalesForce Heroku, Cloudbees, Amazon Elastic Beanstalk, Microsoft ex-Azure, AppFog
§ Single or a few languages, services
§ Start of Multi language Polyglot platforms
§ Enabler for Agile Developers -> Create Business value faster
§ Lack of standards: risk, vendor lock-in
§ Enterprise needs:•Control, customizability
•Private/Hybrid Cloud
•Avoid lock-in
9
Wednesday, May 16, 12
Main Risk: Lock-In
10
Welcome to the hotel californiaSuch a lovely placeSuch a lovely facePlenty of room at the hotel californiaAny time of year, you can find it here
Last thing I remember, I wasRunning for the doorI had to find the passage backTo the place I was before’relax,’ said the night man,We are programmed to receive.You can checkout any time you like,But you can never leave!
Wednesday, May 16, 12
Cloud Foundry: The Open PaaS
11
Clou
d Pr
ovide
r Int
erfa
ce
Application Service Private Clouds
PublicCloud
MicroCloud
Data Services
Other Services
Msg Services
vFabric Postgres
vFabric RabbitMQTM
• Open Source: Apache 2 Licensed
• multi language/frameworks
• multi services
• multi cloud
Wednesday, May 16, 12
12
Open Source
Wednesday, May 16, 12
Open Source Advantage: moves faster
§ http://code.google.com/p/googleappengine/issues/detail?id=13
13
• https://github.com/cloudfoundry/vcap/pull/25
Wednesday, May 16, 12
Open Source Advantage: diverse communities
14
§ Juju Charms to setup a multi node Cloud Foundry in 10 minutes
§ See Brian Thomason & Juan Negron’s UDS 2011 session
http://cloud.ubuntu.com/2011/09/from-zero-to-drawbridge-via-ubuntu-server-ensemble-and-cloudfoundry-in-less-than-10-minutes/
Wednesday, May 16, 12
Open Source Advantage: more innovation
15
Wednesday, May 16, 12
16
CloudFoundry.org
Wednesday, May 16, 12
17
Cloud Foundry Open Source Workflow
Reviewers +2/-2
test verification score ✓ ✗ git push triggers test execution
Gerrit Code Review
reviews.cloudfoundry.org
Jenkins CI
ci.cloudfoundry.org
github
github.com/cloudfoundry
+1/-1 Committers
on +2 and ✓ change pushed to github
cloudfoundry.com
production updated ~2x/week
Wednesday, May 16, 12
Cloud Foundry is 1 year old
18
§ Services• Caldecott – tunnel into your services, explore with standard client tools
• PostgreSQL, RabbitMQ
§ Frameworks, Runtimes, and Tools• Java and Ruby Auto-Reconfiguration
• Scala, node.JS 0.6.*, Erlang, JRuby, PHP, Python, .NET, Spring 3.1, Grails 2.0, Play 2.0
• Multi-Node Chef based deployment tools
• Maven Plugin, Eclipse Integration
• VMC manifests, Java Debugging, Rails Console
• Standalone applications
§ Micro Cloud Foundry x 3
Wednesday, May 16, 12
Cloud Foundry is 1 year old
18
§ Services• Caldecott – tunnel into your services, explore with standard client tools
• PostgreSQL, RabbitMQ
§ Frameworks, Runtimes, and Tools• Java and Ruby Auto-Reconfiguration
• Scala, node.JS 0.6.*, Erlang, JRuby, PHP, Python, .NET, Spring 3.1, Grails 2.0, Play 2.0
• Multi-Node Chef based deployment tools
• Maven Plugin, Eclipse Integration
• VMC manifests, Java Debugging, Rails Console
• Standalone applications
§ Micro Cloud Foundry x 3
Wednesday, May 16, 12
Cloud Foundry Ecosystem
19
Wednesday, May 16, 12
Cloud Foundry Ecosystem
20
Wednesday, May 16, 12
Register today
21
http://cloudfoundry.com/signup
ChefConfUse Promo Code
To avoid approval queue waiting time
Wednesday, May 16, 12
Logical View
22
VMC client STS plugin Browser(user app access)
Routers
CloudControllers App
Services
App
HealthManager
DEA Pool
Messaging
Wednesday, May 16, 12
23
Cloud Foundry BOSH
Wednesday, May 16, 12
24
Chef at VMware
§ Chef is used internally in several projects at VMware
§ To manage the Cloud Foundry deployment and Life Cycle, we developed a more specific system: BOSH
Wednesday, May 16, 12
25
§ 500 – 5,000 VMs
§ 40+ unique node types
§ 75+ unique software packages
§ 75+ unique environments
§ 2x/week cf.com updates
§ 24x7x365 non-stop operation
§ No-downtime deployments
§ Reliable, robust, repeatable deployments, updates, capacity adjustments
§ Small teams manage many instances
Production Grade Cloud Foundry Clusters
Google style problem è Google style solution
cloudfoundry.com
production, staging, stress, qa, dev
Wednesday, May 16, 12
26
§ Cloud Foundry BOSH is an open source tool-chain for release engineering, deployment, and lifecycle management of large scale distributed services• Prescriptive way of creating releases and managing systems and services
• It is not a collection of shell scripts, not a pile of Perl
§ Built to deploy and manage production-class, large scale clusters• Production grade Cloud Foundry clusters: 500+ VMs, 40+ jobs, 75+ packages
• Multi-node, multi-tier, complex clusters: e.g., our Gerrit/Jenkins Cluster
§ Built for devops usage and scale by a crack team of veterans• A project, not a product: command line interface, YAML, etc.
• Continuous improvement, iterative development, rough edges
Cloud Foundry BOSH
github.com/cloudfoundry/bosh
Wednesday, May 16, 12
27
§ Background
§ Concepts
§ Demo
§ Workflow
§ Under the hood
BOSH Agenda
Wednesday, May 16, 12
Background
§ Built from the need to operate cloudfoundry.com• Over 40 different types of VMs
• Hundreds of running instances of VMs
• Pushing at least 2 changes per week
§ End-to-end management• Release engineering
• Deployment
• Lifecycle management
§ Generic solution• Any service
• Any cloud
• Any operating system
28
Wednesday, May 16, 12
29
§ Packages
§ Jobs
§ Releases• http://github.com/cloudfoundry/bosh-sample-release
§ Stemcells
§ Deployments
Concepts
Wednesday, May 16, 12
30
§ Source / blobs• Don’t commit large objects in the SCM
§ Dependencies• Compile time dependencies
§ Versioning• Auto-increment
§ Pre-packaging• Preparation for compilation
§ Packaging• Compiling the source
Packages
Wednesday, May 16, 12
Packages (nginx)
31
src/nginx-rw-r--r-- 1 martin staff 25490 Apr 11 19:17 headers-more-v0.15rc1.tgz-rw-r--r-- 1 martin staff 691501 Apr 11 19:17 nginx-1.0.11.tar.gz-rw-r--r-- 1 martin staff 1556329 Apr 11 19:17 pcre-8.21.tar.gz
Wednesday, May 16, 12
Packages (nginx)
32
packages/nginx/spec---name: nginxfiles: - nginx/headers-more-v0.15rc1.tgz - nginx/nginx-1.0.11.tar.gz - nginx/pcre-8.21.tar.gz
Wednesday, May 16, 12
Packages (nginx)
33
packages/nginx/packaging# abort script on any command that exit with a non zero valueset -e
echo "Extracting pcre..."tar xzf nginx/pcre-8.21.tar.gz
echo "Extracting headers-more module..."tar xzf nginx/headers-more-v0.15rc1.tgz
echo "Extracting nginx..."tar xzf nginx/nginx-1.0.11.tar.gz
echo "Building nginx..."cd nginx-1.0.11./configure \ --prefix=${BOSH_INSTALL_TARGET} \ --with-pcre=../pcre-8.21 \ --with-http_ssl_module \ --add-module=../headers-more-v0.15rc1makemake install
Wednesday, May 16, 12
34
§ Packages• Runtime dependencies
§ Templates• Converting a generalized form to an instance specific file
§ Versioned• Auto increments
§ Supervision• Through monit
§ Monitoring• Reported to the health monitor
§ Lifecycle hooks• E.g. drain workload from job
Jobs
Wednesday, May 16, 12
Jobs (nginx)
35
jobs/nginx/spec---name: nginx
templates: nginx_ctl: bin/nginx_ctl nginx.conf.erb: config/nginx.conf mime.types: config/mime.types
packages: - nginx
Wednesday, May 16, 12
Jobs (nginx)
36
jobs/nginx/monitcheck process nginx with pidfile /var/vcap/sys/run/nginx/nginx.pid start program "/var/vcap/jobs/nginx/bin/nginx_ctl start" stop program "/var/vcap/jobs/nginx/bin/nginx_ctl stop" group vcap
Wednesday, May 16, 12
Jobs (nginx)
37
jobs/nginx/templates/nginx.conf.erbuser root root;worker_processes <%= properties.nginx.workers || 1 %>;
error_log /var/vcap/sys/log/nginx/nginx.error.log;pid /var/vcap/sys/run/nginx/nginx.pid;
events { worker_connections 8192; use epoll;}... upstream wordpress { <% properties.wordpress.servers.each do |server| %> server <%= server %>:<%= properties.wordpress.port %>; <% end %> }
server { listen 80; server_name "<%= properties.wordpress.servername %>"; server_name_in_redirect off;...
Wednesday, May 16, 12
38
§ Packages
§ Jobs
§ Versioned
§ Self contained
§ Incremental
§ Published
Releases
» bosh create release» bosh upload release» bosh delete release» bosh releases
Wednesday, May 16, 12
39
§ VM template
§ BOSH Agent
§ Versioned
§ Built using vmbuilder
§ IaaS Plugin
Stemcells
» bosh upload stemcell» bosh delete stemcell» bosh stemcells
Wednesday, May 16, 12
40
§ Release
§ Networks
§ Resource pools
§ Compilation workers
§ Jobs
§ Update concurrency
§ Properties
§ Cloud properties
Deployments
Wednesday, May 16, 12
Deployment (wordpress)
41
wordpress.yml---name: bosh-wordpressrelease: name: wordpress version: 3
compilation: workers: 4 network: default cloud_properties: ram: 2048 disk: 8096 cpu: 2
update: canaries: 1 canary_watch_time: 60000 update_watch_time: 30000 max_in_flight: 4 max_errors: 1
Wednesday, May 16, 12
Deployment (wordpress)
42
wordpress.yml (continued)networks: - name: default subnets: - reserved: - 172.31.220.2 - 172.31.220.10 static: - 172.31.220.11 - 172.31.220.100 range: 172.31.220.0/22 gateway: 172.31.220.1 dns: - 172.30.22.153 cloud_properties: name: VLAN2220 - name: dmz subnets: - static: - 172.28.5.209 - 172.28.5.210 range: 172.28.5.208/28 dns: - 172.22.22.153 cloud_properties: name: VLAN3093
Wednesday, May 16, 12
Deployment (wordpress)
43
wordpress.yml (continued)resource_pools: - name: infrastructure network: default size: 6 stemcell: name: bosh-stemcell version: 0.4.6 cloud_properties: cpu: 1 disk: 8192 ram: 4096
Wednesday, May 16, 12
Deployment (wordpress)
44
wordpress.yml (continued)jobs: - name: mysql template: mysql instances: 1 resource_pool: infrastructure persistent_disk: 16384 networks: - name: default static_ips: - 172.31.220.20
- name: wordpress template: wordpress instances: 4 resource_pool: infrastructure networks: - name: default static_ips: - 172.31.220.30 - 172.31.220.33
Wednesday, May 16, 12
Deployment (wordpress)
45
wordpress.yml (continued) - name: nginx template: nginx instances: 1 resource_pool: infrastructure networks: - name: default default: [dns, gateway] static_ips: - 172.31.220.40 - name: dmz static_ips: - 172.28.5.209
Wednesday, May 16, 12
Deployment (wordpress)
46
wordpress.yml (continued)properties: wordpress: admin: wordpress@cloudfoundry.com port: 8008 servers: - 172.31.220.30 - 172.31.220.31 - 172.31.220.32 - 172.31.220.33 servername: wordpress.cloudfoundry.com db: name: wp user: wordpress pass: w0rdpr3ss auth_key: random key secure_auth_key: random key logged_in_key: random key mysql: address: 172.31.220.20 port: 3306 password: rootpass nginx: workers: 1
Wednesday, May 16, 12
BOSH Demo
47
Wednesday, May 16, 12
Dev Workflow
48
bosh create release
bosh upload release
bosh deployment~/dev.yml
bosh deploy
write code
run tests
git commit
bosh target dev
Wednesday, May 16, 12
QA Workflow
49
bosh create release
bosh upload release
bosh deployment~/qa.yml
bosh deploy
git pull
run tests
bosh createrelease --final
bosh target qa
bug reports to dev
git commit
Wednesday, May 16, 12
Ops Workflow
50
bosh upload release
bosh deployment~/prod.yml
bosh deploy
git pull
run tests
bosh target prod
bug reports to dev
done!
Wednesday, May 16, 12
51
§ Repeatable deployments
§ Predictable rollout
Control
Wednesday, May 16, 12
52
§ Dev / Staging / Production
§ BOSH targets
Consistency
Wednesday, May 16, 12
53
§ Director / Workers
§ Agent
§ Monitor
§ CLI
§ NATS
§ Redis
§ Database
§ Blobstore
Components
Wednesday, May 16, 12
54
BOSH: under the hood
active jobs
diskIaaS CPI
workers
bosh cli
redis nats
stemcellagent
db
blobs
director healthmon
“BOSH is deployed by BOSH”
cloudfoundry.com
BOSH User
Wednesday, May 16, 12
55
IaaS neutral by design
CPI: code completefunctional status: “work in progress”
vSphere: battle tested implementation, thousands of deploymentsvCloud Director: “work in progress”, 2H 2012
contribute: github.com/cloudfoundry/bosh
Cloud Provider Interface (CPI)
Cloud Foundry BOSH
github.com/piston/openstack-bosh-cpi
Wednesday, May 16, 12
56
Stemcellcreate_stemcell(image, cloud_properties)delete_stemcell(stemcell)
VMcreate_vm(agent_id, stemcell, resource_pool, networks, disk_locality, env)delete_vm(vm)configure_networks(vm, networks)
Diskcreate_disk(size, vm_locality)delete_disk(disk)attach_disk(vm, disk)detach_disk(vm, disk)
Cloud Provider Interface
Wednesday, May 16, 12
57
BOSH and Chef?
§ How should they work together?
§ Talk to us on public mailing lists:• http://groups.google.com/a/cloudfoundry.org/group/bosh-users• http://groups.google.com/a/cloudfoundry.org/group/bosh-dev
Wednesday, May 16, 12
58
§ Mailing lists• http://groups.google.com/a/cloudfoundry.org/group/bosh-users• http://groups.google.com/a/cloudfoundry.org/group/bosh-dev
§ File a Bug• http://cloudfoundry.atlassian.net
§ Browse, Fork & Contribute• https://github.com/cloudfoundry/bosh• gem install gerrit-cli• gerrit clone ssh://reviews.cloudfoundry.org:29418/bosh
Reference / Q&A
Wednesday, May 16, 12
BOSH Community Resources
@DrNichttp://drnicwilliams.com/2012/04/16/creating-a-bosh-from-scratch-on-aws/
59
@BrianMMcClainhttp://www.brianmmcclain.com/2012/05/08/using-bosh-with-vsphere-part-2/
https://github.com/cloudfoundry/bosh
https://groups.google.com/a/cloudfoundry.org/group/bosh-users/topics
https://github.com/piston/openstack-bosh-cpi
Wednesday, May 16, 12
Acknowledgement
§ Drawings from my daughters Eliette
§ Slides from Mark Lucovsky, Dave McCrory, Derek Collison, Martin Englund
60
Wednesday, May 16, 12
top related