hbasecon 2012 | orchestrating clusters with ironfan and chef - runa
DESCRIPTION
This session will discuss how you can represent your complete cluster with one config file and have it deployed to Cloud or Bare Metal. Infochmimps’ Ironfan builds on Opscode Chef to allow you to specify and orchestrate all flavors of your cluster’s deployment, monitoring and growth. Not just the core HBase/HDFS/MapReduce/Hive/Flume, etc. but all the elements including web / app servers, mysql, redis, rabbitmq and whatever other servers needed to implement your service. These same tools can manage variations for development, staging, R&D as well as the target “rendering” to various Clouds, Bare Metal or even Vagrant VMs.TRANSCRIPT
![Page 2: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/2.jpg)
Hassles of Big Data Stack Deployments
• Lots of Moving Parts
• Hadoop/HBase just one sub-system
• Heterogeneous Tech
• Monitoring & Metrics Everywhere
• Details obscure the big picture
• Need repeatability & variations on themes
![Page 3: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/3.jpg)
The Forest for the Trees
![Page 4: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/4.jpg)
Forest
SessionStore HBase MySQL
Dashboard
App Servers Monitor
Hadoop M/RAMQPElastic
Search
![Page 5: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/5.jpg)
Trees
Web
Rails A
ppR
ails App
ELB WebW
eb
Clj A
pp
Clj A
ppC
lj App
Web
Graphite
Ganglia
LogstashSensu
Statsd
GD
ash
ES Server
ES ServerES Server
ES Server
Rabbit
Rabbit
RedisR
edis
Redis
ZooKeeper
ZooKeeper
ZooKeeper
HB
Master
HB
Master
RegionsrvrR
egionsrvr
Regionsrvr
Regionsrvr
Regionsrvr
Regionsrvr
Regionsrvr
Regionsrvr
Regionsrvr
Regionsrvr
Regionsrvr
Master
Sec Master
SlaveSlave
Slave
SlaveSlave
SlaveSlave
Slave SlaveSlave
MasterSlave
Slave
Dashboard App Servers Monitoring
ElasticSearch
AMQP HadoopM/R
SessionStore
HBaseMySQL
![Page 6: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/6.jpg)
LeavesDashboard App Servers Monitoring
Elastic Search AMQP Hadoop M/R
Session Store HBase MySQL
NginxReverse ProxyUnicornRailsDashboard AppJavaPostfixCron jobsSensu clientSensu plugins
Elastic Load BalancerNginxReverse ProxySwarmijiJavaLeiningenJarkClojure AppsHBase ClientPostfixCron jobsSensu clientSensu plugins
MySQL ClientUpstart ConfigLogstash Client
MySQL ClientUpstart ConfigLogstash Client
NginxReverse ProxyJavaLeiningenJarkPostfixCron jobsSensu ServerSensu WebSensu ClientSensu Plugins
Ganglia ServerGanglia WebStatsd ServerGraphite ServerGrpahite WebPythonLogstashMySQL ClientUpstart ConfigLogstash Client
Elastic Search ServerJavaCron jobsSensu clientSensu pluginsUpstart Config
RabbitMQRabbitMQ PluginsCluster ConfigErlangCron jobsSensu clientSensu pluginsUpstart Config
NamenodeSecondary NamenodeTasktrackerJobtrackersBootstrap NamenmodeJavaJMXCron jobsSensu clientSensu pluginsGanglia ClientUpstart Config
NamenodeSecondary NamenodeTasktrackerJobtrackersDatanodesBootstrap NamenmodeJavaJMXCron jobsSensu clientSensu pluginsGanglia ClientUpstart Config
RedisCron jobsSensu clientSensu pluginsUpstart Config
ZookeeperHBase MasterRegionserver
MySQL MasterMySQL SlavesCluster SetupCron jobsSensu clientSensu pluginsUpstart Config
![Page 7: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/7.jpg)
Molecules<configuration> <property> <name>hbase.rootdir</name> <value>hdfs://ip-10-17-57-58.ec2.internal:8020/hadoop/hbase</value> <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR </description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) </description> </property> <property> <name>hbase.zookeeper.quorum</name> <value>master0-cluster0.runa.com,regionserver0-cluster0.runa.com,regionserver1-cluster0.runa.com</value> <description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which we will start/stop ZooKeeper on. </description> </property>
![Page 8: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/8.jpg)
Config Management:Leaves & Molecules
• Chef, Puppet, Cfengine
• Much better than shell scripts or cli jocks
• Infrastructure as code
• Still No Forest Perspective
![Page 9: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/9.jpg)
Ironfan: Forest, Trees, Leaves and Molecules
• Builds on top of Chef
• Cluster Description in Single File
• Your System Diagram Come to Life
• Components announce capabilities
• Service Discovery automates interconnects
• Knife (CLI) extension controls cluster and component life-cycles
![Page 10: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/10.jpg)
Community + OwnCookbooks
VMs and/or ServersRunning Chef-Client
Chef Development Host
N
N
N
N
Basic Chef
Chef Knife:Upload Cookboks to Chef Server
Launch Instances / Bootstrap Servers
Chef Server
Roles / Cookbooks Data BagsNodes
Attributes
Auth & ACLsSearch
![Page 11: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/11.jpg)
NodesAttributes
Search
Chef + IronfanCommunity + Own
Cookbooks
VMs and/or ServersRunning Chef-Client
Chef Development Host
Ironfan Pantry +
Chef Knife + Ironfan Gem:Launch / Bootstrap / Manage Whole Clusters, All Facets or
Specific Instances
N
N
N
N
Chef Server
Roles / Cookbooks Data Bags
NodesAttributesDiscovery
Auth & ACLs
Search / Discovery
![Page 12: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/12.jpg)
Ironfan Components• Ironfan Gem:
• Knife Plugins to orchestrate clusters• Logic to sync Chef Server & Cloud[s]
• Silverware: Coordinate Discovery of Services
• Ironfan-Homebase: Ironfan tuned Chef-Repo
• Ironfan-Pantry: Cookbooks tuned for Clusters
• Ironfan-CI: Testing of Ironfan clusters and Cookbooks
![Page 13: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/13.jpg)
Cluster Config: Forest View
facet 'master' do instances 1 cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge"
role "big_package" role "hadoop_master" role "hbase_master" recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" end
GlobalCloud & Cluster
Configs
Facets are the “Trees”
Roles & Recipesare the “Leaves”
ClusterChef.cluster 'base0-cluster0' do setup_role_implications cloud :ec2 do region 'us-east-1' availability_zones ['us-east-1b'] backing 'ebs' image_name 'natty' security_group(cluster_root) do authorize_port_range(22) end end
role "base_role" role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" role "ebs_volumes_mount"
![Page 14: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/14.jpg)
Global Cloud & Recipe ConfgsClusterChef.cluster 'base0-cluster0' do setup_role_implications cloud :ec2 do region 'us-east-1' availability_zones ['us-east-1b'] backing 'ebs' image_name 'natty' security_group(cluster_root) do authorize_port_range(22) end end
role "base_role" role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" role "ebs_volumes_mount"
Cluster Name
CloudConfigs
SharedRoles
Configure SecurityGroup
![Page 15: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/15.jpg)
Facets add Specifics facet 'master' do instances 1 cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge"
role "hadoop_master" role "hbase_master" recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" end
facet 'regionserver' do instances 7 cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge"
role "hadoop_slave" role "hbase_regionserver" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" server 0 do role "zookeeper_server" end end
Facet Name
Facet Name
Number ofCopies
Number ofCopies
CloudOverrides
FacetRoles &Recipes
FacetRoles &Recipes
CloudOverrides
Make oneinstancespecial
![Page 16: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/16.jpg)
Facets Composed of Components
• Components are Services
• Nginx, MySQL server, Zookeeper, HBMaster, Namenode, etc.
• Chef Cookbooks manage components
• Ironfan Silverware for service discovery
• Auto-Connects components together
![Page 17: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/17.jpg)
Silverware Service Discovery
hbase_config = Mash.new({ :namenode_fqdn => discover(:hadoop, :namenode ).private_hostname), :jobtracker_addr => discover(:hadoop, :jobtracker).private_ip), :zookeeper_addrs => discover_all(:zookeeper, :server).map(&:private_ip).sort, :ganglia => discover(:ganglia, :server), :ganglia_addr => discover(:ganglia, :server).private_hostname), :private_ip => private_ip_of(node) })
announce(:hadoop, :namenode)• Recipe that creates a service , announces it
• Recipe that requires a service , discovers it
![Page 18: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/18.jpg)
Aspects enable Zeroconf Amenities
• A log aspect would enable the following amenities
• logrotated to manage its logs
• flume to archive logs to a location
• A port aspect would enable
• Configuration of firewall
• Monitoring of port uptime & latency
• Remote checks that firewalled ports do NOT respond
![Page 19: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/19.jpg)
Knife Cluster: Lifecycle Management
• A Plugin for Opscode Chef Knife
• Deployment & Lifecycle Operations:• launch, bootstrap, kill, start, stop
• Access and Chef Operations• ssh, kick, proxy
• Utilities• show, sync
![Page 20: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/20.jpg)
Knife Launch Cluster, Facet or Instance[s]
• Launch all the nodes in a clusterknife cluster launch base0-master0
• Launch just a single instance of a single facetknife cluster launch base0-master0 master 0
• Launch all the instances of a facetknife cluster launch base0-master0 regionserver
![Page 21: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/21.jpg)
Stop/Start Cluster, Facet or Instance[s]
• Stop whole clusterknife cluster stop base0-master0
• Stop a single instance of a single facetknife cluster stop base0-master0 master 0
• Stop all instances of a facetknife cluster stop base0-master0 regionserver
![Page 22: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/22.jpg)
Same Knife Command to Launch Vagrant[s]
• Can use the same cluster configurations and knife command to launch Vagrantsknife cluster vagrant up base0-master0
• Still Experimental
![Page 23: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/23.jpg)
Ironfan-CI
• Jenkins based Continuous Integration of Clusters
• Still Experimental
• Uses Discovery to automate baseline test creation
• Leverages Vagrant to create clean test environments
![Page 24: HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef - Runa](https://reader033.vdocuments.net/reader033/viewer/2022052900/5560b7bfd8b42afe3b8b4ba2/html5/thumbnails/24.jpg)
References
• Basic Chef Stuff:http://wiki.opscode.com/display/chef/Home
• Ironfan Screencast:http://vimeo.com/37279372
• Ironfan Wiki for the most complete info : https://github.com/infochimps-labs/ironfan/wiki
• The Forest for the Trees Photo - Ame Otokohttp://www.flickr.com/photos/ameotoko/5383225925/