NoSQL and Big Data for DevOps

Gustavo Fernandes

Sunday, 3 February 13

Agenda

• DevOps?

• Toolset

• BigData

• NoSQL

• Demo

• Q&A

Sunday, 3 February 13

DevOps - Motivations

• Silos development/ops

• Slow release cycles

• Lack of awareness from either side

Sunday, 3 February 13

Developers

• Payed to add new features constantly

• “Works in my laptop”

• Usually IGNORE non-functional requirements

Sunday, 3 February 13

Operations

• Keep it stable

• Reliable

• Monitoring

• Distance from code

Sunday, 3 February 13

Devops - Enablers

• Agile

• Infrastructure as software

Sunday, 3 February 13

What is Devops?

• Development + Operations

• Discipline/Philosophy/Methodology

• Role - coder of non-functional requirements

• Faster, more reliable, continuous releases to production

Sunday, 3 February 13

Devops - Principles

• Automate everything: release, deployment, provision

• Infrastructure as code - TDD, tags, branches, ...

• Agile to ops

Sunday, 3 February 13

Open Source Toolset • Configuration management tool

• Puppet, chef

• Application Lifecycle management tools

• Build tools: Maven, Gradle, Buildr, SBT, Rake

• Maven repository: Nexus, Artifactory

• Provisioning tools

• Vagrant, Boxgrinder

• CI servers: jenkins

Sunday, 3 February 13

Puppet

• Custom Declarative Language• Describe resources and states• Applies states to servers• Standalone/client-server/pub-sub• Testable

Sunday, 3 February 13

Puppet Resourcesfile { "/my/file":

source => "/path/to/", backup => main, mode => “0644”}

cron { logrotate: command => "/usr/sbin/logrotate", user => root, hour => 2, minute => 0}

exec { "tar -xf /Volumes/nfs02/important.tar": cwd => "/var/tmp", creates => "/var/tmp/myfile", path => ["/usr/bin", "/usr/sbin"]}

user { 'opuser': ensure => 'present', password => '$1$9VC1vFFa$GHKWgtdODti8eKqkQ7Ruv.'}

Sunday, 3 February 13

Classesclass mongodb($replicaset = ‘’, $disablenuma = ‘’) {

$mongo_tgz = "mongodb-${arch}- ${version}.tgz"   $base_dir = "${base}"

 group { "mongodb":        "ensure => present" }" user{ "mongodb":         "ensure => present,          gid => "mongodb",        "shell => "/sbin/nologin" " }    file { "$base_dir":" " ensure => "directory"," " owner => "mongodb"," " group => "mongodb"," " alias => "mongo-base"" }}

Sunday, 3 February 13

File Serverfile { "/etc/sudoers": mode => 440, owner => root, group => root, source => "puppet:///modules/name/sudoers"}

file { "$installdir/conf/mongo.conf": mode => 0744, ensure => present, content => template('mongodb/mongodb.conf.erb'),}

Sunday, 3 February 13

ERB Templatesservers = [‘server1.domain’,‘server2.domain’,‘server3.domain’]

file {'/etc/foo.conf': ensure => file, content => template('foo/foo.conf.erb'),}

# foo.conf.erb<% servers.each do |server| -%> <%= server %><% end -%>

# foo.conf.erbserver1.domainserver2.domainserver3.domain

Sunday, 3 February 13

Modules

Static Files

ERB Templates

Module manifest

Sunday, 3 February 13

Site

node "box1.domain" { include java include hadoop_master}

site.pp

Sunday, 3 February 13

Manifest• Puppet entry point

• Nodes

node "box1.domain" { include java include hadoop_master class { 'myapp': version => "1.5-SNAPSHOT", maven_repo => "http://devserver:8081/nexus/content/repositories/snapshots/", }}

node “box2.domain” { include java include hadoop_slave}

Sunday, 3 February 13

Facter Inventory

• Companion ruby utility to puppet

• Collect facts about one environment and expose as a map

• Puppet uses it to decide what to deliver to a node

Sunday, 3 February 13

Facter - Example$ facterarchitecture => x86_64domain => domainfqdn => devserver.domainhardwaremodel => x86_64hostname => devserveripaddress_eth1 => 192.168.95.15ipaddress_lo => 127.0.0.1is_virtual => truekernel => Linuxmemorytotal => 491.11 MBnetwork_eth0 => 10.0.2.0network_eth1 => 192.168.95.0operatingsystem => OpenSuSEoperatingsysrelease => 12.2physicalprocessorcount => 1processor0 => Intel(R) Core(TM) i7-2677M CPU @ 1.80GHzprocessorcount => 1......

Sunday, 3 February 13

Using facter values

... <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value><%= $::processorcount %></value> </property>...

mapred-site.xml.erb

Sunday, 3 February 13

Puppet Extensions

• The language itself

• New types

• New functions

• New resources

Sunday, 3 February 13

The maven resource

maven { "download-artifact": groupid => "com.gustavonalle", artifactid => "$artifact", version => "1.0-SNAPSHOT", repos => "http://devserver:8081/repo/snapshots/", directory => "/opt/myapp",

classifier => “jar” require => File["basedir"], before => Exec["unzip"]}

Sunday, 3 February 13

Maven resource

• Support for SNAPSHOTS

• Support for Releases

• No need to install Maven in the server

• Support Authentication

• Support for http and https

• Support for puppet:// protocol

Sunday, 3 February 13

Installing Hadoop

Sunday, 3 February 13

Hadoop - Processes

• Name Node running on at least one node

• Secondary Name Name node running elsewhere

• Data Node process running on nodes who are part of HDFS

• Job Tracker running in the cluster

• Task Tracker running in each node that can execute map or reduce task

Sunday, 3 February 13

Hadoop HDFS• Name node knows all the slaves from

<HADOOP_HOME>/conf/slaves

• Slaves need to point at namenode in the file <HADOOP_HOME>/conf/core-site.xml

slave1.domain.comslave2.domain.com...

<property> <name>fs.default.name</name> <value>hdfs://namenode:9000</value></property>

Sunday, 3 February 13

Hadoop Map Reduce• Job Tracker knows all the slaves from

<HADOOP_HOME>/conf/slaves

• Slaves need to point at jobtracker in the file <HADOOP_HOME>/conf/mapredsite.xml

slave1.domain.comslave2.domain.com...

<property> <name>mapred.job.tracker</name> <value>master:9001</value> </property>

Sunday, 3 February 13

Hadoop - SSH

• SSH is required to do cluster-wide operations

• ‘ssh localhost’ without asking password

• su hadoop -c ‘ssh slave01’ without password

Sunday, 3 February 13

How puppet can help

• Facter calculates optimal values for memory, cpu, number of maps

• Generate ssh keys on the fly

• Obtain hostnames automatically

• Hide most of the complexity and expose only the bare minimal

• Install in parallel high number of slaves

Sunday, 3 February 13

MongoDB - Replicaset

PrimaryArbiter

Secondaries

(no data)

Sunday, 3 February 13

MongoDB - Creating cluster

• Replicaset must be done with all servers running

• Using cmd tools and a bit of javascript

Sunday, 3 February 13

How puppet can help

• Ensure all servers are running and configured

• Generate .js files and configuration files using templates

Sunday, 3 February 13

Demo

Jenkins

Nexus

Puppet Master

github.com/gustavonalle/puppet

Sunday, 3 February 13

Demo

Name Node

Mongo Primary

JVM

Mongo Secondary

Job Tracker

Data Node

Task Tracker M&R Job

Mongo Arbiter

JVM

box1.domain

box2.domain

devserver.domain** 2 Cpus* 1 Cpu

**

*

*Data Node Task Tracker

Sunday, 3 February 13

site.ppclass mongo_replicaset { class { 'mongodb': replicaset => 'fosdem', primary => 'box1.domain', secondaries => ['box2.domain'], arbiter => 'devserver.domain' }}class customApp { class { 'myapp': version => "1.0-SNAPSHOT", maven_repo => "http://devserver.domain:8081/repo/snapshots/", }}

Sunday, 3 February 13

site.pp (cont.)node "box1.domain" { include java class { 'hadoop': master => box1, slaves => [box1,box2] } include mongo_replicaset include customApp}

node "box2.domain" { include java include mongo_replicaset class { 'hadoop': master => box1, }}

Sunday, 3 February 13

Wrapping up

• Server side software is not getting any simpler

• But infrastructure is now “software”

• Devops is here to stay

Sunday, 3 February 13

Thank you

github.com/gustavonalle/puppet

gustavonalle@gmail.com

Sunday, 3 February 13