webinar - managing files with puppet

OlinData Webinar 2014 - http://bit.ly/olindata-webinar-puppet-files

Managing Fileswith Puppet

An introduction to configuration management


Who am I?

• Walter Heck, Software engineer turned DBA, turned Sysadmin, turned entrepreneur

• Founder of OlinData (http://www.olindata.com) o Puppet Labs training partner for all of Asia and part of

Europeo Node.JS, OpenStack, Linux Foundation trainingo MySQL consulting

http://www.olindata.com


Overview

• What is puppet (for those not aware)?

• Which modules to use for file management?

• Built-in puppet resources for working with files

• How to deal with cross-resource type conflicts

• Exported resources and the concat module

• Questions


What is Puppet and why do we care?

• Configuration management software

• Scales well (1-200k+ nodes)

• Multi-platform (windows, *nix, Mac OS, BSD)

• Commercially supported Open Source

• Infrastructure as code


1. source and content attributes of the file resource type

2. the concat module

3. the file_line resource type from the stdlib module

4. the inifile module

5. the augeas tool

What options do we have?


The file resource type• used to manage whole files (and directories and symlinks)

• content of the file is managed in two mutually exclusive ways:o source => ‘puppet:///modules/apache/httpd.conf’o content => template(apache/httpd.conf.erb’)

• in heavy or large environments, split off file serving to a separate puppetmaster


The file resource type - source

• Source attribute can use o local file on the agent

• syntax: source => ‘/usr/local/myfile.txt’

o remote file on the master• syntax: source => ‘puppet://<puppetmaster>/modules/apache/httpd.conf’

• source attribute copies content from source non-modifiableo great for static files eg. init.d scripts


$master: cat mymodule/manifests/bar.pp

class mymodule::bar {file { '/tmp/file01':ensure => file,owner => 'root',group => 'root',mode => '0644',source => 'puppet:///modules/mymodule/file01',

}file { '/tmp/file02':ensure => file,owner => 'root',group => 'root',mode => '0644',source => '/opt/local/file02',

}}

$master: cat mymodule/files/file01

This is a static file$agent: cat /opt/local/file02

This is a locally sourced static file

The file resource (source example)

$agent: puppet apply -e ‘include mymodule::bar’

$agent: cat /tmp/file01

This is a static file$agent: cat /tmp/file02

This is a locally sourced static file


The file resource type - content

• Content attribute assigns a static string to the content of the file

o use the template() function to parse an erb template and assign the result.

• eg. content => template(‘apache/httpd.conf’)

• erb templates: standard ruby embedded templates• https://docs.puppetlabs.com/guides/templating.html

https://docs.puppetlabs.com/guides/templating.html


$ cat mymodule/manifests/foo.pp class mymodule::foo {

$say_hello_to = 'dude'$myname = 'file03'

file { "/tmp/$myname":ensure => file,content => template('mymodule/polite-file.erb'),

}}

$ cat mymodule/templates/polite-file.erb <% if @say_hello_to -%>Hello <%= @say_hello_to %>,<% end -%>I'm <%= @myname %>, on a <%= @operatingsystem %> system, nice to meet you.

$ puppet apply -e ‘include mymodule::foo’

$ cat /tmp/file03Hello dude,I'm file03, on a Ubuntu system, nice to meet you.

Note: the @operatingsystem variable (replaced here by “Ubuntu”) value is provided by Facter

The file resource (content example)


The concat module

• puppet module developed by R.I.Pienaar originally in 2010o adopted by PuppetLabs:

https://github.com/puppetlabs/puppetlabs-concat• Allows to concatenate separate sections of files

o eg. puppet.conf on your puppet master• Uses a local nifty trick with a shell script

o check out: files/concatfragments.sh• Bonus: use exported resources on fragments to collect for

instance haproxy loadbalancer backends

https://github.com/puppetlabs/puppetlabs-concat


The concat module (example)class motd {$motd = '/etc/motd'

concat { $motd:owner => 'root',group => 'root',mode => '0644'

}

concat::fragment{ 'motd_header':target => $motd,content => "\nPuppet modules on this server:\n\n",order => '01'

}

# local users on the machine can append to motd by just creating# /etc/motd.localconcat::fragment{ 'motd_local':target => $motd,source => '/etc/motd.local',order => '15'

}}

#Note: concat::fragment resources for a single target file can appear anywhere in your repository, across modules and classes


The stdlib module - file_line

• used for managing single lines in a file you otherwise don’t care about

o supports regex matchingo caution: managing the same file with file_line and a file

resource can cause flip-flopping file content• part of the puppetlabs/stdlib module

o https://github.com/puppetlabs/puppetlabs-stdlib• implemented as custom resource type

o lib/puppet/provider/file_line/ruby.rb


class foo::bar {

file_line { 'sudo_rule':path => '/etc/sudoers',line => '%sudo ALL=(ALL) ALL',

}

file_line { 'change_mount':path => '/etc/fstab',line => '10.0.0.1:/vol/data /opt/data nfs defaults 0 0',match => '^172.16.17.2:/vol/old',

}

}

file_line (example)


The inifile module

• used for managing inifile style fileso created by Chris Price, maintained by PuppetLabso https://github.com/puppetlabs/puppetlabs-inifile

• most important resource type is ini_settingo Auto-creates ini section headers

https://github.com/puppetlabs/puppetlabs-inifile


class foo::baz {ini_setting { "sample setting":

ensure => present,path => '/tmp/foo.ini',section => 'foo',setting => 'foosetting',value => 'FOO!',

}}

$ puppet apply -e ‘include foo::baz’

$ cat /tmp/foo.ini[foo]foosetting = FOO!

The inifile module - example


Augeas

• Advanced parsing of config fileso http://augeas.net/

• Uses so-called lenses that define the grammar of known config files. Many available for well known software (http://augeas.net/stock_lenses.html)

o Creating your own lense possible, but tough!• After parsing, a configuration file will be available in tree

format, with read/write capabilities to each node in the tree• Works regardless of order of lines in the file

http://augeas.net/

http://augeas.net/stock_lenses.html


walter@web02:~$ cat -n /etc/hosts1 # HEADER: This file was autogenerated at Sun May 12 07:31:47 +0200 20132 # HEADER: by puppet. While it can still be managed manually, it3 # HEADER: is definitely not recommended.4 ### OlinData installimage5 # nameserver config6 # IPv47 127.0.0.1 localhost8 1.2.3.4 web02.olindata.com9 1.2.3.5 db01.olindata.com10 1.2.3.6 test01.olindata.com11 192.168.100.100 db0112 192.168.100.101 web0213 1.2.3.7 mail01.olindata.com puppet[..SNIP..]

Augeas (example)walter@web02:~$ augtoolaugtool> ls /files/etc/hosts/7ipaddr = 1.2.3.7canonical = mail01.olindata.comalias = puppetaugtool> ls /files/etc/hosts/07augtool> ls /files/etc/hosts/7ipaddr = 1.2.3.7canonical = mail01.olindata.comalias = puppetaugtool> match /files/etc/hosts/*/ipaddr 1.2.3.4/files/etc/hosts/2/ipaddraugtool> print /files/etc/hosts/*/ipaddr/files/etc/hosts/1/ipaddr = "127.0.0.1"/files/etc/hosts/2/ipaddr = "1.2.3.4"/files/etc/hosts/3/ipaddr = "1.2.3.5"/files/etc/hosts/4/ipaddr = "1.2.3.6"/files/etc/hosts/5/ipaddr = "192.168.100.100"/files/etc/hosts/6/ipaddr = "192.168.100.101"/files/etc/hosts/7/ipaddr = "1.2.3.7"


root@web02 /home/walter # cat test.ppclass test {

file { '/tmp/motd':ensure => present,content => "foo \nbar \n"

}

file_line { 'test line':path => '/tmp/motd',line => 'baz',match => '^ba.',

}}

include testroot@web02 /home/walter # puppet apply test.ppNotice: Compiled catalog for web02.olindata.com in environment dev in 0.06 secondsNotice: /Stage[main]/Test/File[/tmp/motd]/ensure: createdNotice: /Stage[main]/Test/File_line[test line]/ensure: createdNotice: Finished catalog run in 0.71 seconds

Trivia - what happens?


root@web02 /home/walter # cat test.ppclass test {

file { '/tmp/motd':ensure => present,content => "foo \nbar \n"

}

file_line { 'test line':path => '/tmp/motd',line => 'baz',match => '^ba.',

}}

include testroot@web02 /home/walter # puppet apply test.ppNotice: Compiled catalog for web02.olindata.com in environment dev in 0.06 secondsNotice: /Stage[main]/Test/File[/tmp/motd]/ensure: createdNotice: /Stage[main]/Test/File_line[test line]/ensure: createdNotice: Finished catalog run in 0.71 seconds

Trivia - what happens?

root@web02 /home/walter # cat /tmp/motdfoobaz


Exported resources

PuppetDB

Puppet

Master

DB01WEB01

@@concat::fragment{ ‘test’:

ensure => present,

target => ‘/etc/motd’

}

concat {‘/etc/motd’:

ensure => present,

}

Concat::Fragment <<| |>>

1. Puppet agent run

6. Send to node

5. Retrieve from PuppetDB3. Store in PuppetDB

2. Export to PM

4. Collect on DB01


Exported resourcesnode /^web\d{3}.olindata.vm$/ {

include role::haproxy::backend

}

node 'proxy01.olindata.vm' {

include role::haproxy

}

class role::haproxy {

concat {‘/etc/haproxy.cfg’:

ensure => present,

}

Concat::Fragment <<| |>>

}

class role::haproxy::backend {

@@concat::fragment { $::fqdn:

ensure => 'present',

target => '/etc/haproxy.cfg'

}

}


Upcoming training

• Puppet Fundamentals Training, Vienna

Monday, November 17, 2014

• Puppet Fundamentals Training, Barcelona


• Puppet Fundamentals Training, Hyderabad


• Puppet Fundamentals Training, Pune

Monday, December 1, 2014

• Puppet Architect Training, Singapore

Wednesday, December 17, 2014

http://olindata.com/training/austria/vienna/puppet-fundamentals-training-vienna-november-2014

http://olindata.com/training/spain/barcelona/puppet-fundamentals-training-barcelona-november-2014

http://olindata.com/training/india/hyderabad/puppet-fundamentals-training-hyderabad-november-2014

http://olindata.com/training/india/sangamvadi/puppet-fundamentals-training-pune-december-2014

http://olindata.com/training/singapore/puppet-architect-training-singapore-december-2014


We’re hiring!EU and Asia based trainers

[email protected]


Questions?

@walterheck / @olindata


[email protected]

http://github.com/olindata


mailto:[email protected]

http://github.com/olindata

webinar - managing files with puppet

Technology