anatomy of a reusable module
DESCRIPTION
A presentation at PuppetConf 2013 about modules reusability, naming standards and stacks.TRANSCRIPT
Anatomy of aReusable Module
Alessandro Franceschigithub.com/example42
PuppetConf 2013
Be����������� ������������������ patientplease
How����������� ������������������ dowe����������� ������������������ usePuppettoday?
Include classes
Set Params & Variables
DefineBusiness &
Integration Logic
ProvideConfiguration Files
ManageResources
ENC
How do we use Puppet today
Include classesmanifests/
site.pp
Set Parameters / Variables
Integration logic
Resources
ENC HIERA
SITE MODULES
SHARED MODULES
BAD
EDG
EConfiguration files
manifests/site.pp ENC HIERA
manifests/site.pp
BAD
?
BAD
?
BAD
SITE MODULES
SITE MODULES
SHARED MODULES
SITE MODULES
SHARED MODULES
manifests/site.pp
BAD
?
ENC
BAD
?
A����������� ������������������ reusable����������� ������������������ module
is����������� ������������������ all����������� ������������������ aboutCHOICE
Operating Systems
Infrastructures
Scales
Node classifiers
Installation methods
Alternative setups
The����������� ������������������ consof����������� ������������������ a
reusablemodule
Harder & LongerDevelopment
Complexity
Verbosity
Not Optimized for performance
THE PARAMETERSDILEMMA
Managed resources attributesApplication specific config options
Application logic and behaviourIntegration with other modules
Parameters: Resources attributesEnough: $package = $redis::params::package, $service = $redis::params::service, $service_ensure = 'running', $service_enable = true, $file = $redis::params::file, $file_notify = "Service['redis']", $file_source = undef, $file_content = undef,
Too much? $package_provider = undef, $file_owner = $redis::params::file_owner, $file_group = $redis::params::file_group, $file_mode = $redis::params::file_mode, $file_replace = $redis::params::file_replace,
Benefits from: A standard naming convention
Parameters: Application optionsEnough: $puppet_server = “puppet.${::domain}”, $syslog_server = “syslog.${::domain}”, $munin_server = “munin.${::domain}”, $dns_servers = [ '8.8.8.8' , '8.8.4.4' ],
Too much! $anonymous_enable = true, $anon_mkdir_write_enable = true, $anon_upload_enable = false, $chroot_list_enable = true, $chroot_list_file = '/etc/vsftpd/chroot_list',
$resourcefile = $nagios::params::resourcefile, $statusfile = $nagios::params::statusfile, $commandfile = $nagios::params::commandfile, $resultpath = $nagios::params::resultpath, $retentionfile = $nagios::params::retentionfile, $p1file = $nagios::params::p1file,
Benefits from: Template + Options Hash pattern
Parameters: Application logicExamples: $install_client = true, $install_stomp_server = true, $install_plugins = true, $use_ssl = false, $munin_autoconfigure = true, $service_autorestart = true, $manage_package_repo = true, $run_initdb = undef,
Benefits from: A standard naming convention
Parameters: Modules IntegrationsExamples: $mongo_db_host = $graylog2::params::mongo_db_host, $mongo_db_port = $graylog2::params::mongo_db_port, $mongo_db_name = $graylog2::params::mongo_db_name, $mongo_user = $graylog2::params::mongo_user, $mongo_password = $graylog2::params::mongo_password, $elasticsearch_template = $graylog2::params::elasticsearch_template, $elasticsearch_path = $graylog2::params::elasticsearch_path,
$database = $puppetdb::params::database, $manage_redhat_firewall = $puppetdb::params::manage_redhat_firewall, $db_type = 'mysql',
Benefits from: Shared Stacks
PATTERNSREUSABILITY
Managing����������� ������������������ files
Let user decide how to manage
configuration files.
Alternatives:sourcecontentconcataugeas
custom types
Managing files: source & contentredis/manifests/init.ppclass redis ( $file = $redis::params::file, $file_source = undef, $file_template = undef, $file_content = undef, ) {
[...]
$managed_file_content = $file_content ? { undef => $file_template ? { undef => undef, default => template($file_template), }, default => $file_content, }
[...]
if $redis::file { file { 'redis.conf': path => $redis::file, source => $redis::file_source, content => $redis::managed_file_content, } }}
Provide the Puppet path of an erb templateclass { ‘redis’: file_template => ‘site/redis/redis.conf.erb’,}
Provide directly the content attributeclass { ‘redis’: file_content => “template(‘site/redis/redis.conf.erb’)”,}
Provide a fileserver source pathclass { ‘redis’: file_source => ‘puppet:///modules/site/redis/redis.conf’,}
Manage the configuration file with other methods(augeas, concat...)class { ‘redis’: }
Multiple����������� ������������������ config����������� ������������������ files����������� ������������������
Add parametersto main class
Use a genericconf define
Manage the whole configuration dir
Multiple files: Add parameterselasticsearch/manifests/init.ppclass elasticsearch (
$file = $elasticsearch::params::file, $file_source = undef, $file_template = undef, $file_content = undef,[...]
$init_script_file = '/etc/init.d/elasticsearch', $init_script_file_template = 'elasticsearch/init.erb',
$init_options_file = $elasticsearch::params::init_options_file, $init_options_file_template = 'elasticsearch/init_options.erb',
Provide custom templates for the main file and the init scriptclass { ‘elasticsearch’: file_template => ‘site/elasticsearch/elasticsearch.yml.erb’, init_script_file_template => ‘site/elasticsearch/elasticsearch.init.erb’,}
Multiple files: Generic conf definenova/manifests/conf.ppdefine nova::conf ( $source = undef, $template = undef, $content = undef, $path = undef,[...] $options_hash = undef, $ensure = present ) {
include nova $managed_path = $path ? { undef => "${nova::config_dir}/${name}", default => $path, }[...] file { "nova_conf_${name}": ensure => $ensure, source => $source, content => $managed_content, path => $managed_path, mode => $managed_mode, owner => $managed_owner, group => $managed_group, require => $managed_require, notify => $managed_notify, replace => $managed_replace, }}
Provide a custom template for an alternative config file in config_dirnova::conf { ‘rootwrap.conf’: template => ‘site/nova/rootwrap.conf.erb’,}
Multiple files: Whole config dirredis/manifests/init.ppclass redis ( $dir = $redis::params::dir, $dir_source = undef, $dir_purge = false, $dir_recurse = true, ) {[...]
$dir_ensure = $ensure ? { 'absent' => 'absent', 'present' => 'directory', }
if $redis::dir_source { file { 'redis.dir': ensure => $redis::dir_ensure, path => $redis::dir, source => $redis::dir_source, recurse => $redis::dir_recurse, purge => $redis::dir_purge, force => $redis::dir_purge, notify => $redis::file_notify, require => $redis::file_require, } }
}
Provide a custom source for the whole config_dirclass { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’,}
Provide a custom source for the whole config_dir and purge any not managed config fileclass { ‘redis’: dir_source => ‘puppet:///modules/site/redis/conf/’, dir_purge => true,}
Managing����������� ������������������ Users
Everyone has his own users...
Leave options to decide
if, how and where to manage the ones
the module requires.
Managing Userselasticsearch/manifests/init.ppclass elasticsearch { $ensure = 'present',[...] $user = 'elasticsearch', $user_uid = undef, $user_gid = undef, $user_groups = undef, $user_class = 'elasticsearch::user',[...] if $elasticsearch::user_class { require $elasticsearch::user_class }
elasticsearch/manifests/user.ppclass elasticsearch::user { @user { $elasticsearch::user : ensure => $elasticsearch::ensure, comment => "${elasticsearch::user} user", password => '!', managehome => false, uid => $elasticsearch::user_uid, gid => $elasticsearch::user_gid, groups => $elasticsearch::user_groups, shell => '/bin/bash', } User <| title == $elasticsearch::user |>}
Do not create the requested userclass { ‘elasticsearch’: user_class => undef,}
Provide the user in a different custom classclass { ‘elasticsearch’: user_class => 'site::users',}
Run elasticsearch with a different userclass { ‘elasticsearch’: user => 'apache',}
Managingextra����������� ������������������
resources
Options to specify custom classes
Options to passan hash to
create_resources
Extra Resources: Custom classeselasticsearch/manifests/init.ppclass elasticsearch (
$dependency_class = 'elasticsearch::dependency', $monitor_class = 'elasticsearch::monitor', $firewall_class = 'elasticsearch::firewall', $my_class = undef,
) {
[...]
if $elasticsearch::dependency_class { include $elasticsearch::dependency_class }
if $elasticsearch::monitor and $elasticsearch::monitor_class { include $elasticsearch::monitor_class }
if $elasticsearch::firewall and $elasticsearch::firewall_class { include $elasticsearch::firewall_class }
if $elasticsearch::my_class { include $elasticsearch::my_class }[...]
Provide the modules dependencies with a custom classclass { ‘elasticsearch’: dependency_class => 'site::dep_elasticsearch',}
Extra Resources: Resources Hashelasticsearch/manifests/init.ppclass elasticsearch ( $create_resource = undef, $resources_hash = undef,
) {
[...]
if $create_resource { create_resources( $create_resource , $resources_hash ) }
Alternative: A single hash that includes resources and resources_hash
Provide the modules dependencies with a custom classclass { ‘elasticsearch’: create_resource => 'file', resources_hash => { path => '/etc/elasticsearch/my_file', content => template('site/elasticsearch/my_file.erb), mode => '0600', },}
ManagingPackages
andServices
Names change
Custom packagesare common
Leave choice,optionally
Managing packagesopenssh/manifests/init.ppclass openssh (
$ensure = 'present', $version = undef, $package = $openssh::params::package,[...] ) {
if $version and $ensure == 'present' { $managed_package_ensure = $version } else { $managed_package_ensure = $ensure }
if $openssh::package { package { $openssh::package: ensure => $openssh::managed_package_ensure, } }
openssh/manifests/params.ppclass openssh::params {
$package = $::osfamily ? { Suse => 'openssh', OpenBSD => '', default => 'openssh-server', }
Install a custom company-openssh packageclass { ‘openssh’: package => 'company-openssh',}
Managing servicesopenssh/manifests/init.ppclass openssh ( $service = $openssh::params::service, $service_ensure = 'running', $service_enable = true,[...] ) { if $ensure == 'absent' { $managed_service_enable = undef $managed_service_ensure = stopped } else { $managed_service_enable = $service_enable $managed_service_ensure = $service_ensure }
if $openssh::service { service { $openssh::service: ensure => $openssh::managed_service_ensure, enable => $openssh::managed_service_enable, } }
openssh/manifests/params.ppclass openssh::params {
$service = $::osfamily ? { Debian => 'ssh', default => 'sshd', }[...]
Manage a custom company-openssh serviceclass { ‘openssh’: service => 'company-openssh',}
ManagingInstallation����������� ������������������ options
Let users decide:
OS Packages
Upstream tarballs
Provider
Installation optionselasticsearch/manifests/init.ppclass elasticsearch ( $package_provider = undef,
$install = 'package', $install_base_url = $elasticsearch::params::install_base_url, $install_source = undef, $install_destination = '/opt', ) {[...] $managed_file = $elasticsearch::install ? { package => $elasticsearch::file, default => "${elasticsearch::home_dir}/config/elasticsearch.yml", }[...]case $elasticsearch::install { package: { package { $elasticsearch::package: ensure => $elasticsearch::managed_package_ensure, provider => $elasticsearch::package_provider, } } upstream: { puppi::netinstall { 'netinstall_elasticsearch': url => $elasticsearch::managed_install_source, destination_dir => $elasticsearch::install_destination, owner => $elasticsearch::user, group => $elasticsearch::user, }[...]
Install elasticsearch from upstream sourceclass { ‘elasticsearch’: install => 'upstream', install_source => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.3.zip',}
Templatesand����������� ������������������ hashes
Managing specific application configs
parameters may get out of control
A single config hashto show them all
A custom templateto use them
Application specific configs
THE PARAMETERSDILEMMA
Options Hash: Setupopenssh/manifests/init.ppclass openssh ([...] $file_template = undef, $options_hash = undef,
site/templates/openssh/sshd_config.erb# File Managed by Puppet[...] Port <%= scope.function_options_lookup(['Port','22']) %> PermitRootLogin <%= scope.function_options_lookup(['PermitRootLogin','yes']) %> UsePAM <%= scope.function_options_lookup(['UsePAM','yes']) %>[...]
* Function options_lookup currently in Example42's Puppi module
Alternative site/templates/openssh/sshd_config.erb Port <%= scope.lookupvar('openssh::options_hash')['Port'] ||='22' %> PermitRootLogin <%= scope.lookupvar('openssh::options_hash')['PermitRootLogin'] ||='yes' %> UsePAM <%= scope.lookupvar('openssh::options_hash')['UsePAM'] ||='yes' %>[...]
Options Hash: UsageUsage (with Hiera):include openssh
/etc/puppet/hieradata/global.yml:---openssh::file_template: 'site/openssh/sshd_config.erb'openssh::file_options_hash: Port: '22222' PermitRootLogin: 'no'
Usage (with parametrized class):class { 'openssh': file_template => 'site/openssh/sshd_config.erb' file_options_hash => { Port => '22222', PermitRootLogin => 'no', }
STANDARDSNAMING
Managed resources attributes
THE PARAMETERSDILEMMA
TheHandy����������� ������������������ Grailof����������� ������������������ Modules����������� ������������������ Standards
A blog post*
Some discussions on Puppet-Users
github.com/stdmod
Naming standardsfor modules parameters
Community driven
(draft 0.0.2)
* http://www.example42.com/?q=The_handy_Grail_of_Modules_Standards
Benefits����������� ������������������ of����������� ������������������ suggested
(and����������� ������������������ shared)naming����������� ������������������
conventions
Saner User Experience
Better modules Interoperability
Reusability Patterns
Predictability in usage and
development
Stdmod Params: Main resources### General parametersensure (enable?) version (package_version?)
### Package - Service - Main configuration filepackage (package_name?)package_ensurepackage_providerpackage_* [any relevant package type attribute]
service (service_name?)service_ensureservice_enableservice_subscribeservice_*
file (file_path? config_file? config?)file_source (source? config_file_source? config_source?)file_template (template? config_file_template? config_template?)file_content (content? config_file_content? config_content?)file_* (config_file_*? config_*?)
file_options_hash (options? options_hash? file_options?)
Stdmod Params: Extra resourcesother_packageother_package_*
client_packageclient_package_*
server_packageserver_package_*
other_serviceother_service_*
log_filelog_file_*
pid_filepid_file_*
init_script_fileinit_script_file_*init_config_fileinit_config_file_*
Stdmod Params: Installation ### Parameter related parameters
installinstall_urlinstall_base_urlinstall_sourceinstall_destinationinstall_pre_execinstall_pre_exec_*install_post_execinstall_post_exec_*
install_script_fileinstall_script_file_*install_response_fileinstall_response_file_*
Stdmod Params: Monitormonitormonitor_toolmonitor_hostmonitor_portmonitor_protocolmonitor_urlmonitor_processmonitor_servicemonitor_config_hash
Stdmod Params: Firewallfirewallfirewall_srcfirewall_dstfirewall_portfirewall_protocol
STACKSMODULES
Integrations of modules
THE PARAMETERSDILEMMA
Why����������� ������������������ reinventing����������� ������������������ the����������� ������������������ stack?
We always use stacks.
We need them to make something
useful with modules.
What about:Sharing?
Best practices?Standardization?
Stacks - A Simple Sample class stack::logs (
$ensure = 'present',
$syslog_server = false, $syslog_server_port = '5544',
$elasticsearch_server = false, $elasticsearch_server_port = '9200', $elasticsearch_cluster = 'logs', $elasticsearch_java_opts = '-Xmx2g -Xms1g',
$install_logstash = false, $install_elasticsearch = false, $install_kibana = false,
$install_graylog2 = false, $install_graylog2_webinterface = false,
$syslog_config_template = 'stack/logs/syslog.conf.erb', $logstash_config_template = 'stack/logs/logstash.conf.erb', $elasticsearch_config_template = 'stack/logs/elasticsearch.yml.erb', $kibana_config_template = 'stack/logs/config.js.erb', $graylog2_config_template = 'stack/logs/graylog2.conf.erb',
) {
[... TO BE CONTINUED ...]
Stacks - A Simple Sample[...] if $syslog_server { rsyslog::config { 'logstash_stack': content => template($syslog_config_template), } }
if $install_logstash { class { 'logstash': template => $logstash_config_template, } }
if $install_elasticsearch { class { 'elasticsearch': java_opts => $elasticsearch_java_opts, template => $elasticsearch_config_template, } }
[...]
Stacks - UsageOn any host:stack::logs { 'central': syslog_server => 'syslog.example42.com',}
On the Logstash (syslog) server:stack::logs { 'central': syslog_server => 'syslog.example42.com', install_logstash => true, elasticsearch_server => 'el.example42.com',}
On the Elasticsearch server(s), with a custom configuration file:stack::logs { 'central': syslog_server => 'syslog.example42.com', install_elasticsearch => true, elasticsearch_server => 'el.example42.com', elasticsearch_config_template => 'site/logs/elasticsearch.yml.erb',}
On the Kibana server:stack::logs { 'central': syslog_server => 'syslog.example42.com', install_kibana => true, elasticsearch_server => 'el.example42.com',}
TheStacksLogic
Stacks are localModules are shared
Higher level interface
Integrate different set of modules
Preserve moduleslocal change
ENC
How do we use Puppet today
Include classesmanifests/
site.pp
Set Parameters / Variables
Integration logic
Resources
ENC HIERA
SITE MODULES
SHARED MODULES
BAD
EDG
EConfiguration files
manifests/site.pp ENC HIERA
manifests/site.pp
BAD
?
BAD
?
BAD
SITE MODULES
SITE MODULES
SHARED MODULES
SITE MODULES
SHARED MODULES
manifests/site.pp
BAD
?
ENC
BAD
?
STACKS
STACKS
STACKS
Standardsfor
Stacks?
Usual benefits:
User Experience
Interoperability
Higher level API exposure
Possible GUI Integrations
Steps
Define stdmod naming conventions
Explore Stacks design and approach
Create templatesfor stacks and
modules
ExploreGUI integrations
SO����������� ������������������ Long and thanksfor all the fish!
Graphics: www.tatlin.net
@alvagante
SO����������� ������������������ Long and thanksfor all the fish!
Graphics: www.tatlin.net
@alvagante