deploying e.l.k stack w puppet
TRANSCRIPT
![Page 1: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/1.jpg)
Deploying
E.L.K.
with Puppets
![Page 3: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/3.jpg)
The ELK Stack - What is it ?
ElasticSearch….for Storage, Indexing & Search
Logstash... For Logs & Filtering
Kibana…. for DataViz & this guy
![Page 4: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/4.jpg)
What you’ll need….
![Page 5: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/5.jpg)
What You’ll Also Need...
Load
Balancer
![Page 6: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/6.jpg)
These too….
elastic/puppet-elasticsearch
elastic/puppet-logstash
puppetlabs/puppetlabs-vcsrepo
puppetlabs/puppetlabs-git
puppetlabs/puppetlabs-concat
puppetlabs/puppetlabs-stdlib
![Page 7: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/7.jpg)
1st Prep a Base Image
Save yourself some headache and just prep an
empty image that sets puppet master in
/etc/hosts
[ec2-user@ip-172-30-0-118 ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain
172.30.0.41 puppet
![Page 8: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/8.jpg)
Prepare your nodes...
Use at minimum a medium instance for
the elasticsearch nodes...
![Page 9: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/9.jpg)
Prep your Load Balancer
![Page 10: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/10.jpg)
![Page 11: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/11.jpg)
The ElasticSearch Confignode 'ip-172-30-0-189.ec2.internal', 'ip-172-30-0-190.ec2.internal','ip-172-30-0-160.ec2.internal','ip-172-30-0-159.ec2.internal','ip-172-30-0-4.ec2.internal' {
class { 'elasticsearch':
ensure => 'present',
package_url => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.3.noarch.rpm',
java_install => true,
config => {
'cluster.name' => 'cluster-name-goeshere-cluster',
'cloud.aws.access_key' => ‘SDFDSGGSDSDGFSRSGsgfse’,
'cloud.aws.secret_key' => ‘WhaTEVerUrKEYHaPp3n5t0B3ItWoodG0h3R3’,
'cloud.aws.region' => 'us-east',
'cloud.node.auto_attributes' => true,
'discovery.type' => 'ec2',
'discovery.ec2.tag.name' => 'elasticsearch',
'discovery.ec2.groups' => 'sg-0d6aaa69',
'http.port' => '9200',
'http.enabled' => true,
…….
![Page 12: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/12.jpg)
package_url => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.3.noarch.rpm',
java_install => true,
config => {
'cluster.name' => 'Frederick-Von-Clusterberg',
'cloud.aws.access_key' => ‘SDFDSGGSDSDGFSRSGsgfse’,
'cloud.aws.secret_key' => ‘WhaTEVerUrKEYHaPp3n5t0B3ItWoodG0h3R3’,
'cloud.aws.region' => 'us-east',
'cloud.node.auto_attributes' => true,
'discovery.type' => 'ec2',
'discovery.ec2.tag.name' => 'elasticsearch',
'discovery.ec2.groups' => 'sg-0d6aaa69',
'http.port' => '9200',
'http.enabled' => true,
'http.cors.enabled' => true,
'http.cors.allow-origin' => 'http://54.152.82.147',
'path.data' => '/opt/elasticsearch/data',
'discovery.zen.ping.multicast.enabled' => false,
'discovery.zen.ping.unicast.hosts' => ["172.30.0.189", "172.30.0.190","172.30.0.159","172.30.0.160","172.30.0.4"],
}
}
exec{'export ES_HEAP_SIZ=2g':}
The ElasticSearch Package you
want to use
Give your cluster a name
![Page 13: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/13.jpg)
'cloud.node.auto_attributes' => true,
'discovery.type' => 'ec2',
'discovery.ec2.tag.name' => 'elasticsearch',
'discovery.ec2.groups' => 'sg-0d6aaa69',
tag your elasticsearch instances the SAME
groups are your security group ID’s
![Page 14: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/14.jpg)
Node Discovery...
'discovery.type' => 'ec2',
![Page 15: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/15.jpg)
Except it Doesn’t work.'discovery.type' => 'ec2',
'http.port' => '9200',
'http.enabled' => true,
'http.cors.enabled' => true,
'http.cors.allow-origin' => 'http://54.152.82.147',
'path.data' => '/opt/elasticsearch/data',
'discovery.zen.ping.multicast.enabled' => false,
'discovery.zen.ping.unicast.hosts' => ["172.30.0.189", "172.30.0.190","172.30.0.159","172.30.0.160","172.30.0.4"],
}
}
![Page 16: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/16.jpg)
CORS… you needs it
'http.cors.enabled' => true,
'http.cors.allow-origin' => 'http://my.kibanabox.whatevs',
Otherwise this happens...
![Page 17: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/17.jpg)
Make your Heap Size Bigger
exec{'export ES_HEAP_SIZ=2g':}
The default is 1GB of
Memory, but apparently
ElasticSearch needs 2GB
![Page 18: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/18.jpg)
You need to declare an instance!!!!
elasticsearch::instance { 'es1': }
![Page 19: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/19.jpg)
Now add some Plugins!!
elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1':
module_dir => 'cloud-aws',
instances => ['es1'],
}
elasticsearch::plugin { 'mobz/elasticsearch-head':
module_dir => 'head',
instances => ['es1'],
}
elasticsearch::plugin { 'lmenezes/elasticsearch-kopf':
module_dir => 'kopf',
instances => ['es1'],
}
elasticsearch::plugin { 'lukas-vlcek/bigdesk':
module_dir => 'bigdesk',
instances => ['es1'],
}
}
And Make
Sure to add
your instance
Name
![Page 20: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/20.jpg)
We’re almost done...
![Page 21: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/21.jpg)
Not Really….
That was just the ElasticSearch Part.
![Page 22: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/22.jpg)
Logstash
raw logs go inpretty formatted logs
come out
![Page 23: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/23.jpg)
Now for Logstash...node 'ip-172-30-0-144.ec2.internal' {
class { 'logstash':
ensure => 'present',
package_url => 'https://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-1.4.2-
1_2c0f5a1.noarch.rpm',
install_contrib => true,
contrib_package_url => 'https://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-contrib-1.4.2-
1_efd53ef.noarch.rpm',
java_install => true,
exec{ ‘openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt -
days 365’}
}
logstash::configfile { ‘somename’:
content => template('files/logstash.conf')
}
}
![Page 24: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/24.jpg)
the Logstash config file
input {
lumberjack {
# The port to listen on
port => 1234
# The paths to your ssl cert and key
ssl_certificate => "/etc/pki/logstash-forwarder.crt"
ssl_key => "/etc/pki/logstash-forwarder.key"
# Set this to whatever you want.
type => "apache-access"
}
}
this is called logstash-forwarder now, but in
logstash config its still called lumberjack...just
so you know.
![Page 25: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/25.jpg)
the Logstash config file
input {
lumberjack {
# The port to listen on
port => 1234
# The paths to your ssl cert and key
ssl_certificate => "/etc/pki/logstash-forwarder.crt"
ssl_key => "/etc/pki/logstash-forwarder.key"
# Set this to whatever you want.
type => "apache-access"
}
}
These need to be placed on the
servers sending the logs !
![Page 26: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/26.jpg)
Filters….
filter {
grok {
type => "apache-access"
match => { message => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => clientip
}
}
![Page 27: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/27.jpg)
Outputs...
output {
elasticsearch { host => 'LoadBalancer.us-east-1.elb.amazonaws.com'
cluster => 'Frederick-Von-Clusterberg'
protocol => 'http'
}
Send logs to your Load Balancer
make sure to give it the cluster name...or don’t, you
didn’t really need those logs anyway.
Set The “elasticsearch” output
![Page 28: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/28.jpg)
Are we there yet ?
![Page 29: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/29.jpg)
And Now for Kibana….
elastic doesn’t provide a kibana module
![Page 30: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/30.jpg)
so use this guy’sechocat/puppet-kibana4
it does the job.
![Page 31: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/31.jpg)
the only config value you need is….
class kibana4 (
$version = '4.0.0-linux-x64',
$download_path = 'http://download.elasticsearch.org/kibana/kibana',
$install_dir = '/opt',
$running = true,
$enabled = true,
$port = 5601,
$host = '0.0.0.0',
$elasticsearch_url = 'http://your.fancy.loadbalancerurl:9200',
$elasticsearch_preserve_host = true,
$kibana_index = '.kibana',
$kibana_elasticsearch_username = '',
$kibana_elasticsearch_password = '',
$default_app_id = 'discover',
$request_timeout = 300000,
$shard_timeout = 0,
$verify_ssl = true,
$ca = '',
$ssl_key_file = '',
$ssl_cert_file = '',
$pid_file = '/var/run/kibana.pid',
$bundled_plugin_ids = [
This one
right here
![Page 32: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/32.jpg)
And Now You Have an ELK Stack!
![Page 33: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/33.jpg)
You Still have to configure your Log Shipper
![Page 34: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/34.jpg)
you need to prepare a few things
![Page 35: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/35.jpg)
like go, the keys you made earlier, logstash
forwarder...
{
"network": {
"servers": [ "ip-172-30-0-144:1234" ],
"ssl key":"/root/.logstash/logstash-forwarder.key",
"ssl ca": "/root/.logstash/logstash-forwarder.crt",
"timeout": 120
},
"files": [
{
"paths": [
"/home/logdir/access*[^.][^g][^z]"
],
"start_position": "beginning",
"fields": { "type": "apache-access" }
}
]
}
![Page 36: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/36.jpg)
Just Use This.
elastic/logstash-forwarder
![Page 37: Deploying E.L.K stack w Puppet](https://reader033.vdocuments.net/reader033/viewer/2022042716/55ac5ce41a28ab72108b4574/html5/thumbnails/37.jpg)
Thanks !