nagios conference 2014 - leland lammert - distributed heirarchical nagios

PowerPoint Presentation

Hierarchical/Distributed Nagios

Leland V. Lammert, PhD
Omnitec Corporation
St. Louis, MO

[email protected]@lvlammert

Introduction & Agenda

The problem:Multiple Remote Sites

System Status/Local Nagios

Central Dispatch for problems

The solution:SRM Hardware

Software Build

Network Issues

Nagios Configuration

Background

Specialists in Open Source Solutions

Core IT Infrastructure

Multi-platform Nagios

Leland V. Lammert, PhD
Chief Scientist

SRM Hardware

SRM Hardware

Raspberry Pi Bor

Raspberry Pi B+

R-Pi Camera

Temperature Probe

Cyntech case w/SD cover

SRM Hardware

SRM Hardware

Raspberry Pi Camera

SRM Hardware

DHT11
Temperature and Humidity

DS18B20
OneWire
Temperature Only

Temperature Probe Options

SRM Hardware

Camera

Temperature
Sensor

SD Card

Status
LEDs

Server Room Monitor [SRM]

Software Build

Software Build

Debian for Pi - Raspianhttp://www.raspbian.org/

Download from Raspberry Pi Projecthttp://www.raspberrypi.org/downloads/

Or, purchase a pre-installed SD

First task, update: aptitude update & upgrade

Software Build

Add packages for Nagios & admin:php5, php5-commin, php5-cli

(apache2-mpm conflicts, will be replaced)

chkconfig

telnet

libgd2-xpm-dev

heirloom-mailx (MUA)

ssmtp (MTA)

Software Build

Setup GPIO for the DS18B20Add to /etc/modules:w1_gpiow1_therm

Use modprobe to install manuallymodprobe [w1_gpio, w1_therm]

Software Build

Check the ID of the installed device:

ll /sys/bus/w1/devices//sys/bus/w1/devices/28-00000489b929/w1_slave
^^^^^^^^^^^^^^^^^^

ID on this device 28-00000489b929

Software Build

Create startup script - /root/SRMStart# Start the camera
raspistill -t 0 -tl 1500 -o \ /usr/local/nagios/share/tmp/snapshot.jpg &>/dev/null &

# Start the NSCA daemon
/usr/local/nagios/libexec/nsca -c /usr/local/nagios/etc/nsca.cfg daemon

# Setup tunnel if required
#/usr/bin/autossh -a -M 9005:1005 -N -R \ :127.0.0.1: [email protected] &

Software Build

Add SRMStart to /etc/rc.local:#
#Start the SRM components
#
/root/SRMstart

Software Build

Install Nagios NagiosCore

Download and build from Source./configure -with-gd-lib=/usr/local/lib/, make, make install

make install-init - Install the init script in /etc/init.d

make install-commandmode - Install and sets permissions on the directory for external commands

make install-config - Install sample config files in /usr/local/nagios/etc

Software Build

Install NSCA./configure && make

nsca deamon, send_nsca client

Copy binaries to /usr/local/nagios/libexec

Configs to usr/local/nagios/etc

Create password and add to both configs

Initial encryption XOR, select as appropriate

Software Build

Build a local copy of plugins./configure, make, make install

Installs to /usr/local/nagios/libexec

Verify operationcd /usr/local/nagios/libexec

./check_

Software Build

Other system pluginsBuild in that environmentLinux

Windows

Mac

Verify operation./check_ [in that environment]

Network Issues

Network Issues

Typical network configuration

?

Network Issues

An ssh tunnel would normally be required to traverse the firewall for
each remote host which is definitely
not ideal

Solution use NSCA!

By forwarding data via NSCA, firewall changes are eliminated, .. greatly reducing load on central system!

Network Issues

How does NSCA allow one Nagios system communicate with another?

By forwarding notifications to a remote system!Notifications for Services

Notifications for Hosts

Using a specific Contact

Only requires one open port, at the Central site!

Network Issues

Central Dispatch

Remote Site(s)

One open
port
required!



Nagios system communications

Central system has NSCA Server which listens for events, normally in 5667

Remote systems have NSCA Client which sends events to the Server on 5667

Notifications are encrypted according to the NSCA configuration


send_nsca Configurationpassword=fV5P0V8ggaFtNE

encryption_method=1

nsca daemon Configurationpassword=fV5P0V8ggaFtNE

encryption_method=1

1 = Simple XOR (Just obfuscation, but very fast)


Encryption Options 0 = None 1 = Simple XOR 2 = DES
3 = 3DES (Triple DES) 4 = CAST-128 5 = CAST-256
6 = xTEA 7 = 3WAY 8 = BLOWFISH
9 = TWOFISH10 = LOKI9711 = RC2
12 = ARCFOUR14 = RIJNDAEL-128
15 = RIJNDAEL-19216 = RIJNDAEL-256
19 = WAKE20 = SERPENT
22 = ENIGMA (Unix crypt)23 = GOST24 = SAFER64
25 = SAFER12826 = SAFER+


define command {
command_name notify-service-by-nsca
command_line /usr/bin/printf
"%s\t%s\t%s\t%s\n"
"$HOSTNAME$" "$SERVICEDESC$"
$SERVICESTATEID$" "$SERVICEOUTPUT$ |
$SERVICEPERFDATA$" | tee -a
/tmp/service_alert.log | /usr/local/nagios/libexec/send_nsca -H
$CONTACTADDRESS1$ -c
/usr/local/nagios/etc/send_nsca.cfg
}


define command {
command_name notify-host-by-nsca
command_line /usr/bin/printf "%s\t%s\t%s\n" \
"$HOSTNAME$" "$HOSTSTATEID$" \
"$HOSTOUTPUT$" | /usr/sbin/send_nsca -H
$CONTACTADDRESS1$ -c
/usr/local/nagios/etc/send_nsca.cfg
}


define contact {
contact_name Central_Dispatch
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r,f,s
host_notification_options d,u,r,f,s
service_notification_commands
notify-service-by-nsca
host_notification_commands
notify-host-by-nsca
address1 nagios.omnitec.net
}


Central SiteSet the service definition or template to passive passive_check_enabled=1, and active_checks_enabled=0.

Note: For any service to be monitored upstream, the hostname and service description must match exactly the data being sent via send_nsca!


Test service template on Central Serverdefine service {
namepassive-service
usegeneric-service
check_freshness1
passive_checks_enabled1
active_checks_enabled0
is_volatile0
flap_detection_enabled0
notification_optionsw,u,c,s
freshness_threshold57600 ;12hr
}


Test service on Central Serverdefine service {
usepassive-service
host_namelocalhost
service_descriptiontest
check_commandcheck_dummy!3!"No Data Received"
}


Test service on Remote SRM#!/usr/bin/perl
#############################################################
# RETURN CODES:
# 0-OK, 1-WARNING, 2-CRITICAL, 3-UNKNOWN
#############################################################
#CONFIG FILES
#$debug=1;
$config="/usr/local/nagios/etc/send_nsca.cfg";
# LOCAL SYSTEM CONFIG OPTIONS
$nsca_host="srm.omnitec.net";
$host="DevelSRM";
$service="test_service";


Test service on Remote SRM (cont'd)# DEFAULT RETURNS
$code=3;
$result="What's going on";
# COMMAND LINE
$send_nsca="/usr/local/nagios/bin/send_nsca -c $config -H $nsca_host";
# Start
# INSERT YOUR FUN CODE HERE, Setting a $code and $result value
# End
if ($debug) {print "SENDING: $host\t$service\t$code\t$result\n";}
open(SEND,"|$send_nsca") || die "Could not run $send_nsca: $!\n";
print SEND "$host\t$service\t$code\t$result\n";
close SEND;


System troubleshooting tipsCheck logs/usr/local/nagios/logs

/var/log/apache2/error.log

Verify UI user permissions/usr/local/nagios/etc/htpssword.users

Distributed/Hierarchical
Architecture

Low cost

Local Nagios Instance

Central monitoring
and dispatch

Conclusion

Ongoing work, ..

Motion VideoB+ hardware

OpenGL accssible (non-proprietary)

Configuration UIAdagios

Not fully NagiosCore 4 compatible yet

Questions?

Thank you!

C:\Users\zbordeau\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.IE5\AV93OOZ6\MP900315598[1].jpg

The End

Lee Lammert

[email protected]
@lvlammert

Click to edit Master title style

Click to edit Master text styles

Second level

Third level

Fourth level

Fifth level

10/17/14


Click to edit Master subtitle style

10/17/14


Second level
Third level
Fourth level
Fifth level

10/17/14



10/17/14


Second level
Third level
Fourth level
Fifth level

Second level
Third level
Fourth level
Fifth level

10/17/14



Second level
Third level
Fourth level
Fifth level


Second level
Third level
Fourth level
Fifth level

10/17/14


10/17/14

10/17/14


Second level
Third level
Fourth level
Fifth level


10/17/14



10/17/14



Second level

Third level

Fourth level

Fifth level

10/17/14



Second level

Third level

Fourth level

Fifth level

10/17/14

PRESENTATION TITLE

Presenter Name

10/17/14

[email protected]



Second level

Third level

Fourth level

Fifth level

10/17/14

PRESENTATION TITLE

Presenter Name

10/17/14

[email protected]


Second level
Third level
Fourth level
Fifth level

10/17/14



10/17/14


Second level
Third level
Fourth level
Fifth level

Second level
Third level
Fourth level
Fifth level

10/17/14



Second level
Third level
Fourth level
Fifth level


Second level
Third level
Fourth level
Fifth level

10/17/14


10/17/14

10/17/14


Second level
Third level
Fourth level
Fifth level


10/17/14



10/17/14



Second level

Third level

Fourth level

Fifth level

10/17/14



Second level

Third level

Fourth level

Fifth level

10/17/14

nagios conference 2014 - leland lammert - distributed heirarchical nagios

Technology