What the #%!@ is wrong with my server?!?

Josh Malone Systems Administrator

National Radio Astronomy Observatory Charlottesville, VA

1

heck

^

• Intro to Monitoring

• Nagios • Install/Config • Usage • Custom plugins

• Intro to Troubleshooting • Tools

• telnet, openssl • grep, sed • ps, lsof, netstat

• Internet protocols 101 • SMTP • IMAP • HTTP

• Packet sniffing for dummies

Agenda

2

MONITORING

3

Automated Monitoring Workflow

4

Monitoring Packages: Open Source•

• Opsview Core

•

•

•

• Core

• Pandora FMS

• Naemon

•

•

• Captialware ServerStatus

• Sensu

5

All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here.

Monitoring Packages: Commercial• Nagios XI

• Groundwork

• PRTG network monitor

• CopperEgg

• WhatsUp Gold

• PRTG network monitor

• op5 (Naemon)

6

All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here.

Why Automatic Service Monitoring?• Spot small problems before they become big ones

• Learn about outages before your users do

• Checklist when restoring from a power outage

• Gives you better problem reports than users

• Problems you might never spot otherwise • Failed HDDs in RAIDs • Full /var partitions • Logs not rotating • System temperature rising

7

Why Automatic Service Monitoring?• Capacity planning

• Performance data can generate graphs of utilization • RAM, Disk, etc.

• Availability reports - CAUTION • Easy to generate -- even easier to generate wrong • Make sure your configurations actually catch problems • Will also include problems with Nagios itself :( • If you’re going to quote your availability numbers (SLAs,

etc.) make sure you understand what you’re actually monitoring.

8

With Monitoring

•dhcp out of leases •dhcp server down •dns server not responding

•ethernet switch down • ISP link down / saturated

9

Without Monitoring

!

“The Internet’s down - fix it!!!”

With Monitoring

•connectivity issues •web server down •apache not running •web server disk full •server load too high

10

Without Monitoring

!

“ZOMG! Our web site is down! O Noes!!!”

• Open source host / service monitoring package

• “Nagios Ain't Gonna Insist On Sainthood”

• Originally released in 1999 as “NetSaint”

• Available in 2 versions: Core and XI

• Nagios Core: Open-source, freely available

• Nagios XI: Commercial • Free license for up to 7 hosts • Available as source installer or VMware appliance

11

Nagios

• Host - A logical (physical / virtual) machine running an OS

• Service - A resource available on a host • Network service (www, dns, imap) • Local resource (free RAM, disk space, system load)

• Plugin - An executable that checks something

• Add-on - An extension to Nagios that adds functionality • Graphing, Trending, SNMP trap reporting, etc.

12

Terminology

What’s a plugin?• Plugins actually run the service or host checks.

• Each plugin monitors a different type of service

• Data from plugin is communicated to Nagios using a (very) simple API

• Plugins can also report “Performance Data” (perfdata) to be graphed or tracked • Requires a perfdata add-on (or Nagios XI)

• Plugins can be written in any language • Perl plugins can run using Nagios’s embedded perl

interpreter for increased performance

13

INSTALLING NAGIOS

14

A word of caution...

15

Installing Nagios on Linux is much easier than Mac. While most Linux distros are still shipping Nagios 3.5.x, they usually come with all the pre-requisite packages available via their native package managers. This makes it much easier to build 4.x from source. Small to medium environments don’t require much of a server to run Nagios.

Mac installations will require you to install numerous pre-requisites, especially for the plug-ins. Many of the pre-reqs can be installed using MacPorts, Homebrew and cpan.

If you can run on Linux, do it.

About Nagios Replacements

16

When Nagios went commercial, the “open-source community” decided that it needed not one, not two, but three replacements for Nagios: Icinga and Naemon (forks of Nagios) and Shinken (a drop-in replacement). Most linux distros are now shipping one or more of these compatible replacements rather than the official Nagios Core. Not a single distro I checked is shipping Nagios 4.

Either Shinken, Naemon or Icinga should work fine using the material covered in the tutorial, but I have only briefly tested Icinga and have not tested Shinken or Naemon at all.

About Nagios Plugins source• On January 15, 2014, Nagios Enterprises “forked” the original

code from the “Nagios plugins” project, effectively creating two sources for the Nagios plugins.

• The original maintainers of the plugins package renamed it to “monitoring plugins” and now host their distribution at monitoring-plugins.org

• Nagios Enterprises appears to be actively maintaining the nagios-plugins package, including security fixes

• This tutorial will use the official Nagios plugins package

17

Requirements • C compiler (Xcode + cmdline tools)

• Perl 5.8+ (for many plugins)

• Web server (for web interface)

• PHP 5.3

• gd (for statusmap and trends CGIs)

• rrdtool, perl-RRD, php-GD (for PNP4Nagios)

18

Nagios 4 - brew or build from source

19

• Latest MacPorts version of nagios is 3.5.0 - boo :( • Uses old worker code and not as efficient as 4.x

• Latest Homebrew version is 4.0.6 - yay! •brew install nagios!

• We want Nagios 4.0.6 at least • fixes lots of bugs and is much faster

Create the user and group

20

• Nagios needs a user / group to run as

• Create user ‘nagios’ with group ‘nagios’

• Can use System Preferences -- no need to resort to command line

Create group ‘nagios’• Go to Users and Groups, click the +

• Change “New Account:” selector to “Group”

• Set name to ‘nagios’

21

Create user nagios• Click + again

• Create a “Standard” account

• Account name: ‘nagios’

• Set a password for security

22

Create user nagios• Right-click the new ‘nagios’ account and select

• Change group to ‘nagios’

• Set shell to

/usr/bin/false

23

Download• Nagios Core

• http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-4.0.7.tar.gz

• Nagios plugins package • http://nagios-plugins.org/downloads/

• PNP4Nagios • http://docs.pnp4nagios.org/pnp-0.6/dwnld

24

Installing Nagios on OSX• Install Xcode and command line tools

• Install gd library using MacPorts (or homebrew) • port install gd2!

• brew install gd!

• cpan!

• install Net::SNMP!

• Extract the tarball

• Configure and install nagios ./configure --with-gd-inc=/opt/local/include \ --with-gd-lib=/opt/local/lib \ --with-httpd-conf=/etc/apache2/other!

25

Compile problems:• ld: archive has no table of contents for architecture

x86_64 • Probably a makefile bug • Solution: run ranlib lib/libnagios.a

26

Installation from sourcesudo -s!

make install!

make install-config!

make install-webconf install-exfoliation!

make install-commandmode!

install -m 755 -o root -g wheel daemon-init \! /usr/local/nagios/nagios.init!

htpasswd -c /usr/local/nagios/etc/htpasswd.users nagios

27

Build and Install Plugins• The plugins have LOTS of prerequisites

• LDAP libs • SNMP utils • samba client • openssl libs

• Installed most prerequisites via MacPorts • Homebrew should work as well

!

./configure --prefix=/usr/local/nagios \!--enable-perl-modules \!--with-mysql=/opt/local/lib/mysql5 \!--with-openssl=/opt/local/

28

Install pnp4nagios• Pre-reqs: rrdtool, perl RRD modules, php-GD

!

./configure --with-nagios-user=nagios \! —with-nagios-group=nagios!

!make!# Install without npcd (bulk mode)!make install-config install-processperfdata \!

install-html install-plugins

29

Configure pnp4nagioscp /usr/local/pnp4nagios/etc/nagios.cfg-sample \! /usr/local/nagios/etc/pnp4nagios.cfg!

cp /usr/local/pnp4nagios/etc/misccommands.cfg-sample \ ! /usr/local/nagios/etc/objects/misccommands.cfg

30

Create LaunchDaemon <?xml version="1.0" encoding="UTF-8"?>!

<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN!

http://www.apple.com/DTDs/PropertyList-1.0.dtd >!

<plist version="1.0">!

<dict>!

<key>Label</key>!

<string>org.nagios.nagios</string>!

<key>ProgramArguments</key>!

<array>!

<string>/usr/local/nagios/bin/nagios</string>!

<string>-d</string>!

<string>/usr/local/nagios/etc/nagios.cfg</string>!

</array>!

<key>KeepAlive</key>!

<true/>!

</dict>!

</plist>

31

Web server / PHP• Enable OSX web server

• Set PHP timezone to avoid a warning echo "date.timezone = 'America/New_York'" >>/etc/php.ini

32

CONFIGURING NAGIOS

33

Configuration

34

• Nagios has 2 parts to configure: the daemon and the cgi’s

• Configure the CGIs to define access levels (authorization)

• Configure the daemon • Commands

• Check commands • Misc commands (how to notify, event handlers)

• Services • Contacts • Monitoring and alerting policies

cgi.conf• Enable authentication in the CGIs

• use_authentication=1!• Expects your web server to authenticate users

• Only allow admin users to issue commands and view config details • authorized_for_system_information=...!• authorized_for_configuration_information=...!• authorized_for_system_commands=...!• authorized_for_all_service_commands=...!• authorized_for_all_host_commands=...!

• Allow any authenticated user to view Nagios status • authorized_for_all_services=*!• authorized_for_all_hosts=*

35

nagios.cfg• Main configuration file

• Typically includes additional cfg files for commands, services, etc. (“objects”) •cfg_file=/path/to/file!•cfg_dir=/path/to/dir/of/configs!

• Interval length • “How many seconds are Nagios’s default unit of time” • If you need to monitor a service more than once per

minute, you can lower this from 60 •interval_length=60

36

nagios.cfg• Timeouts (in seconds):

•service_check_timeout=60!•host_check_timeout=30!•event_handler_timeout=30!•notification_timeout=30!•ocsp_timeout=5!•perfdata_timeout=5!

!

• If you have checks, event handlers, etc. that take a long time to run, increase the appropriate timeout value

37

Included default config files• commands.cfg Define commands called in other files

• contacts.cfg Define contacts to be notified

• localhost.cfg Example services for the local system, • printer.cfg example printer, example network switch, • switch.cfg etc.

• templates.cfg Templates for most objects; useful as a starting point for inheritance

• timeperiods.cfg Def. periods of time (24x7, daytime, etc)

38

Inheritance• All objects in Nagios support inheritance via the

“use <template>” directive.

• Templates can inherit from other templates

• All templates must include the “register 0” directive so Nagios knows it’s just a template

• Templates include common settings for different obj types •normal_check_interval, retry_check_interval, notification_interval, notification_period, notification_options, max_check_attempts, check_period

39

Object Configuration: Overview

40

Inheritance

Reference  /  Assignment

Inheritance Exampledefine service {!!name generic-service!!normal_check_interval! 5!!retry_check_interval!! 1!!max_check_attempts! ! 3!!…! register! ! ! ! ! ! 0!}!!define service {!! use !! ! ! ! generic-service!

!! host_name !! ! webserv!! service_description HTTP!! check_command !! ! check_http!

}

41

Check every 5 minutes. Notify after service is down for ~3 minutes.

Basic Configuration Example• Monitor a mail server that provides IMAP, SMTP and webmail

• Server IP address is 10.42.1.31 — name ‘hornet’

• All services are running on standard ports

• Not concerned with SSL services for this demo

42

Example: Host Templatesdefine host {!! name!! ! ! ! ! ! ! ! ! ! ! generic-host!! notifications_enabled ! 1!! event_handler_enabled ! 1!! flap_detection_enabled ! 1!! process_perf_data ! 1!! retain_status_information ! 1!! retain_nonstatus_information ! 1!! notification_period ! 24x7!! check_period!! ! ! ! ! ! ! ! 24x7!! register ! 0!}

43

Example: Host Templatesdefine host {!! name!! ! ! ! ! ! ! ! ! ! ! generic-server!! use! ! ! ! ! ! ! ! ! ! ! ! generic-host!! check_interval! ! ! ! ! ! ! ! 5!! retry_interval! ! ! ! ! ! ! ! 1!! max_check_attempts! ! ! ! ! ! 5!! check_command! ! ! ! ! ! ! ! check-host-alive!! notification_options!! ! ! ! ! d,u,r!! notification_interval! ! ! ! ! 120!! contact_groups! ! ! ! ! ! ! ! sysadmins!! register! ! ! ! ! ! ! ! ! ! 0!}

44

Example: Define Contactsdefine contact {!! contact_name! ! ! ! ! alice!! use! ! ! ! ! ! ! ! generic-contact!! alias!! ! ! ! ! ! ! Alice Admin!! email!! ! ! ! ! ! ! alice@nrao.edu!}!define contact {!! contact_name! ! ! ! ! bob!! use! ! ! ! ! ! ! ! generic-contact!! alias!! ! ! ! ! ! ! Bob Admin!! email!! ! ! ! ! ! ! bob@nrao.edu!}!define contactgroup {!! contactgroup_name!! ! ! sysadmins!! alias!! ! ! ! ! ! ! All admins!! members! ! ! ! ! ! ! alice,bob!}

45

Example: Define the Serverdefine host {!! name!! ! ! ! ! ! ! ! ! ! hornet!! use! ! ! ! ! ! ! ! ! ! ! generic-server!! host_name!! ! ! ! ! ! ! ! hornet!! host_address!! ! ! ! ! ! ! 10.42.1.31!}

46

• Most configuration inherited from generic-server • Server will be checked every 5 minutes • Problems will be re-checked every minute • Alerts will be sent to contact group ‘sysadmins’ after 5 failed

checks

Example: Service Templatedefine service{! name generic-service! active_checks_enabled 1! passive_checks_enabled 1! parallelize_check 1! check_freshness 0! notifications_enabled 1! event_handler_enabled 1! flap_detection_enabled 1! process_perf_data 1! check_period 24x7! max_check_attempts 5! normal_check_interval 5! retry_check_interval 1! notification_options w,u,c,r! notification_interval 60! notification_period 24x7! register 0! }

47

Example: Define the Servicesdefine service {!! use! ! ! ! ! ! ! ! ! ! ! generic-service!! host_name!! ! ! ! ! ! ! ! hornet!! service_description! ! ! ! ! HTTP!! check_command! ! ! ! ! ! ! check_http!}

48

• Service will be checked every 5 minutes

• Problems will be re-checked every minute

• Alerts will be sent to contacts assigned to the host ‘hornet’

Example: Define check commanddefine command {! command_name check_smb! command_line $USER1$/check_disk_smb -H $HOSTADDRESS$ -s "$ARG1$" -u nagios -p foobar -w "$ARG2$" -c "$ARG3$"!}

49

• Check command uses “macros” • $USER1$ is set in nagios.cfg and is usually the full path to

plugins installation directory • $ARG1$, $ARG2$, etc. are macros that refer to arguments

when calling the check_smb command • $HOSTADDRESS$ is the IP of the host that is being checked

Example: Define New Servicedefine service {!! use! ! ! ! ! ! ! ! ! generic-service!! host_name! ! ! ! ! ! ! hornet!! service_description! ! ! Docs share!! check_command!! ! ! ! ! check_smb!IS docs!80%!95%!}

50

• Passes 3 arguments to the check_smb check command

• SMB share “IS docs” will be checked every 5 minutes

• Problems will be re-checked every minute

• Alerts will be sent to contacts assigned to the host ‘hornet’

Running Plugins on a Remote Host• Sometimes you need the plugin to actually run on the

monitored host • Checking local resources like disk, ram, etc.

• Multiple options for “remote plugin” execution • check_by_ssh - ssh to host and run local plugin • nrpe - the Nagios Remote Plugin Executor • ncpa - a full-featured Nagios monitoring agent

51

USING NAGIOS

52

Overview

53

Navbar Main window

Services View

54

Host summary Service summary

Click on Services - Critical

55

Host and Service Groups• Organize services or hosts into groups by function, etc.

• Can disable alerts, schedule downtime, etc. on whole group

• Can show availability report for a whole group • Group services by desired reporting capability

• Groups get a unique URL so you can send a single link to check on a group of hosts • Great for PHBs! • Also great for delegated IT departments

56

Service Groups

57

Acknowledging an Outage• Click on service name (or hostname) that has the problem

• Under “Service Commands” • Click “Acknowledge this service problem”

• You must enter a comment about why you are acknowledging the problem (i.e., “Bob is working on it”) • Click “Commit”

58

Acknowledging an Outage

59

Click Here

Acknowledging an Outage

60

The Tactical Overview• Displays overview of monitored services and hosts

• Shows if • Any services / hosts have notifications disabled • Any services / hosts are flapping • Active / passive checks enabled / disabled • Warning / Critical / Okay breakdown

61

The Tactical Overview

62

Silencing All Alerts• Don’t ever do this!

• You *will* forget to turn them back on • At least make sure someone is watching the dashboard !

• Hidden at bottom of sidebar

• Click “Process Info”

63

Silencing All Alerts• Disable notifications

• Failed Nagios upgrade causes every plugin to fail • Some other problem with Nagios server itself

• Disable event handlers (in case one is going crazy)

64

CUSTOM PLUGINS

65

Custom Plugins

66

• Nagios can monitor anything you can write a script to check • Simple API: just write text to stdout and exit with a value • You can write plugins in ANY language you choose!

• bash, python, tcl, expect • perl (Nagios has embedded perl interpreter for speed) • C, C++

• Huge collection of plugins available at: http://exchange.nagios.org https://www.monitoringexchange.org

• Be wary of some community plug-ins! • Test first!!!

The API• exit code:

• 0: OK - everything is normal. Carry on. • 1: Warning - a metric’s warning threshold was exceeded • 2: Critical - service not running or a metric’s critical

threshold was exceeded • 3: Unknown - plugin was unable to run properly

• stdout: Brief summary of the state of the service. • Generally included in pages so keep length short • Not mandatory; does not affect Nagios’s interpretation of

the service state • Can include “performance data”

67

Performance Data• Metrics about the state of the service

• Can be used to generate graphs showing trends, etc.

• Performance data processing requires some external add-on like PNP4Nagios

68

Performance Data• Perfdata is separated from main plugin output by vertical bar

character (pipe symbol)

• Format: •name=value[unit];[w thresh];[c thresh];[min];[max]!

• Minimal output: name=value • Examples:

• bytes=245932489 • temp=69.5F;78;85;50;100 • mailq=34Messages;200;400 • disk_root=85%

69

Example Plugin - Time Machine• Let’s write a plugin to check that a machine is being backed up

by Time Machine

• Figure out how to check this via command line • tmutil latestbackup • /Volumes/Time Machine Backups/Backups.backupdb/agrajag/

2014-06-24-215904 • We can parse this last part into a timestamp with

•date -jf '%Y-%m-%d-%H%M%S' '2014-06-24-215904' +%s!

• Beware differences between GNU date and BSD date • Compare to date +%s!• Simple arithmetic to see if we’ve crossed a threshold

70

Minimal Plugin - Time Machine#!/bin/bash!

<read warning and critical thresholds>!

!now=$(date +%s)!

latestbackup=$(tmutil latestbackup | sed -e 's/.*\///' )!

latest=$(date -jf '%Y-%m-%d-%H%M%S' "$latestbackup" +%s)!

delay=$(( now - latest ))!

if [ $delay -gt $critical ]; then state=2!

elif [ $delay -gt $warning ]; then state=1!

else state=0!

fi!

exit $state

71

TROUBLESHOOTING

72

Basic troubleshooting resources• Logs!!!!

• You might have to enable or raise the log level • Verbose / debug output

• --debug --verbose -v or -vvvv options • run servers in the foreground

• Watch the traffic or service output • Telnet • Packet capture (tcpdump, wireshark, etc.)

• Examine processes • ps, lsof, netstat, strace

• Network / DNS tests • ping, dig, traceroute, nmap

73

It’s Log, Log, Log!• /var/log/*

• Common log tools • grep • tail • head • less

• Not sure, which file to look at? Sort by modification time: •ls -lt!

• Learn about “regular expressions” for searching with grep

74

Common Log Actions• Watch a logfile in real-time tail -f logfile!

• Print the last 100 lines tail -100 logfile!

• Search for specific text - case insensitive grep -i -e ‘some string’ logfile!

• Exclude lines with specific text grep -v -e ‘some string’ -e ‘other string’ logfile!

• Truncate (clear) a log file without disturbing file permissions > logfile

75

Common Log Regular Expressions‘^string’ String at beginning of line

‘string$’ String at end of line

‘program: ‘ Find a program in syslog file (colon)

‘8\.8\.8\.8’ IP addresses — must escape the dot

‘progname.*error’ Line that mentions a program and ‘error’

‘prog.*(error|warn)’ Find error or warn on the line

76

Processes• ps aux

• List all processes on the system and the user it’s running as • lsof

• List all open files and which process / user is using it • Great for finding why you can’t unmount a filesystem • Make sure a process is using the files you think it is • lsof also lists listening network processes

lsof | grep ‘IPv4’

77

Netstat• Display info about network sockets and connections

• BSD (Mac) and GNU (Linux) versions differ significantly

• Linux: • -l Show listening processes • -p Show process PID !

• Both: • -n Show numerical output - skip DNS resolution

78

Query DNS with ‘dig’• dig is part of the OSX base and most Linux distros

• Query DNS without involving host’s resolver or cache

• Query for exact record type

• Query a specific DNS server with ‘@<server>’

• Use dig to ensure that your name servers are answering queries properly • Compare to output of ‘host <query>’ to spot resolver

problems !

! dig [@<server>] <query> [<record type>]

79

Common ‘dig’ Queries: jmalone@gargravarr; dig @8.8.8.8 www.nrao.edu!!; <<>> DiG 9.8.3-P1 <<>> @8.8.8.8 www.nrao.edu!

; (1 server found)!

;; global options: +cmd!

;; Got answer:!

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41627!

;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0!!;; QUESTION SECTION:!

;www.nrao.edu. IN A!!;; ANSWER SECTION:!

www.nrao.edu. 11141 IN CNAME quordlepleen.cv.nrao.edu.!

quordlepleen.cv.nrao.edu. 11141 IN A 192.33.115.5!!

80

Common ‘dig’ Queries; <<>> DiG 9.8.3-P1 <<>> @8.8.8.8 nrao.edu in ns!

; (1 server found)!

;; global options: +cmd!

;; Got answer:!

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1187!

;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0!

!;; QUESTION SECTION:!

;nrao.edu. IN NS!

!;; ANSWER SECTION:!

nrao.edu. 13116 IN NS cv3.cv.nrao.edu.!

nrao.edu. 13116 IN NS zia.aoc.nrao.edu.!

nrao.edu. 13116 IN NS sadira.gb.nrao.edu.

81

Telnet & Openssl• We can use telnet and openssl to open a TCP connection to a

service on any port on communicate directly to the service as if we were a client program

• Use telnet for un-encrypted services • telnet <hostname> <port>!

• <port> can be numeric or name from /etc/services

• Use openssl for an SSL- or TLS-protected service • openssl s_client -connect <host>:<port>

82

Telnet Example: web server

83

: jmalone@gargravarr; telnet www.google.com 80!Trying 74.125.225.112...!Connected to www.google.com.!Escape character is '^]'.!GET / HTTP/1.0!↵!HTTP/1.0 200 OK!Date: Mon, 17 Mar 2014 01:03:31 GMT!Expires: -1!Cache-Control: private, max-age=0!Content-Type: text/html; charset=ISO-8859-1!Set-Cookie: PREF=ID=a725ac8...; expires=Wed, 16-Mar-2016 01:03:31 GMT; path=/; domain=.google.com!Set-Cookie: NID=67=O11S-VZ1...; expires=Tue, 16-Sep-2014 01:03:31 GMT; path=/; domain=.google.com; HttpOnly!Server: gws!!<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage">!<head><meta content="Search the world's information, including webpages, images, videos and more....

Same thing but HTTPS

84

$ openssl s_client -connect www.google.com:https!CONNECTED(00000003)!depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA!verify error:num=20:unable to get local issuer certificate!verify return:0!---!Certificate chain! 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com! i:/C=US/O=Google Inc/CN=Google Internet Authority G2! 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2! i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA! 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA! i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority!---!(certificate block)!subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com!issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2!---!SSL handshake has read 3238 bytes and written 444 bytes!---!New, TLSv1/SSLv3, Cipher is RC4-SHA!Server public key is 2048 bit!Secure Renegotiation IS supported!Compression: NONE!Expansion: NONE!SSL-Session:! Protocol : TLSv1!

SMTP• Simple Mail Transfer Protocol

• RFC 821, 2821, 5321

• Simple vocabulary: •HELO!•MAIL FROM: <address>!•RCPT TO: <address>!•DATA!•QUIT

85

Note: the angle brackets are included

around the address

SMTP Example

86

: jmalone@gargravarr; telnet smtp.cv.nrao.edu smtp!Connected to cv3.cv.nrao.edu.!Escape character is '^]'.!220 cv3.cv.nrao.edu ESMTP Sendmail cv-ws-8.12; Sun, 16 Mar 2014 !HELO gargravaar.local!250 cv3.cv.nrao.edu Hello va-xx.dhcp.xx [184.xx], pleased to meet you!MAIL FROM: <user@example.com>!250 2.1.0 <user@example.com>... Sender ok!RCPT TO: <jmalone@nrao.edu>!250 2.1.5 <jmalone@nrao.edu>... Recipient ok!DATA!354 Enter mail, end with "." on a line by itself!Subject: This is a test!!Hi. This is a test message.!.!250 2.0.0 s2H2uDAh001206 Message accepted for delivery!QUIT!221 2.0.0 cv3.cv.nrao.edu closing connection!Connection closed by foreign host.

IMAP Example$ openssl s_client -connect hornet:imaps!* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5 AUTH=X-PLAIN-SUBMIT] Dovecot ready.!. login nagios foobar!. OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND CATENATE UNSELECT …… ] Logged in!. list "" *!* LIST (\HasNoChildren) "." "Deleted Messages"!* LIST (\HasNoChildren) "." "Drafts"!* LIST (\HasNoChildren) "." "Junk"!* LIST (\HasNoChildren) "." "Sent Messages"!* LIST (\HasNoChildren) "." "INBOX"!. OK List completed.!. logout!* BYE Logging out!. OK Logout completed.!closed

87

IMAP Example: Bad auth$ openssl s_client -connect hornet:imaps!* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5 AUTH=X-PLAIN-SUBMIT] Dovecot ready.!. login nagios wrongpassword!. NO [AUTHENTICATIONFAILED] Authentication failed.!!

• Cannot tell if problem is username or password • Check server logs

bash-3.2# grep dovecot /var/log/system.log!Jul 6 14:35:55 hornet dovecot[1533]: auth: Error: od(nagios,10.42.1.33): Credentials could not be verified, username or password is invalid.!bash-3.2# id nagios!uid=1025(nagios) gid=20(staff) groups=20(staff),403(com.apple.sharepoint.group.2),402(com.apple.sharepoint.group.1),404(com.apple.sharepoint.group.3),12(everyone),61(localaccounts)

88

IMAP Example: Bad auth mech$ openssl s_client -connect hornet:imaps!* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=LOGIN AUTH=CRAM-MD5 AUTH=X-PLAIN-SUBMIT] Dovecot ready.!. login nagios foobar!. NO [ALERT] Unsupported authentication mechanism.!!!• In this case, the “plain” mechanism was disabled by the server. • Had to enable this mech to support a legacy client that only

did “plaintext” auth.

89

PACKET SNIFFING

90

Packet Sniffing• Packet sniffing examines live network traffic on a connection

• Watch contents of network packets during a connection to look for problems

• Two main tools to cover • Wireshark: graphical packet sniffing tool • tcpdump: command-line packet sniffing tool

• Other tools available • EtherApe - Linux GUI network monitor • iptraf - Linux TUI connection monitor

91

Wireshark• X11 application - requires XQuartz on 10.8+

• Originally known as “Ethereal”

• Powerful capture filters and protocol decoders

• Warning: In the past, bugs in the protocol decoders have led to remote attack vectors when running Wireshark as root. You have been warned.

• Can also capture traffic on one machine using ‘tcpdump’ and load the cap file into Wireshark on another

92

Wireshark Overview

93

Start new capture

Wireshark Overview

94

Select Interface

Enter capture filter

Start capture

Capture Filters• Capture filters are different from display filters

• Capture filters use libpcap syntax (same as tcpdump) !

• host 8.8.8.8 - Only capture packets involving this host

• ! host 8.8.8.8 - Exclude specific IP address • Useful for chatty hosts you don’t care about

• dst 8.8.8.8 - Traffic send to 8.8.8.8

• port 80 - Any traffic involving the www port

• arp - Just watch ARP traffic (look for storms)

95

Capture Filters• Can use logical operators to AND or OR filters

• dst 8.8.8.8 or dst 8.8.4.4 • host 8.8.8.8 and host 10.0.0.10 • host 8.8.8.8 and (host 10.2.101 or host 10.2.102) !

• tcp portgrange 1-1023 • Traffic from privileged ports !

• http://www.tcpdump.org/tcpdump_man.html

96

Display Filters• Different from capture filters

• ip.addr == 8.8.8.8 - Any traffic involving host

• ! ip.addr == 8.8.8.8 - Traffic not involving host • NOTE: don’t use ip.addr != 8.8.8.8 - not the same filter

• ip.dst == 8.8.8.8 - Traffic destined for 8.8.8.8

• tcp.port == 80 - Any non-SSL web traffic

• arp - Any ARP traffic

97

Display Filters• Can use logical operators to AND or OR filters

• arp or tcp.port == 22 • ip.addr == 8.8.8.8 and tcp.port == 53 • ip.addr == 8.8.8.8 and (tcp.port==53 or tcp.port==5353)

• Other types of matching • ip.addr contains 10.1.1 • http.request.uri matches ‘jsp$’

98

tcpdump• Command-line packet sniffer for unix systems

• Can filter captured traffic based on IP, port, protocol, etc.

• Can save ‘pcap’ files for use with other decoder software • Use options “-s 0 -w <filename>” • The -s 0 tells it to “snarf ” the whole packet not just the

headers • scp file back to your machine and analyze with Wireshark

• Uses the same syntax for capture filters as Wireshark and other libpcap applications.

99

THANK YOU!

100