chapter 11

CT 320: Network and System Administra8on Fall 2014*

Dr. Indrajit Ray Email: [email protected]

Department of Computer Science

Colorado State University Fort Collins, CO 80528, USA

Dr. Indrajit Ray, Computer Science Department CT 320 Network and Systems Administra8on, Fall 2014

* Thanks to Dr. James Walden, NKU and Russ Wakeeld, CSU for contents of these slides

Logging


Topics


1. System logs 2. Logging policies 3. Finding logs 4. Syslog 5. Syslog servers 6. Log monitoring

System Logs


Logs record status and error condi8ons. Where do log messages come from? Kernel Accoun8ng system System services

Logging methods: Service records own logs (apache, cron). Service uses syslog service to manage logs.

Logging Policies


1. Throw away log data. 2. Save for a while, then throw away. 3. Rotate log les 4. Archive log les

How to choose a logging policy?

Are there any data reten8on requirements? How much disk space do you have? How quickly do you need to retrieve logs? Could you nd the source of a security issue

with the logs you keep?


Throwing Away


Not recommended. Leaves you unaware of: So]ware and hardware problems Security incidents

It may take 8me to detect an incident. Keep logs for at least a month or two.

Rota:on


Keep backup les for each day/week logfile logfile.1 logfile.2 logfile.3

Rename les each day/week to move old ones back in list. Compress rotated logs to save disk space. Remove/archive logs that are X days old.

Rota:on


#!/bin/sh cd /var/log mv logfile.2 logfile.3 mv logfile.1 logfile.2 mv logfile logfile.1 cp /dev/null logfile chmod 600 logfile

logrotate


Program to handle log rota8on. Run via /etc/cron.daily. Congured via /etc/logrotate.conf.

Op8ons How o]en to rotate How long to keep logs Compression or not Log le permissions Pre- and post-rotate scripts

logrotate.conf


# rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old create # uncomment if you want your log files compressed #compress # RPM packages drop log rotation information into include /etc/logrotate.d # no packages own wtmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp rotate 1 }

Archiving Logs


Store logs to archival media (tape.) Archive a]er X days/weeks. Should be part of regular backup plan. May want to save logs for all hosts together.

Finding Logs


Most logs are stored under /var/log /var/adm

To nd other logs, read startup scripts /etc/init.d/* and manuals for services started by scripts.

Finding Logs


Log file

Program

Contents

messages

syslog

Various program/kernel logs.

syslog

syslog

Various program/kernel logs.

auth.log

su, ssh, login

Authorization fail/success.

lastlog

login, xdm

Logins, commands.

wtmp

login

Login accounting data.

acct/pacct

kernel

UNIX process accounting.

Xorg.log

X-Windows

X-Windows failures/info.

Syslog

Comprehensive logging system. Frees programmers from managing log les. Gives sysadmins control over log management.

Sorts messages by Sources Importance

Routes messages to des8na8ons Files Network Terminals


Syslog Components


Syslog Daemon that does actual logging. Addi8onal daemon, klog, gets kernel messages.

openlog, syslog, closelog C library rou8nes to submit logs to syslog.

logger User-level program to submit logs to syslog. Can be used from shell scripts.

Example Syslog Messages


Feb 11 10:17:01 localhost /USR/SBIN/CRON[1971]: (root) CMD ( run-parts --report /etc/cron.hourly) Feb 11 10:37:22 localhost -- MARK -- Feb 11 10:51:11 localhost dhclient: DHCPREQUEST on eth1 to 192.168.1.1 port 67 Feb 11 10:51:11 localhost dhclient: DHCPACK

from 10.42.1.1 Feb 11 10:51:11 localhost dhclient: bound to

10.42.1.55 -- renewal in 35330 seconds. Feb 11 14:37:22 localhost -- MARK -- Feb 11 14:44:21 localhost mysqld[7340]: 060211

14:44:21 /usr/sbin/mysqld: Normal shutdown Feb 12 04:46:42 localhost sshd[29093]: Address 218.38.30.101 maps to ns.thundernet.co.kr, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT! Feb 12 04:46:44 localhost sshd[29097]: Invalid user matt from ::ffff:218.38.30.101

Conguring Syslog


Congured in /etc/syslog.conf Format: selector action Ex: mail.info /var/log/mail.log

Selector components Source (facility)

List of facili8es separated by commas or *. Importance (level)

Can be none or *

/etc/syslog.conf


# Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log

Syslog Facili:es


Facility

Used By

kern

The kernel

user

User processes (default)

mail

Mail servers and related software.

daemon

System daemons (except mail, cron)

auth

Security and authorization-related commands.

lpr

Print server and related commands.

cron

Cron daemon.

local0-7

Eight local levels for other programs.

Syslog Levels


Level

Meaning

emerg

Panic situations (hardware failure, crash)

alert

Urgent situations

crit

Critical situations

err

Non-critical errors.

warning

Warnings.

notice

Might merit investigation.

info

Informational messages.

debug

Debugging (typically enabled temporarily.)

Syslog Ac:ons


Action

Meaning

filename

Write message to file on local machine.

@hostname

Send message to syslogd on hostname.

@ip

Send message to syslogd at IP address.

user1,user2

Write message to user screen if logged in.

*

Write message to all logged-in users.

Tes:ng Syslog


stu> for i in {debug,info,no1ce,warning,err,crit,alert,emerg} > do > logger -p daemon.$i "Test message for daemon, level $i" > done stu> tail /var/log/daemon.log Feb 11 15:57:00 localhost stu: Test message for daemon, level debug Feb 11 15:57:00 localhost stu: Test message for daemon, level info Feb 11 15:57:00 localhost stu: Test message for daemon, level no8ce Feb 11 15:57:00 localhost stu: Test message for daemon, level

warning Feb 11 15:57:00 localhost stu: Test message for daemon, level err Feb 11 15:57:00 localhost stu: Test message for daemon, level crit Feb 11 15:57:00 localhost stu: Test message for daemon, level alert Feb 11 15:57:00 localhost stu: Test message for daemon, level emerg

Syslog Variants


Some use m4 macros auth.notice ifdef(LOGHOST, /var/log/authlog, @loghost)

Red Hat Linux variants Allows spaces as separators. New operators: = (this priority only)

Ex: mail.=info New operators: ! (except this pri and higher)

Ex: mail.info,mail.!err

Syslog NG

Free drop-in replacement for syslog. More congurable Save logs to templated loca8on (auto-rotates.) Filter logs based on program, 8me, message, etc. Message format customiza8on. Allows easy logging to remote database.

Improved networking TCP support as well as UDP.

Improved security Doesnt trust hostnames in remote messages. TCP transmission permits encrypted tunneling (stunnel.)


Log Servers

Collect all syslog data on one server. Allows logging to scale to large networks. Logs can be correlated across machines. Security-sensi8ve logs not on compromised host. Routers and diskless-hosts must log to a server.

Need two syslog.conf les Client: sends all logs across network to server. Server: saves logs to database or local les.


Log Monitoring


Too much data for a human to process. Logs arrive 24x7 too.

Use an automa8c monitoring program Triggers on pakerns found in log. Examples: logwatch, swatch # 3ware logs watchfor /(?i)3w-xxxx.+no longer fault tolerant/ mail=root,subject=LW warn: disk 3ware RAID not fault tolerant throttle 1:00:00,use=regex

chapter 11

Documents

computer science department

systems administra8on

indrajit ray email

mv logfile logfile

dayweek logfile logfile

system administra8on

log monitoring system

logfile logrotate