cis 191 - lesson 12 system monitoring 1. cis 191 - lesson 12 system monitoring monitoring log files...
TRANSCRIPT
CIS 191 - Lesson 12
System Monitoring
Monitoring Log Files
• /var/log
‒ Can be used as indication of systematic degradation
• log rotation
‒ logrotate
‒ /etc/logrotate.conf
2
CIS 191 - Lesson 12
System Monitoring
Many important logs (Red Hat family)• Kernel and system boot messages
• dmesg• boot.log (broken – see bugzilla)
• Security and authorization messages• secure, btmp, wtmp, lastlog, audit, …
• System module messages• messages (a good catch-all log), cron, maillog,
… Key Log File Locations:
/var/log directory /etc/syslog.conf
3
CIS 191 - Lesson 12
Logging
The syslog daemon, controlled by /etc/syslog.conf, is a central clearing house for handling all the log messages sent by various system programs
The klogd daemon handles kernel log messages. klogd does not have a configuration file and is controlled by command line switches.
[root@opus ~]# ps -e | grep log 2152 ? 00:00:07 syslogd 2155 ? 00:00:00 klogd[root@opus ~]#
4
CIS 191 - Lesson 12
System Monitoring
Most log files are ascii text
messages: ASCII English textmessages.1: ASCII textmessages.2: ASCII English textmessages.3: ASCII English textppp: directoryprelink: directoryrpmpkgs: ASCII textrpmpkgs.1: ASCII textrpmpkgs.2: ASCII textrpmpkgs.3: ASCII textsamba: directoryscrollkeeper.log: ASCII textsecure: ASCII textsecure.1: emptysecure.2: ASCII English textsecure.3: ASCII textspooler: emptyspooler.1: emptyspooler.2: emptyspooler.3: emptytallylog: emptyvbox: directorywtmp: datawtmp.1: dataXorg.0.log: ASCII English textXorg.0.log.old: ASCII English textyum.log: ASCII text[root@benji log]# 5
CIS 191 - Lesson 12
System Monitoring[root@opus ~]# ls -l /var/logtotal 153572-rw-r----- 1 root root 3665 Nov 11 13:36 acpid-rw------- 1 root root 527440 Jun 16 15:47 anaconda.log-rw------- 1 root root 22282 Jun 16 15:47 anaconda.syslog-rw------- 1 root root 58040 Jun 16 15:47 anaconda.xlogdrwxr-x--- 2 root root 4096 Nov 24 02:03 audit-rw------- 1 root root 0 Nov 23 04:02 boot.log-rw------- 1 root root 0 Nov 16 04:02 boot.log.1-rw------- 1 root root 0 Nov 9 04:02 boot.log.2-rw------- 1 root root 0 Nov 2 04:02 boot.log.3-rw------- 1 root root 0 Oct 26 04:03 boot.log.4-rw------- 1 root utmp 136987008 Nov 29 15:16 btmpdrwxr-xr-x 2 root root 4096 Jun 28 2007 conmandrwxr-xr-x 2 root root 4096 Jun 28 2007 conman.old-rw------- 1 root root 12817 Nov 29 16:01 cron-rw------- 1 root root 13860 Nov 23 04:02 cron.1-rw------- 1 root root 13706 Nov 16 04:02 cron.2-rw------- 1 root root 13843 Nov 9 04:02 cron.3-rw------- 1 root root 14117 Nov 2 04:02 cron.4drwxr-xr-x 2 lp sys 4096 Nov 27 04:02 cups-rw-r--r-- 1 root root 18903 Nov 11 13:35 dmesg-rw------- 1 root root 29256 Nov 11 08:11 faillogdrwxr-xr-x 2 root root 4096 Mar 28 2008 gdmdrwx------ 2 root root 4096 Oct 19 04:02 httpd-rw-r--r-- 1 root root 355948 Nov 29 16:34 lastlogdrwxr-xr-x 2 root root 4096 Jun 16 15:39 mail
-rw------- 1 root root 27520 Nov 29 08:51 maillog-rw------- 1 root root 38980 Nov 23 04:02 maillog.1-rw------- 1 root root 56964 Nov 16 04:02 maillog.2-rw------- 1 root root 74842 Nov 9 04:02 maillog.3-rw------- 1 root root 110136 Nov 2 04:02 maillog.4-rw------- 1 root root 9165 Nov 29 15:35 messages-rw------- 1 root root 11706 Nov 22 21:30 messages.1-rw------- 1 root root 35986 Nov 16 03:22 messages.2-rw------- 1 root root 12430 Nov 8 23:59 messages.3-rw------- 1 root root 6224 Nov 1 16:21 messages.4drwxr-xr-x 2 root root 4096 Jun 17 15:02 pmdrwx------ 2 root root 4096 Dec 1 2006 pppdrwxr-xr-x 2 root root 4096 Jun 27 2007 prelink-rw-r--r-- 1 root root 31559 Nov 29 04:03 rpmpkgs-rw-r--r-- 1 root root 31559 Nov 22 04:03 rpmpkgs.1-rw-r--r-- 1 root root 31559 Nov 15 04:03 rpmpkgs.2-rw-r--r-- 1 root root 31559 Nov 8 04:02 rpmpkgs.3-rw-r--r-- 1 root root 31559 Nov 1 04:02 rpmpkgs.4drwx------ 2 root root 4096 May 20 2008 samba-rw-r--r-- 1 root root 107169 Jun 17 15:07 scrollkeeper.log-rw------- 1 root root 1702726 Nov 29 16:34 secure-rw------- 1 root root 5069529 Nov 23 03:38 secure.1-rw------- 1 root root 1196200 Nov 16 03:30 secure.2-rw------- 1 root root 2404320 Nov 8 23:59 secure.3-rw------- 1 root root 6374517 Nov 1 19:52 secure.4drwxr-xr-x 2 root root 4096 Nov 23 04:02 setroubleshoot-rw------- 1 root root 0 Nov 23 04:02 spooler-rw------- 1 root root 0 Nov 16 04:02 spooler.1-rw------- 1 root root 0 Nov 9 04:02 spooler.2-rw------- 1 root root 0 Nov 2 04:02 spooler.3-rw------- 1 root root 0 Oct 26 04:03 spooler.4drwxr-x--- 2 squid squid 4096 Apr 1 2008 squid-rw------- 1 root root 0 Jun 17 14:57 tallylog-rw-r--r-- 1 root root 34140 Nov 29 16:34 up2date-rw-r--r-- 1 root root 37324 Nov 23 03:34 up2date.1-rw-r--r-- 1 root root 43305 Nov 16 03:34 up2date.2-rw-r--r-- 1 root root 32088 Nov 9 03:49 up2date.3-rw-r--r-- 1 root root 34650 Nov 2 03:49 up2date.4drwxr-xr-x 2 root root 4096 Nov 20 2007 vbox-rw-rw-r-- 1 root utmp 23040 Nov 29 16:34 wtmp-rw-rw-r-- 1 root utmp 1093632 Nov 27 02:13 wtmp.1-rw-rw-r-- 1 root cis90 59894 Oct 24 08:23 Xorg.0.log-rw-rw-r-- 1 root cis90 59894 Sep 16 12:58 Xorg.0.log.old-rw-r--r-- 1 root root 20546 Jun 17 19:32 yum.log
[root@opus ~]#
How many backups are there of each log? 4
How often are these log files rotated? weekly
From observing /var/log ….
Log files are owned by root and have restrictive permissions due to the sensitive information they contain
6
CIS 191 - Lesson 12
/etc/syslog.conf
/etc/syslog.conf on Opus
Each entry is a selector followed by an action 8
CIS 191 - Lesson 12
/etc/syslog.confFacility Description
auth The authorization system. Ex.: login, su, ftpd, rshd
authpriv User access messages use this
cron Used by the cron facility
daemon Other daemon programs without a facility of their own
ftp Used by ftp applications
kern Kernel messages
lpr The line printer spooling system
mail Used by mail applications
mark Used by syslogd to produce timestamps in log files
news Used by news applications
security Same as auth. Should not be used anymore.
syslog
user Messages generated by random user processes. Default.
uucp UUCP messages
local0 – local7 Reserved for local use.
* For all
9http://www.linode.com/wiki/index.php/Syslog_Howto
CIS 191 - Lesson 12
/etc/syslog.conf
Security Level
Priority Keyword Description
0 emergencies emerg, panic A panic condition. This is normally broadcast to all users
1 alerts alert Inmmediate action required. e.g.: Corrupted system database
2 critical crit Critical condition. e.g.: Hard device errors
3 errors err, error Error conditions
4 warning warning, warn Warning conditions
5 notifications notice Normal but significant conditions that need attention
6 informational info Informational messages
7 debugging debug Debugging messages
10http://www.linode.com/wiki/index.php/Syslog_Howto
CIS 191 - Lesson 12
/etc/syslog.confSelector Description
kernel.* kernel facility, any priority
mail.debug mail facility, debug or higher priority (same as *)
lpr,news.* all messages from printer or news
auth.warning all security messages of warning or higher priority
*.info all messages from any facility of level info or higher
ftp.=info ftp facility, info msgs only (and not higher)
*.!err any facility, pri <= err only
*.!=alert any facility, any priority except alert
*.info;mail,news,authpriv.none all msgs with info or higher priority except mail, news, and authpriv
mail.2all mail messages level critical and higher
mail.0 only critical mail messages
11http://www.linode.com/wiki/index.php/Syslog_Howto
CIS 191 - Lesson 12
/etc/syslog.conf
Action Description
/complete/path/of/some/file Messages logged to a file
/dev/console This is a link to the system console
-/complete/path/of/some/file Don't flush (write to disk) file each time; better performance but risks loss of some log info
username1[,username2 ...] Users that will get the message
* All logged in users get the message
@remotehost Log to remote host. Start the remote syslogd with "-r" option
|/path/to/named/pipe
To send output to a command you must create a named pipe, say /var/lib/cmd.pipe with the mkfifo command. Then start the command with cmd </var/lib/cmd.pipe
12http://www.linode.com/wiki/index.php/Syslog_Howto
CIS 191 - Lesson 12
Logging
Note: You can use the severity level to control where messages are sent, but you don't have control over the level a program assigns to a message.
13
CIS 191 - Lesson 12
/etc/syslog.conf
write messages to console
kernel facility (all messages) commented out
14
CIS 191 - Lesson 12
/etc/syslog.conf
All facilties with info (6) or higher priority except mail, authpriv or cron
write messages to this file
15
CIS 191 - Lesson 12
/etc/syslog.conf
mail facility, any priority write messages to this file (- means don't flush file each time)
17
CIS 191 - Lesson 12
/etc/syslog.conf
All emergency level (0) messages from any facility
All logged in users get the message
19
CIS 191 - Lesson 12
/etc/syslog.conf
Critical (2) or higher messages from uucp or news facilities
Messages are written to this file
20
CIS 191 - Lesson 12
/etc/syslog.conf
any messages from local7 (used by Red Hat family for boot messages)
Messages are written to this file
21
CIS 191 - Lesson 12
/etc/syslog.conf
only notification level (5) messages from any facility
Messages are written to this file
For Lab 10
In Lab 10 a new entry is added to /etc/syslog.conf for a custom notices log
22
CIS 191 - Lesson 12
/etc/syslog.conf
Must restart the logging service for the change in /etc/syslog.conf to take effect.
For Lab 10
Create a custom logfile
23
CIS 191 - Lesson 12
/etc/syslog.confFor Lab 10
Login as root on tty2 Login as cis191 on tty3, then su with bad password
The new log will hold root logins and login failures 24
CIS 191 - Lesson 12
Log file rotation
logrotate is normally run out of cron once every day
26The actual program lives in /usr/sbin
This is actually a script that calls the logrotate program
CIS 191 - Lesson 12
[root@opus ~]# cat /etc/logrotate.conf# see "man logrotate" for details# rotate log files weeklyweekly
# keep 4 weeks worth of backlogsrotate 4
# create new (empty) log files after rotating old onescreate
# uncomment this if you want your log files compressed#compress
# RPM packages drop log rotation information into this directoryinclude /etc/logrotate.d
# no packages own wtmp -- we'll rotate them here/var/log/wtmp { monthly minsize 1M create 0664 root utmp rotate 1}
# system-specific logs may be also be configured here.[root@opus ~]#
/etc/logrotate.conf
applie
s to all
files
for
specific
files
logrotate.conf on Opus
27
CIS 191 - Lesson 12
/etc/logrotate.conf[root@benji ~]# cat /etc/logrotate.conf# see "man logrotate" for details# rotate log files weeklyweekly
# keep 4 weeks worth of backlogsrotate 4
# create new (empty) log files after rotating old onescreate
# uncomment this if you want your log files compressed#compress
# RPM packages drop log rotation information into this directoryinclude /etc/logrotate.d
# no packages own wtmp -- we'll rotate them here/var/log/wtmp { monthly create 0664 root utmp rotate 1}
# system-specific logs may be also be configured here.[root@benji ~]#
logrotate.conf on Benji
28
CIS 191 - Lesson 12
/var/log/wtmp and var/log/btmp
[root@opus log]# ls -l /var/logtotal 153576-rw-r----- 1 root root 3665 Nov 11 13:36 acpid-rw------- 1 root root 527440 Jun 16 15:47 anaconda.log-rw------- 1 root root 22282 Jun 16 15:47 anaconda.syslog-rw------- 1 root root 58040 Jun 16 15:47 anaconda.xlogdrwxr-x--- 2 root root 4096 Nov 24 02:03 audit-rw------- 1 root root 0 Nov 23 04:02 boot.log-rw------- 1 root root 0 Nov 16 04:02 boot.log.1-rw------- 1 root root 0 Nov 9 04:02 boot.log.2-rw------- 1 root root 0 Nov 2 04:02 boot.log.3-rw------- 1 root root 0 Oct 26 04:03 boot.log.4
-rw------- 1 root utmp 136987008 Nov 29 15:16 btmpdrwxr-xr-x 2 root root 4096 Jun 28 2007 conmandrwxr-xr-x 2 root root 4096 Jun 28 2007 conman.old-rw------- 1 root root 13117 Nov 29 20:01 cron-rw------- 1 root root 13860 Nov 23 04:02 cron.1-rw------- 1 root root 13706 Nov 16 04:02 cron.2-rw------- 1 root root 13843 Nov 9 04:02 cron.3-rw------- 1 root root 14117 Nov 2 04:02 cron.4drwxr-xr-x 2 lp sys 4096 Nov 27 04:02 cups-rw-r--r-- 1 root root 18903 Nov 11 13:35 dmesg-rw------- 1 root root 29256 Nov 11 08:11 faillogdrwxr-xr-x 2 root root 4096 Mar 28 2008 gdmdrwx------ 2 root root 4096 Oct 19 04:02 httpd-rw-r--r-- 1 root root 355948 Nov 29 18:39 lastlogdrwxr-xr-x 2 root root 4096 Jun 16 15:39 mail-rw------- 1 root root 28085 Nov 29 19:56 maillog-rw------- 1 root root 38980 Nov 23 04:02 maillog.1-rw------- 1 root root 56964 Nov 16 04:02 maillog.2-rw------- 1 root root 74842 Nov 9 04:02 maillog.3-rw------- 1 root root 110136 Nov 2 04:02 maillog.4-rw------- 1 root root 9165 Nov 29 15:35 messages-rw------- 1 root root 11706 Nov 22 21:30 messages.1-rw------- 1 root root 35986 Nov 16 03:22 messages.2-rw------- 1 root root 12430 Nov 8 23:59 messages.3-rw------- 1 root root 6224 Nov 1 16:21 messages.4drwxr-xr-x 2 root root 4096 Jun 17 15:02 pmdrwx------ 2 root root 4096 Dec 1 2006 pppdrwxr-xr-x 2 root root 4096 Jun 27 2007 prelink-rw-r--r-- 1 root root 31559 Nov 29 04:03 rpmpkgs-rw-r--r-- 1 root root 31559 Nov 22 04:03 rpmpkgs.1-rw-r--r-- 1 root root 31559 Nov 15 04:03 rpmpkgs.2-rw-r--r-- 1 root root 31559 Nov 8 04:02 rpmpkgs.3-rw-r--r-- 1 root root 31559 Nov 1 04:02 rpmpkgs.4drwx------ 2 root root 4096 May 20 2008 samba-rw-r--r-- 1 root root 107169 Jun 17 15:07 scrollkeeper.log-rw------- 1 root root 1703877 Nov 29 19:59 secure-rw------- 1 root root 5069529 Nov 23 03:38 secure.1-rw------- 1 root root 1196200 Nov 16 03:30 secure.2-rw------- 1 root root 2404320 Nov 8 23:59 secure.3-rw------- 1 root root 6374517 Nov 1 19:52 secure.4drwxr-xr-x 2 root root 4096 Nov 23 04:02 setroubleshoot-rw------- 1 root root 0 Nov 23 04:02 spooler-rw------- 1 root root 0 Nov 16 04:02 spooler.1-rw------- 1 root root 0 Nov 9 04:02 spooler.2-rw------- 1 root root 0 Nov 2 04:02 spooler.3-rw------- 1 root root 0 Oct 26 04:03 spooler.4drwxr-x--- 2 squid squid 4096 Apr 1 2008 squid-rw------- 1 root root 0 Jun 17 14:57 tallylog-rw-r--r-- 1 root root 34818 Nov 29 19:34 up2date-rw-r--r-- 1 root root 37324 Nov 23 03:34 up2date.1-rw-r--r-- 1 root root 43305 Nov 16 03:34 up2date.2-rw-r--r-- 1 root root 32088 Nov 9 03:49 up2date.3-rw-r--r-- 1 root root 34650 Nov 2 03:49 up2date.4drwxr-xr-x 2 root root 4096 Nov 20 2007 vbox
-rw-rw-r-- 1 root utmp 26112 Nov 29 19:02 wtmp-rw-rw-r-- 1 root utmp 1093632 Nov 27 02:13 wtmp.1-rw-rw-r-- 1 root cis90 59894 Oct 24 08:23 Xorg.0.log-rw-rw-r-- 1 root cis90 59894 Sep 16 12:58 Xorg.0.log.old-rw-r--r-- 1 root root 20546 Jun 17 19:32 yum.log
[root@opus log]#
bad login attempts
good login attempts
30
CIS 191 - Lesson 12
/var/log/wtmp and var/log/btmp
[root@opus ~]# lastb | grep "cool.nju.edu.cn" | headbind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)
[root@opus ~]# lastb | grep "cool.nju.edu.cn" | wc -l3104[root@opus ~]#
31
Shows break in attempt on 11/30/2008
CIS 191 - Lesson 12
/var/log/wtmp and var/log/btmp
[root@opus ~]# lastb | grep "Nov 2 17:45"webadmin ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)webadmin ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)retsu ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)retsu ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sbear ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sbear ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sky ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sky ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)harvey ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)harvey ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)harvey ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)[root@opus ~]#
[root@opus ~]# lastb -i | grep "211.96.97.179" | wc -l598[root@opus ~]#
32
Shows break in attempt by 211.96.97.179 on 11/2/2008
CIS 191 - Lesson 12
/var/log/wtmp and var/log/btmp
[root@opus log]# lastb | sort | cut -f1 -d' ' | grep -v ^$ | uniq –c > bad[root@opus log]# sort –g bad > bad.sort[root@opus log]# [root@opus log]# cat bad.sort | tail -50 471 ftp 472 public 490 test 490 tomcat 498 user 506 service 508 mike 508 username 524 cyrus 530 pgsql 532 test1 544 master 554 linux 554 toor 576 paul 584 support 590 testuser 604 irc
610 test 656 noc 686 www 690 postfix 723 john 734 testing 738 adam 746 alex 754 info 798 tester 832 library 935 guest 990 admin 1002 office 1022 temp 1070 ftpuser
1138 webadmin 1298 nagios 1332 web 1374 a 1384 student 1416 postgres 1690 user 1858 oracle 1944 mysql 2086 webmaste 5324 test 10803 root 10824 admin 18679 root 24064 root[root@opus log]#
Top 50 usernames used by the bad guys 37
CIS 191 - Lesson 12
/var/log/wtmp and var/log/btmp
[root@opus log]# lastb | sort | cut -f1 -d' ' | grep -v ^$| uniq -c | wc -l22128[root@opus log]#
[root@opus log]# lastb | grep root | wc -l 54117
[root@opus log]#
22128 usernames used and failed
53117 failed root logins
Now you know why you need a strong password!
38
CIS 191 - Lesson 12
logwatch[root@opus ~]# mailMail version 8.1 6/6/93. Type ? for help."/var/spool/mail/root": 349 messages 349 new>N 1 [email protected] Mon Jun 16 17:04 43/1587 "Logwatch for opus.cabrillo.edu (Linux)" N 2 [email protected] Mon Jun 16 17:12 18/795 "Anacron job for 'opus.cabrillo.edu' cron." N 3 [email protected] Tue Jun 17 16:14 141/3966 "Logwatch for opus.cabrillo.edu (Linux)" N 4 [email protected] Wed Jun 18 04:02 728/32707 "Logwatch for opus.cabrillo.edu (Linux)" N 5 [email protected] Wed Jun 18 04:05 47/1877 "Cron <root@opus> run-parts /etc/cron.dail" N 6 [email protected] Thu Jun 19 04:02 1007/61932 "Logwatch for opus.cabrillo.edu (Linux)" N 7 [email protected] Thu Jun 19 04:02 47/1889 "Cron <root@opus> run-parts /etc/cron.dail" N 8 [email protected] Fri Jun 20 04:02 168/5533 "Logwatch for opus.cabrillo.edu (Linux)" N 9 [email protected] Fri Jun 20 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail" N 10 [email protected] Sat Jun 21 04:02 274/8886 "Logwatch for opus.cabrillo.edu (Linux)" N 11 [email protected] Sat Jun 21 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 12 [email protected] Sun Jun 22 04:02 156/4722 "Logwatch for opus.cabrillo.edu (Linux)" N 13 [email protected] Sun Jun 22 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 14 [email protected] Mon Jun 23 04:02 241/10770 "Logwatch for opus.cabrillo.edu (Linux)" N 15 [email protected] Mon Jun 23 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 16 [email protected] Tue Jun 24 04:02 3768/316984 "Logwatch for opus.cabrillo.edu (Linux)" N 17 [email protected] Tue Jun 24 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 18 [email protected] Wed Jun 25 04:02 3246/274685 "Logwatch for opus.cabrillo.edu (Linux)" N 19 [email protected] Wed Jun 25 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 20 [email protected] Thu Jun 26 04:02 1390/112446 "Logwatch for opus.cabrillo.edu (Linux)" N 21 [email protected] Thu Jun 26 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 22 [email protected] Fri Jun 27 04:02 72/2185 "Logwatch for opus.cabrillo.edu (Linux)" N 23 [email protected] Fri Jun 27 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 24 [email protected] Sat Jun 28 04:02 91/3228 "Logwatch for opus.cabrillo.edu (Linux)" N 25 [email protected] Sat Jun 28 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 26 [email protected] Sun Jun 29 04:02 150/6673 "Logwatch for opus.cabrillo.edu (Linux)" N 27 [email protected] Sun Jun 29 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 28 [email protected] Mon Jun 30 04:02 247/14351 "Logwatch for opus.cabrillo.edu (Linux)" N 29 [email protected] Mon Jun 30 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 30 [email protected] Tue Jul 1 04:02 395/20660 "Logwatch for opus.cabrillo.edu (Linux)" N 31 [email protected] Tue Jul 1 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail" N 32 [email protected] Wed Jul 2 04:02 481/32664 "Logwatch for opus.cabrillo.edu (Linux)" N 33 [email protected] Wed Jul 2 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail" N 34 [email protected] Thu Jul 3 04:02 102/3197 "Logwatch for opus.cabrillo.edu (Linux)" N 35 [email protected] Thu Jul 3 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail"& 29
You have mail … from logwatch
40
CIS 191 - Lesson 12
logwatch
example email message from logwatch
41
& 11Message 11:From [email protected] Tue Dec 2 10:47:06 2008Date: Tue, 2 Dec 2008 10:47:06 -0800To: [email protected]: [email protected]: Logwatch for benji.localdomain (Linux)MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Type: text/plain; charset="iso-8859-1"
################### Logwatch 7.3 (03/24/06) #################### Processing Initiated: Tue Dec 2 10:47:06 2008 Date Range Processed: yesterday ( 2008-Dec-01 ) Period is day. Detail Level of Output: 0 Type of Output: unformatted Logfiles for Host: benji.localdomain ##################################################################
--------------------- Disk Space Begin ------------------------
Filesystem Size Used Avail Use% Mounted on /dev/sda1 2.9G 2.5G 272M 91% / /dev/sda5 289M 234M 41M 86% /opt /dev/sda3 487M 77M 385M 17% /var /dev/sda7 196M 5.6M 181M 3% /home
---------------------- Disk Space End -------------------------
###################### Logwatch End #########################
CIS 191 - Lesson 12
Configuring logwatch
42
This file shows all the defaults being used by logwatch
Level of detail is Low by default
CIS 191 - Lesson 12
Configuring logwatch
Edit /etc/logwatch/conf/logwatch.conf to modify defaults
43
This line was added to override the default level of Low
Read this for all options to set
CIS 191 - Lesson 12
logwatchMessage 14:From [email protected] Tue Dec 2 10:53:22 2008Date: Tue, 2 Dec 2008 10:53:21 -0800To: [email protected]: [email protected]: Logwatch for benji.localdomain (Linux)MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Type: text/plain; charset="iso-8859-1"
################### Logwatch 7.3 (03/24/06) #################### Processing Initiated: Tue Dec 2 10:53:21 2008 Date Range Processed: yesterday ( 2008-Dec-01 ) Period is day. Detail Level of Output: 10 Type of Output: unformatted Logfiles for Host: benji.localdomain ##################################################################
--------------------- Cron Begin ------------------------
Commands Run: User root: /sbin/dump 0uf /backup/level0/backup-L0-`date +%Y-%d-%m`.dmp /home: 2 Time(s) /sbin/dump 1uf /backup/level1/backup-L1-`date +%Y-%d-%m`.dmp /home: 5 Time(s) /sbin/dump 2uf /backup/level2/backup-L2-`date '+: 4 Time(s) /sbin/dump 2uf /backup/level2/backup-L2-`date +: 5 Time(s) /sbin/dump 2uf /backup/level2/backup-L2-`date +%Y-%d-%m`.dmp /home: 14 Time(s) /sbin/dump 2uf /backup/level2/backup-L2.dmp /home: 2 Time(s) dump 1uf /backup/level1/backup-daily-$(date +: 1 Time(s) dump 2uf /backup/level2/backup-L2.dmp /home: 2 Time(s) dump 2uf /backup/level2/backup-daily-$(date +: 9 Time(s)
logwatch report using High level of detail
44
CIS 191 - Lesson 12
logwatchpersonal crontab deleted: 3 Time(s) personal crontab edited: 6 Time(s) personal crontab listed: 7 Time(s) personal crontab reloaded: 7 Time(s) personal crontab replaced: 11 Time(s) run-parts /etc/cron.daily: 1 Time(s) run-parts /etc/cron.hourly: 24 Time(s) run-parts /etc/cron.monthly: 1 Time(s)
---------------------- Cron End -------------------------
--------------------- sendmail Begin ------------------------
STATISTICS ----------
Bytes Transferred: 90737 Messages Processed: 92 Addressed Recipients: 92
Message recipients per delivery agent: Name # Rcpts local 46 --------------------- TOTAL: 46 in addition to 46 relay submission(s) from MSP
45
logwatch report using High level of detail continued
CIS 191 - Lesson 12
logwatch
Message Size Distribution: Range # Msgs KBytes 0 - 10k 92 88 ---------------------------------- TOTAL 92 88 Avg. Size 0
Top 10 Email Recipients ---------------------------------- [email protected] : 46 emails
Top relays (recipients/connections - min 10 rcpts, max 25 lines): 46/46: benji.localdomain [127.0.0.1] 46/46: root@localhost
---------------------- sendmail End -------------------------
--------------------- Syslogd Begin ------------------------
Syslogd started 1 Time(s)
---------------------- Syslogd End -------------------------
46
logwatch report using High level of detail continued
CIS 191 - Lesson 12
logwatch
--------------------- Disk Space Begin ------------------------
Filesystem Size Used Avail Use% Mounted on /dev/sda1 2.9G 2.5G 272M 91% / /dev/sda5 289M 234M 41M 86% /opt /dev/sda3 487M 77M 385M 17% /var /dev/sda7 196M 5.6M 181M 3% /home
---------------------- Disk Space End -------------------------
###################### Logwatch End #########################
&
47
logwatch report using High level of detail continued
CIS 191 - Lesson 12
logwatch
--------------------- SSHD Begin ------------------------ SSHD Killed: 1 Time(s)
SSHD Started: 1 Time(s)
Disconnecting after too many authentication failures for user: guest90 : 1 Time(s)
Failed logins from: 76.254.22.196 (adsl-76-254-22-196.dsl.pltn13.sbcglobal.net): 2 times 201.7.115.194 (201-7-115-194.spopa302.ipd.brasiltelecom.net.br): 2135 times 210.240.12.14: 20 times
Illegal users from: 201.7.115.194 (201-7-115-194.spopa302.ipd.brasiltelecom.net.br): 564 times 210.240.12.14: 42 times
Users logging in through sshd: guest: 76.254.22.196 (adsl-76-254-22-196.dsl.pltn13.sbcglobal.net): 2 times jimg: 70.132.20.25 (adsl-70-132-20-25.dsl.snfc21.sbcglobal.net): 7 times ordazedw: 76.254.22.196 (adsl-76-254-22-196.dsl.pltn13.sbcglobal.net): 1 time root: 63.249.86.11 (dsl-63-249-86-11.cruzio.com): 3 times 70.132.20.25 (adsl-70-132-20-25.dsl.snfc21.sbcglobal.net): 1 time rsimms: 63.249.86.11 (dsl-63-249-86-11.cruzio.com): 2 times
the bad boys trying to break in … this is why you need strong passwords
48
CIS 191 - Lesson 12
/var/log/secure
The bad boys trying to break in as root … this is why you need strong passwords
Nov 30 06:02:24 opus sshd[27486]: Failed password for root from 202.119.60.132 port 36322 ssh2Nov 30 06:02:24 opus sshd[27487]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:27 opus sshd[27488]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=rootNov 30 06:02:29 opus sshd[27488]: Failed password for root from 202.119.60.132 port 36846 ssh2Nov 30 06:02:29 opus sshd[27489]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:32 opus sshd[27490]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=rootNov 30 06:02:33 opus sshd[27490]: Failed password for root from 202.119.60.132 port 37480 ssh2Nov 30 06:02:34 opus sshd[27491]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:36 opus sshd[27492]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=rootNov 30 06:02:38 opus sshd[27492]: Failed password for root from 202.119.60.132 port 38030 ssh2Nov 30 06:02:39 opus sshd[27493]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:42 opus sshd[27494]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=rootNov 30 06:02:43 opus sshd[27494]: Failed password for root from 202.119.60.132 port 38679 ssh2Nov 30 06:02:43 opus sshd[27495]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:46 opus sshd[27496]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=rootNov 30 06:02:48 opus sshd[27496]: Failed password for root from 202.119.60.132 port 39448 ssh2Nov 30 06:02:48 opus sshd[27497]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:50 opus sshd[27498]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=root
50
CIS 191 - Lesson 12
/var/log/secure
The bad boys trying to break in, guessing usernames … this is why you need strong passwords
Nov 30 06:27:20 opus sshd[28166]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn
Nov 30 06:27:20 opus sshd[28166]: pam_succeed_if(sshd:auth): error retrieving information about user shop
Nov 30 06:27:23 opus sshd[28166]: Failed password for invalid user shop from 202.119.60.132 port 40634 ssh2
Nov 30 06:27:23 opus sshd[28167]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:27:25 opus sshd[28168]: Invalid user lady from 202.119.60.132Nov 30 06:27:25 opus sshd[28169]: input_userauth_request: invalid user ladyNov 30 06:27:25 opus sshd[28168]: pam_unix(sshd:auth): check pass; user unknownNov 30 06:27:25 opus sshd[28168]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cnNov 30 06:27:25 opus sshd[28168]: pam_succeed_if(sshd:auth): error retrieving information about
user ladyNov 30 06:27:28 opus sshd[28168]: Failed password for invalid user lady from 202.119.60.132
port 41408 ssh2Nov 30 06:27:28 opus sshd[28169]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:27:30 opus sshd[28170]: Invalid user lady from 202.119.60.132Nov 30 06:27:30 opus sshd[28171]: input_userauth_request: invalid user ladyNov 30 06:27:30 opus sshd[28170]: pam_unix(sshd:auth): check pass; user unknownNov 30 06:27:30 opus sshd[28170]: pam_unix(sshd:auth): authentication failure; logname= uid=0
euid=0 tty=ssh ruser= rhost=cool.nju.edu.cnNov 30 06:27:30 opus sshd[28170]: pam_succeed_if(sshd:auth): error retrieving information about
user lady
51