Lesson 13Unix Forensics
UTSA IS 6353 Incident Response
Overview
• Basic MO (email sent)• System Tools• Files System Tools• Using the Tools
UTSA IS 6353 Incident Response
Basic MO
• Only use tools you can trust• Consider using tools stored off-line on read-
only medium• Don’t forget a rogue kernel module can make
a compromised system look clean• Hackers who use Unix are considered to be
above average threats, innovative, and persistent
UTSA IS 6353 Incident Response
Basic System Response Tools(1)
• vmstat: provides quick look a memory, CPU, and disk subsystem statistics
• mpstat: provides processor utilization statistics
• ps: process status, shows processes executing on the system and information about them
• top: like ps, but also orders the processes by CPU consumption
UTSA IS 6353 Incident Response
Basic System Response Tools(2)
• iostat: provides statistics on disk subsystem
• sar: displays system perfomance data stored over-time
• lastcom: displays commands that have been run on the system
UTSA IS 6353 Incident Response
Basic File Tools
• lsof: list open files, show all files the operating system has open
• file: outputs a list of hex values that indicate the file contents
• readelf: displays the executable linking and format headers of a binary file so you can determine what functions the executable performs
UTSA IS 6353 Incident Response
Basic File Tools(2)
• ldd: reads the contents of the ELF headers to show what shared object libraries the executable depends
• strace: starts a process or attaches to a currently running process so you can see all the system calls the process is making(Solaris: truss, BSD: ktrace)
UTSA IS 6353 Incident Response
Basic Search Tools(2)
• strings: shows ASCII strings >= 4 characters or longer (user specified) in a file
• find: recursively searches for objects on the file system—can be useful for finding files or directories based on user specified criteria
• grep: global regular expression print—searches a pattern sapce (file) for a user specified pattern
UTSA IS 6353 Incident Response
Basic Premise of the Hack
• Target Machine is s Webserver running Apache
• A Webmaster calls the Incident Response Team to say that the CPU is busy on the server which gets very few daily hits
• Your job to prove or dispel that an incident has occured
UTSA IS 6353 Incident Response
Tool Usage to Uncover Hacker
• Step One: On-line Analysis or off-line analysis? On-line!
• vmstat: shows a user process is running• mpstat: memory not being taxed• iostat: very little disk activity• sar: shows CPU started being used at 03:17• Who could use the CPU at 03:17?• SysAdmin? Easy to confirm
UTSA IS 6353 Incident Response
Tool Usage to Uncover Hacker(2)
• lastcom: shows FTP was run by root several times
• last: shows recent logins, but it doesn’t show where logged in from
• top –d1: shows process Apache using CPU 100%
• ps –eflcym –headers (for linux)– Shows Apache and multiple httpd processes
UTSA IS 6353 Incident Response
Tool Usage to Uncover Hacker(3)
• lsof – p– Shows apache process has a file john.pot open– Google search on john.pot points to John the
Ripper– Good enough for me to say an incident has
occurred—but why stop now?
• file apache– Shows apache is an ELF executable
UTSA IS 6353 Incident Response
Tool Usage to Uncover Hacker(4)
• ldd ./apache– Shows fewer binary links
• ldd /usr/sbin/httpd– Shows many more binary links
• Looks like apache is a rogue process• String apache|grep – i john
– Shows apache is probably the password cracker John the Ripper
• So Why would the Webserver need John the Ripper?
UTSA IS 6353 Incident Response
So What Do We Know?
• The password cracker John the Ripper found on our Web server
• Hack “appears” to have occurred at 03:17 AM
• But don’t be fooled by time hacks—what is server tine synced with?
• Did hacker change the time?
UTSA IS 6353 Incident Response
Things to Consider
• Tools we will use execute in user space• If root kit changed binaries then tool
results could be inaccurate– Do off-line analysis if you suspect
• Best approach on “live” systems is to execute from read only media– Ensures the tools aren’t modified by hacker– Knoppix Linux is another useful tool
UTSA IS 6353 Incident Response
Off-line Analysis
• Unix allows you to mount volumes in read-only mode
mount –o loop=/dev/loop0 /var/tmp/testfilesystem /mnt
• To find out which file systems supportedfind /lib/modules/’uname – r’ /kernel/fs/* - type f|grep – v ‘n/s\/’
• To see what file systems are loaded
grep –v ‘^nodev’ /proc/filesystems
UTSA IS 6353 Incident Response
Mounting A File System Read-Onlymounting a file system image in Linux:
##==##== mount the image read-only so that the image doesn't change on disk# mount -o ro,loop=/dev/loop0 /var/space/images/2003_02_17_linux.bin /mnt##==##== list of files# ls -la /mnttotal 146drwxr-xr-x 19 root root 1024 Feb 17 2003 .drwxr-xr-x 19 root root 4096 Sep 12 11:55 ..-rw-r--r-- 1 root root 0 Sep 12 11:55 .autofsck-rw------- 1 root root 186 Sep 6 19:58 .bash_historydrwxr-xr-x 2 root root 2048 Feb 8 2003 bindrwxr-xr-x 3 root root 1024 Sep 6 19:59 bootdrwxr-xr-x 20 root root 116736 Feb 17 2003 devdrwxr-xr-x 41 root root 3072 Sep 12 14:13 etc[ output deleted ]##==##== unmount /mnt# umount /mnt
UTSA IS 6353 Incident Response
Tools to Detect Changes
NATIVE• Linux
– rpm –Vva
• Solaris (9.0)– pkgchk –vn
• Debian (3.0)– Debsums -ac
3rd Party Tools• AIDE• Integrit• Osiris• Tripwire
Tools run periodically and rely on hashes or checksums
UTSA IS 6353 Incident Response
Tools to Detect in Real-time
• changedfiles
• dnotify
• FAM
• All use kernel modules or poll the file system to detect changes in near realtime
UTSA IS 6353 Incident Response
MAC Times
• Modify, access, change times
• Unix files systems sotre a set of timestamps in their file system metadata
• Timestamps stored in inode
• mtime, atime, ctime
How common commands change MACtimes for a directory (foo):
Action atime ctime mtime
creation (mkdir foo) X X X
directory move (mv foo bar) X X
file creation (touch foo/foo) X X
file creation (dd if=/dev/zero of=foo/foo count=1) X X
list directory (ls foo) X
change directory (cd foo)
file test (-f foo)
file move/rename (mv foo foo_mvd) X X
permissions change (chmod/chown <some_perm> foo) X
file copy (mv foo_mvd foo) X X
file edit (vim foo) X X
file edit (emacs foo) X X X
file edit (nvi/nano foo)
How common commands change MACtimes for a file (f1):
Action atime ctime mtime
creation (touch foo) X X X
creation (dd if=/dev/zero of=foo count=1) X X X
rename (mv foo bar)
permissions change (chmod <some_perm> foo) X
copy (cp foo bar) X
copy overwrite (cp bar foo) X X
append (cat >> foo) X X
overwrite (cat > foo) X X
truncate (cp /dev/null foo) X X
list file (ls foo)
edit (vim/emacs/xemacs/joe/jed foo) X X X
edit (ed/nvi/vi (sun)/vi (obsd)/nano/pico foo) X1 X1 X1
1 - all times changed, but atime is slightly older than mtime and ctime
UTSA IS 6353 Incident Response
Using ls to Display MAC times
displaying MACtimes using ls:
Linux (ls from GNU fileutils) OpenBSD Solaris
mtime ls -latr --full-time ls -latTr ls -latr
atime ls -laur --full-time ls -lauTr ls -laur
ctime ls -lacr --full-time ls -lacTr ls -lacr
UTSA IS 6353 Incident Response
So Now What?
• Search filesystem for files that changed/modified around suspect time
# find / \(-mtime +2 –a –mtime -7\) –a –printf “M:%t C:%c %i\t%p\n”
Or# find / \(-mtime +2 –a –mtime -7\)|perl –ne ‘chomp; ($i,$m,$c)=(stat)[1,9,10]; printf “M:%s\t$i\t$_\n”,localtime($m).”C:”.localtime($c)’
• Finding broken suid or sgid binary to escalate privilegefind / -perm -6000 -ls
UTSA IS 6353 Incident Response
Using Findthe find command in action:
##==##== find all files which have had their status changed in##== the last 24 hrs (display ctime, inode, and filename)# find / -ctime -1 -printf "%c %i\t%p\n"Fri Sep 7 11:55:14 2003 174945 /var/lib/rpmFri Sep 7 11:55:14 2003 174946 /var/lib/rpm/__db.001Fri Sep 7 11:55:17 2003 174947 /var/lib/rpm/__db.002Fri Sep 7 11:55:17 2003 174948 /var/lib/rpm/__db.003Fri Sep 7 11:55:55 2003 160125 /var/lib/random-seedFri Sep 7 11:55:16 2003 222706 /var/logFri Sep 7 12:17:05 2003 224545 /var/log/messages[ output deleted ]##==##== find all files which have had their status changed from##== 5 to 30 (inclusive) days ago and display the ctime, inode, and filename# find / -ctime +4 -ctime -31 -printf "%c %i\t%p\n"Sat Aug 30 19:49:52 2003 414661 /boot/System.map-2.4.20-20.8bigmemSat Aug 30 19:49:52 2003 414662 /boot/config-2.4.20-20.8bigmemSat Aug 30 19:49:52 2003 414663 /boot/module-info-2.4.20-20.8bigmemSat Aug 30 19:49:53 2003 414664 /boot/vmlinux-2.4.20-20.8bigmemSat Aug 30 19:49:53 2003 414665 /boot/vmlinuz-2.4.20-20.8bigmem[ output deleted ]
UTSA IS 6353 Incident Response
File System Debugger
• fsdb – OpenBSD and Solaris
• debugfs – ext2 and ext3 (Linux)
• Debugfs <image path>
UTSA IS 6353 Incident Response
Steps Investigator Took
• First Key Step: on-line or off-line
• If off-line mount in read-only mode
• Use trusted tools
• Use MAC analysis
• Use File System Debugger
UTSA IS 6353 Incident Response
Summary
• Some say Unix Forensics are tougher than Windows Forensics
• I agree the command usage is cryptic at time, but a tool is a tool
• Tools presented are straight forward
• Practice makes perfect