lesson 13 unix forensics. utsa is 6353 incident response overview basic mo (email sent) system tools...

Lesson 13Unix Forensics

UTSA IS 6353 Incident Response

Overview

• Basic MO (email sent)• System Tools• Files System Tools• Using the Tools


Basic MO

• Only use tools you can trust• Consider using tools stored off-line on read-

only medium• Don’t forget a rogue kernel module can make

a compromised system look clean• Hackers who use Unix are considered to be

above average threats, innovative, and persistent


Basic System Response Tools(1)

• vmstat: provides quick look a memory, CPU, and disk subsystem statistics

• mpstat: provides processor utilization statistics

• ps: process status, shows processes executing on the system and information about them

• top: like ps, but also orders the processes by CPU consumption


Basic System Response Tools(2)

• iostat: provides statistics on disk subsystem

• sar: displays system perfomance data stored over-time

• lastcom: displays commands that have been run on the system


Basic File Tools

• lsof: list open files, show all files the operating system has open

• file: outputs a list of hex values that indicate the file contents

• readelf: displays the executable linking and format headers of a binary file so you can determine what functions the executable performs


Basic File Tools(2)

• ldd: reads the contents of the ELF headers to show what shared object libraries the executable depends

• strace: starts a process or attaches to a currently running process so you can see all the system calls the process is making(Solaris: truss, BSD: ktrace)


Basic Search Tools(2)

• strings: shows ASCII strings >= 4 characters or longer (user specified) in a file

• find: recursively searches for objects on the file system—can be useful for finding files or directories based on user specified criteria

• grep: global regular expression print—searches a pattern sapce (file) for a user specified pattern


Basic Premise of the Hack

• Target Machine is s Webserver running Apache

• A Webmaster calls the Incident Response Team to say that the CPU is busy on the server which gets very few daily hits

• Your job to prove or dispel that an incident has occured


Tool Usage to Uncover Hacker

• Step One: On-line Analysis or off-line analysis? On-line!

• vmstat: shows a user process is running• mpstat: memory not being taxed• iostat: very little disk activity• sar: shows CPU started being used at 03:17• Who could use the CPU at 03:17?• SysAdmin? Easy to confirm


Tool Usage to Uncover Hacker(2)

• lastcom: shows FTP was run by root several times

• last: shows recent logins, but it doesn’t show where logged in from

• top –d1: shows process Apache using CPU 100%

• ps –eflcym –headers (for linux)– Shows Apache and multiple httpd processes



• lsof – p– Shows apache process has a file john.pot open– Google search on john.pot points to John the

Ripper– Good enough for me to say an incident has

occurred—but why stop now?

• file apache– Shows apache is an ELF executable



• ldd ./apache– Shows fewer binary links

• ldd /usr/sbin/httpd– Shows many more binary links

• Looks like apache is a rogue process• String apache|grep – i john

– Shows apache is probably the password cracker John the Ripper

• So Why would the Webserver need John the Ripper?


So What Do We Know?

• The password cracker John the Ripper found on our Web server

• Hack “appears” to have occurred at 03:17 AM

• But don’t be fooled by time hacks—what is server tine synced with?

• Did hacker change the time?


Things to Consider

• Tools we will use execute in user space• If root kit changed binaries then tool

results could be inaccurate– Do off-line analysis if you suspect

• Best approach on “live” systems is to execute from read only media– Ensures the tools aren’t modified by hacker– Knoppix Linux is another useful tool


Off-line Analysis

• Unix allows you to mount volumes in read-only mode

mount –o loop=/dev/loop0 /var/tmp/testfilesystem /mnt

• To find out which file systems supportedfind /lib/modules/’uname – r’ /kernel/fs/* - type f|grep – v ‘n/s\/’

• To see what file systems are loaded

grep –v ‘^nodev’ /proc/filesystems


Mounting A File System Read-Onlymounting a file system image in Linux:

##==##== mount the image read-only so that the image doesn't change on disk# mount -o ro,loop=/dev/loop0 /var/space/images/2003_02_17_linux.bin /mnt##==##== list of files# ls -la /mnttotal 146drwxr-xr-x 19 root root 1024 Feb 17 2003 .drwxr-xr-x 19 root root 4096 Sep 12 11:55 ..-rw-r--r-- 1 root root 0 Sep 12 11:55 .autofsck-rw------- 1 root root 186 Sep 6 19:58 .bash_historydrwxr-xr-x 2 root root 2048 Feb 8 2003 bindrwxr-xr-x 3 root root 1024 Sep 6 19:59 bootdrwxr-xr-x 20 root root 116736 Feb 17 2003 devdrwxr-xr-x 41 root root 3072 Sep 12 14:13 etc[ output deleted ]##==##== unmount /mnt# umount /mnt


Tools to Detect Changes

NATIVE• Linux

– rpm –Vva

• Solaris (9.0)– pkgchk –vn

• Debian (3.0)– Debsums -ac

3rd Party Tools• AIDE• Integrit• Osiris• Tripwire

Tools run periodically and rely on hashes or checksums


Tools to Detect in Real-time

• changedfiles

• dnotify

• FAM

• All use kernel modules or poll the file system to detect changes in near realtime


MAC Times

• Modify, access, change times

• Unix files systems sotre a set of timestamps in their file system metadata

• Timestamps stored in inode

• mtime, atime, ctime

How common commands change MACtimes for a directory (foo):

Action atime ctime mtime

creation (mkdir foo) X X X

directory move (mv foo bar) X X

file creation (touch foo/foo) X X

file creation (dd if=/dev/zero of=foo/foo count=1) X X

list directory (ls foo) X

change directory (cd foo)

file test (-f foo)

file move/rename (mv foo foo_mvd) X X

permissions change (chmod/chown <some_perm> foo) X

file copy (mv foo_mvd foo) X X

file edit (vim foo) X X

file edit (emacs foo) X X X

file edit (nvi/nano foo)

How common commands change MACtimes for a file (f1):

Action atime ctime mtime

creation (touch foo) X X X

creation (dd if=/dev/zero of=foo count=1) X X X

rename (mv foo bar)

permissions change (chmod <some_perm> foo) X

copy (cp foo bar) X

copy overwrite (cp bar foo) X X

append (cat >> foo) X X

overwrite (cat > foo) X X

truncate (cp /dev/null foo) X X

list file (ls foo)

edit (vim/emacs/xemacs/joe/jed foo) X X X

edit (ed/nvi/vi (sun)/vi (obsd)/nano/pico foo) X1 X1 X1

1 - all times changed, but atime is slightly older than mtime and ctime


Using ls to Display MAC times

displaying MACtimes using ls:

Linux (ls from GNU fileutils) OpenBSD Solaris

mtime ls -latr --full-time ls -latTr ls -latr

atime ls -laur --full-time ls -lauTr ls -laur

ctime ls -lacr --full-time ls -lacTr ls -lacr


So Now What?

• Search filesystem for files that changed/modified around suspect time

# find / $-mtime +2 –a –mtime -7$ –a –printf “M:%t C:%c %i\t%p\n”

Or# find / $-mtime +2 –a –mtime -7$|perl –ne ‘chomp; ($i,$m,$c)=(stat)[1,9,10]; printf “M:%s\t$i\t$_\n”,localtime($m).”C:”.localtime($c)’

• Finding broken suid or sgid binary to escalate privilegefind / -perm -6000 -ls


Using Findthe find command in action:

##==##== find all files which have had their status changed in##== the last 24 hrs (display ctime, inode, and filename)# find / -ctime -1 -printf "%c %i\t%p\n"Fri Sep 7 11:55:14 2003 174945 /var/lib/rpmFri Sep 7 11:55:14 2003 174946 /var/lib/rpm/__db.001Fri Sep 7 11:55:17 2003 174947 /var/lib/rpm/__db.002Fri Sep 7 11:55:17 2003 174948 /var/lib/rpm/__db.003Fri Sep 7 11:55:55 2003 160125 /var/lib/random-seedFri Sep 7 11:55:16 2003 222706 /var/logFri Sep 7 12:17:05 2003 224545 /var/log/messages[ output deleted ]##==##== find all files which have had their status changed from##== 5 to 30 (inclusive) days ago and display the ctime, inode, and filename# find / -ctime +4 -ctime -31 -printf "%c %i\t%p\n"Sat Aug 30 19:49:52 2003 414661 /boot/System.map-2.4.20-20.8bigmemSat Aug 30 19:49:52 2003 414662 /boot/config-2.4.20-20.8bigmemSat Aug 30 19:49:52 2003 414663 /boot/module-info-2.4.20-20.8bigmemSat Aug 30 19:49:53 2003 414664 /boot/vmlinux-2.4.20-20.8bigmemSat Aug 30 19:49:53 2003 414665 /boot/vmlinuz-2.4.20-20.8bigmem[ output deleted ]


File System Debugger

• fsdb – OpenBSD and Solaris

• debugfs – ext2 and ext3 (Linux)

• Debugfs <image path>


Steps Investigator Took

• First Key Step: on-line or off-line

• If off-line mount in read-only mode

• Use trusted tools

• Use MAC analysis

• Use File System Debugger


Summary

• Some say Unix Forensics are tougher than Windows Forensics

• I agree the command usage is cryptic at time, but a tool is a tool

• Tools presented are straight forward

• Practice makes perfect

lesson 13 unix forensics. utsa is 6353 incident response overview basic mo (email sent) system tools...

Documents

process apache

binary file

user process

file apacheshows apache

file contentsreadelf

file systemcan

compromised system

process status