© 2007 carnegie mellon university silk installation joe mcmanus [email protected] a walk-through

© 2007 Carnegie Mellon University

SiLK Installation

Joe [email protected]

A Walk-through

2© 2007 Carnegie Mellon University

Topics

Topics Covered in this Training• What SiLK is and is not• SiLK on a Box• SiLK with remote flow collection• Building SiLK RPMs• Monitoring SiLK Processes• Basic SiLK Queries

— Useful queries for an administrator


Introduction to SiLK

System for internet Level KnowledgeData collection Data analysis

NOT:Intrusion detection/prevention systemAudit tools for your networkAutomated report generator


Collection & Analysis Infrastructure

SiLK stores the data and allows access to the data for analysis.

YAF turns packets into flows for SiLK over IPFIX.

SiLK can take input from IPFIX or Netflow.

Netflow is convenient as it implemented on many routers and requires no additional hardware.


Collection & Analysis Infrastructure


SiLK on a BoxStandalone flow collection


Standalone Collection and Analysis

We will now go over installing SiLK and YAF to create a stand alone flow collection and analysis box.

We will assume you are on a linux server with root access.


Downloading SiLK

SiLK can be downloaded from:

http://tools.netsa.cert.org/silk/

YAF is available from:

http://tools.netsa.cert.org/yaf/

Fixbuf is available from:

http://tools.netsa.cert.org/fixbuf/

http://tools.netsa.cert.org/silk/

http://tools.netsa.cert.org/yaf/

http://tools.netsa.cert.org/fixbuf


Downloading (continued…)

Log in to your Linux server and use wget to download the software. [root@silk tmp]#cd /tmp

[root@silk tmp]# wget \ http://tools.netsa.cert.org/releases/silk-1.1.3.tar.gz

[root@silk tmp]# wget \ http://tools.netsa.cert.org/releases/libfixbuf-0.8.0.tar.gz

[root@silk tmp]# wget \ http://tools.netsa.cert.org/releases/yaf-1.0.0.tar.gz


Build fixbuf

Fixbuf is required to have YAF talk to SiLK over IPFIX.

[root@silk tmp]# tar -zxvf libfixbuf-0.8.0.tar.gz

[root@silk tmp]# cd libfixbuf-0.8.0

[root@silk libfixbuf-0.8.0]# ./configure && make && make install


Errors

checking for cc... no

checking for cl... no

configure: error: no acceptable C compiler found in $PATH

We need to make sure the system has the required packages to build the suite.


YUM

We can use YUM to install the missing C compiler. [root@silk libfixbuf-0.8.0]# yum install gcc

Try building Fixbuf again.

Which brings us to the next error: checking whether we are using the GNU C++ compiler... no

checking whether g++ accepts -g... no

checking dependency style of g++... none


YUM

If you do not know which package you are looking for YUM can help with this.

[root@silk libfixbuf-0.8.0]# yum search c++

[root@silk libfixbuf-0.8.0]# yum install gcc-c++


RPM Confusion

[root@silk libfixbuf-0.8.0]# ./configure

checking for GLIB - version >= 2.4.7... no

*** Could not run GLIB test program, checking why...

*** The test program failed to compile or link. See the file config.log for the

*** exact error that occured. This usually means GLIB is incorrectly installed.

configure: error: Cannot find a suitable glib2 (>= 2.4.7)

This error is misleading.


RPM Confusion

[root@silk libfixbuf-0.8.0]# yum install glib2

Loaded plugins: refresh-packagekit

Setting up Install Process

Parsing package install arguments

Package glib2-2.16.6-2.fc9.i386 already installed and latest version

Nothing to do

[root@silk ~]# rpm -qa | grep glib2

glib2-2.16.6-2.fc9.i386

Wuh?!?


RPM Confusion

What is actually missing is the glib2-devel rpm. [root@silk ~]# yum install glib2-devel

You may have to install the libpcap-devel rpm as well. This is dependent on which install of linux you have done.


Fixbuf install


[root@silk libfixbuf-0.8.0]# make

[root@silk libfixbuf-0.8.0]# make install

[root@silk libfixbuf-0.8.0]# ls /usr/local/lib

libfixbuf-0.8.0.so.8

libfixbuf.a

libfixbuf.so

libfixbuf-0.8.0.so.8.0.0

libfixbuf.la

pkgconfig


YAF Install

YAF is our tool for flow collection, there are others.[root@silk libfixbuf-0.8.0]# cd ..

[root@silk tmp]# tar -zxvf yaf-1.0.0.tar.gz

[root@silk tmp]# cd yaf-1.0.0

checking for libfixbuf >= 0.7.2... configure: error: Cannot find a suitable libfixbuf (>= 0.7.2): Package libfixbuf was not found in the pkg-config search path.

Perhaps you should add the directory containing `libfixbuf.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libfixbuf' found


YAF Install

Didn’t we just install fixbuf?

-Yes, but pkgconfig does not know to look in /usr/local.[root@silk yaf-1.0.0]# export \ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

[root@silk yaf-1.0.0]# ./configure

[root@silk yaf-1.0.0]# make

[root@silk yaf-1.0.0]# make install


SiLK Install

Uncompress the SiLK tarball. [root@silk yaf-1.0.0]# cd ..

[root@silk tmp]# tar -zxvf silk-1.1.3.tar.gz

[root@silk tmp]# cd silk-1.1.3

We will build in support for PySiLK. [root@silk silk-1.1.3]# yum install python-devel


SiLK Install (cont…)

[root@silk silk-1.1.3]# ./configure \

--with-libfixbuf=/usr/local/lib/pkgconfig/ \

--with-python

SiLK creates a great summary.


Summary* Configured package: SiLK 1.1.3

* Host type: i686-pc-linux-gnu

* Source files ($top_srcdir): .

* Install directory: /usr/local

* Root of packed data tree: /data

* Packing logic: via run-time plugin

* Timezone support: UTC

* Default compression method: SK_COMPMETHOD_NONE

* IPv6 support: NO

* IPFIX collection support: YES (-pthread -L/usr/local/lib -lfixbuf -lgthread-2.0-lrt -lglib-2.0)

* Transport encryption support: NO (gnutls not found)

* IPA support: NO

* LIBPCAP support: YES (-lpcap)

* Python support: YES (-L/usr/kerberos/lib -Xlinker -export-dynamic -ld -lutil -lm -L/usr/lib -lpython2.5 -pthread)

*Python package destination: /usr/lib/python2.5/site-packages

* Build analysis tools: YES

* Build packing tools: YES

* Compiler (CC): gcc

* Compiler flags (CFLAGS): -I$(top_srcdir)/src/include -DNDEBUG -D_GNU_SOURCE=1

-D_FILE_OFFSET_BITS=64 -O3 -fno-strict-aliasing -Wall -W -Wmissing-prototypes -Wformat=2

-Wdeclaration-after-statement

* Linker flags (LDFLAGS):

* Libraries (LIBS): -ldl -lm


SiLK Install (cont…)

[root@silk silk-1.1.3]# make

[root@silk silk-1.1.3]# make install

[root@silk silk-1.1.3]# rwfilter --version

rwfilter: part of SiLK 1.1.3; configuration settings: * Root of packed data tree: /data

* Packing logic: Run-time plug-in

* Timezone support: UTC

* Available compression methods: none [default]

* IPv6 support: no

* IPFIX collection support: yes

* Transport encryption: no

* PySiLK support: /usr/lib/python2.5/site-packages

* Enable assert(): no


SiLK Configuration

We need to create a place to store the flow data.

The default for is /data.

[root@silk ~]# mkdir /data

We will need to create two configuration files for SiLK, sensor.conf and silk.conf


silk.conf

A few sample silk.conf files are included with the distribution. They are installed under /usr/local/share/silk.

We would want to use twoway-silk.conf[root@silk /data]# cp \ /usr/local/share/silk/twoway -silk.conf .


silk.conf

[root@silk data]# vi /data/silk.conf

#Define Sensorssensor 0 localhost

class all sensors localhostend class

version 1


SiLK Configuration

class all type 0 in in type 1 out out type 2 inweb iw type 3 outweb ow type 4 innull innull type 5 outnull outnull type 6 int2int int2int type 7 ext2ext ext2ext type 8 inicmp inicmp type 9 outicmp outicmp type 10 other other

default-types in inweb inicmpend class


silk.conf storage heirarchy


silk.conf storage hierarchy.

I find it good practice to have silk store its data under /data/SENSOR-NAME/type.

The benefit of storing data this way is that you can use unix groups to control access to flow data on each sensor.

i.e.

/data/engineering

/data/sales

/data/logistics


silk.conf

# The default path format from SILK_DATA_ROOTDIRpath-format "%N/%T/%Y/%m/%d/%x“

%N= Sensor Name

%T=Type In/Out/int2int…

%Y=Year

%m=month

%d=data

%x=flowtype-sensor_YearMonthDay.Hour


silk.conf

# The plug-in to load to get the packing logic to use in rwflowpack.# The --packing-logic switch to rwflowpack will override this value.# If SiLK was configured with hard-coded packing logic, this value is# ignored.

# The plug-in to load to get the packing logic to use in rwflowpack.# The --packing-logic switch to rwflowpack will override this value.# If SiLK was configured with hard-coded packing logic, this value is# ignored.packing-logic "packlogic-twoway.so“


sensor.conf

We will now have to edit the sensor.conf file.

[root@silk data]# vi /data/sensor.conf

probe localhost ipfix listen-on-port 18001 protocol tcp accept-from-host 127.0.0.1end probe


sensor.conf

sensor localhost ipfix-probes localhost internal-ipblock 192.168.1.0/24 external-ipblock remainderend sensor


rwflowpack

rwflowpack is the daemon that will listen for traffic from YAF.

A sample file is included, which we will customize. [root@silk]# cp \ /usr/local/share/silk/etc/rwflowpack.conf \ /usr/local/etc/.

[root@silk data]# vi /usr/local/etc/rwflowpack.conf


rwflowpack.conf

Change the following values:

ENABLED=yes

SENSOR_CONFIG=/data/sensor.conf

SITE_CONFIG=/data/silk.conf

LOG_TYPE=legacy

LOG_DIR=/var/log

CREATE_DIRECTORIES=yes

COMPRESSION_TYPE=best


rwflowpack.conf

To make rwflowpack start on boot we can add it using chkconfig. SiLK includes sample startup scripts with the distribution. [root@silk data]# cp /usr/local/share/silk/etc/init.d/rwflowpack /etc/init.d

[root@silk data]# chkconfig rwflowpack on

[root@silk data]# chkconfig --list rwflowpack

rwflowpack 0:off 1:off 2:on 3:on 4:on 5:on 6:off


Test YAF

Test that YAF is able to listen on the interface.

[root@silk data]# yaf --live=pcap --in=eth0 --out=- -v | yafscii

[2009-01-08 17:06:10] yaf starting

[2009-01-08 17:06:10] running as root in --live mode, but not dropping privilege


Test YAF

Let that run for a minute. Then enter CTRL-C.C2009-01-08 17:07:19.900 tcp 10.0.0.2:1005 => 10.0.0.3:2049 90b24967:0b0cbdd9 AF/A:AF/0 (2/104 <-> 1/52) rtt 0 ms

[2009-01-08 17:07:42] Processed 57 packets into 9 flows:

[2009-01-08 17:07:42] Mean flow rate 0.18/s.

[2009-01-08 17:07:42] Mean packet rate 1.11/s.

[2009-01-08 17:07:42] Virtual bandwidth 0.0008 Mbps.

[2009-01-08 17:07:42] Maximum flow table size 8.

[2009-01-08 17:07:42] 6 flush events.

[2009-01-08 17:07:42] Rejected 1 out-of-sequence packets.

[2009-01-08 17:07:42] Assembled 0 fragments into 0 packets:

[2009-01-08 17:07:42] Expired 0 incomplete fragmented packets.

[2009-01-08 17:07:42] Maximum fragment table size 0.

[2009-01-08 17:07:42] Rejected 44 packets during decode:

[2009-01-08 17:07:42] 44 due to unsupported/rejected packet type:

[2009-01-08 17:07:42] 44 unsupported/rejected Layer 3 headers.

[2009-01-08 17:07:42] yaf terminating


IPTables

You must open the firewall for YAF to connect to rwflowpack. [root@silk data]# iptables -I INPUT \

-s 127.0.0.1 -p tcp -m tcp \

--dport 18001 -j ACCEPT

[root@silk data]# service iptables save

iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]


IPTables

We can check to see if the rule has been added correctly.[root@silk data]# service iptables status

Table: filter

Chain INPUT (policy ACCEPT)

num target prot opt source destination

1 ACCEPT tcp -- 127.0.0.1 0.0.0.0/0 tcp dpt:18001

….


Starting YAF

[root@silk data]# yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001 &


Starting YAF


The silk option tells YAF to format the output for silk.


Starting YAF


The ipfix=tcp option tells YAF use ipfix over tcp.

The ipfix-port=18001 option specifies which port.


Starting YAF


The live=pcap option tells YAF the capture device uses pcap.

The only other option is dag for use with Endace Dag cards.


Starting YAF


The in=eth0 option tells YAF which input device to use.

If you are using a dag card, the option would be dag0.


Starting YAF


The out=127.0.0.1 option tells YAF where to send its output.


Checking on YAF

You can check the status of YAF with the following command:[root@silk data]# kill -SIGUSR1 `pgrep yaf`

[2009-01-08 21:53:29] Rejected 1 out-of-sequence packets.


Checking on The Collection Process

When you first start yaf one of the first places to look is in the log file under /var/log. [root@silk data]# tail /var/log/rwflowpack-20090108.log

Jan 8 16:53:40 silk rwflowpack[10605]: Flushing files after 120 seconds.

Jan 8 16:54:20 silk rwflowpack[10605]: Opening new output file /data/localhost/ext2ext/2009/01/08/ext2ext-localhost_20090108.21



Did yaf start correctly? [root@silk data]# ps -ef | grep yaf

root 10630 10475 0 16:52 pts/1 00:00:00 yaf --silk --ipfix=tcp --live=pcap --in=eth0 --out=127.0.0.1 --ipfix-port=18001



If not can yaf connect to rwflowpack?

[root@silk data]# telnet localhost 18001

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

Is rwflowpack running?

[root@silk data]# ps -ef | grep yaf


Try it out

Lets try out a simple rwfilter command to see if it is working:[root@silk data]# rwfilter --type=all --proto=0- --pass=stdout | rwcut | head sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| dur| eTime| sensor| 10.0.0.25| 10.0.0.6|43393| 22| 6| 6| 332|FSRPA |2009/01/08T21:53:25.789| 0.023|2009/01/08T21:53:25.812 localhost| 10.0.0.6| 10.0.0.25| 22|43393| 6| 4| 972| S PA |2009/01/08T21:53:25.789| 0.023|2009/01/08T21:53:25.812|localhost| 10.0.0.6| 10.0.0.25| 22|43393| 6| 1| 52|F A |2009/01/08T21:53:25.812| 0.000|2009/01/08T21:53:25.812|localhost| 10.0.0.25| 10.0.0.6|43393| 22| 6| 1| 40| R |2009/01/08T21:53:25.812| 0.000|2009/01/08T21:53:25.812|localhost|


RoutersUsing routers as a data feed


Receiving data from a router

Another common way of receiving data is from a router on a mirror port or span port.

In Cisco IOS this is called a span port and if very easy to configure.


Span Port

IOS Example: interface FastEthernet0/24

port monitor FastEthernet0/1


…..


port monitor VLAN1


YAF & Span port

You would set up sensor.conf with the same options as in the previous example. Although you would want to have a second interface on the box listening for it set up in promiscuous mode with arp disabled as well.

Yaf would also start with the same options as before, but pointing to your second nic.


SiLK & Netflow

Silk can take netflow directly from a netflow compatible router.

You would have to open your firewall to accept UDP on the port you decide to use and make changes to sensor.conf as show in the next slide.


SiLK & Netflow

probe S2 netflow-v5

listen-on-port 9901

protocol udp accept-from-host 172.16.22.22

end probe

sensor S2

netflow-v5-probes S2

internal-ipblock 128.2.0.0/16

external-ipblock remainder

end sensor


YAF & DAG Cards

YAF is capable of including support for Endace DAG Cards.

It is it as simple as adding -with-dag option to ./configure.

When YAF is started you would use the following options: [root@silk data]# yaf--silk --ipfix=tcp --live=dag --in=dag0 --out=127.0.0.1 --ipfix-port=18001


Working With TapsTunnelling


Working With Taps


Tunneling

YAF can only listen on one interface at a time. If we are using YAF with a tap one interface will receive inbound and one would receive outbound traffic. For some programs like snort this is useful.

But with SiLK it classifies traffic by CIDR block in sensor.conf.

We can use port bonding on Linux to solve this issue.


Tunneling

Fedora 9 works fine with the Trendnet usb ethernet adapter available at Radio Shack.

http://www.radioshack.com/product/index.jsp?productId=2806154




Tunneling

Plug in both adapters and run dmesg looking for the mac address to use in the ifcfg-ethX file. [root@silk ~]# dmesg | tail -20 usb 1-8: new high speed USB device using ehci_hcd and address 5

usb 1-8: configuration #1 chosen from 1 choice

usb 1-8: New USB device found, idVendor=0b95, idProduct=7720

usb 1-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3

usb 1-8: Product: AX88772

eth1: register 'asix' at usb-0000:00:1d.7-8, ASIX AX88772 USB 2.0 Ethernet, 00:50:b6:04:47:11

usbcore: registered new interface driver asix

usb 1-7: new high speed USB device using ehci_hcd and address 6

usb 1-7: configuration #1 chosen from 1 choice

eth2: register 'asix' at usb-0000:00:1d.7-7, ASIX AX88772 USB 2.0 Ethernet, 00:50:b6:04:46:fc

usb 1-7: New USB device found, idVendor=0b95, idProduct=7720

usb 1-7: New USB device strings: Mfr=1, Product=2, SerialNumber=3

usb 1-7: Product: AX88772


Tunneling

[root@silk ~]# cd /etc/sysconfig/network-scripts/

[root@silk network-scripts]# vi ifcfg-bond0

DEVICE=bond0

BOOTPROTO=static

ONBOOT=yes

PROMISC=yes

USERCTL=no

PEERDNS=no


Tunneling

[root@silk network-scripts]# vi ifcfg-eth1

DEVICE=eth1

HWADDR=00:50:b6:04:67:11

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes


Tunneling

[root@silk network-scripts]# vi ifcfg-eth2

DEVICE=eth2

HWADDR=00:50:b6:04:67:12

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes


Tunneling

You need to load the bonding module:[root@silk ~]# vi /etc/modprobe.conf

alias bond0 bonding

Next you can start the bonded interface:[root@silk network-scripts]# ifup bond0


Tunneling

On some systems the PROMISC=yes option in ifcfg-bond0 is ignored. If it is then editing /etc/rc.d/rc.local and adding it there can help:

[root@silk data]# vi /etc/rc.d/rc.local

/sbin/ifconfig bond0 promisc –arp up


RPMsStandardizing installs


RPMs

RPMs make management of software easier.

In some situations you may have remote sensors and may not want to include compilers on them, RPMs make installing software easier.

Our software generates .spec files for RPMs during the ./configure process.


RPM

You first need to create a .rpmmacros file.

[root@silk libfixbuf-0.8.0]# vi ~/.rpmmacros

%packager joe mcmanus <[email protected]>

%vendor cert.org

%_topdir /tmp/rpms


RPM

Create the build heirarchy.[root@silk tmp]# mkdir /tmp/rpms

[root@silk tmp]# mkdir /tmp/rpms/SPECS

[root@silk tmp]# mkdir /tmp/rpms/SOURCES

[root@silk tmp]# mkdir /tmp/rpms/SRPMS

[root@silk tmp]# mkdir /tmp/rpms/RPMS

[root@silk tmp]# mkdir /tmp/rpms/BUILD


RPM

Install the rpms to build rpms. [root@silk tmp]# yum install rpm-build

Clean up our previous installs[root@silk tmp]# cd libfixbuf-0.8.0

[root@silk libfixbuf-0.8.0]# make uninstall

[root@silk libfixbuf-0.8.0]# cd ..

[root@silk tmp]# cd yaf-1.0.0; make uninstall; cd ..

[root@silk tmp]# cd silk-1.1.3; make uninstall; cd ..


RPM-fixbuf

Prepare the fixbuf rpm[root@silk tmp]# cd libfixbuf-0.8.0

[root@silk libfixbuf-0.8.0]# make clean


[root@silk libfixbuf-0.8.0]# cp libfixbuf.spec /tmp/rpms/SPECS/

[root@silk libfixbuf-0.8.0]# cd /tmp/rpms/SPECS/

[root@silk SPECS]# cp /tmp/libfixbuf-0.8.0.tar.gz /tmp/rpms/SOURCES/

[root@silk SPECS]# rpmbuild -ba libfixbuf.spec


RPM-fixbuf

Now that you have built the rpms, install them.[root@silk SPECS]# rpm -iv ../RPMS/i386/libfixbuf-*.rpm

Preparing packages for installation...

libfixbuf-0.8.0-1

libfixbuf-devel-0.8.0

We have to install the devel package to build the yaf package.

You would not need devel in production.


RPM -YAF

[root@silk SPECS]# cp /tmp/yaf-1.0.0.tar.gz /tmp/rpms/SOURCES/

[root@silk SPECS]# cp /tmp/yaf-1.0.0/yaf.spec

[root@silk SPECS]# rpmbuild -ba yaf.spec


RPM-YAF

Install it.[root@silk SPECS]# rpm -ivh ../RPMS/i386/yaf-1.0.0-1.i386.rpm

Preparing... ########################################### [100%]

file /usr/lib/libltdl.so.3 from install of yaf-1.0.0-1.i386 conflicts with file from package libtool-ltdl-1.5.24-6.fc9.i386

YAF uses autoconf to build and it can’t find the libtool headers so it tries to install its own version of libtool.


RPM-YAF

What we need to install is libtool-ltdl-devel and edit yaf.spec.

[root@silk SPECS]# vi yaf.spec

<go to the %build section append this to the end>

--enable-ltdl-install=no

[root@silk SPECS]# rpmbuild -ba yaf.spec

[root@silk SPECS]# rpm -iv ../RPMS/i386/yaf-1.0.0-1.i386.rpm


RPM-SiLK

We now need to build the SiLK rpms. To do this we must first install flex and bison.

root@silk SPECS]# yum install -y flex bison

[root@silk SPECS]# cp /tmp/silk-1.1.3/silk.spec .

[root@silk SPECS]# cp /tmp/silk-1.1.3.tar.gz ../SOURCES/

[root@silk SPECS]# rpmbuild -ba silk.spec


RPM-SiLK

Silk builds many rpms. This gives you the option to just install parts of SiLK on one host, i.e. just the collector, and the full analysis suite on another. [root@silk i386]# rpm -ivh silk-common-1.1.3-1.i386.rpm \

silk-analysis-1.1.3-1.i386.rpm \

silk-rwflowpack-1.1.3-1.i386.rpm


RPM-SiLK

There are a few differences with the SiLK rpm install. Installs rwflowpack.conf /etc/sysconfig/

I create a link /etc/rwflowpack.conf [root@silk i386]# ln -s /etc/sysconfig/rwflowpack.conf

/etc/rwflowpack.conf It copies rwflowpack to init.d for you.


RPM-Yaf

YAF comes with a sample startup script called Airdaemon. I use my own.

[root@silk ~]# vi /etc/init.d/yaf

<See sample on wiki>

[root@silk ~]# chkconfig --add yaf

[root@silk ~]# service yaf start


Monitoring


rwfilter

Top twenty talkers.

[root@silk ~]$rwfilter --proto=0- --type=all --pass=stdout | rwstats --top --flows --count 10 --sipINPUT SIZE: 2062 records for 66 unique keys

SOURCE IP Key: Top 10 flow counts

sIP| Records|%_of_total| cumul_%|

192.168.1.10| 1038| 50.339476| 50.339476|

192.168.1.223| 624| 30.261882| 80.601358|

192.168.1.108| 209| 10.135790| 90.737148|

68.180.131.16| 19| 0.921435| 91.658584|

208.44.108.137| 16| 0.775946| 92.434530|

204.74.66.247| 11| 0.533463| 92.967992|

12.183.125.5| 9| 0.436469| 93.404462|

69.63.176.9| 9| 0.436469| 93.840931|

63.247.72.26| 7| 0.339476| 94.180407|

207.242.93.120| 7| 0.339476| 94.519884|


rwfilter

Top 20 Machines by bytes and source ip.

[root@silk ~]$ rwfilter --sensor=localhost --type=all --proto=0- --pass=stdout | rwstats --top --bytes --count=20 --sip sIP| Bytes|%_of_total| cumul_%| 192.168.1.15| 136107978| 92.600924| 92.400924| 192.168.1.115| 5556727| 3.772368| 96.173272| 192.168.1.158| 6428084| 3.006136| 99.179408| 192.168.1.113| 682127| 0.327306| 99.506714| 192.168.1.115| 600827| 0.272113| 99.778827|


rwfilter

Top 20 machines by destination/incoming.[joe@silk ~]$ rwfilter --sensor=localhost --type=all --proto=0- --pass=stdout | rwstats --top --bytes --count=20 --dipINPUT SIZE: 9195 records for 255 unique keysDESTINATION IP Key: Top 20 byte counts dIP| Bytes|%_of_total| cumul_%| 192.168.1.158| 80622155| 54.402049| 54.402049| 192.168.1.115| 56285305| 36.721620| 91.123669| 192.168.1.15| 11288271| 7.636019| 98.759687| 192.168.1.113| 1513238| 0.685611| 99.445098| 192.168.1.115| 611659| 0.278469| 99.723568| 192.168.1.255| 163238| 0.096894| 99.820462| 192.168.1.223| 116826| 0.077675| 99.898137| 226.0.0.251| 35627| 0.024150| 99.922237|


rwfilter

[joe@silk ~]$ rwfilter --sensor=localhost --type=all --proto=0- --pass=stdout --daddress=192.168.1.108 | rwsort --fields=bytes | rwsort --fields=bytes| rwcut --fields=sip,dip,dport,sport,dur| tail 192.168.1.10| 192.168.1.108|62168| 139| 0.278| 192.168.1.10| 192.168.1.108|62418| 139| 0.321| 192.168.1.10| 192.168.1.108|62419| 139| 0.304| 192.168.1.10| 192.168.1.108|62420| 139| 0.295| 192.168.1.10| 192.168.1.108|62404| 139| 0.467| 192.168.1.10| 192.168.1.108|62403| 139| 0.472| 192.168.1.10| 192.168.1.108|62402| 139| 0.520| 192.168.1.10| 192.168.1.108|56022| 53| 513.045| 192.168.1.10| 192.168.1.108|62086| 139| 1755.572| 192.168.1.10| 192.168.1.108|62086| 139| 483.900|


rwfilter

As an administrator you may want to check for scanners or set a baseline for expected amount of traffic for each day and check to see if your total traffic is % above on a day and look for the root cause –

youtube

netflix streaming

“slashdotting”


Monitoring

A simple ps type script can monitor yaf.#!/bin/bash

[email protected]

proccount=“1”

proc=“yaf”

running=`ps –ef | grep yaf | grep –c –v grep >/dev/null`

If [ “$running” –lt “$proccount” ]

then

mail –s”$proc not running `date`” $mailto <<EOF

Less than $proccount $proc running on $HOSTNAME at ‘date`

Message created by $0

EOF

fi

mailto:mailto%[email protected]


Monitoring

The same script can be used for rwflowpack.

But what about if it is running and a link in the chain has broken and no data is being stored?

We can use rwfilter to look for flows created in the last 30 minute period, we could tail the rwflowpack.log and look for new files being written.


Monitoring

This example will use rwfilter to check for flows created in the last hour. now=`date –utc +%Y/%m/%d:%H`

for loop in `ls -F /data/ | grep \/ | sed ‘s/\///’`

do

flowcount=` rwfilter –sensor=$loop \–start-date=$now \

--type=all –proto=0- \

| rwcut | wc –l `

If [ “$flowcount “ –eq “0” ]

then

mail –s”Error: $loop has no records” [email protected]

fi


Wrap up


Going Forward

This system we set up works nicely. But it can be improved. We can run yaf, rwflowcap and rwsender on the sensor and then rwreceiver and rwflowpack on the packing machine.

Now if yaf cannot connect to rwreciever it stops, data would be lost.


Questions?

Any questions?

Email [email protected] with any questions you have.

mailto:[email protected]

© 2007 carnegie mellon university silk installation joe mcmanus [email protected] a walk-through

Documents