syslogng and splunk

Building Centralized Logging: Syslog

Steven “Maniac” McGrath

Syslog?

• logging service

• UNIX based

• Networkable

Wait a Sec...Network?

• UDP port 514

• Typically limited to 1024bytes

One more thing...

• FIFO Buffers

• First In First Out

• Rolling View of Logs

• Type of Named Pipe

FIFO...Tasty *chomp*

Item 5

Item 4Item 3Item 2

Item 1

3 Line FIFO Buffer

Getting Started...

• Ubuntu 6.06 Server

• Base Install

Installing Syslog...

• Update The Repository

Upgrade the OS

• We need to upgrade the OS to current.

Install Syslog-NG

• Syslog-NG will remove klogd, this is normal.

Reconfiguring Syslog-ng

• Configuration depends on network environment.

• Windows Hosts

• Cisco Devices

• Linux Hosts

• Other Devices and Gear

First off...Global!/etc/syslog-ng/syslog-ng.confoptions { chain_hostnames(0); time_reopen(10); time_reap(360); log_fifo_size(2048); create_dirs(yes); group(admin); perm(0640); dir_perm(0755); use_dns(no); stats_freq(0);};

• Disable Hostname Chaining• Time to wait before re-establishing a dead connection• Time to wait before an idle file is closed• FIFO Buffer size• Create Directories• Permissions• Disable DNS• Disable Statistic Logging

Next, The Source

source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); udp();};

/etc/syslog-ng/syslog-ng.conf

Defining Filters

• Windows Filter

• Cisco Filter

Windows Filter

filter f_windows { program(MSWinEventLog);};

/etc/syslog-ng/syslog-ng.conf

Cisco Filter

filter f_cisco_pix {host(IP.OF.PIX.DEVICE);

};

/etc/syslog-ng/syslog-ng.conf

General Filter

filter f_not_others {not host(IP.OF.PIX.DEVICE)and not program(MSWinEventLog);

};

/etc/syslog-ng/syslog-ng.conf

Destinations

• FIFO Buffers

• One Large File

Windows FIFO

destination d_windows {pipe(“/var/log/buffers/windows”);

};

/etc/syslog-ng/syslog-ng.conf

Cisco FIFO

destination d_cisco {pipe(“/var/log/buffers/cisco”);

};

/etc/syslog-ng/syslog-ng.conf

General FIFO/etc/syslog-ng/syslog-ng.conf

destination d_gen_fifo {pipe(“/var/log/buffers/syslog”);

};

...And the Archive

destination d_all {file(“/var/log/arch/$MONTH$DAY$YEAR”);

};

/etc/syslog-ng/syslog-ng.conf

Tying it all Together!

• Now we tell syslog to handle the configs. ;)

Windows Log

log { source(s_all); filter(f_windows);destination(d_windows);

};

/etc/syslog-ng/syslog-ng.conf

Cisco Log

log { source(s_all); filter(f_cisco_pix);destination(d_cisco);

};

/etc/syslog-ng/syslog-ng.conf

General FIFO

log { source(s_all); filter(f_not_others);destination(d_gen_fifo);

};

/etc/syslog-ng/syslog-ng.conf

Archive Log

log { source(s_all); destination(d_all);

};

/etc/syslog-ng/syslog-ng.conf

Finishing up...

• Making the FIFO buffers

• Creating the directory structure

Run me :)

$ sudo mkdir /var/log/arch$ sudo mkdir /var/log/buffers

$ sudo mkfifo /var/log/buffers/windows$ sudo mkfifo /var/log/buffers/cisco$ sudo mkfifo /var/log/buffers/syslog

Restart Syslog-ng

$ sudo /etc/init.d/syslog-ng restart

Is it working?

• Check your Logfiles (/var/log/arch/*)

• Check your FIFO Buffers

• cat /var/log/buffers/windows

• cat /var/log/buffers/cisco

• cat /var/log/buffers/syslog

Awsome! Wait....

• How are we gonna view this data?

splunk

• Web-based Interface

• Indexes arbitrary data

• Searchable

• Reporting

>

• No, I don’t work for them...I just really like their product.

splunk>

• Download The latest version (3.0b3 as of writing)

• Extract the tarball

• Run the application

• Make it startup with a system boot

Installing splunk>

$ wget 'http://www.splunk.com/index.php/download_track?file=/3.0b3/linux/splunk-3.0b3-20872-Linux-i686.tgz&ac=&wget=true&name=wget'

$ sudo mkdir /opt;cd /opt

$ sudo tar xzvf ~/splunk-3.0b3-20872-Linux-i686.tgz

$ sudo /opt/splunk/bin

Installing splunk>

Configuring splunk>

Configuring splunk>

Configuring splunk>

Configuring splunk>

Configuring splunk>

splunk>

Syslog Agents

• Windows Agents

• UNIX Agents

• Other Devices

Windows Logs?

• SNARE Agent

• Converts Event Logs to Syslog

• Free

UNIX Agents

• Use the syslog service!

• *.* @Syslog Server

Other Devices

• Various systems can be configured

• Cisco, Juniper, Lotus Domino, Apache, IIS, etc. are just a few examples.

Recap

• What is Syslog

• What is FIFO

• Installing and Configuring Syslog-NG

• Installing and Configuring Splunk

• Agents

Questions?