rsyslog futures (2012, RH Mini Summit on Logging)

RSYSLOG update
v7 and beyond

Rainer Gerhards

What's in this talk?

Security improvements in v7

A quick word on Journal integration

v8 engine improvements

Writing plugins in languages other than C

I will probably run out of time - but that's fine, the slides at the end are optional.

The rsyslog doc project

The doc just sucks...

Spawned a new project to create better one: https://github.com/rsyslog/rsyslog-doc

Lead by James Boylan (a sysadmin)

Please helpComplain ;-)

open issues

Write some doc...

We are especially interested to learn what is hard for beginners!

New security features in rsyslog v7

Rainer Gerhards

Remeber, in pre-v7 we have

TLS-encrypted syslog transportRFC5425

Mutual authentication

Trusted propertiesTake log message origin based on SCM_CREDENTIALS

Signed Log Records

Introduced in v7.4

Protects log files on machine

Generic approach by introducing a signature provider interface

Currently provider for Keyless Signature Infrastructure (KSI)

Hash chain for log record is created

Signing via Hash Chains...

Very rough sample (actually Merkle trees!)

No local secret!

Consider chain layer to be operated on a schedule (timer ticks!) by external entity

Source: http://en.wikipedia.org/wiki/File:Hashlink_timestamping.svg

Where did we add Signatures?

filesNetwork
(e.g.TCP)

/dev/logfileDatabaseRemote
systemParsersFor-
matterRules&
Filters

Inputs

Outputs

File Signature Interface in Detail

omfileGeneric interface providing future extensibilityEnables Distros to pack Functionality w/o increasing base system size

File
Stream
Class

Log File

GuardTime
SigProvSigProv
Interface

Sig File
(TLV)

Both files
together
are thesigned log

Activating Log Signing

action(type="omfile" file="/var/log/logfile"sig.provider="gt"sig.keepTreeHashes="on" sig.keepRecordHashes="on")

Parameters except sig.provider are optional

Writesregular log file

plus signature file (*.gtsig)

Signing log records in flight

Best practice is to use TLS with mutual authentication so that the log source can be trusted

no good and practical solution for signatures inside the log record

Experimental module rfc5424addhmac provides HMAC within RFC5424 strucutured data

Log File Encryption

Generic approach by introducing a crypto-provider interface

Currently available a libgcrypt-based crypto provider

Symmetric cryptography, all ciphers & modes supported by libgcrypt

Key can come fromConfig param (testing only, pls!)

File

Script (interface for advanced key exchange options)

Activating Log Encryption

action(type="omfile" file="/var/log/logfile"cry.provider="gcry"
cry.keyprogram=/path/to/binary)

Addtl Parameters for ciphers, etc...

Writesregular log file, encrypted

plus encryption info file (*.encinfo)

Works in conjunction with signatures

Encrypted Disk Queues

Starting with v7.5, disk queue files can also be encrypted

Uses same crypto provider as log files

Can be specififed on a per-queue basis

action(type="omfwd" target="172.123.123.5 Port="10514 queue.type="disk" queue.fileName="enc" queue.cry.provider="gcry" queue.cry.keyprogram="binary" )

Log File Anonymization

Permits to anonymize IP addressesZero-out (based on netmask)

Replace with char

Based on hard German data protection laws

Currently for IPv4

Implemented via the action interfaceCan be applied conditionally

Permits access to original message if desired

No access possible after anonymizer is run

RELP security enhancements

RELP is used to reliably forward messages

Can now be secured like TCP syslogTLS

Mutual authentication via various authentication modes

Implemented at the librelp levelSo this is available to other apps as well

rsyslog Journal Integration

Rainer Gerhards

Integration Modules

Module imjournalProvides ability to pull messages off the journal, just as another event source

Gets into trouble if journal DB is unclean

We currently recommend to use only when absolutely required

Module omjournalstores messages into the journal

Permits to integrate e.g. router messages especially in SOHO environment

Integrating syslog Data into the journal (SOHO env)

/* first, we make sure all necessary modules are present: */module(load="imudp") # input module for UDP syslogmodule(load="omjournal") # output module for journal

/* then, define the actual server that listens to the * router. Note that 514 is the default port for UDP syslog. */input(type="imudp" port="514" ruleset="writeToJournal")

/* inside that ruleset, we just write data to the journal: */ruleset(name="writeToJournal") {action(type="omjournal")}

Writing RSYSLOG error messages to journal

New feature in 7.4.10 and above

Permits to write rsyslog error messages directly to journal

We hope that this will finally help make user notice them, e.g. via
$ systemctl status rsyslog

global( ProcessInternalMessages = "off")

The rsyslog v8 engine

Rainer Gerhards

The v7 rule engine

rsyslogcoreQueue workerQueue workerQueue workerAction instancequeue

Single-thread
compartment

Filter processing

Message formatting

Actual output action, like sending msg

Kept simple & single threaded

Works well with fast actions

Has problems with slow ones, e.g.
via HTTP (like Elasticsearch)

The v8 rule engine

rsyslogcoreQueue workerQueue workerQueue workerAction wrkr inst.queue

Now multiple instances per action!

Queue worker pool automatically
scales outbound connection count
by spawning more worker instances

Works well with Elasticsearch etc.

Inherently serial outputs (e.g. local files!)
must serialize themselves

Action wrkr inst.Action wrkr inst.

Writing external output plugins for RSysLog
IN 2 MINUTES

Rainer Gerhards

Write the plugin itself

Choose any language you like

Implement the pseudocode belowMessages arrive via stdin, one message per line

Read from stdin until EOF

Process each message read as you like

Terminate when EOF is reached

That's it!

While not EOF(stdin) do { Read msg from stdin Process msg}

Make RsysLog call plugin

Regular filtering applies (as with any action)

You can specify message format via a template

Use omprog for the call

module(load=omprog) # needed only once in config!

if $rawmsg contains sometrigger then action(type=omprog binary=/path/to/your/plugin)

Optional: debugging your plugin

If something doesn't work, it's best to debug outside of rsyslog

Do this as you usually debug your programs (e.g. use your favorite debugger!)

For example, do

$ echo testmessage | /path/to/your/plugin

Questions about the plugin interface or plugin integration? Visit http://kb.monitorware.com/external-plugins-f53.html

Want to know more details?

There is an additional presentation available at http://www.slideshare.net/rainergerhards1/external-plugins

The complete interface specification can be found right inside the source repository:
https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md

Check out the copy-templatesAvailable for an increasing number of languages

More advanced interface handling

Ready to be copied

https://github.com/rsyslog/rsyslog/tree/master/plugins/external

Questions?

rgerhards@adiscon.com

www.rsyslog.com

https://github.com/rsyslog

Please fill in the feedback questionnaire:

http://devconf.cz/f/107