Email archiving with piler

The FSF.hu Foundation1 supported and donated piler

within the Free Software Application 20112 project.

First Edition

Publication date Aug 6, 2012

Copyright 2012 Janos SUTO, sj@acts.hu

1 The Hungarian name of the foundation is FSF.hu Alaptvny2 The name of the project is Szabad Szoftver Plyzat 2011

Table of ContentsWhat is piler?........................................................................................................................................3

Business continuity and disaster recovery..................................................................................3Regulatory and compliance requirements...................................................................................3Legal discovery and investigations.............................................................................................3Storage management...................................................................................................................3

Benefits of email archiving..............................................................................................................3Installing piler.......................................................................................................................................4

Prerequisites.....................................................................................................................................4Mandatory:..................................................................................................................................4Highly recommended:.................................................................................................................4Optional.......................................................................................................................................4

Preinstall tasks, recommendations...................................................................................................4Create a dedicated and non-privileged user to run piler..................................................................5Unpack and compile piler................................................................................................................5Edit the configuration file................................................................................................................5Create a key for encryption..............................................................................................................6Create a database for piler...............................................................................................................6Create SQL tables for piler..............................................................................................................6Install and configure sphinx search.................................................................................................7Configure mail relay for piler..........................................................................................................7Set up cron jobs................................................................................................................................7Create a dedicated virtual host for piler...........................................................................................8Website postinstall tasks..................................................................................................................9Configure your mail server to forward a copy of each email it receives or sends to piler............10

Upgrade instructions...........................................................................................................................110.1.18 to 0.1.19 .............................................................................................................................110.1.19 to 0.1.20..............................................................................................................................11

Piler basics..........................................................................................................................................11How to search................................................................................................................................12

How to use simple search..........................................................................................................12How to use expert search..........................................................................................................13Expert search examples.............................................................................................................13

Using the search results.................................................................................................................14Administering piler.............................................................................................................................15

Users, groups, and rights................................................................................................................15admin@local permissions.........................................................................................................15

Archiving rules..............................................................................................................................15Retention rules...............................................................................................................................16What to do with spam....................................................................................................................16How to localize piler?....................................................................................................................17How to make a custom skin to piler?.............................................................................................17

Common tasks....................................................................................................................................18Import emails.................................................................................................................................18

How to import a PST file?........................................................................................................18Export emails.................................................................................................................................19Purge aged emails..........................................................................................................................20

Troubleshooting..................................................................................................................................20Recreate sphinx index data............................................................................................................20

What is piler?

piler is an open source (GNU GPL v3 licensed) email archiving solution with all the necessary features for your enterprise. You have several reasons to archive your emails:

Business continuity and disaster recoverypiler gives you a secure central repository of emails providing the necessary information even if your mail servers are down.

Regulatory and compliance requirementsGovernmental agencies and other regulatory organizations require you to preserve emails for several years. To meet these obligations piler saves your emails in its long term storage.

Legal discovery and investigationspiler helps you to provide relevant information in a timely manner in case of legal discovery, audit or other events.

Storage managementpiler saves storage space by compression and smart deduplication. It also allows you to offload data from the mail server to piler, thus minimizes storage costs.

Benefits of email archivingYour company has lots of knowledge and data in tons of email messages. A proper email archiving solution gives you the following benefits:

fast response to eDiscovery requests compliance reduce mail server load save storage space protect against disaster

Installing piler

Download piler from http://www.mailpiler.org/en/download.html

Prerequisites

Mandatory: openssl3

MySQL 5.1+4

sphinx search 2.0.5+5

php 5.3.x+6

a rewriting rule capable webserver, eg. apache7, lighttpd8, nginx9, an MTA, eg. postfix10 for relaying to piler TRE regex library11

libzip iconv

Highly recommended: memcached12

Optional xcache13

clamav 0.97.x14

libpst15

Preinstall tasks, recommendations

Since piler archives your emails it may need lots of disk space. To keep things tidy, I recommend you to put all piler related data (including your log files) under the /var partition. I strongly

3 http://www.openssl.org/4 http://www.mysql.com/5 http://sphinxsearch.com/6 http://www.php.net/7 http://httpd.apache.org/8 http://www.lighttpd.net/9 http://www.nginx.net/10 http://www.postfix.org/11 http://laurikari.net/tre/12 http://www.memcached.org/13 http://xcache.lighttpd.net/14 http://www.clamav.net/15 http://www.five-ten-sg.com/libpst/

recommend using logical volumes.

You may create the following layout:

/ - 2 GB

/tmp - 128 MB

/var - several hundreds of GBs (or even more)

swap - 512 MB

The used filesystem is not a concern, however if you choose XFS, then be aware of a 32 bit vs. 1 TB issue16.

Create a dedicated and non-privileged user to run piler

groupadd piler

useradd -g piler -s /bin/sh -d /var/piler piler

usermod -L piler

Unpack and compile piler

tar zxvf piler-x.y.z.tar.gz

cd piler-x.y.z