sfscon14: the oss-mailcluster of raiffeisen online gmbh

18.11.2014

Raiffeisen OnLine‘s OSS mail cluster

Raiffeisen OnLine‘s OSS mail cluster [email protected]

18.11.2014

Purpose + requirements

• implementation of a new spam filter system

• replacement of the present Windows solution (3 servers)

• redundant setup

• Active/Active preferred

• easily scalable and flexible

• give our “Service Center” access to

• account configuration setting

• log data for customer support purposes

• should allow further expansion in order to implement new features

and improve hit rate


18.11.2014

Which solution should we use?

• Evaluation of manufacturers

• major differences in price/performance ratio

• not all solutions grant actual redundancy (only standby setup)

• Test installation by a manufacturer works OK …

• … until Friday afternoon, when a sudden deadlock occurs!

• After having spent 3 days searching for the error, the

manufacturer states that:

• …there must be a power outage in the datacenter -> I’m sure there

was no outage


18.11.2014

We finally decided to try out a self-made solution

• But why OSS?

• The experience told us it’s preferable to adopt a solution on which we

are able to investigate by ourselves in an emergency situation.

• We were aiming at a solution that would allow us to meet new

requirements on our own in the future.

• We wanted to be able to develop own new components if necessary.

• Our objective was to reduce dependency on one single manufacturer.


18.11.2014

We found all components in the open source world very quickly

• First implementation in the middle of 2008:

• Postfix (MTA) -> on the front line

• Spamassassin + ClamAV -> as content filter

• AmaVis -> interface between postfix + content filter

• MySQL –> configuration + quarantine (Spam/Virus)

• MySQL -> central logging for “ServiceCenter” + customers

• PHP -> administration front end for “ServiceCenter” + customers


18.11.2014

First phase of implementation

• 2x bare metal servers (8GB RAM, QuadCore, SAS)

• „extreme“ container-virtualization

• Each service had its own container

• 2 Postfix front end MTAs

• 2 AmaVis (Spamassassin, ClamAV) content filters

• 2 MySQL Master/Master Replications (configuration)

• 2 MySQL Master/Master Replications (quarantine)

• 2 MySQL Master/Master Replications (logdatabase)

• 2 DNS Resolvers

• 2 Postfix Backend MTAs (delivering e-mails to storage back ends)


18.11.2014

Each node could go down at any time

• Redundant SMTP paths

IN ---> Postfix ---> AmaVis ---> Postfix ---> OUT


18.11.2014

What are the main problems?

• OSS components are available in heaps, but it was difficult to

• have all different components work together

• process large amounts of log data in order for users to understand them

• We had not much experience with high mail traffic.

• It took us some time to identify the correct performance and

configuration parameters.


18.11.2014

Technical support

• Google

• knows about 90% of all the problems we came across

• Community

• We posted our own questions to mailing lists and forums.

• We kept reading newsgroups.

• Common sense

• logical + analytical approach

• “Don’t worry, it’s just numbers and letters.”*

* cit.: Thomas Gelf


18.11.2014

Continuous enhancements in the last years

• Perl -> small policy daemons we developed in 2012 and 2013

• selective grey listing

• performance improvements, storing non important data into the RAM

• ClamAv content filter replaced (July 2014)

• by ClamAv-Milter: scans viruses directly on front end servers and rejects

with 550 in real time instead of storing in quarantine

• SPF-Policyd (Juli 2014)

• supports Sender Policy Framework in either direction (In/Out)

• OpendDKIM (Juli 2014)

• supports Domain Key Identified Mail in either direction (In/Out)

• OpenDMARC (Juli 2014)

• supports Domain-based Message Authentication, Reporting &

Conformance in either direction (In/Out)


18.11.2014

Current mail traffic (Postfix 1)


18.11.2014

Current mail traffic (Postfix 2, Content filter)


18.11.2014

Some more highlights:

• SSL/TLS available on all front end servers

• IPv6

• active on all front end servers in both directions

• (partial) communication between components

• now facing IMAP/POP3 + SMTP submission

• IMAP + POP3 + SMTP failover

• front end displays processed log data for „ServiceCenter“ and

customers


18.11.2014

IMAP + POP3 failover

• outward: virtial IP + Multicast MAC

• based on IPTables Cluster-IP Module

• [imap,pop3].rolmail.net (Dovecot)

• smtp.rolmail.net (Postfix)

• 2 proxy servers

• imap1.rolmail.net

• imap2.rolmail.net

• more possible

• Mailstore:

• many backend servers (Dovecot)

• Load balancing of I/O


18.11.2014

Processed data for „ServiceCenter“ and customers


18.11.2014

Future plans

• to force SSL/TLS for IMAP/POP3 + SMTP submission clients

• to protect mail traffic with DNS-based Authentication of Named

Entities (called DANE)

• protection of zones with DNSSEC is needed

• to improve quarantine mechanism

• rejecting all e-mails tagged as spam directly on the frond end servers

with 550 -> no mail will be stored in quarantine

• maybe to implement the mailstore backend server pool as an

active/active setup


18.11.2014

### It was a pleasure to be here ###

Many thanks for your attention!

Urban Lösch

Raiffeisen OnLine GmbH

[email protected]

mailto:[email protected]

sfscon14: the oss-mailcluster of raiffeisen online gmbh

Technology