Extracting the HamExtracting the Hamfrom Spamfrom Spam

David J. YoungDavid J. Young

IntroductionIntroduction HistoryHistory SpamSpam TerminologyTerminology ASSPASSP BenchmarksBenchmarks DemoDemo QuestionsQuestions

HistoryHistory

Where did the term Where did the term “spam” come from?“spam” come from?

SPSPiced hiced hAMAM

SPAM sketchSPAM sketchhttp://www.youtube.com/results?search_query=spam+monty+pythonhttp://video.google.com/videosearch?q=spam+monty+python

Scene:Scene:  A cafe.  One table is occupied by a group of Vikings wearing horned helmets.  Whenever   A cafe.  One table is occupied by a group of Vikings wearing horned helmets.  Whenever the word "spam" is repeated, they begin singing and/or chanting.  A man and his wife enter.  The the word "spam" is repeated, they begin singing and/or chanting.  A man and his wife enter.  The man is played by Eric Idle, the wife is played by Graham Chapman (in drag), and the waitress is man is played by Eric Idle, the wife is played by Graham Chapman (in drag), and the waitress is played by Terry Jones, also in drag.played by Terry Jones, also in drag.

Man:Man:You sit here, dear.You sit here, dear.Wife:Wife:All right.All right.Man:Man:Morning!Morning!Waitress:Waitress:Morning!Morning!Man:Man:Well, what've you got?Well, what've you got?Waitress:Waitress:Well, there's egg and bacon; egg sausage and bacon; egg and spam; egg bacon and Well, there's egg and bacon; egg sausage and bacon; egg and spam; egg bacon and spam; egg bacon sausage and spam; spam bacon sausage and spam; spam egg spam spam bacon spam; egg bacon sausage and spam; spam bacon sausage and spam; spam egg spam spam bacon and spam; spam sausage spam spam bacon spam tomato and spam;and spam; spam sausage spam spam bacon spam tomato and spam;Vikings:Vikings:Spam spam spam Spam spam spam spam...spam...Waitress:Waitress:...spam spam spam egg and spam; spam spam spam spam spam spam baked ...spam spam spam egg and spam; spam spam spam spam spam spam baked beans spam spam spam...beans spam spam spam...Vikings:Vikings:Spam! Lovely spam! Lovely spam!Spam! Lovely spam! Lovely spam!Waitress:Waitress:...or Lobster ...or Lobster Thermidor a Crevette with a mornay sauce served in a Provencale manner with shallots and Thermidor a Crevette with a mornay sauce served in a Provencale manner with shallots and aubergines garnished with truffle pate, brandy and with a fried egg on top and spam.aubergines garnished with truffle pate, brandy and with a fried egg on top and spam.Wife:Wife:Have you Have you got anything without spam?got anything without spam?Waitress:Waitress:Well, there's spam egg sausage and spam, that's not got Well, there's spam egg sausage and spam, that's not got much spam in it.much spam in it.Wife:Wife:I don't want ANY spam!I don't want ANY spam!Man:Man:Why can't she have egg bacon spam and Why can't she have egg bacon spam and sausage?sausage?Wife:Wife:THAT'S got spam in it!THAT'S got spam in it!Man:Man:Hasn't got as much spam in it as spam egg sausage and Hasn't got as much spam in it as spam egg sausage and spam, has it?spam, has it?Vikings:Vikings:Spam spam spam spam... (Crescendo through next few lines...)Spam spam spam spam... (Crescendo through next few lines...)Wife:Wife:Could Could you do the egg bacon spam and sausage without the spam then?you do the egg bacon spam and sausage without the spam then?Waitress:Waitress:Urgghh!Urgghh!Wife:Wife:What do What do you mean 'Urgghh'? I don't like spam!you mean 'Urgghh'? I don't like spam!Vikings:Vikings:Lovely spam! Wonderful spam!Lovely spam! Wonderful spam!Waitress:Waitress:Shut up!Shut up!Vikings:Vikings:Lovely spam! Wonderful spam!Lovely spam! Wonderful spam!Waitress:Waitress:Shut up! (Vikings stop) Bloody Vikings! You can't Shut up! (Vikings stop) Bloody Vikings! You can't have egg bacon spam and sausage without the spam.have egg bacon spam and sausage without the spam.Wife:Wife:I don't like spam!I don't like spam!Man:Man:Sshh, dear, don't Sshh, dear, don't cause a fuss. I'll have your spam. I love it. I'm having spam spam spam spam spam spam spam cause a fuss. I'll have your spam. I love it. I'm having spam spam spam spam spam spam spam beaked beans spam spam spam and spam!beaked beans spam spam spam and spam!Vikings:Vikings:Spam spam spam spam. Lovely spam! Spam spam spam spam. Lovely spam! Wonderful spam!Wonderful spam!Waitress:Waitress:Shut up!! Baked beans are off.Shut up!! Baked beans are off.Man:Man:Well could I have her spam instead Well could I have her spam instead of the baked beans then?of the baked beans then?Waitress:Waitress:You mean spam spam spam spam spam spam... (but it is too You mean spam spam spam spam spam spam... (but it is too late and the Vikings drown her words)late and the Vikings drown her words)Vikings:Vikings:Spam spam spam spam. Lovely spam! Wonderful Spam spam spam spam. Lovely spam! Wonderful spam! Spam spa-a-a-a-a-am spam spa-a-a-a-a-am spam. Lovely spam! Lovely spam! Lovely spam! spam! Spam spa-a-a-a-a-am spam spa-a-a-a-a-am spam. Lovely spam! Lovely spam! Lovely spam! Lovely spam! Lovely spam! Spam spam spam spam!Lovely spam! Lovely spam! Spam spam spam spam!

Spam Spam Spam lyricsSpam Spam Spam lyrics Lovely spam, wonderful spa-a-m,Lovely spam, wonderful spa-a-m,

Lovely spam, wonderful S Spam,Lovely spam, wonderful S Spam,Spa-a-a-a-a-a-a-am,Spa-a-a-a-a-a-a-am,Spa-a-a-a-a-a-a-am,Spa-a-a-a-a-a-a-am,SPA-A-A-A-A-A-A-AM,SPA-A-A-A-A-A-A-AM,SPA-A-A-A-A-A-A-AM,SPA-A-A-A-A-A-A-AM,LOVELY SPAM, LOVELY SPAM,LOVELY SPAM, LOVELY SPAM,LOVELY SPAM, LOVELY SPAM,LOVELY SPAM, LOVELY SPAM,LOVELY SPA-A-A-A-AM...LOVELY SPA-A-A-A-AM...SPA-AM, SPA-AM, SPA-AM, SPA-A-A-AM! SPA-AM, SPA-AM, SPA-AM, SPA-A-A-AM!

What is spam?What is spam? Unsolicited Bulk e-mail (UBE)Unsolicited Bulk e-mail (UBE) Unsolicited Commerical Email (UCE)Unsolicited Commerical Email (UCE)

““The abuse of electronic messaging The abuse of electronic messaging systems to send unsolicited, systems to send unsolicited, undesired bulk messages” undesired bulk messages”

The cost of spamThe cost of spam Productivity – It is estimated that 80-Productivity – It is estimated that 80-

85% of all email is spam 85% of all email is spam Payload may contain malware (virus, Payload may contain malware (virus,

worm, trojan, etc.)worm, trojan, etc.) Internet bandwidthInternet bandwidth

How do spammers getHow do spammers gete-mail addresses?e-mail addresses?

Replying to a spam e-mailReplying to a spam e-mail Auto-responders (vacation)Auto-responders (vacation) Viewing HTML spam (web beacons)Viewing HTML spam (web beacons) Clicking on URLs to websites listed in spamClicking on URLs to websites listed in spam Chain e-mail (MUA virus)Chain e-mail (MUA virus) MiningMining

• Usenet postings/message boards/chat roomsUsenet postings/message boards/chat rooms• Usenet article message-IDsUsenet article message-IDs• Company or personal websitesCompany or personal websites• DNS SOA recordsDNS SOA records• whois databasewhois database

Opt-out websitesOpt-out websites E-mail worms harvesting address booksE-mail worms harvesting address books Shady businesses selling addresses to spammersShady businesses selling addresses to spammers Dictionary attacksDictionary attacks ZombiesZombies

Anti-spam best practicesAnti-spam best practices Turn off email “preview”Turn off email “preview” Use throw away email addressesUse throw away email addresses Do not use an auto responderDo not use an auto responder Do not read spamDo not read spam Do not click on URLs in spamDo not click on URLs in spam Give your e-mail address only to closely trusted Give your e-mail address only to closely trusted

acquaintancesacquaintances Use images or other obfuscation techniquesUse images or other obfuscation techniques Googling for your email addressGoogling for your email address Use a good spam filterUse a good spam filter

TerminologyTerminology

Not Identified as Not Identified as SPAMSPAM

Identified as Identified as SPAMSPAM

Not SPAMNot SPAM(Negative)(Negative) True NegativeTrue Negative False NegativeFalse Negative

(*****SPAM*****)(*****SPAM*****)

SPAMSPAM(Positive)(Positive) False PositiveFalse Positive True PositiveTrue Positive

(*****SPAM*****)(*****SPAM*****)

xxxxx Listingxxxxx Listing WhitelistingWhitelisting

A list of email addresses which would generally A list of email addresses which would generally never send you spam never send you spam

BlacklistingBlacklistingA list of email addresses or domains you do not A list of email addresses or domains you do not wish to receive any email fromwish to receive any email from

GreylistingGreylistingTemporarily reject an unknown email by Temporarily reject an unknown email by imposing a fixed delay before accepting email imposing a fixed delay before accepting email (ASSP calls this Delaying due to a name conflict)(ASSP calls this Delaying due to a name conflict)

RedlistingRedlistingKeeps an address off the whitelistKeeps an address off the whitelist

More ASSP termsMore ASSP terms Spam LoverSpam Lover Spam BucketSpam Bucket HoneypotHoneypot PostmasterPostmaster BayesianBayesian MTAMTA MUAMUA SMTPSMTP

Processing matrixProcessing matrix

Filtered MailFiltered Mail Unfiltered MailUnfiltered Mail

Contributes to Contributes to whitelistwhitelist

Normal ASSP Normal ASSP operationoperation Spam LoverSpam Lover

Doesn’t Doesn’t contribute to contribute to

whitelistwhitelist

RedlistRedlist (but (but doesdoes contribute to contribute to

spam/nospam collections)spam/nospam collections)

No processingNo processing(also doesn’t contribute to (also doesn’t contribute to spam/nospam collections)spam/nospam collections)

What is ASSP?What is ASSP?

AAnti-nti-SSpam pam SSMTP MTP PProxyroxy

““An Open Source platform-independent An Open Source platform-independent transparent SMTP proxy server that transparent SMTP proxy server that leverages numerous methodologies and leverages numerous methodologies and technologies to both rigidly and technologies to both rigidly and adaptively identify spam.”adaptively identify spam.”

-- wikipedia.org-- wikipedia.org

Theory of OperationTheory of Operation When you install ASSP a colony of super-When you install ASSP a colony of super-

intelligent thermophilus bacteria takes up intelligent thermophilus bacteria takes up residence on your CPU and begin reading all your residence on your CPU and begin reading all your email. They communicate using radio waves email. They communicate using radio waves directly with the CPU and interface with the ASSP directly with the CPU and interface with the ASSP software choosing between spam and nonspam software choosing between spam and nonspam mail.mail.

If you choose to read further this myth will be If you choose to read further this myth will be sadly dispelled, and I take no responsibility for sadly dispelled, and I take no responsibility for the consequences. the consequences.

However, you can always refer your users to this However, you can always refer your users to this slide to prove to them that their email is actually slide to prove to them that their email is actually being filtered by super-intelligent bacteria. being filtered by super-intelligent bacteria.

True Theory of OperationTrue Theory of Operation ASSP uses three complementary strategies to allow good ASSP uses three complementary strategies to allow good

email and to block unsolicited emailemail and to block unsolicited email• WhitelistingWhitelisting• SpambucketsSpambuckets• Bayesian filteringBayesian filtering

Local mail domain users are not whitelistedLocal mail domain users are not whitelisted

ASSP ImplementationASSP Implementation Version 1.2.5Version 1.2.5 It is a single Perl scriptIt is a single Perl script 360 KB360 KB 10,000 lines10,000 lines Built in web serverBuilt in web server Built in Pseudo-SMTP serverBuilt in Pseudo-SMTP server

ASSP Target User BaseASSP Target User Base ASSP’s primary target audience is mail ASSP’s primary target audience is mail

administrators or system administrators at administrators or system administrators at smallish institutions. If you operate an ISP or a smallish institutions. If you operate an ISP or a mailhost with a heterogeneous user base, you mailhost with a heterogeneous user base, you may not have a good enough consensus about may not have a good enough consensus about what is considered spam or is not. It should work what is considered spam or is not. It should work well with between 1 and 300 client addresses well with between 1 and 300 client addresses and a mail volume of up to around 100,000 and a mail volume of up to around 100,000 messages per day. Testing has not been done to messages per day. Testing has not been done to verify these ranges verify these ranges

ASSP is ASSP is notnot for the following: for the following:1.1. Individual clientsIndividual clients -- ASSP must be installed together -- ASSP must be installed together

with a SMTP serverwith a SMTP server2.2. Domains which receive mail indirectly, for example if Domains which receive mail indirectly, for example if

you use fetchmail you use fetchmail

ASSP PhilosophyASSP Philosophy Reject SPAM before the SMTP serverReject SPAM before the SMTP server Work with any SMTP MTAWork with any SMTP MTA Adapt quickly as spammers change Adapt quickly as spammers change

attack strategiesattack strategies Require low maintenance after initial Require low maintenance after initial

setupsetup

Main ASSP capabilitiesMain ASSP capabilities Automatic WhitelistingAutomatic Whitelisting Spam TrapsSpam Traps Bayesian filteringBayesian filtering GreylistGreylist Whitelist RE MatchingWhitelist RE Matching Email interfaceEmail interface Mail AnalyzerMail Analyzer Automatic StatisticsAutomatic Statistics SPF (Sender Policy Framework)SPF (Sender Policy Framework) DNSBL (DNS Black Lists)DNSBL (DNS Black Lists) ClamAV virus scannerClamAV virus scanner Mail host HeadersMail host Headers

ASSP FeaturesASSP Features Uses existing MTA and MUA’sUses existing MTA and MUA’s Runs on Linux, Unix, Windows, OS X, and moreRuns on Linux, Unix, Windows, OS X, and more Automatic whitelist – no-one you email will ever be blocked Automatic whitelist – no-one you email will ever be blocked Redlist keeps an address off the whitelistRedlist keeps an address off the whitelist Uses honeypot type spambucket addresses to automatically recognize Uses honeypot type spambucket addresses to automatically recognize

spam and update your spam databasespam and update your spam database Bayesian filter intelligently classifies email into spam and non-spamBayesian filter intelligently classifies email into spam and non-spam Supports site-defined regular expressions to identify spam or non-spam Supports site-defined regular expressions to identify spam or non-spam

email email Accepts whitelist submissions and spam error reports by authorized emailAccepts whitelist submissions and spam error reports by authorized email Browser based setupBrowser based setup Keeps spam statistics for your siteKeeps spam statistics for your site Recognizes Mime encoded and other camouflaged spamRecognizes Mime encoded and other camouflaged spam Can listen on more than one smtp portCan listen on more than one smtp port Basic anti-virus filtering using the ClamAV virus databasesBasic anti-virus filtering using the ClamAV virus databases Optionally blocks no mail but adds an email header and/or updates the Optionally blocks no mail but adds an email header and/or updates the

message subject (*****SPAM*****)message subject (*****SPAM*****) Can block spam-bombs (when spammers forge your domain in the from Can block spam-bombs (when spammers forge your domain in the from

field) field) MoreMore

ASSP FlexibilityASSP Flexibility Whitelist-only modeWhitelist-only mode Don’t filter, just tag subject lineDon’t filter, just tag subject line Let specific addresses receive SPAMLet specific addresses receive SPAM Use a mail list behind ASSPUse a mail list behind ASSP Use ASSP with redundant MX Use ASSP with redundant MX

domainsdomains Web based configurationWeb based configuration

ASSP Mail ProcessingASSP Mail ProcessingWhat order does ASSP process mail to check if it is spam?What order does ASSP process mail to check if it is spam?

1.1. Local or whitelisted?Local or whitelisted?2.2. Blacklisted Domain?Blacklisted Domain?3.3. Spam Helo?Spam Helo?4.4. Addressed to spam-bucket?Addressed to spam-bucket?5.5. Mail bomb?Mail bomb?6.6. Blocked attachment?Blocked attachment?7.7. Matches expression to identify non-spam?Matches expression to identify non-spam?8.8. Matches expression to identify spam?Matches expression to identify spam?9.9. Bayesian evaluation Bayesian evaluation

If the message is identified as spam at any step along the If the message is identified as spam at any step along the way it goes to the spam directory. If the message is local or way it goes to the spam directory. If the message is local or whitelisted it goes to the notspam directory. whitelisted it goes to the notspam directory.

Installation OverviewInstallation Overview Install ASSP and dependenciesInstall ASSP and dependencies Configure ASSPConfigure ASSP Put ASSP in test modePut ASSP in test mode Modify mail flow of test user(s)Modify mail flow of test user(s) Test that it is workingTest that it is working Prime the systemPrime the system Create the Bayesian databaseCreate the Bayesian database Automate daily Bayesian database updatesAutomate daily Bayesian database updates Monitor spam filteringMonitor spam filtering Correct false negatives and false positivesCorrect false negatives and false positives Take ASSP out of test modeTake ASSP out of test mode Train user communityTrain user community Modify mail flow of trained usersModify mail flow of trained users

ASSP InstallationASSP Installation Install PerlInstall Perl Install Perl modules from CPANInstall Perl modules from CPAN

• Compress::ZlibCompress::Zlib NEEDED - Standard Perl installationNEEDED - Standard Perl installation• Digest::MD5Digest::MD5 NEEDED - Standard Perl installationNEEDED - Standard Perl installation• Time::HiResTime::HiRes NEEDED - Standard Perl installationNEEDED - Standard Perl installation• Net::DNSNet::DNS NEEDED TO RUN RBL, SPF and 1.2.XNEEDED TO RUN RBL, SPF and 1.2.X• Email::ValidEmail::Valid OPTIONAL, BUT ADVISEDOPTIONAL, BUT ADVISED• File::ReadBackwardsFile::ReadBackwards OPTIONAL, BUT ADVISEDOPTIONAL, BUT ADVISED• Mail::SPF::QueryMail::SPF::Query OPTIONALOPTIONAL• Mail::SRSMail::SRS OPTIONALOPTIONAL• Sys::SyslogSys::Syslog OPTIONALOPTIONAL• Net::LDAPNet::LDAP OPTIONAL :: NEEDED IF YOU RUN LDAPOPTIONAL :: NEEDED IF YOU RUN LDAP• Win32::DaemonWin32::Daemon NEEDED to run as a service on WindowsNEEDED to run as a service on Windows

No installation scriptNo installation script• GUNZIP assp.tar.gz to /usr/local/asspGUNZIP assp.tar.gz to /usr/local/assp• In /usr/local create the following directories: In /usr/local create the following directories:

assp/spam assp/spam assp/notspam assp/notspam assp/errors assp/errors assp/errors/spam assp/errors/spam assp/errors/notspamassp/errors/notspam

Configure ASSPConfigure ASSP Start ASSPStart ASSP

perl assp.plperl assp.pl Configure ASSPConfigure ASSP

http://127.0.0.1:55555http://127.0.0.1:55555Login: <empty>Login: <empty>Password: nospam4me (default)Password: nospam4me (default)

Beware of the “Show Advanced Beware of the “Show Advanced Configuration” OptionConfiguration” Option

ASSP ConfigurationASSP Configuration

Initial ConfigurationInitial Configuration Change values forChange values for

1.1. ““Web Admin Password”Web Admin Password”2.2. ““Accept All Mail”Accept All Mail”3.3. ““Local Domains”Local Domains”4.4. ““Spam Error”Spam Error”5.5. ““Spam Addresses”Spam Addresses”

Addresses of recipients at your site that only Addresses of recipients at your site that only receive spam (website spam-bait, ex-receive spam (website spam-bait, ex-employees)employees)

Mail FlowMail Flow

Internet Mail Svr Clients Inbound

Outbound

Internet ASSP Mail Svr Clients Inbound

OutboundInternet ASSPMail Svr Clients

Internet Mail Svr Clients

with ASSP

Internet Mail SvrASSP Clients Invalid

Email FlowEmail Flow

Internet ASSP GroupWise/Exchange Clients Inbound

Outbound

MTA

Internet GroupWise/Exchange ClientsMTA ASSP

ASSP MTA

smtp0

in out

spam Notspam

white red blackgrey

BayesianDB

Errors

12525

GWIA MTA

POA

GroupWise

19991999

This is an email that is being sent to the Internet. Th This is an email that is

Internet

Internet GWIA MTA

POA

GroupWise

sendmail

Virtusertable aliases

Internet MTADNSBlockList

20032003

20042004

GWIA MTA

POA

GroupWise

sendmail

Virtusertable aliases

sendmail

SpamAssassin

SpamAssassinInternet MTA

Internet

ASSP sendmail

ASSP

spam Notspam

white red blackgrey

BayesianDB

Errors

20062006

GWIA MTA

POA

GroupWise

sendmail

Virtusertable aliases

sendmail

SpamAssassin

SpamAssassinInternet MTA

Internet

ASSP sendmail

ASSP

spam Notspam

white red blackgrey

BayesianDB

Errors

Phase InPhase In

GWIA MTA

POA

GroupWise

sendmail

Virtusertable aliases

sendmail

SpamAssassin

SpamAssassinInternet MTA

Internet

Flow with Anti-VirusFlow with Anti-Virus

Internet ASSP Mail Svr Clients

Internet ASSP Clients

Inbound

Outbound

Antivirus

Mail Svr Antivirus

Flow with GroupwareFlow with Groupware

Internet ASSP Groupware Clients Inbound

Outbound

MTA

Internet Groupware ClientsMTA ASSP

To use ASSP with Exchange, Lotus To use ASSP with Exchange, Lotus Notes or GroupWise, you’ll also need Notes or GroupWise, you’ll also need to implement a “smarthost” relay to implement a “smarthost” relay like sendmail, qmail, postfix, exim or like sendmail, qmail, postfix, exim or one in a number of othersone in a number of others

DNSBL vs GreylistDNSBL vs Greylist The ASSP Greylist supercedes DNSBLThe ASSP Greylist supercedes DNSBL ASSP “Greylist” is not to be confused ASSP “Greylist” is not to be confused

with “Greylisting”with “Greylisting” Use of DNSBL is discouraged (If a Use of DNSBL is discouraged (If a

DNSBL lookup blocks, ASSP will block DNSBL lookup blocks, ASSP will block due to it’s multiplex design)due to it’s multiplex design)

Penalty BoxPenalty Box This will blacklist an SMTP server for This will blacklist an SMTP server for

about 72 hours or so from sending to about 72 hours or so from sending to your server if they violate basic SMTP your server if they violate basic SMTP connection conventions over a connection conventions over a certain threshold.certain threshold.

SMTP PortsSMTP Ports

For example, internet mail needs to For example, internet mail needs to connect to ASSP on port 25 (ASSP's connect to ASSP on port 25 (ASSP's listen port), and ASSP can proxy to listen port), and ASSP can proxy to your mail server on port 125 (or any your mail server on port 125 (or any port you choose) -- ASSP's SMTP port you choose) -- ASSP's SMTP Destination. You need to change your Destination. You need to change your mail server to match. mail server to match.

Sender NotificationSender Notification With most client-based filters (POPFile, With most client-based filters (POPFile,

SpamBayes, SpamAssassin) senders receive NO SpamBayes, SpamAssassin) senders receive NO NOTIFICATION if their mail isn't delivered. With NOTIFICATION if their mail isn't delivered. With most of these solutions, the user bears full most of these solutions, the user bears full responsibility to VERIFY that no good mail is responsibility to VERIFY that no good mail is blocked.blocked.

ASSP’s solution to this is that when spam is ASSP’s solution to this is that when spam is

blocked the SENDER RECEIVES NOTIFICATION, blocked the SENDER RECEIVES NOTIFICATION, and it does this without generating non-delivery and it does this without generating non-delivery reports that bounce and bounce again because reports that bounce and bounce again because spammers forge their from address. spammers forge their from address.

Catch-22Catch-22 Issue:Issue: Let’s say a client receives a non-delivery report, Let’s say a client receives a non-delivery report,

how can he (not in whitelist) send a message to the how can he (not in whitelist) send a message to the organization if he is still not in whitelist? I mean, if the organization if he is still not in whitelist? I mean, if the recipient or assp admin does not receive the notification, recipient or assp admin does not receive the notification, they will not know that there is a false positive and will not they will not know that there is a false positive and will not add the unknown client to whitelist...add the unknown client to whitelist...

Solution:Solution: Set up an email address and put it in the Spam- Set up an email address and put it in the Spam-Lover Address configuration option. Then modify the spam Lover Address configuration option. Then modify the spam error message to direct people to "500 Mail appears to be error message to direct people to "500 Mail appears to be unsolicited (spam) -- please forward this email to not-unsolicited (spam) -- please forward this email to not-spam@mydomain.com if you feel this is in error."spam@mydomain.com if you feel this is in error."Any false positives that bounce back to clients will hopefully Any false positives that bounce back to clients will hopefully be reported to the Mail Admin via the spam lover address be reported to the Mail Admin via the spam lover address (they just forward it), assuming they read the rejected (they just forward it), assuming they read the rejected email.email.

Email InterfaceEmail InterfaceAny user can help to improve ASSP’s spam filtering accuracy. Users can Any user can help to improve ASSP’s spam filtering accuracy. Users can use it to add addresses to the whitelist, report spam, or false-positives. To use it to add addresses to the whitelist, report spam, or false-positives. To use it, you must have it enabeled in the configuration, and have names set use it, you must have it enabeled in the configuration, and have names set for the addresses. The interface only accepts mail addressed to addresses for the addresses. The interface only accepts mail addressed to addresses at any of your localdomains, and only from "Accept All Mail" hosts, or at any of your localdomains, and only from "Accept All Mail" hosts, or authenticated SMTP connections.authenticated SMTP connections.

assp-whiteassp-white -- for whitelist additions -- for whitelist additions assp-spamassp-spam -- to report spam that got through -- to report spam that got through assp-notspamassp-notspam -- to report mis-categorized spam -- to report mis-categorized spam

Whitelisting:Whitelisting: Assuming that your local-domain is yourdomain.com, to add addresses Assuming that your local-domain is yourdomain.com, to add addresses to the whitelist, you’d create a message to to the whitelist, you’d create a message to assp-white@yourdomain.comassp-white@yourdomain.com. You can . You can either put the addresses in the body of the message, or as recipients of the message. either put the addresses in the body of the message, or as recipients of the message. For example, if you wanted to add all the addresses in your address book to the For example, if you wanted to add all the addresses in your address book to the whitelist, create a message to whitelist, create a message to assp-white@yourdomain.comassp-white@yourdomain.com and then add your entire and then add your entire address book to the BCC part of the message and click send. Note that no mail will be address book to the BCC part of the message and click send. Note that no mail will be delivered to any address except delivered to any address except assp-white@yourdomain.comassp-white@yourdomain.com (and that won't actually (and that won't actually be passed to your mail transport). Within a short time you'll receive a response from be passed to your mail transport). Within a short time you'll receive a response from ASSP showing the results of your mail. ASSP showing the results of your mail.

False Negatives:False Negatives: To report a spam that got through, simply forward the mail to To report a spam that got through, simply forward the mail to assp-spam@yourdomain.comassp-spam@yourdomain.com. It's best to forward it as an attachment, but you can . It's best to forward it as an attachment, but you can just forward it normally if you must. In a short time you will receive a confirmation. just forward it normally if you must. In a short time you will receive a confirmation.

False Positives:False Positives: The process is the same to report a miscategorized spam, but send The process is the same to report a miscategorized spam, but send it to it to assp-notspam@yourdomain.comassp-notspam@yourdomain.com..

Spam ReportSpam Report

BenchmarksBenchmarks Spam BucketSpam Bucket Ex-employee that left the company 5 Ex-employee that left the company 5

years agoyears ago Receives 50-80 spam mails per dayReceives 50-80 spam mails per day

Filter effectivenessFilter effectiveness SpamAssassin 60-65% effective in 2004SpamAssassin 60-65% effective in 2004 Deteriorated to 11% by 2006 Deteriorated to 11% by 2006

(267 of 2238 True Positives)(267 of 2238 True Positives) ASSP in first 3 weeks of operation 99.7%ASSP in first 3 weeks of operation 99.7%

(1336 of 1340 True Positives)(1336 of 1340 True Positives)

ASSP vs SpamAssassinASSP vs SpamAssassin SpamAssassinSpamAssassin

• is difficult to installis difficult to install• great investment in hand-made regular expressions and great investment in hand-made regular expressions and

header analysis to identify spamheader analysis to identify spam• Hand-crafted expressions are brittle as spammers adjust Hand-crafted expressions are brittle as spammers adjust

their strategiestheir strategies• Requires frequent updates to accurately identify spamRequires frequent updates to accurately identify spam

ASSPASSP• is low maintenanceis low maintenance• is easy to installis easy to install• is a complete spam blocking solution, not just a filter is a complete spam blocking solution, not just a filter

that must be integrated into your MTAthat must be integrated into your MTA• works with nearly every MTA on any OSworks with nearly every MTA on any OS• Poorly documentedPoorly documented

Before ASSPBefore ASSP

Turning ASSP onTurning ASSP on

With ASSPWith ASSP

stat.pl Statisticsstat.pl Statistics[root@smtp]# perl stat.pl /tmp/m.logAs of Mon Jan 22 21:48:46 2007 the mail logfile shows:0 proxy / smtp connections253 were dropped for attempted relays (0.0% of total).

31523 messages, 16758 were spam (53.2%) in 65 days for 485.0 messages per day or 257.8 spams per day1518 additions to / verifications of the whitelist (23.4 per day)14643 were judged spam by the bayesian filter (87.4% of spam)2115 were to spam addresses (12.6% of spam)0 were rejected for executable attachments (0% of spam)10121 were sent from local clients (68.5% of nonspam)842 were from whitelisted addresses (5.7% of nonspam)0 messages were passed to SPAMLOVERs3802 were ok after a bayesian check (25.8% of nonspam)1498 addresses are on the whitelist

0 hits on the blacklist0 resulted in spam (0.0% of Bayesian spam, 0.0% of blacklist hits)0 resulted in non-spam (0.000% of blacklist hits)

ASSP StatisticsASSP Statistics

IssuesIssues VacationVacation Auto RepliesAuto Replies TLS and secure SMTPTLS and secure SMTP ASSP is site based, not per-userASSP is site based, not per-user

Lessons LearnedLessons Learned Whitelist + spambucket + Bayesian Whitelist + spambucket + Bayesian

is a great spam filtering strategyis a great spam filtering strategy The default is SPF failures will filter The default is SPF failures will filter

even if whitelistedeven if whitelisted Be very careful what you put in the Be very careful what you put in the

relay hosts listrelay hosts list ASSP is not multi-process or multi-ASSP is not multi-process or multi-

threadedthreaded

UtilitiesUtilities rebuildspamdb.plrebuildspamdb.pl repair.plrepair.pl move2num.plmove2num.pl stat.plstat.pl

DemoDemo Web configurationWeb configuration Mail analyzerMail analyzer

Resources on the InternetResources on the Internet http://www.spamland.comhttp://www.spamland.com http://antispam.yahoo.comhttp://antispam.yahoo.com http://www.openspf.orghttp://www.openspf.org

QuestionsQuestions