lessons learned while writing a java mailserver by richard o. hammer
TRANSCRIPT
Lessons Learnedwhile
Writing a Java Mailserver
by Richard O. Hammer
about me
• 70s – BSEE, coded in Fortran and APL.
• 80s – went to CS graduate school at UNC-CH, ABD. Started a building business.
• 90s – started a libertarian think tank.
• Y2K – returned to programming to make a living, focus on Java and Internet programming.
• You can’t get a job without experience.
• You can’t get experience without a job.
• So I needed a project to work on at home.
The Familiar Catch-22
Why this Project?
• Spam is an interesting problem to me. It is a “tragedy of the commons”, an example of lawless behavior in a public space.
• An email service could eliminate spam in a novel way – by charging unlisted senders.
• Along the way I would learn about programming at the level of sockets and network protocols – where the problem of Internet lawlessness seems to start.
Minimal Specification:to offer an email service which
eliminates incoming spam
• Customers can manage a list of “accepted” senders
• All messages from other “unlisted” senders are waylaid. Automatic email is sent back to these senders, explaining the situation, giving them the option to pay for delivery
• Works with usual email client, such as Mozilla Thunderbird or Outlook Express.
Wish-List Specification:(postponed)
• Transform payment into bond, with easy way for customer to order refund.
• Add other ways for unlisted senders to authenticate themselves.
• Make the service scalable, to handle high volume of traffic.
• Install defenses against denial of service attacks.
• And many more features.
This Presentation Is Not
• Entirely up to date (by a few years)
• Comprehensive
Yet It Is
• Basic material that anyone who programs email will need to know
Protocols Specification
• I want people to be able to have email accounts with me. What do I have to offer?
• Protocols required?
• Services required?
Email Clientsuch as
Thunderbird or Outlook Express
simplified SMTP
How an Email message is Transmitted
Sender
SMTPMailserveror
Mail Transfer Agent
Sender’s ISP
Recipient’s ISP
Recipient
POP3
Mailserveror
Mail Transfer Agent
Email Clientsuch as
Thunderbird or Outlook Express
A look at SMTP
• RFC (821, 1982), superseded by 2821, 2001
• client opens a socket to port 25 on the SMTP server
• SMTP server identifies itself
SMTP in Actiontelnet from DOS
C:\>telnet XDomain.com 25
220 gork.XDomain.com ESMTP Postfixhelo mailscreen.net250 gork.XDomain.commail from:<[email protected]>250 Okrcpt to:<[email protected]>250 Okdata354 End data with <CR><LF>.<CR><LF>Here is a message to you..250 Ok: queued as 63235619F2EDquit221 Bye
Connection to host lost.
Content of SMTP dataRFC (822) 2822
data354 End data with <CR><LF>.<CR><LF>Received: from BILL ([73.175.198.24]) by vm024.mailsvcs.com for [email protected]; Wed, 03 Jan 2007 07:07:50 -0600 (CST)Date: Wed, 03 Jan 2007 08:07:49 -0500From: "Bill Hill" <[email protected]>Subject: RE: UFO sightingTo: "Richard Hammer" <[email protected]>MIME-version: 1.0Content-type: text/plain; charset=windows-1250Content-transfer-encoding: 7bit
Hi Rich,
Interesting! Maybe E. Jackson should take a look at it :) Bill.250 Ok: queued as 63235619F2ED
MIME HEADERSRFC 2045
HEADERS
RFC 2822 headers end with a blank line
SMTP (RFC 2821) envelopevs
RFC 2822 headers
mail from:<[email protected]>250 Okrcpt to:<[email protected]>250 Okdata354 End data with <CR><LF>.<CR><LF>From: "Bill Hill" <[email protected]>To: "Richard Hammer" <[email protected]>
My message to you..250 Okquit221 Bye
Addresses in SMTP envelopemay differ from
Addresses in headers
Socket Programming
• java.net.Socket
• java.net.ServerSocket
• On top of TCP
public class GetterServer implements Runnable{ boolean gracefulShutdownOrdered; ServerSocket receiverServerSocket; Socket receiverConnectionSocket;
public GetterServer() throws Exception{ receiverServerSocket = new ServerSocket(25); }
public void run(){ while(!gracefulShutdownOrdered){ try{ try{ receiverConnectionSocket = receiverServerSocket.accept(); }catch (IOException e){ if(gracefulShutdownOrdered) break; else{ logStackTrace("Exiting GetterServer.", e); break; } } Getter myMan = new Getter(); myMan.handleConnection(receiverConnectionSocket); }catch (RuntimeException runt){ logStackTrace(runt); } } } ...}
Top Level Codefor a
SMTP server
protocol programming
• We will look at three methods from class Getter, adapted from:
org.apache.james.smtpserver.SMTPHandler
class Getter{ //adapted from void handleConnection(Socket socket) { remoteHost = socket.getInetAddress().getHostName(); remoteIP = socket.getInetAddress().getHostAddress(); in = new BufferedInputStream(socket.getInputStream(), 7024); smtpCommandLineReader = new CommandLineReader(in); out = new InternetPrintWriter(socket.getOutputStream(), "Getter says: ", true);
out.println("220 " + helloName + " ready ");
String clientCommand = null; try { boolean getAnotherCommand; do { clientCommand = smtpCommandLineReader.readLine(); getAnotherCommand = parseCommand(clientCommand); } while (getAnotherCommand); } catch (Exception e) { if (shutdownOrdered) { //log normal closing return; } else if (e instanceof SocketException) { //log unusual condition } else {// might be major //log error } } closeConnections(); }
method handleConnection(Socket)
private boolean parseCommand(String command) throws Exception { if (command == null) return false; String argument = null; String argument1 = null; command = command.trim(); if (command.indexOf(" ") > 0) { argument = command.substring(command.indexOf(" ") + 1); command = command.substring(0, command.indexOf(" ")); if (argument.indexOf(":") > 0) { argument1 = argument.substring(argument.indexOf(":") + 1); argument = argument.substring(0, argument.indexOf(":")); } } if (command.equalsIgnoreCase("HELO")) doHELO(command, argument, argument1); else if (command.equalsIgnoreCase("EHLO")) doEHLO(command, argument, argument1); else if (command.equalsIgnoreCase("MAIL")) doMAIL(command, argument, argument1); else if (command.equalsIgnoreCase("RCPT")) doRCPT(command, argument, argument1); else if (command.equalsIgnoreCase("AUTH")) doAUTH(argument); else if (command.equalsIgnoreCase("DATA")) { doDATA(); } else if (command.equalsIgnoreCase("QUIT")) doQUIT(command, argument, argument1); else doUnknownCmd(command, argument, argument1); return !(command.equalsIgnoreCase("QUIT") == true || shutdownOrdered); }
method parseCommand(String)
private void doMAIL(String command, String argument, String argument1) { if (!gotEhlo) { out.println("503 bad sequence of commands, send ehlo or helo first"); return; } if (argument == null || !argument.equalsIgnoreCase("FROM") || argument1 == null) { out.println("501 Usage: MAIL FROM:<sender>"); return; } String sender = argument1.trim(); if (sender.length() < 2 || sender.charAt(0) != '<' || sender.charAt(sender.length() - 1) != '>') { out.println("501 Usage: MAIL FROM:<sender>"); return; }
method doMAIL(String, String, String)
//method doMAIL concluded
MailAddress senderAddress = null; sender = sender.substring(1, sender.length() - 1); if (sender.length() > 0) {// the usual case try { senderAddress = new MailAddress(sender); } catch (ParseException e) { out.println("501 Failure to parse InternetAddress from " + "\"" + argument1.trim() + "\". " + e.getMessage()); return; }
// screen senders with mailscreen addresses if (Services.isToLocalDomain(senderAddress)) { if (authenticatedUser == null) { out.println("530 Authentication Required"); return; } else if (!CustomerSet.isMsAddress(senderAddress)) { out.println("550 No such address"); return; } } } else { // sender.length() == 0, must have been <> // let senderAddress remain null, as that is our signal of mail from <> } setEnvelopeFrom(senderAddress); out.println("250 Sender " + sender + " OK"); }
Mailscreen.netas it now runs
• A free offering of the service has been available since March 2004
MAILSCREEN
PAYPAL
TO: [email protected]: Unlisted Sender Payment Required Message
TO: Unlisted SenderFROM: [email protected]
T 0.0
T 0.2
T 1.0
HTTPS URL
Unlisted Sender PaysHTTPS
SMTPCLIENT
SMTPSERVER
POP3MAILBOXES
TOMCATJSP
POSTGRES
Instant PaymentNotificatonHTTP POST
Echo Parameteres
T 1.2
T 1.4
"VERIFIED"
T 1.1
T 1.3
TO: [email protected]: Unlisted Sender
Unlisted Sender
T 0.1T 2.0
Dear [email protected]:
Mailscreen.net has received your email for a Mailscreen customer, [email protected].
Unfortunately, since your email address is not registered as an accepted sender to this customer, Mailscreen will not deliver your message to the customer until you make a payment as described in the next paragraph.
The charge to send this message will be $0.50. To make this payment click on the link below. It will take you to PayPal.com where you can make payment, and where you can open a new PayPal account if you do not already have one. Mailscreen does not accept any other form of payment at present.
https://www.paypal.com/xclick/[email protected]&item_name=accept+email&invoice=060410.204815c5i0&amount=0.50&return=http%3A//mailscreen.net/thanks.jsp
Please make your payment promptly. We cannot promise fulfillment if your payment comes after four calendar days, because we must limit the number of messages that we hold at Mailscreen.
If you would like more specific information to identify the message which we received from you and which we are holding awaiting payment, here are some identifying headers from that message.
Date: 10 April 2006 From: [email protected] Received: FROM nc-60-41-56-62.dyn.yahoo.net ([60.41.56.12]) BY mailscreen.net; Mon, 10 Apr 2006 20:48:15 -0400 (EDT)
Sincerely,Mailscreen.net
Payment Required Message
<% Enumeration en = request.getParameterNames(); StringBuffer replyBuf = new StringBuffer("cmd=_notify-validate"); while(en.hasMoreElements()){ String paramName = (String)en.nextElement(); String paramValue = request.getParameter(paramName); replyBuf.append("&").append(paramName).append("=") .append(URLEncoder.encode(paramValue,"UTF-8")); } // post back to PayPal system to validate URL url = new URL("http://www.paypal.com/cgi-bin/webscr"); URLConnection uc = url.openConnection(); uc.setDoOutput(true); uc.setRequestProperty("Content-Type","application/x-www-form-
urlencoded"); PrintWriter pw = new PrintWriter(uc.getOutputStream()); pw.println(replyBuf.toString()); pw.close(); BufferedReader in = new BufferedReader( new InputStreamReader(uc.getInputStream())); String res = in.readLine(); in.close(); if (!"VERIFIED".equals(res)){ log("Payment failed. res not VERIFIED but "+ res); return; }
Instant Payment Notification, in the JSP that PayPal calls
JSP Control Consoles
• Administrator
• Customer
PROBLEM
How can calls from Web Appreach the objects in the mailserver?
SOLUTION CHOSEN
Run Web App and Mailserver inthe same JVM
public class StartServlet extends HttpServlet {
public void init(){ ServerManager.start(); }
public void destroy(){ ServerManager.myOneServerManager.startGracefulShutdown(); }}
Starting the MailserverIt Is a
Servlet
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd"><web-app> <servlet> <servlet-name>do it damn it</servlet-name> <servlet-class>common.StartServlet</servlet-class> <load-on-startup>3</load-on-startup> </servlet> <!-- security for regular customers --> <security-constraint> <web-resource-collection> <web-resource-name>customerResource</web-resource-name> <url-pattern>/customer/*</url-pattern> </web-resource-collection> <auth-constraint> <role-name>customer</role-name> </auth-constraint> <user-data-constraint> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint>...
mailscreen web.xml file
<Server port="8005" shutdown="SHUTDOWN"> <Service name="Catalina"> <Connector port="80" maxThreads="15" minSpareThreads="1" maxSpareThreads="2" enableLookups="false" redirectPort="443" acceptCount="6" connectionTimeout="20000" disableUploadTimeout="true" /> <Connector port="443" maxThreads="10" minSpareThreads="1" maxSpareThreads="2" enableLookups="false" disableUploadTimeout="true" acceptCount="5" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" /> <Engine name="Catalina" defaultHost="localhost"> <Host name="mailscreen.net" appBase="/usr/website/WebRoot" unpackWARs="true" autoDeploy="true"> <Alias>www.mailscreen.net</Alias> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="msLogs" prefix="access." suffix=".log" pattern="common" resolveHosts="false"/> <Realm className="org.apache.catalina.realm.JDBCRealm" debug="99" driverName="org.postgresql.Driver" connectionURL="jdbc:postgresql://localhost:3246/pqlms" connectionName="mailData" connectionPassword="e3$fe4uii0p" userTable="uLogin" userNameCol="uName" userCredCol="cred" userRoleTable="uRole" roleNameCol="power" /> <Context path="" docBase="." /> </Host> </Engine> </Service></Server>
Tomcat server.xml file
Complexity of SMTP sending, Classification draft
M4 - a message as received in SMTP, with potentially: • many domains among the recipients • many delivery attempts (try or retry) necessary for each domain • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server
succeeds M3 - part of an M4, with recipients at only one domain, but with potentially: • many delivery attempts (try or retry) necessary for each domain • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server
succeeds M2 - one of attempts (try or retry) to deliver a M3, but with potentially: • many servers (IP addresses) to attempt during each delivery attempt • many recipient addresses to attempt when a connection to a server
succeeds M1 - part of an M2, the attempt to deliver an M3 to one server (IP address), but
with potentially: • many recipient addresses to attempt in this connection M0 - part of an M1, the attempt to deliver an M3 to one user address at one
server (IP address) during one try or retry attempt
Mailscreen
Key Classes
DistinctDataMessage DDMString: ddmId, statusboolean: disposable
M4List: envelopeToMailAddress: envelopeFromString: idFile: dataFile
MessageToOneRemoteDomainMORD
List: recipientsString: domainName
RemoteRecipient
RRMailAddress: addressString: status
MessageToMsRecipientMSR
void deliverTermsMet()void stopWaiting()
MsMailBoxEntryvoid writeTo(OutputStream)features
ForwardedMessagetoCustomer
features
MessageToNonMs
MsMessageFeaturesCustomer: customer
MailHeaders: headers
(abstract)
(abstract)
(interface)
1..*
1..*
1
1
1
1
1
1
This notification pertains to a message you sent. You may identify that message by these headers taken from it:
To: [email protected] Subject: coming through Date: Sun, 28 Mar 2004 20:22:21 -0500
This message could not be delivered to one or more recipients. A permanent failure was encountered while attempting to deliver to: [email protected]
No further action will be taken here on your message.
Delivery Status Notification
Lessons about Spam
• Spam is not bad enough to drive people to the tactless solution offered by Mailscreen.
• People do not want to risk the possibility that a cherished contact might receive a “payment required” message.
• The problem is not just technical, because there are many technical solutions.
• The problem is sociological.
References
• RFC Index http://www.rfc-editor.org/rfc-index.html
• Delivery Status Notifications RFC 1891, 1894
• Java Network Programming, 2nd ed., Elliotte Rusty Harold, 2000
• Programming Internet Email, David Wood, 1999
Libraries Used
• org.xbill.DNS
• javax.mail
Source Available to You at
• http://mailscreen.net/jug.zip
• for one week, until Jan. 22, 2007
Thanks To
• Sun Microsystems, for Java, the Java API
• Apache James project
• Ralph Cook
• IETF email list, [email protected]
• Triangle Java Users Group