DovecotIMAPServer

TimoSirainenAugust2008

http://www.dovecot.org/

Dovecot

Pictures from Wikipedia, by Cyril Thomas and Carcharoth

Features

•  O;enhasbe>erperformancethancompeAAon.– OpAmizedforminimizingdiskI/O(index/cachefiles)

•  Highlyconfigurablefordifferentenvironments– Supportforstandardmboxandmaildirformats,aswellasanewDovecot‐specifichigh‐performancedboxformat

– SupportsNFSandclusteredfilesystems,soonsupportforinternalmulA‐masterreplicaAon

– ExtremelyflexibleauthenAcaAon•  PosUixandEximsupportDovecotforSMTPAUTH

•  Admin‐friendly/self‐healing– Allerrorsarelogged– Understandableerrormessages– Detected(index)corrupAongetsfixedautomaAcally

History

•  DovecotdesignwasstartedaroundJune2002– Why?

•  FirstreleasewasJuly2002•  Late2003aredesignstarted•  v1.0.0releasedApril13th2007•  v1.1.0releasedJune21st2008•  v1.2devtreealreadyhasalotofnewfeatures•  95%ofcodeiswri>enbyme–othershavemostlywri>enauthenAcaAonrelatedcode

Development

•  Alldiscussionsinmailinglist– ItrytoanswerallquesAonsothersdon’tanswer

• Mercurialforversioncontrol– DistributedVCSshouldmakeiteasierforotherstocontributecode

•  Currentlynobugtrackingsystem– Ifearitwouldmakemylifemoredifficult– BTSthatfullyintegratedwithmailinglistwouldbenice

CodeDesign• Wri>enwithClanguage

– UsesseveralDovecot‐specificAPIstomakecodingeasierandmoredifficulttocausesecurityholes• Memorypools,datastack–avoidfree()

• Buffers,stringsandtype‐safearrays• Stackableinput/outputstreams

– Somesayit’sveryunlikeanyotherCcode

•  Prefersto(assert‐)crashratherthanconAnuewithpossiblybadstate

•  Unittestsareslowlybeingadded..– Helpwouldbeappreciated.

DovecotProcesses

1.  IMAPcommand:LOGINusernamepassword2.  Forwardusernameandpasswordtoauthprocess3.  Success/Failurereply(reasonisn’treturned–seelogforthat)4.  “Logmein”request–TCPsocketfdsentviaUNIXsocket5.  AuthverificaAon(tomakesurepre‐logindidn’tfakeit)6.  Returnsuserdbfields(home,UNIXUID&GID,etc.)or

“Internalfailure”(pracAcallynever)7.  a)Returnssuccess/failure–pre‐loginstopsIMAPprocessing

b)IMAPprocessforked&fdtransferred8.  IMAPreply:OKLoggedin.

AuthenAcaAon•  AuthenAcaAonmechanisms:PLAIN,CRAM‐MD5,DIGEST‐MD5,Kerberos,etc.

•  Passwordschemes:Plaintext,CRYPT,MD5,SHA1,SHA256,SSHA,etc.

•  Passworddatabases:User<‐>passwordmappingmostly(PAM,SQL,LDAP,etc)

•  Userdatabases:User’shomedir,UNIXUID&GID,othersevngslikequota(passwd,SQL,LDAP,etc.)

•  Passdb/userdbseparaAonallowse.g.passdbPAM+userdbLDAPorpassdbSQL+userdbstaAc

•  SupportformulApledbs:Supportforbothsystem(passwd)andvirtual(e.g.SQL)users(orforanyotherreason)

•  SQL/LDAPlookupsarefullyconfigurable

IMAPProtocol

•  Baseprotocoliscomplex–difficulttoimplementitcorrectly(bothclient&server)

•  Flexible–manydifferentwaystoimplementaclient(online&offline–definedlater)

•  Extensible–therearealotofextensions.IETFgroups:–  imapextcreatedmanyextensionsovermanyyears(ACL,SORT,THREAD,etc).ShutdownonJune2008.

– Lemonadecontainsmanyextensionsmainlyintendedformobileclients(forward‐without‐download,etc)

– MessageOrganizing(morg)groupisstarAngup(e.g.mulA‐mailboxsearch,mailboxmetadata,newcomparator,etc)

•  TalksaboutasimplifiedIMAP5protocolhavestarted

Dovecot’sIMAPExtensions

•  v1.0:SASL‐IRSORTTHREAD=REFERENCESMULTIAPPENDUNSELECTLITERAL+IDLECHILDRENNAMESPACE

•  v1.1:UIDPLUSLIST‐EXTENDEDI18NLEVEL=1STATUS‐IN‐LIST(dra;)

•  v1.2:CONDSTOREQRESYNCWITHINIDSEARCHRESSEARCH=INTHREADESEARCH

•  Future:Lemonadeextensions(CATENATE,URLAUTH,NOTIFY,..)

ImapTestIMAPservertester

•  Wri>enoriginallyforDovecotstresstesAng– Foundalotofcrashes,hangsandmailboxcorrupAononotherIMAPserversaswell

•  TestsIMAPservercompliancewithstaActestsanddynamicrandomstresstesAng.

•  DovecotiscurrentlytheonlyIMAPserverthatfullypassesallofImapTesttests.

•  Mostotherserversfailinmanydifferentways.– “Professional”IMAPserversfromlargecompaniesareamongtheworst.

•  h>p://imapwiki.org/ImapTest

IMAPServerPerformance

•  Difficulttobenchmark•  Dependsalotonclients(onlinevs.offline–moreonnextslides)

• Whatdatatoindex/cache?

Offlineclients

•  Typicallydownloadsthenewlyseenmessages’bodiesonceandcachesthemlocally

•  O;encanbeconfiguredtodownloadimmediatelyvs.downloadwhenreading

•  Someuseserversidesearches(Thunderbird)andsomedon’t(Outlook–ifsomemessageshaven’tbeendownloaded,thosearen’tsearched)

•  Usuallyalsofetchmessages’metadataonce(headers,receiveddate)

•  Cachingmayhelp,butnotthatmuch

Onlineclients

• Webmailso;enkeepaskingforthesameinformaAonoverandoverandoveragain

•  Pineandsomewebmailscachewhatthey’vealreadyseen,butnotpermanently

• Mu>(withoutlocalcache)andsomeothersfetchallmessages’metadataeveryAmewhenopeningamailbox

•  Cachingisveryuseful,butdifferentclientswantdifferentmetadata

DovecotCacheFile

•  Dynamic:cachesonlywhatclientswant.– Specificmessageheaders(From:,Subject:,etc)– MessageMIMEstructureinformaAon– Messagesent/receiveddate– etc.

•  Cachingdecisionsforeachfield:“no”,“temporary”,“permanent”

•  Unusedfieldsdroppeda;eramonth.•  Cacheddataneverchanges(IMAPguarantees)•  Cachefilegets“compressed”onceinawhile•  O;enabout10‐20%ofmailboxsize

DovecotIndexFiles•  dovecot.indexcontainscurrentmetadata

– Fixedsizerecordsonly,onepermessage

– IMAPUniqueIDnumber(UID)idenAfiesmessages– Flags(\Seen,\Answered,etc.)– Keywords(aka.tags,labels,customflags)asabitmask(opAmizedforfewkeywords)

– Extensiondata:mboxfileoffsets,cachefileoffsets,modseqnumber(v1.2CONDSTORE),etc.

•  Lazilycreated/updatedsincev1.1– dovecot.index.loghasallthelatestchanges.dovecot.indexisupdateda;er1kBofnewdatahasbeenwri>entothe.log

DovecotIndexFiles

•  dovecot.index.logcontainstransacAonlog– Somewhatsimilartodatabases’transacAonlogsorfilesystemjournals.

– Containsallchangestobedonetodovecot.index.•  A;erdovecot.indexisreadonce,Dovecotusuallyneverreadsitagainbutonlyupdatesthein‐memorycopyfromdovecot.index.log– VeryefficientwithNFS/clusteredfilesystems!

Locking

•  DovecotusesseveraltechniquestoavoidtradiAonalread/writelocking(nowaiAng!)

•  dovecot.index.logiscurrentlywrite‐lockedwhenwriAng,readsarelockless– O_APPENDcouldbeusedtomakewrite‐lockless

•  dovecot.indexisread‐locked.Ifwrite‐lockingfails,thefileisrecreatedinsteadofwaiAng.

•  dovecot.index.cachedoesshortwritelockstoreservespace.Readsarelockless.

•  Maildirsyncingrequireslocking(orinoAfy)

Plugins

•  DovecotpluginscanhookintoalmostanythingandmodifyDovecot’sbehavior– AccessControlLists– Quota– Fulltextsearchindexes– Readinggzip‐compressedmboxes/maildirfiles

•  CanaddnewIMAPcommands(althoughenhancingexisAngcommandscouldusemorework)

•  Implementnewmailstoragebackends(virtual,SQL,IMAPproxying)

MailboxFormats• mbox

– Oldestformat,widelysupported

– Onemailbox=onefile• Slowtodeletemessagesfromthemiddle

• Maildir– Onefile=onemessage

• Fasttodeletemessages

• Slow(er)toreadthroughallmessages

•  dbox– Dovecot’sextensibleandhigh‐peformancemailboxformat

DboxMailboxFormat

•  Eitheronefile=onemessage– Locklessreads– MaindifferencetoMaildir:filenamedoesn’tchange

•  Oronefile=mulAplemessages– Somelockingnecessaryforreads– Anewfileiscreatedwhenoldonegrowsaboveconfiguredsize(e.g.2MB)orwhenthefileisolderthanndays(usefulforincrementalbackups)

– Changingusedfilesizechangesread/deleteperformance

– Notfullyimplementedyet

DboxMailboxFormat

•  PrimarymetadatastorageisDovecot’sindexfiles– Metadataisbackeduptodboxfilesaboutonceaday,soifindexesarelost,allflagswon’tgetlost

•  Messages’metadataisextensiblewitharbitrarykey=valuepairs.Thiswillusefulinfuture:– SeparaAnga>achmentstoasingleinstancestorage– Storingmessagescompressed

•  ExtremelyeasyandfastmigraAonfromMaildir– CompaAbilitymode:Renamecur/todboxdir,movefilesinnew/andmetadatafiles

MulA‐MasterReplicaAon•  Necessary?•  Notpossibletoimplementreliablywithlow‐levelreplicaAonbecauseofIMAPUniquemessageID(UIDs)

•  IMAPUIDsareincreasing32bitnumbers– Globalsyncrequiredorconflictswillhappen– ConflictsalwayspossiblewithM‐MreplicaAon,butfixingnotpossiblewithlow‐levelreplicaAon

MulA‐MasterReplicaAonGoals

•  SynchronousoperaAon:Neverloseevenasinglemail(if1..nreplicasdie)

•  Performanceshouldbegoodinall‐acAvemulA‐mastersetup

•  DesynchronizaAonshappen:FixthemandconflictscausedbysyncingautomaAcallyandefficiently

MulA‐MasterReplicaAon

•  Savingmails(themostcriAcalpart)•  Expungingmails•  UpdaAngmessageflags/keywords•  CONDSTOREextension:UpdaAngmodificaAonsequences(modseqs)– Per‐msgmodseqincreasesoneveryflagetc.change– ModseqsarealsoveryusefulforreplicaAon

•  Mailboxcreates,renames,deletes,etc.•  Twoverydifferentdata:PotenAallyhugemessagebodiesvs.smallmetadata

ReplicaAonParts

•  3mostlyseparateparts:1.  Incrementalmailboxsync

2.  Fixinga(large)mailboxdesync3.  Syncingmailboxlist(mailboxcreates,deletes,

renames)

•  Implementedindifferentstages(1‐3).Incrementalmailboxsyncisthemostdifficulttogetworkingcorrectlyandfast.

ReplicaAonMaster

•  IMAPUIDsmustbegloballygrowing‐>UIDscanbeallocatedonlybya“mailboxmaster”server

•  Mastermaymovebetweenservers(andatleastiniAallyitalwayswillifaserverwantstosaveanewmail)

•  MastercanalsohandleCONDSTOREextension’sSTOREUNCHANGEDSINCE.

•  Ifnetworkdiesbetweentwoservers,bothmayallocatethesameUID‐>UIDconflictthatmustbefixedlaterwhenserversseeeachothersagain

ReplicaAonProcesses

•  Simplertohaveseparateprocessesforseparatetasks.

•  Be>ersecurity:Lesscodethathaswriteaccesstousers’mailboxes

• WorkerprocesseswhentherecanbewaiAngonlocks,soworksAllconAnueselsewhere