a portable, extensible and efï¬cient implementation of

A Portable, Extensible and Efficient Implementation of Proactor Pattern

Alexander Babu Arulanthu, Irfan Pyarali and Douglas C. Schmidtfalex, irfan, [email protected]

http://www.cs.wustl.edu/f�alex, irfan, schmidtgDepartment of Computer Science

Washington University, St. Louis 63130

Abstract

The Proactor pattern [1] describes how to structure applica-tions and systems that effectively utilize asynchronous mech-anisms supported by operating systems. When an applicationinvokes an asynchronous operation, the OS performs the oper-ation on behalf of the application. This allows the applicationto have multiple operations running simultaneously withoutrequiring the application to have a corresponding number ofthreads. Therefore, the Proactor pattern simplifies concurrentprogramming and improves performance by requiring fewerthreads and leveraging OS support for asynchronous opera-tions.

The Adaptive Communications Environment (ACE) [2] hasimplemented a Proactor framework that encapsulates I/OCompletion Ports of Windows NT operating system. ThisACE Proactor abstraction provides an OO interface to thestandard C APIs supported by Windows NT. We ported thisProactor framework to Unix platforms that support POSIX4asynchronous I/O calls and real-time signals. This pa-per describes the design and implementation of this newPortable Proactor framework and explains how the de-sign and the implementation have been made so that theframework can be extensible, scalable and efficient. Weexplain how our design took care of keeping the old in-terfaces of the framework intact, still making the designhighly extensible and efficient. The source code for thisimplementation can be acquired from the ACE website atwww.cs.wustl.edu/ �schmidt/ACE.html .

1 The Proactor Pattern

1.1 Intent

The Proactor pattern presented in [1] supports the demultiplex-ing and dispatching of multiple event handlers, which are trig-gered by thecompletionof asynchronous events. This patternsimplifies asynchronous application development by integrat-ing the demultiplexing of completion events and the dispatch-

ing of their corresponding event handlers.

1.2 Motivation

The Proactor pattern should be applied when applications re-quire the performance benefits of executing operations concur-rently, without the constraints of synchronous multi-threadedor reactive programming. To illustrate these benefits, considera networking application that needs to perform multiple oper-ations concurrently. For example, a high-performance Webserver must concurrently process HTTP requests sent frommultiple clients. Figure 1 shows a typical interaction betweenWeb browsers and a Web server. When a user instructs a

1: HTTPrequest Web

Server

WebBrowser

FileSystem

3: read file4: send file

2: parse request

Figure 1: Typical Web Server Communication Software Ar-chitecture

browser to open a URL, the browser sends an HTTPGETre-quest to the Web server. Upon receipt, the server parses andvalidates the request and sends the specified file(s) back to thebrowser.

1

Developing high-performance Web servers requires the res-olution of the following forces:

� Concurrency– The server must perform multiple clientrequests simultaneously;

� Efficiency– The server must minimize latency, maximizethroughput, and avoid utilizing the CPU(s) unnecessarily.

� Programming simplicity– The design of the servershould simplify the use of efficient concurrency strate-gies;

� Adaptability– Integrating new or improved transport pro-tocols (such as HTTP 1.1 [3]) should incur minimal main-tenance costs.

A Web server can be implemented using several concur-rency strategies, such as multiple synchronous threads and re-active synchronous event dispatching. But such conventionalapproaches have drawbacks as discussed in [1]. The Proactorpattern provides a powerful technique that supports an effi-cient and flexible asynchronous event dispatching strategy forhigh-performance concurrent applications.

1.3 Concurrency Through Proactive Opera-tions

When the OS platform supports asynchronous operations, anefficient and convenient way to implement a high-performanceWeb server is to useproactive event dispatching. Web serversdesigned using a proactive event dispatching model handlethecompletionof asynchronous operations with one or morethreads of control. Thus, the Proactor patternsimplifies asyn-chronous Web servers by integrating completion event demul-tiplexing and event handler dispatching.

An asynchronous Web server would utilize the Proac-tor pattern by first having the Web server issue an asyn-chronous operation to the OS and registering a callback with aCompletion Dispatcher that will notify the Web serverwhen the operation completes. The OS performs the operationon behalf of the Web server and subsequently queues the resultin a well-known location. TheCompletion Dispatcheris responsible for de-queuing completion notifications andexecuting the appropriate callback that contains application-specific Web server code.

Figures 2 and 3 show how a Web server designed usingproactive event dispatching handles multiple clients concur-rently within one or more threads. Figure 2 shows the se-quence of steps taken when a client connects to the WebServer.

1. The Web Server instructs theAcceptor to initiate anasynchronous accept;

4: connect

Web Server

WebBrowser

Acceptor

CompletionDispatcher

HTTPHandler

OperatingSystem

3: handleevents

5: acceptcomplete

1: acceptconnections

2: accept(Acceptor,Dispatcher)

6:accept

complete

7: create

8: read (Handler,Dispatcher)

Figure 2: Client connects to a Proactor-based Web Server

2. The Acceptor initiates an asynchronous accept with theOS and passes itself as aCompletion Handler anda reference to theCompletion Dispatcher thatwill be used to notify theAcceptor upon completionof the asynchronous accept;

3. The Web Server invokes the event loop of theCompletion Dispatcher ;

4. The client connects to the Web Server;

5. When the asynchronous accept operation completes,the Operating System notifies theCompletionDispatcher ;

6. TheCompletion Dispatcher notifies the Accep-tor;

7. TheAcceptor creates anHTTP Handler ;

8. TheHTTP Handler initiates an asynchronous opera-tion to read the request data from the client and passes it-self as aCompletion Handler and a reference to theCompletion Dispatcher that will be used to no-tify the HTTP Handler upon completion of the asyn-chronous read.

Figure 3 shows the sequence of steps that the proactive WebServer takes to service an HTTPGETrequest. These steps areexplained below:

1. The client sends an HTTPGETrequest;

2. The read operation completes and theOperatingSystem notifies theCompletion Dispatcher ;

3. The Completion Dispatcher notifies theHTTPHandler (steps 2 and 3 will repeat until the entire re-quest has been received);

4. TheHTTP Handler parses the request;

2

Web Server

WebBrowser

FileSystem

CompletionDispatcher

HTTPHandler

OperatingSystem

1: GET/etc/passwd

2: read complete

3: readcomplete

4: parse request

6: write (File, Conn.,Handler, Dispatcher)

7: writecomplete

8: writecomplete5: read (File)

Figure 3: Client Sends requests to a Proactor-based WebServer

5. TheHTTP Handler synchronously reads the requestedfile;

6. TheHTTP Handler initiates an asynchronous opera-tion to write the file data to the client connection andpasses itself as aCompletion Handler and a ref-erence to theCompletion Dispatcher that will beused to notify theHTTP Handler upon completion ofthe asynchronous write;

7. When the write operation completes, the Operating Sys-tem notifies theCompletion Dispatcher ;

8. The Completion Dispatcher then notifies theCompletion Handler (steps 6-8 continue until thefile has been delivered completely).

The primary advantage of using the Proactor pattern is thatmultiple concurrent operations can be started and can run inparallel without necessarily requiring the application to havemultiple threads. The operations are started asynchronouslyby the application and they run to completion within the I/Osubsystem of the OS. The thread that initiated the operation isnow available to service additional requests.

In the example above, for instance, theCompletionDispatcher could be single-threaded. When HTTP re-quests arrive, the singleCompletion Dispatcher threadparses the request, reads the file, and sends the response tothe client. Since the response is sent asynchronously, multi-ple responses could potentially be sent simultaneously. More-over, the synchronous file read could be replaced with an asyn-chronous file read to further increase the potential for concur-rency. If the file read is performed asynchronously, the onlysynchronous operation performed by anHTTP Handler isthe HTTP protocol request parsing.

The primary drawback with the Proactive model is that theprogramming logic is at least as complicated as the Reac-tive model. Moreover, the Proactor pattern can be difficult

to debug since asynchronous operations often have a non-predictable and non-repeatable execution sequence, whichcomplicates analysis and debugging. But Patterns such as theAsynchronous Completion Token [4] can be applied to sim-plify the asynchronous application programming model [1].

1.4 Applicability

The Proactor pattern is used when one or more of the followingconditions hold:

� An application needs to perform one or more asyn-chronous operations without blocking the calling thread;

� The application must be notified when asynchronous op-erationscomplete;

� The application needs to vary its concurrency strategy in-dependent of its I/O model;

� The application will benefit by decoupling theapplication-dependent logic from the application-independent infrastructure;

� An application will perform poorly or fail to meet its per-formance requirements when utilizing either the multi-threaded approach or the reactive dispatching approach.

1.5 Structure and Participants

The structure of the Proactor pattern is illustrated in Figure 4using UML notation.

ProactiveProactiveInitiatorInitiator

CompletionCompletionHandlerHandler

AsynchronousAsynchronousResultResult

AsynchronousAsynchronousOperationOperation

CompletionCompletionDispatcherDispatcher

AsynchronousAsynchronousOperationOperationProcessorProcessor

<< instantiate >><< instantiate >>

<< parameter>><< parameter>><< call>><< call>>

<< call>><< call>> << call>><< call>>

<< parameter>><< parameter>>

<< call>><< call>><< parameter >><< parameter >>

Figure 4: Participants in the Proactor Pattern

The key participants in the Proactor pattern include the fol-lowing:

Proactive Initiator (Web server application’smain thread ):

� A Proactive Initiator is any entity inthe application that initiates anAsynchronousOperation . The Proactive Initiator regis-ters aCompletion Handler and aCompletionDispatcher with an Asynchronous OperationProcessor , which notifies it when the operationcompletes.

3

Completion Handler (the Acceptor and HTTPHandler ):

� The Proactor pattern usesCompletion Handler in-terfaces that are implemented by the application forAsynchronous Operation completion notification.

Asynchronous Operation (the methodsAsynchronousRead, Asynchronous Write , AsynchronousAccept andAsynchronous Transmit File ):

� Asynchronous Operations are used to exe-cute requests (such as I/O and timer operations) onbehalf of applications. When applications invokeAsynchronous Operations , the operations areperformedwithout borrowing the application’s threadof control.1 Therefore, from the application’s per-spective, the operations are performedasynchronously.When Asynchronous Operations complete, theAsynchronous Operation Processor del-egates application notifications to aCompletionDispatcher .

Asynchronous Operation Processor (the OperatingSystem ):

� Asynchronous Operations are run to completionby the Asynchronous Operation Processor .This component is typically implemented by the OS.

Asynchronous Result (the object passed to theCompletion Handler on completion of anAsynchronous Operation )

� For each Asynchronous Operation class,there is oneAsynchronous Result class. TheAsynchronous Operation classes use theAsynchronous Result classes to specify allthe parameters needed to carry out the AsynchronousOperations. TheAsynchronous Result objects arecreated by theAsynchronous Operation classesand are passed to theAsynchronous OperationProcessor on issuing the asynchronous calls. TheAsynchronous Result objects also contain theresults of the asynchronous operations and are used topass the results to theCompletion Handler s. Inaddition, theAsynchronous Result objects haveinformation such asAsynchronous CompletionToken s ACTs [4] which can be used by the applica-tions to uniquely associate the asynchronous methodcompletions with their invocations.

1In contrast, the reactive event dispatching model [5] steals the applica-tion’s thread of control to perform the operation synchronously.

Completion Dispatcher (theNotification Queue )

� The Completion Dispatcher is responsiblefor calling back to the application’sCompletionHandlers when Asynchronous Operationscomplete. When theAsynchronous OperationProcessor completes an asynchronously initiated op-eration, theCompletion Dispatcher performs anapplication callback on behalf of theAsynchronousOperation Processor . The CompletionDispatcher fills the result of the asynchronousoperation in theAsynchronous Result object andpasses that to theCompletion Handler .

2 WIN32 Implementation of the Proac-tor Pattern

In this section, we will discuss the design of the Proactorframework that was built only for the WIN32 I/O Comple-tion Ports. We will discuss how each of the participant of thepattern discussed in the previous section was implemented.

2.1 Asynchronous Operation Processor

The WIN32 I/O subsystem is theAsynchronous OperationProcessor.

It definesOVERLAPPEDstructure which contains informa-tion used in asynchronous input and output (I/O). This struc-ture is passed to the asynchronous APIs to specify the addressof the data, such asOffset etc, in a file.

The WIN32 I/O subsystem provides the following APIs toexecute asynchronous I/O calls.

� ReadFile: To issue asynchronous read on a stream or afile handle.

� WriteFile: To issue asynchronous accept on a stream or afile handle.

� AcceptEx:To issue asynchronous accept from a sockethandle.

� TransmitFile: To initiate transmitting a file asyn-chronously.

� CancelIO:To cancel all pending input and output (I/O)operations that were issued by the calling thread for thespecified file handle.

� GetCompletionStatus:To query the OS for completionsof asynchronous I/O.

� PostQueuedCompletionStatus:To post a completion to aCompletion Port

4

2.2 Asynchronous Operation

Asynchronous Read Stream , AsynchronousWrite Stream , Asynchronous Read File ,Asynchronous Write File , AsynchronousAccept and Asynchronous Transmit File are thedifferent asynchronous operations that are currently supportedin the framework. The UML diagram 5 shows the familyof classes which implement the variousAsynchronousOperation s.

AsynchronousAsynchronousOperation ClassesOperation Classes

AsynchronousAsynchronousOperation BaseOperation Base

AsynchronousAsynchronousRead StreamRead Stream

AsynchronousAsynchronousAcceptAccept

AsynchronousAsynchronousTransmit FileTransmit File

AsynchronousAsynchronousWrite StreamWrite Stream

AsynchronousAsynchronousRead FileRead File

AsynchronousAsynchronousWrite FileWrite File

Figure 5: Asynchronous Operation Classes

2.2.1 Asynchronous Operation Base

The classAsynchronous Operation Base abstractsout all the common code found in the individual asynchronousoperation classes. This class provides the following APIs.

� open : Applications use theopen method to register theI/O handle and theCompletion Handler withthe Completion Dispatcher . This API calls theCreateIoCompletionPort API of the WIN32 op-erating system to register theI/O handle with theCompletion Port of the I/O subsystem.

� cancel : This function cancels all the asynchronouscalls issued on a completion port on a particular I/O han-dle. This is implemented using theCancelIO .

2.2.2 Asynchronous Read Stream

After the open method of the AsynchronousOperation Base is called, applications call thereadAPI of this class to issue an asynchronous read on a stream.The handle to be read from is given via theopen call.read API takes the following parameters

� Handler: The completion handler to be called whenthe operation is completed. ThisHandler defines thecall back methodhandle read stream .

� message block: Buffer where the data has to beread.

� bytes to read: Maximum number of bytes thatcould be read from the socket.

� ACT (Asynchronous Completion Token):Magic cookie to identify the asynchronous read invoca-tion when it completes.

The following steps take place whenread API is called.

� read creates anAsynchronous Read StreamResult object with all the information that are neededto carry out the asynchronous operation such asI/Ohandle andbytes to read and also the informa-tion needed to call back the application such asHandlerandACT.

� read passes this result object to a worker method calledshared read . This worker method is shared by theAsynchronous Read File operation class.

� shared read calls theReadFile WIN32 API to is-sue the asynchronous read.ReadFile takes pointerto theOVERLAPPEDstructure. Since we want to pre-serve more information such asACTandHandler alongwith OVERLAPPEDstructure, we pass the pointer toAsynchronous Read Stream Result object tothe ReadFile call. To achieve this, all the Asyn-chronous Result classes derive from theOVERLAPPEDstructure (refer to Figure 6).

2.2.3 Asynchronous Write Stream

After the open method of the AsynchronousOperation Base is called, applications should callthewrite API of this class to issue an asynchronous write ona stream. Thehandle to write to is already given via theopen call. Thewrite API takes the following parameters

� Handler: The completion handler to be called whenthe operation is completed. ThisHandler defines thecall back methodhandle write stream .

� message block: Buffer that contains the data to besent.

� bytes to write: Number of bytes to be written tothe socket.

� ACT: Magic cookie for this asynchronous write invoca-tion.

The following steps take place whenwrite is called.

� write creates anAsynchronous Write StreamResult object with all the information that are needed

5

to carry out the asynchronous operation such asI/Ohandle andbytes to write and also the informa-tion needed to call back the application such asHandlerandACT.

� write passes this result object to a worker methodcalledshared write . This worker method is sharedby theAsynchronous Write File Operationclass.

� shared write calls theWriteFile WIN32 API to is-sue the asynchronous write.WriteFile takes pointerto the OVERLAPPED structure. As discussed inAsynchronous Read Stream operation, we passthe pointer toAsynchronous Write Stream Resultobjectto theWriteFile so that additional information suchasACT, Handler can be passed around along with theOVERLAPPEDstructure.

2.2.4 Asynchronous Read File

This class extends the functionality of theAsynchronousRead Stream class to do asynchronous read on a file.This operation is very similar toAsynchronous ReadStream except that it reads from a file handle instead ofsteam handle. Thehandle to be read from is given via theopen call. TheHandler ’s handle read file is calledon completion. Whenread is called, aAsynchronousRead File Result class object is created and passed tothe shared read method of theAsynchronous ReadStream .

2.2.5 Asynchronous Write File

This class extends the functionality of theAsynchronousWrite Stream class to do asynchronous write to a file.This operation class is very similar to theAsynchronousWrite Stream except that it writes from a file handle in-stead of steam handle. The streamhandle is given viathe open call. The Handler ’s handlewrite file is calledon completion. Whenwrite is called, aAsynchronousWrite File Result class object is created and passed totheshared read method of theAsynchronous WriteStream .

2.2.6 Asynchronous Accept

After the open method of the AsynchronousOperation Base is called, applications call theac-cept API of this class to issue an asynchronous write on astream. Thelisten handle where the connection is to beaccepted is given via theopen call.Theaccept API takesthe following parameters:

� Handler: This class defines call back hook methodhandle asynch accept which gets called when theaccept completes.

� message block: Buffer to accept initial data that isread on the socket.

� bytes to read: Number of initial bytes to be readfrom the socket.

� ACT: Magic cookie for this asynchronous accept invoca-tion.

The following steps take place whenaccept is called.

� accept creates an Asynchronous AcceptResult object with all the information that are neededto carry out the asynchronous operation which areI/Ohandle and number of bytes to read andalso the information needed to call back the applicationwhich areHandler andACT.

� accept calls theAcceptExWIN32 API to issue theasynchronous accept. AcceptEx takes pointer tothe OVERLAPPEDstructure. Since we want to pre-serve the more information such asACTandHandleralong withOVERLAPPEDstructure, we pass the pointerto Asynchronous Accept Result object to theAcceptEx call.

2.2.7 Asynchronous Transmit File

After the open method of the AsynchronousOperation Base is called, applications should callthe transmitfile API of this class to issue an asynchronoustransmit file operation. Thestream handle is given viathe open call.transmit file API takes the followingparameters

� Handler: This class defines call back hook methodhandle asynch transmit file which gets calledwhen the transmission completes.

� header and trailer: Header and the trailer datafor the transmission.

� The offset of the file from where the data has to beread.

� bytes per send: Size of the block that is sent onthe socket.

� ACT: Magic cookie for this asynchronous transmission.

The following steps take place whentransmit is called.

� The transmit file creates anAsynchronousTransmit File Result object with all the infor-mation that are needed to carry out the asynchronous op-eration such asI/O handle and bytes to read

6

and also the information needed to call back the appli-cation which areHandler andACT.

� transmit file calls the TransmitFile WIN32API to initiate the asynchronous transmission.TransmitFile takes pointer to theOVERLAPPEDstructure. Since we want to preserve additional in-formation such asACT and Handler along withOVERLAPPEDstructure, we pass the pointer toAsynchronous Transmit File Result objectto theTransmitFile call.

2.3 Asynchronous Result

The Asynchronous Result classes derive from theOVERLAPPEDstructure and make more useful classes.For each Asynchronous Operation class, there isan Asynchronous Result class which carries aroundthe additional information besides the information in theOVERLAPPEDstructure. This additional information isneeded to execute that operation. The result classes also con-tain fields to hold the results of the asynchronous operation.The Asynchronous Result objects are finally passedto the Completion Handler s when the completion isdispatched.

Refer to the UML diagram 6 for the family ofAsynchronous Result classes.

AsynchronousAsynchronousResult InterfacesResult Interfaces


AsynchronousAsynchronousRead Stream ResultRead Stream Result

AsynchronousAsynchronousAccept ResultAccept Result

AsynchronousAsynchronousTransmit File ResultTransmit File Result

AsynchronousAsynchronousWrite Stream ResultWrite Stream Result

AsynchronousAsynchronousRead File ResultRead File Result

AsynchronousAsynchronousWrite File ResultWrite File Result

OVERLAPPED

Figure 6: Asynchronous Result Classes

The Asynchronous Result base class derives fromthe OVERLAPPEDand also abstracts out all the commonali-ties found in the individual result classes. The individual re-sult classes derive from theAsynchronous Result classand add more information pertaining to those correspondingasynchronous operations. The Asynchronous Result classescontain the following items.

� Handler: This is application handler that handles com-pletions.

� ACT: The magic cookie given by the application whenthe asynchronous operation is issued.

� complete method: This method is defined in thebase classAsynchronous Result as a pure vir-tual method. The individual Result classes override thismethod and call the correct call back methods for thatoperation.

For example, theAsynch Read Stream Resultclass callshandle read stream call back methodfrom thecomplete method. This is very useful becausethe Completion Dispatcher when it gets back aAsynchronous Result object from the OS, can callthecomplete method on it without knowing the exacttype of the asynchronous operation that has completed.

2.4 Completion Handler

The Handler class defines the call back methods which arecalled by the Proactor framework on completions of asyn-chronous events.

For each Asynchronous Operation , a call backmethod is defined and default implementations are provided.Therefore, currently, theHandler class provides the follow-ing call back methods.

� handle read stream

� handle write stream

� handle read file

� handle write file

� handle accept

� handle transmit file

Applications define their own handlers deriving from theHandler class and fill in the application logic appropriatelyin the call back methods.

2.5 Completion Dispatcher

The Proactor class implements the CompletionDispatcher role of the Proactor pattern. Henceforth,we will be using the terms Proactor and Completion Dis-patcher interchangeably.

TheProactor executes the following steps.

� The event loop on the Proactor executesGetQueuedCompletionStatus to get the comple-tions of asynchronous I/O. When there is a completionof an asynchronous event, it gets back anOVERLAPPEDpointer from the OS. This pointer is down cast to theAsynchronous Result type object.

7

This Asynchronous Result object might comefrom a completion event or it might have been postedthroughPostQueuedCompletionStatus .

� The completion status of the asynchronous oper-ation, which includes the number of bytestransferred, error code are obtained from theOS and filled in theAsynchronous Result object.

� The complete method is invoked on theAsynchronous Result object, which calls back theapplication handler.

� When the call back completes, theAsynchronousResult object is deleted by theProactor .

3 Design of the Portable, Extensibleand Efficient Proactor Framework

In this section, we will explain how we designed the WIN32specific Proactor framework to be a poratable, extensible andefficient one. We will first explain the features that thePOSIX4 operating systems provide to do asynchronous I/O.We will then explain how we ported each of the participantof the Proactor pattern to the POSIX platforms. We present astraight forward solution to show how the POSIX implementa-tion can be integrated to the existing WIN32 implementation.Then, we show how we architected the design to make it moreextensible, scalable and efficient.

3.1 Goals

The following are the goals that we have kept in our mind toguide our design and implementation decisions.

� Backward Compatibility: We should keep theexisting APIs of the framework intact and extend all thefunctionalities to work on POSIX platforms.

� Separation of Interface andImplementation: Irrespective how many dif-ferent implementations are provided for the framework,the APIs of the framework should be simple and commonacross all the platforms or implementations. As far aspossible, applications should be freed from worryingabout which implementation of the framework they areusing etc. But applications should have control overconfiguring which implementation should be used by theframe work.

� Scalability: The design of the frame work is insuch a way that it is easier to extend the framework, forexample to have more asynchronous operations or to port

the framework to moreAsynchronous Processorimplementations or platforms.

� Flexibility: The framework APIs should be flex-ible enough so that applications can easily exploit fea-tures which may be special to some particular platforms.For example, assigningpriority to an asynchronous callis possible on POSIX systems, which is not present onWIN32. The framework should be flexible enough so thatapplications can make use of such features portably.

3.2 POSIX Asynchronous Operation Processor

The POSIX I/O subsystem provides the following APIs to ex-ecute asynchronous operations.

� aiocb: This is a structure defined by the POSIX operatingsystem. This is used to pass the parameters to the vari-ousaio calls, to issue the asynchronous calls and also toquery for the completions of the asynchronous calls. Thestructure has the following parameters.

– int file: File descriptor or stream descriptoron which an asynchronous operation is done.

– void buf: Location of the buffer that containsthe data for the I/O.

– size t nbytes: Length of the data transfer.– offset: Offset for the file from where the file

operation is done.– int priority: Priority of the asynchronous

operation.– struct signal event: Signaling option,

signal number and signal information for the asyn-chronous call.

– struct results: Error code and return valueof the asynchronous operation.

� aio read: This call issues an asynchronous read on astream or a file handle. Theaiocb structure is used tospecify the various parameters to do the read operation.

� aio write: This system call issues asynchronous write ona stream or a file handle.

� aio cancel: This system call is used to cancel all or aparticular asynchronous operation issued on a handle.

� aio suspend:This system call is used to wait on an ar-ray of aiocb s that were used to issueaio read s oraio write s, for their corresponding asynchronous op-erations to complete.

� aio sigtimedwaitor sigwaitinfo: This call is used to waitfor a set of real-time signals. It also delivers the signalinformation used when the signal was raised.

8

� aio error: This system call retrieves the error status foran synchronous operation.

� aio return: This retrieves the return status for an asyn-chronous operation.

3.3 Signal and Asynchronous I/O Blocks BasedCompletion Queue Strategies

On POSIX systems, the asynchronous I/O completions canbe obtained from the Operating system in two different wayseither by using the real-time signals or through the Asyn-chronous I/O Control Blocks (aiocb s) that are used to issuethe asynchronous calls.

We have made use of both of these mechanisms andprovided two different strategies for theCompletionDispatcher i.e. the Proactor class and theAsynchronous Operation classes.

We call the aiocb based completion notifica-tion/dispatching mechanism asAIOCB strategy andthe real-time signals based mechanism asSIG strategy .They are implemented as follows.

� AIOCB Strategy: The asynchronous I/O controlblocks (aiocb s), that are used to issue asynchronousI/O calls (aio read or aio write ) are stored with theProactor class.

When theaio read / aio write are issued, thesignaling option is disabled in theaiocb so that the op-erating system will not raise and queue up the real-timesignal when that operation completes.

The array of aiocb s that were stored with theProactor class are then queried for one or more com-pletions using theaio suspend system call.

This approach needs theCompletion Dispatcher ,i.e., theProactor class, to keep track of all theaiocb swhich are used to issue theaio calls. This implementa-tion adds more complexity on part of theCompletionDispatcher , since they need to maintain the list ofaiocb s and query for completions on them. Also, thisapproach may not scale well, since the number of pend-ing asynchronous operations are limited by the size of theaiocb arraywhich should be decided at compile time.

� SIG Strategy: This completion notifica-tion/dispatching strategy is based on the real-timesignaling feature of the POSIX4 operating systems.

A real-time signal numberand signal information isspecified in theaiocb structure, when the asynchronousI/O call (aio read or aio write ) is invoked. Oncompletion of the operation, the operating system raisesthe real-time signal along with thesignal information

that was specified when theaio call was issued. Ifthat real-time signal is masked, that signal is queued upwhich can then be received throughsigtimedwait orsigwaitinfo system calls.

In our framework implementation, we want the com-pletions to be dispatched only when theevent loopofthe Proactor is invoked. We do not want the con-trol to move around arbitrarily between thesignalhandler and the rest of the code. Therefore, we maskthe real-time signals used for issuing theaio calls andusesigtimedwait andsigwaitinfo in theeventloop of the Proactor class, to receive the real-time sig-nals that are queued by the Operating System. In thisapproach, there is no overhead of maintaining the list ofaiocb s with theProactor .

In the following sections, we will explain how we havemade use of the POSIX4 features to port the framework toPOSIX platforms. We will explain how the various compo-nents in the Asynchronous Operation and Asynchronous Re-sult classes were ported to POSIX platforms.

3.4 Asynchronous Operation Classes

The functionalities of the various Asynchronous Operationclasses shown in Figure 5, are implemented for the POSIXplatforms as follows.

3.4.1 Asynchronous Operation Base

The APIs of this class have been ported as follows.

� open: POSIX subsystem does not have the concept ofthe completion port as in WIN32. Therefore, theopenmethod initializes the data members forI/O handle ,Handler andProactor so that they can be used whileissuing asynchronous calls.

� cancel: aio cancel system call is used to imple-ment this API.

3.4.2 Asynchronous Read Stream

The read API implementation looks similar to the WIN32implementation, except theshared read method, calls theaio read to initiate an asynchronous read.aio read op-eration takes a pointer to the structureaiocb . Since we wantto preserve information such asACT and Handler alongwith aiocb structure for each asynchronous call, we pass thepointer to theAsynchronous Read Stream Resultobject to theaio read call. This is possible since the resultclasses derive from theaiocb structure (refer to Figure 10).

Theshared read has been implemented for the two dif-ferent completion strategies as follows.

9

� AIOCB strategy: Theaiocb objects used to issue theaio readare stored with theProactorso that the Proactorcan doaio suspendon them, to query for the completion.

� SIG strategy: In this strategy, the parametersignalnumber provided through theread API, is used to issuethe aio read operation. Thus, applications can spec-ify signal numbers on a per operation basis. But the sig-nal numbers used to issue the asynchronous operationsshould have been already specified to the Proactor class,so that it can wait for completions with that signal num-ber. TheAsynchronous Read Stream Resultpointer is passed as the signal information, which is re-ceived again on completion.

3.4.3 Asynchronous Write Stream

The write API implementation works similar to theWIN32 implementation. Theshared write makes thecall to aio write passing theAsynchronous WriteStream Result object pointer and initiates the asyn-chronous call. In the case of signal based completion strategy,theAsynchronous Write Stream Result pointer ispassed as the signal information. In the AIOCB strategy, theaiocb object, that is used to issue the asynchronous opera-tion, is stored with the Proactor so that it can be used to queryfor completions.

3.4.4 Asynchronous Read File

This class extends the functionality of theAsynchronousRead Stream class to do asynchronous read on a file.Asynchronous Read File Result class is passed onto theshared read method of theAsynchronous ReadStream class, which invokes theaio read call.

3.4.5 Asynchronous Write File

This class extends the functionality of theAsynchronousWrite Stream class to do asynchronous write on a file.Asynchronous Write File Result class is passedon to theshared write method of theAsynchronousWrite Stream class, which invokes theaio write call.

3.4.6 Asynchronous Accept

Unlike the WIN32 subsystem, which hasAcceptExsystemcall to queue anacceptoperation with the Operating System,POSIX platforms do not provide a system call to doacceptasynchronously on asocket.

When theaccept API of this class is called it should becarried out asynchronously. We looked at the following ap-proaches for implementing this class.

Thread per Accept: For each asynchronousaccept theapplication makes, anAsynchronous Accept Resultobject is created with all the information needed for thatinvocation. Then, a separate thread is spawned and theAsynchronous Accept Result object is passed to thatthread.

The thread will block on theaccept system call. Itnotifiesthe Proactor , when it comes out of the system call. ThepostcompletionAPI of the Proactor is used to notify thecompletions to theProactor .

Refer to the Figure 7 for how this is implemented. But this

CLIENTCLIENT

CLIENTCLIENT CLIENTCLIENT

Accept AcceptHandlerHandler

AsynchAsynchAcceptAccept



2:2: SPAWN A THREAD SPAWN A THREAD

PER ACCEPTPER ACCEPT

ProactorProactor

4:4: POST POST

COMPLETIONCOMPLETION

AcceptAcceptCompletionCompletion

HandlerHandler

3:3: CCONNECTONNECT

5:5: DISPATCH DISPATCH

COMPLETIONCOMPLETION

SERVERSERVER

1:1: ASYCH ASYCH__ACCEPT ACCEPT ((ARGSARGS))

Figure 7: Asynchronous Accept by Thread Per Accept

approach does not scale well since it requires one thread foreach pending accept call.

Reactor with an auxillary thread: Having multiple threadsis overkill for doing asynchronous accept. But we have to ex-ecute theaccept s without borrowing the thread that issuedthe asynchronousaccept call.

In this implementation, We have an auxillary thread runningthe Reactor[6] event loop. We also maintain the Queue ofAsynchronous Accept Result objects to keep trackof theasynchronous accept s issued by the application.Refer to the Figure 8 which explains this implementation.

This model consists of the following components.

� Auxillary Thread: When theopen method on theAsynchronous Accept operation class is called, itspawns the thread which always runs the Reactor’s eventloop. But initially when there is noaccept call issuedby the application, thehandle on which accept isdone is disabled in the Reactor, so that it does not accept

10

CLIENT

CLIENT CLIENT

AcceptHandler

AsynchAccept

2: SPAWN A THREAD

PER ACCEPT

Proactor7: POST

COMPLETION

AcceptCompletion

Handler

5: CONNECT

8: DISPATCH

COMPLETION

SERVER

3: ASYCH ACCEPT (ARGS)1: OPEN (ARGS)

CLIENT

Asynch Accept ResultAsynch Accept ResultQueueQueue

4:4: ENQUEUE ENQUEUE

RESULT OBJECTRESULT OBJECT

6:6: DEQUEUE DEQUEUE

RESULT OBJECTRESULT OBJECT

Figure 8: Asynchronous Accept using Reactor with an Auxil-lary Thread

any connections when there are noasynchronousaccept calls issued by the application.

� Event Handler for the Reactor: TheAsynchronous Accept Handler is the helperclass for implementing theAsynchronous Acceptoperation. It also acts as the Event Handler for theReactor running in the auxillary thread. This classmanages the Queue ofAsynchronous AcceptResult objects.

When an asynchronous accept is issued by the ap-plication, anAsynchronous Accept Result ob-ject is created for that invocation and enqueued in theAsynchronous Accept Result Queue . Thenthe handle on the Reactor is enabled so that connec-tions can be accepted.

When there is a connection at the handle, theReactor which runs in the auxillary thread callsthehandle input of theAsynchronous AcceptHandler class. Thehandle input method does anon-blockingaccept system call on the handle andcompletes the accept call.

� Notifying completions to theProactor: Once theaccept system gets com-pleted in the auxillary thread in thehandle inputmethod, the completion should be notified to the

Proactor .

An Asynchronous Accept Result object de-queued from the Queue and the result of theacceptis filled in the object. The order in which theAsynchronous Accept Result objects are de-queued, can be decided based on thepriority of theasynchronous operation.

TheAsynchronous Accept Result object is thenpostedto theProactor using thepost completionAPI in theProactor class. The implementation of thisAPI is discussed in section 3.8.

3.4.7 Asynchronous Transmit File

WIN32 Operating System provides an API for trans-mitting a file asynchronously on a socket. It is notso on POSIX platforms.Asynchronous TransmitFile operation class has been implemented us-ing an Asynchronous Read File object whichreads from the file and anAsynchronous WriteStream object which writes on data read from the fileto the socket. The Figure 9 depicts this model.

Asynch Transmit FileAsynch Transmit File

AsynchAsynchRead FileRead File

POSIX I POSIX I//OOSUBSYSTEMSUBSYSTEM

Read FileRead FileCompletionCompletion

HandlerHandler

Write StreamWrite StreamCompletionCompletion

HandlerHandler

AsynchAsynchWrite StreamWrite Stream

ProactorProactor

1:1: TRANSMIT FILE TRANSMIT FILE

2:2: READ READ

3:3: AIO READ AIO READ

4:4:READREAD

COMPLETECOMPLETE

6:6: WRITE WRITE

7:7: AIO WRITE AIO WRITE

8:8:WRITEWRITE

COMPLETECOMPLETE

10:10: READ READ5:5: DISPATCH DISPATCH

READREAD

9:9: DISPATCH DISPATCH

WRITEWRITE

DISPATCHDISPATCH

TRANSMITTRANSMIT

Transmit FileTransmit FileCompletionCompletion

HandlerHandler

Figure 9: Asynchronous Transmit File

The Asynchronous Transmit File class has ahelper class calledAsynchronous Transmit File Han-dler which the does the transmission on behalf of theAsynchronous Transmit File operation class.It contains anAsynchronous Read File object

11

and anAsynchronous Write Stream object. Thehelper class acts as theCompletion Handler forboth the operations.

When transmit file API is called, anasynchronous read operation is issued on thefile and the control returns to the caller. After theasynchronous read completes and when the callerexecutes the event loop of the Proactor, the call backmethodhandle read file of the helper class getscalled. In this call back method, anasynchronouswrite is initiated to write all the data read from the fileon to the socket. When theasynchronous writecompletes, handle write stream method getscalled on the helper class. It is possible to have partialwrites on the socket. The state of the writes are takencare of by the helper class. Finally when the write fullycompletes, the helper class initiates an asynchronousread to get the next block of data from the file. Thehelper class keeps the state of the transmission such ascurrent offset of the file, ACT etc.

Special ACT strings are used to send theheader andtrailer before and after the file transmission respec-tively. The ACTs are very useful for the call back meth-ods to differentiate between the completions of header ortrailer transmission and the file data transmission. Thecall back methodhandle write stream , when theheader transmission completes, initiates the file read tostart transmitting the file. It initiates transmitting thetrailer, when the file transmission completes. And whenthe trailer transmission also completes, it calls the dis-patches completion of thetransmit file operation.

3.5 Asynchronous Result Classes

We will explain here how the Asynchronous Resultclasses in the original WIN32 solution were ported toPOSIX platforms.

Since theaio system calls takes a pointer to theaiocbstructure, we derive theAsynchronous Resultclasses from theaiocbstructure. The rest of the inheri-tance hierarchy is the same as in WIN32 implementation.Refer to the UML diagram 10 for the Asynchronous Re-sult classes.

3.6 Completion Handler

This component of the framework does not have any plat-form specific implementations. Hence this is kept com-mon for all the implementations of the framework.









aiocbaiocb

Figure 10: Porting Asynchronous Result Classes to POSIX

3.7 Completion Dispatcher

The Proactor class implements both the completionstrategies discussed in 3.3.

To wait for the completions and the to dispatch them, theevent loop executes the following steps.

– AIOCB Strategy: In this implementation, theaio suspend call is used to wait on the arrayof aiocb objects that are used to issue the asyn-chronous operations.aio suspend call returnswhen there is atleast one completion.aio error is used to find out the correctaiocbobject in the array for which the completion hadoccured. This call also gets the error status of theasynchronous operation.we cast theaiocb pointer to the the derived classAsynchronous Result object, since the dis-patching functionalities are available only in theAsynchronous Result class.

– SIG Strategy: In this strategy,sigwaitinfo is called with the signal setthat has been already masked for all the threads.The sigtimedwait call is used, instead ofsigwaitinfo when the timed event loop iscalled. These wait calls complete on arrival of asignal that is present in the signal set.The Asynchronous Result object associatedwith the signal delivery is retrieved from the signalinformation obtained in the call.aio error is called on this object to get the errorstatus of the asynchronous operation.

aio return is called on thataiocb object to retrivethe return status of the asynchronous operation.

12

To dispatch the completion, Thecomplete methodis invoked on theAsynchronous Result object,which calls the correct call back method in the comple-tion handler.

When the call back completes, theAsynchronousResult object is deleted by the Proactor.

3.8 Posting Completions to CompletionQueue

WIN32 provides the APIPostQueuedCompletionStatus to post a com-pletion (a pointer to the OVERLAPPED structure) to aCompletion Port .

The API postcompletionof the Proactor class takesa pointer to theAsynchronous Result object. OnPOSIX, we have implementedpost completion forthe two different completion strategies as follows.

SIG Strategy: The system callsigqueue is used toqueue up a reserved real-time signal to the current pro-cess. TheAsynchronous Result object is assignedas the signal information in thesigqueue call.

The Proactor’s event loop which waits for the reservedreal-time signal receives the Result block and callscomplete method on it, do dispatch the completion.

AIOCB Strategy: We make use of a notify pipe to sendthe Asynchronous Result objects to the Proac-tor’s completion queue. The Proactor reads the pipeat the other end for theAsynchronous Resultobjects. Reading for theAsynchronous Resultbase class pointer helps because a pointer to anytype that derives from theAsynchronous Resultcan be posted through the notify pipe. For ex-ample, Asynchronous Accept Result object isposted in theAsynchronous Accept operation im-plementation. Also, applications can derive their ownAsynchronous Result classes and use them forposting to the Proactor to fake completions.

Reading from the pipe has to be synchronized with theevent loop of the Proactor, so that the completions are dis-patched only when the even loop is running. To achievethis, the Proactor issues anasynchronous readon the notify pipe using theAsynchronous ReadStream operation class object. A helper class calledNotify Pipe Manager manages the pipe and actsas theCompletion Handler for the asynchronousread operation issued on the pipe.

TheAsynchronous Result pointers are read asyn-chronously from the notify pipe and dispatched duringProactor’s event loop.

When a read from the pipe completes,Notify PipeManager ’s handle read stream gets called. Thismethod receives theAsynchronous Result pointerand callscomplete on it, which dispatches the comple-tion.

After the dispatch, a newasynchronous read is is-sued on the pipe to handle the completions in the future.

The Figure 11 explains this implementation. The ab-stract classHandler provides the default implementa-tions for the completion call back hook methods such ashandle read stream etc.

ProactorProactor

Notify PipeNotify PipeManagerManager

HandlerHandler

PipePipeAsynch ReadAsynch ReadStreamStream

Figure 11: Posting Completions to the Proactor

3.9 A Simple Integrated Proactor Framework

We now integrate the POSIX implementations discussedabove with the existing WIN32 implementation. A straightforward integration makes use of the following tricks.

Conditional compilation: We use#if defined state-ments all over the code to switch between the WIN32 imple-mentation and the POSIX implementations. Class definitions,where we change the inheritance hierarchy depending on theplatform, are guarded by pre-compiler conditional directives.For example, the Asynchronous Result classes derive fromOVERLAPPEDstructure on WIN32, but nthey derive from theaiocb structure on POSIX platforms.

The definition of each function which has platform specificimplementations also has a similar kind of condition compilerdirectives to switch between the implementations based on theplatforms.

Switch statements: With in the POSIX implementation, weneed to switch between, the two different completion strate-gies that we are using. We need to use the run time switches

13

based on a variable to switch between the two different imple-mentation code. If not run time switches, we need to use theconditional compiler directives for this too.

3.9.1 Drawbacks with the Simple Integrated ProactorFramework

We have achieved the portability goal. We did not make anychange to the existing APIs in the framework. We have alsokept the APIs simple. But the integrated framework we havediscussed so far, is not flexible, extensible and scalable, be-cause of the following reasons.

� Unstructured code: Since we have merged the POSIXspecific implementation code on the WIN32 specific im-plementation, the source code is full of #ifdef precom-piler directives which are there to make sure the correctcode is compiled on a platform. This really makes theimplementation unreadable unstructured.

� Scalability still remains an issue. For example porting theimplementation to a new platform will involve definingnew #ifdef pre-compiler directive. The implementationcode becomes more and more ugly and complex. Evenwithin the POSIX implementation, we have two differentimplementations and some platforms may need a differ-ent implementation.

3.10 Extensible Proactor Framework Design

In this section, we will explain how we enhanced our portabledesign to be highly extensible and easy to maintain. This de-sign has totally eliminated the precompiler directives from thesource code. The code for each implementation is totally de-coupled from the other implementations, but all the implemen-tations are bridged by a common simple Interface. The in-terface has not changed much from the original WIN32-onlysolution. Therefore the existing applications do not have tochange drastically. We have applied the following concepts inorder to achieve our goals.

� Applying Bridge Pattern: We have applied theBridge pattern to decouple the concrete implementationsfrom the API. This eliminates all the conditional pre-compiler directives which switched the code between theWIN32 and the POSIX platform implementations. Wehave applied bridging to Asynchronous Operation, Asyn-chronous Result and the Proactor classes. Since the APIshave not changed, the new design is still compatible withthe old applications.

� Inheritance instead of switching: Wehave provided separate inheritance hierarchies for thetwo different POSIX implementations. So all the switch

statements for switching between the implementationshave been removed. The common code among the twoPOSIX implementations have been abstracted out tobase class. Refer to figures 12, 16 and 20 how theAsynchronous Operation, Asynchronous Result andProactor classes have been redesigned.

� Factory Methods: TheProactor class defines thefactory methods [7] to create the correct implementa-tion objects for theAsynchronous Operation andAsynchronous Result classes.

Once the rightProactor implementation is decidedfor the application, the switching between the differentconcrete implementations is decided automatically by theProactor class through the factory methods.

In the next section, we explain how we re-designed each of theparticipant in the Proactor pattern to achieve our goals.

3.11 Design of the Asynchronous OperationClasses

The UML diagram in Figure 12 shows how theAsynchronous Operation classes have been re-designed in order to fulfill our goals.

Asynchronous OperationAsynchronous OperationInterfacesInterfaces

AbsractAbsractAsynchronous OperationAsynchronous Operation

ImplementationImplementation

WIN32WIN32Asynchronous OperationAsynchronous Operation


POSIXPOSIXAsynchronous OperationAsynchronous Operation


Figure 12: Bridged Asynchronous Operation Classes

The Asynchronous Operation Interfacespackage, contains the interface classes. The im-plementation classes are bridged with the interfaceclasses through the Abstract AsynchronousOperation Implementation package. Thepackages WIN32 Asynchronous OperationImplementation and POSIX AsynchronousOperation Implementation provide the imple-

14

mentation classes for the WIN32 and POSIX platformsrespectively.

The UML diagram of theAsynchronous OperationInterfaces package looks exactly similar to the Figure5. But, the interface classes are free from precompilerdirectives now. They forward their methods to the imple-mentation classes which are bridged together byAbstractAsynchronous Operation Implementationclasses.

The UML diagram of the WIN32 AsynchronousOperation Implementation package is shown in theFigure 13.




WIN32WIN32AsynchronousAsynchronousRead StreamRead Stream

WIN32WIN32AsynchronousAsynchronous

AcceptAccept

WIN32WIN32AsynchronousAsynchronousTransmit FileTransmit File

WIN32WIN32AsynchronousAsynchronousWrite StreamWrite Stream


Read FileRead File


Write FileWrite File

Figure 13: WIN32 Asynchronous Operation Implementation

The UML diagram of the POSIX AsynchronousOperation Implementation package is shown in theFigure 14. The implementations of theAsynchronous


ImplementaionImplementaion


POSIX AIOCBPOSIX AIOCBAsynchronous OperationAsynchronous Operation


POSIX SIGPOSIX SIGAsynchronous OperationAsynchronous Operation


Figure 14: POSIX Asynchronous Operation Implementation

Operation classes for the two different completion strate-gies 3.3 are separated in to two packages as shown in theFigure 14.

The package POSIX AIOCB AsynchronousOperation Implementation defines the Asyn-chronous Operation classes for the AIOCB completionstrategy. This package is shown in Figure 15.


POSIX AIOCBPOSIX AIOCBAsynchronousAsynchronous Read Stream Read Stream



POSIX AIOCBPOSIX AIOCBAsynchronousAsynchronous

Accept Accept

POSIX AIOCBPOSIX AIOCBAsynchronousAsynchronous Transmit File Transmit File

POSIX AIOCBPOSIX AIOCBAsynchronousAsynchronous Write Stream Write Stream


Read File Read File


Write File Write File

Figure 15: POSIX AIOCB Asynchronous Operation Imple-mentation

The package POSIX SIG AsynchronousOperation Implementation defines the Asyn-chronous Operation classes for the SIG completion strategy.The UML inheritance hierarchy for this strategy looks similarto the�POSIX AIOCB Asynchronous Implementation packageshown in 15.

3.12 Design of the Asynchronous ResultClasses

The architecture of the new Asynchronous Resultclasses has been shown in the Figure 16. The


AbstractAbstractAsynchronousAsynchronous

ResultResultImplementationImplementation

WIN32 AsynchronousWIN32 AsynchronousResult ImplementationResult Implementation

POSIX AsynchronousPOSIX AsynchronousResult ImplementationResult Implementation

Figure 16: Bridged Asynchronous Result Classes

Asynchronous Result Interfaces package,defines the interfaces to the various AsynchronousResult class implementations. The implementationsare bridged with the interfaces through the abstractclasses in the Abstract Asynchronous ResultImplementation package. The packagesWIN32

15

Asynchronous Result Implementation andPOSIX Asynchronous Result Implementationimplement theAsynchronous Result Interfacesfor the WIN32 and the POSIX4 platforms respectively.

The UML architecture of theAsynchronous ResultInterfaces has been shown in Figure 17. Note that theAsynchronous Result base class does not have to de-rive from platform specificOVERLAPPEDor aiocb struc-ture. Therefore, there is no need for precompiler directives toswitch between the implementation. All the interfaces havethe reference to the implementation objects, where they for-ward all their methods.


AsynchronousAsynchronousResult BaseResult Base







Figure 17: Asynchronous Result Interfaces

The UML diagram of the WIN32 Asynchronous Resultclasses are shown in Figure 18.

WIN32 AsynchWIN32 AsynchResultResult

WIN32 AsynchWIN32 AsynchRead Stream ResultRead Stream Result

WIN32 AsynchWIN32 AsynchAccept ResultAccept Result

WIN32 synchWIN32 synchTransmit File ResultTransmit File Result

WIN32 AsynchWIN32 AsynchWrite Stream ResultWrite Stream Result

WIN32 AsynchWIN32 AsynchRead File ResultRead File Result

WIN32 AsynchWIN32 AsynchWrite File ResultWrite File Result

OVERLAPPEDOVERLAPPED

WIN32 AsnchronousWIN32 AsnchronousResult ImplementationResult Implementation

Figure 18: WIN32 Asynchronous Result Classes

The UML diagram of the POSIX Asynchronous Resultclasses are shown in Figure 19.

Note that the WIN32 and the POSIX implementationclasses do not have to have any precompiler directives now.

3.13 Design of the Proactor class

The UML diagram in Figure 20 shows how the Proactor classhas been redesigned.

The Proactor Interface class acts as the inter-face to the different Proactor class implementations. The

POSIXPOSIXAsynch ResultAsynch Result

POSIX AsynchPOSIX AsynchRead Stream ResultRead Stream Result

POSIX AsynchPOSIX AsynchAccept ResultAccept Result

POSIX AsynchPOSIX AsynchTransmit File ResultTransmit File Result

POSIX AsynchPOSIX AsynchWrite Stream ResultWrite Stream Result

POSIX AsynchPOSIX AsynchRead File ResultRead File Result

POSIX AsynchPOSIX AsynchWrite File ResultWrite File Result

aiocbaiocb

POSIX AsnchronousPOSIX AsnchronousResult ImplementationResult Implementation

Figure 19: POSIX Asynchronous Result Classes

ProactorProactorInterfaceInterface

Proactor AbstractProactor AbstractImplementationImplementation

POSIXPOSIXProactorProactor

POSIX AIOCBPOSIX AIOCBProactorProactor

POSIX SIGPOSIX SIGProactorProactor

WIN32WIN32ProactorProactor

Figure 20: Bridged Proactor Classes

16

Proactor Abstract Implementation bridges theimplementations with the Proactor interface. TheProactorInterface class simply forwards all the methods to the im-plementation classes.

The two different completion notification/dispatchingmechanisms discussed in 3.3, are implemented by thePOSIXAIOCB Proactor class and thePOSIX SIG Proactorclass. ThePOSIX AIOCB Proactor implements theAIOCB strategy and the ThePOSIX SIG Proactor classimplements the SIG Strategy. The common code betweenthese two implementations have been abstracted out in thePOSIX Proactor class.

3.14 Design Analysis

Let us now analyze this design from the view of the goals wehad initially.

� Backward Compatibility: The main interfacesof the framework have not been changed, except that thefollowing minor changes have been done to the APIs ofthe framework to fit with the new design.

– Proactor Constructor: The constructor ofthe Proctor class in the interface level, takes Ab-stract Implementation objects, instead of platformspecific constructor parmeters. When no implemen-tation is given, it creates implementations based onthe predefined constants, with default options. Tooverride this, applications can create the implemen-tation and then create the interface Proactor objectwith that implementation.

– post completion: This API has been takenoff from the main Proactor interface and havebeen moved to theProactor AbstractImplementation class. This is necessarysince theAbstract Asynchronous Resultclasses which are posted as completions havedifferent base classes, in POSIX and WIN32platforms.

– complete: This method is defined in theAsynchronous Result Implementationclasses. This method is used by theProactorImplementation classes to dispatch thecompletions to the correct call back methods.

Applications sometime exploit this feature tofake completions to their handlers. Now, sincethe Asynchronous Result Interfaceclasses are different from theAsynchronousResult Implementation classes, applicationshould make use of thefactory methods

defined in the Proactor class, in order toobtain the correct Asynchronous ResultImplementation s and callcomplete methodon them.

� Separation of Interface andImplementation: We have separated the inter-faces from the implementations. We have also providedseparation between the various implementations.

� Scalability: The framework now has a very cleanarchitecture to scale.

For example, to port the framework to a newAsynchronous Operation Processor im-plementation or to a new Operating System involvesreusing the existing bridging hierarchy and define a newimplementation hierarchy for the new implementation.Factory methods and call back methods have to bedefined appropriately for the new implementation/

In the simple design discussed in 3.9, this would haveinvolved switch statements and #ifdef precompiler direc-tives all over the code to incorporate the new implemen-tation.

� Flexibility: We have made use of platform spe-cific features in the framework so that applications canuse them to their advantage on those platforms with outloosing portability. Default values have been provided tosuch features.

For example, specifyingpriority value for the asyn-chronous operation is provided to the APIs of theAsynchronous Operation so that applications onPOSIX systems can use them successfully. But it is pro-vided with a default value so that applications on WIN32platforms need not be concerned with it.

4 Conclusions

The WIN32 platform specific Proactor framework was ex-tended to work on the POSIX implementations of the Asyn-chronous Operation Processor. We have made our design insuch way that the framework is not only portable but also ex-tensible to include more features, scalable to more implemen-tations of Asynchronous Operations Processor and highly ef-ficient.

References[1] I. Pyarali, T. Harrison, D. C. Schmidt, and T. D. Jordan, “Proac-

tor – An Architectural Pattern for Demultiplexing and Dispatch-ing Handlers for Asynchronous Events,” inThe 4

th Pattern

17

Languages of Programming Conference (Washington Universitytechnical report #WUCS-97-34), September 1997.

[2] D. C. Schmidt, “ACE: an Object-Oriented Framework forDeveloping Distributed Applications,” inProceedings of the6th USENIX C++ Technical Conference, (Cambridge, Mas-

sachusetts), USENIX Association, April 1994.

[3] J. C. Mogul, “The Case for Persistent-connection HTTP,” inPro-ceedings of ACM SIGCOMM ’95 Conference in Computer Com-munication Review, (Boston, MA), pp. 299–314, ACM Press,August 1995.

[4] I. Pyarali, T. H. Harrison, and D. C. Schmidt, “AsynchronousCompletion Token: an Object Behavioral Pattern for EfficientAsynchronous Event Handling,” inPattern Languages of Pro-gram Design(R. Martin, F. Buschmann, and D. Riehle, eds.),Reading, MA: Addison-Wesley, 1997.

[5] D. C. Schmidt, “Reactor: An Object Behavioral Pattern for Con-current Event Demultiplexing and Event Handler Dispatching,”in Pattern Languages of Program Design(J. O. Coplien and D. C.Schmidt, eds.), pp. 529–545, Reading, MA: Addison-Wesley,1995.

[6] D. C. Schmidt, “Reactor: An Object Behavioral Pattern for Con-current Event Demultiplexing and Dispatching,” inProceedingsof the1st Annual Conference on the Pattern Languages of Pro-grams, (Monticello, Illinois), pp. 1–10, August 1994.

[7] E. Gamma, R. Helm, R. Johnson, and J. Vlissides,Design Pat-terns: Elements of Reusable Object-Oriented Software. Reading,MA: Addison-Wesley, 1995.

18

a portable, extensible and efï¬cient implementation of

Documents