The ADAPTIVE Communication EnvironmentAn Object-Oriented Network Programming Toolkit

for Developing Communication Software

Douglas C. Schmidtschmidt@cs.wustl.edu

http://www.cs.wustl.edu/�schmidt/Department of Computer Science

Washington UniversitySt. Louis, MO 63130, (314) 935-7538

Earlier versions of this paper appeared in the11th and

12th Sun User Group conferences in San Jose, California,

Dec. 7–9, 1993 and San Francisco, California, June 14–17,1993.

Abstract

The ADAPTIVE Communication Environment (ACE) is anobject-oriented (OO) toolkit that implements fundamentaldesign patterns for communication software. ACE is tar-geted for developers of high-performance communicationservices and applications on UNIX and Win32 platforms.ACE simplifies the development of OO network applicationsand services that utilize interprocess communication, eventdemultiplexing, explicit dynamic linking, and concurrency.ACE automates system configuration and reconfiguration bydynamically linking services into applications at run-timeand executing these services in one or more processes orthreads.

This paper describes the structure and functionality ofACE and illustrates core ACE features using examples fromdomains like telecommunications, enterprise medical imag-ing, and WWW services. ACE is freely available and is beingused for many commercial projects (such as Ericsson, Bell-core, Siemens, Motorola, Kodak, and McDonnell Douglas),as well as many academic and industrial research projects.ACE has been ported to a variety of OS platforms includingWin32 and most UNIX/POSIX implementations. In addition,both C++ and Java versions of ACE are available.

1 Introduction

1.1 Problem: the Distributed Software Crisis

The demand for robust and high-performance distributedcomputing systems is steadily increasing. Examples of thesetypes of systems include global personal communicationsystems, network management platforms, enterprise medicalimaging systems, online financial analysis systems, and real-time avionics systems. Distributed computing is a promisingtechnology for improving collaboration through connectiv-ity and interworking; performance through parallel process-ing; reliability and availability through replication; scalabil-

ity and portability through modularity; extensibility throughdynamic configuration and reconfiguration; and cost effec-tiveness through resource sharing and open systems.

Although distributed computing offers many potentiallybenefits, developing communication software is expensiveand error-prone. Object-oriented (OO) programming lan-guages, components, and frameworks are widely touted tech-nologies for reducing software cost and improving softwarequality. When stripped of the hype, the primary benefits ofOO stem from the emphasis on modularity and extensibil-ity, which encapsulate volatile implementation details behindstable interfaces and enhance software reuse.

Developers in certain well-traveled domains have success-fully applied OO techniques and tools for years. For in-stance, the Microsoft MFC GUI framework and OCX com-ponents arede factoindustry standards for creating graphicalbusiness applications on PC platforms. Although these toolshave their limitations, they demonstrate the productivity ben-efits of reusing common frameworks and components.

Software developers in more complex domains liketelecommunications, medical imaging, avionics, and onlinetransaction processing have traditionally lacked standard off-the-shelf middleware components. As a result, developerslargely build, validate, and maintain software systems fromscratch. In an era of deregulation and stiff global compe-tition, this in-house development process is becoming pro-hibitively costly and time consuming. Across the industry,this situation has produced a “distributed software crisis,”where computing hardware and networks get smaller, faster,and cheaper; yet distributed software gets larger, slower, andmore expensive to develop and maintain.

The challenges of building distributed software stem frominherentandaccidentalcomplexities [1] associated with dis-tributed systems. Inherent complexities stem from funda-mental challenges of developing distributed software. Chiefamong these is detecting and recovering from networkand host failures, minimizing the impact of communica-tion latency, and determining an optimal partitioning of ser-vice components and workload onto processing elementsthroughout a network.

Accidental complexities stem from limitations with toolsand techniques used to develop telecom software. For in-stance, many standard networking mechanisms (such as

1

THREADTHREAD

LIBRARYLIBRARY

DYNAMICDYNAMIC

LINKINGLINKING

MEMORYMEMORY

MAPPINGMAPPING

SELECTSELECT//POLLPOLL

SYSTEMSYSTEM

VV IPCIPCSTREAMSTREAM

PIPESPIPES

NAMEDNAMED

PIPESPIPES

CAPIS

SOCKETSSOCKETS//TLITLI

COMMUNICATIONCOMMUNICATION

SUBSYSTEMSUBSYSTEM

VIRTUAL MEMORYVIRTUAL MEMORY

SUBSYSTEMSUBSYSTEM

GENERAL POSIX AND WIN32 SERVICES

PROCESSPROCESS//THREADTHREAD

SUBSYSTEMSUBSYSTEM

SYNCHSYNCH

WRAPPERSWRAPPERS

FRAMEWORKS

AND CLASS

CATEGORIES

ACCEPTORACCEPTOR CONNECTORCONNECTOR

DISTRIBUTED

SERVICES AND

COMPONENTS

NAMENAME

SERVERSERVER

TOKENTOKEN

SERVERSERVER

LOGGINGLOGGING

SERVERSERVER

GATEWAYGATEWAY

SERVERSERVER

SOCKSOCK__SAPSAP//TLITLI__SAPSAP

FIFOFIFO

SAPSAP

LOGLOG

MSGMSG

SERVICESERVICE

HANDLERHANDLER

TIMETIME

SERVERSERVER

OS ADAPTATION LAYER

C++WRAPPERS

THREADTHREAD

MANAGERMANAGER

SPIPESPIPE

SAPSAP

CORBACORBA

HANDLERHANDLER

SYSVSYSVWRAPPERSWRAPPERS

REACTORREACTOR

SHAREDSHARED

MALLOCMALLOC

ADAPTIVE SERVICE EXECUTIVE ADAPTIVE SERVICE EXECUTIVE (ASX)(ASX)

SERVICESERVICE

CONFIGCONFIG--URATORURATOR

MEMMEM

MAPMAP

Figure 1: Components in the ADAPTIVE Communication Environment

sockets [2] and TLI [3]) and reusable component libraries(such as X windows and Sun RPC) lack type-safe, portable,re-entrant, and extensibleapplication programming inter-faces(APIs). Likewise, common network programming in-terfaces like sockets and TLI use weakly-typed integer han-dles that can lead to subtle run-time errors [4].

Another source of complexity arises from the widespreaduse of algorithmic decomposition [5], which results in non-extensible and non-reusable software systems [6]. Althoughgraphical user-interfaces (GUIs) are commonly built usingobject-oriented (OO) techniques, distributed software is typ-ically developed using algorithmic decomposition. Thisproblem is exacerbated by the fact that examples in pop-ular network programming textbooks [7, 8, 3] are basedon algorithmically-oriented design and implementation tech-niques.

The lack of extensibility and reuse in-the-large is particu-larly problematic for complex distributed software. Extensi-bility is essential to ensure timely modification and enhance-ment of services and features. Reuse is essential to lever-age the domain knowledge of expert developers to avoid re-developing and re-validating common solutions to recurringrequirements and software challenges.

1.2 Solution: Object-oriented Design Patternsand Frameworks

Object-oriented design patterns and frameworks are well-regarded for their ability to help alleviate costly rediscov-ery and reinvention of core distributed software concepts andabstractions. Patterns provide a way to encapsulate designknowledge that offers solutions to standard distributed soft-ware development problems [9]. For instance, patterns areuseful for describing recurringmicro-architectures(such asReactor [10] and Active Object [11]), which are abstractionsof common object-structures that have proven useful to builddistributed communication software. However, abstractionsdocumented as patterns do not directly yield reusable code.Therefore, it is essential to augment the study of patternswith the creation and use offrameworks.

Frameworks provide reusable software components forapplications by integrating sets of abstract classes and defin-ing standard ways that instances of these classes collabo-rate [12]. Frameworks instantiate families of design pat-terns to help developers avoid costly reinvention of com-mon distributed software components. The results are “semi-complete” application skeletons that can be customized byinheriting and instantiating from reuseable building blockscomponents in the frameworks. Since frameworks are tightlyintegrated with key distributed programming tasks (such asservice initialization, error handling, flow control, event de-multiplexing, concurrency control), the scope of reuse can

2

be significantly larger than by using traditional function li-braries, or even conventional OO class libraries.

This paper is organized as follows: Section 2 presentsan overview of the structure and functionality of the ACEtoolkit; Section 3 describes the ACE C++ wrapper compo-nents and higher-level ACE framework components and pat-terns in detail; Section 4 examines the implementation ofseveral networking applications built using ACE; and Sec-tion 5 presents concluding remarks.

2 Overview of the ADAPTIVE Com-munication Environment (ACE)

To illustrate how OO patterns and frameworks are being suc-cessfully applied to distributed software, this paper exam-ines the ADAPTIVE Communication Environment (ACE)[6]. ACE is a freely available OO toolkit containing a rich setof reusable wrappers, class categories, and frameworks thatperform common network programming tasks across a widerange of OS platforms. The tasks provided by ACE include:

� Event demultiplexing and event handler dispatching[13, 14, 10, 15];

� Connection establishment and service initialization[16,17, 18];

� Interprocess communication[19, 4] andshared memorymanagement;

� Dynamic configuration of distributed communicationservices[20, 21];

� Concurrency/parallelism and synchronization[22, 23,11, 24];

� Components for higher-level distributed services(suchas a Name service, Event service, Logging service,Time service, and Token service).

The ACE toolkit is designed using a layered architecture.Figure 1 illustrates the vertical and horizonal relationship be-tween ACE components. The lower layers of ACE areOOwrappers that encapsulate existing OS network program-ming mechanisms. The higher layers of ACE extend thewrappers to provideOO frameworks and componentsthatcover a broader range of application-oriented networkingtasks and services. The remainder of this section presents anoverview of the structure and functionality of the class cat-egories in ACE (shown in Figure 2). Section 3 provides in-depth coverage of the ACE network programming featuresand components.

Throughout the paper, the ACE components are illustratedvia Booch notation [5]. Solid rectangles indicate class cat-egories, which combine a number of related classes into acommon name space. Solid clouds indicate objects; nest-ing indicates composition relationships between objects; andundirected edges indicate some type of link exists betweentwo objects. Dashed clouds indicate classes; directed edges

StreamFramework

ServiceServiceConfiguratorConfigurator

APPLICATIONSAPPLICATIONSAPPLICATIONAPPLICATION--SPECIFICSPECIFIC

ConcurrencyConcurrencyglobalglobal

InterprocessInterprocessCommunicationCommunication

APPLICATION-INDEPENDENT

ServiceServiceInitializationInitialization

ReactorReactor

NetworkNetworkServicesServices

APPLICATIONSAPPLICATIONSAPPLICATIONSAPPLICATIONS

Figure 2: The Class Categories in ACE

indicate inheritance relationships between classes; and anundirected edge with a small circle at one end indicates eithera composition or uses relation between two classes. The “A”inscribed within a triangle identifies a class as anabstractclass[25]. An abstract class cannot be instantiated directly,but must be subclassed. Subclasses of an abstract class mustprovide definitions for all its abstract methods before any ob-jects of the class may be instantiated.1

2.1 The ACE OS Adaptation Layer

The ACE source tree contains over 85,000 lines of C++. Ap-proximately 9,000 lines of code (i.e.,about 10% of the totaltoolkit) are devoted to theOS Adaptation Layer. This layershields the higher layers of ACE from platform-specific de-pendencies associated with the following OS mechanisms:

� Multi-threading and synchronization

� Interprocess communication

� Event demultiplexing

� Explicit dynamic linking

� Memory-mapped files and shared memory

2.2 The ACE OO Wrappers

Above the OS Adaptation Layer are OO wrappers that en-capsulate and enhance the concurrency, interprocess commu-nication (IPC), and virtual memory mechanisms (illustratedat the bottom of Figure 1) available on modern operating sys-tems like Win32 and UNIX. Applications can combine and

1Abstract methods C++ are commonly calledpure virtual functions.

3

compose these components by selectively inheriting, aggre-gating, and/or instantiating the following ACE wrapper classcategories:

� IPC SAP – which encapsulates local and/or remoteIPC Service Access Point (IPC SAPmechanisms suchas sockets, TLI, UNIX FIFOs and STREAM pipes, andWin32 Named Pipes, [19, 4];

� Service Initialization – ACE provides a set of Connec-tor and Acceptor components [18] that decouple the ac-tive and passive initialization roles, respectively, fromthe tasks a communication service performs once ini-tialization is complete;

� Concurrency mechanisms– ACE abstracts lower-level OS multi-threading and multi-processing mecha-nisms (such as mutexes and semaphores [22]) to createhigher-level OO concurrency abstractions (such as Ac-tive Objects [11]);

� Memory management mechanisms– the ACE mem-ory management components provide a flexible and ex-tensible abstraction for managing dynamic allocationand deallocation of shared memory and local memory;

� CORBA integration – ACE can be integrated withCORBA implementations [26] (such as single-threadedand multi-threaded Orbix).

The use of OO wrappers improves application robustnessby encapsulating OS communication, concurrency, and vir-tual memory mechanisms with type-secure OO interfaces.This alleviates the need for applications to directly accessthe underlying OS libraries, which are written using weakly-typed C interfaces. Therefore, compilers for OO languageslike C++ and Java can detect type system violations atcompile-time, rather than at run-time. The C++ versionof ACE uses inlining extensively to eliminate performancepenalties that would otherwise be incurred from the addi-tional type-security and abstraction provided by the wrapperlayer.

2.3 The ACE Framework

ACE contains a higher layer network programming frame-work that integrates and enhances the lower layer OS wrap-pers. This framework supports the dynamic configuration ofconcurrent network daemons composed of application ser-vices. The framework portion of ACE contains the followingclass categories:

� Reactor – The ACE Reactor [10] provides extensible,object-oriented demultiplexer that dispatches handlersin response to various types of events (e.g.,I/O-based,timer-based, signal-based, and synchronization-basedevents);

� Service Configurator – The ACE Service Configura-tor [21] supports the construction of applications whoseservices may be configured dynamically at installation-time and/or run-time;

IPC_SAP

A

SOCK_SAP TLI_SAP FIFO_SAPSPIPE_SAP

SOCKET

API

TRANSPORT

LAYER

INTERFACE API

STREAM PIPE

API

NAMED PIPE

API

Figure 3:IPC SAP Class Category Relationships

� Streams– The ACE Streams components [6] simplifythe development of concurrent communication softwareapplications composed of one or more hierarchically-related services (such as protocol stacks);

2.4 ACE Network Service Components

In addition to the wrappers and frameworks, ACE provides astandard library of network service components. These com-ponents play two roles in ACE:

1. They illustrate how to utilize the ACE IPC wrappers,Reactor, Service Configurator, Service Initialization,Concurrency, Memory Management, and Streams com-ponents;

2. They provide reusable components for common dis-tributed system tasks such as logging [13, 13], naming,locking, and time synchronization [21];

When combined with OO language features (such asclases, inheritance, dynamic binding, and parameterizedtypes) and design patterns (such as Abstract Factory, Builder,and Service Configurator), the reusable ACE components fa-cilitate the development of communication services and ap-plications that may be updated and extended without modify-ing, recompiling, relinking, or even restarting running soft-ware [20].

3 Detailed Coverage of ACE Compo-nents

3.1 IPC SAP: Local and Remote IPC Mecha-nisms

ACE provides a forest of class categories rooted at theIPCSAP(“InterProcess Communication Service Access Point”)base class.IPC SAP encapsulates the standard I/O handle-based OS local and remote IPC mechanisms that offerconnection-oriented and connectionless protocols. As shownin Figure 3, this forest of class categories includesSOCKSAP(which encapsulates the socket API),TLI SAP (which

4

encapsulates the TLI API),SPIPE SAP (which encapsu-lates the UNIX SunOS 5.x STREAM pipe API), andFIFOSAP(which encapsulates the UNIX named pipe API).

Each class category is organized as an inheritance hierar-chy. Every subclass provides a well-defined interface to asubset of local or remote communication mechanisms. To-gether, the subclasses within a hierarchy comprise the over-all functionality of a particular communication abstraction(such as the Internet-domain or UNIX-domain protocol fam-ilies). The use of classes (as opposed to stand-alone func-tions) helps to simplify network programming as follows:

� Shield applications from error-prone details– For ex-ample, theACEAddr class hierarchy shown in Fig-ure 3 supports several diverse network addressing for-mats via a type-secure OO interface, rather than us-ing the awkward and error-prone C-basedstructsockaddr data structures directly.

� Combine several operations to form a single operation– For example, theSOCK Acceptor constructor per-forms the various socket system calls (such assocket ,bind , andlisten ) required to create a passive-modeserver endpoint.

� Parameterize IPC mechanisms into applications–Classes form the basis for parameterizing an applicationby the type of IPC mechanism it requires. This helps toimprove portability as discussed in Section 3.1.2.

� Enhance code sharing– Inheritance-based hierarchicaldecomposition increases the amount of common codethat is shared amongst the various IPC mechanisms(such as the OO interface to the lower-level OS devicecontrol system calls likefcntl andioctl ).

The following sections discuss each of the class categoriesin IPC SAP.

3.1.1 SOCK SAP

The SOCK SAP[4] class category provides applicationswith an object-oriented interface to the Internet-domainand UNIX-domain protocol families [8]. Applicationsmay access the functionality of the underlying Internet-domain or UNIX-domain socket types by inheriting or in-stantiating the appropriateSOCK SAPsubclasses shownin Figure 4. TheACESOCK* subclasses encapsulateInternet-domain functionality and theACELSOCK* sub-classes encapsulate UNIX-domain functionality. As shownin Figure 4, the subclasses may be further decomposedinto (1) the *Dgram components (which provide unre-liable, connectionless, message-oriented functionality) vs.the *ACE Stream components (which provide reliable,connection-oriented, bytestream functionality) and (2) theACE* Acceptor components (which provide connectionestablishment functionality typically used by servers) vs.the *Stream components (which provide bi-directionalbytestream data transfer functionality used by both clientsand servers).

Using OO wrappers to encapsulate the socket interfacehelps to (1) detect many subtle application type system vio-lations at compile-time, (2) facilitate a platform-independenttransport-level interface that improves application portabil-ity, and (3) greatly reduce the amount of application code anddevelopment effort expended upon lower-level network pro-gramming details. To illustrate the latter point, the follow-ing example program implements a simple client applicationthat uses theACESOCKDgram Bcast class to broadcast amessage to all servers listening on a designated port numberin a LAN subnet::

intmain (int argc, char *argv[]){

ACE_SOCK_Dgram_Bcast b_sap (sap_any);char *msg;unsigned short b_port;

msg = argc > 1 ? argv[1] : "hello world\n";b_port = argc > 2 ? atoi (argv[2]) : 12345;

if (b_sap.send (msg, strlen (msg),b_port) == -1)

perror ("can’t send broadcast"), exit (1);exit (0);

}

It is instructive to compare this concise example with thedozens of lines of C source code required to implementbroadcasting using the socket interface directly.

3.1.2 TLI SAP

The TLI SAP class category provides an OO interface tothe System V Transport Layer Interface (TLI). TheTLISAP inheritance hierarchy for TLI is almost identical to theSOCK SAPwrappers for sockets. The primary differenceis that TLI andTLI SAP do not define an interface to theUNIX-domain protocol family. In addition, TLI is not cur-rently ported to Win32 platforms.

By combining C++ features (such as default parame-ter values and templates) together with thetirdwr (theread/write compatibility STREAMS module), it be-comes relatively straight-forward to develop applicationsthat may be parameterized at compile-time to operate cor-rectly over either a socket-based or TLI-based transport in-terface. For instance, the following code illustrates howC++ templates may be applied to parameterize the IPCmechanisms used by an application. This code was ex-tracted from the distributed logging facility described inSection 4.1. In the code below, a subclass derived fromACEEvent Handler is parameterized by a particulartype of transport interface and its corresponding protocol ad-dress class:

/* Logging_Handler header file */template <class PEER_STREAM, class ADDR>class Logging_Handler : public ACE_Event_Handler{public:

Logging_Handler (void);virtual ˜Logging_Handler (void);

virtual int handle_input (ACE_HANDLE);virtual ACE_HANDLE get_handle (void) const{

5

AACCEELLSSOOCCKK

AACCEEIIPPCCSSAAPP

GGRROOUUPP

CCOOMMMM

DDAATTAAGGRRAAMM

CCOOMMMM

SSTTRREEAAMM

CCOOMMMM

CCOONNNNEECCTTIIOONN

EESSTTAABBLLIISSHHMMEENNTT

AACCEESSOOCCKK

AACCEESSOOCCKK

CCOODDggrraamm

AACCEESSOOCCKKDDggrraamm

ACESOCKDgramBcast

AACCEESSOOCCKK

AAcccceeppttoorr

AACCEESSOOCCKK

CCoonnnneeccttoorr

AACCEESSOOCCKKSSttrreeaamm

AACCEELLSSOOCCKKCCOODDggrraamm

AACCEELLSSOOCCKKDDggrraamm

AACCEELLSSOOCCKK

CCoonnnneeccttoorr

AACCEELLSSOOCCKK

AAcccceeppttoorr

AACCEELLSSOOCCKKSSttrreeaamm

ACESOCKDgramMcast

Figure 4: TheSOCK SAPClass Categories

return this->peer_stream_.get_handle ();}

protected:PEER_STREAM peer_stream_;

};

Depending on certain properties of the underlying OS plat-form (such as whether it is BSD-based SunOS 4.x or SystemV-based SunOS 5.x), the logging application may instantiatethe Client Handler class to use eitherSOCK SAPorTLI SAP , as shown below:

/* Logging application */class Logging_Handler#if defined (MT_SAFE_SOCKETS): public Logging_Handler<ACE_SOCK_Stream, ACE_INET_Addr>#else: public Logging_Handler<ACE_TLI_Stream, ACE_INET_Addr>#endif /* MT_SAFE_SOCKETS */{

/* ... */};

The increased flexibility offered by this template-basedapproach is extremely useful when developing an applicationthat must run portability across multiple OS platforms. Inparticular, the ability to parameterize applications accordingto transport interface is necessary across variants of SunOSplatforms since the socket implementation in SunOS 5.2 isnot thread-safe and the TLI implementation in SunOS 4.xcontains a number of serious defects.

TLI SAP also shields applications from many peculiari-ties of the TLI interface. For example, the subtle application-level code required to properly handle the non-intuitive,error-prone behavior oft listen andt accept in a con-current server with aqlen > 1 [3] is encapsulated withinthe accept method in theTLI Acceptor class. Thismethod accepts incoming connection requests from clients.Through the use of C++ default parameter values, the stan-dard method for calling theaccept method is syntacticallyequivalent for bothTLI SAP -based andSOCK SAP-basedapplications.

3.1.3 SPIPE SAP

The SPIPE SAP class category provides a OO wrapperinterface for high-performance local IPC. On Win32 plat-forms, theSPIPE SAPclass category is implemented atopNamed Pipes. The Win32 Named Pipes mechanism is pri-marily used to transfer data efficiently among processes onthe same machine. It is typically more efficient than socketsfor local IPC [27].

On UNIX platforms, theSPIPE SAP class category isimplemented with mounted STREAM pipes andconnld[28]. SunOS 5.x provides thefattach system call thatmounts a pipe handle at a designated location in the UNIXfile system. A server application is created by pushing theconnld STREAM module onto the mounted end of thepipe. When a client application running on the same hostmachine as the server subsequently opens the filename asso-ciated with the mounted pipe, the client and server each re-ceive an I/O handle that identifies a unique, non-multiplexed,bi-directional channel of communication.

The SPIPE SAP inheritance hierarchy mirrors the oneused for SOCK SAPand TLI SAP . It offers function-ality that is similar to theSOCK SAP ACELSOCK*classes (which themselves encapsulate UNIX-domain sock-ets). However, on SunOS 5.x platformsSPIPE SAP ismore flexible than theACELSOCK* interface since it en-ables STREAM modules to be “pushed” and “popped” toand from SPIPE SAP endpoints, respectively.SPIPESAPalso supports bi-directional delivery of byte-stream andprioritized message-oriented data between processes and/orthreads executing within the same host machine [29].

3.1.4 FIFO SAP

The FIFO SAP class category encapsulates the UNIXnamed pipe mechanism (also called FIFOs). UnlikeSTREAM pipes, named pipes offer only a uni-directionaldata channel from one or more senders to a single receiver.

6

Moreover, messages from different senders are all placedinto the same communication channel. Therefore, sometype of demultiplexing identifier must be included explicitlyin each message to enable the receiver to determine whichsender transmitted the message.

The STREAMS-based implementation of named pipes inSunOS 5.x provides both message-oriented and bytestream-oriented data delivery semantics. In contrast, some plat-forms, (such as SunOS 4.x) only provides bytestream-oriented named pipes. Therefore, unless fixed length mes-sages are always used, each message sent via a named pipe inSunOS 4.x must be distinguished by some form of byte countor special termination symbol that allows a receiver to ex-tract messages from the communication channel bytestream.To alleviate this limitation, the ACEFIFO SAP implemen-tation contains logic that emulates the message-oriented se-mantics available in SunOS 5.x.

3.1.5 Other Communication Mechanisms

In addition to encapsulating handle-based I/O communica-tion mechanisms such as sockets and TLI, ACE also providesOO wrappers for memory-mapped files and System V UNIXIPC mechanisms:

� Memory-Mapped Files: TheACEMemMap class pro-vides an OO interface to other memory-mapped file mech-anisms available on Win32 and UNIX (such as themmapfamily of system calls). These calls utilize the underlyingOS virtual memory facilities [30] to map files into the ad-dress space of a process. The contents of mapped files maybe accessed directly via pointers. A pointer interface is oftenmore convenient and efficient than accessing blocks of dataindirectly via the standardread /write I/O system calls.In addition, contents of memory-mapped files may be sharedconveniently between two or more processes.

Existing Win32 and UNIX interfaces for memory-mappedfiles are somewhat baroque. For instance, developers mustperform many bookkeeping details manually (such as ex-plicitly opening a file, determining its length, performingmultiple mappings, etc.). In contrast, theACEMemMapOO wrapper offers an interface that employs default valuesand multiple constructors with several type signature vari-ants (e.g., “map from an open file handle,” “map from a file-name,” etc.) to simplify typical memory-mapped file usagepatterns.

For example, thefollowing program uses theACEMemMap OO wrapper tomap a file specified via the command-line and print its linesin reverse:

static voidputline (const char *s){

while (putchar (*s++) != ’\n’)continue;

}

intmain (int argc, char *argv[]){

char *filename = argv[1];

char *file_p;Mem_Map mmap (filename);

if (mmap (file_p) != -1){

size_t size = mmap.size () - 1;

if (file_p[size] == ’\0’)file_p[size] = ’\n’;

while (--size >= 0)if (file_p[size] == ’\n’)

putline (file_p + size + 1);

putline (file_p);return 0;

}else

return 1;}

It is instructive to compare the use of this OO wrapper in-terface with the much more verbose C interface necessary touse I/O systems calls likeread directly.

� System V IPC Mechanisms: SunOS UNIX providesa suite of shared memory, synchronization, and messagepassing mechanisms known colloquially as “System V IPC”[29]. Most of the functionality offered by these mechanismshas been subsumed by more recent SunOS UNIX facilities(such asmmap, thread synchronization [31], and STREAMpipes primitives, respectively). However, certain types ofapplications (such as database engines) may benefit fromcharacteristics of System V IPC mechanisms (such as thepeer-to-peer nature of Message Queues, the efficient multi-operation atomicity semantics of Semaphores, and the wide-spread availability of System V IPC across a range of UNIXOS platforms). However, it is somewhat challenging to un-derstand and use System V IPC mechanisms (particularlysemaphores) correctly since their interfaces are quite generaland their behavior has traditionally been documented rathersparsely until recently [8, 29].

The ACE System V IPC wrapper interfaces shield devel-opers from a myriad of unnecessary details. For example,the ACE OO wrapper version of System V IPC semaphoresis more intuitive and simpler to use for applications that uti-lize standardwait andsignal semaphore operations, asshown in the following code fragment from a typical pro-ducer/consumer example:

typedef ACE_SV_Semaphore_Simple SEMA;SEMA prod (1, SEMA::CREATE, 1);SEMA cons (2, SEMA::CREATE, 0);

void producer (void){

for (;;) {prod.wait ();// produce resource...cons.signal ();

}}

void consumer (void){

for (;;) {cons.wait ();// consume resource...prod.signal ();

}}

7

: Initiation Dispatcher: Initiation Dispatcher

REGISTEREDREGISTERED

OBJECTSOBJECTS

FR

AM

EW

OR

KF

RA

ME

WO

RK

LE

VE

LL

EV

EL

KE

RN

EL

KE

RN

EL

LE

VE

LL

EV

EL

AP

PL

ICA

TIO

NA

PP

LIC

AT

ION

LE

VE

LL

EV

EL

: Event: EventHandlerHandler

: Logging: LoggingHandlerHandler

: Event: EventHandlerHandler

: Logging: LoggingHandlerHandler

1: handle_input()1: handle_input()

5: handle_input()5: handle_input()6: recv(msg)6: recv(msg)7:process(msg)7:process(msg)

: Event: EventHandlerHandler

: Logging: LoggingAcceptorAcceptor

2: sh = new Logging_Handler2: sh = new Logging_Handler3: accept (sh->peer())3: accept (sh->peer())4: sh->open()4: sh->open()

OS EVENT DEMULTIPLEXING INTERFACE

:Timer:TimerQueueQueue : Reactor: Reactor

: Handle: HandleTableTable

Figure 5: Software Architecture of the WWW Server

It is instructive to compare this concise OO wrapper interfacewith the much more verbose C interface necessary to useSystem V semaphores directly.

3.2 Reactor: Event Demultiplexing and EventHandler Dispatching

Communication software demultiplexes and processes manydifferent types of events (such as timer-based, I/O-based,signal-based, and synchronization-based events). For exam-ple, a WWW server is commonly structured internally usingan event loop that monitors a well-known Internet port (typ-ically port 80). This port is associated with an application-specific handler that listens for clients to connect on port 80.When clients connect, the WWW server accepts the connec-tion and creates an event handler to service the HTTP re-quest. For instance, if a Netscape browser sends aGETre-quest the WWW server will return the requested content tothe browser.

To consolidate and automate event-driven processing ac-tivities, ACE provides an event demultiplexing and eventhandler dispatching framework called theACEReactor[10]. TheReactor encapsulates the functionality of UNIXand Windows NT event demultiplexing mechanisms (suchasselect andpoll ) within a portable and extensible OOwrapper [10]. These OS event demultiplexing system callsdetect the occurrence of different types of input and outputevents on one or more I/O handles simultaneously.

To facilitate application portability, theACEReactorprovides the same interface regardless of what eventdemultiplexing mechanism is used.2 In addition, the

2An extended version of theACEReactor , calledACEReactorEx ,is used on Win32 platforms to encapsulatethe WaitForMultipleObjects event demultiplexing call. Since thisfunctionality is not portable across OS platforms, it is not covered in thisdocument.

: Initiation Dispatcher: Initiation Dispatcher

REGISTEREDREGISTERED

OBJECTSOBJECTS

FR

AM

EW

OR

KF

RA

ME

WO

RK

LE

VE

LL

EV

EL

KE

RN

EL

KE

RN

EL

LE

VE

LL

EV

EL

AP

PL

ICA

TIO

NA

PP

LIC

AT

ION

LE

VE

LL

EV

EL

: Event: EventHandlerHandler

: Logging: LoggingHandlerHandler

: Event: EventHandlerHandler

: Logging: LoggingHandlerHandler

1: handle_input()1: handle_input()

5: handle_input()5: handle_input()6: recv(msg)6: recv(msg)7:process(msg)7:process(msg)

: Event: EventHandlerHandler

: Logging: LoggingAcceptorAcceptor

2: sh = new Logging_Handler2: sh = new Logging_Handler3: accept (sh->peer())3: accept (sh->peer())4: sh->open()4: sh->open()

OS EVENT DEMULTIPLEXING INTERFACE

:Timer:TimerQueueQueue : Reactor: Reactor

: Handle: HandleTableTable

Figure 6: TheACEReactor Class Category Components

ACEReactor encapsulates the mutual exclusion mecha-nisms necessary to perform callback-style dispatching cor-rectly and efficiently in a multi-threaded event processingenvironment.

The structure of objects in theACEReactor is illus-trated in Figure 6. These objects are responsible for (1) de-multiplexing of events (such astemporal eventsgenerated bya timer-driven callout queue,I/O eventsreceived on commu-nication ports, andsignal events) and (2) dispatching the ap-propriate methods of pre-registered event handler(s) to pro-cess these events. As shown in Figure 6, all the event han-dler objects derive from theACEEvent Handler abstractbase class. This class specifies an interface that is used bythe ACEReactor to dispatch certain application-specificmethods in response to the arrival of certain events.

The ACEReactor uses the virtual methods de-clared in the Event Handler interface to integratethe demultiplexing of I/O handle-based, timer-based, andsignal-based events. I/O handle-based events are dis-patched via thehandle input , handle output , andhandle exceptions methods; timer-based events aredispatched via thehandle timeout method; and Signal-based events are dispatched via thehandle signalmethod.

Subclasses ofACEEvent Handler (such as theLogging Handler described in Section 4.1) may aug-ment the base class interface by defining additional meth-ods and data members. In addition, virtual methods inthe ACEEvent Handler interface may be selectivelyoverridden by subclasses to implement application-specificfunctionality. For example, application-specific subclassesin the PBX monitoring server presented in Section 4.2define Event Handler objects that communicate withclients by inheriting and/or instantiating objects of theSOCK SAPor TLI SAP transport interface classes de-scribed in Section 3.1. After the virtual methods in the

8

ACEEvent Handler base class have been defined by asubclass, an application may instantiate the resulting eventhandler object.

The following example implements a simple programthat continuously exchanges messages back and forth be-tween two processes using a bi-directional communica-tion channel. This example illustrates how services in-herit from the ACEEvent Handler . It also depictshow the ACEReactor is used to demultiplex and dis-patch I/O-based, signal-based, and timer-based events. ThePing Pong class shown below inherits the interface fromACEEvent Handler and implements its application-specific functionality as follows:

class Ping_Pong : public ACE_Event_Handler{public:

Ping_Pong (char *b): len (min (strlen (b) + 1, BUFSIZ)) {strncpy (this->buf, b, BUFSIZ);

}virtual int handle_input (ACE_HANDLE handle) {

return read (handle, this->buf, BUFSIZ);}virtual int handle_output (ACE_HANDLE handle) {

return write (handle, this->buf, this->len);}virtual int handle_signal (int signum) {

this->finished = 1;}virtual int handle_timeout (const Time_Value &,

const void *) {this->finished = 1;

}bool done (void) {

return this->finished == 1;}

private:sig_atomic_t finished;char buf[BUFSIZ];size_t len;

};

The bi-directional communication channel is created us-ing SVR4 UNIX STREAM pipes:

static intinit_handles (ACE_HANDLE handles[]){

if (pipe (handles) == -1)LM_ERROR ((LOG_ERROR, "%p\n%a", "pipe", 1));

// Enable message-oriented mode instead of// bytestream mode.int arg = RMSGN;

if (ioctl (handles[0], I_SRDOPT, arg) == -1|| ioctl (handles[1], I_SRDOPT, arg) == -1)

return -1;}

The main program begins by opening the appropriatecommunication channel. Following this, the program forksa child process, instantiates aPing Pong event handler ob-ject namedcallback in each of the two processes, regis-ters thecallback object for I/O-based, signal-based, andtimer-based events with an instance of theACEReactor ,and then enters an event loop, as follows:

int main (int argc, char *argv[]){

ACE_HANDLE handles[2];ACE_Reactor reactor;

mainprogram

INITIALIZE

REGISTER HANDLER

callback :Ping_Pong

START EVENT LOOP

DATA ARRIVES

OK TO SEND

reactor: Reactor

handle_events()

FOREACH EVENT DO

handle_input()

select()

Reactor::Reactor ()

register_handler(callback)

handle_output()

SIGNAL ARRIVES

TIMER EXPIRES

handle_signal()

handle_timeout()

Figure 7: ACEReactor Interaction Diagram

init_handles (handles);

pid_t pid = fork ();

Ping_Pong callback (argv[1]);

// Register I/O-based event handlerreactor.register_handler (

handles[pid == 0],&callback,ACE_Event_Handler::READ_MASK| ACE_Event_Handler::WRITE_MASK);

// Register signal-based event handlerreactor.register_handler (SIGINT, &callback);

// Register timer-based event handlerreactor.schedule_timer (&callback, 0, 10);

/* Main event loop (run in each process) */while (callback.done () == false)

reactor.handle_events ();

return 0;}

The callback event handler for the timer-based andsignal-based events is stored with the appropriate tables in-side theACEReactor . Likewise, theACEReactorstores the appropriate handle in an internal table whenthe register handler method is invoked to registerthe I/O-based event handler. When the application sub-sequently performs its main event loop by calling theACEReactor::handle events method, this handle ispassed as an argument to the underlying OS I/O demultiplex-ing system call (e.g.,select or poll ).

As input, output, signal, and timer events associatedwith the pre-registered event handlercallback object oc-cur at run-time, theACEReactor automatically detectsthese events and dispatches the appropriate method(s) ofthe event handler object. The dispatched method of thecallback object is responsible for performing application-specific functionality (such as writing a message to the com-munication channel, reading a message from the channel, orsetting a flag that triggers termination of the program). Thecollaboration between these components is depicted via theobject interaction diagram shown in Figure 7.

9

3.3 Concurrency: Multi-threading and Syn-chronization Mechanisms

The ACE Concurrency class category contains OO wrappers(e.g.,ACEMutex , ACECondition , ACESemaphore ,and ACERWMutex ) that encapsulate the correspondingSolaris [31] and POSIX Pthreads [32] multi-threading andsynchronization mechanisms. These wrappers automate theinitialization of synchronization objects that appear as fieldsin classes and also simplify typical usage patterns for thethreading and synchronization mechanisms. For instance,the following code illustrates the use of the ACE wrap-pers for the SunOSmutex t andcond t synchronizationmechanisms for a typical shared resource management class:class Resource_Manager{public:

Resource_Manager (u_int initial_resources): resource_add_ (this->lock),

resources_ (initial_resources) {}

int acquire_resources (u_int amount_wanted){

this->lock_.acquire ();

while (this->resources_ < amount_wanted) {this->waiting_++;// Block until resources are released.this->resource_add_.wait ();

}this->resources_ -= amount_wanted;this->lock_.release ();

}

int release_resources (u_int amount_released){

this->lock_.acquire ();this->resources_ += amount_released;if (this->waiting_ == 1) {

this->waiting_ = 0;this->resource_add_.signal ()

}else if (this->waiting_ > 1) {

this->waiting_ = 0;this->resource_add_.broadcast ();

}this->lock_.release ();

}// ...

private:ACE_Mutex lock_;ACE_Condition<ACE_Mutex> resource_add_;u_int resources_;u_int waiting_;// ...

};

Note how the constructor for theACECondition objectresource add binds theACEMutex object lock to-gether with theCondition object. This simplifies theACECondition::wait calling interface, in comparisonwith the underlying SunOScond t cond wait interface.

Although theACEMutex wrappers provide a relativelyelegant method for synchronizing multiple threads of con-trol, they are potentially error-prone since it is possible toforget to call therelease method (either due to program-mer negligence or due to the occurrence of C++ exceptions).To improve application robustness, the ACE synchronizationfacilities leverage off the semantics of C++ class constructorsand destructors. To ensure thatACEMutex locks will beautomatically acquired and released, ACE provides a helperclass calledACEGuard , which is defined as follows:

template <class MUTEX>class ACE_Guard{public:

ACE_Guard (MUTEX &m): lock (m) {this->lock_.acquire ();

}˜ACE_Guard (void) {

this->lock_.release ();}

private:MUTEX &lock_;

}

An object of theACEGuard class defines a block of codeover which aACEMutex is acquired and then released au-tomatically when the block is exited.

Note that theACEGuard class is defined as a templatethat is parameterized by mutual exclusion mechanism. Thereare several different types of mutex semantics [33]. Eachtype of mutual exclusion shares a common interface (i.e.,acquire /release ), but possesses different serializationand performance properties. Two types of mutual exclusionsupported by ACE arenon-recursiveandrecursivelocks.

� Non-recursive locks: A non-recursive lock provides anefficient form of mutual exclusion that define acritical sec-tion, where only a single thread may execute at a time.They are non-recursive in the sense that the thread currentlyowning a lock may not reacquire the lock without releas-ing it first. Otherwise, deadlock will occur immediately.SunOS 5.x provides support for non-recursive locks via itsmutex t , rwlock t , andsema t types (POSIX Pthreadsdoesn’t provide the latter two synchronization mechanisms).TheASX framework provides theMutex , RWMutex , andSemaphore wrappers to encapsulate these semantics, re-spectively.

� Recursive lock: A recursive lock, on the other hand, al-lows acquire method invocations to nest as long as thethread that owns the lock is the one trying to re-acquire it.Recursive locks are particularly useful for callback-drivenevent dispatching frameworks (such as theReactor de-scribed in Section 3.2), where the framework event-loop per-forms callbacks to pre-registered user-defined objects. Sincethe user-defined objects may subsequently re-enter the dis-patching framework via its method entry points, recursivelocks are necessary to prevent deadlock from occurring onlocks held within the framework during callbacks.

The following C++ template class implements recursivelock semantics for the synchronization mechanisms in So-laris threads and POSIX Pthreads whose native behaviordoes not provide recursive locking semantics:

template <class MUTEX>class ACE_Recursive_Thread_Mutex{public:

// Initialize a recursive mutex.ACE_Recursive_Thread_Mutex (void);

// Implicitly release a recursive mutex.˜ACE_Recursive_Thread_Mutex (void);

// Acquire a recursive mutex.int acquire (void) const;

// Conditionally acquire a recursive mutex.int tryacquire (void) const;

// Releases a recursive mutex.

10

int release (void) const;

private:ACE_Mutex nesting_mutex_;ACE_Condition<ACE_Mutex> mutex_available_;thread_t owner_id_;int nesting_level_;

};

Note that the interface for this class is consistent with theother locking mechanisms available in ACE [22].

The following code illustrates how theACEGuard andACERecursive Thread Mutex might be used within acallback mechanism:

intCallback::dispatch (const Event_Handler *eh,

Event *event){

// Constructor acquires the lock on entry.ACE_Guard<ACE_Recursive_Thread_Mutex<ACE_Mutex> >

m (this->lock_);

eh->handle_event (event);// Destructor of Guard releases the lock on exit.

}

This code ensures that registering anEvent Handler ob-ject executes as a critical section. This example also illus-trates the use of a C++ idiom [34] where the constructor ofa classacquire s the lock on a synchronization object au-tomatically when theACEGuard object is created. Like-wise, the class destruction automatically unlocks the objectwhen themon object goes out of scope. Moreover, the de-structor formonwill be invoked automatically to release theMutex lock regardless of which arm of theif/else state-ment returns from the method. In addition, the lock will alsobe released automatically if a C++ exception is raised dur-ing processing within the body of theregister handlermethod. TheACERecursive Thread Mutex is used toensure that application-specifichandle event callbacksthat are dispatched by thedispatch method do not causedeadlock if they reenter theCallback object.

ACE also provides aACEThread Manager class thatcontains a set of mechanisms to manage groups of threadsthat collaborate to implement collective actions. For in-stance, theACEThread Manager class provides mech-anisms (such assuspend all andresume all ) that al-low any number of participating threads to be suspended andresumed atomically.

3.4 Service Configurator: Explicit DynamicLinking Mechanisms

Static linkingis a technique for composing a complete ex-ecutable program by binding together all its object files atcompile-time and/or static link-time.Dynamic linking, incontrast, enables the addition and/or deletion of object filesinto the address space of a process at initial program invoca-tion or at any point later during run-time. SunOS 4.x and 5.xsupport bothimplicit andexplicit dynamic linking:

� Implicit dynamic linkingis used to implement sharedobject files, also known as shared libraries [35]. Shared

object files reduce primary and secondary storage uti-lization since only one copy of shared object code ex-ists in memory and on disk, regardless of the number ofprocesses that are executing this code. Moreover, cer-tain address resolution and relocation operations maybe deferred until a dynamically linked function is firstreferenced. This “lazy evaluation” scheme minimizeslink editing overhead during process start-up.

� Explicit dynamic linkingprovides interfaces that al-low applications to obtain, utilize, and/or removethe run-time address bindings of symbols definedin shared object files [36]. Explicit dynamic link-ing mechanisms significantly enhance the functional-ity and flexibility of communication software sinceservices may be inserted and/or deleted at run-time without terminating and/or restarting the en-tire application. SunOS 5.x supports explicit dy-namic linking via the dlopen/dlsym/dlcloseroutines and Win32 supports this feature via theLoadLibrary/GetProcAddress routines.

ACE provides theService Configurator class cat-egory to encapsulate the explicit dynamic linking mecha-nisms of SunOS within a set of classes and inheritance hier-archies. TheService Configurator leverages uponthe other ACE components to extend the functionality ofconventional daemon configuration and control frameworks[20] (such aslisten [3], inetd [2], and the WindowsNT Service Control Manager [37]) that provide au-tomated support for (1) static and dynamic configuration ofconcurrent, multi-service communication software, (2) mon-itoring sets of communication ports for I/O activity, and (3)dispatching incoming messages received on monitored portsto the appropriate application-specified services. The re-mainder of this section discusses the primary components intheService Configurator class category.

3.4.1 The ACEServiceObject Inheritance Hierarchy

The primary unit of configuration in theServiceConfigurator is the service. A service may be sim-ple (such as returning the current time-of-day) or highlycomplex (such as a distributed, real-time router for PBXevent traffic [6, 38]). To provide a consistent environ-ment for defining, configuring, and using communicationsoftware, all application services are derived from theACEService Object inheritance hierarchy (illustratedin Figure 8 (1)).

The ACEService Object class is the focal pointof a multi-level hierarchy of types related by inheritance.The standard interfaces provided by the abstract classesin this type hierarchy may be selectively implementedby application-specific subclasses in order to access cer-tain application-independentService Configuratormechanisms. These mechanisms provide transparent dy-namic linking, event handler registration, event demultiplex-ing, and service dispatching. By decoupling the application-

11

1

1

1

1

ServiceConfig

REACTOR

n

ServiceRepository

1

(2) The Service_RepositoryClass

(3) The Service_ConfigClass

ServiceRepository

ServiceRepository

Iterator

F

1

n

1

1

1

ServiceObject

A

(1) Service_ObjectInheritance Hierarchy

EventHandler

A

SharedObject

A

ServiceObject

Task

A

n

ServiceObject

EventHandler

APPLICATION-SPECIFIC

FUNCTIONALITY

Task

A

Task

AAPPLICATION-

INDEPENDENT

FUNCTIONALITY

Figure 8: Component Relationships for theService Configurator Class Category

specific portions of a handler object from the underly-ing application-independentService Configuratormechanisms, the effort necessary to insert and remove ser-vices from a running application is significantly reduced.

TheACEService Object inheritance hierarchy con-sists of the ACEEvent Handlerand ACEShared Object abstract base classes, as wellas theACEService Object abstract derived class. TheACEEvent Handler class was described above in Sec-tion 3.2. The behavior of the other classes is outlined below.

� The ACE Shared Object Abstract Base Class: Thisbase class specifies an interface for dynamically link-ing service handler objects into the address space of anapplication. TheACEShared Object abstract baseclass exports three abstract methods:init , fini ,and info . These methods impose a contract betweenthe application-independent reusable components providedby the Service Configurator and the application-specific functionality that utilizes these components. By us-ing abstract methods, theService Configurator en-sures that a service handler implementation honors its obli-gation to provide certain configuration-related information.This information is subsequently used by theServiceConfigurator to automatically link, initialize, identify,and unlink a service at run-time.

The ACEShared Object base class is defined inde-pendently from theACEEvent Handler class to clearlyseparate their two orthogonal sets of concerns. For example,certain applications (such as a compiler or text editor) mightbenefit from dynamic linking, though they might not requirecommunication port event demultiplexing. Conversely, otherapplications (such as anftp server) may require event de-

multiplexing, but might not require dynamic linking. By sep-arating these interfaces into two base classes, applicationsare able to select a subset ofService Configuratormechanisms without incurring unnecessary storage costs.

� The ACE ServiceObject Abstract Derived Class: Ingeneral, installing and administering complex distributedsystems requires support for dynamic linking, event demul-tiplexing, and service dispatching in order to automate thedynamic configuration and reconfiguration of applicationservices. Therefore, theService Configurator de-fines theACEService Object class, which combinesthe interfaces from both theACEEvent Handler and theACEShared Object abstract base classes. The resultingabstract derived class supplies an interface that developersuse as the basis for implementing and configuring a serviceinto theService Configurator .

During development, the application-specific subclassesof ACEService Object must implement the ab-stract suspend and resume methods specified by theACEService Object class interface. These methods areinvoked automatically by theService Configuratorin response to external events. An application developer maycontrol the actions the object undertakes in order to suspenda service without removing and unlinking it completely, aswell as to resume a previously suspended service.

In addition, application-specific subclasses must also im-plement the four abstract methods (init , fini , info ,and get handle ) that are inherited (but not defined) bythe ACEService Object subclass. Theinit methodserves as the entry-point to a service handler during run-time initialization. This method is responsible for perform-ing application-specific initialization when an instance of a

12

ACEService Object is dynamically linked. Likewise,the fini method is called automatically by theServiceConfigurator when aACEService Object is un-linked and removed from an application at run-time. Thismethod typically performs termination operations that re-lease dynamically allocated resources (such as memory, I/Ohandles, or synchronization locks). Theinfo method for-mats a humanly-readablestring that concisely reports serviceaddressing information and documents service functionality.Clients may query an application to retrieve this informationand use it to contact a particular service running in the ap-plication. Finally, theget handle method is used by theReactor to extract the underlying I/O handle from a ser-vice handler object. This I/O handle identifies a transportendpoint that may be used to accept connections or receivedata from clients.

� Application-Specific Concrete Derived Subclasses:Service Object is an abstract class since its interfacecontains the abstract methods inherited from theEventHandler and Shared Object abstract base classes.Therefore, developers must supply concrete subclasses that(1) define the six abstract methods described above and (2)implement the necessary application-specific functionality.To accomplish the latter task subclasses typically define cer-tain virtual methods exported by theService Object in-terface. For example, thehandle input method is oftenimplemented to accept connections or data that are receivedfrom clients.

TheACEAcceptor class depicted in Figure 8 (1) is anexample of an application-independent subclass that acceptsconnection requests as part of a distributed logging facility.This class is described further in the example presented inSection 4.1.

3.4.2 The ACE ServiceRepository Class

The Service Configurator class category supportstheconfiguration of both single-service and multi-service com-munication software. Therefore, to simplify run-time admin-istration, it is often necessary to individually and/or collec-tively control and coordinate theACEService Object sthat comprise an application’s currently active1 services.The ACEService Repository is an object managerthat coordinates local and remote queries involving the ser-vices offered by aService Configurator -based ap-plication. A search structure within the object managerbinds service names (represented as ASCII strings) with in-stances ofACEService Object s (represented as objectcode). A service name uniquely identifies an instance of aACEService Object stored in the repository.

Each entry in theACEService Repository con-tains a pointer to theACEService Object portion ofan application-specific derived class (shown in Figure 8 (2)).This enables theService Configurator to load, en-able, suspend, resume, or unloadACEService Object s

<svc-config-entries> ::=svc-config-entries svc-config-entry| NULL

<svc-config-entry> ::= <dynamic> | <static>| <suspend> | <resume> | <remove>| <stream> | <remote>

<dynamic> ::= DYNAMIC <svc-location>[ <parameters-opt> ]

<static> ::= STATIC <svc-name>[ <parameters-opt> ]

<suspend> ::= SUSPEND <svc-name><resume> ::= RESUME <svc-name><remove> ::= REMOVE <svc-name><stream> ::= STREAM <stream_ops>

’{’ <module-list> ’}’<stream_ops> ::= <dynamic> | <static><remote> ::= STRING ’{’ <svc-config-entry> ’}’<module-list> ::= <module-list> <module>

| NULL<module> ::= <dynamic> | <static>

| <suspend> | <resume> | <remove><svc-location> ::= <svc-name> <type>

<svc-initializer> <status><type> ::= SERVICE_OBJECT ’*’ | MODULE ’*’

STREAM ’*’ | NULL<svc-initializer> ::= <object-name>

| <function-name><object-name> ::= PATHNAME ’:’ IDENT<function-name> ::= PATHNAME ’:’ IDENT ’(’ ’)’<status> ::= ACTIVE | INACTIVE | NULL<parameters-opt> ::= STRING | NULL

Figure 9: EBNF Format for a Service Config Entry

Symbol Description

dynamic Dynamically link and enable a servicestatic Enable a statically linked serviceremove Completely remove a servicesuspend Suspend service without removing itresume Resume a previously suspended servicestream Configure a Stream into a daemon

Table 1: Service Config Directives

from an application statically or dynamically. For dy-namically linked ACEService Objects , the reposi-tory also maintains a handle to the underlying shared ob-ject file. This handle is used to unlink and unload aACEService Object from a running application whenthe service it offers is no longer required.

An iterator class is also supplied in conjunction with theACEService Repository . This class is used to visiteveryACEService Object in the repository without un-duly compromising data encapsulation.

3.4.3 The ACEServiceConfig Class

The ACEService Config class is the unifying com-ponent in the Service Configurator framework.As illustrated in Figure 8 (3), this class integratesthe otherService Configurator framework compo-nents (such as theACEService Repository and theACEReactor ) to automate the static and/or dynamic con-figuration of concurrent, multi-service communication soft-ware.

The ACEService Config class uses a configurationfile (known assvc.conf ) to guide its configuration and re-

13

INITIALIZED

CONFIGURE/Service_Config::process_directives()

NETWORK EVENT/Reactor::dispatch()

RECONFIGURE/Service_Config::process_directives()

SHUTDOWN/Service_Config::close() AWAITING

EVENTS

START EVENT LOOP/Service_Config::run_event_loop()

CALL HANDLER/Event_Handler::handle_input()

IDLE

PERFORMCALLBACK

Figure 10: State Transition Diagram for Service Configura-tion, Execution, and Reconfiguration

configuration activities. Each application may be associatedwith a distinctsvc.conf configuration file. Likewise, aset of applications may be described by a singlesvc.conffile. Figure 9 describes the primary syntactical elementsin a svc.conf file using extended-Backus/Naur Format(EBNF). Eachservice config entryin the file begins with aservice config directivethat specifies the configuration activ-ity to perform. Table 1 summarizes the valid service configdirectives.

Each service config entry contains attributes that indicatethe location of the shared object file for each dynamicallylinked service, as well as the parameters required to initializethe service at run-time. By consolidating service attributesand initialization parameters into a single configuration file,the installation and administration of the services in an appli-cation is significantly simplified. Thesvc.conf file helpsto decouple the structure of an application from the behaviorof its services. This decoupling also permits the “lazy” con-figuration and reconfiguration of mechanisms provided bythe framework, based on the application-specific attributesand parameters specified in thesvc.conf file.

Figure 10 depicts a state transition diagram illustrat-ing the methods in theService Configurator classcategory that are invoked in response to events occurringduring service configuration, execution, and reconfigura-tion. For example, when theCONFIGURE and RECONFIG-URE events occur, theprocess directives method ofthe ACEService Config class is called to consult thesvc.conf file. This file is first consulted when a new in-stance of a application is initially configured. The file isconsulted again whenever application reconfiguration is trig-gered upon receipt of a pre-designated external event (suchas the UNIX SIGHUP signal or a notification arriving froma socket).

3.5 Stream: Layered Serivce Integration

The Stream class category is the primary focal point of theADAPTIVE Communication Environment. This class cat-egory contains the ADAPTIVE Service eXecutive (ASX)framework [6] (ASX) framework, which integrates the lower-level OO wrapper components (likeIPC SAP) and higher-level class categories (like theReactor and theServiceConfigurator ). The ASX framework helps to simplifythe development of hierarchically-integrated communica-tion software, particularly user-level communication proto-col stacks and network servers. TheASX framework is de-signed to improve the modularity, extensibility, reusability,and portability of both the application-specific services andthe underlying OS concurrency, IPC, explicit dynamic link-ing, and demultiplexing mechanisms upon which these ser-vices are built.

The ASX framework provides the following two benefitsto developers of communication software:

1. It embodies, encapsulates, and implements key de-sign patternsthat are commonly used to develop com-munication software. Design patterns help to en-hance software quality by addressing fundamental chal-lenges in large-scale system development. These chal-lenges include communication of architectural knowl-edge among developers; accommodating new de-sign paradigms or architectural styles; resolving non-functional forces such as reusability, portability, and ex-tensibility; and avoiding development traps and pitfallsthat are usually learned only by experience.

2. It strictly separates key development concerns– ACEseparates communication software development intotwo distinct categories: (1)application-independentconcerns, which are common to most or all com-munication software (such as port monitoring; mes-sage buffering, queueing, and demultiplexing; servicedispatching; local/remote interprocess communication;concurrency control; and application configuration, in-stallation, and run-time service management) and (2)application-specific concerns, which depend on an in-dividual application. By reusing the OO wrappers andframeworks provided by ACE, developers are freedfrom spending their time reinventing solutions to com-monly recurring tasks. In turn, this enables them to con-centrate on the key higher-level functional requirementsand design concerns that constitute particular applica-tions.

3.5.1 Primary ASXFeatures

The ASX framework increases the flexibility of communi-cation software by decoupling application-specific process-ing policies from the following configuration-related devel-opment activities and mechanisms:

� The type and number of services associated with eachapplication process: The ASX framework permits appli-

14

cations to consolidate one or more services into a single ad-ministrative unit. This multi-service approach to configuringcommunication software helps to (1) simplify developmentand reuse code by performing common service initializationactivities automatically, (2) reduce the consumption of OSresources (such as process table slots) by spawning servicehandlers “on-demand,” (3) allow application services to beupdated without modifying existing source code or terminat-ing an executing dispatcher process (such as theinetd su-perserver), and (4) consolidate the administration of networkservices via a uniform set of configuration management op-erations.

� The point of time at which a service is configuredinto an application: TheASXframework leverages off theService Configurator framework (described in Sec-tion 3.4.1) to provide an extensible object-oriented interfacethat automates the use of OS mechanisms for explicit dy-namic linking. Dynamic linking enhances the extensibilityof communication software by permitting internal servicesto be configured when an application first begins executingor while it is running. This feature enables an application’sservices to be dynamic reconfiguredwithout requiring themodification, recompilation, relinking, or restarting of activeservices. In theASXframework, the choice between static ordynamic configuration may selected on a per-service basis.Furthermore, this choice may be deferred until an applicationbegins execution.

� The type of execution agents: In the ASX framework,services may be performed at run-time via several differenttypes of process and thread execution agents. By decouplingservice functionality from the execution agent used to invokethe service, theASX framework increases the range of ap-plication concurrency configuration alternatives available todevelopers.

An efficient application concurrency configuration oftendepends upon certain service requirements and platformcharacteristics. For example, a process-based configurationmay be appropriate for implementing long-duration services(such as the Internetftp and telnet ) that base their se-curity mechanisms on process ownership. In this case, eachservice (or each active instance of a service) may be mappedonto a separate process and executed in parallel on a multi-processor platform. Different configurations may be moresuitable in other circumstances, however. For instance, it isoften simpler and more efficient to implement cooperatingservices (such as those found in end-systems of distributeddatabase engines) in separate threads since they frequentlyreference common data structures. In this approach, eachservice may be executed on a separate thread within thesame process to reduce the overhead of scheduling, contextswitching, and synchronization [6].

� The order in which hierarchically-related services arecombined into an application: Complex services may becomposed using an interconnected series of independent ser-vice objects that communicate by passing messages. These

objects may be joined together in essentially arbitrary con-figurations to satisfy application requirements and enhancecomponent reuse.

� The I/O handle-based and timer-based event demulti-plexing mechanisms: These mechanisms are used to dis-patch incoming connection requests and data onto a pre-registered application-specific handler. TheASXframeworkuses theReactor class category to integrate the demulti-plexing of I/O handle-based, timer-based, and signal-basedevents via an extensible and type-safe object-oriented inter-face.

� The underlying IPC mechanisms: Application servicesmay use theIPC SAP mechanisms described in Section 3.1to exchange data with participating communication entitieson local or remote end-systems. Unlike the weakly-typed,“handle-based” socket and TLI interfaces, theIPC SAPwrappers enable applications to access the underlying OSIPC mechanisms via a type-safe, portable interface.

The ASX framework incorporates concepts from severalmodular communication frameworks including System VSTREAMS [39], thex-kernel [40], and the Conduit frame-work [41] from the Choices object-oriented operating system(a survey of these and other communication frameworks ap-pears in [42]). These frameworks all contain features thatsupport the flexible configuration of communication subsys-tems by inter-connecting “building-block” protocol and ser-vice components. In general, these frameworks encouragethe development of standard communication-related com-ponents (such as message managers, timer-based event dis-patchers, demultiplexors [40], and assorted protocol func-tions [43]) by decoupling processing functionality from thesurrounding framework infrastructure. As described below,theASX framework contains additional features that furtherdecouple processing functionality from the underlying pro-cess architecture.3

Unlike STREAMS, application services configured intothe ASX framework execute in user-space rather than inkernel-space. There are several advantages to develop-ing general-purpose communication software in user-space,rather than within the OS kernel:

� Access to general OS features: Applications developedto run in user-space have access to the full range of OS mech-anisms (such as dynamic linking, memory-mapped files,multi-threading, large virtual address spaces, interprocesscommunication mechanisms, file systems, and databases). Incontrast, kernel-resident components are often restricted to alimited set of kernel-specific mechanisms. Although thereare idioms that overcome some of these limitations (e.g.,maintaining a user-level daemon that performs file I/O onbehalf of a kernel-resident component), these workaroundstend to be somewhat inelegant and non-portable.

3A process architecture represents a binding between one or more CPUstogether with the application tasks and messages that implement services ina communication system [6].

15

� Enhanced development environment: Higher-levelprogramming tools (such as symbolic debuggers) may beused to develop applications in user-space. Conversely, de-veloping network services within a OS kernel is a complexand challenging task, due to the primitive debugging toolsand subtle timing interactions in a kernel programming envi-ronment [44]. It is risky to expect application developers toprogram effectively in such a constrained environment.

� Increased system robustness: Exceptional conditions(such as dereferencing NULL pointers, dividing by 0, etc.)generated in a user-level process or thread should only affectthe offending process or thread. However, exceptional con-ditions within a OS kernel may cause the entire operatingsystem to panic and crash. Moreover, rebooting the OS afterevery crash quickly becomes tedious.

� Portability: Porting kernel-level drivers between differ-ent OS platforms (and even different variants of the sameOS platform) typically entails many more complicationsthan porting user-level application components between plat-forms. The SunOS 5.x DDI/DKI API is intended to alle-viate some of the portability problems on UNIX platforms,but that does not solve the portability problem for applica-tions running on other platforms such as OS/2, Windows NT,VMS, and Novell Netware.

The primary rationale for implementing distributed ser-vices in a OS kernel is to improve performance. For ex-ample, a kernel-resident implementation of a communicationprotocol stack often helps reduce scheduling, context switch-ing, and protection-domain boundary crossing overhead andmay exhibit more predictable response time due to the useof “wired” (rather than paged) memory. However, for manycommunication software applications, the increased flexibil-ity, simplicity, robustness, and portability offered by user-space development are key requirements that offset potentialperformance degradations.

3.5.2 Stream Class Category Components

Components in theStream class category are responsiblefor coordinating the configuration and run-time execution ofone or moreACE Streamobjects. AnACEStream is anobject that user applications collaborate with to configureand execute application-specific services in theASX frame-work. As illustrated in Figure 3.5, anACEStream containsa series of inter-connectedACEModule objects that maybe linked together by (1) developers at installation-time or(2) applications at run-time.ACEModule s are objects usedto decompose the architecture of an application into a seriesof inter-connected, functionally distinct layers. Each layertypically implements a cluster of related service functional-ity (such as an end-to-end transport service or a presentationlayer formatting service). EveryACEModule contains apair of ACETask objects that partition a layer into its con-stituent read-side and write-side processing functionality.

OO language features (such as classes, inheritance, dy-namic binding, and parameterized types) enable develop-

NETWORK INTERFACE

OR PSEUDO-DEVICES

STREAMTail

Multiplexor

APPLICATION

Stream

STREAMHead

APPLICATION

Stream

UP

ST

RE

AMD

OW

NS

TR

EA

MMESSAGEMESSAGE WRITEWRITE

TASKTASK

READREAD

TASKTASKMODULEMODULE

open()=0close()=0put()=0svc()=0

Figure 11: Components in theASXFramework

ers to incorporate application-specific functionality into aStream without modifying the application-independentASXframework components. For example, adding a servicelayer into a Stream involves (1) inheriting from the defaultACETask interface and selectively overriding several vir-tual methods in the subclass to provide application-specificfunctionality, (2) allocating a newACEModule that con-tains two instances (one for the read-side and one for thewrite-side) of the application-specificACETask subclass,and (3) inserting theACEModule into a Stream. Servicesin adjacent inter-connectedACETask s collaborate by ex-changing typed messages via a message passing interface.

To avoid reinventing familiar terminology, many classnames in the Stream class category correspond to similarcomponentry available in the System V STREAMS frame-work [39]. However, the techniques used to support exten-sibility and concurrency in the two frameworks are signif-icantly different. For example, adding application-specificfunctionality to theASX Stream classes is performed byinheriting several well-defined interfaces and implementa-tions from existing framework components. Using inheri-tance to add new functionality provides greater type-safetythan the pointer-to-function techniques used in System VSTREAMS. TheASXStream classes also employ a differ-ent concurrency control scheme to reduce the likelyhoodof deadlock and to simplify flow control betweenTask sin a Stream. TheASXStream classes completely redesignand reimplement the co-routine-based, “weightless”4 service

4A weightless process executes on a run-time stack that is also used by

16

processing mechanisms used in System V STREAMS [45].TheseASXchanges are intended to utilize multiple PEs on ashared memory multi-processor platform more effectively.

The remainder of this section discusses the primary com-ponents of the Stream class category (e.g., ACEStreamclass, theACEModule class, theACETask class in de-tail:

� The ACE Stream Class: The ACEStream class de-fines the application interface to a Stream. AnACEStreamobject contains a stack of one or more hierarchically-relatedservices that provide applications with a bi-directionalget /put -style interface for sending and receiving data andcontrol messages through the inter-connectedModules thatcomprise a particular Stream. TheACEStream class alsoimplements apush /pop -style interface that allows applica-tions to configure a Stream at run-time by inserting and re-moving objects of theACEModule class described below.

� The ACE Module Class: The ACEModule class de-fines a distinct layer of application-specific functional-ity. A Stream is formed by inter-connecting a seriesof ACEModule objects. ACEModule objects in aStream are loosely coupled, and collaborate with adjacentACEModule objects by passing typed messages. EachACEModule object contains a pair of pointers to objectsthat are application-specific subclasses of theACETaskclass described shortly below.

As shown in Figure 3.5, two defaultACEModule ob-jects (ACEStream Head and ACEStream Tail ) areinstalled automatically when a Stream is opened. These twoACEModule s interpret pre-definedASX framework con-trol messages and data messages that circulate through aStream at run-time. TheACEStream Head class providesa message buffering interface between an application and aStream. TheACEStream Tail class typically transformsincoming messages from a network or from a pseudo-deviceinto a canonical internal message format that may be pro-cessed by higher-level components in a Stream. Likewise,for outgoing messages it transforms messages from their in-ternal format into network messages.

� The ACE Task Abstract Class: The ACETask classis the central mechanism in ACE for creating user-definedactive objects[11] andpassive objectsthat process applica-tion messages. An ACETask can perform the followingactivities:

� Can be dynamically linked;

� Can serve as a demultiplexing endpoint for I/O opera-tions;

� Can be associated with multiple threads of control (ı.e.,become a so-called “active object”);

� Can store messages in a queue for subsequent process-ing;

other processes, which complicates programming and increases the poten-tial for deadlock. For example, a weightless process may not suspend exe-cution to wait for resources to become available or events to occur.

� Can execute user-defined services.

The ACETask abstract class defines an interface thatis inherited and implemented by derived classes in orderto provide application-specific functionality. It is an ab-stract class since its interface defines the abstract methods(open , close , put , andsvc ) described below. Defin-ing ACETask as an abstract class enhances reuse by de-coupling the application-independent components providedby the ACEStream class category from the application-specific subclasses that inherit from and use these compo-nents. Likewise, the use of abstract methods allows the com-piler to ensure that a subclass ofTask honors its obligationto provide the following functionality:

� Initialization and Termination Methods– Subclassesderived fromACETask must implementopen andclose methods that perform application-specificACETask initialization and termination activities.These activities typically allocate and free resourcessuch as connection control blocks, I/O handles, and syn-chronization locks.

ACETasks can be defined and used either to-gether withACEModule s or separately. When usedwith ACEModules they are stored in pairs: oneACETask subclass handles read-side processing formessages sent upstream to itsACEModule layer andthe other handles write-side processing messages senddownstream to itsModule layer. The open andclose methods of aModule ’s write-side and read-side ACETask subclasses are invoked automaticallyby theASX framework when theACEModule is in-serted or removed from a Stream, respectively.

� Application-Specific Processing Methods– In additionto open and close , subclasses ofACETask mustalso define theput and svc methods. These meth-ods perform application-specific processing functional-ity on messages. For example, when messages arriveat the head or the tail of a Stream, they are escortedthrough a series of inter-connectedACETasks as aresult of invoking theput and/orsvc method of eachACETask in the Stream.

A put method is invoked when aACETask at onelayer in a Stream passes a message to an adjacentACETask in another layer. Theput method runssynchronouslywith respect to its caller,i.e., it borrowsthe thread of control from theTask that originally in-voked its put method. This thread of control typi-cally originate either “upstream” from an applicationprocess, “downstream” from a pool of threads that han-dle I/O device interrupts [40], or internal to the Streamfrom an event dispatching mechanism (such as a timer-driven callout queue used to trigger retransmissions in aconnection-oriented transport protocolACEModule ).

If an ACETask executes as apassive object(i.e.,it always borrows the thread of control from the

17

SVC A

SVC B

SVC C

SVC A

SVC B

SVC C

2:svc()

1:put()

4:svc()

3:put()

2:put()

1:put()

(1) TASK-BASEDCONCURRENCY ARCHITECTURE

(2) MESSAGE-BASEDCONCURRENCY ARCHITECTURE

MSG

MSG MSG

MSG

MSGMSG

MSG

Figure 12: Alternative Methods for Invokingput andsvc Methods

caller), then theACETask::put method is the en-try point into theACETask and serves as the con-text in which ACETask executes its behavior. Incontrast, if an ACEACETask executes as anactiveobject the ACETask::svc method is used to per-form application-specific processingasynchronouslywith respect to otherACETask s. Unlike put , thesvc method is not directly invoked from an adja-cent ACETask . Instead, it is invoked by a separatethread associated with itsACETask . This thread pro-vides an execution context and thread of control for theACETask ’s svc method. This method runs an eventloop that continuously waits for messages to arrive ontheACETask ’s ACEMessage Queue (see next bul-let).

Within the implementation of aput or svc method, amessage may be forwarded to an adjacentACETaskin the Stream via theput next ACE Task utilitymethod.Put next calls theput method of the nextACETask residing in an adjacent layer. This invo-cation ofput may borrow the thread of control fromthe caller and handle the message immediately (i.e.,the synchronous processing approach illustrated in Fig-ure 12 (1)). Conversely, theput method may enqueuethe message and defer handling to itssvc method thatis executing in a separate thread of control (i.e., theasynchronous processing approach illustrated in Fig-ure 12 (2)). As discussed in [6], the particular process-ing approach that is selected has a significant impact onperformance and ease of programming.

� Message Queueing Mechanisms– In addition to theopen , close , put , andsvc abstract method inter-faces, eachACETask also contains anACEMessage Queue.An ACEMessage Queue is a standard compo-nent in ACE that is used pass information between

ACETasks . Moreover, when aACETask exe-cutes as an active object, itsACEMessage Queue isused to buffer a sequence of data messages and con-trol messages for subsequent processing in thesvcmethod. As messages arrive, thesvc method dequeuesthe messages and performs theACETask subclass’sapplication-specific processing tasks.

Two typesof messages may appear on aACEMessage Queue:simple and composite. A simple message contains asingle ACEMessage Block and a composite mes-sage contains multipleACEMessage Block s linkedtogether. Composite messages generally consist of acontrol block followed by one or moredatablocks. Acontrol block contains bookkeeping information (suchas destination addresses and length fields), whereas datablocks contain the actual contents of a message. Theoverhead of passingACEMessage Block s betweenTask s is minimized by passing pointers to messagesrather than copying data.

ACEMessage Queues contain a pair of high and lowwater mark variables that are used to implement layer-to-layer flow control between adjacentACEModulesin a Stream. The high water mark indicates the amountof bytes of messages theACEMessage Queue iswilling to buffer before it becomes flow controlled. Thelow water mark indicates the level at which a previouslyflow controlledACETask is no longer considered tobe full.

4 ACE Examples

The ACE components are currently being used in several re-search [46] and commercial environments [6, 38, 47] to en-hance the configuration flexibility and software component

18

NETWORK

int spawn (void) { if (fork () == -1) Log_Msg::log (LOG_ERROR, "unable to fork in function spawn");

SERVER

CLIENTCRIMEE

NAMED PIPECLIENT

LOGGING

DAEMON

P1

P3P2

{ if (Options::debugging) Log_Msg::log (LOG_DEBUG, "sending request to server %s", server_host); Comm_Manager::send (request, len);

HOST A HOST BCRIMEE ZOLA

SERVER LOGGING

DAEMON

STORAGE DEVICE

PRINTER

CLIENTZOLANAMED PIPE

CLIENT

LOGGING

DAEMON

P1

P3P2

TCP

CO

NN

ECTI

ON

CONSOLE

Oct 29 14:50:13 1992@crimee.ics.uci.edu@22677@7@client-test::unable to fork in function spawnOct 29 14:50:28 1992@zola.ics.uci.edu@18352@2@drwho::sending request to server bastilleOct 29 14:48:13 1992@crimee.ics.uci.edu@38491@7@client::unable to fork in function spawn

TCP CONNECTION

Figure 13: The Distributed Logging Facility

reuse of communication software that operate efficiently andportably across multiple hardware and software platforms.To illustrate how theASXframework is used in practice, thissection examines the architecture of two commercial appli-cations currently being developed using ACE components: adistributed logging facility and a distributed monitoring sys-tem for telecommunication switch devices.

4.1 Distributed Logging Example

Debugging distributed communication software is frequentlychallenging since diagnostic output appears in different win-dows and/or on different remote host systems. Therefore,ACE provides a distributed logging facility that simplifiesdebugging and run-time tracing. This facility is currentlyused in a commercial on-line transaction processing system[14] to provide logging services for a cluster of worksta-tions and multi-processor database servers in a high-speednetwork environment.

As shown in Figure 13, the distributed logging facility al-lows applications running on multiple client hosts to sendlogging records to a server logging daemon running on adesignated server host. This section focuses on the ar-chitecture and configuration of the server daemon portionof the logging facility, which is based on theServiceConfigurator andACEReactor class categories pro-vided by theASXframework. The complete design and im-plementation of the distributed logging facility is describedin [10].

ServiceServiceConfiguratorConfigurator

StreamStreamConnectionConnection

Logging

Acceptor

Logging_HandlerSOCK_Acceptor

Logging

Handler

SOCK_StreamNull_Synch

Svc

HandlerAcceptor

SVC_HANDLERPEER_ACCEPTOR

PEER_STREAMSYNCH_STRAT

CO

NN

EC

TIO

N-

OR

IEN

TE

D

CO

MP

ON

EN

TS

AP

PL

ICA

TIO

NA

PP

LIC

AT

ION

--S

PE

CIF

ICS

PE

CIF

IC

CO

MP

ON

EN

TS

CO

MP

ON

EN

TS

AC

EA

CE

FR

AM

EW

OR

KF

RA

ME

WO

RK

CO

MP

ON

EN

TS

CO

MP

ON

EN

TS

PEER

ACCEPTORPEER

STREAM

IPC_SAPIPC_SAP

ReactorReactor

ACTIVATES

1 n

ConcurrencyConcurrency

Figure 14: Class Components in the Server Logging Daemon

4.1.1 Server Logging Daemon Components

The server logging daemon is a concurrent, multi-servicedaemon that processes logging records received from oneor more client hosts simultaneously. The object-orienteddesign of the server logging daemon is decomposed intoseveral modular components (shown in Figure 14) that per-form well-defined tasks. The application-specific compo-nents (Logging Acceptor and Logging Handler )are responsible for processing logging records received fromclients. The connection-oriented application components(Acceptor and Client Handler ) are responsible foraccepting connection requests and data from clients. Fi-nally, the application-independent ASX framework compo-nents (theACEReactor , Service Configurator ,andIPC SAP class categories) are responsible for perform-ing IPC, explicit dynamic linking, event demultiplexing, ser-vice dispatching, and concurrency control.

The Logging Handler subclass is a parameterizedtype that is responsible for processing logging records sentto the server logging daemon from participating client hosts.Its communication mechanisms may be instantiated with ei-ther theSOCK SAPor TLI SAP wrappers, as follows:

class Logging_Handler : public Client_Handler <#if defined (MT_SAFE_SOCKETS)

ACE_SOCK_Stream,#else

ACE_TLI_Stream,#endif /* MT_SAFE_SOCKETS */

ACE_INET_Addr>{

/* ... */};

The Logging Handler class inherits from EventHandler (indirectly via Client Handler ) rather than

19

Service Object since it is not dynamic linked into theserver logging daemon.

When logging records arrive from the client host as-sociated with a particularLogging Handler object,the ACEReactor automatically dispatches the object’shandle input method. This method formats and dis-plays the records on one or more output devices (such as theprinter, persistent storage, and/or console devices illustratedin Figure 13).

TheLogging Acceptor subclass is also a parameter-ized type that is responsible for accepting connections fromclient hosts participating in the logging service:

class Logging_Acceptor :public Client_Acceptor<Logging_Handler,

#if defined (MT_SAFE_SOCKETS)ACE_SOCK_Acceptor,

#elseACE_TLI_Acceptor,

#endif /* MT_SAFE_SOCKETS */ACE_INET_Addr>

{/* ... */

};

Since the Logging Acceptorclass inherits fromACEService Object (indirectly viaits ACEAcceptor base class), it may be dynamicallylinked into the server logging daemon and manipulatedat run-time via the server logging daemon’ssvc.confconfiguration file. Likewise, sinceLogging Acceptorindirectly inherits from theACEEvent Handler inter-face, its handle input method will be invoked au-tomatically by theACEReactor when connection re-quests arrive from clients. When a connection requestarrives, the Logging Acceptor subclass allocates aLogging Handler object and registers this object withtheACEReactor .

The modularity, reusability, and configurability of the dis-tributed logging facility is significantly enhanced by de-coupling the functionality of connection establishment andlogging record reception into the two distinct class hi-erarchies shown in Figure 14. This decoupling allowsthe ACEAcceptor class to be reused for other types ofconnection-oriented services. In particular, to provide com-pletely different processing functionality, only the behaviorof theACEClient Handler portion of the service wouldneed to be reimplemented. Furthermore, the use of parame-terized types decouples the reliance on a particular type IPCmechanism.

4.1.2 Server Logging Daemon Configuration

TheASX framework uses theService Configuratorto enable the dynamic and static configuration of logging ser-vices into the server logging daemon. Dynamically config-ured services may be inserted, modified, or removed at run-time, thereby improving service flexibility and extensibility.The following svc.conf file entry is used to to dynami-cally configure the logging service into the server loggingdaemon:

LoggingHandler

ServiceRepository

ServiceConfig

Reactor

SERVER

SERVERSERVER

LOGGINGLOGGING

DAEMONDAEMON

Service ServiceManagerManager

LoggingHandler

LoggingAcceptor

CONNECTION

REQUEST

REMOTE

CONTROL

OPERATIONS

CLIENT

LOGGING

RECORDS

CLIENT CLIENTCLIENT

Figure 15: ACE Components in the Distributed Logging Fa-cility

dynamic Logger Service_Object *./Logger.so:_alloc() "-p 7001"

The <svc-name > token Logger specifies the ser-vice name that is used at installation and run-time toidentify the correspondingService Object within theACEService Repository . Service Object *is the return type of the alloc method that is lo-cated in the shared object file indicated by the pathname./Logger.so . TheService Configurator frame-work locates and dynamically links this shared object fileinto the logging daemon’s address space. The service lo-cation also specifies the name of the application-specificobject derived fromService Object . In this case,the alloc function is used to dynamically allocate anewLogging Acceptor object. The remaining contentson the line ("-p 7001" ) represent a application-specificset of configuration parameters. These parameters arepassed to theinit method of the service asargc/argv -style command-line arguments. Theinit method for theLogging Acceptor class interprets"-p 7001" as theport number where the server logging daemon listens forclient connection requests.

Statically configured services are always available toa daemon when it first begins execution. For exam-ple, the Service Manager is a standardServiceConfigurator framework component that clients use toobtain a listing of active daemon services. The following en-try in thesvc.conf file is used to statically configure theService Manager service into the server logging dae-

20

: ServiceConfig

L :Logging_Handler

LoggerDaemon

REGISTER SERVICE

A :Logging Acceptor

START EVENT LOOP

CONNECTION EVENT

DATA EVENT

REGISTER NEW HANDLER

FOR CLIENT I/O

PROCESS LOGGING

RECORD

register_handler(A)

: Reactor

run_event_loop()

handle_events()FOREACH EVENT DO

handle_input()

L = new Logging_HandlerL.open (A);register_handler(L)

handle_input()

write()

LOAD SERVICE

CONFIGURE

FOREACH SVC ENTRY DO

open()

: ServiceRepository

process_directives()

load_service()

Figure 16: Interaction Diagram for the Server Logging Daemon

mon during initialization:

static ACE_Svc_Manager "-p 911"

In order for thestatic directive to work, the objectcode that implements theACESvc Manager service mustbe statically linked together with the main daemon driverexecutable program. In addition, theACESvc Managerobject must be inserted into theService Repositorybefore dynamic configuration occurs (this is done automati-cally by theACEService Config constructor). Due tothese constraints, a statically configured service may not bereconfigured at run-time without being removed from theService Repository first.

The main driver program for the server logger daemon isimplemented by the following code:

intmain (int argc, char *argv[]){

ACE_Service_Config loggerd;

// Configure server logging daemon.if (loggerd.open (argc, argv) == -1)

return -1;

// Perform logging service.loggerd.run_reactor_event_loop ();return 0;

}

Figure 16 depicts the run-time interaction between the vari-ous framework and application-specific objects that collabo-rate to provide the logging service. Daemon configuration isperformed in theACEService Config::open method.This method consults the followingsvc.conf file, whichspecifies the services to configure into the daemon:

static ACE_Svc_Manager -p 911dynamic Logger Service_Object *

./Logger.so:_alloc() "-p 7001"

Each of the service config entries in thesvc.conf file ispro-cessed by inserting the designatedACEService Object

into theACEService Repository and registering theACEEvent Handler portion of the service object handlerwith theACEReactor .

When all the configuration activities have been com-pleted, the main driver program shown above in-vokes the run reactor event loop method of theACEService Config . This method enters an event loopthatcontinuously calls theACEReactor::handle eventsservice dispatch method. As shown in Figure 10, this dis-patch function blocks awaiting the occurrence of events(such as connection requests or I/O from clients). As theseevents occur, theACEReactor automatically dispatchespreviously-registered event handlers to perform the desig-nated application-specific services.

TheASXframework also responds to external events thattrigger daemon reconfiguration at run-time. The dynamicconfiguration steps outlined above are performed wheneveran executingASX-based daemon receives a pre-designatedexternal event (such as the UNIX SIGHUP signal). Depend-ing on the updated contents of thesvc.conf file, servicesmay be added, suspended, resumed or removed from the dae-mon.

The ASX framework’s dynamic reconfiguration mecha-nisms enable developers to modify server logging daemonfunctionality or fine-tune performance without extensive re-development and reinstallation effort. For example, debug-ging a faulty implementation of the logging service simplyrequires the dynamic reinstallation of a functionally equiva-lent service that contains additional instrumentation to helpisolate the source of erroneous behavior. Note that this re-installation process may be performed without modifying,recompiling, relinking, or restarting the currently executingserver logging daemon.

21

4.2 Distributed PBX Monitoring System

Figure 17 illustrates the client/server architecture of a privatebranch exchange (PBX) telecommunication switch monitor-ing system implemented usingASXframework components[20]. In this distributed communication system, the serverreceives and processes status information generated by oneor more PBXs attached to the server via a high-speed com-munication link. The server transforms and forwards this sta-tus information across a network to client end-systems thatgraphically display the information to end-users. End-usersare typically supervisors who use the PBX status informa-tion to monitor the performance of personnel and forecastthe allocation of resources to meet customer demands.

The PBX devices attached to the server are con-trolled by the Device Adapter ACE Module . ThisACEModule shields the rest of the server from PBX-specific communication characteristics. The read-side of theDevice Adapter ACE Module maintains a collection ofDevice Handler objects (one per-PBX) that are respon-sible for parsing and transforming incoming device eventsinto a canonical PBX-independent message object built atopa flexible message management class described in [6].

After being initialized, incoming canonical message ob-jects are passed to the read-side of theEvent AnalyzerACEModule . This Module implements the application-specific functionality for the server. An internal addressingtable maintained within theEvent Analyzer is used todetermine which client(s) should receive each message ob-ject. After theEvent Analyzer determines the properdestination(s), the message object is forwarded to the read-side of theMulticast Router ACE Module .

The Multicast Router ACE Module is a reusablecomponent that shields the rest of the application-specificserver code from knowledge of the client/server interactionsand from the particular choice of communication protocols.Clients subscribe to receive events published by the server byestablishing a connection with theMulticast RouterACEModule . The write-side of theMulticast RouterACEModule accepts connection requests from clients andcreates a separateClient Handler object to manageeach client connection. ThisClient Handler object han-dles all subsequent data transfer and control operations be-tween the server and its associated client. Once a clienthas connected with the server, it indicates the type of PBXevent(s) it is interested in monitoring. From that point, whenthe read-side of theMulticast Router receives a mes-sage object from theEvent Analyzer , it automaticallymulticasts the message to all clients that have subscribed toreceive the particular type of event encapsulated in the mes-sage object.

The ACEService Config object is used by theserver to control the initialization and termination ofStreamACEModule components that are configured stat-ically at installation-time or dynamically during run-time.The ACEService Config object contains an instanceof the ACEReactor event demultiplexor that is used

: Reactor

: ServiceRepository

: ServiceConfig

CCMStream

: EventFilter

ASX RUN-TIME: SwitchHandler

: SwitchHandler

: SessionHandler

: SessionHandler

: SessionHandler

TELECOMSWITCHES

SUPER-VISOR

EVENTSERVER

SUPER-VISOR

SUPER-VISOR

: SessionRouter

: SwitchAdapter

: EventAnalyzer

upstream

downstream

Figure 17:ASXComponents for the PBX Application

to dispatch incoming client messages to the appropri-ate Client Handler or Device Handler event han-dler. Control messages arriving from clients are sentdown the write-side of the Stream, starting with theMulticast Router and continuing through the inter-connected write-side of the StreamACEModule s to theDevice Adapter , which sends them to the appropriatePBX device. Likewise, theReactor detects incomingevents from PBX devices and dispatches them up the Streamstarting at theDevice Adapter ACE Module .

4.3 Server Configuration

The ACEModule s that comprise the PBX server may beconfigured into the server at any time. TheASX frameworkprovides this degree of flexibility via the use of explicit dy-namic linking driven by thesvc.conf configuration script.The following configuration script indicates which servicesare to be dynamically linked into the address space of theserver:

stream Server_Stream dynamicSTREAM * /svcs/Server_Stream.so : _alloc()

{dynamic Device_Adapter

Module * /svcs/DA.so:_alloc() "-p 2001"dynamic Event_Analyzer

Module * /svcs/EA.so:_alloc()dynamic Multicast_Router

Module * /svcs/MR.so:_alloc() "-p 2010"}

22

This configuration script indicates the order in which thehierarchically-related services are dynamically linked andpushed onto theServer Stream . During application ini-tialization, theService Config class parses this config-uration script and carries out the directives that describe eachentry.

The Server Stream iscomposed of threeACEModule s (theDevice Adapter ,the Event Analyzer , and theMulticast Router )that are dynamically configured into the server. The indi-cated shared object file is linked dynamically into the server(as specified by thedynamic directive). An instance ofModule object is then extracted from the shared object li-brary by calling the alloc function. As described be-low, theseModule s may be subsequently updated and re-linked if necessary (e.g., to install an updated version of aACEModule ) without completely terminating the execut-ing PBX server.

4.4 Server Reconfiguration

There are a number of drawbacks associated with staticallyconfiguring services into a communication software applica-tion. For example, performance bottlenecks may result if toomany services are configured into the server-side an applica-tion and too many active clients simultaneously access theseservices. Conversely, configuring too many services into theclient-side may also result in bottlenecks since clients oftenexecute on less powerful end-systems. In general, it is diffi-cult to determine the appropriate partitioning of applicationservicesa priori since processing characteristics and work-loads may vary over time. Therefore, a primary objective ofthe ASX framework was to develop object-oriented serviceconfiguration mechanisms that allow developers to defer un-til very late in the development cycle (i.e., installation-timeor run-time) decisions regarding which services ran in theclient-side and which ran in the server-side.

To facilitate flexible reconfiguration, the run-time controlenvironment provided by theASX framework enables de-velopers to alter their application service configurations ei-therstaticallyat installation-time ordynamicallyduring run-time. This is useful since different OS/hardware platformsand different network characteristics often require differentservice configurations. For example, in some configurationsthe server performs most of the work, whereas in others theclients do more of the work. Moreover, different end-systemconfigurations may be appropriate under different circum-stances (such as whether multi-processor server platforms orhigh-speed networks are available). Figure 18 illustrates howthe configuration shown in Figure 17 may be altered to oper-ate efficiently in a distributed environment where the serverprocessing constitutes the primary bottleneck.

This reconfiguration process is accomplished via the fol-lowing script:

suspend Server_Streamstream Server_Stream{

TELECOMSWITCHES

: EventFilter

: SessionHandler

: SessionHandler

: SessionHandler CCM

Stream: SessionRouter

: SwitchAdapter

ASX RUN-TIME

: Reactor

: ServiceRepository

: ServiceConfig

SUPER-VISOR

SUPER-VISOR

: SwitchHandler

: SwitchHandler

: EventAnalyzer

upstream

downstream

: EventFilter: Event

Filter

SUPER-VISOR

EVENTSERVER

Figure 18: Reconfiguration of PBX Monitoring System

remove Event_Analyzer}remote "-h all -p 911"{

stream Server_Stream{

dynamic Event_AnalyzerModule * /svcs/EA.so : _alloc()

}}resume Server_Stream

This new script migrates processing functionality fromthe server to the clients by dynamically unlinkingACEModule s from the server’s Stream and dynamicallylinking them into each client’s Stream [20].

TheASXframework replaced a previous architecture thatused ad hoc techniques (such as parameter passing andshared memory) to exchange messages between related ser-vices in the server. In contrast to the previous approach, thehighly uniformACEModule inter-connection mechanismsprovided by theASXframework greatly simplify portabilityand configurability.

5 Concluding Remarks

The ADAPTIVE Communication Environment (ACE) is atoolkit containing OO components that help reduce dis-tributed software complexity by reifying successful designpatterns and software architectures. ACE consolidates com-mon communication-related activities (such as local and re-

23

mote IPC [4], event demultiplexing and service handler dis-patching [14], service initialization [16, 17] configurationmechanisms for distributed applications containing mono-lithic and layered services [20], distributed logging [13], andintra- and inter-service concurrency [46]) into reusable OOcomponents and frameworks.

ACE is freely available via the World Wide Web at URLwww.cs.wustl.edu/˜schmidt/ACE.html . Thisdistribution contains the source code, documentation, andexample test drivers developed at Washington University, St.Louis. ACE is currently being used in communication soft-ware at many companies including Bellcore, Siemens, DEC,Motorola, Ericsson, Kodak, and McDonnell Douglas. ACEhas been ported to Win32 (i.e., WinNT and Win95), mostversions of UNIX (e.g., SunOS 4.x and 5.x, SGI IRIX, HP-UX, OSF/1, AIX, Linux, and SCO), and POSIX systems(such as VxWorks and MVS OpenEdition). There are bothC++ [6] and Java [48] versions of ACE available.

References[1] F. P. Brooks, The Mythical Man-Month. Reading, MA:

Addison-Wesley, 1975.

[2] S. J. Leffler, M. McKusick, M. Karels, and J. Quarterman,TheDesign and Implementation of the 4.3BSD UNIX OperatingSystem. Addison-Wesley, 1989.

[3] S. Rago,UNIX System V Network Programming. Reading,MA: Addison-Wesley, 1993.

[4] D. C. Schmidt, T. H. Harrison, and E. Al-Shaer, “Object-Oriented Components for High-speed Network Program-ming,” in Proceedings of the1st Conference on Object-Oriented Technologies and Systems, (Monterey, CA),USENIX, June 1995.

[5] G. Booch, Object Oriented Analysis and Design with Ap-plications (2nd Edition). Redwood City, California: Ben-jamin/Cummings, 1994.

[6] D. C. Schmidt, “ACE: an Object-Oriented Framework forDeveloping Distributed Applications,” inProceedings of the6th USENIX C++ Technical Conference, (Cambridge, Mas-

sachusetts), USENIX Association, April 1994.

[7] D. E. Comer and D. L. Stevens,Internetworking with TCP/IPVol III: Client – Server Programming and Applications. En-glewood Cliffs, NJ: Prentice Hall, 1992.

[8] W. R. Stevens,UNIX Network Programming, First Edition.Englewood Cliffs, NJ: Prentice Hall, 1990.

[9] E. Gamma, R. Helm, R. Johnson, and J. Vlissides,Design Pat-terns: Elements of Reusable Object-Oriented Software. Read-ing, MA: Addison-Wesley, 1995.

[10] D. C. Schmidt, “Reactor: An Object Behavioral Pattern forConcurrent Event Demultiplexing and Event Handler Dis-patching,” in Pattern Languages of Program Design(J. O.Coplien and D. C. Schmidt, eds.), pp. 529–545, Reading, MA:Addison-Wesley, 1995.

[11] R. G. Lavender and D. C. Schmidt, “Active Object: an ObjectBehavioral Pattern for Concurrent Programming,” inPatternLanguages of Program Design(J. O. Coplien, J. Vlissides,and N. Kerth, eds.), Reading, MA: Addison-Wesley, 1996.

[12] R. Johnson and B. Foote, “Designing Reusable Classes,”Journal of Object-Oriented Programming, vol. 1, pp. 22–35,June/July 1988.

[13] D. C. Schmidt, “The Reactor: An Object-Oriented Interfacefor Event-Driven UNIX I/O Multiplexing (Part 1 of 2),”C++Report, vol. 5, February 1993.

[14] D. C. Schmidt, “The Object-Oriented Design and Implemen-tation of the Reactor: A C++ Wrapper for UNIX I/O Multi-plexing (Part 2 of 2),”C++ Report, vol. 5, September 1993.

[15] I. Pyarali, T. H. Harrison, and D. C. Schmidt, “AsynchronousCompletion Token: an Object Behavioral Pattern for Effi-cient Asynchronous Event Handling,” inThe3rd Annual Con-ference on the Pattern Languages of Programs (WashingtonUniversity technical report #WUCS-97-07), (Monticello, Illi-nois), pp. 1–7, February 1997.

[16] D. C. Schmidt, “Design Patterns for Initializing Network Ser-vices: Introducing the Acceptor and Connector Patterns,”C++ Report, vol. 7, November/December 1995.

[17] D. C. Schmidt, “Connector: a Design Pattern for ActivelyInitializing Network Services,”C++ Report, vol. 8, January1996.

[18] D. C. Schmidt, “Acceptor and Connector: Design Patterns forInitializing Communication Services,” inThe1

st EuropeanPattern Languages of Programming Conference (WashingtonUniversity technical report #WUCS-97-07), July 1997.

[19] D. C. Schmidt, “IPCSAP: An Object-Oriented Interface toInterprocess Communication Services,”C++ Report, vol. 4,November/December 1992.

[20] D. C. Schmidt and T. Suda, “An Object-Oriented Frameworkfor Dynamically Configuring Extensible Distributed Commu-nication Systems,”IEE/BCS Distributed Systems EngineeringJournal (Special Issue on Configurable Distributed Systems),vol. 2, pp. 280–293, December 1994.

[21] P. Jain and D. C. Schmidt, “Service Configurator: A Pat-tern for Dynamic Configuration and Reconfiguration of Com-munication Services,” inThe3rd Pattern Languages of Pro-gramming Conference (Washington University technical re-port #WUCS-97-07), February 1997.

[22] D. C. Schmidt, “An OO Encapsulation of Lightweight OSConcurrency Mechanisms in the ACE Toolkit,” Tech. Rep.WUCS-95-31, Washington University, St. Louis, September1995.

[23] D. C. Schmidt and C. D. Cranor, “Half-Sync/Half-Async: anArchitectural Pattern for Efficient and Well-structured Con-current I/O,” inPattern Languages of Program Design(J. O.Coplien, J. Vlissides, and N. Kerth, eds.), Reading, MA:Addison-Wesley, 1996.

[24] D. C. Schmidt and T. Harrison, “Double-Checked Locking– An Object Behavioral Pattern for Initializing and Access-ing Thread-safe Objects Efficiently,” inThe3rd Pattern Lan-guages of Programming Conference (Washington Universitytechnical report #WUCS-97-07), February 1997.

[25] Bjarne Stroustrup and Margret Ellis,The Annotated C++ Ref-erence Manual. Addison-Wesley, 1990.

[26] I. Pyarali, T. H. Harrison, and D. C. Schmidt, “Design andPerformance of an Object-Oriented Framework for High-Performance Electronic Medical Imaging,”USENIX Comput-ing Systems, vol. 9, November/December 1996.

[27] R. Davis, Win32 Network Programming. Reading, MA:Addison-Wesley, 1996.

[28] D. L. Presotto and D. M. Ritchie, “Interprocess Communica-tion in the Ninth Edition UNIX System,”UNIX Research Sys-tem Papers, Tenth Edition, vol. 2, no. 8, pp. 523–530, 1990.

24

[29] W. R. Stevens,Advanced Programming in the UNIX Environ-ment. Reading, Massachusetts: Addison Wesley, 1992.

[30] R. Gingell, J. Moran, and W. Shannon, “Virtual Memory Ar-chitecture in SunOS,” inProceedings of the Summer 1987USENIX Technical Conference, (Phoenix, Arizona), 1987.

[31] J. Eykholt, S. Kleiman, S. Barton, R. Faulkner, A. Shivalin-giah, M. Smith, D. Stein, J. Voll, M. Weeks, and D. Williams,“Beyond Multiprocessing... Multithreading the SunOS Ker-nel,” in Proceedings of the Summer USENIX Conference, (SanAntonio, Texas), June 1992.

[32] IEEE, Threads Extension for Portable Operating Systems(Draft 10), February 1996.

[33] D. C. Schmidt, “Transparently Parameterizing Synchroniza-tion Mechanisms into a Concurrent Distributed Application,”C++ Report, vol. 6, July/August 1994.

[34] G. Booch and M. Vilot, “Simplifying the Booch Compo-nents,”C++ Report, vol. 5, June 1993.

[35] R. Gingell, M. Lee, X. Dang, and M. Weeks, “SharedLibraries in SunOS,” inProceedings of the Summer 1987USENIX Technical Conference, (Phoenix, Arizona), 1987.

[36] W. W. Ho and R. Olsson, “An Approach to Genuine Dy-namic Linking,” Software: Practice and Experience, vol. 21,pp. 375–390, Apr. 1991.

[37] H. Custer,Inside Windows NT. Redmond, Washington: Mi-crosoft Press, 1993.

[38] D. C. Schmidt and P. Stephenson, “Experiences Using DesignPatterns to Evolve System Software Across Diverse OS Plat-forms,” in Proceedings of the9th European Conference onObject-Oriented Programming, (Aarhus, Denmark), ACM,August 1995.

[39] D. Ritchie, “A Stream Input–Output System,”AT&T BellLabs Technical Journal, vol. 63, pp. 311–324, Oct. 1984.

[40] N. C. Hutchinson and L. L. Peterson, “Thex-kernel: An Ar-chitecture for Implementing Network Protocols,”IEEE Trans-actions on Software Engineering, vol. 17, pp. 64–76, January1991.

[41] J. M. Zweig, “The Conduit: a Communication Abstraction inC++,” in Proceedings of the2nd USENIX C++ Conference,pp. 191–203, USENIX Association, April 1990.

[42] D. C. Schmidt and T. Suda, “Transport System ArchitectureServices for High-Performance Communications Systems,”IEEE Journal on Selected Areas in Communication, vol. 11,pp. 489–506, May 1993.

[43] D. C. Schmidt, B. Stiller, T. Suda, A. Tantawy, and M. Zit-terbart, “Language Support for Flexible, Application-TailoredProtocol Configuration,” inProceedings of the18th Con-ference on Local Computer Networks, (Minneapolis, Min-nesota), pp. 369–378, IEEE, Sept. 1993.

[44] A. McRae, “Hardware Profiling of Kernels,” inUSENIX Win-ter Conference, (San Diego, CA), USENIX Association, Jan.1993.

[45] S. Saxena, J. K. Peacock, F. Yang, V. Verma, and M. Krish-nan, “Pitfalls in Multithreading SVR4 STREAMS and otherWeightless Processes,” inProceedings of the Winter USENIXConference, (San Diego, CA), pp. 85–106, Jan. 1993.

[46] D. C. Schmidt, D. F. Box, and T. Suda, “ADAPTIVE: A Dy-namically Assembled Protocol Transformation, Integration,and eValuation Environment,”Journal of Concurrency: Prac-tice and Experience, vol. 5, pp. 269–286, June 1993.

[47] D. C. Schmidt, “A Family of Design Patterns for Application-level Gateways,”The Theory and Practice of Object Systems(Special Issue on Patterns and Pattern Languages), vol. 2,no. 1, 1996.

[48] P. Jain and D. Schmidt, “Experiences Converting a C++Communication Software Framework to Java,”C++ Report,vol. 9, January 1997.

25