Practical Protocol Composition, Encapsulation and Sharing in Kompics∗

Cosmin Arad†

†Royal Institute of Technology (KTH)

{icarad,haridi}@kth.se

Seif Haridi†,‡

‡Swedish Institute of Computer Science (SICS)

seif@sics.se

Abstract

At the core of any distributed system is a set of concur-

rent distributed algorithms that coordinate the functionality

of the distributed system. We present a software architec-

ture, Kompics that is component-based and compositional

which facilitates building distributed protocols. The under-

lying computation model subsumes that of event-based sys-

tems, SEDA (staged event-driven architecture) and thread-

based models. We illustrate various salient features of Kom-

pics such as ease of use, compositionality and configurabil-

ity through a series of well chosen distributed protocols.

1. Introduction

Programming distributed systems, applications and ser-

vices is difficult due to concurrency and partial failures,

which are fundamental characteristics of distributed sys-

tems, but also due to the complexity of the software archi-

tecture of any non-trivial distributed system. One way to

simplify the understanding and programming of complex

distributed protocols is to split them into their orthogonal

aspects and encapsulate each of these aspects in reusable

abstractions [6]. Combining multiple interdependent dis-

tributed protocols that are concurrently working together

gives rise to complex interactions.

We introduce Kompics, a novel software framework for

programming, configuring, and executing distributed proto-

cols as software components that interact asynchronously

by passing data-carrying events. Kompics components are

reactive, concurrent, and they can be composed into com-

plex architectures of composite components that are safely

reconfigurable at runtime and allow for sharing of common

subcomponents at various levels in the component hierar-

chy. Software faults occurring in components are isolated

and handled by supervisor components organized in flexi-

ble supervisor hierarchies [2].

∗This work was funded by the European Union in the SELFMAN

project (contract 34084). We would like to thank Roberto Roverso for

his contribution to the development of earlier versions of Kompics.

The component programming style encompasses both

the event-based and the thread-based styles to facilitate

programming most distributed protocols, and the execu-

tion model allows a flexible allocation of components to

threads which enables various resource allocation policies,

and readily takes advantage of multi-core hardware archi-

tectures.

The Kompics component model targets the development

of reliable and adaptable long-lived, dynamic, and self-

managing distributed systems. Such systems are com-

posed of many software modules which implement vari-

ous distributed protocols (e.g. failure detectors, reliable

group communication, agreement protocols, gossip proto-

cols, etc.) and interact in complex ways. Kompics aims to

facilitate the construction of complex distributed systems by

providing a computation model that accommodates their re-

active nature and makes their programming as easy as pos-

sible.

The rest of this paper is structured as follows: in the next

section we present the Kompics component model and some

example component architectures. In Section 3 we describe

the semantics of component interaction and execution in

Kompics. In Section 4 we give a qualitative evaluation of

the model. In Section 5 we survey related work. We discuss

future work and we conclude in Section 6.

2. Component model

In Kompics, distributed abstractions are encapsulated

into components that can be composed into hierarchical ar-

chitectures of composite components. Subcomponents can

be safely shared by multiple components at any level in

the component hierarchy. Kompics components interact by

passing asynchronous data-carrying events and they are de-

coupled by a flexible event publish-subscribe system.

Components A component is a unit of functionality and

management. Components are active entities that inter-

act with each other by triggering (sending) and handling

(receiving) events. Components react to events by ex-

ecuting event specific procedures to handle the received

2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops

978-0-7695-3553-1/08 $25.00 © 2008 IEEE

DOI 10.1109/SASOW.2008.67

266

events. Components are decoupled (by channels) which

makes them independent and reusable.

Every component contains some internal state and a set

of event-handling procedures. Composite components also

contain subcomponents and thus form a component hier-

archy. We sometimes call subcomponents children compo-

nents and the containing composite component parent com-

ponent. We say that the parent component is at a higher

level in the component hierarchy than its children compo-

nents.

Events Events are passive objects that contain a set of im-

mutable attributes. Events are typed and they can form type

hierarchies. Components subscribe for events to channels

and publish events into channels.

Channels Channels are interaction links between compo-

nents. They carry events from publisher components to sub-

scriber components. Every channel is parameterized with a

set of event types that can be subscribed for or published

into the channel. Channels exist in the context of the com-

posite components which create them. However, references

to channels can be passed between components through

events.

Event handlers An event handler is an event-specific pro-

cedure that a component executes as a reaction to a re-

ceived event. An event handler takes as argument one event

of a certain type. While being executed, event handlers

may trigger new events. Event handlers can be guarded by

boolean guards.

Event subscriptions Components subscribe their event

handlers to channels by registering event subscriptions at

the respective channels. Event subscriptions can be made

either by event type or by both event type and event at-

tributes, whereby a subscription contains a set of (attribute,

value) attribute filters. Events published into a channel are

delivered to all subscriber components which registered at

the channel subscriptions that match the published events.

Components can publish or subscribe for subtypes of the

event types carried by the channel.

Component types A component interacts with its envi-

ronment (other components) by triggering (output) and han-

dling (input) events. The component is subscribed for the

input events to input channels and publishes output events

into output channels. Kompics components are parameter-

ized by their input and output channels. The types of input

and output events of a component together with the input

and output channel parameters that carry them represent the

component’s type. Component types are useful for express-

ing a composite component’s dependencies on subcompo-

nents in terms of what they do and not on how they do it.

Component membranes A component membrane is the

runtime correspondent of the component’s type. The com-

ponent membrane is a set of references to the actual chan-

nels that the component is using as input and output chan-

nel parameters. The membrane maps every pair (event type,

in/out direction) to an actual channel reference.

Component sharing A component is shared between

multiple composite components essentially by sharing the

channels in its membrane. To share one of its subcompo-

nents, a composite component registers the subcomponent’s

membrane under a name, in a registry of shared compo-

nents. Other composite components can retrieve the mem-

brane (by name) from the registry and use its channels to

communicate with the subcomponent. This registry is hi-

erarchical in the following sense: (1) names registered at

some level in the component hierarchy (the level of the par-

ent component of the shared component) are not visible at

higher levels, and (2) names registered at a lower level in the

component hierarchy shadow the same names registered at

a higher level.

2.1. Examples

We illustrate some of the concepts in Kompics by exam-

ples from Guerraoui et al. [6]. Figure 1 shows an example

graphical representation of Kompics components. Here we

have two components: A and B. They communicate through

channel x which carries events of type E1. Component A

has an output channel formal parameter and component B

has an input channel formal parameter. Both A and B are

parameterized by channel x. A uses x for its output channel

formal parameter and B uses x for its input channel formal

parameter. Component B has an event handler that is sub-

scribed to B’s input channel and handles events of type E1.

Component A has an event handler that publishes events of

type E1 in A’s output channel. Both components A and B

and channel x exist in the context of a parent component,

Node.

���

� �

����

��

���� �

Figure 1. Graphical representation of Kom-

pics components.

267

Figure 2 shows an example graphical representation of

a Kompics composite component, namely a Best-Effort

Broadcast (BEB) [6] component that contains a Perfect

Point-to-point Links (PP2P) [6] subcomponent. The BEB

component is parameterized by an input channel carry-

ing BebBroadcast events and an output channel carrying

BebDeliver events. The PP2P component is parameter-

ized by an input channel carrying Pp2pSend events and an

output channel carrying Pp2pDeliver events. The BEB

component contains two local channels that are used as

the actual channels that parameterize the PP2P subcompo-

nent and two event handlers that handle BebBroadcast and

Pp2pDeliver events respectively, and trigger Pp2pSend

and BebDeliver events respectively. The arrows indicate

event subscriptions and publications.

�����

���

�� ����� � ��

� ��������

������

���������������������������

������ ���������

Figure 2. Graphical representation of a Kom-

pics composite component.

Figure 3 shows an example of two composite compo-

nents sharing a common subcomponent. A Perfect Point-to-

point Links (PP2P) [6] component and a Fair-Loss Point-

to-point Links (FLP2P) [6] component share a Network

subcomponent. The shared component as well as the chan-

nels in its membrane are represented with double borders.

The Network component accepts NetSend events (in the

sender process) and triggers NetDeliver events (in the re-

ceiver process). The NetDeliver event that is to be deliv-

ered in the receiver process is encapsulated in a NetSend

event on the sender process.

NetSend events triggered by PP2P in the sender pro-

cess are to result in NetDeliver events handled only by

PP2P in the receiver process. Correspondingly, NetSend

events triggered by FLP2P in the sender process are to

result in NetDeliver events handled only by FLP2P in

the receiver process. Filtering NetDeliver events between

PP2P and FLP2P is done by event subtyping, i.e., PP2P

and FLP2P subscribe to the Network component’s output

channels for different subtypes of the NetDeliver event

type.

We refer to this pattern of sharing a subcomponent be-

tween parent components of different types, whereby the

filtering of events delivered to the parent components is

done by event subtyping, as static sharing. Another pat-

tern of sharing manifests when a subcomponent is shared

��������

������

���

�������

������ ������

� ��� ��

���������

� �������

����

�������

������ ������

��� ����� ��

�� ���� ���� ���� ����

�!�����"��"�!����"��# �����"#�#�����

�!����"��# �����

Figure 3. Two different composite compo-

nents sharing a subcomponent.

among multiple parent components of the same type. In this

case, the filtering of events is done by event attributes. The

parent components subscribe for the output events of the

shared component by event attributes with distinguishing

values. We call this sharing pattern, instance sharing. An

example of instance sharing is illustrated in Figure 4. Here

we have multiple Virtual Node (VN) components running

on the same physical node and sharing a Network compo-

nent. Each VN component has subscribed for NetDeliver

events with a destination address attribute equal to its own

address. Consequently, each VN component receives only

the NetDeliver events that it is interested in.

$�%

�������

������ ������

$�

�������

������ ������

$��

�������

������ ������

Figure 4. Multiple similar composite compo-

nents sharing a subcomponent.

3. Component execution and interaction se-

mantics

Event handlers are executed on behalf of components by

worker threads from a worker pool. Components that have

received events are scheduled for execution to one of the

worker threads. Components that embed third-party soft-

ware or use blocking services may use extra threads. We

call these threaded components.

268

Concurrent component execution The event handlers of

the same component instance are guaranteed to be executed

sequentially, but different component instances can exe-

cute event handlers concurrently (or in parallel on multi-

core machines). In other words, the event handlers of the

same component instance are mutually exclusive, while the

event handlers of different component instances are not.

However, the execution of an event handler is not atomic

(in the all-or-nothing sense). That means that events trig-

gered by one event handler are visible to the corresponding

subscriber components (thus, executable) immediately after

they are triggered. This entails that the execution of an event

handler is not failure atomic, i.e., it can fail before comple-

tion with observable partial side effects (some of the events

that are supposed to be triggered by the handler are indeed

triggered while others are not). The motivation for this de-

sign decision is twofold: first, making events available for

execution as soon as they are triggered increases the poten-

tial parallelism in the system. Second, it provides a uniform

observable behavior for threaded and non-threaded compo-

nents.

Event subscription Components subscribe their event

handlers to input channels for a particular event type. When

component a subscribes to channel x an event handler h

for events of type T with attributes matching filters(T ),a subscription of the form (a, T , h, filters(T )) is regis-

tered at channel x. At the same time, a FIFO work queue,

qx, is created at a and is associated with channel x (if qx is

not already existing from a previous subscription of a to x).

Thus, a channel y has an associated work queue qy in every

component that has subscribed to y. A component can sub-

scribe multiple event handlers to the same channel and can

subscribe the same event handler to multiple channels.

Component scheduling A component a can be in exclu-

sively one of the following three scheduling states: BUSY,

READY, or IDLE. We say that a is BUSY if one of the

worker threads is being actively executing one of a’s event

handlers. We say that a is READY if it is not BUSY and at

least one of its work queues qx is not empty, so a is ready to

execute some event. We say that a is IDLE if it is not BUSY

and all its work queues qx are empty, so a has no event to

execute.

Event publication While executing event handlers, com-

ponents may publish events into output channels. As-

sume component a triggers event e of type T in channel

x. Let S be the subset of all subscriptions of the form

(b, T ′, h, filters(T ′)), to channel x, where T ′ is either T

or a super-type of T and e matches filters(T ′). For each

subscription (b, T ′, h, filters(T ′)) in S, a work item of the

form (e, h) is enqueued at b in work queue qx and if b was

IDLE then b becomes READY.

Channel FIFO guarantees The execution model guaran-

tees the following FIFO semantics for channels. Each com-

ponent a subscribed to a channel x, receives events pub-

lished in x, in the same order in which they are published.

Events triggered sequentially by one component instance

will be published in the channel in the order in which they

were triggered. A channel serializes the concurrent publi-

cation of events into the channel, i.e., events that are pub-

lished concurrently in the same channel by different com-

ponent instances. This means that all subscribers to channel

x for event type T observe the same order of publications

of events of type T in their local work queues qx.

Event handler execution Worker threads execute event

handlers on behalf of components. Worker threads atom-

ically pick READY components and make them BUSY.

When a worker picks READY component a, it immediately

makes a BUSY. An invariant of the execution model is that

at this point a has at least one work queue qx that is not

empty. After making a BUSY, the worker dequeues one

work item (e, h) from some work queue qx selected ac-

cording to some fairness criteria. Thereafter, the worker

proceeds to execute a’s event handler h by passing it as an

argument the event e. Upon completing the execution of h,

if all a’s work queues qx are empty, then the worker makes

a IDLE. Otherwise it makes a READY.

Worker threads loop Worker threads wait for compo-

nents to become READY. When a component a becomes

READY, a worker w picks it and executes one work item

(e, h), the head of some work queue qx of a. The execu-

tion of event handler h may trigger new events ei of types

Ti, published in channels xi. All components subscribed to

channels xi for event types Ti become READY if they were

not BUSY. Upon completing the execution of event handler

h, worker w picks another READY component, if one exists,

and it repeats the above steps. If no component is READY,

worker w starts waiting for a component to become READY

and it repeats the above steps.

4. Evaluation

We implemented Kompics in Java. The component

framework library, source code and documentation is pub-

licly available at http://kompics.sics.se. In this

section we present two case studies of event-based hierar-

chical component architectures that we implemented with

Kompics.

269

The example in Figure 5 shows composition, encapsu-

lation and sharing of protocols in Kompics. For example

the Abortable Consensus [6] component makes use of Best-

Effort Broadcast and Perfect Links and the Consensus In-

stance uses the Leader Detector. The example also shows

how different components can be used to separate the imple-

mentation of functional and non-functional aspects of an ab-

straction. The Consensus Instance implements a Paxos uni-

form consensus algorithm. The Consensus Port offers to an

application component a sequence of consensus instances

while garbage collecting the already decided instances. The

Consensus Service component allocates Consensus Ports to

different applications.

Network

Lossy Point-to-point Links Perfect Point-to-point Links

Failure DetectorBest-Effort Br.

Consensus Service

Boot

Leader DetectorUnreliable Br.

Consensus Port

Application

Abortable Consensus

Consensus Instance

Reliable Broadcast

Group Membership

Virtual Synchrony

Distributed Shared Memory

Figure 5. Static membership distributed ab-

stractions system architecture.

In Figure 2 we showed a graphical representation of a

Best-Effort Broadcast (BEB)[6] component. In Figure 6

we give the pseudo-code, from Guerraoui et al. [6], and

our corresponding Java code for the BebBroadcast event

handler of the BEB component. One can observe that Kom-

pics component definitions naturally match the pseudo-code

specifications which become straight forward to implement.

The Java code corresponds to a static sharing implemen-

tation, which assumes that the BEB component shares

the PP2P component with other components and is itself

shared by multiple parent components. Thus, BEB has sub-

scribed for BebPp2pDeliver (a subtype of Pp2pDeliver)

events and both Pp2pSend and BebBroadcast events en-

capsulate Pp2pDeliver and BebDeliver events respec-

tively. The pseudo-code assumes no sharing of protocols,

hence the slight difference.

The example in Figure 7 shows a peer-to-peer system ar-

chitecture that supports multiple virtual peers in one node.

This is an example of hierarchical sharing where we want

to share the Perfect Network abstraction among the pro-

tocols of one virtual peer, but have different Perfect Net-

works in different virtual peers. On the other hand, the Net-

upon event bebBroadcast | m dofor all pi do trigger pp2pSend | pi, m ; end for

end event��������� �������������� ���������� ����������

� ����� ��� ������������������ ����������������������������������������������������

�������������������������������������� �������� ���������

���� ���������������!��� ������������"����������������� ���! ���� ���#��������

$$

Figure 6. Pseudo-code and Java code for an

event handler of a BEB component.

work component is shared and used by all Perfect Network

abstractions in an instance sharing pattern. Every Peer

component contains subcomponents implementing various

protocols like failure detectors, structured and unstructured

overlays, gossip protocols, group communication, agree-

ment protocols, transactional storage, etc. The advantage of

this architecture is that it allows the execution of the same

protocol implementations both in a simulation scenario1 or

in a real deployment. We have executed up to 128 virtual

peers implementing the Chord [10] overlay in one address

space, where the Perfect Network and Lossy Network

components in each peer implemented network emulation

by delaying sent messages in the sender peers with laten-

cies extracted from the King [7] matrix, based on the source

and destination peer addresses. Exactly the same architec-

ture can be executed in multiple address spaces to scale the

network size. The Network component provides communi-

cation between the different address spaces.

Boot P2P Application (Simulator)

TimerWeb Server (Jetty)Network (Apache MINA)

Lossy Network

Peer Cluster / Peer Set

Perfect Network Web Handler

Failure Detector Peer MonitorBootstrap Client

Cyclon

Aggregation / Slicing

T-Man / T-Chord

Gradient / Gnutella

Structured Overlay

Symmetric replication

Bulk Operations Peer Application

Transactional DHT

Fast Paxos

Group MulticastRing Router

Figure 7. Kompics peer-to-peer system archi-

tecture.

1by replacing the Application component with a simulator for user ac-

tions and peer dynamism and replacing the Network component with a

network simulator.

270

5. Related work

We can position our work with respect to component

models, protocol composition frameworks and software ar-

chitectures.

The Fractal [5] component model allows the specifica-

tion of components that are reflective, hierarchical, and dy-

namically reconfigurable. However, the Fractal model is ag-

nostic with respect to the execution model of components.

Kompics is a reactive component model that has these desir-

able properties of Fractal but it enforces a particular execu-

tion and component interaction model, that facilitates pro-

gramming of distributed protocols.

Protocol composition frameworks like Appia [8], Cac-

tus [4] or Coyote [3] allow the building of protocol stacks by

composing smaller building blocks called protocol modules

that interact through events. However, these systems focus

on the flow of events though the protocol stack rather than

on the encapsulation and abstraction of low-level protocols.

As a result the degree of protocol composability offered by

these systems is limited2 which prevents the construction of

complex hierarchical architectures.

The most notable piece of related work on event-based

software architectures is SEDA (staged event-driven archi-

tecture) [11]. Here the focus is on performance, namely on

self-tuning resource management that adapts dynamically

to changes in load to provide graceful degradation in per-

formance, rather than on hierarchical architectures or dy-

namic reconfiguration. Kompics possesses these properties

due to the ability to allocate different worker pools for dif-

ferent groups of components. Additionally, Kompics offers

compositionality and encapsulation that are not present in

SEDA.

6. Concluding remarks and future work

We presented Kompics, a component model and frame-

work, that facilitates the construction of complex distributed

systems by structuring protocols as reactive components

and composing them in hierarchical architectures that are

reconfigurable at runtime. We have shown the advantages

of protocol composition with sharing in two real-world ex-

ample component architectures. We reported on our expe-

rience of using Kompics as a teaching tool which showed

that Kompics is very easy to use for prototyping and testing

distributed protocols.

Using Kompics we have developed a library of reusable

protocol components for building large-scale, decentral-

ized, dynamic distributed systems. We continue to add more

protocols to the library with the goal to provide a complete

middleware for peer-to-peer systems including: structured

2to a set of stacks in Appia and to a 2-level hierarchy in Cactus.

and unstructured overlay networks, agreement protocols,

scalable transactional storage, gossip protocols for aggrega-

tion, slicing, random peer sampling, topology maintenance,

information dissemination, etc. Further, we plan to extend

the Kompics model with distribution transparency, to work

on remote component deployment and distributed architec-

ture reconfiguration, followed by transactional reconfigura-

tion.

We propose Kompics as a tool for teaching and research

in distributed algorithms and as a framework for construct-

ing and executing real-world complex distributed systems.

References

[1] The Kompics component framework.

http://kompics.sics.se, 2008.[2] J. Armstrong. Making reliable distributed systems in the

presence of software errors. PhD thesis, Swedish Institute

of Computer Science (SICS), 2003.[3] N. T. Bhatti, M. A. Hiltunen, R. D. Schlichting, and W. Chiu.

Coyote: a system for constructing fine-grain configurable

communication services. ACM Transactions on Computer

Systems, 16(4):321–366, 1998.[4] N. T. Bhatti and R. D. Schlichting. A system for construct-

ing configurable high-level protocols. In SIGCOMM, pages

138–150, 1995.[5] E. Bruneton, T. Coupaye, M. Leclercq, V. Quema, and J.-

B. Stefani. The fractal component model and its support

in java: Experiences with auto-adaptive and reconfigurable

systems. Softw. Pract. Exper., 36(11-12):1257–1284, 2006.[6] R. Guerraoui and L. Rodrigues. Introduction to Reliable

Distributed Programming. Springer-Verlag New York, Inc.,

Secaucus, NJ, USA, 2006.[7] K. P. Gummadi, S. Saroiu, and S. D. Gribble. King: es-

timating latency between arbitrary internet end hosts. In

IMW ’02: Proceedings of the 2nd ACM SIGCOMM Work-

shop on Internet measurment, pages 5–18, New York, NY,

USA, 2002. ACM.[8] A. Pinto. Appia: A flexible protocol kernel supporting mul-

tiple coordinated channels. In ICDCS ’01: Proceedings of

the The 21st International Conference on Distributed Com-

puting Systems, page 707, Washington, DC, USA, 2001.

IEEE Computer Society.[9] O. Rutti, P. T. Wojciechowski, and A. Schiper. Service in-

terface: a new abstraction for implementing and composing

protocols. In SAC ’06: Proceedings of the 2006 ACM sym-

posium on Applied computing, pages 691–696, New York,

NY, USA, 2006. ACM.[10] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Bal-

akrishnan. Chord: A scalable peer-to-peer lookup service

for internet applications. In SIGCOMM ’01: Proceedings

of the 2001 conference on Applications, technologies, archi-

tectures, and protocols for computer communications, pages

149–160, New York, NY, USA, 2001. ACM.[11] M. Welsh, D. Culler, and E. Brewer. Seda: an architecture

for well-conditioned, scalable internet services. SIGOPS

Oper. Syst. Rev., 35(5):230–243, 2001.

271