rapid simulator development for multicast protocol analysis

SOFTWARE—PRACTICE AND EXPERIENCESoftw. Pract. Exper. 2002; 32:1–23 (DOI: 10.1002/spe.425)

Rapid simulator developmentfor multicast protocol analysis

Robert Simon∗,† and Paul Rabinovich

Department of Computer Science, George Mason University, Fairfax, VA 22030, U.S.A.

SUMMARY

When evaluating new protocols and algorithms it is often desirable to be able to rapidly build a specializedsimulator from scratch. The advantage of this approach is that software designers can avoid the often-steeplearning curve associated with using existing simulators, and that specialized simulators can have significantcomputational advantages over general-purpose programs. The drawback is that simulators can be complexto develop and debug. This paper describes our experiences developing SESAME, a Java-based simulatordesigned to analyze multicast protocol performance. By using an appropriate set of design patterns we wereable to rapidly develop a reusable and easy-to-implement framework for multicast algorithm and protocolanalysis. We developed several useful approaches for rapid simulation development, including a multicastvisualization tool and techniques for simulating concurrent processing within a computer network thatsubstantially simplified code development and maintenance. Owing to performance concerns, we also ranan extensive series of performance tests of different priority queue implementations. Our design approachand the lessons that we learned in developing SESAME can be applied to rapid simulation development ingeneral. Further, our performance results suggest that by using commercial off-the-shelf Java developmentenvironments it is possible to obtain sufficient performance for many time-sensitive applications such asdiscrete event simulation. Copyright 2001 John Wiley & Sons, Ltd.

KEY WORDS: simulator design; design patterns; secure multicast; priority queue performance analysis

1. INTRODUCTION

This paper reports on the design, implementation and performance evaluation of a multicast protocolsimulator using the Java programming language. The simulator is called SESAME, for SEquentialSimulAtor for Multicast message Exchange. The initial motivation for developing SESAME grewout of a desire to study the algorithmic message complexity of different multicast protocols andapplications. Since multicasting is a critical network technology for the next generation of distributedapplications, it is essential that protocol developers carefully evaluate and compare multicast protocol

∗Correspondence to: Robert Simon, Department of Computer Science, George Mason University, Fairfax, VA 22030, U.S.A.†E-mail: [email protected]

Copyright 2001 John Wiley & Sons, Ltd.Received 1 August 2000Revised 28 August 2001

Accepted 28 August 2001

2 R. SIMON AND P. RABINOVICH

performance and overhead. Owing to its inherent complexity, multicast routing performance measuresmust be obtained via simulation. For reasons described in Section 2, we decided to write our ownsimulator, rather than use one of a number of existing simulation toolkits or network simulationpackages. We required a simulator that could be developed extremely rapidly, could easily abstractout non-essential host- and network-level details without compromising the results of our experiments,and had good performance. To accomplish these goals we had to overcome several conceptual andimplementation hurdles as well as performance concerns, the later due to SESAME’s pure Javaimplementation.

SESAME’s software architecture provides a framework for studying multicast and unicast messagecomplexity. One of the advantages of the SESAME framework is reusability. The stable interfacesprovided by frameworks enhance the reusability of generic components that can be reapplied tocreate new applications. Framework reusability leverages the domain knowledge and prior effort ofexperienced developers in order to avoid recreating and revalidating common solutions to recurringapplication requirements and software design challenges. Frameworks represent part code reuse andpart design reuse. Design patterns [1] are the architectural elements of a framework.

An important characteristic of the SESAME framework is the inversion of simulator control. Atypical scenario for designing extensible simulators is for developers to reuse components from existingsimulation libraries by writing a main program calling these components whenever necessary. InSESAME, the main program is reused, and the network architect decides what is plugged into it.

A difficulty in using frameworks is that they can have steep development curves. One of the goalsof this paper is to explain the techniques we used for rapid framework development. For instance, wehave limited the number of classes that must be subclassed before they can be used, and minimizedthe number of methods that must be overridden. We believe that our approach can be used in otherdomains where rapid simulator development is desirable.

To (re)use SESAME, the developer reuses the main body and writes the specific multicast messagepassing protocols it calls. We used design patterns to facilitate and speed up SESAME’s development,and to make our framework design as general-purpose as possible. In subsequent sections we describehow we used the facade, the adapter, the factory and the state design patterns. Using these patternsenabled us to rapidly overcome several conceptual hurdles. For instance, to accomplish our goal ofdevelopment simplicity and for performance reasons, we did not use any concurrency mechanismssuch as threads. However, we still needed to simulate concurrent events in the network, such asmultiple routers simultaneously receiving different packets. Using the state design pattern we wereable to quickly develop a concurrency emulation package. Another example is our use of the facadepattern as a way of significantly simplifying the introduction of new protocols.

There has been concern about Java’s run-time performance. In particular, for performance sensitiveapplications such as SESAME, Java is often considered an unsuitable choice. As we developedSESAME we therefore decided to conduct a careful performance evaluation of our design andimplementation. We focused on the performance of our priority queue classes, because priority queueshave the greatest impact on overall simulator performance. Our purpose in conducting this performancestudy is threefold. First, we want to provide actual performance measurements of different priorityqueue implementations under a variety of usage patterns. Second, we want to compare our findingswith earlier (non-Java) performance studies of different priority queue implementations [2]. Finally,we want to explain some of the methodological issues involved in priority queue evaluation.

Copyright 2001 John Wiley & Sons, Ltd. Softw. Pract. Exper. 2002; 32:1–23

RAPID SIMULATOR DEVELOPMENT 3

We have used SESAME to analyze the implementation and message overhead for a multicastsecurity protocol called KHIP [3]. For dynamic groups, the overhead required by many types ofmulticast security protocols is of particular interest, because security parameters for the entire groupmust be completely updated after any member either joins or leaves a group. This analysis helped usvalidate an analytical model we had developed. It also allowed us to uncover a protocol race conditionthat could lead to a security violation.

SESAME’s performance and ease of use make it a powerful tool for studying a wide range ofmulticast and other network routing protocols. We believe that our framework and pattern-basedapproach to design and implementation provide meaningful insight for developers who need to rapidlydevelop a simulator. Further, our performance studies indicate that for a wide range of priority queuedata structures even current Java and PC technology provide surprisingly good performance for timesensitive applications.

2. BACKGROUND AND RELATED WORK

This section provides some necessary background in multicasting and multicast routing analysis. Wealso provide an overview of simulator design and performance analysis.

2.1. Multicasting

Multicasting is a network level technology that efficiently supports distributed multi-party applicationsby reducing the level of network traffic. Unlike unicast communication, which requires a separate sendoperation per receiver, multicasting enables senders to send a packet just once. Multicast routers thenautomatically forward the packet to each receiver, while simultaneously minimizing the number ofpacket duplicates in the network. This efficiency is achieved via the construction of multicast routingtrees. Two types of multicast routing methods can be distinguished. In the source-rooted tree approachthere is a shortest path tree rooted at each source [4,5]. Owing to the overhead of maintaining multipletrees, the source-rooted tree approach is suitable for groups with a small to medium number of senders.

The other approach for multicast routing is the shared-tree approach [6,7]. Here, a single treespans all receivers, while senders use one-to-one unidirectional paths to reach any node. An importantsubclass of shared-tree routing is known as core-based (CB) multicast [8,9]. This approach associates acenter or core node or nodes with each multicast group. Messages destined for the group are first routedto the core and then distributed to the other group members. This approach is more scalable then thesource-rooted tree approach. However, the performance of all CB multicast protocols is very sensitiveto the location and selection of the core, since all messages must traverse this node‡. The problemof picking a good center node is therefore of central importance to the widespread deployment ofmulticasting. Notice the problem of good center node selection is dependent upon a number of factors,including network topology, the pattern and quantity of sender/receiver data, and protocol traffic. Statedanother way, center point selection is a strongly coupled network layer and application layer problem.

‡There are adaptive multicast protocols such as Sparse/Dense mode Protocol Independent Multicast [6,7] that can address thisproblem by defining circumstances when the core node is bypassed. However, our interest here is in a very broad class ofcore-based protocols.



Our motivation in designing SESAME was to study a wide range of center point or core selectionheuristics for a wide class of network topologies and distributed applications. This paper describes, inSection 4, how we used SESAME to study core selection in multicast security protocols.

The objective of a multicast security mechanism is to provide for member authentication and dataintegrity and secrecy. By meeting this objective the communication architecture guarantees that onlylegitimate users can send packets to, or receive packets from, the group.

Enforcing security over the Internet requires data encryption at the packet level. For multicastapplications it is necessary to use group-key management protocols [10]. These are protocolsresponsible for ensuring that only currently legitimate users have access to the group keys necessaryfor data transmission and reception. Because multicast groups may be distributed over a large area,and because multicast group members may be continually joining and leaving the group, multicastgroup-key management techniques can be quite complex and difficult to implement effectively. Theefficiency of these protocols is judged by a range of factors, including scalability, the number of keysrequired by each group member, and the number of message exchanges required for each group join andleave. A wide variety of approaches have been suggested in recent years. One particularly promisingclass of solutions uses a hierarchical tree-based management approach [11]. These approaches addressscalability and message complexity by organizing group keys according to a logical hierarchy. Takingadvantage of this logical tree structure reduces the complexity of required key updates. Many of theselogical tree algorithms use a CB approach, so the problem of center selection is particularly important.

2.2. Simulator design and performance analysis

As explained in the previous section, multicast protocols must be carefully examined before they aredeployed. Owing to their intrinsic complexity, it may be very hard to evaluate them via purely analyticalmeans. One approach could be to develop a prototype and conduct empirical studies on it. Anotherpopular approach is to use one of a large number of simulation tools.

There are many existing network simulation packages. Two of the better known ones are OPNETand ns-2. The Optimized Network Engineering Tools (OPNET) is a commercial simulation productdeveloped by MIL3 [12]. It uses the discrete event simulation approach that allows a large numberof closely spaced events in a sizable network to be represented accurately and efficiently. In OPNETnetworks are built of components connected by perfect links that can be degraded at will. OPNETuses a layered architecture where each layer provides services to the layer above it. Each component’sbehavior is modeled as a state transition diagram. A program written in the C programming languagedescribes the process, taking place in each state.

ns-2 is a simulator developed under the auspices of the MASH Project at the University of California-Berkley and the VINT Project at the University of Southern California/Information Sciences Institute[13]. ns-2 emulates a WAN with high fidelity, using a system of workstations connected by a LAN.Protocols studied using ns-2 are programmed to run on a real network. The simulator includes modelsof point-to-point wide-area links so that the overall ns-2 system emulates an environment very similarto an actual WAN. The ns-2 environment is not subject to physical problems existing in real WANs,such as sudden link degradation or difficulty of observing behavior of a physically remote system.However, in ns-2 users program real, working protocols running on real, working machines. This is agood way to implement new protocols, but it has the drawback that a new concept must be implementedin a working protocol in order to study it by simulation.



Using an existing high-quality simulator, such as OPNET or ns-2, is an attractive option. However,in practice we have found that it may be difficult and time consuming to learn these tools. The veryopenness of these systems that allows them to be adaptable to a variety of environments may make itharder and more laborious to customize them to solve a particular problem. Furthermore, simulatorssuch as those described above are designed to be able to simulate, with high fidelity, the conditionsthat actually occur inside of the network. This often includes effects at the physical, data link, network,and transport and application layer. However, simulating data link, network, transport and applicationevents at the scale of a wide area network can be computationally expensive. Further, simulation of allof these layers and their associated protocols does not add insight in an effective simulation study forproblems such as message complexity for multicast security.

This makes the other option, development of a simulator from scratch, appealing as well, especiallyif some low-level details can be eliminated, and the complexity of the implementation reduced. Infact, a rapidly developed simulator is very important during the initial stages of the design andimplementation of a protocol, because it often uncovers protocol errors not found during the initialdesign, and because it provides an early indication of protocol performance. For these reasons wedecided to write SESAME. In addition to performing correctly, we wanted to ensure that SESAMEcould be developed rapidly, would easily allow assessment of different multicast protocols, and wouldperform adequately. Section 3 describes SESAME’s construction, and Section 4 explains how we usedSESAME to evaluate a multicast security protocol. The rest of this section provides brief backgroundon priority queue performance analysis, and the use of design patterns as a technique for simplifyingsoftware design and architecture.

Priority queues

The way a priority queue performs has the critical impact on the overall performance of anysimulator. Understanding how priority queues perform is therefore essential in the deployment andimplementation of any simulation toolkit. The priority queue holds the pending event set, which is theset of events that have been generated but not yet evaluated. The most important metric for judgingpriority queue performance is the access time, the time required to perform the most basic operations.For priority queues the basic operations are enqueue and dequeue. An enqueue inserts a new elementinto the priority queue. A dequeue removes the element with the highest priority from the queue. Inmost cases, the measure of interest is the amortized (or average) time, although in some applications,such as real-time systems, the worst-case access time is studied. Since the actual performance ofdifferent priority queues is a key factor for the above applications, there have been numerous studiescomparing access times under different systems and implementations.

To illustrate these concepts and to understand how our run-time environment performed, wedeveloped SESAME classes that represent a binary heap, a skew heap, a splay tree, a calendar queueand a skip list, and compared their performance in various settings. We ran experiments using theClassic Hold, Markov Hold, and Up/Down priority queue access patterns. These three access patternsdetermine the distribution of enqueues and dequeues; they are fully described in Section 6. Wealso developed several priority increment distribution functions to study their effect on the overallperformance of a priority queue. A priority increment determines the relative basis of where a newitem will be inserted into the priority queue data structure.



Design patterns

A design pattern represents a recurring solution to a software development problem. Design patternshelp software engineers and architects to move away from having to continually re-invent conceptsand components by capturing standard approaches to repeated situations. Patterns are particularlyimportant in framework-based systems such as SESAME. A framework is a set of cooperating classesthat make up a reusable design for a specific class of software [14], such as a message-complexitysimulator as represented by SESAME. The framework defines a software architecture. The advantageof a framework is that you can build applications faster, and applications have similar structures [14].This makes them easier to maintain and easier for new application developers to understand. For thesereasons a framework-based approach is ideal for customized simulation design. The danger in using aframework is that they lock users into certain design decisions.

One way to alleviate the above problem is to use design patterns within a framework. Since designpatterns capture general solutions to whole classes of problems, a well-thought-out framework withdesign patterns can achieve a higher, more general level of design and code reuse. For instance, inSESAME we used the facade pattern. This pattern provides a unified interface to a set of componentsin a subsystem, thus making the subsystem easier to use. We used this pattern to implement protocol-independent packet dispatching by network nodes (see Section 3.1).

3. SESAME DESIGN

In designing SESAME, our goal was to develop an extensible framework for algorithmic analysis thatwas simple to implement and debug, could easily generate application and network topology scenarios(including the use of a visualization tool), and had good performance.

Since we are interested in analyzing multicast message complexity via simulation, SESAMErequired three major components: a discrete-event simulation (DES) component, a network simulationcomponent and a multicast component. The discrete-event simulation component provides aframework for implementing a variety of discrete event simulators, independent of the specificapplication areas being simulated. The network component passes messages between network nodesand hosts. The multicast component generates specific message workloads, depending on the type ofapplication and the CB protocol. The top-down diagram of the simulator is shown in Figure 1.

Taken together, the DES, network and multicast components produce the framework within whichto analyze multicast message complexity. This framework is extensible: new processes and networkprotocols can be plugged into it. Since only some of them may be applicable to a given simulationexperiment, SESAME provides a generic facility to load a particular configuration desired by the user.When the developers want to simulate a new combination of protocols and behaviors, they simplysubclass the abstract class Configuration, code it to construct the required environment, and runthe simulator setting the property cfg to the name of the newly created configuration class.

As Figure 1 indicates, the simulator’s main class, Driver, reads its configuration file, establishesthe name of the configuration class that defines the simulation environment, instantiates it, and letsit create a configuration instance. The initialization process uses the factory design pattern [1]: theclass Configuration is the abstract factory, and the class Configuration-Instance is the



Multicast Sim ulation Layer

Network Simulation Layer

Discrete Event Simulation Layer

ConfigFiles

LogFiles

D river S im ulatorP end ing

E vent S et

P riorityQ ueue

C onfigurationC onfiguration

Ins tance

R outerH os t

H os t H os t

N et

H os t H os t

N et

H os t H os t

N et

H os t

R outer

K H IPR outerA pp

H os t

A ppH os t

A ppH os t

N et

A ppH os t

A ppH os t

N et

A ppH os t

A ppH os t

N et

A ppH os t

K H IPR outer

c reate

dequeuerun

c reate

E vent

enqueue

enqueue

enqueue

seed

fireseed

Figure 1. Top-down diagram of the SESAME simulator. The discrete-event-simulation layer provides the core ofthe simulation framework all other components are plugged into. The class Simulator runs the event loop afterSESAME is initialized. The initialization is driven by one or more configuration files that select components to beused, define parameters for them, and stipulate how they are to be attached to the framework. Unlabeled arrowsleading from configuration files to the Driver and Configuration objects indicate that these objects read thefiles and configure themselves accordingly. Arrows connecting networks and routers imply physical connectionthrough a network card. Lastly, unlabeled arrows emanating from the Event object point to the target of the event,

i.e. the object that activated by the event’s firing.



abstract product. This approach allows a SESAME simulation session to be configured for one of manyfamilies of environments.

Among other things, the configuration instance contains the topology of the simulated network. Tostart a simulation session, the configuration instance seeds the pending event set (see Section 3.1) withone or more events to bootstrap the process, and the driver passes control to the simulator object. Anyinformation necessary to evaluate results of a simulation session can be passed to a logger object thatrecords it in a file. SESAME is capable of supporting several simultaneous logging streams each withits own output file. The logger API is open to all standard and user-defined components in a runningsimulator.

The remainder of this section describes our design of the DES and network components and oursolution to the problem of efficiently emulating multi-threading within the single-threaded SESAMEsimulator.

3.1. DES and network components

In the discrete event simulator component, the abstract class Event is the root of the hierarchy ofclasses representing events. All events include the target object of the event, the object that generatedthe event, an additional wrapper object to pass any parameters required by the event, and the event’sscheduled time. Event and other classes comprising the DES component are shown in Figure 2.

The class Simulatormaintains a pending event set (PES), and running in a loop, extracts the nextscheduled event object from the PES (if there is any) and calls that object’s method fire() to actuallyexecute the event. Simulator is a singleton class [1]: there should be exactly one instance of thisclass in the application. Additional methods are provided to start, stop, restart, suspend and resume thesimulation process.

When designing the DES layer, we initially considered directly using a Priority-Queue toimplement the pending event set. This would seem to be a logical way to save on implementation time.This approach, however, has the following problems.

• The Simulator will need to cast the result of a dequeue operation (a Comparable) to theclass Event. If for whatever reason (say, due to a bug) this object does not happen to be anEvent, an exception will be thrown by the JVM. This situation can only be discovered at runtime.

• The method enqueue() allows any Comparable object as its parameter; therefore, objectsother than Events may be erroneously included in the pending event set. Since many classeswill generate events, it may be difficult to locate and fix this bug.

• Events are scheduled relative to the current simulation time, but priority queues deal withabsolute priorities. Thus, a mechanism is needed to convert the former into the latter.

We resolved these issues by introducing the adapter design pattern into SESAME. The intentof this pattern is to ‘convert the interface of a class into another interface clients expect’ [1]. Theclass Pending-Event-Set, an adapter for various implementations of priority queues, narrows thePriority-Queue interface allowing only objects of the class Event into the queue, and convertsrelative scheduled times of events supplied by the user into absolute times required by priority queues.Using the adapter pattern enabled rapid development and debugging of the DES layer, and in particularfacilitated our experiments with different priority queue implementations.



Simulator

+enqueue(in evt : Event)+dequeue() : Event

PendingEventSet

BinaryHeap

CalendarQueue

SkewHeap

SkipList

SplayTree+fire()

-target : Object-param : Object-source : Object

Event

Comparable

-prty : long-seqNo : int

Priority

+enqueue(in cmp : Comparable)+dequeue() : Comparable

PriorityQueue

uses

Figure 2. Class diagram for the DES component of the simulator. Legend: —abstract class

or interface; —concrete class; —generalization (‘is-a’) relationship; —composition(‘has-a’) relationship; —dependency; + —public method/attribute; - —private method/attribute.

Figure 3 shows the sequence diagram for the DES component. After initialization that, among otherthings, includes seeding the pending event set, the Simulator runs in a loop, retrieving the nextscheduled event and passing control to it. The event (an instance of a situation-specific subclass ofthe class Event) calls its target object, which in turn schedules zero or more future events whileperforming its ‘useful’ task.

The network-simulation layer is another part of the SESAME architecture where careful designtradeoffs were necessary. This component includes classes and interfaces necessary to simulate thefunctionality of the OSI data link and network layers. A data-link layer contains protocols for handlingtraffic at the hardware level, over local area networks (LANs) or direct device-to-device (point-to-point)links. The network layer contains end-to-end protocols for routing and congestion control over widearea networks. Multicasting is a network-layer protocol that uses services of IP, the Internet Protocol.It therefore may appear that a reasonable level of modeling abstraction is to provide only networklayer support in SESAME. However, the SESAME network component includes both network anddata-link functionalities. Although for a network simulator this is an unusual design and appears tobe a so-called ‘layer violation’, combining both layers has certain advantages. First, it is simpler toimplement and has the performance advantage of not requiring multiple component processing permessage event. Second, it extends the ability of the simulator to study message complexity under a



Simulator PES PriorityQueue Event Target Object

dequeue()

dequeue()

fire()

some method()

enqueue()

enqueue()

enqueue()

enqueue()

Figure 3. Sequence diagram for the discrete-event simulation layer. Legend: —object lifeline; —activation

(method duration); —blocking method call; —object destruction.



wide variety of network architectures. For instance, SESAME can be easily extended to study theeffect of multi-homed hosts (i.e. hosts with multiple data link interfaces).

SESAME combines the data-link and network layer in such a way as to easily enable the introductionof new protocols and new sources of performance delay within the simulator. This is done by definingtwo interface hierarchies and using a facade pattern to combine them. Further, the fundamentaldata structure that is processed by the network-simulation layer, the packet, is actually a combinedmessage that contains both network- and application-layer data. Although this also appears to be alayer violation, this feature enabled us to rapidly emulate the workings of a multi-threaded simulationpackage. Further, this layer violation has no effect on SESAME’s operation, since the design goal is tostudy message complexity and protocol correctness.

The SESAME network component includes the classes Node and Packet. A Node represents anetwork node, currently a host or a router. As described above the abstract class Packet serves a dualpurpose, representing both an application-level message and a network packet. It contains a bare-bonesset of fields: the source and destination addresses, and a sequence number and an in-reply-to-sequence-number required to support remote calling, described below.Nodes may be of two types: a Host or a Router. The class Router provides minimal facilities

for packet forwarding. By default it uses a variation of a centralized routing protocol with a staticrouting table implemented by the class Unicast-Routing-Table. More specialized router classesmay use different routing algorithms and incorporate routing-table updates if needed. The class Framerepresents OSI data link layer frames. Frames are responsible for transporting packets within a LAN.The class Net represents LANs in SESAME. When a Net receives a frame it schedules a deliveryevent and returns. This approach allows introduction of delays into the simulation process: any Netobject may optionally get a random-number generator to produce delays with a desired distribution.When a network node needs to send a packet and creates a frame for it, it needs to find out whatphysical (MAC) address corresponds to the packet’s destination address. For example, nodes in IPnetworks use the Address-Resolution Protocol (ARP) to dynamically request mappings between IPand MAC (e.g. Ethernet) addresses. By default, the network component uses a static address resolutiontable (the class Address-Resolution-Table). Finally, we provided a simple mechanism tosupport flooding protocols (the classes Tiny-Flood, Tiny-Flood-Implementor and Tiny-Flood-Packet). To implement a specific flooding protocol (e.g. TinySAP discussed in Section 3.3)the programmer will have to subclass the classes provided in this package.

We next describe the use of dual class hierarchies to allow rapid introduction of new protocols.The interfaces Protocol and Protocol-Implementor are provided to become roots of the twointerface (or class) hierarchies: the first is expected to provide meta-information about a new protocol,and the second, a collection of methods to process different packets associated with it. When a newprotocol, My-Protocol, is introduced with packet types My-Packet1, My-Packet2, etc., theprogrammer should create a new specialization of the interface Protocol-Implementor, such asthe one shown in Figure 4. Then all classes-participants in the protocol must implement the interfaceMy-Protocol-Implementor.

The classes Node and Packet were designed in such a way as to allow programmers to introducenew protocols without affecting the rest of the simulator. Any given node type may implement a varietyof sub-interfaces of the interface Protocol-Implementor, i.e. support multiple protocols. Theproblem is that we do not want to modify the class Node every time a new protocol is added. Yet,we would also want to dispatch packets received by a node, based on the packets’ type. Since Java



public interface MyProtoImplementor extends ProtocolImplementor {public void packet(MyPacket1 pkt, Net net);public void packet(MyPacket2 pkt, Net net);...

}Figure 4. A Java interface to support a newly introduced protocol in SESAME.

public void accept(ProtocolImplementor protoImp, Net net) {try {MyProtocolImplementor myProtoImp =

(MyProtocolImplementor) protoImp;myProtoImp.packet(this, net);

}catch (java.lang.ClassCastException cce) {super.accept(protoImp, net);

}}

Figure 5. A typical implementation of the method accept() in a subclass of the class Packet. Calling thesuper-class implementation allows co-existence of concrete implementations of protocols and their specializations,

such as multicast group management, in the same configuration.

(and most other object-oriented languages) does not support double dispatching, the virtual method-resolution mechanism that takes into account both the class of the object being called and the classof the parameter it is called with, it is difficult to translate this interaction into working Java code.However, careful class design allowed us to mimic the functionality we need. We decided to use thefacade design pattern [1] to achieve this goal. Facades are used to provide a simple interface to complexsubsystems, especially when you need to layer your subsystems. Figure 5 shows the algorithm thatshould be used by all subclasses of the class Packet to support the protocol-independent dispatchingby Nodes.

3.2. Concurrency emulation package

Distributed multicast protocols often require a request/reply pair of messages, in the form of a remotecall with the caller blocking until the reply is available. For example, a typical way to performsecure multicasting is for a multicast group initiator to obtain authorization from an authenticationservice. One way to implement and simulate this action is to use multiple threads and some formof concurrency control. However, when designing SESAME we wanted to eliminate the potentialdebugging complexity that multi-threaded applications sometimes entail. Since we implementedSESAME as a single-threaded application, we needed a mechanism that would allow ‘swapping out‘of a blocked task and ‘loading’ of another one ready for processing. This bears an obvious similarity



with a multi-tasking operating system. The approach we took is based on the state design pattern[1]. The intent of the state design pattern is to allow an object to alter its behavior when its internalstate changes. Properly implemented, this pattern allows run-time objects to appear to change theirclasses. In our case, any task involving one or more request/reply exchanges was represented by a finiteautomaton that changed its state every time it received a packet. Since packets traverse the simulatedinter-network asynchronously, chances are a different automaton (if any) advances towards its finalstate in each iteration of the event loop. This provides the desired illusion of simultaneous blockingof remote calls. We call the SESAME package rpc; however, the rpc package provides classes andinterfaces to manage requests and replies, and does not implement any true remote procedure call(RPC) paradigm. The class diagram for the package is shown in Figure 6.

The abstract class Subroutine represents any task that may require blocking calls, i.e. anysequence of actions that may make blocking calls must be organized as a subclass of Subroutine.Every instance of the class Subroutine carries a reference to its owner (e.g. a multicast router) and aunique sequence number. When a Subroutine sends packets to other network nodes, this sequencenumber is included in every packet (the field seqNo). Every reply is tagged with this number as well(the field inReplyToSeqNo). (Note that these sequence numbers are distinct from any sequencenumbers used for packet ordering.) Thus, if a node has multiple active subroutines, and a packet arrives,it can always find which subroutine should receive the packet, if any.

Every Subroutine is organized as a sequence of steps. Every step is represented by an instanceof the class Step. One common scenario for a Step is to run in response to a packet, perform somelocal operations, send a request, and terminate. The next Step in the Subroutine will process thereply. The last Step in a Subroutine must not make any calls since there are no other Steps topick up the reply. A special type of Subroutines, Sequence-Subroutines, may be used whena Subroutine doesn’t have any flow control logic in it. Steps and Subroutines provide anexample of the state design pattern [1]: a Subroutine ‘modifies’ its behavior every time it receivesa Packet.

Usually network nodes do not interact directly with instances of the class Subroutine. Instead,they rely on the class Subroutine-Manager. An object of this class can keep track of allSubroutines active in a node and dispatch replies to them.

In some cases a series of protocols may share a sub-protocol. For example, in KHIP [3] a trustedrouter performs the same sequence of steps to join a multicast tree whether it was prompted by anapplication host on an attached LAN or by another router on whose path to the tree it happens to lay. Toaccommodate this, we allow a Subroutine to call another Subroutine thus providing another vehiclefor reuse in SESAME. Objects of the abstract class Pseudo-Packet may be used to pass returnvalues from Subroutines. A Pseudo-Packet never leaves the network entity that generated it.In all other respects (notably type-based packet dispatching) it behaves like a regular packet.

The class Timeout-Pseudo-Packet, a subclass of Pseudo-Packet, represents ‘signals’delivered to a Subroutine if the reply it is waiting for failed to arrive within a specified periodof time. The interface Timeout-Pseudo-Packet-Receiver should be implemented by anyStep that expects a reply to a request sent by its predecessor within a Subroutine. The predecessorStep should schedule a timeout packet when it sends a request. If the reply arrives within the specifiedinterval, it will be processed, and the timeout packet when it is delivered, ignored; if the reply arrives toolate, it will be ignored since by that time the Timeout-Pseudo-Packet will have been deliveredand the Subroutine will have moved on to other Steps or terminated.



Packet

PseudoPacket

+Subrout ineManager( in node : Node)+putSub(in sub : Subroutine)+callSub(in sub : Subroutine, in pkt : Packet, in net : Net)+callSubIfExists(in pkt : Packet, in net : Net) : bool

« implementat ion c lass»SubroutineManager

+Subroutine(in node : Node, in seqNo : int)+hasMoreSteps() : bool+getFirstStep() : Step+getNextStep() : Step+getSeqNo() : int+getNode() : Node#setNextStep(in step : Step)

Subroutine

+SequenceSubroutine(in node : Node, in seqNo : int)#setSteps(in steps : Step[])

-pc : int

SequenceSubroutine

Step

«inter face»ProtocolImplementor

'Program counter '

«uses»

«uses»

Figure 6. The logical view of the package rpc. If a Sequential-Subroutine calls another Subroutine,it holds a reference to the latter for the duration of the call. Each Sequence-Subroutine contains a sequentialset of Steps. The Subroutine-Manager may have several active subroutines executing ‘concurrently’.Legend: —generalization (‘is-a’) relationship; —‘implements’ relationship; —composition (‘has-

a’) relationship; —dependency; —note; + —public method/attribute; # —protected method/attribute;- —private method/attribute; —method signature including its name,

parameters (their names, types, and category: input, output, or both), and return type (if any).



As the discussion at the beginning of this section indicates, the requirement to create a concurrencysimulation mechanism came from the network simulation layer. Thus, the package rpc has a distinctpacket-switched network flavor. If SESAME’s approach were to be used to simulate a system notrelated to computer communications a developer would represent its components as nodes in a networkto reuse the currently available concurrency simulation mechanism.

3.3. SESAME multicast layer

This section describes SESAME’s multicast component. It is designed to provide the basic servicesrequired by protocol designers to measure message overhead and performance. SESAME providesclasses representing services and applications common to all distributed multiparty applications, suchas directory services and session announcement tools. SESAME also has a topology visualizationtool. When designing or evaluating a new multicast protocol it is only necessary to write a new classrepresenting the protocol-specific actions of a router.

SESAME’s standard multicast classes include an App-Host, which represents any multicastapplication host. This class sends and receives SESAME packets, and also receives group formationannouncements via the TinySAP protocol, described below. Embedded in the App-Host class is theworkload generator. There are two types of workload generated within SESAME. The first is for groupmembership changes, including the decision when to join a group, when to leave a group and when toform a group. The second is the data-packet generation module, which decides when to generate newdata packets to send to the group.

Other SESAME classes include Authentication-Server, the entity that provides permissionto start a new group or join an existing one. Another server is the Directory-Server, whichprovides a directory of distributed group services (e.g. names and locations of database or videoservices) and security principals (e.g. hosts, routers, services, and users with their public keys).SESAME also has a Location-Server, which allows the lookup of the group’s center-pointaddress. Finally, we wrote a generic protocol/class for session (group) announcements, called TinySAP,or Tiny Session Advertisement Protocol. It is a flooding protocol, and therefore is guaranteed to reachall hosts and nodes. Specific CB protocols will subclass this class. Packets of this type often do notneed to be counted when analyzing message complexity.

In order to evaluate a specific CB protocol using the SESAME framework it is only necessary todesign and implement a SESAME class that provides protocol-specific logic for packet processing androuting. The next section describes a specific CB protocol we implemented, and then describes theSESAME visualization tool.

4. KHIP

The Keyed HIP (KHIP) multicast security protocol was introduced in [3] as a secure enhancement of ahierarchical multicasting protocol. KHIP provides for interoperation of multicast domains employingdifferent communication and security protocols. Security and core selection in KHIP are tightlycoupled. KHIP is based on HIP, a hierarchical multicast routing method that routes traffic betweenheterogeneous multicast domains using the Ordered Core-Based Tree (OCBT) protocol [9]. HIPrepresents each domain as a virtual router by having the border routers of the domain emulate behavior



of a single OCBT router. HIP is a core-based protocol; in HIP the core is referred to as the center point.Different multicast groups may have different center points. However, KHIP, HIP and OCBT do notspecify exactly how the center point is chosen.

In order to study KHIP within SESAME we only needed to implement a class called Khip-Router. The primary function of this class is to process packets produced by groups running theKHIP protocol. Khip-Router implements the policies and algorithms required by KHIP. It must beable to handle KHIP-specific packets, including:

• the ‘you-are-the-center-point-of-the-group’ message sent by the group initiator to the centerpoint it selected;

• core requests sent by routers wishing to join some multicast group;• multicast data packets;• center point distance requests that may be sent by a border router to other border routers in its

domain in order to locate the exit router (in KHIP a domain’s exit router is the border routerclosest to the center point);

• requests from application hosts to join or leave a multicast group;• join requests sent by routers wishing to join a multicast group.

In SESAME, KHIP routers use the rpc package. Every time a subroutine gets a packet (or a pseudo-packet), it processes it and moves to the next step. As a general rule, each step in a subroutine starts withreceiving a reply and ends with sending a request, whose reply will be used as input into the next step.For instance, when a KHIP router receives a membership request from one of its application hosts, itruns an instance of the class Host-Join-Subroutine. This subroutine in turn calls a Trusted-Router-Join-Subroutine to help the router join the requested multicast group, When it returnswith a Group-Info object, the Host-Join-Subroutine adds the host–originator to the list ofapplication hosts—members of the group.

We used our KHIP implementation to test an analytical model for the message overhead requiredto construct a delivery tree that includes all routers wishing to join a new multicast group. The detailsof the model are outside the scope of this paper and are available in [9]; briefly, the model uses acombinatorial approach that considers the expected number of ways a particular router in a highlyconnected domain can be designated a core. We ran over 400 different experiments and were able toverify our model with a high confidence level.

During our implementation we also uncovered a loophole in the KHIP: it does not guaranteebackward confidentiality, i.e. the ability to prevent newly-joining members from obtaining earlier datamessages. In the KHIP protocol data communications, join and leave events, and tree-maintenanceoperations occur simultaneously. KHIP does not provide any primitives for coordination betweendifferent processes happening in the inter-network at the same time. As a new router joins a groupit may decide to eavesdrop on the group’s communications passing though its network and stashthem until it actually joins the group. Then it will be able to decrypt data packets multicast beforeit joined the group. The details are presented in [15]. Therefore, in its current state KHIP is notbackward confidential. SESAME allowed us to spot this problem, thus highlighting the advantageof early simulation in protocol design.



(a)

(b)

Figure 7. A sample KHIP topology: (a) the top half; (b) the bottom half. Legend: —application host;—authentication server; —directory server; —initiator; —location server; —LAN; —KHIP router;

—domain boundary.



5. MULTICAST VISUALIZATION TOOL

Network- and distributed-system topology visualization tools are important instruments forunderstanding the behavior of different protocols. This is certainly true when analyzing distributedmulticast systems that involve multiple router, server (directory, location, etc.) and host types. For ourproblem domain, the visualization problem is reducible to a graph-drawing problem. There are severalgeneral guidelines or principles for graph drawing, including ‘distribute edges uniformly’, ‘avoid edgecrossings’ and ‘display symmetries’ [16]. In general, the optimization problems associated with theseaesthetics are NP-hard. One commonly-used technique, like the one used in the ns-2 simulator, treatsnodes as balls connected together with springs. The springs represent links or edges. The balls repulseeach other, the springs pull them together, and the system eventually converges into a readable layout.We adopted a simpler approach for SESAME, one that takes advantage of our focus on hierarchicaldomains. We have found that our simple design generally produces aesthetically pleasing results forour purposes.

Figure 7 shows a sample network topology (divided into a top and bottom half) rendered in thevisualization tool.

In this case we made partial use of an existing tool. The SESAME visualization tool was built usingthe package Diva produced at the University of California at Berkeley [17]. Diva provides a powerfulframework for building, rendering, and manipulation of graphs. It also provides a set of abstract classesand interfaces to manage the layout of graphs, and several concrete classes that implement simplestrategies for doing it. None of these classes, however, provided a means to produce a meaningful andreadable rendering of a hierarchical network topology. We developed the class Mcast-Layout thatprovides this functionality.Mcast-Layout implements the interface Global-Layout, the common ancestor of all layout

managers in Diva. To display a multicast layer network, this class represents each multicast domainas a set of concentric rings, including (a) the internal router ring, (b) the internal LAN ring, and(c) the border router ring. Each network has a semi-circle with hosts (i.e. application hosts and, whenapplicable, the authentication, directory and location servers and the group initiator) hanging off it.Domains, in turn, are also arranged in a circle. Finally, external LANs (i.e. LANs that do not belong toany domain) are placed between the domains they connect.

Every ring is given its minimum radius (or its minimum distance to the preceding concentric ring)and the minimum distance between vertices on that ring. If there are only a few vertices on the ring, thenthe minimum radius is used as the final radius; if there are many vertices, the final radius is calculatedin such a way as to guarantee the minimum distance between vertices. Finally, each ring is rotated bya random angle in an attempt to get rid of overlapping or almost overlapping straight connectors.

6. PERFORMANCE STUDY

Since SESAME was implemented in Java, analyzing its performance was especially important for us.As described in Section 2 the performance bottleneck for discrete event simulators are priority queueevents. For instance, our initial experiments with SESAME using a linked list-based priority queueimplementation showed that the simulator spent about 95% of its CPU time just inserting and extractingevents. It meant that the system could not be used even for simplest simulations, and another data



0

5

10

15

20

25

30

10 18 32 56 100 178 316 562 1000 1778 3162 5623 10000 17783 31623 56234 100000

Queue size

Mea

n ac

cess

tim

e, m

icro

secs

Binary heap

Calendar queue

Skew heap

Skip list

Splay tree

Figure 8. Mean access time in Classic Hold experiments with different priority-queue implementations; andpriority increments distributed according to the exponential law.

structure was required. Indeed, our analysis of SESAME’s performance confirmed that the pendingevent set (i.e. the priority queue) is the most serious bottleneck in a discrete-event simulator. Forinstance, we found that even a good priority queue implementation (based on splay tree) SESAMEstill spent 54% of its CPU time accessing the priority queue. These figures were obtained from anetwork topology containing 220 nodes.

We decided to study behavior of five different priority queue implementations—binary heap [18],calendar queue [19], skew heap, skip list [20], and splay tree [21]—under various access patternsand with various priority increment distributions. This section presents the methodology in studyingpriority queue performance and describes the highlights of our results.

There are three classes of access patterns to priority queues [2]; (a) steady state, (b) transient, and(c) random. The steady access pattern is primarily modeled by an algorithm called Classic Hold [22].A hold operation is a dequeue followed by an enqueue. Another algorithm, Up/Down [23] is used toanalyze the transient behavior of priority queues. In Up/Down experiments the queue grows to a certainsize by a series of enqueues, and then gets emptied by a series of dequeues. Finally, in a random accesspattern enqueues and dequeues alternate at random. The Markov Hold algorithm is one way to model



0

0.5

1

1.5

2

2.5

3

3.5

10 18 32 56 100 178 316 562 1000 1778 3162 5623 10000 17783 31623 56234 100000

Queue size

Mea

n ac

cess

tim

e, m

icro

secs

Bimodal

Exponential mix

Exponential

Negative triangular

Triangular

Uniform

Uniform 2

(a)

Figure 9. Mean access time in Classic Hold experiments with the calendar queue: (a) with all but the Zipfdistribution; (b) with the Zipf and bimodal distributions (in logarithmic scale).

random access to priority queues. Since [2] and [15] showed that Markov Hold experiments do notprovide much additional insight into the behavior of priority queues, in this section we will focus onthe Classic Hold and Up/Down algorithms.

In our experiments we were interested in mean access time, worst access time, and standarddeviation. To measure the mean we used the two-loop approach: one of the loops included actualqueue access, the other did not. Because in the first case the queue-access time and the overheadwere measured, and in the second, only the overhead; their difference provides the desired value. Tocapture the worst access time, we had to separately time each queue access. Since Sun’s JDK providesonly millisecond-resolution timers, we developed a JNI-based [24] library in C that utilizes the UNIXgettimeofday(3) function to fetch time measurements with microsecond precision.

The following coding conventions are used in SESAME.

• All priority queue algorithms are implemented iteratively.



1

10

100

1000

10000

10 18 32 56 100 178 316 562 1000 1778 3162 5623 10000 17783 31623 56234 100000

Queue size

Mea

n ac

cess

tim

e, m

icro

secs

(log

scal

e)

Bimodal

Zipf

(b)

Figure 9. Continued.

• Since some of the algorithms cannot be made stable (e.g. the binary heap), each event objectis assigned a unique sequence number used to break ties between events of equal priority. (Analgorithm is stable if elements with the same value are extracted in the order in which they wereinserted.)

• Memory is allocated before the timer starts running.

In this section we present some the highlights of our experiments. A fuller set of experimentalresults is available in [15]. All experiments were performed on a Sun SPARC Ultra60 dual-processorworkstation (clock speed 366 MHz) with 256 MB of memory, running Solaris 7. We used Sun’sstandard Java virtual machine with a just-in-time (JIT) compiler from Symantec.

In our performance studies we varied three parameters: (a) the access pattern to the priority queue,(b) the priority queue implementation, and (c) the priority increment distribution function. The priorityincrement distribution function controls event inter-arrival times, i.e., intervals between consecutivefirings of events. Studying different increment distributions is important since not all priority queue



algorithms work equally well with all priority increment distributions [2]. A more detailed discussionis given in Reference [15].

Figure 8 gives a comparison of different priority queue implementations for the same priorityincrement distribution. The calendar queue, as expected, by far outperforms the other data structures.It’s followed by the skew heap and binary heap. The skip list is the slowest one of all. The splay treeperforms very well for queue sizes more than 100 000 (second only to the calendar queue), mediocrefor sizes between 10 000 and 100 000, and poorly for smaller queues.

Our results for Classic Hold access to the calendar queue are presented in Figure 9(a). Thetheoretically expected mean access time is O(1) [19]. As can be seen, the actual mean access timeremains constant only for queue sizes between 30 and 10 000 elements. Our experiments have shownthat the calendar queue is heavily influenced by the priority increment distribution: unlike the otheralgorithms where most distributions ‘stay together’, the calendar queue ‘fans them out’ so much so thatthe Zipf distribution is out of scale with the rest of them and warrants a separate figure in logarithmicscale (see Figure 9(b)). This observation means that calendar queue is not a good match for cases wherethe majority of priority increments have Zipf or, perhaps, other heavy-tailed distributions.

In conclusion, with the exception of the unstable initial phase in the Up/Down experiments and thecase of calendar queue with Zipf-distributed priority increments, all mean access times are well under30 microseconds. Compared with the results reported by Roenngren and Ayani [2] (they used softwareimplemented in the C programming language and ran it on a Sun SPARC 10 workstation and an IntelPentiumPro-based PC), it shows a tenfold increase in the simulation speed. It means that a simulatordeveloped in Java and run with a JIT compiler is a viable alternative to simulators written in compiledlanguages such as C.

7. CONCLUSIONS

In this paper we described our experiences developing an object-oriented network simulator, SESAME,and using it for simulation and analysis of KHIP, a multicast security protocol. It took us approximately300 hours to scope, design, implement, and test the simulator. The fact that we were able todevelop the system very quickly and use it to make meaningful observations about KHIP’s behaviorand performance of different priority-queue implementations demonstrates that rapid simulatordevelopment using modern software-engineering tools—design patterns, Unified Modeling Language,and a powerful object-oriented language—is a viable alternative to using an existing simulationpackage and adapting it to new protocols and processes.

Our performance analysis of different data structures used to implement the pending event setshowed that off-the-shelf software (e.g. Java) and general-purpose hardware (e.g. a Sun SPARCworkstation) can be used to perform interesting simulation experiments. As we pointed out in Section 6,calendar queue was the best performer among the priority queue implementations, followed by splaytree, skew heap, binary heap, and skip list. We demonstrated that some of the implementations performdifferently with different priority increment distributions. The Zipf distribution especially stood out: inmost cases event streams with Zipf-distributed priority increments produced lower mean access timevalues than the other distributions; however, with calendar queue it exhibited linear behavior whereasthe other distributions produced either constant or logarithmic mean access times.



Finally, SESAME provided us with valuable experience in design and development of a complexsoftware system. The software-engineering tools mentioned above allowed us to capture concepts fromdiverse disciplines: discrete-event simulation, computer communications at the data link and networklevel, multicast protocols, security, advanced data structures, and performance analysis. We believethat SESAME provides an excellent framework for study of new network protocols, their structuralsoundness, and their message complexity.

REFERENCES

1. Gamma E, Helm R, Johnson R, Vlissides J. Design Patterns. Addison-Wesley: Reading, MA, 1995.2. Roenngren R, Ayani R. A comparative study of parallel and sequential priority queue algorithms. ACM Transactions on

Modeling and Computer Simulation 1997; 7(2).3. Shields C, Garcia-Luna-Aceves JJ. KHIP—a scalable protocol for secure multicast routing. Computer Communication

Review 1999; 29(4).4. Moy J. Multicast extensions to OSPF. Internet RFC 1584, March 1994.5. Waitzman D, Partridge C, Deering S. Distance vector multicast routing protocol. Internet RFC 1075, November 1988.6. Deering S et al. The PIM architecture for wide-area multicast routing. IEEE/ACM Transactions on Networking 1996;

4:153–162.7. Estrin D et al. Protocol independent multicast sparse mode (PIM-SM): Protocol specification. Internet RFC 2362, June

1998.8. Ballardie A. Core based trees (CBT) multicast routing architecture. Internet RFC 2201, September 1997.9. Shields C, Garcia-Luna-Aceves JJ. The ordered core-based tree protocol. Proceedings of the ACM SIGCOMM’99,

Cambridge, MA, August 1999; 53–64.10. Canetti R et al. Multicast security: A taxonomy and efficient authentication. Proceedings IEEE INFOCOMM, April 1999;

708–716.11. Wong CK, Gouda M, Lam SS. Secure group communication using key graphs. Proceedings of the ACM SIGCOMM, 1998;

68–79.12. MIL3, Inc. Modeling and Simulating Communications Networks: A Hands-On Approach Using OPNET. Prentice Hall:

Upper Saddle River, NJ, 1999.13. The Network Simulator—ns-2. Palo Alto, CA. http://www.isi.edu/nsnam/ns/.14. Deutsch LP. Design reuse and frameworks in the Smalltalk-80 system. Software Reusability, Vol. II: Applications and

Experience. Addison-Wesley: Reading, MA, 1989; 57–71.15. Rabinovich P. Design and analysis of object-oriented simulation techniques. MS Thesis, Department of Computer Science,

George Mason University, Fairfax, VA, April 2000.(Available as GMU Technical Report TR00-05 at http://www.cs.gmu.edu.)

16. Fruchterman T, Reingold E. Graph drawing by force-directed placement. Software Practice and Experience 1991;21(11):1129–1164.

17. The Regents of the University of California, Diva API. Berkley, CA.http://www-cad.eecs.berkeley.edu/Respep/Research/diva/api/ [1999].

18. Cormen TH, Leiserson CE, Rivest RL. Introduction to Algorithms. MIT Press: Cambridge, MA, 1997.19. Brown R. Calendar queues: A fast O(1) priority queue implementation for the simulation event set problem.

Communications of the ACM 1988; 31(10):1220–1227.20. Pugh W. Skip lists: A probabilistic alternative to balanced trees. Communications of the ACM 1990; 33(6).21. Sleator DD, Tarjan RE. Self-adjusting binary search trees. Journal of the ACM 1985; 32(3).22. McCormack WM, Sargent RG. Analysis of future event set algorithms for discrete event simulation. Communications of

the ACM 1981; 24(12).23. Roenngren R, Riboe J, Ayani R. Lazy queue: New approach to implementing the pending event set. International Journal

of Computer Simulation 1993; 3.24. JavaSoft, Inc. Java Native Interface Specification. Palo Alto, CA.

http://java.sun.com/products/jdk/1.2/docs/guide/jni/spec/jniTOC.doc.html [1997].


rapid simulator development for multicast protocol analysis

Documents