moves: a framework for parallel and distributed simulation...

Available online at www.sciencedirect.com

Computer Networks 52 (2008) 155–179

www.elsevier.com/locate/comnet

MoVES: A framework for parallel and distributed simulationof wireless vehicular ad hoc networks

Luciano Bononi *, Marco Di Felice, Gabriele D’Angelo, Michele Bracuto,Lorenzo Donatiello

Department of Computer Science, University of Bologna, Mura Anteo Zamboni 7, 40127 Bologna, Italy

Available online 29 September 2007

Abstract

In this paper, we illustrate a Mobile Wireless Vehicular Environment Simulation (MoVES) framework for the paralleland distributed simulation of vehicular wireless ad hoc networks (VANETs). The proposed framework supports extensi-ble, module-based and layered modeling, and scalable, accurate and efficient simulation of vehicular scenarios integratedwith wireless communication and mobile services/applications. The vehicular layer includes models for vehicles, syntheticand trace-driven mobility, driver behavior, GPS-based street maps, intersection policies and traffic lights. The wirelesscommunication layer currently includes models for physical propagation, and a network protocol stack including IEEE802.11 Medium Access Control, up to the Application layer. MoVES provides a platform for microscopic modelingand simulation-based analysis of wireless vehicular scenarios and communication-based services and applications, likeIntelligent Transportation Systems, communication-based monitoring/control and info-mobility services. The frameworkincludes design solutions for scalable, accurate and efficient parallel and distributed simulation of complex, vehicular com-munication scenarios executed over cost-effective, commercial-off-the-shelf (COTS) simulation architectures. Dynamicmodel partition and adaptation-based load balancing solutions have been designed by exploiting common assumptionsand model characteristics, in a user-transparent way. Test-bed performance evaluation for realistic scenarios has shownthe effectiveness of MoVES in terms of simulation efficiency, scalability, adaptation and simulation accuracy.� 2007 Elsevier B.V. All rights reserved.

Keywords: Wireless vehicular ad hoc networks; Parallel and distributed simulation; Intelligent transportation systems; Mobile services;Mobile communication

1389-1286/$ - see front matter � 2007 Elsevier B.V. All rights reserved

doi:10.1016/j.comnet.2007.09.015

* Corresponding author. Tel.: +39 051 2094972; fax: +39 0512094510.

E-mail addresses: [email protected] (L. Bononi), [email protected] (M. Di Felice), [email protected] (G. D’Angelo),[email protected] (M. Bracuto), [email protected] (L. Dona-tiello).

1. Introduction

Technology advances in portable devices, embed-ded systems and wireless networking have contrib-uted to the marriage of mobile computing andcommunication for supporting new pervasive ser-vices and applications. As an example, implementa-tion guidelines for a new class of mobile services for

.

mailto:[email protected]

mailto:difelice@ cs.unibo.it

mailto:difelice@ cs.unibo.it




156 L. Bononi et al. / Computer Networks 52 (2008) 155–179

vehicle-drivers assistance and safety are the missionof worldwide Intelligent Transportation System(ITS) research programs [1,2]. The integration ofpositioning and multiple sensor capabilities,installed on-board vehicles [3], and specific wirelesstechnologies for vehicular environment will enablecar-to-car, car-to-infrastructure mobile communica-tion and Vehicular Ad Hoc Networks (VANETs)[1,7,3,8–10]. These technologies include DedicatedShort Range Communication (DSRC) IEEE802.11p Wireless Access for Vehicular Environment(WAVE) [4], and ISO TC204 Continuous Air inter-faces-Long and Medium Range (CALM) [5,6].VANETs can be considered as a mix of mobile adhoc and sensor networks [3]: opportunistic commu-nication up to intermittent/stable access to theInternet is possible, depending on the deployedwireless infrastructures. Distributed and coordi-nated communication-based services realized ontop of VANETs are expected to improve safety(e.g., car accident reduction), real-time monitoringand equalization of traffic flows and parkingresources, and the support of real-time info-mobil-ity services about traffic jams, accidents, weatherand dynamic travel route optimization [11,12].Moreover, infrastructure-based inter-vehicle com-munication (IVC) systems may enable a wide setof vehicular killer applications, ranging from Inter-net access, e-payment and infotainment services.

A new generation of adaptation-based protocolsand applications for VANETs are currentlyexplored by some recent research projects headedby research institutes and car manufacturers [7–10]. The complex protocol architecture and conver-gence of wireless communications in vehicular sys-tems is the subject of current research projects,e.g., the Cooperative Vehicle to Infrastructure Sys-tem (CVIS) project [2] and the Network on Wheels(NOW) project [8]. In order to promote the intro-duction of novel applications based on IVC systems,it is essential to demonstrate the effectiveness ofsuch protocols and service architectures over realis-tic environments by using accurate and fine-grainedmethodologies for the analysis [13]. Experimentalcharacterization of IVC [14] may confirm the feasi-bility of ITS systems, under possibly different sce-narios and service paradigms (e.g., client/server vs.P2P).

Computer simulation is used as a fine-grainedand cost-effective support for analysis: the modelingof each area, vehicle and driver behavior can beaccurate, and this would make it possible to evalu-

ate both complex and hypothetical scenarios, whichare impractical to realize as a real testbed. On theother hand, the adoption of simulation techniquesfor the analysis of large-scale, dynamic and complexvehicular networks, and related wireless communi-cation-based services, creates severe problems interms of simulation resources and simulation per-formance. Large data structures defining the stateof complex model entities have an impact on mem-ory requirements. Fine-grained simulation of urbantraffic and wireless communication scenarios mayeasily involve thousands of causally interacting enti-ties, each one characterized by local policies andattributes [15]. In a simulation, the individual evolu-tion of a massive number of entities, the dynamiccorrelation effects due to mobility and to the localbroadcast nature of the wireless transmissions, andthe wireless signal propagation effects in urban sce-narios [16] greatly increase computational require-ments. Unfortunately, a sequential simulation of areal scenario with a large number of complex modelentities (like vehicles running distributed services,and their wireless communications) may becomecomputationally infeasible on a single physical exe-cution unit (PEU) [17,18]. Solutions based on paral-lel computing architectures, like Single InstructionsMultiple Data (SIMD) architectures have been con-sidered in PARAMICS [19]: this approach may beuseful under homogeneous modeling assumptions(that is, concurrently executed code updates differ-ent data structures each one representing the stateof a homogeneous instance of the model entities).On the other hand, parallel and High-PerformanceComputing (HPC) architectures have high costand require modelers to have expertise in parallelprogramming, to exploit the architecture potential.Parallel and distributed simulation (PADS) [20]offers a cost-effective technology to exploit concur-rent computation over different PEUs. In this way,the simulation can be supported by low-cost aggre-gate computation and memory resources. In addi-tion, the approval in 2000 of the IEEE 1516Standard for Modeling and Simulation High LevelArchitecture (HLA) [6] has fostered the creationof middleware frameworks for PADS. Advantagesof HLA-based PADS middleware include the sup-port for integration of heterogeneous simulators,model reuse, and the solution of main simulationissues in a transparent way, from the user’sviewpoint.

In this paper, we present a novel frameworknamed Mobile Wireless Vehicular Environment

L. Bononi et al. / Computer Networks 52 (2008) 155–179 157

Simulator (MoVES), for the parallel and distributedsimulation of urban mobility and wireless communi-cation over vehicular ad hoc networks. A paralleland distributed simulation is implemented as a con-current execution of Logical Processes (LP). In ourapproach, each physical processor runs an LP incharge of managing the execution of a subset ofmodel entities: static entities like streets, trafficlights, intersections, and mobile vehicle entities withtheir wireless communication stack. In the simula-tion, interactions among model entities are imple-mented by exchanging time-stamped eventmessages (that is, message passing) between the cor-responding LPs. If the message passing and syn-chronization overheads are designed and managedin a judicious way, a parallel or distributed simula-tion offers potential speedup thanks to the concur-rent execution of computation tasks, and mayresult in high scalability at low-cost, thanks to thepossible aggregation of computation, memory andcommunication resources of commercial off-the-shelf (COTS) systems. This would be useful forurban traffic simulation, as explored in [17,21,18].The first version of the MoVES simulator presentedin [22] was implemented from scratch as a concur-rent execution of sequential simulators, coordinatedby the kernel of a PADS middleware calledAdvanced RTI System (ARTIS) [23–25]. ARTIS isa middleware for generalized parallel and distrib-uted simulation partially compliant with the HighLevel Architecture (HLA) standard IEEE 1516 [6].In the first version of MoVES, each simulator wasin charge of managing the coordinated executionof a portion of the simulation model. Initializationpolicies for uniform model allocation, load balanc-ing, and entity migration functions were defined asa part of the MoVES framework. The main draw-backs of this approach were: the initial model parti-tion, load balancing and entity allocation werebased on static policies in the initialization phase,with some limitations that will be illustrated in Sec-tion 4, and the entity migration function was notcompletely user-transparent, being partiallyincluded in the model code. To cope with these lim-itations, the MoVES framework has been com-pletely re-designed in this paper, with supportprovided by the ARTIS simulation middleware[23–25]. In the following, we will refer to theMOVES framework in general when discussingcommon features and implementation issues. Morespecifically, we will refer to the MOVES I imple-mentation, and to the MOVES II implementation,

to illustrate the differences between the first and sec-ond framework implementations, respectively. Thenew implementation of MoVES (MoVES II)defined in this paper has many additional advanta-ges, illustrated in detail in Section 3, includinguser-transparent, dynamic and adaptation-basedentity allocation and load balancing (both commu-nication and computation), and synchronizationand communication overhead reduction. This trans-lates to simple model implementation, dynamicadaptation-based load balancing under generalizedmodel assumptions, and simulation efficiency, scala-bility and accuracy, as shown in testbed perfor-mance evaluation for realistic model scenarios inSection 6.

The paper is organized as follows: Section 2describes related work and existing solutions forPADS and monolithic simulation of vehicular andwireless systems; Section 3 illustrates the layeredframework architecture designed for modeling andsimulation of vehicular traffic and wireless commu-nication; Section 4 illustrates the design and imple-mentation of the proposed PADS architecture,including adaptation-based computation- and com-munication-load balancing and communicationoverhead reduction solutions; Section 5 sketchesthe definition of a testbed vehicular and wirelesscommunication model; Section 6 illustrates someperformance analysis results, based on realistic sim-ulation scenarios (downtown area of the city ofBologna) to show the scalability and performanceof the proposed PADS framework; Section 7 pre-sents conclusions and future work.

2. Related work

Many vehicular-mobility simulation studies arebased on fine-grained models that describe thebehavior of individual entities instead of the behav-ior of (macroscopic) aggregate vehicular flows. Sev-eral simulators have been developed with varioussolutions to support both model complexity andscalability characteristics: VISSIM [26], PARAM-ICS [19,27] and SimTraffic [28] are some of the mostsuccessful commercial tools for simulation of com-plex, dynamic traffic scenarios, with high level ofmodeling detail, and support for realistic models,analysis and rendering. In general, to the best ofour knowledge, such simulators do not includemodeling support for wireless communication. Inaddition, the execution of simulation for very largevehicular models coupled with wireless radio


communication models, and network-based appli-cations and services, may have a significant impacton scalability and performance issues. The integra-tion of vehicular traffic and wireless communicationmodels in VANET simulation environments havebeen proposed in recent works. CARISMA [10] isa traffic simulator which has been coupled withthe Java-based AppSim tool to obtain a traffic andVANET simulation environment [29]. CORSIM[30] is a tool defined to model and simulate trafficin Atlanta, GA, which has been federated in adistributed simulation framework with Qualnet, acommercial wireless network simulator [31].STRAW [13] is a traffic mobility and wireless net-work model simulated using the SWANS simulationtool [32]. A vehicular-mobility model for the net-work simulator (ns-2) is proposed in [33].

Only a few simulation packages currently sup-port fine-grained event-based traffic models forPADS simulation. In some cases, the models aretightly-coupled with a PADS synchronizationalgorithms, by reducing the model portability andreuse. Alternative distributed implementations ofPARAMICS have been proposed: in [17,34]; thedivide-and-conquer concept is applied to thePARAMICS traffic simulation, by splitting the sim-ulation map into sub-areas (called tiles), and execut-ing the simulations of tiles on a cluster of PEUsconnected by high-speed Ethernet links. Vehicleentities migration among different PEUs has beenimplemented to control synchronization overheads.A similar concept is presented in [18], in which syn-chronization among the LPs is managed by anexternal entity. HLA-based distributed simulationof traffic scenarios is considered in [21], and in[30]: a simulation is composed of federated compo-nents, each one executed in a common event-spacewith coordinated time-management scheme.

A set of simulation tools and models have beenproposed to specifically assist in the modeling andsimulation of wireless systems [35]. The list includesthe open source monolithic network simulator(ns-2) [36], Omnet++ [37], and some powerful com-mercial tools like Opnet [38] and Qualnet [31]. Ingeneral, the scalability problem of massive wirelesssystems simulation has been investigated underPADS implementations in preliminary work[39,31]. The Parallel Distributed Network Simulator(PDNS) [40] is a parallel and enhanced version ofthe widely adopted Network Simulator (ns) [36],developed by the PADS research group at GeorgiaTech. The simulator is based on the concept of fed-

erated simulation of multiple instances of ns sched-ulers. The Georgia Tech Network Simulator(GTNetS) [41] is a network simulation environmentfor large-scale networks. GTNetS supports the dis-tribution of a single simulation topology on eithera network of loosely coupled workstations, ashared-memory symmetric multiprocessing system,or a combination of both. The Dartmouth ScalableSimulation Framework (DaSSF) [42] is a C++ par-allel and distributed simulation framework based onmessage passing (using MPI) running a combina-tion of shared-memory and distributed-memoryexecution architectures. The new version (namediSSF, [43]) will support HLA interoperability, real-time simulation, and human-interaction capabili-ties. MAISIE [39] is a C-based simulation languageproposed for sequential and parallel execution ofdiscrete-event simulation of mobile wireless systems.The Telecommunications Description Language(TED) [44] is an object-oriented simulation lan-guage that can be used to model communicationsystems, supported by the Georgia Tech Time Warp(GTW) [44], an optimized parallel simulation kernelfor parallel and distributed simulation. TED hasserved as the basis for the implementation of WiP-PET [45], a parallel simulator designed to modelthe radio propagation, mobility and protocols ofwireless networks. In SwiMNet [46], mobility isassumed to be not correlated to wireless communi-cation: a pre-computation of the mobility patternsof mobile hosts is done to know in advance theeffects of mobility, and to optimize parallel and dis-tributed simulation of wireless PCS networks.GloMosim [47] is a simulation environment forwireless and wired network systems based on theParsec parallel discrete-event simulation kernel.QualNet [31] is a complete and efficient commercialtool derived from GloMoSim. It supports bothmonolithic simulations and parallel simulations. Itis reported to scale up to 10’s of thousands of nodesand it includes a complete and detailed library ofprotocol models for wireless systems. Qualnetoptions include the support for HLA & ThreadedCommunication Modules. SWANS [32] is a Java-based scalable wireless network simulator built onthe top of JiST, a high-performance discrete-eventsimulation engine. OPNET [38] is a commercialframework for modeling and simulation of wiredand wireless communication systems. It includes alarge library of implemented protocol models. Thetool supports discrete-event, hybrid and analyticalsimulation; it runs in both sequential and parallel


simulation mode, and supports HLA and co-simu-lation technologies.

A summary of motivations, new characteristicsand advanced methodologies specifically imple-mented in the MOVES framework for the paralleland distributed simulation of VANET scenarios isreported in Section 3.1.

3. The parallel and distributed simulation architecture

In this section we illustrate the aims, the motiva-tions, and the logical structure of a PADS architec-ture for modeling and simulation of wirelessvehicular networks. Our PADS framework has thefollowing aims:

• Scalability: by exploiting aggregate computationand memory resources of different PEUs, simula-tion of massive models can be supported;

• Transparency: by exploiting PADS primitivesimplemented by the ARTIS middleware, the dis-tributed model implementation should be astransparent as possible to the modeler;

• Adaptation-based overhead reduction and loadbalancing: by exploiting PADS primitives andload balancing heuristics implemented by theARTIS middleware, we defined adaptation-basedsolutions to initialize, allocate and migrate modelentities over COTS simulation architectures.Dynamic adaptation improves the utilization ofshared resources and the simulation performanceby reducing message passing and synchronizationoverheads between LPs;

• Software modularity and reuse: the simulationtool and modeling architecture allow the compo-sition and reuse of module-based model compo-nents, including modeling and simulationmanagement modules.

3.1. Motivation and previous work

A number of available vehicular traffic simula-tors (i) are commercial products, (ii) do not supportdistributed/parallel simulation, or (iii) do notinclude models for the wireless networking simula-tion. A federation of existing traffic and wirelesssimulators, like the ones described in Section 2, ispossible only if all simulators implement a commoninteroperability standard, like IEEE 1516 [6]. Ingeneral, even if the performance and compatibilitycharacteristics of federated simulators are good,

the performance of such an approach may be poordue to: (i) sub-optimal utilization of resources, (ii)lack of a coordination scheme for resources adapta-tion and load balancing, and (iii) lack of exploita-tion of model characteristics and assumptions. Forthese reasons, we developed a novel framework(named Mobile Wireless Vehicular Environmentsimulator, MoVES) for the fine-grained (micro-scopic) modeling and simulation of realistic trafficscenarios, wireless networks and communication-based services and applications. The first implemen-tation of the MoVES framework (MoVES I in thefollowing) proposed in [22] was realized as a self-contained simulation framework, based on parallelexecution of coordinated sequential simulators.Each simulator was synchronized by exploiting theARTIS parallel and distributed simulation middle-ware [24]. Many assumptions, and runtime optimi-zation solutions, were implemented at themodeling framework layer. The MoVES I imple-mentation has some drawbacks: (i) the optimizationapproach is based on strong assumptions, (ii) theoptimization functions are model-dependent, and(iii) the optimization functions are local to eachsequential simulator, and may lead to sub-optimalglobal performance. Details about these issues willbe illustrated in Sections 4.3.2, 4.4.1 and 6.1. In thispaper, MoVES has been completely re-designed andre-implemented on a layered PADS architecture (seeFig. 1). A new implementation of the MoVESframework (MoVES II in the following) is now real-ized as a distributed simulation, based on additionalARTIS middleware functionalities. This new lay-ered approach has potential advantages, illustratedin the following Section.

3.2. The PADS architecture description

By looking at the PADS architecture (Fig. 1),three separate layers are defined: the MoVES frame-work layer, the ARTIS middleware layer, and thephysical execution units (PEUs) layer.

Starting from the bottom of Fig. 1, the PEUlayer represents the physical computation and com-munication system supporting the parallel and dis-tributed simulation. In general, a physicalexecution unit (PEU) is defined as an autonomousprocessing unit, connected to other PEUs by net-work hardware. A PEU may have shared-memorymultiprocessor architecture and multi-threadingCPUs, or a single processor architecture. In ourapproach, each physical processor runs a Logical

MoVES

application

(wireless) network

vehicular

P.E.U.

Cpu

Cpu Cpu

Cpu

LAN

MoVES

application

vehicular

ARTÌSMigrationmanager

Synchronizationmanager

Data Distrib.manager

Communicationmanager

P.E.U.

Cpu

Cpu

Internet

MoVES

application

vehicular





P.E.U.

Cpu

R-UDP TCP/IP





(wireless) network (wireless) network

SharedMemory

SharedMemory memory

Fig. 1. The PADS simulation framework architecture.


Process (LP) in charge of managing the execution ofa subset of model entities. In general, a single Log-ical Process execution cannot be split over multipleCPUs. Each CPU can allocate many different LPs.The communication support is provided byshared-memory message passing between LPs onthe same PEU, and by LAN network communica-tion up to Internet-based communication, betweenLPs on different PEUs.

The ARTIS layer includes the middleware for themanagement of parallel and distributed simulationsover heterogeneous execution architectures. ARTISis written in C, it provides bindings for C/C++ andJava modeling frameworks. It implements manysolutions to support and to optimize the paralleland distributed simulation of massive, dynamicand complex system models over heterogeneousexecution architectures [25]. ARTIS design andimplementation is still an ongoing activity, andmore details can be found in [24,25]. For the pur-poses of this paper, the ARTIS middleware issketched for illustrating the support provided tothe MoVES framework. The ARTIS middlewareprovides service APIs to MoVES, associated withthe functions implemented in middleware modules.The ARTIS Communication Manager implementsscalable communication functions between LPs,see Fig. 1. This is obtained by exploiting data mar-

shalling solutions and selection of efficient networkprotocols for the physical communication layer.As an example, the communication is implementedby low-overhead message passing between CPUswith shared memory, Reliable-UDP between PEUsconnected by reliable high-performance LANs, andTCP/IP between remote PEUs connected by theInternet. The Synchronization Manager implementspolicies to coordinate the events execution and stateupdates of distributed LPs, that is, to ensure thatsimulated events are processed in causal order.The Data Distribution Manager (DDM) module isin charge of managing the data distribution amongLPs implementing a parallel and distributed simula-tion. In our approach, the distributed simulationarchitecture is based on a global sharing of a subsetof model state information (as an example, the posi-tion of vehicle model entities). Appropriate DDMtechniques are implemented to reduce the updatecost of the shared information. This is a major dif-ference with respect to the MOVES I implementa-tion in [22], realized as a coordinated execution ofsequential simulators, without information-sharingsupport. The new approach introduces controlledcommunication overheads, due to the additionalamount of communication required when periodicstate sharing is performed. On the other hand, thenew approach introduces advantages in (i) the sim-

Digital Areaand Street

Map

PARSER

Initializer

vehicle model

Vehicletype

WirelessInfrastructure

Map

Trace drivendata

position

Driver model

Psycho-physicalmodel

Destination

Wirelessdevices

(Wireless) Network model

Medium and Propagation model

Application

Mobilitymodel

Pathmanagement

street model

streets lanes

intersections

Intersectionpolicy

Traffic light

Device model & technology

Application/Service model

Device type and position

traffic model

Inter-segment model

Intra-segment model

Route management model

status

Synthetic Datadistributions

ARTÌS Simulation Middleware

ApplicationLayer

(Wireless)Network

Layer

VehicularLayer

Transport

Network

MAC/LLC

Physical

MoVES Serialization&

De-serialization Module

ARTÌS MigrationManagement

Fig. 2. The MoVES layer architecture.


plification of real-time monitoring functions, (ii) theexploitation of ARTIS load balancing heuristics, ina way transparent to the MoVES framework, (iii)performance gain, under realistic scenarios andassumptions, as shown in the performance evalua-tion Section 6.1. The Migration Manager moduleimplements different functions, including communi-cation- and computation-load balancing based onruntime migration of model entities between thePEUs. The ARTIS migration implementation andload balancing heuristics are general and parame-ter-based. They have been carefully designed toreduce overheads and to ensure adaptation to themodel and to the simulation architecture character-istics, as will be described in Section 4.

The MoVES layer is written in Java, and itincludes three modeling sub-layers: (1) the applica-tion sub-layer allows the user to model communica-tion-based services and applications, (2) thenetworking sub-layer allows the user to model wire-less (and wired) technologies and protocol architec-tures, network infrastructures (if any), static

devices, medium and signal propagation character-istics, and (3) the vehicular sub-layer allows the userto model vehicles with and without mobile networkdevices, realistic street topologies, driver behavior,traffic scenarios and traffic management policies. Adetailed architecture of the MoVES layer is shownin Fig. 2. The MoVES application sub-layerincludes models of communication-based servicesand applications. Services and applications runningon top of virtual devices can be easily modeled byusing an approach based on socket-like APIs pro-vided by the model of the network protocol suite.At the Network sub-layer, MoVES implements themodel components defining a protocol stack for net-work communication devices. Device technologies,and static device infrastructures can be defined, ifany are needed. The Vehicular sub-layer providesextensible modeling of realistic traffic scenarios,and mobile wireless devices installed on vehicles.Specifically, street model components includestreets, lanes, intersections, intersection policiesand traffic lights. Vehicle model components may


include wireless devices. Driver model componentsinclude psycho-physical attitude, mobility modeland path preferences. The traffic model componentsinclude route management policies, intra- and inter-segment models, sketched in Section 5.1. In addi-tion, the MoVES layer includes service modules: aparser component allows initializing and feedingmodel entities with data obtained from digitalGPS maps, wireless infrastructure maps, and syn-thetic and trace-driven data sources. The serializa-tion and de-serialization module is in charge ofserializing (as byte-streams) the state informationof model entities, to be migrated between differentPEUs by ARTIS. Not shown in Fig. 2, a loggingand data monitoring utility supports saving andanalyzing traces and pre-defined performance fig-ures. A prototype rendering application supportstrace animation (as an example, car mobility andwireless transmissions), both at runtime, and offline.

4. Design and implementation of MoVES framework

In the following we illustrate the design and imple-mentation of the PADS architecture for modelingand simulation of wireless vehicular networksdescribed in Section 3. Verification and validationof the proposed simulation framework and relatedmodels have been performed during the design andimplementation phases, and have demonstrated thecorrect functionality of the simulation executionplatform, and the expected degree of model accuracy.

The simulation framework proposed in thispaper may potentially support different tasks undera layered modeling architecture (see Fig. 2). At thevehicular layer, (i) modeling and analysis of realis-tic, detailed areas of interest, including GPS-basedstreet maps, lanes, intersections, traffic lights andrelated policies, (ii) modeling of vehicles and driverpsycho-physical attitude, (iii) modeling of syntheticor realistic trace-driven mobility models [15], trafficpatterns and positive/negative attraction points (likedense work areas at 8.00am and 6.00pm, respec-tively), (iv) modeling of traffic policies [19,13,29,26] and the impact of ‘‘what if’’ managementassumptions [48]. At the wireless communicationlayer: (v) modeling of wireless communication sce-narios, supported by opportunistic VANETs andinfrastructure-based communication networks[13,14,48], wireless propagation models, and a net-work protocol stack including physical and MACtechnology libraries (e.g., IEEE 802.11) up to theapplication layer. At the application layer, simu-

lated processes of target applications/services canbe virtually immersed in the simulated scenario.

4.1. Time-stepped synchronization scheme

Different PADS communication and synchroni-zation solutions have been proposed. In this workwe consider a conservative time-stepped synchroni-zation scheme [20]: simulated time t is advancedby a time-step only after events scheduled at timet have been managed. A synchronization phase isneeded to deliver new event-messages and to updatemodel entities in order to maintain a causal order ofevents before the next time-step execution. This syn-chronization, managed by the ARTIS Synchroniza-tion module, is one overhead factor of the PADSapproach. We selected the conservative time-stepped scheme because it is simple, it offers theopportunity to implement scalable solutions formulti-layered models, and to implement migration-based load balancing schemes specifically designedto exploit the characteristics of the considered mod-els. Optimistic solutions [20] would need costlystate-saving management, which would not beappropriate under the migration scheme we adoptedfor load balancing in this middleware framework. Inaddition, the time-step duration can be adapted bythe simulation middleware, depending on the gran-ularity of events of interest for the analysis. As anexample, in a vehicular traffic simulation, the time-step could be quite large, since the granularity ofevents can be assumed in the order of a tenth of asecond (e.g., the reaction time of a vehicle driver).In a simulation of a wireless scenario the time-stepcould be in the order of microseconds (e.g., a slot-time of IEEE 802.11 Medium Access Control proto-col). When both models are coupled under the samesimulation framework, as an example, to study theeffects of mobility on wireless communication, orthe effect of wireless communication info-serviceson the mobility of vehicles, the time-step lengthcould be dynamically managed by the simulationmiddleware. This enhancement is considered afuture work activity for the MoVES framework.

4.2. Simulation performance issues

Beyond scalability, one motivation of the PADSapproach is to obtain simulation performancespeedup. The main performance problem of a paral-lel and distributed simulation is to obtain the max-imum advantage of concurrent computation of


events, while reducing communication and synchro-nization overheads needed for PADS coordination.The speedup can be measured as the ratio betweenthe amount of Wall Clock Time (WCT) requiredfor completing the same simulation test under differ-ent simulation approaches. Speedup is a normalizedindex of the overall time required for simulation:speedup values greater than one imply a gain in sim-ulation performance. A speedup can be obtained byincreasing hardware performance, by increasing thedegree of computation parallelism (that is, the num-ber of PEUs), and by reducing the computation,communication and synchronization overheads.Unfortunately, from a speedup viewpoint, tradeoffsmust be considered in the implementation of PADS:in general, the adoption of high-performance hard-ware is costly, and the increase on parallelism trans-lates in high communication and synchronizationoverheads. The optimal balancing of speedup trade-offs is hard to define a priori, because dynamic fac-tors bias the optimal balancing at runtime. As anexample, the rate of interactions between modelentities, and the performance of hardware commu-nication resources (e.g., latency of a shared LAN)may show unpredictable variations during the simu-lation execution, biasing the balancing of communi-cation and synchronization overheads. The amountof computation required to manage simulatedevents and model state updates, and the utilizationof computation resources (e.g., CPU load of sharedmulti-programmed PEU architectures) can vary inan unpredictable way during the simulation execu-tion, biasing the load balancing of computationresources. For these reasons, in the proposed mid-dleware, we do not assume the system and networkresources to be allocated for the simulation in anexclusive way. In addition, we considered two clas-ses of generalized and low-cost solutions to improvethe simulation efficiency: (i) communication- andcomputation-overhead reduction, and (ii) dynamiccommunication- and computation-load balancing.

4.3. Communication and computation overhead

reduction

We summarize two approaches that have beenfollowed for overhead reduction in preliminaryand in current implementation of MoVES.

4.3.1. Terminology

For the sake of clarity, we define some formalterminology. We call a ‘‘simulated area’’ or simply

‘‘area’’ the model of the space populated by simu-lated entities. As an example, in our experiments,the simulated area is the downtown area of the cityof Bologna. A simulated area can be partitioned inregular units called ‘‘cells’’. An arbitrary collectionof adjacent cells is called a ‘‘region’’. A ‘‘mobile

entity’’ is a model entity which moves in the simu-lated area (as an example, a vehicle). The conceptof ‘‘model entity migration’’ is the transfer of (staticor mobile) model-entity’s data-structures andmanagement from one ‘‘home-LP’’ to another ‘‘for-

eign-LP’’. Based on the communication architectureconnecting the PEUs, we define three classes of con-nectivity for LPs: ‘‘SHM-connected LPs’’ are LPsinteracting via (very fast) shared memory, ‘‘LAN-

connected LPs’’ are LPs interacting via (fast) LANtechnology, and ‘‘I-connected LPs’’ are remote LPsinteracting via (slow) Internet connections. The con-cept of ‘‘space proximity’’ of model entities indicatesthat the model entities are near to each other (thatis, Space-neighbors, ‘‘S-neighbors’’) in the simulatedarea. The concept of ‘‘allocation proximity’’ ofmodel entities (that is, Allocation-neighbors, ‘‘A-

neighbors’’ model entities) is intended as the factthat model entities are allocated (and efficientlycommunicating) on the same ‘‘home-LP’’, or ontightly-coupled SHM-connected LPs. A ‘‘Causal

Set’’ (C-set) for event E is the collection of modelentities causally influenced by the occurrence ofevent E. As an example, the C-set of a wirelesstransmission event contains all the wireless receiversin the transmission range.

4.3.2. MoVES I: Communication and computation

overhead reduction

The MoVES I implementation was based on acoordinated execution of concurrent simulators.Each simulator was implemented as a single LP exe-cuted over a single CPU. Each LP was assumed tomanage the set of model entities positioned in a stat-ically defined region of the simulated area. Compu-tation overhead reduction was based on the use ofefficient data structures and smart algorithms forevent-management and the state-update of modelentities. Communication overhead reduction solu-tions were implemented to reduce the cost (latency)of communication and synchronization betweenLPs. To this end, we assume that the hardwarearchitecture supporting LP communication is com-posed of communication resources with differentperformance: as an example, shared memorybetween LPs on the same PEU, local area network


technology between a local cluster of PEUs, up toInternet communication between remote PEUs(see Fig. 1). In general, the support of a PADS mid-dleware includes message passing primitives, fornotifying events and updates to LPs in charge ofmanaging the simulation of model entities. As anexample, an event-message notification is neededbetween two different LPs A and B, when the simu-lated event generated by a model entity X, managedby LP A, has a causal effect on at least one modelentity Y managed by LP B.

In wireless and vehicular models, the amount ofevent correlation and causality between simulatedmodel entities depends on their ‘‘space proximity’’.In other words, the S-neighbor entities in the simu-lation area show a strong causal correlation ofevents. We illustrate this concept with an exampletaken from our model of interest: in a scenariowhere a wireless transmission event is originatedby a vehicle, the event influences all the S-neighborvehicles positioned in the simulated area, withinthe wireless transmission range. In other words,the C-Set of wireless transmission events includesa significant subset of the S-neighbor entities of thetransmitting vehicle. A possible solution to reducecommunication overhead is to send event-messagenotifications only to destinations in the C-Set ofthe generated event. In general, we can assume thisresult is accomplished by well-designed PADS mid-dleware solutions for Data Distribution manage-ment. One additional solution to further reducecommunication overheads is the following: the C-

Fig. 3. Example of map partitions created by three diff

Set of events should be managed by a minimal num-ber of LPs (possibly, one LP), and such LPs shouldbelong to the class of the best connected communi-cation architectures, like SHM-connected LPs. Ifthis condition could be maintained, few efficientcommunications between a few LPs would be suffi-cient to satisfy all the event-communicationrequirements.

Intuitively, one solution is to allocate the modelentities on the LPs, and the LPs on the PEUarchitecture, such that the maximum number ofS-neighbor model entities in the simulated areabecome A-Neighbor model entities, that is, entitiesmanaged over the best connected LPs. This wouldreduce the communication overhead. To implementthis solution, in the MoVES I implementation, wedefined three static policies for dividing the simu-lated area in cells, and to cluster adjacent cells intoregions (see Fig. 3).

The regions assigned to the LPs are defined byaggregating adjacent cells with three possible poli-cies: (P1) Equal Stripes approach, which aggregatesthe map cells into a set of stripes with the same area(number of cells), (P2) Balanced Stripes approach,which produces different regions (stripes) containingthe same number of intersections, and (P3) GreedyApproach, which creates macro-cells of clusteredadjacent cells by equalizing the number of intersec-tions in efficient way. The greedy algorithm pro-duces an equalized N-partition of the map (beingN the number of LPs) with a cell-merging approachdescribed in [22].

erent area mapping algorithms used in MoVES I.


In this approach, each intersection and vehicleentity was assumed as a homogeneous computationunit. By balancing the number of crossings in theregions, and by assuming a uniform distributionof vehicle entities in the area, both balanced stripesand greedy algorithms would create map-depen-dent, computation-balanced partitions. Such finalpartitions (called regions) are depicted as coloredareas in Fig. 3.

To summarize, each region is statically allocatedon one single LP, such that a high number of S-neigh-

bor model entities are allocated as A-neighbors modelentities over SMH-connected LPs. Unfortunately,this scheme has to cope with two problems: (p1) sinceeach LP only manages entities in its allocated region,mobile entities crossing the region boundaries haveto be migrated to a different LP managing the desti-nation region, and (p2) any static ideal allocation ofregions (and their model entities) on LPs, based on‘‘space proximity’’ communication-overheads opti-mization, would be quickly disrupted due to mobileentities changing their position. As an example, amobile vehicle X initially positioned in a cell of anurban area, managed by LP A at simulated time t,may change position at time (t + k) by approachingthe area boundary. In this way, the mobile vehicleentity could be ‘‘surrounded’’ by foreign modelentities managed by the foreign LP B. If the LP Bis allocated on a PEU with low communication per-formance, this would originate a flow of externalevent-messages dynamically increasing the commu-nication overheads.

The migration of model entities was defined as aservice of the MoVES I implementation [22], pri-marily because it was needed for implementing thecoordinated simulators execution (see the problemp1 above). Migration was implicitly used as a com-munication overhead reduction solution, becausethe mobile entity migration maintains the clusteringof the S-neighbor entities under the same LP (thatis, contributing to solve the problem p2).

The migration policy of MoVES I was not basedon heuristics. It was model-based, with a simple andstraightforward definition: when a mobile entity(that is, a vehicle) crosses the area boundary man-aged by its own home-LP, the model entity migra-tion is performed to the destination-LP.

4.3.3. MoVES II: Communication and computation

overhead reduction

Some facts can be summarized before proceedingwith the illustration of the MoVES II implementa-

tion guidelines: (i) in the MoVES I implementation,the migration scheme was model- and area- (region)dependent, and only mobile entities crossing thearea boundaries were migrated; (ii) model causalityin wireless vehicular models may be considered asan abstraction of the ‘‘spatial proximity’’ concept;(iii) mobile entities, like vehicles in simulated vehic-ular scenarios, dynamically change their S-neighborset during the simulation, and in this way, (iv) com-munication overhead reduction schemes based on astatic association between LP and pre-definedregions (approximating a clustering of the S-neigh-bor sets), will see a decrease in their effectiveness,eventually; (v) dense event-generation is producedby wireless communication scenarios, comparedwith traffic simulation scenarios, hence the wirelessscenario assumptions may have a dominating effecton cost reduction policies. All these guidelines havebeen considered in the MoVES II implementation,as illustrated in the following.

In the MoVES II implementation, the modelentity migration has been made almost transparentto the modeling layer: only the serialization andde-serialization of migrating entities (data struc-tures) are provided by MoVES, and migration prim-itives are implemented by the ARTIS middleware.In addition, in the MoVES II implementation, themigration scheme has been extended to all modelentities (that is, not only the mobile ones). More-over, the assumption of a static allocation of arearegions on the LPs has been relaxed. The newapproach allows extending the use of the entitymigration functionality, as a dynamic adaptationscheme, under the control of multiple heuristicsdefined in the ARTIS middleware. This can beexploited to perform dynamic communication over-head reduction, and dynamic computation- andcommunication-load balancing. The new MoVESII implementation allows relaxing many assump-tions made in the first implementation, includingthe fact that now migration can be triggered by heu-ristics, and not only triggered by entities crossingstatic area boundaries. This allows defining newadaptation-based policies, for clustering model enti-ties on LPs under more general and adaptive com-munication reduction guidelines, as an example,based on ‘‘spatial proximity’’ and ‘‘interactionrate’’. Fig. 4 illustrates an example of the newapproach implemented in MoVES II to realizemigration-based adaptive communication-overheadreduction. By looking at Fig. 4, the MoVES simu-lated area shows the position of mobile vehicles

PEU A PEU B

LAN delay

model entities of vehiclesin the simulated area.

Vehicle X executed over PEU Aoriginates a “transmission-event”

LAN delay

Same scenario after migrationof X from A to B

XX

ARTIS

before migration after migration

SHMdelay

SHMdelay

LP A LP BMoVES simulated area MoVES simulated area

PEU A PEU B

ARTISLP A LP B

Fig. 4. The MoVES II migration based on ARTIS heuristics for communication overhead reduction.


(as dots). The color of dots identifies the LP (andthe corresponding PEU) managing the vehicleentity. Black vehicles are managed by LP A onPEU A, and white vehicles are managed by LP Bon PEU B. In this example, we assume that the allo-cation of black and white vehicles was initiallydefined in an optimal way, based on a static area-partition and a static allocation-scheme (that is, allwhite/black vehicles were in the proximity of eachother and all white/black vehicles were managedon LP B/A). Anyway, the simulated mobility ofvehicles could quickly produce a scenario like theone depicted in Fig. 4, during the simulation. InFig. 4(left) the X vehicle is surrounded by whitevehicles. The X vehicle simulates a wireless trans-mission event which must be notified as an event-message to all the S-neighbors entities within thetransmission range (that is, the C-set of the trans-mission event generated from X, denoted with thinarrows departing from X). One local event notifica-tion message is exchanged via shared memorybetween X and the other black vehicle, implementedon the same LP, in Fig. 4(left). The remaining 4external event notification messages will reach theexternal LP B on PEU B, being transmitted viaLAN technology. The Local Communication Ratio(LCR) is defined as the ratio between the number ofcommunications internally performed by an LP,with respect to the total of communications of thesame LP. Intuitively, the LCR illustrates how themigration-based clustering of model entities is ableto reduce the communication overhead of the simu-lation. The LCR for the event management in Fig. 4is defined as LCR = 1/(4 + 1) on the LP A manag-ing X. This means that low-overhead communica-

tion is exploited only with 20% probability by LPB. If we assume that a migration-based heuristic isdefined to implement a communication overheadreduction, the migration of vehicle X could beimplemented from LP A on PEU A, to LP B onPEU B. This would lead to the scenario depictedin Fig. 4(right). In this case, the new LCR isobtained as LCR = 4/(1 + 4) = 80% for the samescenario, and for future execution of eventsoriginated by X on LP B. The idea of maintaininghighly-interacting neighbor entities under themanagement of the same LP can be implemented,by migrating model entities between LPs at run-time, based on dynamic estimates like the LCR. Inthis way, the mobile vehicle X could be dynami-cally migrated on the new home-LP B, whichincludes a majority of interacting entities (seeFig. 4, right).

Figs. 5 and 6 show a visual snapshot of the trafficsimulation in the city of Bologna, by illustrating theeffect of the MoVES II migration scheme. The streetmap shows vehicles positions as dots, and colorsindicate which LP manages the vehicle entity simu-lation. Fig. 5 shows the initial random allocation ofvehicle entities over three simulation LPs. Fig. 6shows the effects of the MoVES II migration basedon ARTIS heuristics for communication overheadreduction obtained after a few simulated time-steps.The migration scheme keeps vehicles clustered, bymaintaining highly-interacting neighbors under themanagement of the same LP.

The new migration scheme is also adopted forload balancing mechanisms in MoVES II, undergeneral heuristics, assumptions and scenarios, asillustrated in the following Section 4.4.2.

Fig. 5. Initial random vehicles allocation over three simulationLPs (dots colors indicate LPs).

Fig. 6. Runtime allocation of vehicles over three simulation LPs,under the effect of MoVES II migration scheme.


4.4. Dynamic communication- and computation-load

balancing

In this section we shortly summarize twoapproaches that have been followed for dynamiccommunication and computation load balancing,in the MoVES I and in the MoVES II implementa-tions, respectively.

4.4.1. MoVES I: Dynamic communication and

computation load balancing

In the MoVES I implementation [22], some modeland system assumptions were considered as the basisfor the proposed computation-load balancing solu-tion. In MoVES I, each LP was assumed to managea statically defined region of the whole simulatedarea. The main assumption concerning load balanc-ing was that the LP’s computation load was propor-tional to the size of respectively simulated regions.More specifically, we assumed that the number ofcomputation-intensive entities in the region man-aged by an LP, like street-crossings and vehicles,were statically balanced in the initialization phaseof model allocation. Three initial area-allocationpolicies were investigated (see Section 4.3.2) to findsolutions for static computation-load balancing in[22]. In addition, computation-load balancing wasbased on the assumption that mobile model-entities(vehicles) had homogeneous computation costand uniform distribution in the simulated area.Pseudo-random mobility models, like the RandomWaypoint, were used to maintain the assumptionson uniform distribution of mobile model entities.As a consequence of the above assumptions, the ini-tial region-allocation phase (Section 4.3.2) wasimplicitly a static policy for computation-load bal-ancing between the LPs. It is worth noting thatCPUs and memory resources supporting the execu-tion of simulation LPs were assumed to be homoge-neous. These assumptions may be difficult to meet inreal simulation system implementations and scenar-ios. As an example, at the system level, PEUs withheterogeneous performance and variable back-ground load would violate the assumptions consid-ered. The same violation may occur if the hot-spottraffic scenarios (like congested traffic areas) andattraction points are present in the model.

4.4.2. MoVES II: Dynamic communication- and

computation-load balancing

MoVES II required a significant re-implementa-tion effort: a distributed simulation framework,managed by the ARTIS PADS middleware, hasreplaced the coordinated execution of independentsimulators realized in MOVES I. The main differ-ence is that the middleware functions now imple-ment a shared state of the whole simulation onevery LP. This fact has advantages and drawbacks.Advantages include the possibility to implementsimulation monitoring functions on each LP, andnew middleware solutions for overhead reduction


and load balancing, based on general assumptions.The main drawback is the additional amount ofcommunication overhead needed to maintain theshared state information. As we will show in thedesign illustration and performance evaluationsections, under worst case scenarios, advantages willbalance the overhead of this new implementation.

In the new implementation, a dynamic computa-tion-load balancing is controlled by the heuristicsdefined in the ARTIS Migration module. Theimplemented approach is simple and general, inorder to capture the essence of the problem at themiddleware layer, without critical assumptions.Each LP is monitored during the simulation bythe load balancing heuristics. The heuristics imple-mentations are discussed in Section 4.5. In a time-stepped synchronization scheme, like the oneadopted in this framework, unified computation-and communication-load balancing can be definedas follows: each LP must complete a round of com-putation and communication (synchronization)with the other LPs, at the same Wall Clock Time.Intuitively, it is important that all LPs get synchro-nized for advancing to the next time-step of simu-lated time, with the minimum Wall Clock Timedifference. In fact, all LPs advance the simulatedtime synchronously with the slowest LP. If one LPis early with respect to others reaching the synchro-nization barrier, this means that some more compu-tation load could be used to fill the gap and toalleviate overloaded LPs. On the other hand, ifone LP is late with respect to others, its computa-tion overload should be eliminated. It is worth not-ing that the above condition is sufficient for loadbalancing, without discriminating between compu-tation- and communication-resources performance.In other words, the real achievement is not givenby a symmetry in the utilization of resources, andby the utilization of all available resources, but isgiven by the homogeneous result in the compositionof two costs: the computation and the communica-tion needed for performing the parallel and distrib-uted simulation. This approach can be effective on aCOTS, heterogeneous, shared and distributed com-putation and communication architecture like theone we assume for our testbed simulations. One ofthe best results achievable with this scheme is thatthe distributed simulation architecture can proac-tively allocate new computations on more availablePEUs, or dismiss adopted PEUs. The proactivemanagement policy depends on the fact that allo-cated PEUs positively or negatively contribute in

the tradeoff between concurrent execution and com-munication overheads. As an example, computa-tionally efficient PEUs could be dismissed becausetheir communication latency is slowing down theoverall system performance. Conversely, one CPUwith only 10% residual CPU time could be includedbecause its small computation potential is effectivewhen coupled with highly efficient communicationarchitectures.

4.5. ARTIS migration heuristics

A detailed illustration of the ARTIS heuristicscan be found in [23,25]. In this work we sketch theimplementation of heuristics used for controllingmigrations. Migrations are the basis for load bal-ancing and communication overhead reductionsolutions implemented in MoVES II. MoVES IIallows the migration of all model entities. In otherwords, in MoVES II, the LPs are no longer stati-cally initialized and associated with regions. Now,the LPs can incrementally migrate a set of entities,under the control of migration heuristics. Thiswould lead to an adaptive clustering of model enti-ties, whose allocation on the LPs is independentwith respect to entity positions in the simulatedarea, and completely based on the ‘‘entity interac-tion’’ concept, as shown in Fig. 4. The set of modelentities allocated on the LPs should dynamicallymap the subset of entities having the highest rateof interactions during the current phase of the sim-ulation. This dynamic mechanism will be exploitedto realize adaptation-based clustering, that is, tocapture the dense interactions (event-messages com-munication) without speculating on the densityassumption of model entities, and by optimizingboth load balancing and communication overheads.In conclusion, the load balancing mechanism isimplemented, without assuming a static partitionand allocation of the simulated area, and by main-taining the ‘‘proximity’’ clustering of model entitiesfor the reduction of communication overheads. Thiswould enable load balanced execution of scenarioscharacterized by non-uniform entity distributionsand event-generation density, like in dynamic hotspots originated at simulation runtime, for whichit was impossible to maintain load balancing inthe MoVES I implementation. As an example, it isworth noting that Italian towns usually have con-centric streets, variable street sizes and intersectiondensity, due to the Middle Ages origins (seeFig. 6); hence hot-spot distribution of vehicles


may easily appear in quite an unpredictable way,caused by traffic jams.

4.5.1. The ARTIS communication overhead reduction

(COR) heuristic

The Communication Overhead Reduction(COR) heuristic is based on a data structure associ-ated with each model entity by the ARTIS middle-ware. A sliding window approach, whose size is asimulation parameter, is used to limit the time-hori-zon of the heuristic. To limit the overhead, the heu-ristic function is called only after a communicationis performed, involving the model entity. After atime-step including at least a communication event,the heuristic evaluates the external LP which wasdestination of the majority of external model entityinteractions, during the recent sliding windowobservation period. If the rate of the external (for-eign-LP) with respect to internal (home-LP) interac-tions is greater than a threshold parameter (whichcan be tuned at runtime), then a migration of themodel entity is performed towards the external-LP. This COR heuristic aims at maintaining a highLocal Communication Ratio (LCR) in the commu-nications between LPs, resulting in lightweight com-munication overheads.

4.5.2. The ARTIS load balancing (LB) heuristicCombined with the COR heuristic, a Load Bal-

ancing (LB) heuristic is implemented by the ARTISmiddleware. The LB heuristic monitors fast and slowLPs, respectively, when implementing a simulatedtime-step synchronization. A distributed voting-likeprocedure is performed to share the local rankingof each LP. In this way, based on a threshold param-eter, an LP can realize if it can accommodate moreload with respect to the average, or if it should de-allocate some load. In both cases, the LP also identi-fies the target LP to migrate model entities to.

5. Testbed models for vehicular and wireless scenarios

In the following we illustrate the main character-istics of the vehicular and wireless communicationmodels implemented in MoVES II, as a perfor-mance analysis testbed of the PADS architecture.

5.1. Vehicular model

The mobility- and area-entities in our model areroad segments, lanes, intersections, traffic lightsand the mobile vehicles. A Parser module can

import maps from the GPS TrackMaker program[49] by creating and allocating data structures inmemory according to selected map partition strat-egy. As proposed in STRAW [13], we identify thecomponents of the road model as: an intra-segmentcomponent, an inter-segment component and aroute management model. The intra-segment modelcontrols vehicular movement into a road segmentaccording to a bottom-up, microscopic approach.Several car-following models have been proposedin the literature to model the cars’ behavior on a sin-gle-lane or multi-lane street. For driver character-ization, we implement the psycho-physical modeldefined in [15], which is based on different parame-ters (speed of the vehicle, speed of the vehicle ahead,safe distance, driver’s mood). Lane-change policiesare currently not supported, but can be easilyincluded in the model. Intersection entities areresponsible for controlling the flow of vehiclesfrom/to connected segments. Inter-segment mobil-ity is modeled by using a kind of admission con-trol/reservation scheme: when the leading vehiclein a lane reaches a threshold distance from thestop-line, it requests permission to occupy the inter-section. The intersection entity then replies based onthe current state of correlated entities (traffic lights,directions of vehicles currently being in the intersec-tion, etc.). If it is allowed to, the vehicle entity passesthe intersection, otherwise, it stops and waits for anotification. The MoVES II Route Managementcomponent manages the road segments that a vehi-cle traverses during the simulation run. A RandomWalk model is considered in the performance anal-ysis reported in this paper: vehicles randomly selectthe next road at each intersection. Other syntheticmobility models, like random waypoint patternsand more realistic traffic-aware and shortest-pathstrategies could be easily be integrated, and will beconsidered in future work. The additional amountof computation required to implement such strate-gies has been synthetically shaped in the perfor-mance analysis section, to measure the relativeimpact of variable distributed computation load.

5.2. Wireless model

In this paper, a prototype definition of a VANETmodel is used to study the analysis potential ofinter-vehicle communication among mobile nodes,and the performance impact of the composition ofvehicular and wireless models on the PADS archi-tecture. Only a subset of the modeling potential of


MoVES II is exploited in this work, and the comple-tion of model libraries is an ongoing activity. In thispaper, the IEEE 802.11 standard [50] is consideredas the enabling wireless technology for VANETs.A mobile wireless device defines a protocol-stackincluding the application layer, the data link andMAC layers. Two wireless signal propagation mod-els are currently available: open-space and two-rayground [35]. Some works on intelligent transporta-tion systems (ITS) analysis consider the real-timedissemination of traffic information as the targetapplication for VANETs. As an example, theSOTIS and the TrafficView projects [11,34] proposescalable traffic information systems that work byallowing a vehicle to periodically send and receivetraffic information from neighbor vehicles. Thisinformation is shown on-board on display systemsfor augmented safety and driver assistance.

In our MoVES model, at the application layer, weconsider two different approaches for data dissemi-nation in vehicular systems [51]. In the first method,a Data Diffusion Protocol (DDP) is modeled as aconstant flow of ping messages, as follows: each node(i) maintains an environment representation (ER) ofthe surroundings, (ii) updates the ER upon receptionof new data, and (iii) periodically broadcasts the ERto the neighbor nodes. Another method for the dis-semination of information is flooding: each nodereceiving a message immediately forwards it in itszone, using sequence-number lists and time-to-livemechanisms to limit useless retransmissions and net-work congestion. The MAC layer of each wirelessdevice implements the Carrier Sense Multiple Accesswith Collision Avoidance (CSMA/CA) MediumAccess Control protocol of the IEEE 802.11 Distrib-uted Coordination Function (DCF). A realistic sim-ulation of inter-vehicle communication in urbanscenarios would require realistic and accurate wire-less propagation models, including the effects ofobstacles like buildings, path-loss, multi-path, shad-owing and diffractions effects. In the analysis shownin this work, a simple two-ray ground propagationmodel is used [35,9], since we are interested on theMoVES migration management, and on PADS per-formance effects, influenced by model factors likethe local causality of broadcast communication,and the vehicle mobility and distribution.

6. Performance analysis

In this section, we present a preliminary illustra-tion of MoVES performance. For space reasons, we

only include a small subset of the results obtainedfrom the simulation experiments, aiming to showdifferent aspects of this work: (i) a general compar-ison of the MoVES I and the MoVES II implemen-tations, from a performance viewpoint, (ii) theimpact of the parallel/distributed approach on themicro-simulation of large-scale highly-populatedurban traffic scenarios, (iii) benefits and drawbacksof adaptation mechanisms included in the PADSmiddleware.

A real-world urban area (the Middle Ages down-town of Bologna) is the target scenario (shown inFig. 6): 1450 road segments and 760 intersectionsin a 5.75 km2 area. Vehicle density is varied, in arange from 2000 to 16000 vehicles in the simulatedarea, both under uniform and hot-spot distribu-tions. This is made in order to evaluate the effective-ness of sequential vs. parallel/distributed simulationapproaches under different model assumptions andcomputation load.

We define both parallel and distributed simula-tion architectures for the analysis. The parallel sim-ulation is implemented over one multiprocessorPEU with shared-memory CPUs. The distributedsimulation is implemented over PEUs connectedvia a LAN. All the parallel simulation experimentshave been executed on a Quadral Intel Pentium IVXeon CPU 1.50 GHz, with Hyper-Threading sup-port activated and 1 GB RAM (Quadral ParallelArchitecture). All the distributed simulation experi-ments have been executed on a cluster of threehomogeneous Dual Intel Pentium IV Xeon,2.8 GHz, 3 GB RAM, interconnected by a Fast-Ethernet (100 Mb/s) LAN (3 · Dual DistributedArchitecture). In all our experiments, each CPUexecutes one single LP. We performed multiple runsof each experiment, and the confidence intervalsobtained with a 90% confidence level are lower than5% the average value of the performance indexesshown.

6.1. MoVES I vs. MoVES II implementations

In this section, we show a performance compari-son of the MoVES I and MoVES II implementa-tions. The PADS execution is parallel, with 4 LPsimplemented on the Quadral Parallel Architecture.The Wall Clock Time (WCT) is used as the perfor-mance index, because speedup is not a correct indexto compare different models/implementations. InFig. 7, we show the WCT needed to complete a test-bed simulation run in a scenario with vehicles uni-

0

2000

4000

6000

8000

10000

2000 3000 4000 5000 6000 7000 8000

Wal

l Clo

ck T

ime

(sec

)

Number of Vehicles

MoVES Architecture Performance: MoVES I vs. MoVES II

MoVES IIMoVES I

Fig. 7. MoVES I vs. MoVES II, uniform vehicle distribution, parallel simulation with 4 LPs (4 CPUs).


formly distributed in the simulated area. TheMoVES I framework design is based on uniformdistribution assumption, hence MoVES I is workingunder its most favorable conditions in this experi-ment. Vehicles are mobile and wireless-enabled(IEEE 802.11 technology). In addition, vehiclesare performing significant computation due to theimplementation of the Data Diffusion Protocol(DDP) described in Section 5.2. In all the tests, tosupport the detailed modeling of wireless protocols,the time-step was fixed to 10�5 s of simulated time.The complex migration management and state shar-ing communication needed by the MoVES II imple-mentation is expected to introduce overheads withrespect to the MOVES I implementation. Fig. 7shows that a small overhead is introduced byMoVES II, but the impact of the overhead is undercontrol for the considered scenario. Fig. 8 shows thesame performance index, obtained under a differentscenario: in the initialization phase, all vehicles areuniformly distributed in the area, and the mobilitymodel of vehicles follows a non-uniform distribu-tion. All vehicles start moving to a small region ofthe simulated area (hot-spot distribution). In thiscase, MoVES I cannot react to computation-loadunbalancing, because a majority of vehicles will beeventually clustered in a small region of the simu-lated area. This means that a subset of LPs will be

managing high computation loads. This effect hasa significant impact on the MoVES I simulation per-formance, as shown in Fig. 8. The MoVES II imple-mentation is faster than MoVES I, in terms ofsimulation WCT. The same qualitative results, notshown for space reasons, have been obtained underdistributed execution architectures.

Results shown in Figs. 7 and 8 demonstrate thatthe MoVES II implementation is scalable and adaptsin an automatic fashion to realistic model dynamics,with low-overheads and significant performancegain, with respect to the MOVES I implementation.For this reason, in the following we will concentrateour analysis on the MoVES II implementation, only.

6.2. Analysis of mobile vehicular scenarios

In the following we have tested the scalability ofthe MoVES II framework, in parallel and in distrib-uted execution environments, when only the vehicu-lar-mobility is simulated (that is, withoutconsidering the computation and synchronizationeffect introduced in the simulation by wireless prop-agation, communications and the DDP applica-tion). In all tests, the time-step was fixed to 1 s ofsimulated time, while the number of LPs involvedin a simulation run is varied between 1 and 6 (inthe distributed scenario), and between 1 and 4 (in

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

2000 3000 4000 5000 6000 7000 8000

Wal

l Clo

ck T

ime

(sec

)

Number of Vehicles

MoVES Architecture Performance: MoVES I vs. MoVES II

MoVES IMoVES II

Fig. 8. MoVES I vs. MoVES II, hot-spot vehicle distribution, parallel simulation with 4 LPs (4 CPUs).

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 1.5 2 2.5 3 3.5 4

Spe

edU

p V

alue

Number of Logical Processes

MoVES II SpeedUp Value. Parallel Scenario

16000 Vehicles8000 Vehicles2000 Vehicles

Fig. 9. Speedup value, parallel execution, variable number of mobile vehicles.


the parallel scenario). The scalability evaluation islimited to 4 and 6 LPs, respectively, due to thecurrent availability of dedicated and homoge-neous resources in our execution platform. This sca-lability analysis will be possibly extended as a future

work. For a given simulation run, the speedup isdefined as the ratio of execution time of a mono-lithic simulation, divided by the execution time ofparallel/distributed simulation [20]. This is consid-ered the main index to evaluate the performance


gain resulting from parallel/distributed computa-tion. The speedup is affected by different factors,including load balancing, memory utilization, syn-chronization and message passing overheadsrequired for the coordination of LPs [20]. Fig. 9

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 2 3

Spe

edU

p V

alue

Number of Log

MoVES II SpeedUp Va

Fig. 10. Speedup value, distributed executio

0

0.5

1

1.5

2

2.5

3

3.5

4

1 1.5 2 2

Spe

edU

p V

alue

Number of Lo

MoVES II SpeedUp V

Fig. 11. Speedup value, parallel execution, variable num

shows the speedup value for a parallel execution(on the Quadral Parallel Architecture), as a functionof the number of LPs. The figure shows that a smallspeedup can be obtained by increasing the numberof LPs. The difference between the scenarios with

4 5 6

ical Processes

lue. Distributed Scenario


n, variable number of mobile vehicles.

.5 3 3.5 4

gical Processes

alue, Parallel Scenario


ber of mobile vehicles, high computation model.


2000, 8000 and 16000 vehicles is not significant. Thisdemonstrates that the computation load in thismodel is marginal, resulting in a low advantage ofthe parallel execution.

Fig. 10 shows the speedup value of a distributedexecution (on the 3 · Dual Distributed Architec-ture), as a function of the number of LPs. In thisdistributed scenario, no speedup is obtained forthe considered simulated model, and the differencesobtained by varying the number of simulated vehi-cles are marginal. No speedup is obtained in thiscase because the modeled scenario has a high degreeof communication overheads (to maintain a sharedstate and to maintain synchronization) associatedwith a low degree of computation. The parallelimplementation in Fig. 9 has slightly better speedupthan the distributed implementation, because thecost of communication (and synchronization) isaffected by the latency of a LAN, which is greaterthan the latency of a shared memory communica-tion. For this scenario, a similar performance isobtained under a monolithic and a PADS simula-tion. Beyond performance results, the scalabilityresulting from resources aggregation in a PADSsimulation for the latter scenario would be useful(i) to support a huge number of vehicles in a widesimulation area, or (ii) to support computationintensive protocol stacks and user applications onthe modeled entities.

0

1

2

3

4

5

6

1 2 3

Spe

edU

p V

alue

Number of Log

MoVES II SpeedUp Val

Fig. 12. Speedup value, distributed execution, variable n

The potential speedup that can be obtained byexploiting concurrent computation in a PADS hasbeen tested by increasing the computation load inthe previous model, as shown in Figs. 11 and 12.Each vehicle now performs ‘‘high computation’’by calculating an adaptive-path between its currentlocation and a target destination, which requires thefrequent execution of a modified version of theDijkstra algorithm on a weighted street graph. Inother words, this has been defined as a simulationtestbed aiming to stress the computation load inthe system. Fig. 11 shows the speedup of a parallelexecution of the ‘‘high computation’’ model: thespeedup obtained approximates the maximum par-allelism achievable with up to four parallel LPs.Fig. 12 shows the speedup index obtained for a dis-tributed execution of the ‘‘high computation’’ modelin a range up to six distributed LPs: again, thespeedup obtained approximates the maximum par-allelism achievable for the execution architectureconsidered. In this case, the speedup is smaller thanthe one obtained in the parallel execution due to theoverhead of communication in distributed (LAN)environment, which is higher than shared memorycommunication. In Figs. 11 and 12 the number ofsimulated vehicles has a remarkable effect on thespeedup, due to the multiplication of the high per-vehicle model computation introduced. In Figs. 9and 10 the effect of the variable number of vehicles

4 5 6ical Processes

ue, Distributed Scenario


umber of mobile vehicles, high computation model.

Table 1Wireless model parameters

Simulation time step 10 ls (IEEE 802.11 SIFS)MAC layer IEEE 802.11 DCFPR factor (%wireless vehicles) 50%Packet size 64 BytePacket rate 1 pkt/sPropagation model

(Avg. Transmission range)(Avg. Carrier Sense range)

Two-Ray Ground100 m300 m

Vehicle Density 1 vehicle · 3125 m2

Nominal Channel Bitrate 2 Mbps


is negligible due to the low model computationrequired.

The speedup obtained in Figs. 11 and 12 is basedon high computation assumption. In the followingwe will show a testbed analysis of a model includingboth vehicular-mobility and IEEE 802.11 wirelesscommunication, that is, a more natural model forthe MoVES II design. Since the implementation ofthe detailed simulation of wireless protocolsrequires a computation intensive activity, we expectthat this simulated scenario would obtain typicalspeedup results in the range of the results obtainedwith previous model assumptions.

6.3. Analysis under mobile vehicular and wireless

scenarios

In this section we test the scalability of theMoVES framework in the simulation of VANETs,that is, when both the mobility factor and thewireless inter-vehicle communication are modeled.Inter-vehicle communication is performed for theimplementation of the Data Diffusion Protocol(DDP) described in Section 5.2. Other modelingparameters of wireless communication appear inTable 1. The time-step of the simulation is deter-mined by the smallest time parameter of the IEEE802.11 MAC protocol [50], that is, a SIFS Time

0

0.5

1

1.5

2

2.5

3

3.5

4

8000 2000

Spe

edup

Val

ue

Number o

MoVES II Speedup: C

Fig. 13. Distributed simulation, six LPs, VANET scenario,

(10 ls): this means a 105 reduction of the time-step size with respect to pure mobility simulationscenarios. Since it is reasonable that in futurevehicular ad hoc networks only a fraction of vehi-cles will be equipped with a wireless interface, weintroduce a penetration rate factor (PR) represent-ing the fraction of vehicles which have a wirelessdevice.

Fig. 13 shows the speedup value when the num-ber of vehicles varies between 2000 and 16000, andthe wireless penetration rate is 50%. In order to havea correct comparison in terms of scalability, the sce-narios with 2000, 8000 and 16000 vehicles have beendefined such that the vehicle density in the simulatedarea is constant, and vehicles are uniformly distrib-uted. The execution architecture is a distributed

16000

f vehicles

OR Heuristic ON/OFF

COR Heuristic ONCOR Heuristic OFF

communication overhead reduction (COR) ON/OFF.


simulation with six LPs, executed on a 3 · Dual Dis-tributed Architecture. In general, parallel architec-tures have the best performance results for thescenario considered. Since the distributed architec-ture is the most critical and interesting for simula-tion, under the communication overheadsviewpoint, we will focus on distributed implementa-tion results. By looking at Fig. 13, we observe thatthe speedup obtained by the distributed implemen-tation when the MoVES II Communication Reduc-tion heuristic is disabled (COR OFF) is 1.0 (that is,equivalent to the monolithic approach) with 2000vehicles, 2.59 with 8000 vehicles, and 3.05 with16000 vehicles. This confirms that a computationintensive activity, like the wireless protocol imple-mentation in the vehicular scenario, could obtainsignificant speedup results. Since the distributed sce-nario is sensitive to the cost of communication (andsynchronization), which is influenced by highlatency of LANs, a communication overhead reduc-tion heuristic (COR) like the one defined in Section4.5.1 would give additional advantages. This is dem-onstrated in Fig. 13, by looking at the speedup val-ues obtained in the vehicular wireless scenario withMoVES II and the COR heuristic activated (CORON): 1.67 with 2000 vehicles, 3.29 with 8000 vehi-cles, and 3.45 with 16000 vehicles. The speedup gaindue to the COR heuristic can be obtained as the dif-ference between the two values, resulting in 0.67,

10

20

30

40

50

60

70

80

90

0 100 200 300 400 50

Loca

l Com

mum

inat

ion

Rat

io (

%)

Simulated

Local Communicatio

Fig. 14. Average LCR of six distributed LPs, vehicu

0.70 and 0.40 for 2000, 8000 and 16000 vehicles,respectively.

Fig. 14 shows the average LCR index (see Sec-tion 4.3.3) for the six LPs realizing the distrib-uted simulation of the vehicular wirelessscenario. This figure demonstrates the effective-ness of the COR heuristic in the reduction ofthe communication overhead, obtained withdynamic, migration-based clustering of mobilemodel entities. The COR-based migration schemeis devoted to maintain an elevated Local Com-munication Ratio (LCR) in the communicationbetween LPs, that is, lightweight communicationoverheads. Fig. 14 shows the static LCR valueobtained between six LPs without the effect ofthe COR heuristic. This is equivalent to theprobability that a local entity allocated on LPX randomly selects a destination entity locatedon the same LP. By assuming that the allocationof model entities on six LPs follows a uniformdistribution, the expected value is LCR = 1/6.Given the initial random allocation of modelentities, the migration-based overhead communi-cation reduction implemented by the COR heuris-tic is able to dynamically cluster many interactingentities on the same LPs. As shown in Fig. 14,the steady state percentage of LCR with CORheuristic activated reaches values between 75%and 80% after a short initial transient phase.

0 600 700 800 900 1000Time (sec)

n Ratio (LCR), 6 LPs

COR ON, 2000 VehiclesCOR ON, 8000 Vehicles

COR ON, 16000 VehiclesCOR OFF

lar wireless scenario, COR heuristic ON/OFF.


7. Conclusions and future work

In this paper, we have illustrated the architectureand design issues of a novel, scalable and efficientframework for the parallel and distributed simula-tion of vehicular ad hoc networks, named MobileWireless Vehicular Environment Simulation(MoVES). MoVES is implemented on top of theARTIS simulation middleware, by including solu-tions for communication overhead reduction, andfor computation- and communication-load balanc-ing based on model-entity migration heuristics forparallel and distributed simulation. The proposedsolutions have shown effective performanceimprovements in the simulation of realistic wirelessvehicular scenarios. We have defined and testedvehicle mobility in a real urban area model, includ-ing driver psycho-physical model, inter-vehicle wire-less communication, signal propagation, IEEE802.11 MAC (DCF) protocol, and traffic dissemina-tion applications. The performance analysis hasdemonstrated the scalability, efficiency and accuracyof MoVES, and some guidelines about the possibledrawbacks and benefits of a parallel/distributedsimulation of communication solutions for mobilewireless VANETs.

Future work includes the improvements of theARTIS middleware support, and MoVES frame-work model libraries. In addition, we will investi-gate solutions to increase the adaptation of thesimulation framework to the characteristics of themodel and the execution architecture. The new solu-tions will be designed to increase the performanceand scalability of the PADS simulation frameworkand the support for layered and module-based mod-eling and simulation of dynamic and massive wire-less vehicular, sensor and mesh networks.

Acknowledgements

This work is partially supported by ItalianMIUR funds under the PRIN project NADIR (De-sign and performance evaluation of protocols anddistributed algorithms for quality of service in meshnetworks) and by University of Bologna funds, un-der the project ESOV (Enabling Services onVehicles).

Authors wish to thank Prof. Nael Abu-Ghazalehand all the anonymous reviewers for their sugges-tions that significantly helped to improve the paperclarity and contribution.

References

[1] J. Blum, A. Eskandarian, L. Hoffman, Challenges ofintervehicle ad hoc networks, IEEE Transactions on Intel-ligent Transportation Systems 5 (4) (2004) 347–351.

[2] Cooperative Vehicle Infrastructure System (CVIS) Inte-grated Project, European Commission, Sixth FrameworkProgramme of Research, 02/2006-01/2010. <http://www.cvisproject.org/>.

[3] M. Nekovee, Sensor networks on the road: the promises andchallenges of vehicular adhoc networks and vehicular grids,in: Proceedings of Workshop on Ubiquitous Computing ande-Research, Edinburgh, UK, May 2005.

[4] IEEE 802.11 TGp, Wireless Access for Vehicular Environ-ments (WAVE).

[5] CALM, Continuous Air interface for Long and Mediumrange, ETSI, ISO TC204 WG16.

[6] IEEE Std 1516-2000: IEEE Standard for modelling andsimulation (M&S) high level architecture (HLA).

[7] W. Franz, R. Eberhardt, T. Luckenbach, FleetNet- Interneton the road, in: Proceedings of ITS’01, Sydney, Australia,October 2001.

[8] Network on Wheels Project. <http://www.prevent-ip.org>.[9] D. Reichardt, M. Miglietta, L. Moretti, P. Morsink, W.

Schulz, CarTalk 2000: safe and comfortable driving basedupon inter-vehicle communication, in: Proceedings ofIVS’02, Versailles, France, June 2002, pp. 545–550.

[10] M. Schulze, G. Nocker, K. Bohm, PReVENT: a Europeanprogram to improve active safety, in: Proceedings ofInternational Conference on ITS Telecom., Brest, France,June 2005.

[11] T. Nadeem, S. Dashtinezhad, C. Liao, L. Iftode, Traffic-View: traffic data dissemination using car-to-car communi-cation, ACM Sigmobile Mobile Computing andCommunications Review 8 (3) (2004) 6–19.

[12] L. Wischhof, A. Ebner, M. Lott, H. Rohling, R. Halfmann,SOTIS: a self-organizing traffic information system, in:Proceedings of VTC’03, Jeju, South Korea, April 2003, pp.2242–2246.

[13] D. Choffnes, F.E. Bustamante, STRAW: an integratedmobility and traffic model for VANETs, in: Proceedings ofCCRTS’05, Mclean, VA, USA, June 2005.

[14] M. Moske, H. Fussler, H. Hartenstein, W. Franz, Perfor-mance measurements of a vehicular ad hoc network, in:Proceedings of VTC’04, Los Angeles, CA, USA, September2004, pp. 2116–2120.

[15] S. Panwai, H. Dia, Comparative evaluation of microscopiccar-following behavior, IEEE Transactions on IntelligentTransportation Systems 6 (3) (2005) 314–325.

[16] T.S. Rappaport, Wireless Communications: Principles andPractice, Prentice Hall, 2002.

[17] R. Klefstad, Y. Zhang, M. Lai, R. Jayakrishnan, R.Lavanya, A scalable, synchronized, and distributed frame-work for large-scale microscopic traffic simulation, in:Proceedings of ITSC’05, Vienna, Austria, September 2005.

[18] H. Liu, W. Ma, R. Jayakrishnan, W. Recker, Distributedlarge-scale network modeling with paramics implementation,in: Proceedings of ITS’05, Vienna, Austria, September 2005,pp. 232–238.

[19] G.D.B. Cameron, G.I.D. Duncan, PARAMICS: parallelmicroscopic simulation of road traffic, Springer Journal ofSupercomputing 10 (1) (1996) 25–53.

http://www.cvisproject.org/

http://www.cvisproject.org/

http://www.prevent-ip.org


[20] R.M. Fujimoto, Parallel and Distributed Simulation Sys-tems, Wiley-Interscience, 1999.

[21] U. Klein, T. Schulze, S. Strassburger, Traffic simulationbased on the high level architecture, in: Proceedings of Proc.of WSC’98, Washington, D.C., USA, December 1998, pp.1095–1104.

[22] L. Bononi, M. Di Felice, M. Bertini, E. Croci, Parallel anddistributed simulation of wireless vehicular ad hoc networks,in: Proceedings of MSWiM’06, Torremolinos, Spain, Octo-ber 2006, pp. 28–35.

[23] L. Bononi, M. Bracuto, G. D’Angelo, L. Donatiello,Performance analysis of a parallel and distributed frame-work for large scale wireless systems’ simulation, in:Proceedings of MSWiM’04, Venice, Italy, October 2004,pp. 52–61.

[24] L. Bononi, M. Bracuto, G. D’Angelo, L. Donatiello,ARTIS: a parallel and distributed simulation middlewarefor performance evaluation, in: Proceedings of ISCIS’04,Kemer-Antalya, Turkey, October 2004, pp. 627–637.

[25] PADS research group, Department of computer science,University of Bologna, Italy, The ARTIS documentationand distribution webpage. <http://pads.cs.unibo.it/>.

[26] VISSIM PTV Simulation Tool. <http://www.english.ptv.de/cgi-bin/traffic/traf_vissim.pl>.

[27] Quadstone PARAMICS, Paramics suite of traffic simulationtools. <http://www.paramics-online.com/>.

[28] Traffic Group inc. SimTraffic 7.0 (Traffic Engineering andTransportation Planning). <http://www.trafficgroup.com/>.

[29] C. Schroth, F. Dotzer, T. Kosch, B. Ostermaier, M.Strassberger, Simulating the traffic effects of vehicle-to-vehicle messaging systems, in: Proceedings of InternationalConference on ITS Telecom., Brest, France, June 2005.

[30] H. Wu, J. Lee, M. Hunter, R. Fujimoto, R.L. Guensler, J.Ko, Efficiency of simulated vehicle-to-vehicle message prop-agation in Atlanta, Georgia, l75 corridor, Journal of theTransportation Research Board 1910 (2005) 82–89.

[31] Scalable Network Technologies, The first natively parallelnetwork simulator, Qualnet 4.0. <http://www.scalable-net-works.com/>.

[32] JiST/SWANS. <http://jist.ece.cornell.edu/>.[33] A.K. Saha, D.B. Johnson, Modeling mobility for vehicular

ad-hoc networks, in: Proceedings of VANET’04, Philadel-phia, PA, USA, October 2004, pp. 91–92.

[34] D.-H. Lee, P. Chandrasekar, A framework for parallel trafficsimulation using multiple instancing of a simulation pro-gram, Intelligent Transportation Systems 7 (3–4) (2002) 279–294.

[35] A. Boukerche, L. Bononi, Simulation and modeling ofwireless, mobile and ad hoc networks, in: S. Basagni, M.Conti, S. Giordano, I. Stojmenovic (Eds.), Mobile Ad HocNetworking, IEEE/Wiley, 2004.

[36] The Network Simulator, ns-2. <http://nsnam.isi.edu/nsnam/>.

[37] Omnet++ Community Site. <http://www.omnetpp.org/>.[38] Opnet Technologies. <http://www.opnet.com/>.[39] R.L. Bagrodia, W.T. Liao, Maisie: a language for the design

of efficient discrete-event simulations, IEEE Transactions onSoftware Engineering 20 (4) (1994) 225–238.

[40] Parallel/ Distributed ns. <http://www.cc.gatech.edu/com-puting/compass/pdns/>.

[41] GTNetS (Georgia Tech Network Simulator). <http://www.ece.gatech.edu/research/labs/MANIACS/GTNetS/>.

[42] Dartmouth SSF (DaSSF). <http://www.cs.dartmouth.edu/research/DaSSF/>.

[43] iSSF Home Page. <http://www.crhc.uiuc.edu/~jasonliu/pro-jects/issf/>.

[44] GTW/TeD/PNNI. <http://www.cc.gatech.edu/computing/pads/teddoc.html>.

[45] O.E. Kelly, J. Lai, N.B. Mandayam, A.T. Ogielski, J.Panchal, R.D. Yates, Parallel simulations of wireless net-works with TED: radio propagation, mobility and protocols,SIGMETRICS 25 (4) (1998) 30–39.

[46] A. Boukerche, S.K. Das, A. Fabbri, SWiMNet: a scalableparallel simulation testbed for wireless and mobile networks,Wireless Networks 7 (5) (2001) 467–486.

[47] R.L. Bagrodia, X. Zeng, M. Gerla, GloMoSim: a library forparallel simulation of large-scale wireless networks, in:Proceedings of PADS’98, Banff, Canada, May 1998, pp.154–161.

[48] S. Eichler, B. Ostermaier, C. Schroth, T. Kosch, Simulationof car-to-car messaging: analyzing the impact on road traffic,in: Proceedings of MASCOTS’05, Atlanta, GA, USA,September 2005, pp. 507–510.

[49] GPSTrackMaker Homepage. <http://gpstm.cpm/>.[50] IEEE 802.11 WG, IEEE Std. 802.11, 1999 ed, Part II:

Wireless LAN MAC and PHY layer specs. 1999.[51] J. Bronsted, L. Krinstensen, Specification and performance

evaluation of two zone dissemination protocols for vehicularad hoc networks, in: Proceedings of ANSS’06, Huntsville,AL, USA, April 2006, pp. 68–79.

Luciano Bononi received the Laureadegree (Summa Cum Laude) in Com-puter Science in 1997, and the PhD inComputer Science in 2002, both from theUniversity of Bologna, Italy. In spring2000 he was a visiting researcher at theDepartment of Electrical Engineering ofthe University of California at LosAngeles. From March 2001 to Septem-ber 2002 he was a postdoc researcher,and since October 2002 he is an assistant

professor at the Department of Computer Science of the Uni-versity of Bologna. His research interests include mobile and
wireless network protocols, standards and architectures, QoS andsecurity, network on chip architectures, communication protocoldesign and analysis, modeling and simulation, parallel and dis-tributed simulation.
Marco Di Felice received the Laureadegree (summa cum laude) in ComputerScience in 2004, from the University ofBologna, Italy. Currently, he is a PhDstudent in the Department of ComputerScience at the University of Bologna,Italy. His research activity includesmobile networking and computing,design, analysis and performance evalu-ation of protocols and architectures forwireless ad hoc networks, vehicular net-

works, modeling and simulation of wireless systems.

http://pads.cs.unibo.it/

http://www.english.ptv.de/cgi-bin/traffic/traf_vissim.pl

http://www.english.ptv.de/cgi-bin/traffic/traf_vissim.pl

http://www.paramics-online.com/

http://www.trafficgroup.com/

http://www.scalable-networks.com/

http://www.scalable-networks.com/

http://jist.ece.cornell.edu/

http://nsnam.isi.edu/nsnam/

http://nsnam.isi.edu/nsnam/

http://www.omnetpp.org/

http://www.opnet.com/

http://www.cc.gatech.edu/computing/compass/pdns/

http://www.cc.gatech.edu/computing/compass/pdns/

http://www.ece.gatech.edu/research/labs/MANIACS/GTNetS/

http://www.ece.gatech.edu/research/labs/MANIACS/GTNetS/

http://www.cs.dartmouth.edu/research/DaSSF/

http://www.cs.dartmouth.edu/research/DaSSF/

http://www.crhc.uiuc.edu/~jasonliu/projects/issf/

http://www.crhc.uiuc.edu/~jasonliu/projects/issf/

http://www.cc.gatech.edu/computing/pads/teddoc.html

http://www.cc.gatech.edu/computing/pads/teddoc.html

http://gpstm.cpm/

r Networks 52 (2008) 155–179 179

Gabriele D’Angelo received the Laureadegree (summa cum laude) in Computer
Science in 2001, and a PhD degree inComputer Science in 2005, both from theUniversity of Bologna, Italy. He is anAssistant Professor at the Department ofComputer Science of the University ofBologna. His research interests includeparallel and distributed simulation, dis-tributed systems, Internet games andcomputer security. He is the author of
several publications on these topics. During the last few years hehas worked on the design and implementation of the ARTIS

L. Bononi et al. / Compute

parallel and distributed simulation middleware.

Michele Bracuto received the Laureadegree in Computer Science in 2002,from the University of Bologna, Italy.He is a research associate at theDepartment of Computer Science of theUniversity of Bologna, Italy. Hisresearch activity includes parallel anddistributed simulation, distributed sys-tems and Quality-Of-Service of multi-media on Internet. During the last fewyears he worked on the design and

implementation of the ARTIS parallel and distributed simulationmiddleware.

Lorenzo Donatiello received the Laureadegree in computer science from Uni-versity of Pisa. From 1984 to 1990, hewas with the Department of ComputerScience, University of Pisa. SinceNovember 1990, he has been a professorof Computer Science at the University ofBologna. From March 1983 to April1984 and from August 1986 to October1986, he was a visiting scientist at theIBM T.J. Watson Research Center,

Yorktown Heights, NY, USA. His research interests includeperformance models of computer and communication systems,
software architecture, parallel and distributed simulation, wire-less networks.

moves: a framework for parallel and distributed simulation...

Documents