qualnet-opnet-comparison lb 06 06

Confidential and Proprietary - Scalable Network Technologies, Inc. 1

Comparisons of OPNET and QualNet by Third Parties

2Confidential and Proprietary - Scalable Network Technologies, Inc.

SummaryObjective: Fair comparisons of two commercial simulators via studies conducted by third parties for their own projectsThree studies:

Selection of the best simulator on the market for MANET studies:BISON report (by a European research project)Evaluation of OPNET and QualNet on their runtime (sequential) performance: OSPFv2 report (by Boeing Phantom Works)Practicality of parallelizing existing simulators designed for sequential execution: WSC paper (by Georgia Institute of Technologies)

Benchmark projectQuotes from users of both QualNet and OPNET


Architecture of the Simulation Environment*

From a report (June 2003) for the BISON project(Slides prepared by SNT)

A. Montresor, G. D. Caro and P. E. HeegaardUniversita di Bologna (Italy), Telenor Communication AS (Norway),

Technische Universitat Dresden (Germany), IDSIA (Switzerland), Santa Fe Institute (USA)

* Funded by the European Commission under the InformationSociety Technologies Programme of the 5th Framework (1998-2002)


Study 1: OverviewBISON: Biology-Inspired Techniques for Self Organization in Dynamic NetworksDiscrete event simulation as the main tool to study and predict the behavior of communication networksSimulation requirements completely different for different types of networks

Mobile ad hoc networks (MANETs): routing, traffic and mobility patternsOverlay networks: millions of nodes

Identifying the best simulation tool to meet various requirements for the BISON project(SNT note: this slide set summarizes only the MANET section)


Selection CriteriaSelection criteria for mobile ad hoc networks (1-8)


Selection Criteria (Cont’d)Selection criteria for mobile ad hoc networks (9-15)


State-of-the-art of Network Simulators

Possible candidates to build up our simulation environment:

OPNETGloMoSimQualNetNS-2OMNeT++

Investigate the characteristics of these simulators and their general “compliance” to the selection criteria


Study 1 ConclusionsQualNet as Simulation Framework

None of reviewed simulators possesses set of characteristics required by BISON’s research plans and objectivesHowever, QualNet appears as the best compromise:

An extensive set of pre-built models, protocols and algorithmsA good level of acceptance from the scientific communityAn excellent scalabilityA rather good, highly modular, software designA satisfactory level of usability, modifiability and expandabilityAdvanced graphical and mathematical tools for experiment building, monitoring and post-processingGood documentationPossibility of parallel and/or distributed implementations


Conclusions (Cont’d)Other simulators

OMNeT++ does not include an extensive set of models, protocols and algorithmsOnly QualNet and OMNeT++ can scale up to thousands of nodesFor the ease to use/modify/extend, NS-2 scores poorly, QualNet and OPNET are comparable, and OMNeT++ seems the best among all

We leave open the possibility to use either NS-2 or OMNeT++ in the future

If QualNet limits for our research


Study 2: QualNet and OPNET Evaluation

From a report (December 2002) approved for release

(Slides prepared by SNT)T. Henderson and J. Kim

Communications Network TechnologiesBoeing Phantom Works


OSPFv2 model comparison Software

OPNET Modeler 8.0QualNet ver. 3.1e

Hardware1 GHz Pentium 4 machine (single processor) with 1 GB RAM

•No support for virtual paths•No support for AS external LSA•No support for authentication and checksum processing•No support of incremental LSA update•No support of equal cost multipath

QualNet, based on RFC 2328

OSPFv2

•No support for virtual paths•No support for AS external routes•No support for authentication and checksum processing•No support for Type of Service (TOS)-based routing

OPNET library, based on RFC 2328

OSPFv2

Limitations or issuesSourceProtocol


Target ScenarioA simplified topology analogous to the current Naval afloat scenario

Ship_1 (3)

Host (14)

Satellite(23)

Antenna_SAT_1(17)

Host (15)

Antenna_SAT_1(18)

Antenna_LOS_1(21)

Host (16)

Antenna_SAT_1(19)

Antenna_SAT_2(20)

Antenna_LOS_1(22)

Ship_Router(11)

Ship_Router(13)

Ship_Router(12)

NOC_Router_0(6)

NOC_Antenna_0(7)

NOC_Antenna_3(10)

NOC_Antenna_2(9)

NOC_Antenna_1(8)

192.4.1.2

192.1.3.2

192.2.3.2192.1.2.2

192.1.1.2

192.2.3.1

192.3.2.2

192.1.3.1

192.4.3.1

192.4.3.2

192.3.2.1

192.1.2.1

192.4.2.1

192.4.2.2

192.1.1.1

192.4.1.1

Ship_0 (2)

Ship_2 (4)

NOC (1)

Fleet (5)

A0

A0

A0

A0

A0

A0

A0A0

A0

A0

A3

A2

A1


Target Scenario (Cont’d)Two clusters of simulated topology:

SHF=256 kb/s, metric 800 (SAT)

DWTS =256 kb/s, metric 700 (LOS)

INMARSAT=64 kb/s, metric 1300 (SAT)

EHF LDR=32 kb/s, metric 2700 (LOS)

HTTP

CBR FTP

HTTP

CBR FTP

HTTP

FTP

HTTP

CBR FTP

HTTP

CBR

HTTP

CBR FTP

HTTP

CBR FTP

HTTP

CBR

HTTP

CBR FTP

HTTP

CBR

192.5.1.1 192.5.2.1

192.5.3.1 192.5.4.1

192.5.5.1

192.5.6.1 192.5.7.1

192.5.8.1 192.5.9.1

192.5.10.1

192.1.1.1 192.1.2.1

192.1.3.1 192.1.4.1

192.1.5.1

192.1.6.1 192.1.7.1

192.1.8.1192.1.9.1

192.1.10.1

192.3.1.1

192.3.1.2

192.3.2.1

192.3.2.2

192.3.4.1 192.3.4.2

192.3.6.1

192.3.6.2

192.3.7.1

192.3.7.2

192.1.10.2

192.1.8.2192.1.9.2

192.6.1.3192.6.1.2192.6.1.1

192.1.4.2

192.1.2.2

192.1.1.2

192.1.3.2

192.1.5.2

192.1.6.2

192.1.7.2

Ship 0 Ship 1

Ship 2 Ship 3

Ship 4

Ship 5 Ship 6

Ship 7 Ship 8

Ship 9

NOC (Network Operating Center)

FTP

CBR FTP

FTP

FTP

SHF=256 kb/s, metric 800 (SAT)

DWTS =256 kb/s, metric 700 (LOS)

INMARSAT=64 kb/s, metric 1300 (SAT)

EHF LDR=32 kb/s, metric 2700 (LOS)

HTTP

CBR FTP

HTTP

CBR FTP

HTTP

FTP

HTTP

CBR FTP

HTTP

CBR

HTTP

CBR FTP

HTTP

CBR FTP

HTTP

CBR

HTTP

CBR FTP

HTTP

CBR

192.5.1.1 192.5.2.1

192.5.3.1 192.5.4.1

192.5.5.1

192.5.6.1 192.5.7.1

192.5.8.1 192.5.9.1

192.5.10.1

192.1.1.1 192.1.2.1

192.1.3.1 192.1.4.1

192.1.5.1

192.1.6.1 192.1.7.1

192.1.8.1192.1.9.1

192.1.10.1

192.3.1.1

192.3.1.2

192.3.2.1

192.3.2.2

192.3.4.1 192.3.4.2

192.3.6.1

192.3.6.2

192.3.7.1

192.3.7.2

192.1.10.2

192.1.8.2192.1.9.2

192.6.1.3192.6.1.2192.6.1.1

192.1.4.2

192.1.2.2

192.1.1.2

192.1.3.2

192.1.5.2

192.1.6.2

192.1.7.2

Ship 0 Ship 1

Ship 2 Ship 3

Ship 4

Ship 5 Ship 6

Ship 7 Ship 8

Ship 9

NOC (Network Operating Center)

FTP

CBR FTP

FTP

FTP


Validation of Simulation ResultsTraffic load comparison for 20 clusters (500-600 nodes)

Considerable amount of similarity between results of the two simulators except for the OSPF database description traffic(SNT note: while both are valid, a database description packet likely contains multiple LSA headers in real implementation)(SNT note: database description packets: 3.3% of total number of packets)

232,170,190233,095,429434,356442,315TOTAL

10,075,93212,317,74762,52456,450HTTP

185,473,106184,046,684217,929222,285FTP

23,490,00023,490,00015,66015,660CBR

8887921822LSRs

1,134,1801,743,24416,42012,416LSAs

6,996,7886,853,83446,84446,708LSUs

1,123,496767,2481,01114,824DBASE_DESCs

3,875,8003,875,88073,95073,950HELLOs

QualNetOPNETQualNetOPNET

Total Number of BytesTotal Number of PacketsPacket Type


Memory ConsumptionMemory Usage vs Number of Clusters (log scale)

QualNet provides one order of magnitude memory usage reduction over OPNET

1

10

100

1000

2 4 6 8 10 12 14 16 18 20

Number of Clusters

Mem

ory

Usa

ge (M

byte

s)

QualNetOPNET


RuntimeSimulation Time vs Number of Clusters (log scale)

QualNet provides two orders of magnitude simulation time reduction over OPNETEven with 20 clusters, the QualNet simulations executed in better than real-time

5.5e-4

5.5e-3

0.055

0.55

5.5

55.5

50 100 150 200 250 300 350 400 450 500Number of Nodes

Sim

ulat

ion

time/

Rea

l tim

e ra

tio QualNet

OPNET

Fasterthanreal time

5.5e-4

5.5e-3

0.055

0.55

5.5

55.5

50 100 150 200 250 300 350 400 450 500Number of Nodes

Sim

ulat

ion

time/

Rea

l tim

e ra

tio QualNet

OPNET

Fasterthanreal time


Study 3: Parallel Simulations Using OPNET*

Prof. Richard FujimotoGeorgia Institute of Technology

[email protected](excerpted and reformatted)

* Funded by a grant from the National Science Foundation


Executive SummaryKey issues in federating network simulators

Event exchange and synchronizationImplemented using proxies

Static global variablesImplemented using ghost nodes

Dynamic global variables and zero lookahead eventsProblematic: requires substantial revision to model

OPNETSimple models can be readily parallelized (UDP/IP)Many models require substantial revision because of global state and zero lookahead events

[Federating: running the simulator on parallel architectures.]


The Global State Problem

Network simulators are often designed assuming complete, global information of the network; e.g., computing routesA federate simulating a subnetwork has incomplete information

Federate 1 Federate 2

read

?read


Solution Approach: Ghost Nodes

Assume state information does not change during execution (static state)Ghost node: a skeleton model representing a remote LP which caches remote state informationInstantiate ghost node object for each remote LP referenced by an LP

read

read

ghost nodeactual nodeuninstantiated


LookaheadLookahead is defined as the minimum simulation time into the future that a federate can schedule an event

Federate A

Federate B

Federate C

Federate D

Simulation Time

problem: limited concurrencyeach federate must process events in time stamp order

TA

possible messageOK to process

event

not OK to process yet

without lookahead

TA+LA

possible messageOK to process

with lookahead

Lookahead is necessary to allow concurrent processing of events with different time stamps (unless optimistic event processing is used)

SNT Proprietary


Zero Lookahead Events

OPNET: Interrupts generate zero lookahead eventsFTP schedules an event for destination indicating end of transmission

Zero lookahead events essential serialize executionZero lookahead does not happen “in nature;” artifact of the way the network model was developedSolutions

Modify model to eliminate zero lookahead interactionsRequire substantial revision to existing software

Interrupt @ time 100

SimulationTime = 100


Study 3:ConclusionsOPNET models will require significant revision to federate with other network simulations (even, other OPNET federates)Because of the above difficulties, work on OPNET for NSF project has been stoppedExtension to include Opnet appears to be problematic, will add significant risk to the project


Study 4.Tool Comparison by a Commercial European

Organization


Study 4: Comparison

ANSI C / C++Procedural model

design!

ANSI C / C++Procedural model

design!

OO-Design / C++OO-Designed models

Combination of TCL and ANSI C

Language “Model”

No access to simulation kernel

source code.Full access to

modules source code.

No access to simulation kernel

source code.Full access to

modules source code.

Full access to simulation kernel

and modules source code.

Full access to simulation kernel

and modules source code.

Access to Source Code

Sim Platform and modules

documented. “Source code

reading” necessary.

Very good documentation.

Sim. Platform and modules good documented.

Sim. Platform well documented.Rudimentary

modules documentation

available.

Available Documentation

Commercial License (Floating License)Sim. Platform: $

24000Runtime license: $

12000

Commercial License (Floating License)Sim. Platform: $

40000.-Runtime license: $

4000

Commercial license:Sim. Platform: US $

15000.-Support: US $

1800.- p.aNo costs for runtime

license

Open SourceFree Available

License Model / Costs

QualNetProduct 3 OPNET

Product 2 OMNET++Product 1 NS-2


Study 4: Comparison continued…

Yes(Each job requires

one runtime license!)NoYesYes

Usage in Linux GRID Possible

Windows NT/2000Linux

Sun OS (SPARC)

Windows NT/2000Sun OS (SPARC)

Windows NT/2000Linux

Windows 95/98/MEWindows NT/2000

Linux

Supported Platforms

ModerateModerate to highModerate to highVery highAssumed Period

of Vocational Adjustment

Main modules available.

(IPv6 still trail with limited functionality)

Necessary Investment:

QoS Module: US $ 10000

Main modules available.Necessary

Investment:IPv6: US $ 25000.-MPLS: US $ 25000.-


(IPv4, IPv6, MPLS)Routing Protocols missing! (OSPF)


Interoperability between different

modules not given!

Model Library

QualNetProduct 3 OPNET

Product 2 OMNET++Product 1 NS-2


Study 5.Benchmark study by a major defense contractor


Study 5. Benchmarking ProjectNumber of Nodes

A single division is to be modeledDivision - has 4 brigadesBrigade – each brigade has 4 battle groupsBattle group – each has 2 companies and 2 squadronsCompany – each has 16 Warriors organized into 4 platoon commandsSquadron – each has 12 tanks organized into 3 troop commands


Study 5. Benchmarking tools

Voice traffic will use VHF or HF. Data traffic will use HCDR if available. The division HQ, Brigade HQ, and Battle Group HQ will be equipped with HCDR. If an entity is not equipped with HCDR, data will be sent over VHF / HF. Voice traffic always takes priority over data traffic.


Study 5. Benchmarking toolsRadio Parameters

Radio Type, Transmit power, Frequency Range, Data Rate, VHF16W, 30-88 MHz, 2400 bps, HF100W, 1.5-30 MHz, 2400 bps, HCDR 20W, 225-450 MHz, 244 KbpsAll radios have an antenna height of 2 meters. Each radio runs 802.11 and utilizes OSPFv2 routing.


Study 5. Benchmarking toolsData Traffic

All vehicles send a position report every minute. These reports get routed to all other vehicles.Size of position report: assume 100 bytes200 users send a 50 kbyte message every 5 minutes

Voice TrafficVoice traffic on each of the defined voice nets:Division: 50% of the timeBrigade: 45% of the timeBattle group: 45% of the time


Study 6. Benchmarking toolsProtocols required are as follows:

Protocol Simple Jammer models (repeater, look-thru, follow, spot, barrage)RS-423FED-STD-1052 (ARQ)STANAG-4538 HF Data Link, Clustering Protocol, QBL-MSK ModulationTCP, UDP, IPv4, IEEE 802.11 CSMA/CA, EthernetOSPFv2DHCPFECSNMPv1 Interface


Study 5. Benchmarking toolsTerrainGaming area: 120 km by 70 kmTerrain model: Irregular Terrain Model

(Longley-Rice)


Study 5. Benchmarking toolsMovement

Division HQ – noneBrigade HQ - noneBattle group HQ - noneCompany / Squadron group – yes (60% of entities in constant movement)Assume the moving entities move in a square 1 km x 1 km

Scenario DurationThe scenario will run for 30 minutes.


Study 5. Benchmarking toolsScenario preparation

SNT 1.5 weeks and working flawlesslyOPNET 8 weeks and modeled only fractions of the network with no terrain effects.

Problems with OSPF…long run times and crashesNo Linux implementationNo GUI implementation


Study 5. Benchmark Setup

none-111 dBmPropagation Limit

5 mElevation update resolution

01DTED level

Longley-Rice (ITM)Longley-Rice (ITM)Propagation Path Loss

20 km/h30 km/hVehicle speed

100 m100 mPosition update resolution

NE 34.99 -119.01; Fort Hood, display: SW 34.01 -119.7772; contour lines cross

Terrain

802.11b, routing OSPFv2 (RFC 2328), IPv4, TCP, UDP

802.11b, routing OSPFv2 (RFC 2328)

Protocols

19301866Nodes

Windows 2000Windows 2000OS

Dual 1.2GHz Pentium III1GBDual 1.2GHz Pentium III 1GBPC

OPNETQualNet


Study 5. Benchmark Setup (Cont.)

gmsk - BER tablegmsk - BER tableEncoding

nonenonePropagation fading model

statisticalPropagation model

2 meters, omnidirectional2 meters, omnidirectionalAntenna height

20 W20 WTransmit power HCDR

16 W16 WTransmit power VHF

100 bytes / min, staggered100 bytes / min, staggeredPosition reports

200200Data users

50 kbytes every 5 min50 kbytes every 5 minData equivalent

1.25 kbytes every 7.5 sData traffic

16 kbit / s16 kbit / sVoice equivalent

OPNETQualNet


Study 5. BenchmarkTiming Results

Despite using a full OSPF implementation, 10 times the number of terrain posts along a line of sight, and 50% more terrain calculations, QualNet was still more than twice as fast as OPNET.

Slowly increases to 1.2 GB800 MBCore memory usage

3-20 minutes22 secInitialization time

2966 secExecution time (2 CPUs)

1h 46m 7s = 6367 sec3130 secExecution time (1 CPU)

156.5 million175 millionEvents

OPNETQualNet


Study 5. BenchmarkConclusion

QualNet twice as fast as OPNETOPNET numerous crashes, questionable computed delays, twice as slowQualNet delivered on time, no crashes, and computed results were as expected…twice as fast.QualNet executed faster than real timeQualNet is the clear winner


QualNet Customer Experiences“We chose QualNet because it is the most promising tool to realistically simulate battlefield communications in real-time.”

-John Powers, Raytheon

“The parallel simulation kernel has the potential to allow our proposed simulation environment to execute in near real time. The serial simulation execution appears to be very efficient.”

-Highland Systems


Customer Experiences“We ran a model in QualNet of a hybrid mobile

and satellite network with over 500 nodes, running the Open Shortest Path First (OSPF) routing protocol and application traffic. The model ran faster than real time and was about 100 times faster than another widely used network simulator modeling the same scenario. Furthermore, the output from QualNet was validated at a packet-by-packet level.”

Tom Henderson, Boeing Phantom Works


Customer ExperiencesAbout Scalability“I use QualNet because scalability is my #1 concern. My typical simulation scenarios have thousands of nodes. There is no other simulator that can deliver that kind of scalability.”

Chien-Chung Chen, University of Delaware


Customer ExperiencesAbout High Fidelity Models“We recently finished porting our MANET routing protocol

into QualNet. It turns out that QualNet also makes a good stress-testing tool for MANET code. Larger scale tests in QualNet uncovered a bug in one of the routing table calculation routines that did not manifest itself in any of our previous testing.”

Andreas Yankopolus, Scientific Research Corporation

“QualNet is interesting to our team because we do a lot of our own coding. The QualNet code base is clean and intuitive and has easy-to-follow APIs.”

Tom Henderson, Boeing Phantom Works


Customer ExperiencesAbout the QualNet Modeling Environment“[QualNet’s] fixed layer architecture approach

makes adding, deleting, inserting layers possible, but difficult to maintain for non-expert users –this will be important in proposed simulation environment as layer structure needs to be flexible and include layers other than traditional communications layers – SNT is working on improving this in the next few months.”

-Highland Systems


Customer Experiences"My focus is building protocols for reliable and scalable networks in mobile

wireless environments. My productivity is high with QualNet because of the product's detailed physical layer models and diverse set of protocol models. I can build my own models fast using QualNet."

-Chien-Chung Chen, University of Delaware

“Currently a fixed set of layers is in place in QualNet (application, transport, network, link, physical & antenna). A set of API’s is defined to allow communications across the layers. This can be limiting if there is a need/desire to include an intermediate layer such as an adaptation layer –or as an example, if the network layer is providing control directly to an antenna by going around the link layer.”

-Highland Systems(QualNet’s layer-based modeling paradigm is intuitive because it matches the

ISO stack. Models based on layers enable and enforce high fidelity protocol models. There is a learning curve in switching to any new technology, and QualNet is not an exception.)


Customer Experiences“QualNet does not allow use of global variables in parallel

simulations. This is for purposes of running parallel simulations so that strict partitions can be drawn between objects. This can be limiting when it comes to keeping statistics across the network.”

-Highland Systems(QualNet outlaws global variables to enable fast execution

speeds. In QualNet’s defense, there are ways to collect statistics throughout the simulation that still allow parallel execution of a model. In other words, solutions exist for this limitation.)


Customer ExperiencesAbout the Physical Layer Modeling“SNT is focused on the wireless, mobile, DoD environment

and new product developments directly support this.”-Highland Systems

“QualNet’s channel modeling includes path loss, antenna, transmit power, interference, SINR vs. BER, transmission delay, propagation delay. QualNet also includes a TIREM interface, 2-ray reflection model, and turbo-coding. Appears sufficient for proposed simulation environment.”

-Highland Systems


Customer Experiences“Antenna Support [includes] custom 3-D antenna

pattern support with full 3-D specification. SNR Curve/FEC Support--QualNet provides a general SNR vs. BER input capability and has facilities for providing turbo-code inclusion. Potentially useful in detailed communications simulations.”

-Highland Systems


Customer ExperiencesAbout the Company“QualNet is an up and coming tool that is

improving rapidly. The SNT staff is talented and motivated. SNT is also very responsive to user needs and has been very willing to listen to customers.”

-Highland Systems

qualnet-opnet-comparison lb 06 06

Documents

networks manets

simulation framework

simulation environment

conclusions qualnet

best simulation tool

bison report

reviewed simulators

commercial simulators