network simulator survey

Qualitative Comparison Qualitative Comparison of Network Simulation of Network Simulation

ToolsTools

Author: Björn SchillingAuthor: Björn Schilling

Supervisor: Jörg HähnerSupervisor: Jörg Hähner

Qualitative Comparison of Network Simulation Tools University of Stuttgart

ContentsContents

1.1. IntroductionIntroduction

2/19Qualitative Comparison of Network Simulation Tools University of Stuttgart

ContentsContents


2.2. The Simulators: OPNET Modeler & NS-2The Simulators: OPNET Modeler & NS-2


ContentsContents



3.3. ComparisonsComparisons• NS-2 vs. OPNET ModelerNS-2 vs. OPNET Modeler


ContentsContents



3.3. ComparisonsComparisons• NS-2 vs. OPNET ModelerNS-2 vs. OPNET Modeler• Simulators vs. RealWorld testbedSimulators vs. RealWorld testbed


ContentsContents




4.4. Results & DiscussionResults & Discussion


ContentsContents





5.5. Trade-off with simulation detailsTrade-off with simulation details


ContentsContents






6.6. Summary & ConclusionsSummary & Conclusions


About SimulationAbout Simulation

• Experiments can be expensive, impossible, complexExperiments can be expensive, impossible, complex



• Experiments can be expensive, impossible, complexExperiments can be expensive, impossible, complex• Additionally, most measurements are not Additionally, most measurements are not

repeatablerepeatable




repeatablerepeatable► We need simulatorsWe need simulators





• Simulators model the real worldSimulators model the real world• Ease understanding of itEase understanding of it• Surge its behaviorSurge its behavior• Research its reactions on particular eventsResearch its reactions on particular events






► „„as real as possible“as real as possible“






► „„as real as possible“as real as possible“► Details play an important roleDetails play an important role


Level of detail in simulationLevel of detail in simulation

Correct level of detail is one of the most challenging Correct level of detail is one of the most challenging problemsproblems




Pro:Pro:• More details -> more realismMore details -> more realism




Pro:Pro:• More details -> more realismMore details -> more realism• Too less details can lead to wrong or deluding resultsToo less details can lead to wrong or deluding results





Contra:Contra:• Time-consuming implementation & simulationTime-consuming implementation & simulation





Contra:Contra:• Time-consuming implementation & simulationTime-consuming implementation & simulation• MaintenanceMaintenance






In wireless networks two main points are important:In wireless networks two main points are important:• Algorithms & protocols have to be implemented detailedAlgorithms & protocols have to be implemented detailed


Level of detail in simulationLevel of detail in simulationCorrect level of detail is one of the most challenging Correct level of detail is one of the most challenging

problemsproblems



In wireless networks two main points are important:In wireless networks two main points are important:• Algorithms & protocols have to be implemented detailedAlgorithms & protocols have to be implemented detailed• Simulation environment must be realisticSimulation environment must be realistic


ComparisonsComparisonsWhy are they important?Why are they important?

• Significant differences between simulatorsSignificant differences between simulators




• Only way to see which simulator is more accurate is Only way to see which simulator is more accurate is to make comparisonsto make comparisons





• Important for usersImportant for users Results can influence the choice of the simulatorResults can influence the choice of the simulator





• Important for usersImportant for users Results can influence the choice of the simulatorResults can influence the choice of the simulator

• Important for developersImportant for developers Results help to improve the simulatorResults help to improve the simulator


ContentsContents








OPNET ModelerOPNET Modeler

• Core is a finite state machineCore is a finite state machine• Analytical model is responsible for speeding up the Analytical model is responsible for speeding up the

simulationsimulation• Licences needed -> no open source productLicences needed -> no open source product• GUIGUI• Lots of documentationLots of documentation• Many pre-implemented algorithms and protocolsMany pre-implemented algorithms and protocols





► Primal intention was network simulation for Primal intention was network simulation for companiescompanies





► Primal intention was network simulation for Primal intention was network simulation for companiescompanies

► But comes along with some developer toolsBut comes along with some developer tools


NS-2NS-2

• Second version able to deal with wireless networksSecond version able to deal with wireless networks• Combination of C++ & OTclCombination of C++ & OTcl

• Compromise between performance and user-friendly Compromise between performance and user-friendly designdesign

• Event-based simulatorEvent-based simulator• Open-sourceOpen-source• Less documentationLess documentation• Help can be found in one of the numerous user-Help can be found in one of the numerous user-

groupsgroups


NS-2NS-2

• Second version able to deal with wireless networksSecond version able to deal with wireless networks• Combination of C++ & OTclCombination of C++ & OTcl

• Compromise between performance and user-friendly Compromise between performance and user-friendly designdesign

• Event-based simulatorEvent-based simulator• Open-sourceOpen-source• Less documentationLess documentation• Help can be found in one of the numerous user-Help can be found in one of the numerous user-

groupsgroups

► Mainly used among developersMainly used among developers


ContentsContents








NS-2 vs. OPNET ModelerNS-2 vs. OPNET Modeler

1.1. Simple algorithm needed:Simple algorithm needed:► Flooding implemented in NS-2, OPNET, GloMoSimFlooding implemented in NS-2, OPNET, GloMoSim



1.1. Simple algorithm needed:Simple algorithm needed:► Flooding implemented in NS-2, OPNET, GloMoSimFlooding implemented in NS-2, OPNET, GloMoSim

2.2. SetupSetup• 50 nodes, 10 of them initiating 100 broadcasts50 nodes, 10 of them initiating 100 broadcasts• 1km*1km area1km*1km area• 300 seconds runtime300 seconds runtime• Random Waypoint mobility model (controllable)Random Waypoint mobility model (controllable)• 802.11 MAC layer protocol802.11 MAC layer protocol• Freespace propagationFreespace propagation



• Effective Effective transmission rangetransmission range

• Maximum Maximum difference of factor difference of factor 22

• Complete different Complete different behavior of behavior of simulator GloMoSimsimulator GloMoSim



• Overhead, i.e. Overhead, i.e. produced produced duplicatesduplicates

• Again different Again different behaviorbehavior

• Difference of factor Difference of factor 2,52,5


ContentsContents








Simulators vs. RealWorld testbedSimulators vs. RealWorld testbedWhich simulator comes closest to reality?Which simulator comes closest to reality?

1.1. Simple (wired) Simple (wired) testbed:testbed:

• Possibility to create Possibility to create background trafficbackground traffic

• All the traffic has to All the traffic has to pass a 10Mbit linkpass a 10Mbit link


Simulators vs. RealWorld testbedSimulators vs. RealWorld testbedWhich simulator comes closest to reality?Which simulator comes closest to reality?

1.1. Simple (wired) Simple (wired) testbed:testbed:

• Possibility to create Possibility to create background trafficbackground traffic

• All the traffic has to All the traffic has to pass a 10Mbit linkpass a 10Mbit link

2.2. Two different easy applications make up Two different easy applications make up simulation: CBR & FTPsimulation: CBR & FTP►CBR for simplicity, FTP for popularity & common usageCBR for simplicity, FTP for popularity & common usage


Simulators vs. RealWorld testbedSimulators vs. RealWorld testbedCBR, 5MBit/s, Background = 6MBit/sCBR, 5MBit/s, Background = 6MBit/s

• Network testbed inaccurateNetwork testbed inaccurate• NS-2 more constant than OPNETNS-2 more constant than OPNET• Sudden drop of NS-2 curve after 60 secondsSudden drop of NS-2 curve after 60 seconds


Simulators vs. RealWorld testbedSimulators vs. RealWorld testbedFTP, 10MB File, no background trafficFTP, 10MB File, no background traffic

• 2 simulations (normal / after simulator fine-tuning)2 simulations (normal / after simulator fine-tuning)• First results barely accurateFirst results barely accurate• Better results after fine-tuning, esp. OPNET ModelerBetter results after fine-tuning, esp. OPNET Modeler


ContentsContents








Results & DiscussionResults & DiscussionReasons for the differencesReasons for the differences



• Main factor: Level of detail of physical and MAC Main factor: Level of detail of physical and MAC layer (1 and 2)layer (1 and 2) These layers are implemented in the simulatorThese layers are implemented in the simulator Other layers are implemented by the Other layers are implemented by the

developer/userdeveloper/user





• Level of detail in simulation is decisive for the Level of detail in simulation is decisive for the resultsresults






• Identical implementation of algorithms in bothIdentical implementation of algorithms in bothsimulators is not possiblesimulators is not possible






• Identical implementation of algorithms in bothIdentical implementation of algorithms in bothsimulators is not possiblesimulators is not possible

• Bugs & failures can occurBugs & failures can occur Regular software updates, esp. NS-2Regular software updates, esp. NS-2


ContentsContents








Trade-off with simulation detailsTrade-off with simulation details

• Often integration is difficult & can have far reaching Often integration is difficult & can have far reaching consequences:consequences: DebuggingDebugging Later changeLater change




► Which details do make sense, which do not?Which details do make sense, which do not?► Compromise between performance & correctness Compromise between performance & correctness

must be found!must be found!




► Which details do make sense, which do not?Which details do make sense, which do not?► Compromise between performance & correctness Compromise between performance & correctness

must be found!must be found!

• Typically in todays simulators:Typically in todays simulators: Protocols rather detailedProtocols rather detailed Developers try to integrate as few details as possibleDevelopers try to integrate as few details as possible


ExampleExample

One scenario simulated twice…One scenario simulated twice…• Without idle energy consumptionWithout idle energy consumption• With idle energy consumptionWith idle energy consumption

……with 4 different ad-hocwith 4 different ad-hoc

routing protocolsrouting protocols• AODVAODV• DSRDSR• DSDVDSDV• TORATORA


ExampleExample




► Deluding results in the first scenarioDeluding results in the first scenario


ExampleExample




► Deluding results in the first scenarioDeluding results in the first scenario► Idle energy consumption turns out to be an Idle energy consumption turns out to be an

important and necessary detail!important and necessary detail!


ContentsContents








Summary & ConclusionsSummary & Conclusions

• There are differences between simulators and their There are differences between simulators and their simulation resultssimulation results




• The simulator design influences the simulationThe simulator design influences the simulation





• Too few comparisons have been made to make Too few comparisons have been made to make statements about quality and accuracy of the statements about quality and accuracy of the simulatorssimulators





• Too few comparisons have been made to make Too few comparisons have been made to make statements about quality and accuracy of the statements about quality and accuracy of the simulatorssimulators

• Each of the simulators showed strengths and Each of the simulators showed strengths and enervationsenervations Usage of a specific simulator depending on application Usage of a specific simulator depending on application

areaarea



• Behavior of simulation can depend on small detailsBehavior of simulation can depend on small details



• Behavior of simulation can depend on small detailsBehavior of simulation can depend on small details

• Necessity of a detail is not known before simulation Necessity of a detail is not known before simulation is done. Solutions for this problem?is done. Solutions for this problem?►Use experiences made before for future simulationsUse experiences made before for future simulations►Share knowledge: User-groupsShare knowledge: User-groups



Thank you for your Thank you for your attention!attention!

Are there any questions?Are there any questions?

network simulator survey

Technology

simulation experiments

simulation correct level

simulators simulators

opnet modeler simulators

opnet modeler ns

comparisons ns

possible details

contents introduction