1 per gunningberg© a real-world test-bed for mobile ad hoc networks: methodology, experimentations,...

1Per Gunningberg©

A Real-World Test-bed for Mobile Ad hoc Networks:

Methodology, Experimentations, Simulation and Results.

Per Gunningberg, Erik Nordström, Christian Rohner, Oskar WiblingUppsala University

2Per Gunningberg©

Background and problem

IETF is standardizing MANET (Mobile Adhoc NETwork) routing protocols:

– One proactive protocol - knowledge about all nodes– One reactive protocol - path on the need basis

Based on experiences from three protocols:– AODV - Adhoc On Demand Distance Vector

(reactive)– DSR - Dynamic Source Routing (reactive)– OLSR - OptiMized Link State Routing(proactive)

Problem: But majority of research done through simulations...

3Per Gunningberg©

Part One

A test-bed for evaluating ad hoc routing protocols.

Close to reality

What to measure and how to analyze

Repeatable experiments

Grey Zone Phenomena

Conclusion

4Per Gunningberg©

The Uppsala Ad hoc Protocol Evaluation Testbed (APE)

People carrying laptops with 802.11b

Suitable for indoor experiments that are hard to model in simulation

5Per Gunningberg©

6Per Gunningberg©

The Ad hoc Protocol Evaluation Testbed (APE)

Execution environment on top of existing OS.– Runs on Win and Linux

Scenarios with movement choreography.

Emphasizes easy management for scaling.

800++ downloads.

7Per Gunningberg©

Laptop instructions (choreography)

node.11.action.0.msg=Test is starting... node.11.action.0.command=start_spyd node.11.action.0.duration=1 node.11.action.1.command=my_iperf c 2 t 330 node.11.action.1.msg=Stay at this location.node.11.action.1.duration=30 node.11.action.2.msg=Start moving! Go to Point A, the end of building. node.11.action.2.duration=75 node.11.action.3.msg=You should have arrived at Point A. Please stay. node.11.action.3.duration=30

8Per Gunningberg©

Measurement procedures

Every node collects SNR from every other node it can hear during the test session

Every event is time stamped

Received Packets/Application results are collected at all nodes

Routing state snapshots are collected

Analysis is done after the test session.

9Per Gunningberg©

Replaying a scenario

• SNR mapped to virtual distance• Each time interval corresponds to a

topological map

T5025 12510

075

150

Point A

Point D

10Per Gunningberg©

APE is a Testbed for…

1. Relative protocol performance comparisons

2. Radio channel effects on ad hoc routing protocols

3. Interactions between hardware, software, protocol, mobility and radio environmentExample: Grey Zone Phenomena

4. Validation of simulation models

5. Generation of traces

11Per Gunningberg©

802.11 Gray Zone Phenomena

AA

10 2

3

Broadcast

Unicast

3

12Per Gunningberg©

Challenge

Results should be reproducible and comparable between tests

It follows that experiments must be repeatable...

...and therefore stochastic factors need to be dealt with

So – what can we achieve?

13Per Gunningberg©

Stochastic Factors in Real World Experiments

Node mobility adds frequent changes in the network topology.– We use choreography and “measure

topology differences”

Variations in hardware and software configuration.– We use identical hardware and software.

Time varying radio environment affects link quality and error rates.

14Per Gunningberg©

Topology differences - visual check

RED = Average mobilityGREEN = 25% with lowest mobilityBLUE = 25% with highest mobility

Experiment 1 Experiment 2

15Per Gunningberg©

Part Two

Evaluating MANET protocols with the APE testbed, simulation and emulation.

Scenarios

UDP, Ping and TCP

Side-by-side comparison

Faulty protocol constructs

Conclusion

16Per Gunningberg©

Coupling Simulation, Emulation and Real World

17Per Gunningberg©

Routing protocols ability to adapt

OLSR - Proactive Link state routing. Monitors neighbors and exchange link state info.

AODV - broadcasts to set up path. HELLO or Link feedback to detect link failure.

DSR - broadcasts with source route. Listens to other traffic to find shorter route. RTT measurements and network ACKs.

React to connectivity changes

18Per Gunningberg©

Emulation

•Same configuration as Real world

•Table-top emulation

•MAC filters force connectivity changes

•Reduces radio and mobility factors

•Interference reduces bandwidth

19Per Gunningberg©

Simulation

•Scenarios recreated in a ns2-simulation using “default” models:

– Transmission range tuned to better match indoors– Mobility with jitter modeled after real world

measurements– Results averaged over 10 runs

•Results provide a baseline

•Can simulations using default (simple) models be used to predict routing protocol performance in complex real world environments?

20Per Gunningberg©

Multidimensional Comparison

Three MANET routing protocol implementations:– OOLSR, AODV-UU, DSR-UU

Three traffic types:– UDP (20 pkts/s CBR)– Ping (20 pkts/s CBR)– TCP (File transfer)

Three mobility scenarios:– End node swap, Relay node swap, Roaming node

Three environments (dimensions):– Simulation, Emulation, Real world

3x3x3x(10 runs) = 270 runs

21Per Gunningberg©

Experimental Test Environment

•Indoors with offices and corridors

•Four nodes (0, 1, 2, 3)

•Four waypoints (A, B, C, D)

•One data stream from node 3 to node 0

22Per Gunningberg©

Relay Node swap

AAA B C D

0 1 2 3

23Per Gunningberg©

Scenarios – Relay Node Swap

•End nodes stationary

•Intermediate nodes changes position

•Hop count never smaller than 2

24Per Gunningberg©

End node swap

AAA B C D

0 1 2 3

25Per Gunningberg©

Scenarios – End Node Swap

•End nodes change positions

•Intermediary nodes stationary

•Hop count changes from 3 to (2) and 1 and back

26Per Gunningberg©

Roaming node

AAA B C D

0 1 2

3

27Per Gunningberg©

Scenarios – Roaming Node

•Roaming node is source node

•All other nodes stationary

28Per Gunningberg©

Results – Relay Node Swap

Simulation Emulation Real World

0

0.2

0.4

0.6

0.8

1

1.2

UDP (Delivery ratio)

AODV-UU DSR-UU OOLSR


0

0.2

0.4

0.6

0.8

1

1.2

Ping (Delivery ratio)



0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

TCP (Mbps)


29Per Gunningberg©

Results – End Node Swap


0

0.2

0.4

0.6

0.8

1

1.2




0

0.5

1

1.5

2

2.5

3

TCP (Mbps)


30Per Gunningberg©

Results – Roaming Node


0

0.2

0.4

0.6

0.8

1

1.2




0

0.2

0.4

0.6

0.8

1

1.2

Ping (Delivery ratio)



0

0.5

1

1.5

2

2.5

3

3.5

4

TCP (Mbps)


31Per Gunningberg©

AODV - UDP - End Node Swap

32Per Gunningberg©

OLSR - UDP - End Node Swap

34Per Gunningberg©

Observations

Simulation and Emulation similar in absolute CBR performance but not in relative protocol ranking

Real world CBR performance is significantly lower

Discrepancy grows with traffic complexity and scenario

TCP performance is orders of magnitude lower for real world compared to simulation

periods of no-progress time in real world

35Per Gunningberg©

Observations (continued)

OLSR tries less hard to re-route and therefore achieves more even performance

•Radio factors account for most of the discrepancy between simulation and real world...

•...but secondary effects, such as cross-layer interactions that are protocol specific, dominate, e.g.:

– Lost HELLOs (AODV)– Excessive buffering (DSR)

36Per Gunningberg©

Protocol comparison conclusion

If one protocol performs better than another in simulation, is it possible to assume the same for

the real world?

NO

37Per Gunningberg©

Latency - Ping - Relay Node

Simulation Real World

0

500

1000

1500

2000

2500

3000

3500

4000

Relay node swap (Ping) RTT std. dev.


41Per Gunningberg©

Conclusions

APE aims to address the lack of real world ad hoc experimental research test-beds

Repeatability addressed at a level that allows relative protocol comparisons

The value of cross-environment evaluation

Revealing of sensing problems leading to instabilities and poor performance

Not visible in simulations

42Per Gunningberg©

The End

Paper:

http://www.it.uu.se/research/group/core/publications/GC_technical_report.pdf

APE testbed:http://apetestbed.sourceforge.net/

The Research group:http://www.it.uu.se/research/group/core/

1 per gunningberg© a real-world test-bed for mobile ad hoc networks: methodology, experimentations,...

Documents

gunningberg slide

gunningberg ape

simulation slide

gunningberg background

point d slide

gunningberg challenge

gunningberg stochastic

spyd node