teacup v0.9 -- a system for automated tcp testbed...

TEACUP v09 ndash A System for Automated TCPTestbed Experiments

Sebastian Zander Grenville ArmitageCentre for Advanced Internet Architectures Technical Report 150414A

Swinburne University of TechnologyMelbourne Australia

szanderswineduau garmitageswineduau

AbstractmdashOver the last few decades several TCP congestion control algorithms weredeveloped in order to optimise TCPrsquos behaviour in certain situations While TCPwas traditionally used mainly for file transfers more recently it is also becomingthe protocol of choice for streaming applications for example video streaming fromYouTube and Netflix is TCP-based [1] [2] and there is an ISO standard for DynamicAdaptive Streaming over HTTP (DASH) [3] However the impact of different TCPcongestion control algorithms on TCP-based streaming flows (within a mix of othertypical traffic) is not well understood Experiments in a controlled testbed allowshedding more light on this issue This report describes TEACUP (TCP ExperimentAutomation Controlled Using Python) version 09 ndash a software tool for runningautomated TCP experiments in a testbed Based on a configuration file TEACUPcan perform a series of experiments with different traffic mixes different bottleneckconfigurations (such as bandwidths queue mechanisms) different emulated networkdelays andor loss rates and different host settings (eg used TCP congestion controlalgorithm) For each experiment TEACUP automatically collects relevant informationthat allows analysing TCP behaviour such as tcpdump files and TCP stack information

Index TermsmdashTCP experiments automated control

CONTENTS

I Introduction 2

II TEACUP Requirements and Design 3II-A Requirements 3II-B Overall design 3II-C Experiment process flow 4II-D Fabric ndash overview and installation 5

III Traffic Generation and Logging 5III-A Traffic sources and sinks 5III-B Loggers 6III-C Host information logged 6III-D Experiment config information

logged 7III-E Log file naming 7

IV Host and Router Configuration 8IV-A Host setup 8IV-B Linux router setup 8

IV-C FreeBSD router setup 10

V Config File 11V-A Config file location 11V-B V_variables 11V-C Fabric configuration 11V-D Testbed configuration 12V-E General experiment settings 12V-F tcpdumppcap configuration 13V-G Topology configuration 13V-H Rebooting OS selection and

power cycling 14V-I Clock offset measurement (broad-

cast pings) 15V-J Custom host init commands 16V-K Router queue setup 16V-L Traffic generator setup 17V-M Available traffic generators 18V-N Mandatory experiment variables 21V-O Experiment-specific variables 22V-P Defining parameters to vary 23

CAIA Technical Report 150414A April 2015 page 1 of 44

V-Q Adding new V_ variables 23

VI Running Experiments 24VI-A Initial steps 24VI-B Example config 24VI-C Running experiments 24VI-D TEACUP variable information

logged 26

VII Host Control Utility Functions 26VII-A Remote command execution 27VII-B Copying files to testbed hosts 27VII-C Installing ssh keys 27VII-D Topology configuration 27VII-E Initialise hosts to a specific oper-

ating system 27VII-F Power cycling 28VII-G Check software installations on

testbed hosts 28VII-H Check testbed host connectivity 28VII-I Check TEACUP config file 28

VIII Experiment Examples 29VIII-A Overview 29VIII-B Scenario 1 Two TCP flows from

data sender to receiver 29VIII-C Scenario 2 Scenario 1 plus auto-

matic booting of hosts 33VIII-D Scenario 3 Scenario 1 plus power

control 34VIII-E Scenario 4 TCP flows with same

bottleneck queue but different delay 35VIII-F Scenario 5 Partially overlapping

TCP flows with different TCP CC 36VIII-G Scenario 6 Two HTTP-based

video streaming clients 37VIII-H Scenario 7 Incast problem scenario 38

IX Extending TEACUP Functionality 39IX-A Additional host setup 39IX-B New TCP congestion control al-

gorithm 39IX-C New traffic generator 40IX-D New data logger 40IX-E New analysis method 41

X Known Issues 42

XI Conclusions and Future Work 43

References 43

I INTRODUCTION

Over the last few decades several TCP congestion controlalgorithms were developed in order to optimise TCPrsquosbehaviour in certain situations While TCP was tradi-tionally used mainly for file transfers more recently itis also becoming the protocol of choice for streamingapplications for example video streaming from YouTubeand Netflix is TCP-based [1] [2] and there is an ISOstandard for Dynamic Adaptive Streaming over HTTP(DASH) [3] However the impact of different TCPcongestion control algorithms on TCP-based streamingflows (within a mix of other typical traffic) is not wellunderstood Experiments in a controlled testbed allowshedding more light on this issue

This report describes TEACUP (TCP Experiment Au-tomation Controlled Using Python) version 09 ndash asoftware tool for running automated TCP experimentsin a testbed It updates the previous tech reports de-scribing TEACUP version 04 version 06 and version08 [4] Based on a configuration file TEACUP canperform a series of experiments with different trafficmixes different bottlenecks (such as bandwidths queuemechanisms) different emulated network delays andorloss rates and different host settings (eg TCP conges-tion control algorithm) For each experiment TEACUPautomatically collects relevant information that allowsanalysing TCP behaviour such as tcpdump files SIFTR[5] and Web10G [6] logs The related technical report [7]describes the design and implementation of our testbedin which we use TEACUP

TEACUP also provides a number of tasks for the anal-ysis of data collected during experiments The analysisaspect of TEACUP is quite distinct from the tasks thatactually run experiments and gather testbed data hencethese are described in the accompanying technical report[8]

This report is organised as follows Section II describesthe overall design of TEACUP ndash the testbed networkmodel process flow and use of Fabric Section IIIoutlines the traffic generators and logging available underTEACUP while host and bottleneck router configurationis summarised in Section IV Section V describes theconfiguration options available when running experi-ments as described in Section VI Section VII describesutility functions that can be used for host maintenanceSection IX outlines how to extend TEACUP Section X


lists known issues Section XI concludes and outlinesfuture work

II TEACUP REQUIREMENTS AND DESIGN

This section describes the design of TEACUP We firstlist the requirements Then we describe the overallfunctional block design and the process flow Finallywe describe the naming scheme for log files

A Requirements

The following paragraphs describe the requirements thatmotivated our design of TEACUP

1) General Create a system to automate performinga series of TCP experiments with varying parame-ters

2) Topology TEACUP runs experiments in a controlledenvironment like that shown in Figure 1 [7] where onebottleneck router interconnects two experiment networksThe experiment networks contain hosts that can actas traffic sources and sinks The router all hosts andTEACUP (on a control server) are also connected toa separate control network1 TEACUP configures hostsbefore each experiment and collects data from hosts aftereach experiment2 Where the control network is a privatenetwork the control server may act as a NAT gatewayenabling all testbed hosts to access the public Internet ifnecessary

3) TCP algorithms The following TCP congestion con-trol algorithms must be supported NewReno and CU-BIC (representing classic loss-based algorithms) Com-poundTCP (Microsoftrsquos hybrid) CDG (CAIArsquos hybrid)and HD (Hamilton Institutesrsquo delay-based TCP) Option-ally other TCPs may be supported All the noted TCPalgorithms are sender-side variants so the destinationcan be any standard TCP implementation

4) Path characteristics The system must be able tocreate bottleneck bandwidth limits to represent likelyconsumer experience (eg ADSL) and some data centrescenarios Emulation of constant path delay and loss ineither direction is required to simulate different condi-tions between traffic sources and sinks The emulationis implemented by the bottleneck node (router)

1The control server also runs a DHCP+TFTP server for the PXEboot setup described in [7]

2If the networks are implemented as VLANs on a suitable Ethernetswitch TEACUP can also automate the assignment of hosts to one orthe other experiment network and associated VLAN (Section V-G)

5) Bottleneck AQM The following Active QueuingManagement (AQM) mechanisms are required Tail-DropFIFO CoDel PIE RED Optionally other AQMmechanisms may be supported Since FreeBSD does notsupport some of the required AQMs the router must runLinux (but to allow comparison TEACUP also has somebasic support for a FreeBSD router) The buffer size mustbe configurable

6) ECN Support It must be possible to enabledisableExplicit Congestion Notification (ECN) on hosts andorrouter

7) Host OS We are OS-agnostic However to coverthe various TCP algorithms and their common im-plementations TEACUP must support scenarios wheresources andor destinations are Windows (Compound)Linux (NewReno CUBIC) FreeBSD (NewReno CU-BIC CDG HD) or Mac OS X (NewReno) Cygwin isused to instrument Windows [7]

8) Traffic loads The following traffic loads must besupported Streaming media over HTTPTCP (DASH-like) TCP bulk transfer UDP flows (VoIP-like) anddata centre queryresponse patterns (one query to Nresponders correlated return traffic causing incast con-gestion)

B Overall design

TEACUP is built on Fabric [9] (Section II-D) a Python(25 or higher) library and command-line tool for stream-lining the remote application deployment or systemadministration tasks using SSH Our design is basedon multiple small tasks that are combined to run anexperiment or a series of experiments (where somemay also be executed directly from the command line)Functions which are not Fabric tasks are ordinary Pythonfunctions Currently we do not make use of the object-oriented capabilities of Python

Figure 2 shows the main functional building blocks Allthe blocks in the diagram have corresponding Pythonfiles However we have summarised a number of Pythonfiles in the helper_functions block The fabfile blockis the entry point for the user It implements tasksfor running a single experiment or a series of similarexperiments with different parameters The fabfile blockalso provides access to all other tasks via Python im-ports


13

13

13

$ $

Figure 1 Testbed topology

The experiment block implements the main functionthat controls a single experiment and uses a number offunctions of other blocks The sanity_checks block im-plements functions to check the config file the presenceof tools on the hosts the connectivity between hostsand a function to kill old processes that are still runningThe host_setup block implements all functions to setupnetworking on the testbed hosts (including basic setup ofthe testbed router) The router_setup block implementsthe functions that set up the queues on the router andthe delayloss emulation The loggers block implementsthe start and stop functions of the loggers such astcpdump and SIFTRweb10g loggers The traffic_gensblock implements the start and stop functions of alltraffic generators

The util block contains utility tasks that can be executedfrom the command line such as executing a command ona number of testbed hosts or copying a file to a number oftestbed hosts The analysis block contains all the post-processing functions that extract measurement metricsfrom log files and plot graphs

C Experiment process flow

The following list explains the main steps that areexecuted during an experiment or a series of experi-ments

I) Initialise and check config fileII) Get parameter combination for next experiment

III) Start experiment based on config and parameterconfiguration

1) Log experiment test ID in fileexperiments_startedtxt

2) Get host information OS NIC names NICMAC addresses

3) Reboot hosts reboot hosts as required giventhe configuration

4) Topology reconfiguration (assign hosts tosubnets)

5) Get host information again OS NIC namesNIC MAC addresses

6) Run sanity checksbull Check that tools to be used existbull Check connectivity between hostsbull Kill any leftover processes on hosts

7) Initialise hostsbull Configure NICs (eg disable TSO)bull Configure ECN usebull Configure TCP congestion controlbull Initialise router queues

8) Configure router queues set router queuesbased on config

9) Log host state log host information (seeSection III-C)

10) Start all logging processes tcpdumpSIFTRWeb10G etc

11) Start all traffic generators start traffic gener-ators based on config

12) Wait for experiment to finish


1313

13 13

13

13

1313

1313

Figure 2 TEACUP main functional blocks

13) Stop all running processes on hosts14) Collect all log files from logging and traffic

generating processes15) Log experiment test ID in file

experiments_completedtxt

IV) If we have another parameter combination to rungo to step III otherwise finish

D Fabric ndash overview and installation

Fabric [9] provides several basic operations for executinglocal or remote shell commands and uploadingdown-loading files as well as auxiliary functions such asprompting the user for input or aborting execution of thecurrent task Typically with Fabric one creates a Pythonmodule where some functions are marked as Fabric tasks(using a Python function decorator)

These tasks can then be executed directly from thecommand line using the Fabric tool fab The entry pointof the module is a file commonly named fabfilepywhich is typically located in a directory from whichwe execute Fabric tasks (if the file is named differentlywe must use fab -f ltnamegtpy) The complete listof tasks available in fabfilepy can be viewed withthe command fab -l Parameters can be passed toFabric tasks however a limitation is that all parametervalues are passed as strings A Fabric task may alsoexecute another Fabric task with Fabricrsquos execute()

function

Sections VI and VII contain a number of examples ofhow to run various TEACUP tasks

TEACUP was developed with Fabric versions 18ndash110but it should also run with newer versions of Fabric Theeasiest way to install the latest version of Fabric is using

the tool pip Under FreeBSD pip can be installed withportmaster

gt portmaster develpy-pip

On Linux pip can be installed with the package man-ager for example on openSUSE it can be installed asfollows

gt zypper install python-pip

Then to install Fabric execute

gt pip install fabric

You can test that Fabric is correctly installed

gt fab --versionFabric 180Paramiko 1120

The Fabric manual provides more information aboutinstalling Fabric [10]

III TRAFFIC GENERATION AND LOGGING

This section describes the implemented trafficsourcessinks and loggers the range of informationlogged and the log file naming schemes

A Traffic sources and sinks

We now describe the available traffic generator functionsHow these can be used is described in more detail inSection V-M

1) iperf The tool iperf [11] can be used to generateTCP bulk transfer flows Note that the iperf client pushesdata to the iperf server so the data flows in the oppositedirection compared to httperf iperf can also be used


13

13131313

Figure 3 TCP video streaming (DASH) behaviour

to generate unidirectional UDP flows with a specifiedbandwidth and two iperfs can be combined to generatebidirectional UDP flows

2) ping This starts a ping from one host to another(ICMP Echo) The rate of pings is configurable forFreeBSD and Linux but limited to one ping per secondfor Windows

3) lighttpd A lighttpd [12] web server is started Thiscan be used as traffic source for httperf-based sinksThere are also scripts to setup fake content for DASH-like streaming and incast scenario traffic However forspecific experiments one may need to setup web servercontent manually or create new scripts to do this Wechoose lighttpd as web server because it is leaner thansome other popular web servers and hence it is easier toconfigure and provides higher performance

4) httperf The tool httperf [13] can be used to simulatean HTTP client It can generate simple request patternssuch as accessing some html file n times per secondIt can also generate complex workloads based on worksession log files (cf httperf man page at [13])

5) httperf_dash This starts a TCP video streaminghttperf client [14] that emulates the behaviour of DASHor other similar TCP streaming algorithms [1] [2] In theinitial buffering phase the client will download the firstpart of the content as fast as possible Then the client willfetch another block of content every t seconds Figure 3shows an illustration of this behaviour The video rateand the cycle length are configurable (and the size ofthe data blocks depends on these)

6) httperf_incast This starts an httperf client for theincast scenario The client will request a block of contentfrom n servers every t seconds The requests are sent

as close together as possible to make sure the serversrespond simultaneously The size of the data blocks isconfigurable Note that emulating the incast problem ina physical testbed is difficult if the number of hosts(number of responders) is relatively small

7) nttcp This starts an nttcp [15] client and an nttcpserver for simple unidirectional UDP VoIP flow emu-lation The fixed packet size and inter-packet time canbe configured Note nttcp also opens a TCP controlconnection between client and server However on thisconnection only a few packets are exchanged before andafter the data flow

B Loggers

Currently there are two types of loggers that log infor-mation on all hosts All traffic is logged with tcpdumpand TCP state information is logged with differenttools

1) Traffic logger tcpdump is used to capture the trafficon all testbed NICs on all hosts All traffic is capturedbut the snap size is limited to 68 bytes by default

2) TCP statistics logger Different tools are used to logTCP state information on all hosts except the routerOn FreeBSD we use SIFTR [5] On Linux we useWeb10G [6] which implements the TCP EStats MIB[16] inside the Linux kernel with our own loggingtool based on the Web10G library For Windows 7 weimplemented our own logging tool which can access theTCP EStats MIB inside the Windows 7 kernel SIFTRhas not been ported to Mac OS X so for Mac OSX we implemented our own logging tool that outputsTCP statistics logs in SIFTR format based on DTrace[17] Note that the DTrace tool collects most but not allstatistics collected by SIFTR (the toolrsquos documentationdescribes the limitations)

The statistics collected by SIFTR are described in theSIFTR README [18] The statistics collected by ourWeb10G client and the Windows 7 EStats logger aredescribed as part of the web100 (the predecessor ofWeb10G) documentation [19]

C Host information logged

TEACUP does not only log the output of traffic genera-tors and loggers but also collects per-host informationThis section describes the information collected for each


host participating in an experiment The following infor-mation is gathered before an experiment is started (wherelttest_id_prefixgt is a test ID prefix and lttest_idgt is a testID)

bull lttest_idgt_ifconfigloggz This file contains the out-put of ifconfig (FreeBSD Linux or MacOSX) oripconfig (Windows)

bull lttest_idgt_unameloggz This file contains the out-put of uname -a

bull lttest_idgt_netstatloggz This file contains informa-tion about routing obtained with netstat -r

bull lttest_idgt_ntploggz This file contains informationabout the NTP status based on ntpq -p (FreeBSDLinux MacOSX or Windows with NTP daemoninstalled) or w32tm (Windows)

bull lttest_idgt_procsloggz This file contains the list ofall running processes (output of ps)

bull lttest_idgt_sysctlloggz This file contains the outputof sysctl -a (FreeBSD Linux or MacOSX) andvarious information for Windows

bull lttest_idgt_config_varsloggz This file contains in-formation about all the V_ parameters in configpy(see Section V) It logs the actual parameter valuesfor each experiment It also provides an indicationof whether a variable was actually used or not(caveat this does not work properly with variablesused for TCP parameter configuration they arealways shown as used)

bull lttest_idgt_host_tcploggz This file contains infor-mation about the TCP congestion control algorithmused on each host and any TCP parameters speci-fied

bull lttest_idgt_tcpmodloggz This file contains the TCPcongestion control kernel module parameter settings(Linux only)

bull lttest_idgt_ethtoolloggz This file contains the net-work interface configuration information providedby ethtool (Linux only)

bull lttest_id_prefixgt_nameip_maploggz This file logshost names and IP addresses (control interface IPaddresses) of all hosts and routers participating inthe series of experiments

The following information is gathered after an experi-ment

bull lttest_idgt_queue_statsloggz Information about therouter queue setup (including all queue disciplineparameters) as well as router queue and filteringstatistics based on the output of tc (Linux) or ipfw

(FreeBSD) Of course this information is collectedonly for the router

D Experiment config information logged

TEACUP also logs information about the configurationof each series of experiments and variables set for eachexperiment The following information is gathered beforean experiment is started (where lttest_id_prefixgt is a testID prefix and lttest_idgt is a test ID)

bull lttest_idgt_config_varsloggz This file logs the val-ues of all V_ variables for each experiment (seeSection VI-D)

bull lttest_id_prefixgt_configtargz This file containscopies of the config file(s) ndash there can be multiplefiles if Pythonrsquos execfile() is used to separate theconfiguration into multiple files

bull lttest_id_prefixgt_tpconf_varsloggz This file listsall TPCONF_ parameters specified in the configfile(s) in a format that can be imported into Python

bull lttest_id_pfxgt_varying_paramsloggz This filecontains a list of all V_ variables varied during theexperiment(s) and their short names used in filenames (see Section VI-D)

E Log file naming

The log file names of TEACUP follow a naming schemethat has the following format

lttest_ID_pfxgt_[ltpar_namegt_ltpar_valgt_]_lthostgt_

[lttraffgen_IDgt_]_ltfile_namegtltextensiongtgz

The test ID prefix lttest_ID_pfxgt is the start ofthe file name and either specified in the config file(TPCONF_test_id) or on the command line (as describedin Section VI)

The [ltpar_namegt_ltpar_valgt_] is the zero to nparameter names and parameter values (separated by anunderscore) Parameter names (ltpar_namegt) should notcontain underscores by definition and all underscoresin parameter values (ltpar_valgt) are changed to hy-phens (this allows later parsing of the names and valuesusing the underscores as separators) If an experimentwas started with run_experiment_single there arezero parameter names and values If an experiment wasstarted with run_experiment_multiple there are asmany parameters names and values as specified inTPCONF_vary_parameters We also refer to the partlttest_ID_pfxgt_[ltpar_namegt_ltpar_valgt_] (thepart before the lthostgt) as test ID


The lthostgt part specifies the IP or name of the testbedhost a log file was collected from This corresponds toan entry in TPCONF_router or TPCONF_hosts

If the log file is from a traffic generator specifiedin TPCONF_traffic_gens the traffic generator numberfollows the host identifier ([lttraffgen_IDgt]) Other-wise lttraffgen_IDgt does not exist

The ltfile_namegt depends on the process whichlogged the data for example it set to lsquounamersquo for unameinformation collected it is set to lsquohttperf_dashrsquo for anhttperf client emulating DASH or it set to lsquoweb10grsquo fora Web10G log file tcpdump files are special in that theyhave an empty file name for tcpdumps collected on hosts(assuming they only have one testbed NIC) or the filename is ltint_namegt_router for tcpdumps collected onthe router where ltint_namegt is the name of the NIC(eg eth1)

The ltextensiongt is either lsquodmprsquo indicating a tcpdumpfile or lsquologrsquo for all other log files All log files are usuallycompressed with gzip hence their file names end withlsquogzrsquo

Figure 4 shows an example name for a tcpdump filecollected on host testhost2 for an experiment where twoparameters (dash tcp) where varied and an examplename for the output of one httperf traffic generator(traffic generator number 3) executed on host testhost2for the same experiment

All log files for one experiment (eg fabrun_experiment_single) or a series of experiments(eg fab run_experiments_multiple) are stored under asub directory named lttest_ID_pfxgt created insidethe directory where fabfilepy is located3

IV HOST AND ROUTER CONFIGURATION

This section provides a general description of how thehosts and router are configured for each experimentand test within an experiment The router implements abottleneck with configurable one-way delays rate limitsand AQM (active queue management)

A Host setup

The setup of hosts other than the router is relativelystraight-forward First each host is booted into the

3Prior to version 047 TEACUP stored all log files in the directorywhere fabfilepy was located

selected OS Then hardware offloading such as TCPsegmentation offloading (TSO) is disabled on testbedinterfaces (all OS) the TCP host cache is disabled(Linux) or configured with a very short timeout andpurged (FreeBSD) and TCP receive and send buffersare set to 2 MB or more (FreeBSD Linux)

Next ECN is enabled or disabled depending on the con-figuration Then the TCP congestion control algorithmis configured for FreeBSD and Linux (including loadingany necessary kernel modules) Then the parametersfor the current TCP congestion control algorithm areconfigured if specified by the user (FreeBSD Linux)Finally custom user-specified commands are executed onhosts as specified in the configuration (these can overrulethe general setup)

B Linux router setup

The router setup differs between FreeBSD (where ipfwand Dummynet is used) and Linux (where tc and netemis used) Our main focus is Linux because Linux sup-ports more AQM mechanisms than FreeBSD and someof the required AQM mechanisms are only implementedon Linux

First hardware offloading such as TCP segmentation of-floading (TSO) is disabled on the two testbed interfacesThen the queuing is configured In the following twosub sections we first describe our overall approach tosetup rate limiting AQM and delayloss emulation forthe Linux router Then we describe an example setup toillustrate the approach in practice

1) Approach We use the following approach ShapingAQM and delayloss emulation is done on the egressNIC (as usual) To filter packets and direct them intothe lsquopipesrsquo we use netfilter [20] The hierarchical tokenbucket (HTB) queuing discipline is used for rate limitingwith the desired AQM queuing discipline (eg pfifocodel) as leaf node (this is similar to a setup mentionedat [21]) After rate shaping and AQM constant loss anddelay is emulated with netem [22] For each pipe we setup a new tc class on the two testbed NICs of the router Ifpipes are unidirectional a class is only used on one of thetwo interfaces Otherwise it is used on both interfacesIn future work we could optimise the unidirectional caseand omit the creation of unused classes

The traffic flow is as follows (also see Figure 10)


tcpdump file collected on testhost2 for an experiment where two parameters where varied20131206-170846_windows_dash_1000_tcp_compound_testhost2dmpgz output of httperf traffic generator (traffic generator 3) executed on testhost220131206-170846_windows_dash_1000_tcp_compound_testhost2_3_httperf_dashloggz

Figure 4 Example file names

1) Arriving packets are marked at the netfilter mangletablersquos POSTROUTING hook depending on sourceand destination IP address with a unique mark foreach pipe4

2) Marked packets are classified into the appropriateclass based on the mark (a one-to-one mappingbetween marks and classes) and redirected to apseudo interface With pseudo device we refer tothe so-called intermediate function block (IFB)device [23]

3) The traffic control rules on the pseudo interface dothe shaping with HTB (bandwidth as per config)and the chosen AQM (as a leaf queuing discipline)

4) Packets go back to the actual outgoing interface5) The traffic control rules on the actual interface

do network delayloss emulation with netem Westill need classes here to allow for pipe specificdelayloss settings Hence we use a HTB again butwith the bandwidth set to the maximum possiblerate (so there is effectively no rate shaping or AQMhere) and netem plus pfifo are used as leaf queuingdiscipline5

6) Packets leave the router via the stack and networkcard driver

The main reason for this setup with pseudo interfaces isto cleanly separate the rate limiting and AQM from thenetem delayloss emulation One could combine both onthe same interface but then there are certain limitationsuch as netem must be before the AQM and [21] reportedthat in such a setup netem causes problems Also a bigadvantage with our setup is that it is possible to emulatedifferent delay or loss for different flows that share thesame bottleneckAQM

2) Example We now show an example of the setupbased on partial (and for convenience reordered) output

4There also is a dummy rule MARK and 0x0 inserted first whichis used to count all packets going through the POSTROUTING hookNote that since this dummy rule has lsquoanywherersquo specified for sourceand destination it also counts packets going through the routerrsquoscontrol interface

5The netem queue has a hard-coded size of 1000 packets whichshould be large enough for our targeted experimental parameter space

of a queue_statsloggz file for a scenario with twounidirectional pipes 8 Mbps downstream and 1 Mbpsupstream both with 30 ms delay and 0 loss

First Figure 5 shows the netfilter marking rules Ourupstream direction is 1721610024 to 1721611024and all packets are given the mark 0x1 Our downstreamdirection is 1721611024 to 1721610024 and allpackets are given the mark 0x2

In the upstream direction our outgoing interface is eth3and we have the tc filters shown in Figure 6 which puteach packet with mark 0x1 in class 11 and redirect itto pseudo interface ifb1

Note that the class setting is effective for eth3 but it willnot lsquostickrsquo across interfaces Hence we need to set theclass again on ifb1 as shown in Figure 7 (again class 11is set if the mark is 0x1)

On ifb1 we use the queuing discipline setup as shownin Figure 8 The HTB does the rate limiting to 1 MbpsHere he leaf queuing discipline is a bfifo (byte FIFO)with a buffer size of 1875 kB

After packets are through the bfifo they are passed backto eth3 where we have an HTB with maximum rate andnetem as leaf queuing discipline (here netem emulates30 ms constant delay) as shown in Figure 9

After leaving netem the packets are passed to the stackwhich then passes them to the NIC driver For the sake ofbrevity we are not describing the downstream directionhere but the principle is exactly the same The onlydifferences are the interfaces used (eth2 and ifb0 insteadof eth3 and ifb1) and the different HTB AQM and netemparameters

Figure 10 shows the flow of packets with the differentsteps carried out in the order of the numbers in paren-thesis The markingclassifying is not shown explicitlyit takes place between step 1 and 2 (netfilter and classon actual interface) and between step 2 and 3 (class onpseudo interface)


gt iptables -t mangle -vLChain POSTROUTING (policy ACCEPT 52829 packets 69M bytes) pkts bytes target prot opt in outsource destination52988 69M MARK all -- any any anywhere anywhere MARK and 0x022774 1202K MARK all -- any any 1721610024 1721611024 MARK set 0x128936 66M MARK all -- any any 1721611024 1721610024 MARK set 0x2

Figure 5 Netfilter marking rules

gt tc -s filter show dev eth3filter parent 1 protocol ip pref 49152 fwfilter parent 1 protocol ip pref 49152 fw handle 0x1 classid 11action order 33 mirred (Egress Redirect to device ifb1) stolenindex 3266 ref 1 bind 1 installed 99 sec used 12 secAction statisticsSent 1520865 bytes 22774 pkt (dropped 0 overlimits 0 requeues 0)backlog 0b 0p requeues 0

Figure 6 tc filter on outgoing network interface

gt tc -d -s filter show dev ifb1filter parent 1 protocol ip pref 49152 fwfilter parent 1 protocol ip pref 49152 fw handle 0x1 classid 11

Figure 7 tc filter on pseudo interface

gt tc -d -s class show dev ifb1class htb 11 root leaf 1001 prio 0 quantum 12500 rate 1000Kbit ceil 1000Kbit burst 1600b1mpu 0b overhead 0b cburst 1600b1 mpu 0b overhead 0b level 0Sent 1520865 bytes 22774 pkt (dropped 0 overlimits 0 requeues 0)rate 62112bit 117pps backlog 0b 0p requeues 0lended 22774 borrowed 0 giants 0tokens 191750 ctokens 191750gt tc -d -s qdisc show ifb1qdisc htb 1 dev ifb1 root refcnt 2 r2q 10 default 0 direct_packets_stat 0 ver 317Sent 1520865 bytes 22774 pkt (dropped 0 overlimits 0 requeues 0)backlog 0b 0p requeues 0qdisc bfifo 1001 dev ifb1 parent 11 limit 18750bSent 1520865 bytes 22774 pkt (dropped 0 overlimits 0 requeues 0)backlog 0b 0p requeues 0

Figure 8 Queuing discipline setup on pseudo interface

We can see that with our setup it is possible to emulatedifferent delay or loss for different flows that sharethe same bottleneckAQM Multiple tc filters on the ifbinterface can classify different flows as the same classso they share the same bottleneck However on the ethinterface we can have one class and one netem queue perflow and the tc filters classify each flow into a differentclass

3) Notes Note that in addition to the buffers mentionedearlier according to [21] the HTB queuing discipline hasa built-in buffer of one packet (that cannot be changed)and the device drivers also have separate buffers

C FreeBSD router setup

While a Linux router is our main focus we also imple-mented a basic router queue setup for FreeBSD

On FreeBSD each pipe is realised as one Dummynetpipe which does the rate shaping lossdelay emulationand queuing (FIFO or RED only) ipfw rules are used toredirect packets to the pipes based on the specified sourceand destination IP parameters If a pipe is unidirectionalthen there is a single pipe ltnumgt ip from ltsourcegtto ltdestgt out rule If a pipe is bidirectional there isan additional pipe ltnumgt ip from ltdestgt to ltsourcegtout rule The pipe number ltnumgt is automatically


gt tc -d -s class show dev eth3class htb 11 root leaf 1001 prio 0 rate 1000Mbit ceil 1000Mbit burst 1375b cburst 1375bSent 1520865 bytes 22774 pkt (dropped 0 overlimits 0 requeues 0)rate 62184bit 117pps backlog 0b 0p requeues 0lended 22774 borrowed 0 giants 0tokens 178 ctokens 178gt tc -d -s qdisc show eth3qdisc htb 1 dev eth3 root refcnt 9 r2q 10 default 0 direct_packets_stat 3 ver 317Sent 1520991 bytes 22777 pkt (dropped 0 overlimits 66602 requeues 0)backlog 0b 0p requeues 0qdisc netem 1001 dev eth3 parent 11 limit 1000 delay 300msSent 1520865 bytes 22774 pkt (dropped 0 overlimits 0 requeues 0)backlog 0b 0p requeues 0

Figure 9 Queuing discipline setup on outgoing interface (netem)

13

13

$

amp

13

()$

Figure 10 Flow of packets through our queue setup

determined by TEACUP A more sophisticated setup forFreeBSD remains future work

V CONFIG FILE

This section describes TEACUPrsquos top-level configpy filethat controls the experiments

A Config file location

By default TEACUP will load the configpy file that is inthe directory where fabfilepy is located ndash the directoryfrom which experiments are started Since TEACUPversion 09 alternative config files can be specifiedusing fabrsquos --set parameter For example if we wantto run a series of experiments with the configuration

in myconfigpy we simple need to run TEACUP asfollows

gt fab --set teacup_config=myconfigpy

run_experiment_multiple

B V_variables

To iterate over parameter settings for each experimentTEACUP uses V_variables These are identifiers ofthe form V_ltnamegt where ltnamegt must consist ofonly letters numbers hyphens (-) or underscores (_)V_variables can be used in router queue settings (seeSection V-K) traffic generator settings (see SectionV-M) TCP algorithm settings (see Section V-N) or hostsetup commands (see Section V-J) Section V-P describeshow to define V_variables

C Fabric configuration

The following settings in the config file are Fabricsettings For a more in-depth description refer to theFabric documentation [9] All Fabric settings are part ofthe Fabric env dictionary and hence are Python variables(and must adhere to the Python syntax)

The user used for the SSH login is specified withenvuser For example

envuser = rsquorootrsquo

The password used for the SSH login is specified withenvpassword The password can be empty if public-keyauthorisation is set up properly eg the public SSH keyof the control PC running TEACUP has been added toall hosts ltusergtsshauthorized_keys files (and the cor-responding private key on the control host is ~sshid_rsaor a file ltkey_filegt specified with fab -i ltkey_filegt or theenvkey_filename configuration parameter [9])


envpassword = rsquotestrootrsquo

The shell used to execute commands is specified withenvshell By default Fabric uses Bash but Bash is notstandard on FreeBSD So TEACUPrsquos default settingis

envshell = rsquobinsh -crsquo

The timeout for an SSH connection is specified withenvtimeout

envtimeout = 5

The number of concurrent processes used for parallelexecution is specified with envpool_size The numbershould be at least as high as the number of hosts unlessthe number of hosts is large in which case we may wantto limit the number of concurrent processes

envpool_size = 10

D Testbed configuration

All TEACUP settings start with the TPCONF_ prefix andare Python variables (and must adhere to the Pythonsyntax)

TPCONF_script_path specifies the path to the TEACUPscripts and is appended to the Python path

TPCONF_script_path = rsquohometestsrcteacuprsquo

Two lists specify the testbed hosts TPCONF_routerspecifies the list of routers Note that prior to TEACUPversion 09 the TPCONF_router list was limited to onlyone router Since version 09 a list of routers can bespecified TPCONF_hosts specifies the list of hostsRouter and hosts can be specified as IP addresses or hostnames (typically for convenience just the name withoutthe domain part)TPCONF_router = [ rsquotesthost1rsquo ]TPCONF_hosts = [ rsquotesthost2rsquo rsquotesthost3rsquo ]

The dictionary TPCONF_host_internal_ip specifies thetestbed IP addresses for each host The hosts (keys)specified must match the entries in the TPCONF_routerand TPCONF_hosts lists exactly The current code doessimple string matching it does not attempt to resolvehost identifiers into some canonical form

TPCONF_host_internal_ip = rsquotesthost1rsquo [rsquo17216101rsquorsquo17216111rsquo]rsquotesthost2rsquo [rsquo17216102rsquo]rsquotesthost3rsquo [rsquo17216103rsquo]

E General experiment settings

TPCONF_test_id specifies the default test ID prefixNote that if the test ID prefix is specified on thecommand line the command line overrules this set-tingnow = datetimedatetimetoday()TPCONF_test_id = nowstrftime(Ymd-HMS)

TPCONF_remote_dir specifies the directory on the re-mote host where the log files (eg tcpdump SIFTR) arestored during an experiment Files are deleted automat-ically at the end of an experiment but if an experimentis interrupted files can remain Currently there is noautomatic cleanup of the directory to remove left-overfiles

TPCONF_remote_dir = rsquotmprsquo

TEACUP performs a very simple time synchronisationcheck at the start (after waiting for more than 30 sec-onds after the hosts were rebooted) It checks the timeoffset between the control host (the host that executesTEACUP) and each testbed host listed in the configfile TPCONF_max_time_diff specifies the maximumtolerable clock offset ie the maximum allowed timedifference in seconds If the clock offset for one ofthe testbed hosts is too large TEACUP will abort theexperiment or series of experimentsTPCONF_max_time_diff = 1

TPCONF_debug_level specifies the debug level for ex-periments At the default level of 0 no debug informationis generated If the variable is set to a higher value debuginformation is generated for example at a debug level of1 Rout files are generated when plotting graphsTPCONF_debug_level = 0

The parameter TPCONF_web10g_poll_interval allowsto specify the poll interval in milliseconds for web10gloggers on Linux or Windows The minimum value is1 ms and the maximum value is 1000 ms The defaultvalue is 10 ms Note that the control of the interval lengthis not very accurate and with small intervals less then10 ms the actual interval will likely be larger than thespecified intervalTPCONF_web10g_poll_interval = 10


F tcpdumppcap configuration

The variable TPCONF_pcap_snaplen sets the snaplength (number of bytes captured by tcpdump per Eth-ernet frame) If not specified the default is 80 bytesSetting TPCONF_pcap_snaplen=0 means lsquocapture allbytesrsquo The following shows an example where we setthe snap length to 128 bytesTPCONF_pcap_snaplen = 128

G Topology configuration

Since version 08 TEACUP can automate the assignmentof each host into one of the two subnets The topol-ogy configuration is EXPERIMENTAL and based on anumber of assumptions around the network setup6 Foran experiment or a series of experiment the topology(re)configuration is (optionally) carried out after themachines haven been rebooted and before any sanitychecks are done

Automatic topology configuration is enabled by settingthe variable TPCONF_config_topology to lsquo1rsquo (If thisvariable is undefined or set to lsquo0rsquo there is no topologyconfiguration)TPCONF_config_topology = rsquo1rsquo

When topology configuration is enabled TEACUP ac-tively (re)configures each hostrsquos test subnet IP ad-dress to that specified in TPCONF_host_internal_ipthen changes the VLAN configuration of the testbedswitchrsquos ports so all hosts are connected to the rightsubnets

Automated VLAN (re)configuration requires SSH ac-cess to the switch with the same SSH user name andpassword as used on all other hosts As many switcheslimit the number of concurrent SSH sessions allowedTEACUP currently performs the configuration of theports on switch sequentially (host by host) Howeverafter the switch has been configured TEACUP carriesout the setup of each host (network interface and routingconfiguration) in parallel to reduce the overall time fortopology configuration

When mapping switch ports to VLANs TEACUP as-sumes we are using 24 subnets and that the third octet ofthe IP address is identical to the VLAN name configured

6The most critical restriction is that switch reconfiguration has onlybeen tested with Dell 5324 and Dell N3000 series switches

on the switch For example a host being configuredwith address 17216102 is mapped to a correspondingVLAN named lsquo10rsquo on the switch

TEACUP also assumes that all hosts are differentiated byappending consecutive numbers to their hostnames Thelowest number h can be chosen arbitrarily For exampleif we have two hosts they could be named rsquotesthost1rsquoand rsquotesthost2rsquo TEACUP further assumes that hosts areconnected to consecutive ports of the switch in the sameorder as their hostname numbering For example if wehave testhost1 and testhost2 and testhost1 is connectedto switch port n then testhost2 must be connected toswitch port n+ 1

The address or host name of the switch can beconfigured with the variable TPCONF_topology_switchTo map a host to the corresponding port thevariables TPCONF_topology_switch_port_prefixand TPCONF_topology_switch_port_offset (alsocalled o) can be defined The name of theswitch port used will be the string specified inTPCONF_topology_switch_port_prefix concatenatedwith the the number h + o The following shows thedefault configurationTPCONF_topology_switch = rsquoswitch2rsquoTPCONF_topology_switch_port_prefix = rsquoGi10rsquoTPCONF_topology_switch_port_offset = 5

The above configuration assumes that testhost1 is con-nected to switch port lsquoGi106rsquo and testhost2 is connectedto switch port lsquoGi107rsquo on a switch with the host namelsquoswitch2rsquo (which has SSH configuration enabled)

Topology configuration works with all supported OS(Linux FreeBSD WindowsCygwin and Mac OS X)However the switch reconfiguration code has onlybeen tested with Dell 5324 and Dell N3000 seriesswitches

Note that the network interfaces on the hosts are cur-rently hard-coded inside TEACUP For example forFreeBSD hosts the topology setup method assumes thatem1 is the testbed interface The interface names canonly be changed by modifying the Python code

1) Link speed selection As of version 09 TEACUPcan set the link speed of the Ethernet links betweenthe switch and testbed NICs There are four settings forthe speed lsquo10rsquo (10 Mbps 10baseT) lsquo100rsquo (100 Mpbs100baseT) lsquo1000rsquo (1000 Mbps 1000baseT) and lsquoautorsquo(default which should result in 1000baseT) The variable


TPCONF_linkspeed allows to configure the link speedfor all hosts For example if all hosts should use a linkspeed of 100 Mbps (100baseT) we can specifyTPCONF_linkspeed = rsquo100rsquo

However we can also configure host-specific link speedswith the dictionary TPCONF_host_linkspeed Each entrymust have as index a host name (as defined in TP-CONF_router or TPCONF_hosts) and as value one of thepossible speed settings explained above For example iftesthost1 should be set 100 Mbps and testhost2 to 1000Mbps we can defineTPCONF_host_linkspeed =

rsquotesthost1rsquo rsquo100rsquorsquotesthost2rsquo rsquo1000rsquo

TPCONF_host_linkspeed entries always overrule TP-CONF_linkspeed If neither variable is specified thedefault link speed setting is lsquoautorsquo which should resultin 1000baseT assuming Gigabit Ethernet NICs

Note that the link speed setting is persistent for LinuxFreeBSD and Windows However for Mac OS X thespeed will reset to 1000baseT after a reboot (normallythis should not be an issue as hosts are only rebootedbefore experiments)

H Rebooting OS selection and power cycling

With suitable external support TEACUP enables auto-mated rebooting of testbed hosts into different operatingsystems and forced power cycling of hosts that havebecome unresponsive

1) PXE-based rebooting TEACUPrsquos PXE-based re-booting process is explained in [7]

The IP address and port of the TFTP or HTTP serverthat serves the ipxe configuration files during thePXE boot process is specified with the parameter TP-CONF_tftpserver Both IP address and port number mustbe specified separated by a colon For exampleTPCONF_tftpserver = rsquo1011118080rsquo

The path to the directory that is served bythe TFTP or HTTP server is specified withTPCONF_tftpboot_dir

TPCONF_tftpboot_dir = rsquotftpbootrsquo

Omitting TPCONF_tftpboot_dir or setting it to an emptystring disables PXE booting TPCONF_host_os and TP-CONF_force_reboot (see below) are then ignored

2) OS and kernel selection The TPCONF_host_os dic-tionary specifies which OS are booted on the differenthosts The hosts (keys) specified must match the en-tries in the TPCONF_router and TPCONF_hosts listsexactly (any host specified in TPCONF_host_os must bespecified in either TPCONF_router or TPCONF_hosts)The current code does simple string matching it doesnot attempt to resolve host identifiers in some canonicalform

TEACUP currently supports selecting from four dif-ferent types of OS lsquoLinuxrsquo lsquoFreeBSDrsquo lsquoCYG-WINrsquo (Windows) and lsquoDarwinrsquo (Mac OS X) Se-lecting the specific Linux kernels to boot is sup-ported with the TPCONF_linux_kern_router and TP-CONF_linux_kern_hosts parameters (see below) TheOS selection occurs once during reboot and cannotsubsequently be varied during a given experimentTPCONF_host_os = rsquotesthost1rsquo rsquoLinuxrsquorsquotesthost2rsquo rsquoFreeBSDrsquorsquotesthost3rsquo rsquoLinuxrsquo

TPCONF_linux_kern_router specifies the Linux kernelbooted on the router and TPCONF_linux_kern_hostsspecifies the Linux kernel booted on all other hostsThe name is the kernel image name minus the startinglsquovmlinuz-rsquo so the name starts with the version numberIt is also possible to specify the keywords lsquorunningrsquo orlsquocurrentrsquo to tell TEACUP to use the same kernel that iscurrently running on the host (this requires that the hostis already running Linux)TPCONF_linux_kern_router = rsquo31418-10000hzrsquoTPCONF_linux_kern_hosts = rsquo398-web10grsquo

The parameter TPCONF_os_partition allows us to spec-ify the partitions for the different operating systems onthe hosts (in GRUB4DOS format since our TEACUP-based testbed uses GRUB4DOS to reboot into the desiredOS) For example if we have the configuration belowthen TEACUP will attempt to boot from the first partitionof the first disk if WindowsCygwin is selected bootfrom the second partition of the first disk if Linux isselected and boot from the third partition of the firstdisk if FreeBSD is selectedTPCONF_os_partition = rsquoCYGWINrsquo rsquo(hd00)rsquorsquoLinuxrsquo rsquo(hd01)rsquorsquoFreeBSDrsquo rsquo(hd02)rsquo


3) Boot behaviour and timeout If TP-CONF_force_reboot is set to lsquo1rsquo all hosts willbe rebooted If TPCONF_force_reboot is set tolsquo0rsquo only hosts where the currently running OS(or kernel in case of Linux) is not the desiredOS (or kernel in case of Linux) as specified inTPCONF_host_os (and TPCONF_linux_kern_router orTPCONF_linux_kern_hosts) will be rebooted

TPCONF_force_reboot = rsquo1rsquo

TPCONF_boot_timeout specifies the maximum time inseconds (as integer) a reboot can take If the rebootedmachine is not up and running the chosen OS afterthis time the reboot is deemed a failure and the scriptaborts unless TPCONF_do_power_cycle is set to lsquo1rsquo(see below)

TPCONF_boot_timeout = 100

4) Forced power cycling If TEACUP-supported powercontrollers are installed and TPCONF_do_power_cycleis set to lsquo1rsquo a host is power cycled if it does notreboot within TPCONF_boot_timeout seconds If TP-CONF_do_power_cycle is omitted or set to lsquo0rsquo there isno power cycling

TPCONF_do_power_cycle = rsquo0rsquo

If TPCONF_do_power_cycle=1 then parameters TP-CONF_power_ctrl_type TPCONF_host_power_ctrlportTPCONF_power_admin_name and TP-CONF_power_admin_pw must also be set

TPCONF_power_ctrl_type must be used to specify thetype of power controller ndash lsquo9258HPrsquo (for an ldquoIP Power9258HPrdquo) or lsquoSLP-SPP1008rsquo (for a ldquoServerlink SLP-SPP1008-Hrdquo) The default is rsquo9258HPrsquo A mix of dif-ferent types of power controllers is currently not sup-portedTPCONF_power_ctrl_type = rsquo9258HPrsquo

TPCONF_host_power_ctrlport is a dictionary that foreach host specifies the IP (or host name) of the respon-sible power controller and the number of the controllerrsquosport the host is connected to (as integer starting from1)TPCONF_host_power_ctrlport = rsquotesthost1rsquo ( rsquo1921681178rsquo rsquo1rsquo )rsquotesthost2rsquo ( rsquo1921681178rsquo rsquo2rsquo )rsquotesthost3rsquo ( rsquo1921681178rsquo rsquo3rsquo )

TPCONF_power_admin_name specifies the name of thepower controllerrsquos admin user

TPCONF_power_admin_name = rsquoadminrsquo

TPCONF_power_admin_pw specifies the password ofthe power controllerrsquos admin user (which in the belowexample is identical to the SSH password used by Fabricbut in general it can be different)

TPCONF_power_admin_pw = envpassword

I Clock offset measurement (broadcast pings)

All the participating machines should have synchro-nised clocks (for example by running NTP) so we canaccurately compare data measured on different hostsHowever clocks on consumer-grade PC hardware drifteven while using NTP TEACUP provides an additionalmechanism to evaluate the quality of the time synchro-nisation and (optionally) correct for clock offsets in thepost-analysis

When TPCONF_bc_ping_enable is set to lsquo1rsquo TEACUPconfigures the router host to send a broadcast or multicastping over the control network These ping packets willbe multicasted or broadcasted to the control interfacesof all other hosts (almost) simultaneously

During experiments there is no other traffic on thecontrol network and typically network jitter introducedby the sender (router) the network switch or the receiveris small Thus one can estimate the offsets of the clocksof different hosts with relatively high accuracy by com-paring the arrival times of the broadcast or multicast pingpackets Here we explain how to enable and configurethe broadcast pings

TPCONF_bc_ping_enable must be set to lsquo1rsquo to en-able the broadcastmulticast ping (default is lsquo0rsquo)TPCONF_bc_ping_rate controls the rate in ping pack-ets per second (default is one packet per second)TPCONF_bc_ping_address is the address the ping issent to This must be either a multicast address (eg22401199) or a broadcast address (eg 1921680255if the control interfaces are in the 1921680024 sub-net) The following shows an example configuration forenabled multicast pings

TPCONF_bc_ping_enable = rsquo1rsquoTPCONF_bc_ping_rate = 1

TPCONF_bc_ping_address = rsquo22401199rsquo


In technical report [8] we explain TEACUPrsquos functionsto analyse the clock offsets and use them for the correc-tion of timestamps

J Custom host init commands

TPCONF_host_init_custom_cmds allows to execute cus-tom per-host init commands This allows to changethe host configuration for example with sysctls TP-CONF_host_init_custom_cmds is a dictionary wherethe key specifies the host name and the value is alist of commands The commands are executed inexactly the order specified after all default build-inhost initialisation has been carried out This meansTPCONF_host_init_custom_cmds makes it possible tooverrule default initialisation The commands are spec-ified as strings and are executed on the host exactly asspecified (with the exception that V_variables if presentare substituted with values) V_variables can be used incommands but the current limitation is there can onlybe one V_variable per command

The custom commands are executed before the routerconfiguration So when using for example a FreeBSDrouter we can use TPCONF_host_init_custom_cmdsto increase the maximum allowable Dummynet queuelength (using sysctl) before Dummynet is actually con-figured

Note that the commands are executed in the foregroundwhich means that for each command TEACUP will waituntil it has been executed on the remote host beforeexecuting the next command for the same host It iscurrently not possible to execute background commandsHowever commands on different hosts are executed inparallel ie waiting for a command to finish on hosttesthost1 does not block executing the next commandon host testhost2 In summary commands on differenthost are executed in parallel but commands on the samehost are executed sequentially

The following config file part shows an example wherewe simply execute the command lsquoecho TESTrsquo on hosttesthost1TPCONF_host_init_custom_cmds = rsquotesthost1rsquo [ rsquoecho TESTrsquo ]

K Router queue setup

The variable TPCONF_router_queues specifies therouter pipes (also referred to as queues here) Each entry

is a 2-tuple The first value specifies a unique integer IDfor each queue The second value is a comma-separatedstring specifying the queue parameters The queues donot necessarily need to be defined in the order of queueID but it is recommended to do so The following queueparameters exist

bull source Specifies the source IPhostname or sourcenetwork (ltipgt[ltprefixgt]) of traffic that is queuedin this queue If a host name is specified there canbe no prefix One can specify an internaltestbed orexternalcontrol IPhostname If an external IPhost-name is specified this will be automatically trans-lated into the first internal IP specified for the hostin TPCONF_host_internal_ip

bull dest Specifies the destination IPhostname orsource network (ltipgt[ltprefixgt]) of traffic that isqueued in this queue If a host name is speci-fied there can be no prefix One can specify aninternaltestbed or externalcontrol IPhostname Ifan external IPhostname is specified this will beautomatically translated into the first internal IPspecified for the host in TPCONF_host_internal_ip

bull delay Specifies the emulated constant delay inmilliseconds For example delay=50 sets the delayto 50 ms

bull loss Specifies the emulated constant loss rate Forexample loss=001 sets the loss rate to 1

bull rate Specifies the rate limit of the queue On Linuxwe can use units such as lsquokbitrsquo or lsquombitrsquo Forexample queue_size=lsquo1mbitrsquo sets the rate limit to1 Mbitsecond

bull queue_size Specifies the size of the queue OnLinux queue size is defined in packets for mostqueuing disciplines but for some queuing disci-plines it needs to be specified in bytes For exampleif we have a Linux queue with size specified inpackets queue_size=1000 sets the queue size to1000 packets On FreeBSD the queue size is alsospecified in packets typically but one can specifythe size in bytes by adding a lsquobytesrsquo or lsquokbytesrsquo forexample queue_size=lsquo100kbytesrsquo specifies a queueof size 100 kbytes If lsquobdprsquo is specified the queuesize will be set to the nominal bandwidth-delay-product (BDP) (this does only work for queuing dis-ciplines where TEACUP knows whether the queuesize is specified in bytes or packets) The minimumqueue size is one packet (if the size is specified inpackets) or 2048 bytes (if the size is specified inbytes)


bull queue_size_mult The actual queue size is thequeue size multiplied with this factor This shouldonly be used if queue_size if set to lsquobdprsquo Thisallows to vary the queue size in multiples of thenominal BDP

bull queue_disc Specifies the queuing discipline Thiscan be the name of any of the queuing disciplinessupported by Linux such as lsquofq_codelrsquo lsquocodelrsquolsquoredrsquo lsquochokersquo lsquopfiforsquo lsquopiersquo etc On FreeBSD theonly queuing disciplines available are lsquofiforsquo andlsquoredrsquo For example queue_disc=lsquofq_codelrsquo sets thequeuing discipline to the fair-queuing+codel modelFor compatibility with FreeBSD one can specifylsquofiforsquo on Linux which is mapped to lsquopfiforsquo (lsquopfiforsquois the default for HTB classes which we use forrate limiting) The queue_disc parameter must bespecified explicitly

bull queue_disc_params This parameter allows topass parameters to queuing disciplines For exam-ple if we wanted to turn ECN on for fq_codelwe would specify queue_disc_params=lsquoecnrsquo (cffq_codel man page)

bull bidir This allows to specify whether a queue isunidirectional (set to lsquo0rsquo) or bidirectional (set tolsquo1rsquo) A unidirectional queue will only get the trafficfrom source to destination whereas a bidirectionalqueue will get the traffic from source to dest andfrom destination to source

bull attach_to_queue This parameter works on Linuxonly It allows to direct matching packets into anexisting queue referenced by the specified queueID but to emulate flow-specific delayloss (differ-ent from the delay and loss of other traffic) Ifattach_to_queue is specified the matching trafficwill go through the already existing queue butthe emulated delay or loss is set according to thecurrent queue specification This means we can omitthe rate queue_disc and queue_size parametersbecause they do not have any effect

bull rtt This parameter allows to explicitly specify theemulated RTT in milliseconds This parameter onlyneeds to be specified if queue_size is set to lsquobdprsquoand the RTT is not twice the delay of the currentTEACUP queue (eg if we set up asymmetric delaywith attach_to_queue)

All parameters must be assigned with either a con-stant value or a TEACUP V_variable V_variable namesmust be defined in TPCONF_parameter_list and TP-CONF_variable_defaults (see below) V_variables are

replaced with either the default value specified in TP-CONF_variable_defaults or the current value from TP-CONF_parameter_list if we iterate through multiple val-ues for the parameter

Figure 11 shows an example queue setup with the samedelay and loss for every host and the same delay andloss in both directions (all the parameters are variableshere)

Figure 12 shows an example that illustrates the at-tach_to_queue parameter Traffic between 17216103and 17216113 goes through the same queues as trafficbetween 17216102 and 17216112 but in both direc-tions it experiences twice the delay

Since version 09 TEACUP supports multiple routersIf multiple routers are specified in TPCONF_router butthere is only one TPCONF_router_queues specification(for example the one shown above) TEACUP willapply the single TPCONF_router_queues specificationto all routers However in many cases with multiplerouters we want to specify router-specific queue setupsThis can be done by making TPCONF_router_queuesa dictionary with the keys being router names (asspecified in TPCONF_router) and the values being TP-CONF_router_queues specifications

As an example let us assume there are two routerstestrouter1 and testrouter2 For each of the routers weneed to specify the queue setup (for brevity we use aplaceholder here instead of showing the actual queuespecificationstestrouter1_queues = [ltqueue_spec1gt]testrouter2_queues = [ltqueue_spec2gt]

Then we need to define TPCONF_router_queues asexplained aboveTPCONF_router_queues = TPCONF_router_queues[rsquotestrouter1rsquo] =

testrouter1_queuesTPCONF_router_queues[rsquotestrouter2rsquo] =

testrouter2_queues

L Traffic generator setup

Traffic generators are defined with the variableTPCONF_traffic_gens This is a list of 3-tuples The firstvalue of a tuple is the start time relative to the starttime of the experiment The second value of the tupleis a unique ID The third value of the tuple is a list ofstrings with the function name of the start function of the


TPCONF_router_queues = [( rsquo1rsquo source=rsquo1721610024rsquo dest=rsquo1721611024rsquo delay=V_delay loss=V_loss rate=V_urate

queue_disc=V_aqm queue_size=V_bsize )( rsquo2rsquo source=rsquo1721611024rsquo dest=rsquo1721610024rsquo delay=V_delay loss=V_loss rate=V_drate

queue_disc=V_aqm queue_size=V_bsize )]

Figure 11 Router queue definition example

TPCONF_router_queues = [( rsquo1rsquo source=rsquo17216102rsquo dest=rsquo17216112rsquo delay=V_delay loss=V_loss rate=V_up_rate

queue_disc=V_aqm queue_size=V_bsize )( rsquo2rsquo source=rsquo17216112rsquo dest=rsquo17216102rsquo delay=V_delay loss=V_loss rate=V_down_rate

queue_disc=V_aqm queue_size=V_bsize )( rsquo3rsquo source=rsquo17216103rsquo dest=rsquo17216113rsquo delay=2V_delay loss=V_loss attach_to_queue=rsquo1rsquo )( rsquo4rsquo source=rsquo17216113rsquo dest=rsquo17216103rsquo delay=2V_delay loss=V_loss attach_to_queue=rsquo2rsquo )]

Figure 12 Router queue definition with attach_to_queue example

traffic generator as first entry followed by the parametersThe name of the functions and the parameters for eachfunction are described in Section V-M

Client and server parameters can be external (controlnetwork) addresses or host names An external addressor host name is replaced by the first internal addressspecified for a host in TPCONF_host_internal_ip Clientand server parameters can also be internal (testbed net-work) addresses which allows to specify any internaladdress

Each parameter is defined as ltparame-ter_namegt=ltparameter_valuegt Parameter names mustbe the parameter names of traffic generator functions(and as such be valid Python variable names) Parametervalues can be either constants (string or numeric)or TEACUP V_variables that are replaced by theactual values depending on the current experiment TheV_variables must be defined in TPCONF_parameter_listand TPCONF_variable_defaults Numeric V_variablescan be modified using mathematical operations suchas addition or multiplication with constants Forexample if a variable lsquoV_delayrsquo exists one can specifylsquo2middotV_delayrsquo as parameter value

Figure 13 shows a simple example At time zero a webserver is started and fake DASH content is created 05seconds later a httperf DASH-like client is started Theduration and rate of the DASH-like flow are specifiedby variables that can change for each experiment Incontrast the cycle length and prefetch time are set tofixed values

The example config files in the source code distributioncontain more examples of setting up traffic genera-tors

M Available traffic generators

This section describes the traffic generators (listed bytheir start function names) that can be used in TP-CONF_traffic_gens

1) start_iperf This starts an iperf client and serverNote that the client sends data to the server It has thefollowing parameters

bull port port number to use for client and server(passed to iperf -p option)

bull client IP or name of client (passed to iperf -coption)

bull server IP or name of server (passed to iperf -Boption)

bull duration time in seconds the client transmits(passed to iperf -t option)

bull congestion_algo TCP congestion algorithm to use(works only on Linux)

bull kill By default this is lsquo0rsquo and the iperf client willterminate after duration seconds If this is set tolsquo1rsquo the iperf client will be killed approximately1 second after duration This is a work-around fora ldquofeaturerdquo in iperf that prevents it from stoppingafter the specified duration (If set to lsquo1rsquo the iperfserver is also killed approximately 2 seconds afterduration)

bull mss TCP maximum segment size (passed to iperf-M option)


TPCONF_traffic_gens = [( rsquo00rsquo rsquo1rsquo start_http_server server=rsquotesthost3rsquo port=80 )( rsquo00rsquo rsquo2rsquo create_http_dash_content server=rsquotesthost3rsquo duration=2V_duration

rates=V_dash_rates cycles=rsquo5 10rsquo )( rsquo05rsquo rsquo3rsquo start_httperf_dash client=rsquotesthost2rsquo server=rsquotesthost3rsquo port=80

duration=V_duration rate=V_dash_rate cycle=5 prefetch=2 )]

Figure 13 Traffic generator example

bull buf_size Send and receive buffer size in bytes(passed to iperf -j and -k which only exist for iperfwith the CAIA patch [7])

bull proto Protocol to use lsquotcprsquo (default) or lsquoudprsquo (setsiperf -u option for lsquoudprsquo)

bull rate The bandwidth used for TCP (passed to iperf-a option) or UDP (passed to iperf -b option) Canend in lsquoKrsquo or lsquoMrsquo to indicate kilo bytes or megabytes

bull extra_params_client Command line parameterspassed to iperf client

bull extra_params_server Command line parameterspassed to iperf server

2) start_ping This starts a ping and has the followingparameters

bull client IP or name of machine to run ping onbull dest IP or name of machine to pingbull rate number of pings per second (Windows ping

only supports 1 pingsecond) (default = 1)bull extra_params Command line parameters passed to

ping

3) start_http_server This starts an HTTP server(lighttpd) and has the following parameters

bull port port to listen on (currently one server canlisten on only one port)

bull config_dir directory where the config file(lighttpdconf) should be copied to

bull config_in local template for config filebull docroot document root on server (FreeBSD

default usrlocalwwwdata MacOSX defaultoptlocalwwwhtdocs LinuxCYGWIN defaultsrvwwwhtdocs)

4) create_http_dash_content This creates fake contentfor the DASH-like client It has the following parame-ters

bull server IP or name of HTTP server

bull docroot document root of HTTP server (FreeBSDdefault usrlocalwwwdata MacOSX defaultoptlocalwwwhtdocs LinuxCYGWIN default s-rvwwwhtdocs)

bull duration number of seconds of fake contentbull rates comma-separated list of DASH rates in kBbull cycles comma-separated list of cycle lengths in

seconds

5) create_http_incast_content This creates fake contentfor incast experiments It has the following parame-ters

bull server IP or name of HTTP serverbull docroot document root of HTTP server (FreeBSD

default usrlocalwwwdata MacOSX defaultoptlocalwwwhtdocs LinuxCYGWIN default s-rvwwwhtdocs)

bull sizes comma-separated list of content file sizes

6) start_httperf This starts an httperf HTTP client Ithas the following parameters

bull client IP or name of clientbull server IP or name of HTTP server (passed to

httperf --server)bull port port server is listening on (passed to httperf

--port)bull conns total number of connections (passed to

httperf --num-conns)bull rate rate at which connections are created (passed

to httperf --rate)bull timeout timeout for each connection httperf will

give up if a HTTP request does not complete withinthe timeout (passed to httperf --timeout)

bull calls number of requests in each connection(passed to httperf --num-calls default = 1)

bull burst length of burst (passed to httperf --burst-length)

bull wsesslog session description file (passed to thirdparameter of httperf --wsesslog)


bull wsesslog_timeout default timeout for each wsess-log connection (passed to second parameter ofhttperf --wsesslog)

bull period time between creation of connectionsequivalent to 1rate if period is a number but periodcan also specify inter-arrival time distributions (seehttperf man page)

bull sessions number of sessions (passed to first param-eter of httperf --wsesslog default = 1)

bull call_stats number of entries in call statistics array(passed to httperf --call-stats default = 1000)

bull extra_params Command line parameters passed tohttperf

7) start_httperf_dash This starts a DASH-like httperfclient It has the following parameters



--port)bull duration duration of DASH flow in secondsbull rate data rate of DASH-like flow in kbpsbull cycle interval between requests in secondsbull prefetch prefetch time in seconds of content to

prefetch (can be fractional number) (default = 00)bull prefetch_timeout like timeout for start_httperf but

only for the prefetch (by default this is set to cycle)bull extra_params Command line parameters passed to

httperf

The behaviour is as follows

1) The client opens a persistent TCP connection tothe server

2) If prefetch is gt 00 the client will fetch the speci-fied number of seconds of content and right afterthat send a request for the next block (step 3)

3) The client will request a block of content wait forsome time (cycle minus download time) and thenrequest the next block The size of one block iscyclemiddotratemiddot10008 bytes

8) start_httperf_incast This starts an httperf clientfor the incast scenario It has the following parame-ters

bull client IP or name of clientbull servers comma-separated list where each entry

specifies one server in the form of lsquo[IP|name]portrsquoIP or name is a serverrsquos IP address or name and

port is the port the server is listening on (a separatesession for each server is created via httperfrsquossession log)

bull duration duration of incast session in secondsbull period period between requests in seconds (float-

ing point number)bull burst_size number of queries sent at each period

start (this is to increase the number of queries ina testbed that only has a few physical respondermachines)

bull response_size size of response from each respon-der in kB


9) start_nttcp This starts an nttcp client and an nttcpserver for some simple unidirectional UDP VoIP flowemulation It has the following parameters

bull client IP or name of clientbull server IP or name of HTTP serverbull port control port server is listening on (passed to

nttcp -p)bull duration duration of the flow (based on this and

the interval TEACUP computes the number ofbuffers to send passed to nttcp -n)

bull interval interval between UDP packets in millisec-onds (passed to nttcp -g)

bull psize UDP payload size (excluding UDPIP header)(passed to nttcp -l)

bull buf_size send buffer size (passed to nttcp -w)bull extra_params_client Command line parameters

passed to nttcp clientbull extra_params_server Command line parameters

passed to nttcp server

Note that nttcp also opens a TCP control connectionbetween client and server However on this connectiononly a few packets are exchanged before and after thedata flow

10) start_httperf_incast_n This starts an httperf client(querier) and n HTTP servers (responders) for an in-cast scenario It also generates the fake content forall n servers Basically this generator is a shortcutto setting up n servers with n start_http_server andcreate_http_incast_content It has the following parame-ters

bull client IP or name of clientbull servers comma-separated list of servers where

each entry is either the IP or the host name of an


HTTP server (a separate session for each server iscreated via httperfrsquos session log)

bull server_port_start the port the first server is lis-tening on All further servers will be started onconsecutive port numbers in the order they arespecified in servers








bull sizes comma-separated list of content file sizes onthe servers

bull num_responders The querier will only sendqueries to the first num_responders servers Thiscan be used to vary the number of responders ina series of experiments


N Mandatory experiment variables

We now describe the mandatory experiment variablesthat must be in every config file There are twotypes singulars and lists Singulars are fixed param-eters while lists specify the different values used insubsequent experiments based on the definitions of TP-CONF_parameter_list and TPCONF_vary_parameters(see below)

1) Traffic duration The duration of the traffic inseconds (must be an integer) is specified with TP-CONF_duration Note that currently the actual dura-tion of an experiment is the number of seconds spec-ified by TPCONF_duration plus the number of sec-onds until the last traffic generator is started (basedon TPCONF_traffic_gens) plus some warmup time TP-CONF_duration is specified as follows

TPCONF_duration = 30

2) Number of repeats (runs) The number of repetitions(runs) carried out for each unique parameter combinationis specified with TPCONF_runs

TPCONF_runs = 1

3) Enabling ECN on hosts TPCONF_ECN specifieswhether ECN is used on the hosts that are the trafficsourcessinks If set to lsquo1rsquo ECN is enabled for all hostsIf set to lsquo0rsquo ECN is disabled for all hosts Currentlyper-host configuration is only possible with custom com-mands

TPCONF_ECN = [ rsquo0rsquo rsquo1rsquo ]

(Note that TPCONF_ECN only enables ECN on thehosts For full ECN support the AQM mechanism onthe router must also be configured to use ECN)

4) Congestion control algorithms TP-CONF_TCP_algos specifies the TCP congestionalgorithms used The following algorithms can beselected lsquonewrenorsquo lsquocubicrsquo lsquohdrsquo lsquohtcprsquo lsquocdgrsquolsquocompoundrsquo

TPCONF_TCP_algos = [ rsquonewrenorsquo rsquocubicrsquo ]

However only some of these are supported dependingon the OS a host is running

bull Windows newreno (default) compoundbull FreeBSD newreno (default) cubic hd htcp cdgbull Linux cubic (default) newreno htcpbull Mac OS X newreno (default)

Instead of specifying a particular TCP algorithm onecan specify lsquodefaultrsquo This will set the algorithm tothe default algorithm depending on the OS the host isrunning

Using only TPCONF_TCP_algos one is limited to eitherusing the same algorithm on all hosts or the defaultsTo run different algorithms on different hosts one canspecify lsquohostltNgtrsquo where ltNgt is an integer numberstarting from 0 The ltNgt refers to the Nth entry foreach host in TPCONF_host_TCP_algos

TPCONF_host_TCP_algos defines the TCP congestioncontrol algorithms used for each host if the lsquohostltNgtrsquodefinitions are used in TPCONF_TCP_algos In the fol-lowing example a lsquohost0rsquo entry in TPCONF_TCP_algoswill lead to each host using its default A lsquohost1rsquo entrywill configure testhost2 to use lsquonewrenorsquo and testhost3to use lsquocdgrsquoTPCONF_host_TCP_algos =


rsquotesthost2rsquo [ rsquodefaultrsquo rsquonewrenorsquo ]rsquotesthost3rsquo [ rsquodefaultrsquo rsquocdgrsquo ]

With TPCONF_host_TCP_algo_params we can specifyparameter settings for each host and TCP congestioncontrol algorithm The settings are passed directly tosysctl on the remote host We can use V_variables toiterate over different settings (similar as for pipes andtraffic generators) and these are replaced with the actualcurrent value before passing the string to sysctl Forexample we can specify settings for CDG for hosttesthost2TPCONF_host_TCP_algo_params = rsquotesthost2rsquo rsquocdgrsquo [rsquonetinettcpcccdgbeta_delay = V_cdgbdelrsquorsquonetinettcpcccdgbeta_loss = V_cdgblossrsquorsquonetinettcpcccdgexp_backoff_scale = 3rsquorsquonetinettcpcccdgsmoothing_factor = 8rsquo ]

5) Emulated delays loss rates and bottleneck band-widths TPCONF_delays specifies the emulated networkdelays in milliseconds The numbers must be chosenfrom [0 max_delay) For most practical purposes themaximum delay max_delay is 10 seconds although itcould be more (if supported by the emulator) Thefollowing shows an example

TPCONF_delays = [ 0 25 50 100 ]

TPCONF_loss_rates specifies the emulated network lossrates The numbers must be between 00 and 10 Thefollowing shows an example

TPCONF_loss_rates = [ 0 0001 001 ]

TPCONF_bandwidths specifies the emulated bandwidthsas 2-tuples The first value in each tuple is the down-stream rate and the second value in each tuple is theupstream rate Note that the values are passed through tothe router queues and are not checked by TEACUP Unitscan be used if the queue setup allows this eg in thefollowing example we use lsquombitrsquo to specify Mbitsecondwhich Linux tc understandsTPCONF_bandwidths = [( rsquo8mbitrsquo rsquo1mbitrsquo )( rsquo20mbitrsquo rsquo14mbitrsquo )]

6) Selection of bottleneck AQM and buffer sizes TP-CONF_aqms specifies the list of AQMqueuing tech-niques This is completely dependent on the router OSLinux supports lsquofiforsquo (mapped to lsquopfiforsquo) lsquopfiforsquo lsquobfiforsquo

lsquofq_codelrsquo lsquocodelrsquo lsquopiersquo lsquoredrsquo etc (refer to the tc manpage for the full list) FreeBSD support only lsquofiforsquo andlsquoredrsquo Default on Linux and FreeBSD are FIFOs (withsize in packets) The following shows an example

TPCONF_aqms = [ rsquopfiforsquo rsquofq_codelrsquo rsquopiersquo ]

Note that all underscores in parameter values used in logfile names are changed to hyphens to allow for easierparsing of log file names For example lsquofq_codelrsquo willbecome lsquofq-codelrsquo in the file name

TPCONF_buffer_sizes specifies the bottleneck buffersizes If the router is Linux this is mostly in packetsslotsbut it depends on the AQM technique (eg for bfifoit is bytes) If the router is FreeBSD this would bein slots by default but we can specify byte sizes (egwe can specify 4Kbytes) The following example for aLinux router would result in buffers of 100 200 and 600packets long

TPCONF_buffer_sizes = [ 100 200 600 ]

7) Specifying which parameters to varyTPCONF_vary_parameters specifies a list of parametersto vary over a series of experiments ie parametersthat will take on multiple values The listed parametersmust be defined in TPCONF_parameter_list (seeSection V-P) The total number of experiments carriedout is the number of unique parameter combinations(multiplied by the number of TPCONF_runs if lsquorunsrsquo isalso specified in TPCONF_vary_parameters)

The TPCONF_vary_parameters specification also de-fines the order of the parameters in the log file namesWhile not strictly necessary if used lsquorunsrsquo shouldbe last in the list If lsquorunsrsquo is not specified thereis a single experiment for each parameter combina-tion TPCONF_vary_parameters is only used for mul-tiple experiments When we run a single experiment(run_experiment_single) all the variables are set to fixedvalues based on TPCONF_variable_defaults The follow-ing shows an example for parameters included in TP-CONF_parameter_list in the example config filesTPCONF_vary_parameters = [

rsquotcpalgosrsquo rsquodelaysrsquo rsquolossrsquorsquobandwidthsrsquo rsquoaqmsrsquo rsquobsizesrsquorsquorunsrsquo

]

O Experiment-specific variables

Some variables defined in the example config file(s) areonly used with certain traffic generators


TPCONF_dash_rates specifies the DASH content ratesin kbitsecond and TPCONF_dash_rates_str is a stringwith a comma-separated list of rates (the latter is used bythe content creation function create_http_dash_content)DASH rates must be integers The following shows anexampleTPCONF_dash_rates = [ 500 1000 2000 ]TPCONF_dash_rates_str = rsquorsquojoin(map(str

TPCONF_dash_rates))

TPCONF_inc_content_sizes specifies the content sizesin kB (as integer) for the replies sent in an incastscenario TPCONF_inc_periods specifies the length ofa period between requests by the querier in seconds (asfloating point) The following shows an exampleTPCONF_inc_content_sizes= rsquo64 512 1024rsquoTPCONF_inc_periods = [ 10 ]

P Defining parameters to vary

TEACUP provides a flexible mechanism for definingwhat parameters are varied during experiments whatvalues (or ranges of values) those parameters maytake and how those parameters are then used to drivehost traffic generator and bottleneck configuration TP-CONF_parameter_list is a Python dictionary at the coreof this mechanism

The keys are names that can be used in TP-CONF_vary_parameters The values are 4-tuples Thefirst parameter of each tuple is a list of V_variablesthat can be used for example in the queue configurationor traffic generator configuration The second parameterof each tuple is a list of lsquoshort namesrsquo used in the filenames of created log files The third parameter of eachtuple is the list of parameter values these are usuallyreferences to the lists defined in the previous section Thelast parameter is a dictionary of extra V_variables to set(this can be empty) where the keys are variable namesand the values are the variable values The length of thefirst three tuple parameters (V_variable identifiers shortnames and V_variable values) must be equal

When a series of experiments is started with lsquofabrun_experiment_multiplersquo the following happens Foreach parameter combination of the parameters defined inTPCONF_vary_parameters one experiment is run wherethe parameter settings are logged in the file name usingthe short names and the V_ variables are set to theparameter combination given by the value lists

TPCONF_parameter_list can handle groupedV_variables where in each experiment a specificcombination of the grouped V_variables is used Anexample of this is the parameter bandwidths which usesTPCONF_bandwidths as values

TPCONF_variable_defaults is a dictionary that specifiesthe defaults for V_ variables The keys are V_ variablenames and the values are the default values (often thefirst value of the parameter lists) For each parameterthat is not varied the default value specified in TP-CONF_variable_defaults is used

We now discuss a simple example where we focus onthe variables to vary delay and TCP algorithms Assumewe want to experiment with two delay settings and twodifferent TCP CC algorithms So we haveTPCONF_delays = [ 0 50 ]TPCONF_TCP_algos = [ rsquonewrenorsquo rsquocubicrsquo ]

We also need to specify the two parameters to be variedand the default parameters for the variables as shown inFigure 14

V_delay can then be used in the router queue settingsV_tcp_cc_algo is passed to the host setup functionWhen we run lsquofab run_experiment_multiplersquo this willrun the following experiments here represented by thestart of the log file names (we assume the test ID prefixis the default from Section V-E)20131206-170846_del_0_tcp_newreno20131206-170846_del_0_tcp_cubic20131206-170846_del_50_tcp_newreno20131206-170846_del_50_tcp_cubic

Q Adding new V_ variables

New V_variables are easy to add Say we want to createa new V_variable called V_varx We need to do thefollowing

1) Add a new list with parameter values letrsquos sayTPCONF_varx = [ x y ]

2) Add one line in TPCONF_parameters_list thekey is the identifier that can be used in TP-CONF_vary_parameters (letrsquos call it varx_vals)and the value is a 4-tuple variable name as stringlsquoV_varxrsquo variable name used in file name (saylsquovarxrsquo) pointer to value list (here TPCONF_varx)and optionally a dictionary with related variables(here empty)


TPCONF_parameter_list = rsquodelaysrsquo ( [ rsquoV_delayrsquo ] [ rsquodelrsquo ] TPCONF_delays )rsquotcpalgosrsquo ( [ rsquoV_tcp_cc_algorsquo ] [ rsquotcprsquo ] TPCONF_TCP_algos )TPCONF_variable_defaults rsquoV_delayrsquo TPCONF_delays[0]rsquorsquoV_tcp_algorsquo TPCONF_TCP_algos[0]TPCONF_vary_parameters = [ rsquodelaysrsquo rsquotcpalgosrsquo ]

Figure 14 Specifying the parameters to be varied

3) Add one line in TPCONF_variable_defaults spec-ifying the default value used (when not iterat-ing over the variable) the key is the V_variablename as string (here lsquoV_varxrsquo) and the valueis the default value from TPCONF_varx (sayTPCONF_varx[0])

Technically step 1 is not necessary as the list of val-ues can be put directly in TPCONF_parameters_listand the default value can be put directly in TP-CONF_variable_defaults However defining a separatelist improves the readability

VI RUNNING EXPERIMENTS

This section describes how to run experiments Firstwe describe the initial steps needed Then we outlinea simple example config file Finally we describe howto run the experiments

A Initial steps

First you should create a new directory for the experi-ment or series of experiments Copy the files fabfilepyand runsh (and run_resumesh) from the TEACUP dis-tribution into that new directory Then create a configpyfile in the directory An easy way to get a configpy fileis to start with one of the provided example config filesas basis and modify it as necessary

B Example config

Listing 1 shows a minimal but complete configpyfile The testbed consists of three machines twohosts (19216812 19216813) connected by a router(19216814) The two hosts will run FreeBSD for theexperiment while the router will run Linux On therouter we configure two pipes one in each direction withdifferent rates but the same AQM mechanism buffersize and emulated delay and loss The test traffic consists

of two parallel TCP sessions generated with iperf bothstart at the start of the experiment (time 00) With iperfthe client sends data to the server so the data is sentfrom 19216812 to 19216813 Each experiment lasts30 seconds and we run a series of experiments varyingthe TCP congestion control algorithm network delayand loss upstream and downstream bandwidths AQMtechnique and buffer size There is one experiment foreach combination of parameters (one run)

C Running experiments

There are two Fabric tasks to startexperiments run_experiment_single andrun_experiment_multiple

To run a single experiment with the default test ID prefixTPCONF_test_id type

gt fab run_experiment_single

To run a series of experiment based on the TP-CONF_vary_parameters settings with the default test IDprefix TPCONF_test_id type

gt fab run_experiment_multiple

In both cases the Fabric log output will be printed outon the current terminal (stdout) and can be redirectedwith the usual means The default test ID prefix (TP-CONF_test_id) is specified in the config file Howeverthe test ID prefix can also be specified on the commandline (overruling the config setting)

gt fab run_experiment_multipletest_id=lsquodate

+Ymd-HMSlsquo

The last command will run a series of experimentswhere the test ID prefix is YYYYMMDD-HHMMSSusing the actual date when the fab command is run Forconvenience TEACUP provides a shell script (runsh)that logs the Fabric output in a lttest_ID_prefixgtlog fileinside the lttest_ID_prefixgt sub directory and is startedwith


import sysimport datetimefrom fabricapi import env

envuser = rsquorootrsquoenvpassword = rsquopasswordrsquoenvshell = rsquobinsh -crsquoenvtimeout = 5envpool_size = 10

TPCONF_script_path = rsquohometestsrcteacuprsquosyspathappend(TPCONF_script_path)TPCONF_tftpboot_dir = rsquotftpbootrsquoTPCONF_router = [ rsquo19216814rsquo ]TPCONF_hosts = [ rsquo19216812rsquo rsquo19216813rsquo ]TPCONF_host_internal_ip = rsquo19216814rsquo [ rsquo17216101rsquo rsquo17216111rsquo ]rsquo19216812rsquo [ rsquo17216102rsquo ]rsquo19216813rsquo [ rsquo17216112rsquo ]

now = datetimedatetimetoday()TPCONF_test_id = nowstrftime(Ymd-HMS) + rsquo_experimentrsquoTPCONF_remote_dir = rsquotmprsquoTPCONF_host_os = rsquo19216814rsquo rsquoLinuxrsquorsquo19216812rsquo rsquoFreeBSDrsquorsquo19216813rsquo rsquoFreeBSDrsquo TPCONF_linux_kern_router = rsquo31418-10000hzrsquoTPCONF_force_reboot = rsquo1rsquoTPCONF_boot_timeout = 100TPCONF_do_power_cycle = rsquo0rsquoTPCONF_host_power_ctrlport = TPCONF_power_admin_name = rsquorsquoTPCONF_power_admin_pw = rsquorsquoTPCONF_max_time_diff = 1

TPCONF_router_queues = [( rsquo1rsquo source=rsquo1721610024rsquo dest=rsquo1721611024rsquo delay=V_delay loss=V_loss rate=V_urate queue_disc=V_aqm queue_size=V_bsize )( rsquo2rsquo source=rsquo1721611024rsquo dest=rsquo1721610024rsquo delay=V_delay loss=V_loss rate=V_drate queue_disc=V_aqm queue_size=V_bsize ) ]

traffic_iperf = [( rsquo00rsquo rsquo1rsquo start_iperf client=rsquo19216812rsquo server=rsquo19216813rsquo port=5000 duration=V_duration )( rsquo00rsquo rsquo2rsquo start_iperf client=rsquo19216812rsquo server=rsquo19216813rsquo port=5001 duration=V_duration ) ]TPCONF_traffic_gens = traffic_iperf

TPCONF_duration = 30TPCONF_runs = 1TPCONF_ECN = [ rsquo0rsquo rsquo1rsquo ]TPCONF_TCP_algos = [ rsquonewrenorsquo rsquocubicrsquo rsquohtcprsquo ]TPCONF_host_TCP_algos = TPCONF_host_TCP_algo_params = TPCONF_host_init_custom_cmds = TPCONF_delays = [ 0 25 50 100 ]TPCONF_loss_rates = [ 0 0001 001 ]TPCONF_bandwidths = [ ( rsquo8mbitrsquo rsquo1mbitrsquo ) ( rsquo20mbitrsquo rsquo14mbitrsquo ) ]TPCONF_aqms = [ rsquopfiforsquo rsquocodelrsquo rsquopiersquo ]TPCONF_buffer_sizes = [ 1000 1 ]

TPCONF_parameter_list = rsquodelaysrsquo ( [ rsquoV_delayrsquo ] [ rsquodelrsquo ] TPCONF_delays )rsquolossrsquo ( [ rsquoV_lossrsquo ] [ rsquolossrsquo ] TPCONF_loss_rates )rsquotcpalgosrsquo ( [ rsquoV_tcp_cc_algorsquo ] [ rsquotcprsquo ] TPCONF_TCP_algos )rsquoaqmsrsquo ( [ rsquoV_aqmrsquo ] [ rsquoaqmrsquo ] TPCONF_aqms )rsquobsizesrsquo ( [ rsquoV_bsizersquo ] [ rsquobsrsquo ] TPCONF_buffer_sizes )rsquorunsrsquo ( [ rsquoV_runsrsquo ] [ rsquorunrsquo ] range(TPCONF_runs) )rsquobandwidthsrsquo ( [ rsquoV_dratersquo rsquoV_uratersquo ] [ rsquodownrsquo rsquouprsquo ] TPCONF_bandwidths ) TPCONF_variable_defaults = rsquoV_ecnrsquo TPCONF_ECN[0]rsquoV_durationrsquo TPCONF_durationrsquoV_delayrsquo TPCONF_delays[0]rsquoV_lossrsquo TPCONF_loss_rates[0]rsquoV_tcp_cc_algorsquo TPCONF_TCP_algos[0]rsquoV_dratersquo TPCONF_bandwidths[0][0]rsquoV_uratersquo TPCONF_bandwidths[0][1]rsquoV_aqmrsquo TPCONF_aqms[0]rsquoV_bsizersquo TPCONF_buffer_sizes[0]

TPCONF_variable_defaults TPCONF_vary_parameters = [ rsquotcpalgosrsquo rsquodelaysrsquo rsquolossrsquo rsquobandwidthsrsquo rsquoaqmsrsquo rsquobsizesrsquo rsquorunsrsquo ]

Listing 1 Example configpy file


gt runsh

The shell script generates a test ID prefix and thenexecutes the command

gt fab run_experiment_multipletest_id=

lttest_ID_pfxgt gt lttest_ID_pfxgtlog 2gtamp1

The test ID prefix is set to lsquodate

+Ymd-HMSlsquo_experiment The outputis unbuffered so one can use tail -f on the log fileand get timely output The fabfile to be used can bespecified ie to use the fabfile myfabfilepy insteadof fabfilepy run

gt runsh myfabfilepy

TEACUP keeps track of experiments using two files inthe current directory

bull The file experiment_startedtxt logs the testIDs of all experiments started

bull The file experiment_completedtxt logs thetest IDs of all experiments successfully completed

Note that TEACUP never resets either of these files ndashnew test IDs are simply appended to the files (which arecreated if they donrsquot already exist)7

A run_experiment_multiple task that was interruptedpart-way through may be restarted with the resume

parameter TEACUP will perform all experiments of theseries that were not previously completed (not logged inexperiments_completedtxt)

For example the following command resumesa series of experiments with test ID prefix20131218-113431_windows (and appends thelog output to the existing log file)


test_id=20131218-113431_windowsresume=1

gtgt 20131218-113431_windowslog 2gtamp1

The resume parameter also enables redoing of previ-ously completed experiments by first editing them outof experiments_completedtxt

If a series of experiments is interrupted by non-deterministic errors ie each experiment may fail withsome small probability the run_resumesh shell scriptcan be used to ensure the whole series of experimentsis completed The script runs the experiments using

7The user must manually delete or edit these files if actualexperiment results are later deleted

the run_experiment_multiple task and uses the resume

option to automatically restart and continue each time anexperiment was not successfully completed The scriptis used by executing

gt run_resumesh

D TEACUP variable information logged

TEACUP logs configuration information for conductedexperiments in two files In the following lttest_idgtis a test ID and lttest_id_pfxgt is the correspondingtest ID prefix For each experiment TEACUP logs allV_ variables and their values (in alphabetical order ofvariable names) in a file lttest_idgt_config_varsloggzThe following is an example (with header and legendlines omitted for brevity)U V_aqm pfifoU V_bsize 1000U V_delay 100

Each line starts with an lsquoUrsquo or an lsquoNrsquo indicating whethera variable is Used or Not used After the use indicatorfollows the variable name and the value the variable wasset to in the particular experiment

TEACUP also logs all varying parametersfor a series of experiments in a file namedlttest_id_pfxgt_varying_paramsloggz The followingshows a file as example (excluding the headerline)V_aqm aqm aqmsV_bsize bs bsizesV_delay del delays

In each line the first column is the V_ variable namethe second column is the short name used in the test ID(in the file names) and the last column is the identifierthat is used in TPCONF_vary_parameters (and links toan entry in TPCONF_parameter_list)

VII HOST CONTROL UTILITY FUNCTIONS

This section describes a number of utility functionsavailable as Fabric tasks As mentioned previously thefab utility has an option to list all available tasks

gt fab -l


A Remote command execution

The exec_cmd task can be used to execute one commandon one or more testbed hosts For example the followingcommand executes the command uname -s on a numberof hosts

gt fab -H testhost1testhost2testhost3

exec_cmdcmd=uname -s

If no hosts are specified on the command line theexec_cmd command is executed on all hosts listed inthe config file (the union set of TPCONF_router andTPCONF_hosts) For example the following commandis executed on all testbed hosts

gt fab exec_cmdcmd=uname -s

B Copying files to testbed hosts

The copy_file task can be used to copy a local file toone or more testbed hosts For example the followingcommand copies the web10g-logger executable to alltestbed hosts except the router (this assumes all the hostsrun Linux when the command is executed)

gt fab -H testhost2testhost3

copy_filefile_name=usrbinweb10g-logger

remote_path=usrbin

If no hosts are specified on the command line thecommand is executed for all hosts listed in the config file(the union set of TPCONF_router and TPCONF_hosts)For example the following command copies the file toall testbed hosts

gt fab copy_filefile_name=

usrbinweb10g-loggerremote_path=usrbin

The parameter method controls the method used forcopying By default (method=rsquoputrsquo) copy_file will usethe Fabric put function to copy the file Howeverthe Fabric put function is slow For large files settingmethod=rsquoscprsquo provides much better performance usingthe scp command While scp is faster it may prompt forthe password if public key authentication has not beenconfigured

C Installing ssh keys

The authorize_key task can be used to append the currentuserrsquos public SSH key to the ~sshauthorized_keys fileof the remote user The user can then login via SSHwithout having to enter a password For example the

following command enables password-less access for theuser on three testbed hosts


authorize_key

Note the authorize_key task assumes the user has a~sshid_rsapub key file This can be created withssh-keygen -t rsa Also note that the task does notcheck if the public key is already in the remote userrsquosauthorized_keys file so executing this task multipletimes may lead to duplicate entries in the remote userrsquosauthorized_keys file

D Topology configuration

Where the testbed meets the requirements describedin Section V-G the init_topology task can be usedto ldquomoverdquo hosts from one test subnet to the otherby configuring the VLAN membership of the switchports in conjunction with IP addresses and staticroutes on each host The task uses the configpyfile to get the testbed IP addresses of the hosts tobe configured (from TPCONF_host_internal_ip) Italso uses the values of TPCONF_topology_switchTPCONF_topology_switch_port_prefix andTPCONF_topology_switch_port_offset specified inconfigpy to reconfigure the switch The last threeparameters can also be overridden by the taskrsquosparameters switch port_prefix and port_offsetLimitations of the current implementation are describedin Section V-G

The following shows an example where we configure thehosts testhost1 and testhost2 using the taskrsquos parametersto specify the switch-relevant settings


init_topologyswitch=switch2

port_prefix=Gi10port_offset=5

E Initialise hosts to a specific operating system

The init_os task can be used to reboot hosts into specificoperating systems (OSs) For example the followingcommand reboots the hosts testhost1 and testhost2 intothe OSs Linux and FreeBSD respectively


init_osos_list=LinuxFreeBSD

force_reboot=1

Note that the commas in os_list need to be escapedwith backslashes () since otherwise Fabric interprets


the commas as parameter delimiters Note that os_listcan be shorter than the number of specified hosts inwhich case it will be padded to the length of the numberof hosts by duplicating the last entry This allows toreboot a large number of hosts into the same OS whilespecifying an os_list with only a single entry (thedesired OS)

For Linux the kernel to boot can be chosenwith the parameters linux_kern_hosts andlinux_kern_router linux_kern_hosts specifiesthe kernel for normal hosts (not routers) andlinux_kern_router specifies the kernel for routersPlease refer to the description in Section V-H2 on howto specify the kernel name

By default force_reboot is 0 which means hosts thatare already running the desired OS are not rebootedSetting force_reboot to 1 enforces a reboot Bydefault the script waits 100 seconds for a host to rebootIf the host is not responsive after this time the script willgive up unless the do_power_cycle parameter is set to1 This timeout can be changed with the boot_timeoutparameter which specifies the timeout in seconds (asinteger) A minimum boot timeout of 60 seconds will beenforced

The do_power_cycle parameter can be set to 1 toforce a power cycle if a host does not respond afterthe boot timeout (assuming TEACUP-compatible powercontroller(s) are configured) The script will then waitfor boot_timeout seconds again for the host to comeup If the host is still unresponsive after the timeout thescript will give up (there are no further automatic powercycles) The following command shows an example withdo_power_cycle set to 1



force_reboot=1do_power_cycle=1

F Power cycling

The power_cycle task can be used to power cycle hostsie if hosts become unresponsive (assuming TEACUP-compatible power controller are configured) After thepower cycle the hosts will boot the last selected OS Forexample the following command power cycles the hoststesthost1 and testhost2

gt fab -H testhost1testhost2 power_cycle

G Check software installations on testbed hosts

The check_host command can be used to check if therequired software is installed on the hosts The task onlychecks for the presence of necessary tools but it doesnot check if the tools actually work For example thefollowing command checks all testbed hosts


check_host

H Check testbed host connectivity

The check_connectivity task can be used to check con-nectivity between testbed hosts with ping This taskonly checks the connectivity of the internal testbednetwork not the reachability of hosts on their controlinterface For example the following command checkswhether each host can reach each other host across thetestbed network


check_connectivity

I Check TEACUP config file

The check_config task can be used to check theTEACUP config file This task will perform a numberof checks and abort with an error message if it findsany errors in the config file (This task is automaticallyrun at the start of each experiment or series of experi-ments)

gt fab check_config

1) Print version information The get_version taskprints out the TEACUP version number revision numberand data and the copyright

gt fab get_version

The output will look like

TEACUP Version 09Revision 1175Date 2015-04-01 110050 +1100 (Wed 01Apr 2015)Copyright (c) 2013-2015 Centre forAdvanced Internet Architectures

Swinburne University of Technology All

rights reserved


VIII EXPERIMENT EXAMPLES

A Overview

Here we provide a few example scenarios of howTEACUP can be used Each scenario involves trafficsenders and receivers distributed across two subnets(subnet A and subnet B) connected by a bottleneckrouter All hosts and the router are time synchronisedusing NTP We have the following scenarios

bull Scenario 1 We have two TCP flows with dataflowing from a source in subnet A to a destinationin subnet B We emulate different delays on therouter In this scenario the rebooting functionalityof TEACUP is not used which means the exper-imenter has to boot all three machines into thedesired OS before the experiment is started

bull Scenario 2 As with scenario 1 with TEACUP alsoautomatically booting the desired OS

bull Scenario 3 As with scenario 2 with TEACUP usinga power controller to power cycle hosts if they donrsquotrespond after the reboot

bull Scenario 4 As with scenario 1 with each TCPflow between a different senderreceiver pair anddifferent network path delay emulated for each flow(both flows still go through the same bottleneck)

bull Scenario 5 We have three staggered bulk-transferflows going through the router each between adifferent senderreceiver pair We use different em-ulated delay for each senderreceiver pair and alsovary the bandwidths We now also vary the TCPcongestion control algorithm

bull Scenario 6 We have three hosts plus the routerOne host in subnet A acts as web server The twoother hosts in subnet B act as clients that both useDASH-like video streaming over HTTP We emulatedifferent network delays and AQM mechanisms

bull Scenario 7 Investigating the incast problem Onhost in subnet A queries 10 responders in subnet BAgain we emulate different network delays AQMmechanisms and TCP algorithms We also vary thesize of the responses

B Scenario 1 Two TCP flows from data sender toreceiver

1) Topology In this scenario we have two hostsnewtcp20 connected to the 1721610024 network andnewtcp27 connected to the 1721611024 network Themachine newtcprt3 connects the two experiment subnets

All three machines also have a second network interfacethat is used to control the experiment via TEACUPnewtcp20 and newtcp27 must run Linux FreeBSD Win-dows 7 or Mac OS X and newtcprt3 must run FreeBSDor Linux newtcp20 and newtcp27 must have the trafficgenerator and logging tools installed as described in [7]However PXE booting or a multi-OS installation is notneeded for this scenario

2) Test Traffic Two TCP bulk transfer flows are createdusing iperf

3) Variable Parameters We emulate three different de-lays and two different bandwidth settings ndash six differentexperiments in total We do also define a variable param-eter for AQM but define only one AQM (default pfifo)This causes the used AQM to be logged as part of theexperiment ID

4) TEACUP Config File You can download the config-uration file from [24] To use the configuration renameit to configpy In the following we explain the configu-ration file

Note that TEACUP configuration files can be split acrossmultiple files (using Pythonrsquos execfile()) This allowsto split a large config file into multiple files for betterreadability It also allows to include part of configurationfiles into multiple different config files which makes itpossible to reuse parts of the config that is not changedacross multiple experiments An example of a this isthis config file which is functionally identical to theabove config file Only here the main config file includesseparate files to specify the testbed machines the routersetup and the traffic generated

At the top of the configuration file we need to importthe Fabric env structure and we also need to import anyother Python functionality used (see Listing 2) Firstwe need to configure the user and password used byFabric via Fabrics env parameters A password is notrequired if public key authentication is set up We alsoneed to tell Fabric how to execute commands on theremote machines TEACUP uses Bourne shell (binsh)by default

The next part of the config file defines the path tothe TEACUP scripts and the testbed hosts (see Listing3) TPCONF_router is used to define the router andTPCONF_hosts is used to define the list of hosts Foreach host and the router the testbed network interfacesneed to be defined with TPCONF_host_internal_ip The


User and passwordenvuser = rsquorootrsquoenvpassword = rsquorootpwrsquo Set shell used to execute commandsenvshell = rsquobinsh -crsquo

Listing 2 Scenario 1 Fabric configuration

router obviously has two testbed network interfaceswhereas the hosts have only one

In the next part of the configuration file we need todefine some general experiment settings (see Listing 4)TEACUP_max_time_diff specifies the maximum clockoffset in seconds allowed This is a very coarse thresholdas the offset estimation performed by TEACUP is notvery accurate Currently TEACUP simply attempts tofind out if the synchronisation is very bad and if yesit will abort it does try to enforce high accuracy forthe time synchronisation TPCONF_test_id defines thename prefix for the output files for an experiment ora series of experiments TPCONF_remote_dir specifieswhere on the hosts the log files are stored until they aremoved to the control host running TEACUP after eachexperiment

Then we define the router queuespipes using TP-CONF_router_queues (see Listing 5) Each entry of thislist is a tuple The first value is the queue numberand the second value is a comma separated list ofparameters The queue numbers must be unique Notethat variable parameters must be either constants orvariable names defined by the experimenter Variablesare evaluated during run-time Variable names muststart with a lsquoV_rsquo Parameter names can only containnumbers letter (upper and lower case) underscores (_)and hyphenminus (-) All V_variables must be definedin TPCONF_variable_list (see below)

Next we need to define the traffic generated duringthe experiments (see Listing 6) TEACUP implementsa number of traffic generators In this example weuse iperf to generate TCP bulk transfer flows TP-CONF_traffic_gens defines the traffic generator but herewe use a temporary variable as well which allows tohave multiple traffic generator definitions and switchbetween them by changing the variable assigned toTPCONF_traffic_gens

Each entry in is a 3-tuple The first value of the tuplemust be a float and is the time relative to the start of

the experiment when tasks are executed If two taskshave the same start time their start order is arbitraryThe second entry of the tuple is the task number andmust be a unique integer (used as ID for the process)The last value of the tuple is a comma separated list ofparameters The first parameter of this list must be thetask name The TEACUP manual lists the task name andpossible parameters Client and server can be specifiedusing the externalcontrol IP addresses or host namesIn this case the actual interface used is the first internaladdress (according to TPCONF_host_internal_ip) Alter-natively client and server can be specified as internaladdresses which allows to use any internal interfacesconfigured

Next we define all the parameter values used in theexperiments (see Listing 7) TPCONF_duration definesthe duration of the traffic TPCONF_runs specifies thenumber of runs carried out for each unique com-bination of parameters TPCONF_TCP_algos specifiesthe congestion control algorithms Here we only useNewreno

In this simple case all hosts use the same TCP conges-tion control algorithm but TEACUP allows to specifyper-host algorithms with TPCONF_host_TCP_algos Pa-rameter settings for TCP congestion control algorithmscan be specified with TPCONF_host_TCP_algo_params(assuming parameters can be controlled with sysctl) buthere we do not make use of that and simply use thedefault settings

TPCONF_delays specifies the delay values (delay ineach direction) TPCONF_loss_rates specifies the pos-sible packet loss rates and TPCONF_bandwidths spec-ifies the emulated bandwidths (in downstream and up-stream directions) TPCONF_aqms specifies the AQMmechanism to use (here pfifo which is the default)TPCONF_buffer_sizes specifies the size of the queueThis is normally specified in packets but it depends onthe type of AQM For example if bfifo was used thesize would need to be specified in bytes


Path to teacup scriptsTPCONF_script_path = rsquohometeacupteacup-08rsquo DO NOT remove the following linesyspathappend(TPCONF_script_path) Set debugging level (0 = no debugging info output)TPCONF_debug_level = 0 Host listsTPCONF_router = [rsquonewtcprt3rsquo ]TPCONF_hosts = [ rsquonewtcp20rsquo rsquonewtcp27rsquo ] Map external IPs to internal IPsTPCONF_host_internal_ip =

rsquonewtcprt3rsquo [rsquo17216101rsquo rsquo17216111rsquo]rsquonewtcp20rsquo [rsquo172161060rsquo]rsquonewtcp27rsquo [rsquo172161167rsquo]

Listing 3 Scenario 1 general settings and host configuration

Maximum allowed time difference between machines in seconds otherwise experiment will abort cause synchronisation problemsTPCONF_max_time_diff = 1 Experiment name prefix used if not set on the command line The command line setting will overrule this config settingnow = datetimedatetimetoday()TPCONF_test_id = nowstrftime(Ymd-HMS) + rsquoexperimentrsquo Directory to store log files on remote hostTPCONF_remote_dir = rsquotmprsquo

Listing 4 Scenario 1 test ID prefix and directory configuration

TPCONF_router_queues = [ Set same delay for every host(rsquo1rsquo source=rsquo1721610024rsquo dest=rsquo1721611024rsquo delay=V_delay loss=V_loss rate=V_up_rate queue_disc=V_aqm queue_size=V_bsize )

(rsquo2rsquo source=rsquo1721611024rsquo dest=rsquo1721610024rsquo delay=V_delay loss=V_loss rate=V_down_rate queue_disc=V_aqm queue_size=V_bsize )

]

Listing 5 Scenario 1 router queue configuration

Finally we need to specify which parameters will bevaried and which parameters will be fixed for a seriesof experiments and we also need to define how ourparameter ranges above map to V_variables used for thequeue setup and traffic generators and the log file namesgenerated by TEACUP (see Listing 8)

TPCONF_parameter_list is a map of the parameters wepotentially want to vary The key of each item is theidentifier that can be used in TPCONF_vary_parameters(see below) The value of each item is a 4-tuple contain-ing the following things First a list of variable namesSecond a list of short names uses for the file names Foreach parameter varied a string lsquo_ltshort_namegt_ltvaluegtrsquois appended to the log file names (appended to chosen

prefix) Note short names should only be letters fromandashz or AndashZ do not use underscores or hyphens Thirdthe list of parameters values If there is more than onevariable this must be a list of tuples each tuple havingthe same number of items as the number of variablesFourth an optional dictionary with additional variableswhere the keys are the variable names and the values arethe variable values

The parameters that are actually varied are specifiedwith TPCONF_vary_parameters Only parameters listedin TPCONF_vary_parameters will appear in the nameof TEACUP output files So it can make sense to add aparameter in TPCONF_vary_parameters that only has a


traffic_iperf = [(rsquo00rsquo rsquo1rsquo start_iperf client=rsquonewtcp27rsquo server=rsquonewtcp20rsquo port=5000

duration=V_duration )(rsquo00rsquo rsquo2rsquo start_iperf client=rsquonewtcp27rsquo server=rsquonewtcp20rsquo port=5001

duration=V_duration ) Or using internal addresses( rsquo00rsquo rsquo1rsquo start_iperf client=rsquo17216112rsquo server=rsquo17216102rsquo port=5000 duration=V_duration )( rsquo00rsquo rsquo2rsquo start_iperf client=rsquo17216112rsquo server=rsquo17216102rsquo port=5001 duration=V_duration )

] THIS is the traffic generator setup we will useTPCONF_traffic_gens = traffic_iperf

Listing 6 Scenario 1 traffic configuration

Duration in seconds of trafficTPCONF_duration = 30 Number of runs for each settingTPCONF_runs = 1 TCP congestion control algorithm usedTPCONF_TCP_algos = [rsquonewrenorsquo ] Emulated delays in msTPCONF_delays = [0 25 50] Emulated loss ratesTPCONF_loss_rates = [0] Emulated bandwidths (downstream upstream)TPCONF_bandwidths = [ (rsquo8mbitrsquo rsquo1mbitrsquo) (rsquo20mbitrsquo rsquo14mbitrsquo) ] AQMTPCONF_aqms = [rsquopfiforsquo ] Buffer sizeTPCONF_buffer_sizes = [100]

Listing 7 Scenario 1 experiment parameter value configuration

single value because only then will the parameter shortname and value be part of the file names

TPCONF_variable_defaults specifies the default valuefor each variable which is used for variables that arenot varied The key of each item is the parameter nameThe value of each item is the default parameter valueused if the variable is not varied

5) Running Experiment Create a directory with theabove configpy and copy fabfilepy and runsh from theTEACUP code distribution into this directory To run theseries of experiments with all parameter combinationsgo into the experiment directory (that contains fabfilepyand runsh) and execute

gt runsh

This will create a sub directory with the name of thetest ID prefix and store all output files in this subdirectory

6) Analysing Results TEACUP provides a range oftasks for analysing the results of a given experiment de-scribed in an accompanying technical report [8] Here webriefly summarise how to extract some key results

TEACUP will create a file names lttest_id_prefixgtlogfile in the experiment data sub directory It willalso create a file experiments_startedtxt and experi-ments_completedtxt in the parent directory The fileexperiments_startedtxt contains the names of all ex-periments started and the file experiments_completedtxtcontains the names of all experiments successfully com-pleted If some experiments were not completed checkthe log file for errors

Assuming all experiments of a series completed success-fully we can now start to analyse the data To createtime series of CWND RTT as estimated by TCP RTTas estimated using SPP and the throughput over timefor each experiment of the series use the following


TPCONF_parameter_list = Vary name V_ variable file name values extra varsrsquodelaysrsquo ([rsquoV_delayrsquo] [rsquodelrsquo] TPCONF_delays )rsquolossrsquo ([rsquoV_lossrsquo] [rsquolossrsquo] TPCONF_loss_rates )rsquotcpalgosrsquo ([rsquoV_tcp_cc_algorsquo][rsquotcprsquo] TPCONF_TCP_algos )rsquoaqmsrsquo ([rsquoV_aqmrsquo] [rsquoaqmrsquo] TPCONF_aqms )rsquobsizesrsquo ([rsquoV_bsizersquo] [rsquobsrsquo] TPCONF_buffer_sizes )rsquorunsrsquo ([rsquoV_runsrsquo] [rsquorunrsquo] range(TPCONF_runs) )rsquobandwidthsrsquo ([rsquoV_down_ratersquo rsquoV_up_ratersquo] [rsquodownrsquo rsquouprsquo] TPCONF_bandwidths )

TPCONF_variable_defaults =

V_ variable valuersquoV_durationrsquo TPCONF_durationrsquoV_delayrsquo TPCONF_delays[0]rsquoV_lossrsquo TPCONF_loss_rates[0]rsquoV_tcp_cc_algorsquo TPCONF_TCP_algos[0]rsquoV_down_ratersquo TPCONF_bandwidths[0][0]rsquoV_up_ratersquo TPCONF_bandwidths[0][1]rsquoV_aqmrsquo TPCONF_aqms[0]rsquoV_bsizersquo TPCONF_buffer_sizes[0]

Specify the parameters we vary through all values all others will be fixed according to TPCONF_variable_defaultsTPCONF_vary_parameters = [rsquodelaysrsquo rsquobandwidthsrsquo rsquoaqmsrsquo rsquorunsrsquo]

Listing 8 Scenario 1 variable parameters configuration

command which creates intermediate files and the pdfplot files in a sub directory named results inside the testdata directory (the command needs to be executed in thedirectory where fabfilepy is)

gt fab analyse_allout_dir=results

Figure 15 shows the throughput and TCPrsquos smoothedRTT estimates produced by TEACUP for the experimentwith an emulated RTT of 50 ms a bandwidth of 8 mbitsdownstream and 1 mbits upstream and a buffer size of100 packets The hosts and the router ran Linux kernel3174 The throughput graph shows that both flows sharethe downstream link equally while the upstream linkcarrying only ACKs is not fully utilised The RTT graphshows that the total RTT reaches almost 100 ms due tothe buffering (the plotted RTT estimates for the ACKstreams are useless)

C Scenario 2 Scenario 1 plus automatic booting ofhosts

1) Topology Like in scenario 1 we have two hostsnewtcp20 connected to the 1721610024 network andnewtcp27 connected to the 1721611024 network Themachine newtcprt3 connects the two experiment subnetsAll three machines also have a second network interfacethat is used to control the experiment via TEACUP In

this scenario we assume that the hosts have been installedaccording to [7] including PXE booting and a multi-OS installation on each host as described in the techreport

2) Test Traffic Like in scenario 1 two TCP bulk transferflows are created using iperf

3) Variable Parameters Like in scenario 1 we emulatethree different delays and two different bandwidth set-tings ndash six different experiments in total We do alsodefine a variable parameter for AQM but define onlyone AQM (default pfifo) This causes the used AQM tobe logged as part of the experiment ID


Most of the configuration is identical to the configurationused in scenario 1 The only difference is that nowTEACUP will reboot the machines automatically into thespecified OS which is very useful if machines have amulti-boot setup ie can run different OS The automaticrebooting of multi-OS machines requires that machinesare configured with PXE+GRUB as described in the techreport If machines have only a single OS the rebootfunction is still useful to put machines into a clean state


0 5 10 15 20 25 30

0

1000

2000

3000

4000

20150211minus171242_experiment_del_25_down_8mbit_up_1mbit_aqm_pfifo_run_0

Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT

Figure 15 Throughput and TCP RTT estimate measured in Scenario 1

before an experiment and in this case PXE booting isnot needed

Listing 9 shows the additional configuration neededcompared to scenario 1 TPCONF_tftpboot_dir speci-fies the directory where TEACUP will put the filesthat GRUB will read (after loaded via PXE) whichspecify from which hard disk partition to boot fromTPCONF_host_os specifies the operating for each hostand router For Linux TEACUP allows to specifythe kernel that is booted TPCONF_linux_kern_routerspecifies the kernel booted on the router and TP-CONF_linux_kern_hosts specifies the kernel booted onthe other hosts If TPCONF_force_reboot is not setto lsquo0rsquo TEACUP will only reboot a host if the cur-rently running OS is different from the OS specifiedin TPCONF_host_os (Linux hosts will also be rebootedif the currently running kernel is different from thekernel specified in TPCONF_linux_kern_router or TP-CONF_linux_kern_hosts) TPCONF_boot_timeout spec-ifies the amount of time TEACUP will wait for a hostto be rebooted and accessible via SSH again If a hostis not rebooted and running the desired OS within thistime TEACUP will abort with an error

Currently by default TEACUP expects Windows onpartition 1 Linux on partition 2 FreeBSD on partition3 on the first hard disk However the variable TP-CONF_os_partition can be used to specify the partitionsin GRUB4DOS format PXE booting of MacOS is notsupported currently

5) Run and analyse experiment See scenario 1

D Scenario 3 Scenario 1 plus power control

1) Topology Like in scenario 1 we have two hostsnewtcp20 connected to the 1721610024 network andnewtcp27 connected to the 1721611024 network Themachine newtcprt3 connects the two experiment subnetsAll three machines also have a second network interfacethat is used to control the experiment via TEACUP Weassume that the hosts have been installed according to[7] Furthermore this scenario only works with installedTEACUP-compatible power controllers that are set up tocontrol the power of the three hosts




Most of the configuration is identical to the configurationused in scenario 2 The only different is that nowTEACUP will automatically power cycle machines thatto not come up within the reboot timeout TEACUPwill only power cycle machines once If after a powercycle and reboot the machines are still unresponsive


Path to tftp server handling the pxe boot Setting this to an empty string rsquorsquo means no PXE booting and TPCONF_host_os and TPCONF_force_reboot are simply ignoredTPCONF_tftpboot_dir = rsquotftpbootrsquo Operating system config machines that are not explicitly listed are left as they are (OS can be rsquoLinuxrsquo rsquoFreeBSDrsquo rsquoCYGWINrsquo rsquoDarwinrsquo)TPCONF_host_os =

rsquonewtcprt3rsquo rsquoLinuxrsquorsquonewtcp20rsquo rsquoLinuxrsquorsquonewtcp27rsquo rsquoLinuxrsquo

Specify the Linux kernel to use only used for machines running Linux (basically the full name without the vmlinuz-)TPCONF_linux_kern_router = rsquo3174-vanilla-10000hzrsquoTPCONF_linux_kern_hosts = rsquo3174-vanilla-web10grsquo Force reboot If set to rsquo1rsquo will force a reboot of all hosts If set to rsquo0rsquo only hosts where OS is not the desired OS will be rebootedTPCONF_force_reboot = rsquo0rsquo Time to wait for reboot in seconds (integer) Minimum timeout is 60 secondsTPCONF_boot_timeout = 120 Map OS to partition on hard disk (note the partition must be specified in the GRUB4DOS format _not_ GRUB2 format)TPCONF_os_partition =

rsquoCYGWINrsquo rsquo(hd00)rsquorsquoLinuxrsquo rsquo(hd01)rsquorsquoFreeBSDrsquo rsquo(hd02)rsquo

Listing 9 Scenario 2 OS and reboot configuration

TEACUP will give up Power cycling can only usedif the machines are connected via a power controllersupported by TEACUP Currently TEACUP supportstwo power controllers IP Power 9258HP (9258HP) andServerlink SLP-SPP1008-H (SLP-SPP1008)

Listing 10 shows the additional configuration neededcompared to scenario 1 TPCONF_do_power_cycle mustbe set to lsquo1rsquo to perform power cycling If power cyclingis used TPCONF_host_power_ctrlport must define theIP address of the responsible power controller for eachmachine as well as the power controllerrsquos port themachine is connected to TPCONF_power_admin_nameand TPCONF_power_admin_pw must specify the adminuserrsquos name and password required to login to the powercontrollerrsquos web interface TPCONF_power_ctrl_typespecifies the type of power controller


E Scenario 4 TCP flows with same bottleneck queuebut different delay

1) Topology We now have two hosts in each subnetnewtcp20 and newtcp21 connected to the 1721610024network newtcp27 and newtcp28 connected to the1721611024 network The machine newtcprt3 con-nects the two experiment subnets All three machinesalso have a second network interface that is used tocontrol the experiment via TEACUP Like scenario 1this scenario requires that hosts have the traffic generatorand logging tools installed as described in [7] but PXEbooting or a multi-OS installation is not needed

2) Test Traffic Two TCP bulk transfer flows are cre-ated using iperf One flow is between newtcp20 andnewtcp27 and the second flow is between newtcp21 andnewtcp28

3) Variable Parameters Like in scenario 1 we emulatetwo different bandwidth settings However we setupdifferent delays for each flow Since each flow can havetwo different values we have eight different experimentsin total We do also define a variable parameter for AQM


If host does not come up within timeout force power cycle If set to rsquo1rsquo force power cycle if host not up within timeout If set to rsquo0rsquo never force power cycleTPCONF_do_power_cycle = rsquo1rsquo Maps host to power controller IP (or name) and power controller port numberTPCONF_host_power_ctrlport =

rsquonewtcprt3rsquo (rsquo1000100rsquo rsquo1rsquo)rsquonewtcp20rsquo (rsquo1000100rsquo rsquo2rsquo)rsquonewtcp27rsquo (rsquo1000100rsquo rsquo3rsquo)

Power controller admin user nameTPCONF_power_admin_name = rsquoadminrsquo Power controller admin user passwordTPCONF_power_admin_pw = envpassword Type of power controller Currently supported are only IP Power 9258HP (9258HP) and Serverlink SLP-SPP1008-H (SLP-SPP1008)TPCONF_power_ctrl_type = rsquoSLP-SPP1008rsquo

Listing 10 Scenario 3 power controller configuration

but define only one AQM (default pfifo) This causesthe used AQM to be logged as part of the experimentID

4) TEACUP Config File You can download the config-uration file from [24] To use the configuration renameit to configpy In the following we explain the partsof the configuration file that have changed compared toscenario 1

Our host configuration now looks as in Listing 11

To configure different delays for each flow we also needto change the router setup (see Listing 12)

We also need to define both of the V_ variables andmake sure we iterate over both in the experiments (seeListing 13)

Finally we need to define the traffic generators to createone TCP bulk transfer flow for each host pair as shownin Listing 14


F Scenario 5 Partially overlapping TCP flows withdifferent TCP CC

1) Topology We now have three hosts in each subnetnewtcp20 newtcp21 and newtcp22 connected to the1721610024 network and newtcp27 newtcp28 andnewtcp29 connected to the 1721611024 network Themachine newtcprt3 connects the two experiment subnetsAll three machines also have a second network interfacethat is used to control the experiment via TEACUP

Like scenario 1 this scenario requires that hosts have thetraffic generator and logging tools installed as describedin [7] but PXE booting or a multi-OS installation is notneeded

2) Test Traffic Three TCP bulk transfer flows are cre-ated using iperf One flow is between newtcp20 andnewtcp27 the second flow is between newtcp21 andnewtcp28 and the third flow is between newtcp22 andnewtcp29 The flows do not longer start at the same timeFlow one start at the start of the experiment while flow2 starts 10 seconds after the first flow and flow 3 starts10 seconds after flow 2 All flows have a duration of 30seconds as in scenario 1

3) Variable Parameters Like in scenario 1 we emulatethree different delays (same delays for all flows) andtwo different bandwidth settings However we now alsovary the used TCP congestion control algorithm betweenNewreno and Cubic This means we have 12 differentexperiments in total We do also define a variable pa-rameter for AQM but define only one AQM (defaultpfifo) This causes the used AQM to be logged as partof the experiment ID


Our host configuration now looks as shown in Listing15


TPCONF_router = [rsquonewtcprt3rsquo ]TPCONF_hosts = [ rsquonewtcp20rsquo rsquonewtcp21rsquo rsquonewtcp27rsquo rsquonewtcp28rsquo ]TPCONF_host_internal_ip =

rsquonewtcprt3rsquo [rsquo17216101rsquo rsquo17216111rsquo]rsquonewtcp20rsquo [rsquo172161060rsquo]rsquonewtcp21rsquo [rsquo172161061rsquo]rsquonewtcp27rsquo [rsquo172161167rsquo]rsquonewtcp28rsquo [rsquo172161168rsquo]

Listing 11 Scenario 4 host configuration

TPCONF_router_queues = [(rsquo1rsquo source=rsquo172161060rsquo dest=rsquo172161167rsquo delay=V_delay loss=V_loss rate=V_up_rate queue_disc=V_aqm queue_size=V_bsize )


(rsquo3rsquo source=rsquo172161061rsquo dest=rsquo172161168rsquo delay=V_delay2 loss=V_loss rate=V_up_rate queue_disc=V_aqm queue_size=V_bsize attach_to_queue=rsquo1rsquo )

(rsquo4rsquo source=rsquo172161168rsquo dest=rsquo172161061rsquo delay=V_delay2 loss=V_loss rate=V_down_rate queue_disc=V_aqm queue_size=V_bsize attach_to_queue=rsquo2rsquo )

]


The traffic generator setup now creates three staggeredflows as shown in Listing 16

We also need to configure the different TCP congestioncontrol algorithms and instruct TEACUP to vary thisparameter (see Listing 17)


G Scenario 6 Two HTTP-based video streamingclients

1) Topology We now have hosts newtcp20 newtcp21connected to the 1721610024 network and hostnewtcp27 connected to the 1721611024 network Themachine newtcprt3 connects the two experiment subnetsAll three machines also have a second network interfacethat is used to control the experiment via TEACUPLike scenario 1 this scenario requires that hosts have thetraffic generator and logging tools installed as describedin [7] but PXE booting or a multi-OS installation is notneeded

2) Test Traffic In this scenario we simulate DASH-like HTTP video streaming Host newtcp27 runs a webserver Host newtcp20 and newtcp21 are clients and usehttperf to emulate DASH-like streaming ndash there is one

DASH-like flow between each client and the server Inthis scenario we have fixed the video rates and onoffcycle times but the rate and cycle time differs for bothstreams

3) Variable Parameters Like in scenario 1 we emulatethree different delays (same delays for all flows) and twodifferent bandwidth settings We now also vary the AQMmechanism used on the router between FIFO CoDeland FQ CoDel This means we have 18 experiments intotal



The traffic generator setup now starts a web serverand generates fake streaming content on newtcp27 be-fore starting the DASH-like flows as shown in Listing19

Finally we need to configure the differentAQM mechanisms used (see Listing 20)TPCONF_vary_parameters already included the lsquoaqmsrsquo


TPCONF_delays = [5 50]TPCONF_delays2 = [5 50]TPCONF_parameter_list =

Vary name V_ variable file name values extra varsrsquodelaysrsquo ([rsquoV_delayrsquo] [rsquodel1rsquo] TPCONF_delays )rsquodelays2rsquo ([rsquoV_delay2rsquo] [rsquodel2rsquo] TPCONF_delays2 )rsquolossrsquo ([rsquoV_lossrsquo] [rsquolossrsquo] TPCONF_loss_rates )rsquotcpalgosrsquo ([rsquoV_tcp_cc_algorsquo][rsquotcprsquo] TPCONF_TCP_algos )rsquoaqmsrsquo ([rsquoV_aqmrsquo] [rsquoaqmrsquo] TPCONF_aqms )rsquobsizesrsquo ([rsquoV_bsizersquo] [rsquobsrsquo] TPCONF_buffer_sizes )rsquorunsrsquo ([rsquoV_runsrsquo] [rsquorunrsquo] range(TPCONF_runs) )rsquobandwidthsrsquo ([rsquoV_down_ratersquo rsquoV_up_ratersquo] [rsquodownrsquo rsquouprsquo] TPCONF_bandwidths )


V_ variable valuersquoV_durationrsquo TPCONF_durationrsquoV_delayrsquo TPCONF_delays[0]rsquoV_delay2rsquo TPCONF_delays2[0]rsquoV_lossrsquo TPCONF_loss_rates[0]rsquoV_tcp_cc_algorsquo TPCONF_TCP_algos[0]rsquoV_down_ratersquo TPCONF_bandwidths[0][0]rsquoV_up_ratersquo TPCONF_bandwidths[0][1]rsquoV_aqmrsquo TPCONF_aqms[0]rsquoV_bsizersquo TPCONF_buffer_sizes[0]

TPCONF_vary_parameters = [rsquodelaysrsquo rsquodelays2rsquo rsquobandwidthsrsquo rsquoaqmsrsquo rsquorunsrsquo]

Listing 13 Scenario 4 experiment parameter configuration



duration=V_duration )]TPCONF_traffic_gens = traffic_iperf


parameters in scenario 1 so TPCONF_parameter_listand TPCONF_variable_defaults look like in scenario 1We only have to change TPCONF_aqms


H Scenario 7 Incast problem scenario

1) Topology We now have the host newtcp20 connectedto the 1721610024 network and hosts newtcp22ndash30connected to the 1721611024 network The machinenewtcprt3 connects the two experiment subnets All threemachines also have a second network interface thatis used to control the experiment via TEACUP Likescenario 1 this scenario requires that hosts have thetraffic generator and logging tools installed as described

in [7] but PXE booting or a multi-OS installation is notneeded

2) Test Traffic In this scenario we emulate traffic toinvestigate the incast problem The host newtcp20 isthe querier and in regular intervals sends simultane-ous queries to hosts newtcp21 newtcp22 newtcp23newtcp24 newtcp25 newtcp26 newtcp27 newtcp28newtcp29 and newtcp30 which then respond simultane-ously The querier uses httperf to send the queries to webservers running on all of the responders

3) Variable Parameters Like in scenario 6 we emulatethree different delays (same delays for all flows) twodifferent bandwidth settings and three different AQMmechanism (FIFO CoDel and FQ CoDel) We now also


TPCONF_router = [rsquonewtcprt3rsquo ]TPCONF_hosts = [ rsquonewtcp20rsquo rsquonewtcp21rsquo rsquonewtcp22rsquo rsquonewtcp27rsquo rsquonewtcp28rsquo rsquonewtcp29rsquo ]TPCONF_host_internal_ip =

rsquonewtcprt3rsquo [rsquo17216101rsquo rsquo17216111rsquo]rsquonewtcp20rsquo [rsquo172161060rsquo]rsquonewtcp21rsquo [rsquo172161061rsquo]rsquonewtcp22rsquo [rsquo172161062rsquo]rsquonewtcp27rsquo [rsquo172161167rsquo]rsquonewtcp28rsquo [rsquo172161168rsquo]rsquonewtcp29rsquo [rsquo172161169rsquo]







vary the size of the response between six different valuesThis means we have 108 experiments in total



The traffic generator setup now starts a web server andgenerates fake streaming content on each responderThen after a delay of one second it starts the querierListing 22 shows the configuration Setting up eachserver individually requires a large number of entriesin the traffic configuration Since version 09 TEACUPsupports the start_httperf_incast_n traffic generator thatwill setup the n servers and the querier with a singleentry

Next we need to configure the value ranges for responsesizes and the time period between sending queries (seeListing 23)

Finally we need to configure TEACUP to make surethe V_ variables used in the traffic generator setup aredefined and we vary the response sizes (see Listing24)


IX EXTENDING TEACUP FUNCTIONALITY

This section contains some notes on extending thecurrent implementation We refer to Python functions(which can be Fabric tasks) using the notation ofltpython_filegtpyltfunctiongt()

A Additional host setup

Any general host setup (eg sysctl settings for all experi-ments) should be added in hostsetuppyinit_host()Note that in this function there are different sectionsone for each OS (FreeBSD Linux WindowsCygwinMac OS X) Commands that shall only be executedin certain experiments can be set in the config (TP-CONF_host_init_custom_cmds)

B New TCP congestion control algorithm

Adding support for a new TCP conges-tion control algorithm requires modifyinghostsetuppyinit_cc_algo() The new algorithmneeds to be added to the list of supported algorithmsand in the OS-specific sections code need to be addedto load the corresponding kernel module (if any)


TPCONF_TCP_algos = [rsquonewrenorsquo rsquocubicrsquo ]TPCONF_parameter_list =

rsquodelaysrsquo ([rsquoV_delayrsquo] [rsquodelrsquo] TPCONF_delays )rsquolossrsquo ([rsquoV_lossrsquo] [rsquolossrsquo] TPCONF_loss_rates )rsquotcpalgosrsquo ([rsquoV_tcp_cc_algorsquo][rsquotcprsquo] TPCONF_TCP_algos )rsquoaqmsrsquo ([rsquoV_aqmrsquo] [rsquoaqmrsquo] TPCONF_aqms )rsquobsizesrsquo ([rsquoV_bsizersquo] [rsquobsrsquo] TPCONF_buffer_sizes )rsquorunsrsquo ([rsquoV_runsrsquo] [rsquorunrsquo] range(TPCONF_runs) )rsquobandwidthsrsquo ([rsquoV_down_ratersquo rsquoV_up_ratersquo] [rsquodownrsquo rsquouprsquo] TPCONF_bandwidths )


rsquoV_durationrsquo TPCONF_durationrsquoV_delayrsquo TPCONF_delays[0]rsquoV_lossrsquo TPCONF_loss_rates[0]rsquoV_tcp_cc_algorsquo TPCONF_TCP_algos[0]rsquoV_down_ratersquo TPCONF_bandwidths[0][0]rsquoV_up_ratersquo TPCONF_bandwidths[0][1]rsquoV_aqmrsquo TPCONF_aqms[0]rsquoV_bsizersquo TPCONF_buffer_sizes[0]

TPCONF_vary_parameters = [rsquotcpalgosrsquo rsquodelaysrsquo rsquobandwidthsrsquo rsquoaqmsrsquo rsquorunsrsquo]

Listing 17 Scenario 5 variable parameter configuration

TPCONF_router = [rsquonewtcprt3rsquo ]TPCONF_hosts = [ rsquonewtcp20rsquo rsquonewtcp21rsquo rsquonewtcp27rsquo ]TPCONF_host_internal_ip =

rsquonewtcprt3rsquo [rsquo17216101rsquo rsquo17216111rsquo]rsquonewtcp20rsquo [rsquo172161060rsquo]rsquonewtcp21rsquo [rsquo172161061rsquo]rsquonewtcp27rsquo [rsquo172161167rsquo]


C New traffic generator

Adding a new traffic generator requires adding a newstart task in trafficgenspy The current start tasksalways consist of two methods the actual start method isa wrapper around an internal _start method This allowshaving the host on which the generator is started asexplicit parameter (and not as Fabric hosts parameter)and having multiple traffic generators that actually usethe same underlying tool (for example this is the casefor httperf) The new start method must be added to theimports in experimentpy

The traffic generator start function must after thetraffic generator process has been started registerthe started process with its process ID by callingbgprocregister_proc() This ensures that the pro-cess will be stopped at the end of the experimentand the traffic generatorrsquos log file is collected whenrunbgpystop_processes() is called A current lim-

itation is that there can only be one log file per trafficgenerator process

Some traffic generators also have stop methods Initiallythe idea was that traffic generators could be started andstopped from the command line directly but this is notsupported at the moment ie some stop methods are notimplemented (empty)

D New data logger

To add a new data logger a start method andpossibly a stop method need to be added inloggerspy The new loggerrsquos start method shouldbe called from loggerspystart_loggers() viaFabricrsquos execute() but could also be called fromexperimentpyrun_experiment() if required (in thelatter case it must be added to the imports inexperimentpy)


traffic_dash = [ Start server and create content (server must be started first)(rsquo00rsquo rsquo1rsquo start_http_server server=rsquonewtcp27rsquo port=80 )(rsquo00rsquo rsquo2rsquo create_http_dash_content server=rsquonewtcp27rsquo duration=2V_duration

rates=rsquo500 1000rsquo cycles=rsquo5 10rsquo ) Create DASH-like flows(rsquo05rsquo rsquo3rsquo start_httperf_dash client=rsquonewtcp20rsquo server=rsquonewtcp27rsquo port=80

duration=V_duration rate=500 cycle=5 prefetch=20 prefetch_timeout=20 )

(rsquo05rsquo rsquo4rsquo start_httperf_dash client=rsquonewtcp20rsquo server=rsquonewtcp27rsquo port=80 duration=V_duration rate=1000 cycle=10 prefetch=20 prefetch_timeout=20 )

]TPCONF_traffic_gens = traffic_dash


TPCONF_aqms = [rsquopfiforsquo rsquocodelrsquo rsquofq_codelrsquo ]TPCONF_vary_parameters = [rsquodelaysrsquo rsquobandwidthsrsquo rsquoaqmsrsquo rsquorunsrsquo]

Listing 20 Scenario 6 parameter configuration

TPCONF_router = [rsquonewtcprt3rsquo ]TPCONF_hosts = [

rsquonewtcp20rsquo rsquonewtcp21rsquo rsquonewtcp22rsquo rsquonewtcp23rsquo rsquonewtcp24rsquorsquonewtcp25rsquo rsquonewtcp26rsquo rsquonewtcp27rsquo rsquonewtcp28rsquo rsquonewtcp29rsquo rsquonewtcp30rsquo

]TPCONF_host_internal_ip =

rsquonewtcprt3rsquo [rsquo17216101rsquo rsquo17216111rsquo]rsquonewtcp20rsquo [rsquo172161060rsquo] querierrsquonewtcp21rsquo [rsquo172161161rsquo] respondersrsquonewtcp22rsquo [rsquo172161162rsquo]rsquonewtcp23rsquo [rsquo172161163rsquo]rsquonewtcp24rsquo [rsquo172161164rsquo]rsquonewtcp25rsquo [rsquo172161165rsquo]rsquonewtcp26rsquo [rsquo172161166rsquo]rsquonewtcp27rsquo [rsquo172161167rsquo]rsquonewtcp28rsquo [rsquo172161168rsquo]rsquonewtcp29rsquo [rsquo172161169rsquo]rsquonewtcp30rsquo [rsquo172161170rsquo]


If the logger is a userspace process such astcpdump at the end of the start function it shouldregister itself (including its process ID) usingbgprocregister_proc_later() Then it is ensuredthat the logging process will be stopped at the endof the experiment and the log file is collected whenrunbgpystop_processes() is called In this case nostop method needs to be implemented

If the logger is not a userspace process for exam-ple SIFTR on FreeBSD start and stop methods needto be implemented The start method must still call

bgprocregister_proc_later() but the process IDmust be set to zero The stop method must be calledfrom runbgpystop_processes() if the process ID iszero and the internal TEACUP name of the process isthe name of the new logger

E New analysis method

To add a new analysis method add an analysis taskin analysispy If the new analysis should be carriedout as part of the analyse_all task the new task must


traffic_incast = [ Start servers and create contents (server must be started first)(rsquo00rsquo rsquo1rsquo start_http_server server=rsquonewtcp21rsquo port=80 )(rsquo00rsquo rsquo2rsquo start_http_server server=rsquonewtcp22rsquo port=80 )(rsquo00rsquo rsquo3rsquo start_http_server server=rsquonewtcp23rsquo port=80 )(rsquo00rsquo rsquo4rsquo start_http_server server=rsquonewtcp24rsquo port=80 )(rsquo00rsquo rsquo5rsquo start_http_server server=rsquonewtcp25rsquo port=80 )(rsquo00rsquo rsquo6rsquo start_http_server server=rsquonewtcp26rsquo port=80 )(rsquo00rsquo rsquo7rsquo start_http_server server=rsquonewtcp27rsquo port=80 )(rsquo00rsquo rsquo8rsquo start_http_server server=rsquonewtcp28rsquo port=80 )(rsquo00rsquo rsquo9rsquo start_http_server server=rsquonewtcp29rsquo port=80 )(rsquo00rsquo rsquo10rsquo start_http_server server=rsquonewtcp30rsquo port=80 )(rsquo00rsquo rsquo11rsquo create_http_incast_content server=rsquonewtcp21rsquo duration=2V_duration

sizes=V_inc_content_sizes_str )(rsquo00rsquo rsquo12rsquo create_http_incast_content server=rsquonewtcp22rsquo duration=2V_duration









sizes=V_inc_content_sizes_str ) Start querier(rsquo10rsquo rsquo30rsquo start_httperf_incast client=rsquonewtcp20rsquo

servers=rsquonewtcp2180newtcp2280newtcp2380newtcp2480newtcp2580newtcp2680 newtcp2780newtcp2880newtcp2980newtcp3080rsquo duration=V_duration period=V_inc_period response_size=V_inc_size)

]TPCONF_traffic_gens = traffic_incast


TPCONF_inc_content_sizes = [8 16 32 64 128 256]TPCONF_inc_content_sizes_str = rsquorsquojoin( str(x) for x in TPCONF_inc_content_sizes)TPCONF_inc_periods = [10]

Listing 23 Scenario 7 incast content parameter configuration

be called from analysispyanalyse_all() via Fab-rics execute() function The new task should imple-ment the common parameters test_id out_dir pdf_dirout_name replot_only source_filter min_values etimestime ymin ymax (see existing analyse tasks as exam-ples) The new task must be added to the imports infabfilepy

X KNOWN ISSUES

During the host setup phase TEACUP enables anddisables NICs On Windows the enable and disableNIC commands have permanent effect If TEACUPis interrupted or aborts between a disable and enable

command the NIC will stay disabled TEACUP willautomatically enable all testbed NICs on Windows priorto each experiment however in the unlikely event thatthe previous aborted NIC configuration left the NIC inan inconsistent state it may be necessary to reconfigurethe NIC manually

TEACUP logs all output from traffic generators suchas iperf or httperf Some of the tools used only generateoutput after they completed If an experiment ends beforea tool completed its task the resulting log file may beempty Possibly this issue could be mitigated by turningthe stdout and stderr buffering off for these tools in futureversions


TPCONF_parameter_list = rsquodelaysrsquo ([rsquoV_delayrsquo] [rsquodelrsquo] TPCONF_delays )rsquolossrsquo ([rsquoV_lossrsquo] [rsquolossrsquo] TPCONF_loss_rates )rsquotcpalgosrsquo ([rsquoV_tcp_cc_algorsquo][rsquotcprsquo] TPCONF_TCP_algos )rsquoaqmsrsquo ([rsquoV_aqmrsquo] [rsquoaqmrsquo] TPCONF_aqms )rsquobsizesrsquo ([rsquoV_bsizersquo] [rsquobsrsquo] TPCONF_buffer_sizes )rsquorunsrsquo ([rsquoV_runsrsquo] [rsquorunrsquo] range(TPCONF_runs) )rsquobandwidthsrsquo ([rsquoV_down_ratersquo rsquoV_up_ratersquo] [rsquodownrsquo rsquouprsquo] TPCONF_bandwidths )rsquoincast_periodsrsquo ([rsquoV_inc_periodrsquo] [rsquoincperrsquo] TPCONF_inc_periods )rsquoincast_sizesrsquo ([rsquoV_inc_sizersquo] [rsquoincszrsquo] TPCONF_inc_content_sizes)


rsquoV_durationrsquo TPCONF_durationrsquoV_delayrsquo TPCONF_delays[0]rsquoV_lossrsquo TPCONF_loss_rates[0]rsquoV_tcp_cc_algorsquo TPCONF_TCP_algos[0]rsquoV_down_ratersquo TPCONF_bandwidths[0][0]rsquoV_up_ratersquo TPCONF_bandwidths[0][1]rsquoV_aqmrsquo TPCONF_aqms[0]rsquoV_bsizersquo TPCONF_buffer_sizes[0]rsquoV_inc_periodrsquo TPCONF_inc_periods[0]rsquoV_inc_sizersquo TPCONF_inc_content_sizes[0]rsquoV_inc_content_sizes_strrsquo TPCONF_inc_content_sizes_str

TPCONF_vary_parameters = [rsquoincast_sizesrsquo rsquodelaysrsquo rsquobandwidthsrsquo rsquoaqmsrsquo rsquorunsrsquo]


XI CONCLUSIONS AND FUTURE WORK

In this report we described TEACUP a Python-basedsoftware we developed to run automated TCP perfor-mance tests in a controlled testbed In the future we willcontinue to extend TEACUP with more features

ACKNOWLEDGEMENTS

TEACUP v09 was developed as part of a project fundedby Cisco Systems and titled ldquoStudy in TCP CongestionControl Performance In A Data Centrerdquo This is acollaborative effort between CAIA and Fred Baker ofCisco Systems

REFERENCES

[1] A Finamore M Mellia M M Munafograve R Torres and S GRao ldquoYoutube everywhere Impact of device and infrastructuresynergies on user experiencerdquo in Proceedings of the 2011 ACMSIGCOMM Conference on Internet Measurement Conferenceser IMC rsquo11 2011 pp 345ndash360

[2] A Rao A Legout Y-s Lim D Towsley C Barakat andW Dabbous ldquoNetwork characteristics of video streaming traf-ficrdquo in Proceedings of the Seventh COnference on EmergingNetworking EXperiments and Technologies ser CoNEXT rsquo112011 pp 251ndash2512

[3] ldquoDynamic adaptive streaming over HTTP (DASH) ndash Part 1 Me-dia presentation description and segment formatsrdquo ISO 2012iSOIEC 23009-12012 httpwwwisoorgisoiso_cataloguecatalogue_tccatalogue_detailhtmcsnumber=57623

[4] S Zander G Armitage ldquoTEACUP v08 ndash A System forAutomated TCP Testbed Experimentsrdquo Centre for AdvancedInternet Architectures Swinburne University of TechnologyTech Rep 150210A 2015 [Online] Available httpcaiaswineduaureports150210ACAIA-TR-150210Apdf

[5] L Stewart ldquoSIFTR ndash Statistical Information For TCPResearchrdquo [Online] Available httpcaiaswineduauurpnewtcptoolshtml

[6] ldquoThe Web10G Projectrdquo [Online] Available httpweb10gorg

[7] S Zander G Armitage ldquoCAIA Testbed for TCPExperiments Version 2rdquo Centre for Advanced InternetArchitectures Swinburne University of Technology Tech Rep150210C 2015 [Online] Available httpcaiaswineduaureports150210CCAIA-TR-150210Cpdf

[8] mdashmdash ldquoTEACUP v09 ndash Data Analysis FunctionsrdquoCentre for Advanced Internet Architectures SwinburneUniversity of Technology Tech Rep 150414B 2015[Online] Available httpcaiaswineduaureports150414BCAIA-TR-150414Bpdf

[9] ldquoFabric 18 documentationrdquo [Online] Available httpdocsfabfileorgen18

[10] ldquoFabric 18 documentation ndash Installationrdquo [Online] Availablehttpdocsfabfileorgen18installationhtml

[11] ldquoiperf Web Pagerdquo [Online] Available httpiperffr


[12] ldquoLIGHTTPD Web Serverrdquo [Online] Available httpwwwlighttpdnet

[13] HP Labs ldquohttperf homepagerdquo [Online] Available httpwwwhplhpcomresearchlinuxhttperf

[14] J Summers T Brecht D Eager B Wong ldquoModified versionof httperfrdquo 2012 [Online] Available httpscsuwaterlooca~brechtpapersnossdav-2012httperftgz

[15] ldquonttcp-147 ndash An improved version of the popular ttcpprogramrdquo [Online] Available httphpuxconnectorgukhppdhpuxNetworkingAdminnttcp-147

[16] M Mathis J Heffner and R Raghunarayan ldquoTCP ExtendedStatistics MIBrdquo RFC 4898 (Proposed Standard) InternetEngineering Task Force May 2007 [Online] Availablehttpwwwietforgrfcrfc4898txt

[17] Wikipedia ldquoDTracerdquo httpenwikipediaorgwindexphptitle=DTraceampoldid=637195163

[18] L Stewart ldquoSIFTR v123 READMErdquo July 2010[Online] Available httpcaiaswineduauurpnewtcptoolssiftr-readme-123txt

[19] M Mathis J Semke R Reddy J Heffner ldquoDocumentationof variables for the Web100 TCP Kernel Instrumentation Set(KIS) projectrdquo [Online] Available httpwwwweb100orgdownloadkerneltcp-kistxt

[20] ldquonetfilter ndash firewalling NAT and packet mangling for Linuxrdquo[Online] Available httpwwwnetfilterorg

[21] J Gettys ldquoBest practices for benchmarking bufferbloatrdquo [On-line] Available httpwwwbufferbloatnetprojectscodelwikiBest_practices_for_benchmarking_Codel_and_FQ_Codel

[22] Linux Foundation ldquonetem ndash Network Emula-tion Functionalityrdquo November 2009 [Online] Avail-able httpwwwlinuxfoundationorgcollaborateworkgroupsnetworkingnetem

[23] mdashmdash ldquoIFB ndash Intermediate Functional Block devicerdquo November2009 [Online] Available httpwwwlinuxfoundationorgcollaborateworkgroupsnetworkingifb

[24] S Zander ldquoTCP Experiment Automation Controlled UsingPython (TEACUP) ndash Usage Examplesrdquo 2014 [Online]Available httpcaiaswineduautoolsteacupusageexampleshtml


Contents

Introduction

TEACUP Requirements and Design

Requirements

Overall design

Experiment process flow

Fabric -- overview and installation

Traffic Generation and Logging

Traffic sources and sinks

Loggers

Host information logged

Experiment config information logged

Log file naming

Host and Router Configuration

Host setup

Linux router setup

FreeBSD router setup

Config File

Config file location

V_variables

Fabric configuration

Testbed configuration

General experiment settings

tcpdumppcap configuration

Topology configuration

Rebooting OS selection and power cycling

Clock offset measurement (broadcast pings)

Custom host init commands

Router queue setup

Traffic generator setup

Available traffic generators

Mandatory experiment variables

Experiment-specific variables

Defining parameters to vary

Adding new V_ variables

Running Experiments

Initial steps

Example config

Running experiments

TEACUP variable information logged

Host Control Utility Functions

Remote command execution

Copying files to testbed hosts

Installing ssh keys


Initialise hosts to a specific operating system

Power cycling

Check software installations on testbed hosts

Check testbed host connectivity

Check TEACUP config file

Experiment Examples

Overview

Scenario 1 Two TCP flows from data sender to receiver

Scenario 2 Scenario 1 plus automatic booting of hosts

Scenario 3 Scenario 1 plus power control

Scenario 4 TCP flows with same bottleneck queue but different delay

Scenario 5 Partially overlapping TCP flows with different TCP CC

Scenario 6 Two HTTP-based video streaming clients

Scenario 7 Incast problem scenario

Extending TEACUP Functionality

Additional host setup

New TCP congestion control algorithm

New traffic generator

New data logger

New analysis method

Known Issues

Conclusions and Future Work

References

V-Q Adding new V_ variables 23

VI Running Experiments 24VI-A Initial steps 24VI-B Example config 24VI-C Running experiments 24VI-D TEACUP variable information

logged 26

VII Host Control Utility Functions 26VII-A Remote command execution 27VII-B Copying files to testbed hosts 27VII-C Installing ssh keys 27VII-D Topology configuration 27VII-E Initialise hosts to a specific oper-

ating system 27VII-F Power cycling 28VII-G Check software installations on

testbed hosts 28VII-H Check testbed host connectivity 28VII-I Check TEACUP config file 28

VIII Experiment Examples 29VIII-A Overview 29VIII-B Scenario 1 Two TCP flows from

data sender to receiver 29VIII-C Scenario 2 Scenario 1 plus auto-

matic booting of hosts 33VIII-D Scenario 3 Scenario 1 plus power

control 34VIII-E Scenario 4 TCP flows with same

bottleneck queue but different delay 35VIII-F Scenario 5 Partially overlapping

TCP flows with different TCP CC 36VIII-G Scenario 6 Two HTTP-based

video streaming clients 37VIII-H Scenario 7 Incast problem scenario 38

IX Extending TEACUP Functionality 39IX-A Additional host setup 39IX-B New TCP congestion control al-

gorithm 39IX-C New traffic generator 40IX-D New data logger 40IX-E New analysis method 41

X Known Issues 42

XI Conclusions and Future Work 43

References 43

I INTRODUCTION

Over the last few decades several TCP congestion controlalgorithms were developed in order to optimise TCPrsquosbehaviour in certain situations While TCP was tradi-tionally used mainly for file transfers more recently itis also becoming the protocol of choice for streamingapplications for example video streaming from YouTubeand Netflix is TCP-based [1] [2] and there is an ISOstandard for Dynamic Adaptive Streaming over HTTP(DASH) [3] However the impact of different TCPcongestion control algorithms on TCP-based streamingflows (within a mix of other typical traffic) is not wellunderstood Experiments in a controlled testbed allowshedding more light on this issue

This report describes TEACUP (TCP Experiment Au-tomation Controlled Using Python) version 09 ndash asoftware tool for running automated TCP experimentsin a testbed It updates the previous tech reports de-scribing TEACUP version 04 version 06 and version08 [4] Based on a configuration file TEACUP canperform a series of experiments with different trafficmixes different bottlenecks (such as bandwidths queuemechanisms) different emulated network delays andorloss rates and different host settings (eg TCP conges-tion control algorithm) For each experiment TEACUPautomatically collects relevant information that allowsanalysing TCP behaviour such as tcpdump files SIFTR[5] and Web10G [6] logs The related technical report [7]describes the design and implementation of our testbedin which we use TEACUP

TEACUP also provides a number of tasks for the anal-ysis of data collected during experiments The analysisaspect of TEACUP is quite distinct from the tasks thatactually run experiments and gather testbed data hencethese are described in the accompanying technical report[8]

This report is organised as follows Section II describesthe overall design of TEACUP ndash the testbed networkmodel process flow and use of Fabric Section IIIoutlines the traffic generators and logging available underTEACUP while host and bottleneck router configurationis summarised in Section IV Section V describes theconfiguration options available when running experi-ments as described in Section VI Section VII describesutility functions that can be used for host maintenanceSection IX outlines how to extend TEACUP Section X





A Requirements












B Overall design




13

13

13

$ $





















1313

13 13

13

13

1313

1313









function


















13

13131313










B Loggers





























E Log file naming















A Host setup
















































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References




A Requirements












B Overall design




13

13

13

$ $





















1313

13 13

13

13

1313

1313









function


















13

13131313










B Loggers





























E Log file naming















A Host setup
















































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References

13

13

13

$ $





















1313

13 13

13

13

1313

1313









function


















13

13131313










B Loggers





























E Log file naming















A Host setup
















































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References

1313

13 13

13

13

1313

1313









function


















13

13131313










B Loggers





























E Log file naming















A Host setup
















































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References

13

13131313










B Loggers





























E Log file naming















A Host setup
















































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References






















E Log file naming















A Host setup
















































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References









A Host setup
















































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References






































13

13

$

amp

13

()$



V CONFIG FILE







B V_variables












envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References





envtimeout = 5


envpool_size = 10





































































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References






















































































testrouter2_queues








































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References





































ping










seconds



























httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References












httperf













































TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References


















TPCONF_runs = 1















TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References




TPCONF_delays = [ 0 25 50 100 ]













]





TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References


TPCONF_dash_rates))






















A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References







A Initial steps


B Example config











+Ymd-HMSlsquo














gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References












gt runsh






















gt run_resumesh








gt fab -l












remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References











remote_path=usrbin









authorize_key












force_reboot=1










F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References








F Power cycling






check_host




check_connectivity



gt fab check_config


gt fab get_version




rights reserved



A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References


A Overview






































]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References




















]

















gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References











gt runsh






















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References

















0 5 10 15 20 25 30

0

1000

2000

3000

4000


Time (s)

Thr

ough

put (

kbps

)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(a) Throughput

0 5 10 15 20 25 30

0

20

40

60

80

100


Time (s)

Sm

ooth

ed T

CP

RT

T (

ms)

17216102_5000_17216112_5831417216102_5001_17216112_54807

17216112_58314_17216102_500017216112_54807_17216102_5001

(b) RTT




















































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References








































]





































































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References























































D New data logger







































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References





































X KNOWN ISSUES












ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References








ACKNOWLEDGEMENTS


REFERENCES



























Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References















Contents

Introduction


Requirements

Overall design





Loggers



Log file naming


Host setup

Linux router setup


Config File


V_variables









Router queue setup







Running Experiments

Initial steps

Example config

Running experiments





Installing ssh keys



Power cycling




Experiment Examples

Overview












New data logger

New analysis method

Known Issues


References

teacup v0.9 -- a system for automated tcp testbed...

Documents