the elephant the mice - divakau.diva-portal.org/smash/get/diva2:1251705/fulltext01.pdf · found:...

Print & layout University Printing Office, Karlstad 2018

DOCTORAL THESISKarlstad University Studies, 2018:42ISBN 978-91-7063-878-7 (Print) ISBN 978-91-7063-973-9 (pdf)

The topic of this thesis is the performance of computer networks in general, and the internet in particular. While network performance has generally improved with time, over the last several years we have seen examples of performance barriers limiting network performance. In this work we explore such performance barriers and look for solutions.

Our exploration takes us through three areas where performance barriers are found: The bufferbloat phenomenon of excessive queueing latency, the performance anomaly in WiFi networks and related airtime resource sharing problems, and the problem of implementing high-speed programmable packet processing in an operating system. In each of these areas we present solutions that significantly advance the state of the art.

The work in this thesis spans all three aspects of the field of computing, namely mathematics, engineering and science. We perform mathematical analysis of algorithms, engineer solutions to the problems we explore, and perform scientific studies of the network itself. All our solutions are implemented as open source software, including both contributions to the upstream Linux kernel, as well as the Flent test tool, developed to support the measurements performed in the rest of the thesis.

2018:42

The elephant symbo-lises a big download, commonly referred to

as an ”elephant flow”, which blocks the link with its bulk so everything has to wait behind it.

The mice are smaller flows, such as web

pages, that take up little space but should be able to move freely. Through better queue management, the mice are guided around the elephant, so they don’t have to wait; and interactivity is restored.

The penguin is Tux, the Linux mascot, who stands at the bottleneck

and directs traffic onto the right path.

Bufferbloat and BeyondRemoving Performance Barriers in Real-World Networks

Toke Høiland-Jørgensen

Toke Høiland-Jørgensen | B

ufferbloat and Beyond | 2018:42

Bufferbloat and Beyond




DOCTORAL THESIS | Karlstad University Studies | 2018:42

Faculty of Health, Science and Technology

Computer Science


ISSN 1403-8099

ISBN 978-91-7063-973-9 (pdf)

ISBN 978-91-7063-878-7 (Print)


Bufferbloat and BeyondRemoving Performance Barriers in Real-World Networks


Print: Universitetstryckeriet, Karlstad 2018

Distribution:Karlstad University Faculty of Health, Science and TechnologyDepartment of Mathematics and Computer ScienceSE-651 88 Karlstad, Sweden+46 54 700 10 00

© The author

ISSN 1403-8099

urn:nbn:se:kau:diva-69416

Karlstad University Studies | 2018:42

DOCTORAL THESIS


Bufferbloat and Beyond - Removing Performance Barriers in Real-World Networks

WWW.KAU.SE

ISBN 978-91-7063-973-9 (pdf)

ISBN 978-91-7063-878-7 (Print)

iii

Bufferbloat and Beyond: Removing PerformanceBarriers in Real-World NetworksToke Høiland-Jørgensen

Department of Mathematics and Computer ScienceKarlstad University

AbstractThe topic of this thesis is the performance of computer networks. While net-work performance has generally improved with time, over the last severalyears we have seen examples of performance barriers limiting network per-formance. In this work we explore such performance barriers and look forsolutions.

The problem of excess persistent queueing latency, known as bufferbloat,serves as our starting point; we examine its prevalence in the public internet,and evaluate solutions for better queue management, and explore how toimprove on existing solutions to make them easier to deploy.

Since an increasing number of clients access the internet through WiFinetworks, examining WiFi performance is a natural next step. Here wealso look at bufferbloat, as well as the so-called performance anomaly, wherestations with poor signal strengths can severely impact the performance of thewhole network. We present solutions for both of these issues, and additionallydesign a mechanism for assigning policies for distributing airtime betweendevices on a WiFi network. We also analyse the “TCP Small Queues” latencyminimisation technique implemented in the Linux TCP stack and optimise itsperformance over WiFi networks.

Finally, we explore how high-speed network processing can be enabled insoftware, by looking at the eXpress Data Path framework that has been gradu-ally implemented in the Linux kernel as a way to enable high-performanceprogrammable packet processing directly in the operating system’s networkingstack.

A special focus of this work has been to ensure that the results are carriedforward to the implementation stage, which is achieved by releasing imple-mentations as open source software. This includes parts that have been accep-ted into the Linux kernel, as well as a separate open source measurement tool,called Flent, which is used to perform most of the experiments presented inthis thesis, and also used widely in the bufferbloat community.

Keywords: Bufferbloat, AQM, WiFi, XDP, TSQ, Flent, network measure-ment, performance evaluation, fairness, queueing, programmable packet pro-cessing

v

AcknowledgementsThe journey is almost over. It has been challenging, exciting, annoying,incredible and above all interesting. And it would not have been the samewithout the amazing people who have accompanied and supported me alongthe way, all of whom deserve a huge thank you.

First of all, thank you to the many collaborators I have had these pastyears. Foremost among them of course my advisors, Anna Brunström andPer Hurtig, who have always been available with their competent feedback,ideas, suggestions and support. A special thank you also to Dave Täht, withwhom I have collaborated extensively, on both the papers we have co-authored,and a list of other projects too long to reproduce here. Dave was also, alongwith Jim Gettys, the people who initially introduced me to the concept ofbufferbloat, and they helped set me on the path towards the work representedin this thesis, for which I am exceedingly grateful. Thank you to Jesper Brouerfor his collaboration, and for convincing me to continue the work with Linuxand open source post-thesis. Many thanks to Fanny Reinholtz for designingthe awesome dust jacket for the printed version of this thesis. And finally,thank you to my other co-authors, and all the members of the bufferbloat,make-wifi-fast, Linux and OpenWrt communities that I have collaborated withover the last six years.

A journey such as this would not have been possible without the friendswho have shared my joys and sorrows along the way. Thank you to mycolleagues at the university, my old friends in Denmark, and my new(er)friends in Karlstad. Thank you to Lea, Stefan, Stina and Leonardo for climbing,skiing and everything else. To Martin, Daniel, Alex and Miriam for hangingout and having fun. To Timo, Leo and Ricardo for bringing the band (back)together. To the innebandy team in its various incarnations. To the dwarves,and to Sunni, Morten, Sidsel and Torben for keeping in touch through myfive-year Swedish exile.

Last, but certainly not least, thank you so much to my wonderful lovingfamily. Thank you mum and dad for helping me get to this point, for alwaysbeing there for me, and for your love and respect. And thank you Sascha, forcoming with me to Sweden and brightening up my days.

Karlstad, October 2018 Toke Høiland-Jørgensen

vii

List of appended papersI. Toke Høiland-Jørgensen, Bengt Ahlgren, Per Hurtig and Anna Brun-

strom. Measuring Latency Variation in the Internet. ACM CoNEXT’16, December 12–15, 2016, Irvine, CA, USA.

II. Toke Høiland-Jørgensen, Per Hurtig and Anna Brunstrom. The Good,the Bad and the WiFi: Modern AQMs in a Residential Setting. ComputerNetworks, vol 89, pg 90–106, October 2015.

III. Toke Høiland-Jørgensen. Analysing the Latency of Sparse Flows inthe FQ-CoDel Queue Management Algorithm. IEEE CommunicationLetters, October 2018.

IV. Toke Høiland-Jørgensen, Dave Täht and Jonathan Morton. Piece ofCAKE: A Comprehensive Queue Management Solution for HomeGateways. IEEE International Symposium on Local and MetropolitanArea Networks (LANMAN 2018), 25–27 June 2018, Washington, DC.

V. Toke Høiland-Jørgensen, Michal Kazior, Dave Täht, Per Hurtig andAnna Brunstrom. Ending the Anomaly: Achieving Low Latency andAirtime Fairness in WiFi. 2017 USENIX Annual Technical Conference(USENIX ATC 17), July 12–14 2017, Santa Clara, CA.

VI. Toke Høiland-Jørgensen, Per Hurtig and Anna Brunstrom. PoliFi: Air-time Policy Enforcement for WiFi. Under Submission.

VII. Carlo Augusto Grazia, Natale Patriciello, Toke Høiland-Jørgensen, Mar-tin Klapez and Maurizio Casoni. Adapting TCP Small Queues for IEEE802.11 Networks. The 29th Annual IEEE International Symposium onPersonal, Indoor and Mobile Radio Communications (IEEE PIMRC2018), 9-12 September 2018, Bologna, Italy.

VIII. Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann,John Fastabend, Tom Herbert, David Ahern and David Miller. The eX-press Data Path: Fast Programmable Packet Processing in the OperatingSystem Kernel. ACM CoNEXT ’18, December 4–7, 2018, Heraklion,Greece.

IX. Toke Høiland-Jørgensen, Carlo Augusto Grazia, Per Hurtig and AnnaBrunstrom. Flent: The FLExible Network Tester. 11th EAI Interna-tional Conference on Performance Evaluation Methodologies and Tools(VALUETOOLS 2017), December 5–7, 2017, Venice, Italy.

Comments on my participationFor all papers, the ideas have been developed in cooperation with my co-authors. However, I have done the bulk of all writing, implementation,experimental work and evaluation, with the following exceptions:

viii

• In Paper I, the data analysis and most of the text describing the accessnetwork dataset was done by Bengt Ahlgren.

• In Paper IV, most of the initial implementation was done by JonathanMorton. I have been responsible for the upstream submission to theLinux kernel, and have made major revisions in the course of thisprocess. The design of CAKE has been heavily community-driven, withmany members of the bufferbloat community taking part by testing andrequesting features.

• In Paper V, a large part of the MAC layer queue rework implementationwas done by Michal Kazior, and the 30 station tests were performed bySven Eckelmann.

• In Paper VII, I was responsible for the ath9k and ath10k testing, and forsubmitting the final patch for inclusion in the upstream Linux kernel.The rest of the experiments and most of the writing was done by myco-authors.

• In Paper VIII, the design and implementation of XDP has been anongoing process in the Linux development community, with most ofthe work being done by my co-authors. I have been responsible fordesigning and performing the experiments in the paper, together withJesper Danggaard Brouer. I have also been responsible for writing andstructuring the paper.

Other relevant publicationsThe following is a list of the other relevant publications I have co-authoredover the course of my PhD work, and how they relate to the work presentedin this thesis:

• Jesper Dangaard Brouer and Toke Høiland-Jørgensen. “XDP - challengesand future work”. Linux Plumbers Conference 2018 Networking Track,Vancouver, Canada, November 2018. Here we present parts of the workdescribed in Paper VIII.

• Toke Høiland-Jørgensen et al. “The Flow Queue CoDel Packet Sched-uler and Active Queue Management Algorithm”. RFC 8290 (Experi-mental), January 2018. This is the standards document describing theFQ-CoDel algorithm also treated in Paper III.

• Toke Høiland-Jørgensen. “On The Bleeding Edge: Debloating InternetAccess Networks”. Licentiate Thesis, Karlstad University, November2016. This contains parts of the work presented in this thesis; specifically,Papers I, II and earlier versions of Papers V and IX as well as parts of theintroductory summary.

ix

• Toke Høiland-Jørgensen. “Bufferbloat mitigation in the WiFi stack -status and next steps”. NetDev 2.2, Seoul, South Korea, November 2016.This includes early versions of the work presented in Papers V and VI.

• Toke Høiland-Jørgensen. “Flent: The FLExible Network Tester”. The11th Swedish National Computer Networking Workshop (SNCNW),Karlstad, Sweden, May 28–29, 2015. This is an earlier version of Pa-per IX.

xi

Contents

Introductory Summary 1

1 Introduction 3

2 Research objective 4

3 Background 53.1 Bufferbloat and its mitigation . . . . . . . . . . . . . . . . . 53.2 The 802.11 WiFi standards . . . . . . . . . . . . . . . . . . . 63.3 High-performance software packet processing . . . . . . . . . 8

4 Research questions 11

5 Contributions 13

6 Research method 16

7 Summary of appended papers 18

8 Conclusions and future work 22

Paper I:Measuring Latency Variation in the Internet 31

1 Introduction 31

2 Datasets and methodology 322.1 The M-Lab NDT data . . . . . . . . . . . . . . . . . . . . . 332.2 The access aggregation link data . . . . . . . . . . . . . . . . 342.3 Sources of latency variation . . . . . . . . . . . . . . . . . . 36

3 Latency variation over time and geography 363.1 Geographic differences . . . . . . . . . . . . . . . . . . . . . 363.2 Development over time . . . . . . . . . . . . . . . . . . . . . 383.3 Different measures of latency span . . . . . . . . . . . . . . . 38

4 Latency variation in the access network 38

5 Examining queueing latency 405.1 Latency reductions after a drop . . . . . . . . . . . . . . . . 405.2 Delay correlated with load . . . . . . . . . . . . . . . . . . . 425.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

xii

6 Related work 446.1 Large-scale active measurements . . . . . . . . . . . . . . . . 446.2 Targeted active measurements . . . . . . . . . . . . . . . . . 456.3 Passive measurements . . . . . . . . . . . . . . . . . . . . . . 45

7 Conclusions 46

8 Acknowledgements 46

Paper II:The Good, the Bad and the WiFi: Modern AQMs in aResidential Setting 51

1 Introduction 51

2 Related work 53

3 Experimental methodology 55

4 Tested algorithms 564.1 pfifo_fast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.2 ARED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.3 PIE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.4 CoDel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.5 SFQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.6 fq_codel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.7 fq_nocodel . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5 The Good: steady-state behaviour 605.1 The Real-time Response Under Load test . . . . . . . . . . . 605.2 VoIP test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.3 Web test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

6 The Bad: fairness and transient behaviour 696.1 Inter-flow fairness . . . . . . . . . . . . . . . . . . . . . . . . 696.2 Transient behaviour . . . . . . . . . . . . . . . . . . . . . . . 726.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

7 The WiFi: adding a wireless link 737.1 The RRUL test . . . . . . . . . . . . . . . . . . . . . . . . . 757.2 VoIP traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . 767.3 Web results . . . . . . . . . . . . . . . . . . . . . . . . . . . 777.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

8 Conclusions and future work 77

xiii

Paper III:Analysing the Latency of Sparse Flows in the FQ-CoDelQueue Management Algorithm 87

1 Introduction 87

2 Related work 88

3 The sparse flow optimisation 89

4 Analytical framework 894.1 One sparse flow . . . . . . . . . . . . . . . . . . . . . . . . . 894.2 Multiple sparse flows . . . . . . . . . . . . . . . . . . . . . . 914.3 Impact on bulk flows . . . . . . . . . . . . . . . . . . . . . . 924.4 Impact of changing the quantum . . . . . . . . . . . . . . . . 92

5 Real-world examples 92

6 Conclusion 95


Paper IV:Piece of CAKE: A Comprehensive Queue ManagementSolution for Home Gateways 99

1 Introduction 99

2 Background and Related Work 1002.1 Bandwidth Shaping . . . . . . . . . . . . . . . . . . . . . . . 1012.2 Queue Management . . . . . . . . . . . . . . . . . . . . . . 1012.3 DiffServ Handling . . . . . . . . . . . . . . . . . . . . . . . 1022.4 TCP ACK Filtering . . . . . . . . . . . . . . . . . . . . . . . 102

3 The Design of CAKE 1033.1 Bandwidth Shaping . . . . . . . . . . . . . . . . . . . . . . . 1033.2 Flow Isolation and Hashing . . . . . . . . . . . . . . . . . . 1043.3 DiffServ handling . . . . . . . . . . . . . . . . . . . . . . . . 1053.4 ACK filtering . . . . . . . . . . . . . . . . . . . . . . . . . . 106

4 Performance Evaluation 1074.1 Host Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . 1074.2 DiffServ Handling . . . . . . . . . . . . . . . . . . . . . . . 1094.3 ACK Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . 109

5 Conclusions 110

xiv

Acknowledgements 111

Paper V:Ending the Anomaly: Achieving Low Latency and Air-time Fairness in WiFi 117

1 Introduction 117

2 Background 1182.1 Bufferbloat in the context of WiFi . . . . . . . . . . . . . . . 1182.2 Airtime fairness . . . . . . . . . . . . . . . . . . . . . . . . . 120

3 Our solution 1233.1 A bloat-free queueing structure for 802.11 . . . . . . . . . . . 1233.2 Airtime fairness scheduling . . . . . . . . . . . . . . . . . . . 1253.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 128

4 Evaluation 1294.1 Validation of effects . . . . . . . . . . . . . . . . . . . . . . . 1294.2 Effects on real-world application performance . . . . . . . . . 1354.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

5 Related work 139

6 Conclusion 140


Paper VI:PoliFi: Airtime Policy Enforcement for WiFi 147

1 Introduction 147

2 Related work 148

3 The PoliFi Design 1493.1 Userspace Policy Daemon . . . . . . . . . . . . . . . . . . . 1503.2 Weighted Airtime DRR . . . . . . . . . . . . . . . . . . . . 1523.3 Kernel Airtime Scheduler . . . . . . . . . . . . . . . . . . . 154

4 Evaluation 1544.1 Steady state measurements . . . . . . . . . . . . . . . . . . . 1554.2 Dynamic measurements . . . . . . . . . . . . . . . . . . . . 1574.3 DASH Traffic Test . . . . . . . . . . . . . . . . . . . . . . . 159

5 Conclusion 159

xv

Paper VII:Adapting TCP Small Queues for IEEE 802.11 Networks 165

1 Introduction 165

2 TCP Small Queues in a Nutshell 166

3 Controlled TSQ 169

4 Testbed 169

5 Results 173

6 Conclusions 178

Paper VIII:The eXpress Data Path: Fast Programmable Packet Pro-cessing in the Operating System Kernel 183

1 Introduction 183

2 Related work 185

3 The design of XDP 1873.1 The XDP Driver Hook . . . . . . . . . . . . . . . . . . . . . 1903.2 The eBPF Virtual Machine . . . . . . . . . . . . . . . . . . . 1913.3 BPF Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1923.4 The eBPF Verifier . . . . . . . . . . . . . . . . . . . . . . . . 1933.5 Example XDP program . . . . . . . . . . . . . . . . . . . . . 1953.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

4 Performance evaluation 1964.1 Packet Drop Performance . . . . . . . . . . . . . . . . . . . 1984.2 CPU Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . 1994.3 Packet Forwarding Performance . . . . . . . . . . . . . . . . 1994.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

5 Real-world use cases 2025.1 Software Routing . . . . . . . . . . . . . . . . . . . . . . . . 2035.2 Inline DoS Mitigation . . . . . . . . . . . . . . . . . . . . . 2035.3 Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . 205

6 Limitations and future work 2066.1 Limitations on eBPF programs . . . . . . . . . . . . . . . . . 2066.2 User Experience and Debugging . . . . . . . . . . . . . . . . 2076.3 Driver Support . . . . . . . . . . . . . . . . . . . . . . . . . 2076.4 Performance Improvements . . . . . . . . . . . . . . . . . . 208

xvi

6.5 QoS and Rate Transitions . . . . . . . . . . . . . . . . . . . 2086.6 Accelerating Transport Protocols . . . . . . . . . . . . . . . 2086.7 Zero-copy to userspace . . . . . . . . . . . . . . . . . . . . . 2086.8 XDP as a building block . . . . . . . . . . . . . . . . . . . . 209

7 Conclusions 209

Paper IX:Flent: The FLExible Network Tester 217

1 Introduction 217

2 Experimental challenges 2182.1 Reproducing experiments . . . . . . . . . . . . . . . . . . . 2182.2 Testbed configuration and test automation . . . . . . . . . . 2192.3 Storing and analysing measurement data . . . . . . . . . . . . 219

3 How Flent helps 2193.1 Reproducing experiments . . . . . . . . . . . . . . . . . . . 2203.2 Configuration and automation . . . . . . . . . . . . . . . . . 2203.3 Storing and analysing measurement data . . . . . . . . . . . . 221

4 Showcasing Flent: A look at the BBR congestion control 2214.1 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . 2224.2 Testbed results . . . . . . . . . . . . . . . . . . . . . . . . . 2234.3 Public internet results . . . . . . . . . . . . . . . . . . . . . . 226

5 Related work 229

6 Conclusions and Future Work 229

Introductory Summary

“To Infinity. . . And Beyond!”

Buzz Lightyear, Toy Story

3

1 IntroductionInternet usage is steadily increasing and our world is becoming increasinglyconnected. At the same time, new applications place ever higher performancerequirements on the network, which must support both high bandwidth andlow latency. Latency in particular has become more important as applicationsturn more interactive, and emerging applications such as augmented reality andubiquitous video conversations are only going to make this more important.

While network performance has increased significantly, and continues todo so, over the last several years we have seen examples of performance barrierslimiting the network performance. One example of this is the bufferbloatphenomenon, which is a term coined to describe the effect that occurs whena network bottleneck is congested and large buffers fill up and do not drain,thus inducing a persistent queueing delay that can be much larger than thepath round-trip time [1, 2]. Excessive queueing delay is by no means a newphenomenon, but even so, bufferbloat has been found to be widespread indeployed systems, thus forming a practical performance barrier.

In this thesis we will explore such performance barriers, and look for waysto remove or mitigate them. Bufferbloat serves as our starting point. An onlinecommunity has formed around the need to develop technical solutions tomitigate bufferbloat,1 and the work in this thesis should be seen in the contextof this community effort, which I have taken part in over the last several years.

The bufferbloat effort has been focused on last-mile access networks, wherebufferbloat has been both extremely prevalent, but also fixable due to cheaphome routers that can have their firmware replaced. Replacing the router firm-ware makes it possible to apply state of the art queue management algorithmson the bottleneck link connecting the home to the internet. The efficacy ofsuch queue management solutions is one of the topics we examine in this work.

As we improve the access link itself, and as bandwidths increase, the newbottleneck link, and source of performance barriers, becomes the WiFi link.Thus, exploring the performance of WiFi networks is a natural extension ofthe bufferbloat work. It turns out that WiFi network performance suffers notonly from bufferbloat, but also from the so-called performance anomaly, wherestations with poor signal strengths can severely impact the performance of thewhole network. We explore a range of performance issues with WiFi networksand present solutions for them.

Both the solutions for bufferbloat and the improvements to WiFi networksare characterised by being enabled by malleable open source software imple-mentations which can be inspected and improved upon. This has been possiblein the relatively low-speed home network devices where CPUs can keep upwith the packet rate. However, for truly high-speed networking (going intotens and hundreds of gigabits per second), programmable packet processinghas been out of reach until quite recently. Now, not only is special-purposenetworking hardware gaining programmability, general-purpose computersare also beginning to reach a level of performance where they can realistically

1See https://www.bufferbloat.net.

4

process packets at the 100-Gbps level. The software architecture of commonoperating systems forms a bottleneck when trying to achieve such speeds,which is the last barrier that we explore solutions to in this work. Specifically,we look at the eXpress Data Path framework that has been gradually imple-mented in the Linux kernel over the last several releases, which we describeand evaluate in detail.

This thesis explores the performance barriers mentioned above, in the formof the nine appended papers. Before we get this far, though, we will first set thestage and present some context. This is the role of this introductory summary,the rest of which is structured as follows: Section 2 (below) outlines the mainresearch objective we are pursuing in this thesis. Section 3 contains a shortbackground primer on the subject areas we touch upon, and summarises relatedwork. Section 4 outlines our research questions and Section 5 outlines themain contributions of this thesis. Section 6 relates the work to the traditionsof the field of computer science and Section 7 presents a short summary of theappended papers. Finally, Section 8 concludes and outlines future work.

2 Research objectiveThe overarching research objective of this work is to improve network per-formance by exploring specific performance barriers in modern real-worldnetworks, and developing ways of removing them. We use the bufferbloatphenomenon as a starting point for this exploration and continue into severalrelated areas. In particular, we are interested in the following:

1. Improving the understanding of the specific manifestations of bufferbloatand of its prevalence in the internet.

2. Evaluating the efficacy of existing solutions for bufferbloat, and identify-ing ways to optimise them for deployment in edge networks.

3. Addressing bufferbloat in WiFi networks and exploring other relatedperformance problems and their solutions.

4. Exploring solutions for removing the performance barrier posed by theoperating system in software packet processing systems.

5

3 BackgroundThis section provides some background on the subject areas of this thesis. Sec-tion 3.1 outlines the previous work on bufferbloat assessment and mitigation;Section 3.2 gives an overview of the operation of 802.11 WiFi networks, andoutlines previous work on network performance in the WiFi space; and Sec-tion 3.3 outlines previous work on high-performance packet processing insoftware.

3.1 Bufferbloat and its mitigationWhile bufferbloat is straightforward to demonstrate in a lab setting, there hasbeen some argument over how widespread it is in the real world. Due toits distributed nature, no one has a global view of the internet, so one islimited to analysing data from a subset of the network. This can be doneeither by generating measurement traffic to look for specific behaviour (activemeasurements), or by capturing traffic at a specific point in the network andtrying to infer path characteristics from that (passive measurements).

Several studies based on both active and passive measurements have beenperformed, with various focus areas. Examples of large scale studies thatinclude latency measurements are [3], which uses 130,000 tests from theNetalyzr test suite to measure queueing latency; [4], which uses data fromthe BISMark and SamKnows measurement platforms to measure baseline andunder-load latency; and [5], which also uses BISMark data (as well as othersources) to measure broadband performance in South Africa. Smaller scaleactive measurements are described in [6], where active probing of residentialhosts is used to measure network connections; in [7], which is specificallytargeted at assessing queueing latency; and in [8, 9], both of which use clientsunder the experimenters’ control to measure bufferbloat in cellular networks.Fewer passive measurement studies specifically look at latency, but a fewexamples can be found, e.g. [10, 11], both of which employ captures taken at anetwork edge for their analysis.

Several of the studies mentioned above find examples of latency increasesover the baseline, but few of them specifically discuss bufferbloat as a possiblecause. In Paper I we seek to contribute to the discussion, by exploring specific-ally how latency varies over time in the internet, and how much of this can beattributed to bufferbloat, with the data available.

Turning to bufferbloat mitigation procedures, the problem of unmanagedFIFO queues and their impact on latency has been known for a long time,and the introduction of Active Queue Management (AQM) has been knownto be an effective way of reducing delays, if deployed correctly. The mostwell-known classical AQM algorithm, Random Early Detection (RED) [12],is more than two decades old, and its successor, Adaptive RED (ARED) [13],is over fifteen. Package scheduling algorithms, which can also help mitigatequeueing latency, have even older well-known examples, such as the Stochastic

6

Fairness Queueing (SFQ) algorithm [14], and other fairness queueing al-gorithms that schedule individual flows to ensure fairness between them.

Many other algorithms have been proposed over the years [15], howeverfew have seen widespread deployment in the internet. This is mostly due tothe difficulty of tuning the algorithms, and the adverse effect they can have onperformance if configured incorrectly [16]. And so, for many years no easilydeployable queue management techniques were readily available.

As the bufferbloat issue started receiving increased attention, however,we started seeing a resurgence of interest in developing algorithms that caneffectively control queues without requiring extensive tuning. And so, anumber of new algorithms appeared: Controlled Delay (CoDel) [17] andProportional Integral controller Enhanced (PIE) [18] are both new AQMalgorithms, while FQ-CoDel [19] combines the CoDel AQM with a fairnessqueueing scheme and an optimisation for improving the latency of “sparse”flows (i.e., flows that don’t use a lot of bandwidth). These algorithms are thefocus of the evaluation presented in Paper II.

To supplement the experimental evaluation of different queueing algorithms,an analytical approach can be useful to explore specific performance char-acteristics of an algorithm. There have been examples of this, such as thatperformed for the Shortest Queue First (SQF) algorithm by its authors in [20],or for the Quick Fair Queueing (QFQ) algorithm in [21]. Another exampleof such an analytical approach is the analysis of the number of active flowsin a fairness queueing system that is provided in [22]. However, no such ana-lysis has been performed for the new queue management algorithms discussedabove. Remedying this is the subject of Paper III.

While effective algorithms for controlling bufferbloat are certainly ne-cessary to solve the problem, they are not themselves sufficient. A queuemanagement algorithm is only effective if it is in control of the bottleneckqueue. This can be achieved by eliminating queues in lower layers, such as hasbeen achieved in Linux for Ethernet drivers [23]. If this is not possible, thebottleneck queue can be moved through the use of a bandwidth shaper. Themost common way to do this is by a token bucket-based shaper (e.g., [24]), orby a rate-based shaper (e.g., [25]). Integrating bandwidth shaping with a queuemanagement algorithm complicates deployment, which can be a barrier toadoption. Exploring how to improve on this situation is the topic of Paper IV.

3.2 The 802.11 WiFi standardsThe IEEE 802.11 standards [26] are a series of documents that describe thephysical and MAC layers of the wireless network protocol that is commonlyknown as WiFi. The protocol acts like a virtual Ethernet link, but becausethe radio waves are propagating through a shared medium, there are severalimportant differences in how the protocol operates. In this section, we sum-marise the main operational principles of the 802.11 WiFi MAC. We omit thephysical layer and focus on the 802.11n revision of the specification, whichis what we have used in Paper V and Paper VI. The newer 802.11ac revision

7

mainly differs at the physical layer, and in some constants (such as the maxaggregation size) at the MAC layer. The overall operational principles arealmost identical between 802.11n and 802.11ac.

When a node in a WiFi network (or “station”, as non-access point nodesare typically called) has data to send, it will first attempt to determine whetheror not the channel is free, by listening for other transmissions. If it determinesthat no other devices are currently transmitting, it will wait a random amountof time (in the order of several microseconds), and then begin transmitting itsdata frame. After transmission has completed, it will wait for the receivingdevice to acknowledge the transmission. If no acknowledgement is heard,the sender will assume that a collision occurred (i.e., that another sendertransmitted at the same time and the transmissions interfered with each otherand were lost), and so will initiate another, longer, random wait period, beforethe procedure is repeated. This mechanism of arbitration for the opportunityto send is known as the 802.11 Distributed Coordination Function (DCF).The standard additionally specifies several possible transmission rates (withcorresponding physical encodings of the data), and leaves it up to the devices topick the best rate for a given transmission opportunity, based on the observedsignal-to-noise ratio at the time of transmission.

The main advantage of the DCF is that it does not require a centralcoordinator in the network. However, this comes at a cost: The random back-off periods can lead to inefficiency in the network (i.e., periods of time in whichthe channel is entirely idle). To limit the overhead of the DCF, aggregation hasbeen introduced into the MAC protocol, allowing several data packets to besent at once when a device gets a chance to transmit. The standard specifies amaximum size and a maximum transmission time for aggregates, but not howdevices should build aggregates. From a queueing latency perspective, however,it is clear that some amount of queueing is necessary in order to have enoughpackets ready to build a reasonably sized aggregate when given a chance totransmit. The introduction of aggregation also makes the transmission bursty(there will be periods of no data being transmitted, followed by short burstsof several packets when an aggregate is formed). This introduces its ownchallenges for a low-latency queueing structure, as we will see in Paper V.

Another fundamental property of the DCF is that it gives each device onthe network the same probability of winning the contention for the medium,and thus being able to transmit. However, this is independent of the time eachdevice spends transmitting when it gets a chance. Since the transmission ratecan vary over a large interval in the same network, one station transmitting ata low rate can take up a large fraction of the total transmission time, and soeffectively limit the throughput of all stations in the network, which hurts theaggregate efficiency of the network as a whole. This phenomenon is known asthe 802.11 performance anomaly [27].

The final property of WiFi networks worth mentioning here is the Qual-ity of Service (QoS) standards defined by the 802.11e document in the 802.11standards series. This specifies that a station can transmit data in different prior-ity tiers depending on their QoS markings. There are four QoS levels, labelled

8

(in decreasing priority order) as Voice, Video, Best Effort and Background.The standard specifies different constants for the DCF for each QoS level, giv-ing the higher priority levels a higher probability of winning the contentionfor the transmission medium. The standard also specifies different aggregationsize limits on the different QoS levels, which in theory prevents abuse by trad-ing higher contention win probability for lower maximum throughput (due tothe lower aggregation). From a latency perspective, the QoS mechanism canin theory be a big win, but unfortunately there are no agreed-upon standardsfor how to assign QoS levels to different traffic flows, and implementationstend to be bug-ridden, significantly lowering the usefulness of this feature ofthe standard.

While there is nothing fundamental in WiFi stating that queues cannot bemanaged to avoid excess queueing latency, there is little in the literature thatdeals with this aspect of the problem. A few examples of work that do treatqueueing latency in WiFi networks are [28], which features a WiFi componentin a larger evaluation of bufferbloat mitigation techniques; [29], which looks atbuffer sizing issues in WiFi networks; and [30], which touches upon congestionin the WiFi hop and uses different queueing schemes to address it, but in thecontext of a centralised solution that also seeks to control fairness in the wholenetwork. However, none of these studies provides a solution at the WiFi hopitself.

The performance anomaly, on the other hand, has been studied extens-ively, and several solutions have been proposed, falling roughly into three cat-egories: Those that modify the back-off parameters of WiFi nodes to achievefairness [31–34], those that change the transmission size [35–37], and thosethat employ a scheduler at the access point [38, 39]. However, most of thesolutions have never made it to actual implementations, whether because theimplementation has never left the simulation stage, or because the approachhas other tradeoffs that make them impractical to implement. This meansthat the performance anomaly is still very much present today, as we show inPaper V.

The decentralised nature of WiFi also means that it lacks protocol supportfor enforcing policies on resource usage (notably airtime usage), which wouldotherwise be an obvious way to increase performance for some applications,especially in contested scenarios. There are, however, some attempts at apply-ing policies to WiFi networks [40–42] but none of them are at a state wherethey can realistically be deployed on today’s networks. Some enterprise accesspoints do offer features related to airtime fairness and policy configuration [43],but being a proprietary system, not much information is available for studyof the mechanisms involved. How to design a workable policy enforcementmechanism for WiFi is discussed in Paper VI.

3.3 High-performance software packet processingPacket processing in the network has historically been limited to the mostbasic operations, simply because high traffic volume means very tight bounds

9

on the time spent processing each packet. However, as hardware becomes morecapable, and costs drop, it becomes increasingly feasible to perform advancedprocessing as part of the data path. This has lead to an increasing demand forprogrammable capabilities in networking hardware. One example of this isthe advent of of programmable hardware devices, such as the NetFPGA [44],which enables custom packet processing on specialised hardware, by exposingan API that makes it possible to run arbitrary packet processing tasks on theFPGA-based dedicated hardware. Another example is the P4 language [45],which seeks to extend this programmability to a wider variety of packetprocessing hardware.

However, even though specialised hardware has gained data plane program-mability, it still cannot match the flexibility and low cost of common off-the-shelf (COTS) hardware. Several packet processing systems running on COTShardware have appeared over the last several years. Examples of this includeapplications performing single functions, such as switching [46], routing [47],named-based forwarding [48], classification [49], caching [50] or traffic gen-eration [51]. They also include more general solutions which are highlycustomisable and can operate on packets from a variety of sources [52–57].

One challenge with using COTS hardware, is that in order to achievehigh performance it is necessary to remove any bottlenecks between thenetworking interface card (NIC) and the program performing the packetprocessing. Since one of the main sources of performance bottlenecks is theinterface between the operating system kernel and the userspace applicationsrunning on top of it (because of the high overhead of a system call andcomplexity of the underlying feature-rich generic networking stack in theoperating system), low-level packet processing frameworks have to managethis overhead in one way or another. The approaches taken to this fall intothree broad categories: (a) implementing parts of the application as a moduleinside the kernel itself, with examples such as the Open vSwitch [57] virtualswitch and the Click [55] virtual router framework; (b) providing an interfacefor userspace to access packet data with lower overhead than with traditionalsockets, such as in PF_RING [58], the Packet I/O engine that is part ofPacketShader [47] or the Netmap [59] framework, as well as special-purposeoperating systems such as Arrakis [60] and ClickOS [61]; or (c) bypassing thekernel entirely and handing over control of the networking device directlyto userspace, where examples include the PF_RING ZC module [62], thehardware-specific Solarflare OpenOnload [63] or, most prominently, theDataPlane Development Kit (DPDK) [64].

Each of the three categories of solutions have different drawbacks andtradeoffs, and while all of them offer better performance than the regular oper-ating system network stack, the relative performance of each solution varieswidely. The highest raw performance numbers are seen with kernel bypasssolutions, of which DPDK represents the state of the art. DPDK is commonlyused with higher-level processing frameworks such as Cisco’s VPP [54], whichoffers a programming model that helps maximise performance for many ap-plications.

10

Because kernel bypass solutions need direct access to the hardware toachieve the high performance numbers, it is difficult to integrate with systemswhere the kernel plays a dominant role in resource abstraction and isolation,such as container-based and virtualised workloads. A different solution, thatallows more fine-grained integration into the operating system, has been de-veloped by the Linux kernel community over the last several years. This solu-tion is called XDP, but has been lacking a high-level architectural description.Remedying this lack is the topic of Paper VIII.

11

4 Research questionsIn order to achieve the overall research objectives outlined in Section 2, weformulate the following research questions that we seek to answer in this work:

1. How bloated is the internet?Before one can address a problem, one must first be aware that it exists.In particular, before spending the considerable resources required todeploy upgrades to the edge of the internet (with its billions of connecteddevices), one must be convinced of the gain. While the bufferbloatproblem can be easily demonstrated in a lab, there has been some debateover how prevalent it is in the real world (as discussed in the previoussection). Since no one has a global view of the internet, finding the rightanswer is no trivial task. However, we can make a dent in the problemby analysing the data we do have available, and by looking at smallerscale representative slices of the network. In this work we set out to dojust that, and thereby further the understanding of the problem.

2. Can we solve bufferbloat by smarter queue management?Knowing that there is a problem to be solved, one naturally startslooking for solutions. In the previous years, several possible solutionsto the bufferbloat issue have been proposed, often in the form of queuemanagement and packet scheduling algorithms that seek to ensure lowlatency even when a link is heavily loaded.However, such algorithms are often evaluated only by their inventors, inscenarios that reflect the design goals of the algorithms. Thus, comparat-ive studies of several algorithms are needed to fully assess their relativeefficacy. We set out to provide such a comparison, thereby answering thequestion of how well the bufferbloat issue can be resolved today. Thisexperimental comparison clearly shows that the FQ-CoDel algorithmcan offer tremendous performance benefits, especially for low-rate flowsthat do not build a large queue themselves. We seek to examine this inmore detail by supplementing our experimental results with an analysisof the conditions under which flows can benefit from the low latencyguarantees of the algorithm, and what exactly those guarantees are.Finally, we seek to improve the deployment aspects of new bufferbloatsolutions. An important aspect of this is ease configuration, especiallyin devices at the edge of the internet, where the problem tends tobe worse. Often, several different parts, such as queue managementand bandwidth shaping, need to be integrated into a solution before itbecomes deployable. To ease such deployment, we seek to develop asolution that integrates state of the art algorithms into an easy to deploysolution for bufferbloat at the network edge.

12

3. How can we improve the performance of WiFi?As explained in the previous section, two major performance issuesaffect modern WiFi networks: The bufferbloat issue, and the perform-ance anomaly. We set out to find workable solutions for both theseissues, using the Linux kernel as our example platform. In addition, weset out to explore how these solutions can be used to implement an-other performance-enhancing feature: Policy-based airtime assignmentto devices on a WiFi network. Finally, we set out to improve the wayWiFi networks interact with other bufferbloat mitigation solutions im-plemented in Linux, focusing on the TCP Small Queues optimisation inthe TCP stack.

4. How can we integrate high-performance programmable packet processinginto the operating system?As outlined in the previous section, while several solutions for high-performance packet processing exist, to avoid the performance penaltyimposed by interacting with the operating system kernel for everypacket, they all either heavily modify the operating system, or bypass itentirely. However, this limits flexibility in packet processing and makesit harder to integrate it into a mixed environment where applicationsrelying on the regular operating system environment is also present. Weseek to investigate how the performance barrier of the operating systemcan be lifted, allowing programmable packet processing to be integratedinto the operating system in a cooperative way.

13

5 ContributionsThe main contributions of this work are the following:

1. A better understanding of the magnitude of bufferbloat in the internetWe contribute to a better understanding of bufferbloat and its magnitudein the internet. We do this by combining a large scale active meas-urement dataset from the public internet with a smaller scale passivemeasurement dataset from an internet service provider access network.We show that significant latency variation occurs, with large differencesbetween regions, but negligible development over time, despite consist-ent increases in bandwidth over the same time span. In addition, weuse the observed variation in latency as a way to quantify excess latency,and combine it with a novel approach to identify bufferbloat from TCPRTT samples, which we employ on a subset of the data to show that atleast some of the latency variation can be attributed to bloat.

2. An evaluation of existing solutions for bufferbloat mitigationWe contribute an extensive experimental evaluation of several mod-ern queue management algorithms in a Linux testbed setup, designedto model a home network setup. This evaluation provides importantreal-world performance data on modern queue management algorithms.Our evaluation covers a range of simulated access network connectionspeeds, as well as a WiFi bottleneck scenario. We show that modernqueue management algorithms can significantly reduce bufferbloat atthe access link; however, the tested AQM algorithms have some issueswith transient delay spikes as flows start up, and they exacerbate TCPflow unfairness, while the tested fairness queueing algorithms provideconsistently low latency and almost perfect fairness. Our web and VoIPapplication tests show that the improvements in latency afforded bythe algorithms translate to real-world application performance benefits.Finally, we show that the algorithms are less effective on a WiFi bottle-neck, due to buffering in lower layers of the operating system networkstack.

3. An analysis of the sparse flow behaviour of FQ-CoDelOur evaluation of queue management algorithms clearly shows that theFQ-CoDel algorithm achieves the lowest latency for many workloads,because of its optimisation for prioritisation of flows that do not build aqueue, so-called “sparse flows”. We contribute to the understanding ofthis mechanism by performing an analysis of which conditions a packetflow must fulfil to be considered “sparse” by the FQ-CoDel queuemanagement algorithm, and thus given priority and low latency. Wecombine analytical expressions derived from the algorithm descriptionwith a numerical simulation, and formulate conditions that a flow needsto fulfil to keep the “sparse” status, as well as expressions for the expectedlatency of such a flow.

14

4. An integrated queueing solution targeting home network gatewaysWe present an integrated queue management solution for home networkgateways. This solution, called CAKE, builds on the basic fairnessqueueing design of FQ-CoDel, but adds several features are useful in ahome gateway. These features include traffic shaping, host-based fairnessqueueing, DiffServ handling and TCP ACK filtering. Our evaluationshows that these features provide compelling benefits for their respectiveuse cases. In addition, the integration of these features into a singlequeueing discipline significantly eases deployment and configuration,which is an important aspect of defeating bufferbloat in real-worldnetworks.

5. A solution for improving the performance of WiFiWe design and implement workable measures to resolve both the buf-ferbloat issue and the performance anomaly in WiFi, using Linux as animplementation platform. Our solution reduces latency in the stackby an order of magnitude by improving queue management, and itincreases the efficiency of the network with up to a factor of five byenforcing airtime fairness between devices. We have worked with theLinux community to get our solution incorporated into the mainstreamLinux kernel, to make sure it is not just a theoretical solution, but ratherone that will find its way into deployed devices. Our solution builds onstate of the art queue management algorithms, and showcases new waysof adapting these techniques to the WiFi domain.

6. An airtime policy enforcement system for WiFiBuilding on our airtime fairness system for solving the performanceanomaly, we design and implement a solution for enforcing configurableairtime policy assignment between stations in an infrastructure WiFinetwork. This enables several interesting use cases, from a limited guestnetwork use case, to a full network slicing solution as is being discussedfor future 5G network environments. We implement the system inLinux and show how it can be used to improve performance of real-world applications.

7. An improvement to the TCP Small Queues mechanism for WiFi networksThe “TCP Small Queues” feature included in the Linux kernel limitsthe size of the queue the TCP stack keeps for each flow, in order toreduce queueing latency. We show that this has an adverse impact on theperformance of TCP over WiFi, because there is not enough data queuedto build aggregates at the WiFi MAC level. We analyse the magnitude ofthis degradation and implement a solution which selectively tunes theTCP Small Queues system when flow egress is over a WiFi link.

15

8. A description and evaluation of the eXpress Data Path programmable packetprocessing system in LinuxThe Linux kernel networking community have designed and imple-mented the eXpress Data Path system in the Linux kernel over the lastseveral years. This system allows high-performance packet processingto be integrated into the kernel in a cooperative way, increasing flexibil-ity compared to other systems, while retaining high performance. Wecontribute a description of the architecture of the XDP system, alongwith a thorough performance evaluation featuring both synthetic bench-marks and real-world application examples. We make the case that XDPpresents a compelling tradeoff in terms of features, flexibility and per-formance compared to other related systems.

9. A tool for automation and re-usability of experimentsIn the course of answering the main research questions and developingthe contributions outlined above, we have developed and described atesting tool that helps facilitate future experimental work, by makingtests repeatable and results sharable. The tool is called The FLExibleNetwork Tester (Flent), and contains several new features for easy dataexploration and automation of experimental work. Flent has shownreal-world utility and is widely used in the bufferbloat community.

16

6 Research methodThis thesis is written within the field of computer science, more specificallyin the area of computer networking. Computer science, in turn, is a part ofthe broader field of computing. This field is quite diverse, but can be viewed asbeing comprised of three different aspects, each drawing on different traditions,namely mathematics, engineering and science [65]. In this section, we situatethe work presented in this thesis in the broader field of computing and thesethree aspects that comprise the field.

The mathematical origins of computing are tied to the theoretical founda-tions of the field, most notably the theory of computation and the question ofwhat can be computed. In addition, mathematical analysis and proof theoryplays an important role in many branches of computing. The engineeringaspect has also played a significant role throughout the history of computing,simply because the practical realisation of computation (i.e., the building ofactual computers) has been an integral part of the development of the field.Today, the engineering influences is most clearly seen in the term “softwareengineering”. Finally, “computer science” is commonly used when naminguniversity departments (as is the case at Karlstad University), but determiningwhat exactly it is a science of has been the subject of quite some discussion.

As mentioned above, the research presented in this thesis concerns com-puter networks and their function. It wasn’t long after the first computerswere built that it became evident that it was useful to have different com-puters communicate with each other. This soon led to the development of theArpanet, which in turn has developed into the modern internet [66]. Today,the internet has become such an integral part of society that many functionscannot exist without it. Like the rest of the field of computing, networkingrests on the three legs of mathematical theory (e.g., queueing theory, distrib-uted algorithms, etc.), engineering (the internet as a whole is by far the largestand most complicated machine built by humankind) and science (where thenetwork itself can become the subject of study).

The research presented in this thesis contains elements of all three aspectscomprising the broader field. Mathematical analysis of an algorithm is theprimary focus of Paper III, and is also incorporated into Paper V, where itserves to inform the design and implementation of the presented solution.

The engineering aspect is present in most of the work presented here.Indeed, an important motivation of the work has been to ensure the ideas aredeveloped all the way to the point where they can be implemented in realsystems, and thus improve today’s networks. The primary avenue for thishas been open source software, most notably the Linux kernel. The solutionsdeveloped in Papers IV, V, VI, VII and VIII are all included in the upstreamLinux kernel, or are in the process of being incorporated at the time of writing.And the Flent tool presented in Paper IX is released as its own open sourceproject, and has been used in the bufferbloat community for years.

Finally, the aspect of scientific study is included where the functioningof the network itself becomes the object of study. In networking, this is

17

usually achieved in one of three ways: By simulation (where a computerprogram will simulate an entire network and the packets flowing throughit), emulation (where the network is simulated, but interacts with, e.g., realoperating system network stacks) and experiments (where real networkingequipment and software is used in the experiments). All too often, newconcepts in networking are never developed past the simulation stage, whichmeans they languish unused and do not make it to the stage where they can bedeployed on the real internet. In addition, simulation is built on idealisationswhich get ever further from reality as optimisations are added to networkingdevices, often leading to very different behaviour than those assumed by thesimulation tools. To counteract this, the focus in this thesis is very deliberatelyon the latter two kinds of study. Papers I and II are examples of this, whilemany of the others include experimental studies as part of the evaluation.

18

7 Summary of appended papers

Paper I – Measuring Latency Variation in the Internet

In this paper we examine two complementary datasets to assess to what extentbufferbloat is a real problem in the internet. We do this by analysing latencyvariation in a large-scale active measurement dataset from the MeasurementLab Network Diagnostic Tool, combined with a passive measurement data setfrom an access link.

The former dataset allows us to look at large scale trends, but because it con-sists of data from active measurements performed over the public internet,we can only use it to infer the potential for bufferbloat, not the frequencywith which it occurs. The other dataset is much smaller in scale, but is basedon passive measurements and comes from a network that has known pathcharacteristics. This means that we can make conclusions about what the datashows with higher certainty. The combination of these two datasets allows usto say something meaningful of the latency characteristics of the internet as awhole.

We find that significant latency variation is present in both datasets. Addition-ally, a more detailed analysis of a subset of the data shows that at least some ofit can be attributed to bufferbloat.

Paper II – The Good, the Bad and the WiFi

In this paper we evaluate a selection of bottleneck queue management schemesin a test-bed representative of residential internet connections of both sym-metrical and asymmetrical bandwidths as well as WiFi. Latency under loadand the performance of VoIP and web traffic patterns are evaluated understeady state conditions. Furthermore, the impact of the algorithms on fairnessbetween TCP flows with different RTTs, and also the transient behaviour ofthe algorithms at flow startup is examined.

We show that the tested AQM algorithms can significantly improve the steadystate performance, but that they exacerbate TCP flow unfairness and severelystruggle to contain queueing delay in transient conditions, such as when flowsstart up. The tested fairness queueing algorithms, on the other hand, almostcompletely eliminate these problems and give consistently low latency andhigh throughput in the tested scenarios.

Finally, we show that all the tested algorithms perform worse on a WiFi bot-tleneck because they are limited by significant queueing in lower layers of thestack, and thus outside the control of the algorithms.

19

Paper III – Analysing the Latency of Sparse Flows in the FQ-CoDelQueue Management Algorithm

In this paper we analyse an aspect of the FQ-CoDel queue managementalgorithm that has thus far not been well-explored: The conditions underwhich a particular flow is considered “sparse”, and thus receives preferentialtreatment by FQ-CoDel. We formulate a set of constraints that a sparse flowmust satisfy, and also formulate a set of expressions for the expected queueinglatency of sparse flows.

To verify the analytical expressions, we use a numerical example to show howmany of a given type of sparse flows (Voice over IP traffic) a given link cansupport, and verify the analytical predictions in a numerical simulation. Weshow that the number of sparse flows that a given bottleneck can service withlow latency is only dependent on the number of backlogged bulk flows at thebottleneck. Furthermore, we show that as long as the maximum number ofsparse flows is not exceeded, all sparse flows can expect a very low queueinglatency through the bottleneck.

Paper IV – Piece of CAKE

In this paper we present the design and implementation of the CAKE queuemanagement algorithm, which is a comprehensive network queue managementsystem designed specifically for home internet gateways. CAKE is built uponthe FQ-CoDel hybrid AQM and flow queueing algorithm, but adds severalfeatures targeted specifically at home gateways while being easy to configureand deploy. These features include: bandwidth shaping with overhead com-pensation for various link layers; reasonable DiffServ handling; improved flowhashing with both per-flow and per-host queueing fairness; and filtering ofTCP ACKs.

Our evaluation shows that each of these features offer compelling advantagesfor their respective use cases, giving CAKE the potential to significantly im-prove the performance of last-mile internet connections. CAKE has beenaccepted into the upstream Linux kernel and is included from Linux v4.19,released in October 2018.

Paper V – Ending the Anomaly

In this paper we present workable solutions to both bufferbloat at the WiFilink and the 802.11 performance anomaly. We implement a queueing schemein Linux that is based on FQ-CoDel and tightly integrated with the MAClayer to solve the bufferbloat issue, and a scheduler-based solution to achievingairtime fairness.

We formulate an analytical model for achievable 802.11n throughput with andwithout airtime fairness and use that to evaluate our solution in combinationwith a series of testbed experiments. We achieve an order of magnitude reduc-tion in latency under load, large improvements in multi-station throughput,

20

and nearly perfect airtime fairness for both UDP and TCP traffic. Furtherexperiments with application traffic confirm that our modifications provide asignificant performance gain for real-world traffic.

Paper VI – PoliFi

In this paper we present the design and implementation of PoliFi, an airtimepolicy solution for WiFi. PoliFi builds on the previous work presented inPaper V, but generalises the airtime fairness solution and adds a mechanismfor specifying policies for how airtime is assigned to different stations. Inaddition, it includes a policy daemon that enforces the user-configured policiesin real time, and is able to implement a variety of useful policies. Theseinclude prioritisation of specific devices; balancing groups of devices for sharingbetween different logical networks or network slices; and limiting groups ofdevices to implement guest networks or other low-priority services.

Our evaluation shows that that PoliFi successfully enforces the desired policies;and we show how these can be used to improve the performance of a real-world DASH video streaming application.

Paper VII – Adapting TCP Small Queues for IEEE 802.11 Networks

In this paper we examine the interactions between WiFi networks and theTCP Small Queues (TSQ) mechanism that is a part of the Linux TCP stackand is designed to keep queues small for TCP sockets originating on the localmachine. We show that the default settings for TSQ prevent enough data frombeing buffered to properly utilise the packet aggregation mechanism in WiFilinks. We evaluate a range of tuning parameters for 802.11n and 802.11ac andshow that it is possible to double the throughput for 802.11n and increase itby an order of magnitude for 802.11ac, with a negligible increase in queueinglatency.

Paper VIII – The eXpress Data Path

In this paper we describe the eXpress Data Path (XDP) programmable packetprocessing framework that has been introduced into the Linux networkingstack over that last several years. XDP represents a novel approach to program-mable packet processing, where the processing is integrated into the kerneldevice drivers in a cooperative way, by way of an execution environment thatexecutes custom byte code which the kernel statically analyses for safety, andtranslates into native instructions.

This approach allows user programs to selectively process some packets, whileletting the operating system networking stack handle others. In addition,packet processing programs can make use of kernel features to perform certainoperations, and access state from both the networking stack, userspace pro-grams and other parts of the kernel. Together, these features offer compelling

21

advantages by enabling programmable packet processing without the all-or-nothing approach imposed by user-space networking applications that need totake over the networking device entirely.

Our evaluation shows that XDP achieves single-core packet processing per-formance as high as 24 million packets per second. We also illustrate theflexibility of the programming model through three example use cases: layer-3routing, inline DDoS protection and layer-4 load balancing.

Paper IX – Flent

In this paper we present a tool designed to make experimental evaluationsof networks more reliable and easier to perform. This tool, called Flent,works by composing well-known benchmarking tools to, for example, runtests consisting of several bulk data flows combined with simultaneous latencymeasurements. Tests are specified in source code, and several common tests areincluded with the tool. In addition, Flent contains features to automate testruns, and to interactively plot and explore data collected from experiments.

22

8 Conclusions and future workIn this thesis we have explored a range of performance barriers that can limitreal-world network performance, and identified solutions for them. The firstsuch issue is bufferbloat, where we have contributed to understanding theextent of the problem through an assessment of the magnitude of excess latencyin the public internet; performed a thorough evaluation of existing queuemanagement solutions targeted at mitigating bufferbloat; and analysed theperformance characteristics of the state of the FQ-CoDel algorithm in detail.Following this, we have shown how deployment of bufferbloat solutions canbe made easier through the CAKE integrated queue management and trafficshaping system.

Having identified WiFi networks as an area where existing bufferbloat solu-tions fell short, we have designed and implemented a solution for bufferbloattailored specifically to WiFi networks. In doing so, we identified several otherperformance barriers in WiFi networks, that we also offer solutions for. Theseinclude the 802.11 performance anomaly, the interaction between WiFi net-works and latency-reducing measures for TCP, and the lack of a mechanismfor airtime policy enforcement in WiFi networks.

Turning to a more general, but related, performance barrier in software-based network processing, we have investigated how the overhead that theoperating system imposes on high-performance packet processing applicationscan be reduced. We have contributed a description and thorough evaluation ofthe Linux community’s answer to this problem, which allows integration ofpacket processing programs into the networking stack in a cooperative way,achieving both flexibility and high performance.

Finally, over the course of implementing our solutions for the researchproblems, we have implemented a measurement tool designed to make exper-imental evaluations of networks more reliable and easier to perform. Thistool is already widely used in the bufferbloat community, and can significantlyincrease the quality and availability of sophisticated network measurements inthe future.

Looking forward, while the contributions presented here represent a signi-ficant step forward, there are certainly still performance barriers that preventreal-world networks from achieving their maximum performance potential.In the bufferbloat space, the most significant road-block is no doubt the effortneeded to get the new technology deployed to the hundreds of millions ofexisting devices all over the world, many of which have an upgrade cycle meas-ured in years. However, it is encouraging to see that awareness of the problemis at least increasing, with prominent examples being deployment in OS X,2and the inclusion of a bufferbloat measure in more online speed tests.3

As far as WiFi performance is concerned, there are several areas thatwarrant further study. Adopting the solutions presented here to the casewhere more functionality is offloaded into hardware is one such issue; either

2https://lists.bufferbloat.net/pipermail/bloat/2018- September/008615.html3E.g., https://www.dslreports.com/speedtest and https://fast.com.

23

solutions have to be implemented in the hardware or firmware itself, or theoperating system-level solutions need to take potential queueing at lowerlayers into account. Simultaneous transmission to multiple devices will likelyplay an increasing role in future WiFi deployments, which means schedulingalgorithms need to be adapted to deal with the case where several queues needto be serviced simultaneously. In addition, re-evaluating the 802.11e QoSmechanism in the light of modern queueing mechanisms could be anotherinteresting area of inquiry. Finally, while the solutions we have seen so farhas been limited to single access point cases, it is quite probable that betterresults could be achieved by cooperation between neighbouring access pointsin a dense deployment scenario (which is becoming increasingly common asWiFi becomes ever more ubiquitous).

The high-performance software packet processing space is perhaps themost active of the subject areas we touch upon in this thesis. As we haveseen, there are several competing architectures being explored; and futurework includes adding features such as transport acceleration and full or partialhardware offload. Finally, there are no doubt many application areas that willbenefit from the programmable capabilities enabled by frameworks such asXDP in the future.

In conclusion, the work presented here represents a significant contribu-tion to improving the performance of real-world networks today and in thefuture. But it is clear that future improvements will continue to push the per-formance envelope towards ever higher capacity, lower latency and ubiquitousconnectivity everywhere.

References[1] J. Gettys and K. Nichols, “Bufferbloat: Dark buffers in the internet,”

ACM Queue, vol. 9, no. 11, pp. 40:40–40:54, Nov. 2011.

[2] C. Staff, “BufferBloat: what’s wrong with the internet?” Communicationsof the ACM, vol. 55, no. 2, pp. 40–47, Feb. 2012.

[3] C. Kreibich et al., “Netalyzr: illuminating the edge network,” in InternetMeasurement Conference. ACM, 2010, pp. 246–259.

[4] S. Sundaresan et al., “Broadband internet performance: a view from thegateway,” in ACM SIGCOMM computer communication review, vol. 41.ACM, 2011, pp. 134–145.

[5] M. Chetty et al., “Measuring broadband performance in South Africa,”in 4th Annual Symposium on Computing for Development. ACM, 2013.

[6] M. Dischinger et al., “Characterizing residential broadband networks,”in Internet Measurement Conference. ACM, 2007, pp. 43–56.

[7] C. Chirichella and D. Rossi, “To the moon and back: are internet buf-ferbloat delays really that large?” in 2013 IEEE Conference on Computer

24

Communications Workshops (INFOCOM WKSHPS). IEEE, 2013, pp.417–422.

[8] H. Jiang et al., “Tackling bufferbloat in 3g/4g networks,” in InternetMeasurement Conference. ACM, 2012, pp. 329–342.

[9] S. Alfredsson et al., “Impact of TCP congestion control on bufferbloat incellular networks,” in IEEE 14th International Symposium and Workshopson a World of Wireless, Mobile and Multimedia Networks (WoWMoM),2013.

[10] J. Aikat et al., “Variability in TCP round-trip times,” in ACM conferenceon Internet measurement. ACM, 2003, pp. 279–284.

[11] M. Allman, “Comments on bufferbloat,” ACM SIGCOMM ComputerCommunications Review, vol. 43, no. 1, pp. 31–37, January 2013.

[12] S. Floyd and V. Jacobson, “Random early detection gateways for conges-tion avoidance,” IEEE/ACM Transactions on Networking, vol. 1, no. 4, pp.397–413, 1993.

[13] S. Floyd, R. Gummadi, and S. Shenker, “Adaptive RED: An algorithmfor increasing the robustness of REDs active queue management,” 2001.http://www.icir.org/floyd/papers.html

[14] P. McKenney, “Stochastic fairness queueing,” in INFOCOM ’90. NinthAnnual Joint Conference of the IEEE Computer and Communication Soci-eties. ’The Multiple Facets of Integration’. IEEE, jun 1990, pp. 733 –740vol.2.

[15] R. Adams, “Active Queue Management: A survey,” IEEE Communica-tions Surveys & Tutorials, vol. 15, no. 3, pp. 1425–1476, 2013.

[16] M. May et al., “Reasons not to deploy RED,” in 1999 Seventh Interna-tional Workshop on Quality of Service (IWQoS ’99), 1999, pp. 260–262.

[17] K. Nichols and V. Jacobson, “Controlling queue delay,” Communicationsof the ACM, vol. 55, no. 7, pp. 42–50, Jul. 2012.

[18] R. Pan et al., “PIE: A lightweight control scheme to address the buf-ferbloat problem,” in 2013 IEEE 14th International Conference on HighPerformance Switching and Routing (HPSR), July 2013, pp. 148–155.

[19] T. Høiland-Jørgensen et al., “The Flow Queue CoDel Packet Schedulerand Active Queue Management Algorithm,” RFC 8290 (Experimental),RFC Editor, Jan. 2018.

[20] G. Carofiglio and L. Muscariello, “On the impact of TCP and per-flowscheduling on internet performance,” IEEE/ACM Trans. Netw., vol. 20,no. 2, pp. 620–633, 2012.

25

[21] F. Checconi, L. Rizzo, and P. Valente, “QFQ: Efficient packet schedulingwith tight guarantees,” IEEE/ACM Transactions on Networking (TON),vol. 21, no. 3, pp. 802–816, 2013.

[22] A. Kortebi et al., “Evaluating the number of active flows in a schedulerrealizing fair statistical bandwidth sharing,” in ACM SIGMETRICS Per-formance Evaluation Review, vol. 33. ACM, 2005, pp. 217–228.

[23] J. Corbet, “Network transmit queue limits,” LWN Article, August 2011.https://lwn.net/Articles/454390/

[24] G. Niestegge, “The "leaky bucket" policing method in the atm (asyn-chronous transfer mode) network,” International Journal of Communica-tion Systems, vol. 3, no. 2, pp. 187–197, 1990.

[25] A. Eleftheriadis and D. Anastassiou, “Constrained and general dynamicrate shaping of compressed digital video,” in International Conference onImage Processing, vol. 3. IEEE, 1995, pp. 396–399.

[26] “IEEE standard for information technology–telecommunications andinformation exchange between systems local and metropolitan areanetworks–specific requirements part 11: Wireless LAN medium accesscontrol (MAC) and physical layer (PHY) specifications,” IEEE Std 802.11-2012 (Revision of IEEE Std 802.11-2007), pp. 1–2793, March 2012.

[27] M. Heusse et al., “Performance anomaly of 802.11 b,” in INFOCOM2003. Twenty-Second Annual Joint Conference of the IEEE Computer andCommunications. IEEE Societies, vol. 2. IEEE, 2003, pp. 836–843.

[28] N. Khademi, D. Ros, and M. Welzl, “The new AQM kids on the block:Much ado about nothing?” Oslo University, Tech. Rep. 434, 2013.

[29] A. Showail, K. Jamshaid, and B. Shihada, “Buffer sizing in wireless net-works: challenges, solutions, and opportunities,” IEEE CommunicationsMagazine, vol. 54, no. 4, pp. 130–137, Apr 2016.

[30] K. Cai et al., “Wireless Unfairness: Alleviate MAC Congestion First!”in Proceedings of the Second ACM International Workshop on WirelessNetwork Testbeds, Experimental Evaluation and Characterization, ser.WinTECH ’07. New York, NY, USA: ACM, 2007, pp. 43–50.

[31] L. B. Jiang and S. C. Liew, “Proportional fairness in wireless LANs and adhoc networks,” in Wireless Communications and Networking Conference,2005 IEEE, vol. 3. IEEE, 2005, pp. 1551–1556.

[32] P. Lin, W.-I. Chou, and T. Lin, “Achieving airtime fairness of delay-sensitive applications in multirate IEEE 802.11 wireless LANs,” IEEECommunications Magazine, vol. 49, no. 9, pp. 169–175, 2011.

26

[33] L. Sanabria-Russo et al., “Future evolution of CSMA protocols for theIEEE 802.11 standard,” in 2013 IEEE International Conference on Com-munications Workshops (ICC). IEEE, 2013, pp. 1274–1279.

[34] T. Joshi et al., “Airtime fairness for IEEE 802.11 multirate networks,”IEEE Transactions on Mobile Computing, vol. 7, no. 4, pp. 513–527, Apr2008.

[35] J. Dunn et al., “A practical cross-layer mechanism for fairness in 802.11networks,” in First International Conference on Broadband Networks(BroadNets 2004). IEEE, 2004, pp. 355–364.

[36] T. Razafindralambo et al., “Dynamic packet aggregation to solve perform-ance anomaly in 802.11 wireless networks,” in Proceedings of the 9th ACMinternational symposium on Modeling analysis and simulation of wirelessand mobile systems. ACM, 2006, pp. 247–254.

[37] M. Kim, E.-C. Park, and C.-H. Choi, “Adaptive two-level frame aggrega-tion for fairness and efficiency in IEEE 802.11n wireless LANs,” MobileInformation Systems, vol. 2015, 2015.

[38] R. G. Garroppo et al., “Providing air-time usage fairness in IEEE 802.11networks with the deficit transmission time (DTT) scheduler,” WirelessNetworks, vol. 13, no. 4, pp. 481–495, Aug 2007.

[39] R. Riggio, D. Miorandi, and I. Chlamtac, “Airtime Deficit Round Robin(ADRR) packet scheduling algorithm,” in 2008 5th IEEE InternationalConference on Mobile Ad Hoc and Sensor Systems, Sep. 2008, pp. 647–652.

[40] M. Richart et al., “Resource allocation for network slicing in WiFi accesspoints,” in 13th International Conference on Network and Service Manage-ment, CNSM, 2017, 2017.

[41] K. Katsalis et al., “Virtual 802.11 wireless networks with guaranteedthroughout sharing,” in 2015 IEEE Symposium on Computers and Com-munication (ISCC), Jul 2015.

[42] Y. Yiakoumis et al., “Slicing home networks,” in Proceedings of the 2NdACM SIGCOMMWorkshop on Home Networks, ser. HomeNets ’11. NewYork, NY, USA: ACM, 2011.

[43] “Air Time Fairness (ATF) Phase1 and Phase 2 Deployment Guide,”Cisco systems, 2015. https://www.cisco.com/c/en/us/td/docs/wireless/technology/mesh/8-2/b_Air_Time_Fairness_Phase1_and_Phase2_Deployment_Guide.html

[44] J. W. Lockwood et al., “NetFPGA–an open platform for gigabit-ratenetwork switching and routing,” in IEEE International Conference onMicroelectronic Systems Education. IEEE, 2007.

27

[45] P. Bosshart et al., “P4: Programming protocol-independent packet pro-cessors,” ACM SIGCOMM Computer Communication Review, vol. 44,no. 3, 2014.

[46] L. Rizzo and G. Lettieri, “Vale, a switched ethernet for virtual machines,”in Proceedings of the 8th international conference on Emerging networkingexperiments and technologies. ACM, 2012.

[47] S. Han et al., “PacketShader: a GPU-accelerated software router,” in ACMSIGCOMM Computer Communication Review, vol. 40, no. 4. ACM,2010.

[48] D. Kirchner et al., “Augustus: a CCN router for programmable net-works,” in Proceedings of the 3rd ACM Conference on Information-CentricNetworking. ACM, 2016.

[49] P. M. Santiago del Rio et al., “Wire-speed statistical classification ofnetwork traffic on commodity hardware,” in Proceedings of the 2012Internet Measurement Conference. ACM, 2012.

[50] R. B. Mansilha et al., “Hierarchical content stores in high-speed ICNrouters: Emulation and prototype implementation,” in Proceedings of the2nd ACM Conference on Information-Centric Networking. ACM, 2015.

[51] P. Emmerich et al., “Moongen: A scriptable high-speed packet generator,”in Proceedings of the 2015 Internet Measurement Conference. ACM, 2015.

[52] S. Han et al., “MegaPipe: A new programming interface for scalablenetwork I/O,” in USENIX OSDI ’12, 2012.

[53] T. Marian, K. S. Lee, and H. Weatherspoon, “NetSlices: scalablemulti-core packet processing in user-space,” in Proceedings of the eighthACM/IEEE symposium on Architectures for networking and communica-tions systems. ACM, 2012.

[54] L. Linguaglossa et al., “High-speed software data plane via vectorizedpacket processing,” Telecom ParisTech, Tech. Rep., 2017.

[55] R. Morris et al., “The Click modular router,” ACM SIGOPS OperatingSystems Review, vol. 33, no. 5, 1999.

[56] M. Dobrescu et al., “RouteBricks: exploiting parallelism to scale softwarerouters,” in Proceedings of the ACM SIGOPS 22nd symposium on Operatingsystems principles. ACM, 2009.

[57] B. Pfaff et al., “The design and implementation of Open vSwitch,” inUSENIX NSDI ’15, 2015.

[58] L. Deri, “Modern packet capture and analysis: Multi-core, multi-gigabit,and beyond,” in the 11th IFIP/IEEE International Symposium on IntegratedNetwork Management (IM), 2009.

28

[59] L. Rizzo, “Netmap: a novel framework for fast packet I/O,” in 21stUSENIX Security Symposium (USENIX Security 12), 2012.

[60] S. Peter et al., “Arrakis: The operating system is the control plane,” ACMTransactions on Computer Systems (TOCS), vol. 33, no. 4, 2016.

[61] J. Martins et al., “ClickOS and the art of network function virtualization,”in Proceedings of the 11th USENIXConference on Networked Systems Designand Implementation. USENIX Association, 2014.

[62] “PF_RING ZC (Zero Copy),” Ntop project, 2018. https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/

[63] “OpenOnload,” Solarflare Communications Inc, 2018. https://www.openonload.org/

[64] “Data plane development kit,” Linux Foundation, 2018. https://www.dpdk.org/

[65] M. Tedre, The science of computing: Shaping a discipline. CRC Press,2014.

[66] H. Bidgoli, The Internet Encyclopedia, Volume 2 (G - O), ser. The InternetEncyclopedia. Wiley, 2004.

Print & layout University Printing Office, Karlstad 2018

DOCTORAL THESIS Karlstad University Studies, 2018:42ISBN 978-91-7063-878-7 (Print) ISBN 978-91-7063-973-9 (pdf)




2018:42

The elephant symbo-lises a big download, commonly referred to

as an ”elephant flow”, which blocks the link with its bulk so everything has to wait behind it.

The mice are smaller flows, such as web

pages, that take up little space but should be able to move freely. Through better queue management, the mice are guided around the elephant, so they don’t have to wait; and interactivity is restored.

The penguin is Tux, the Linux mascot, who stands at the bottleneck

and directs traffic onto the right path.

the elephant the mice - divakau.diva-portal.org/smash/get/diva2:1251705/fulltext01.pdf · found:...

Documents