testing applications with traffic control in containers / alban crequy (kinvolk)

In containers

Alban Crequy

Testing applicationswith traffic control

HighLoad++ - November 2016https://goo.gl/knaz6m

Alban Crequy

∘ Worked on traffic control for multimedia applications in cars (tcmmd)

∘ Worked on the container run-time rkt∘ Working on Weave Scope

https://github.com/alban

Berlin-based software company building foundational Linux technologies

Some examples of what we worked on...

OSTreegit for operating system binaries

Who works with...

Find out more about us…

Blog: https://kinvolk.io/blog

Github: https://github.com/kinvolk

Twitter: https://twitter.com/kinvolkio

Email: hello@kinvolk.io

∘ What is traffic control and how does it work on Linux∘ How it can be used for testing a microservices application∘ Demo

∘ With Minikube, Kubernetes, Weave Scope, Weave Socks

Plan

What is traffic control?

How does it work on Linux?

Traffic control, why?

web server client

client

client

THEINTERNET

∘ fair distribution of bandwidth

∘ reserve bandwidth to specific applications

∘ avoid bufferbloat

∘ Network scheduling algorithm∘ which packet to emit next?∘ when?

∘ Configurable at run-time:∘ /sbin/tc∘ Netlink

∘ Default on new network interfaces: sysctl net.core.default_qdisc

Queuing disciplines(qdisc)

eth0 THE INTERNETqdisc

Stochastic FairnessQueueing (sfq)

eth0

THE INTERNET

FIFO n

FIFO 1

FIFO 0

...

round robin

Traffic control for testing?

Developers & InternetDevelopers usually have good Internet connection

- Low latency- High bandwidth- Not much packet loss or packet corruption

Difficult to see problems

Network emulator(netem)

eth0 THE INTERNETnetem

bandwidth

latency packet loss

corrupt...

Testing with containers

container 1 container 2

eth0eth0

Testing framework

configure “netem” qdiscs:bandwidth, latency, packet drop...

The demo application

microservices-demo

https://github.com/microservices-demo/microservices-demo

Some micro-services

front-end Firefox

catalogue

ordersorders-db

payment

Kubernetes

PodsGroup of container(s) running together, in the same context

Kubernetesnode 1

Kubernetesnode 2

pod

container

container

podcontainer

podcontainer

podcontainer

Each pod has

- Its own network namespace- Its own IP address

Replica sets- Control the number of “Pods” replicas running- Deploy pods

Kubernetesnode 1

Kubernetesnode 2

podcatalogue

podfront-end

podorders

In this example:

- 2x front-end- 1x catalogue- 1x orders

podfront-end

Replica set“catalogue”

ServicesDirecting the traffic to “Pods”

- Cluster IP- DNAT catalogue1

catalogue2

catalogue3

Service“catalogue”

front-end

Testing with traffic control in Kubernetes

Kubernetesnode 1

pod

pod

Kubernetesnode 2

pod

pod

tc tccontrols∘ Latency∘ Bandwidth∘ Packet drop

∘ configure network simulator

∘ play scenarios

Weave Scope

Weave Scope

Testing with Weave Scope

Kubernetesnode 1

tcScopeProbe

pod pod

pod pod

KubernetesNode 2

tcScopeProbe

pod pod

pod pod

ScopeApp

<Demo>

Reproduce the demo yourself: https://github.com/kinvolk/demo

</Demo>

Reproduce the demo yourself: https://github.com/kinvolk/demo

Testing frameworkfor web apps

Selenium

Plugins in Scope

tc pluginScopeProbe

ScopeApp

report

reportcontrol

control

- Unix socket in /var/run/scope/plugins/- Protocols: report and control- Write your own plugins for your testing needs

Testing more complex scenarios

(my “wishlist”)

Add latency on a specific connection

front-end Firefox

catalogue

ordersorders-db

payment

latency=100ms

How to define classes of traffic

eth0

netem

interface

latency=100ms

dest_ip=10.0.4.* dest_ip=10.0.5.* other

u32: filter on contenteth0

HTB

HTB

HTBHTB HTB

netemnetem netem

interface

root qdisc (type = HTB)

root class (type = HTB)

leaf qdiscs (type = netem)

leaf classes (type = HTB)

filters (type=u32)

otherip=10.0.5.*ip=10.0.4.*

latency=10ms

Filtering with cBPF/eBPF

eth0

BPF

netemnetem

kernel

userspace

BPF_JMP...BPF_LD...BPF_RET...

if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation

clang... -march=bpf

uploadin the kernel:

- bpf()- Netlink

x86_64 codeJIT compilation

eBPF maps

eth0

BPF

netemnetem

kernel

userspace

x86_64 code

eBPF map

tc

∘ Build statistics∘ Make them available to

the testing framework

- Another Weave Scope plugin could handlemore filters: u32 or eBPF

Advanced tc plugin

Anotherplugin

ScopeProbe

ScopeApp

control

controleth0

u32,eBPF, ...

The EndTry the demos yourself: https://github.com/kinvolk/demo

Weave Scope’s traffic control plugin: https://github.com/weaveworks-plugins/scope-traffic-control

The slides: https://goo.gl/knaz6m

Questions?