testing applications with traffic control in containers / alban crequy (kinvolk)
TRANSCRIPT
![Page 1: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/1.jpg)
In containers
Alban Crequy
Testing applicationswith traffic control
HighLoad++ - November 2016https://goo.gl/knaz6m
![Page 2: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/2.jpg)
Alban Crequy
∘ Worked on traffic control for multimedia applications in cars (tcmmd)
∘ Worked on the container run-time rkt∘ Working on Weave Scope
https://github.com/alban
![Page 3: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/3.jpg)
Berlin-based software company building foundational Linux technologies
Some examples of what we worked on...
OSTreegit for operating system binaries
![Page 4: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/4.jpg)
Who works with...
![Page 5: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/5.jpg)
Find out more about us…
Blog: https://kinvolk.io/blog
Github: https://github.com/kinvolk
Twitter: https://twitter.com/kinvolkio
Email: [email protected]
![Page 6: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/6.jpg)
∘ What is traffic control and how does it work on Linux∘ How it can be used for testing a microservices application∘ Demo
∘ With Minikube, Kubernetes, Weave Scope, Weave Socks
Plan
![Page 7: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/7.jpg)
What is traffic control?
How does it work on Linux?
![Page 8: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/8.jpg)
Traffic control, why?
web server client
client
client
THEINTERNET
∘ fair distribution of bandwidth
∘ reserve bandwidth to specific applications
∘ avoid bufferbloat
![Page 9: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/9.jpg)
∘ Network scheduling algorithm∘ which packet to emit next?∘ when?
∘ Configurable at run-time:∘ /sbin/tc∘ Netlink
∘ Default on new network interfaces: sysctl net.core.default_qdisc
Queuing disciplines(qdisc)
eth0 THE INTERNETqdisc
![Page 10: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/10.jpg)
Stochastic FairnessQueueing (sfq)
eth0
THE INTERNET
FIFO n
FIFO 1
FIFO 0
...
round robin
![Page 11: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/11.jpg)
Traffic control for testing?
![Page 12: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/12.jpg)
Developers & InternetDevelopers usually have good Internet connection
- Low latency- High bandwidth- Not much packet loss or packet corruption
Difficult to see problems
![Page 13: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/13.jpg)
Network emulator(netem)
eth0 THE INTERNETnetem
bandwidth
latency packet loss
corrupt...
![Page 14: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/14.jpg)
Testing with containers
container 1 container 2
eth0eth0
Testing framework
configure “netem” qdiscs:bandwidth, latency, packet drop...
![Page 15: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/15.jpg)
The demo application
![Page 16: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/16.jpg)
microservices-demo
https://github.com/microservices-demo/microservices-demo
![Page 17: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/17.jpg)
Some micro-services
front-end Firefox
catalogue
ordersorders-db
payment
![Page 18: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/18.jpg)
Kubernetes
![Page 19: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/19.jpg)
PodsGroup of container(s) running together, in the same context
Kubernetesnode 1
Kubernetesnode 2
pod
container
container
podcontainer
podcontainer
podcontainer
Each pod has
- Its own network namespace- Its own IP address
![Page 20: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/20.jpg)
Replica sets- Control the number of “Pods” replicas running- Deploy pods
Kubernetesnode 1
Kubernetesnode 2
podcatalogue
podfront-end
podorders
In this example:
- 2x front-end- 1x catalogue- 1x orders
podfront-end
![Page 21: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/21.jpg)
Replica set“catalogue”
ServicesDirecting the traffic to “Pods”
- Cluster IP- DNAT catalogue1
catalogue2
catalogue3
Service“catalogue”
front-end
![Page 22: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/22.jpg)
Testing with traffic control in Kubernetes
Kubernetesnode 1
pod
pod
Kubernetesnode 2
pod
pod
tc tccontrols∘ Latency∘ Bandwidth∘ Packet drop
∘ configure network simulator
∘ play scenarios
![Page 23: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/23.jpg)
Weave Scope
![Page 24: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/24.jpg)
Weave Scope
![Page 25: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/25.jpg)
Testing with Weave Scope
Kubernetesnode 1
tcScopeProbe
pod pod
pod pod
KubernetesNode 2
tcScopeProbe
pod pod
pod pod
ScopeApp
![Page 26: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/26.jpg)
<Demo>
Reproduce the demo yourself: https://github.com/kinvolk/demo
![Page 27: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/27.jpg)
![Page 28: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/28.jpg)
![Page 29: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/29.jpg)
![Page 30: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/30.jpg)
![Page 31: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/31.jpg)
![Page 32: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/32.jpg)
![Page 33: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/33.jpg)
![Page 34: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/34.jpg)
![Page 35: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/35.jpg)
![Page 36: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/36.jpg)
![Page 37: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/37.jpg)
![Page 38: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/38.jpg)
![Page 39: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/39.jpg)
![Page 40: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/40.jpg)
![Page 41: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/41.jpg)
![Page 42: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/42.jpg)
![Page 43: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/43.jpg)
![Page 44: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/44.jpg)
![Page 45: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/45.jpg)
![Page 46: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/46.jpg)
![Page 47: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/47.jpg)
</Demo>
Reproduce the demo yourself: https://github.com/kinvolk/demo
![Page 48: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/48.jpg)
Testing frameworkfor web apps
Selenium
![Page 49: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/49.jpg)
Plugins in Scope
tc pluginScopeProbe
ScopeApp
report
reportcontrol
control
- Unix socket in /var/run/scope/plugins/- Protocols: report and control- Write your own plugins for your testing needs
![Page 50: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/50.jpg)
![Page 51: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/51.jpg)
Testing more complex scenarios
(my “wishlist”)
![Page 52: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/52.jpg)
Add latency on a specific connection
front-end Firefox
catalogue
ordersorders-db
payment
latency=100ms
![Page 53: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/53.jpg)
How to define classes of traffic
eth0
netem
interface
latency=100ms
dest_ip=10.0.4.* dest_ip=10.0.5.* other
![Page 54: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/54.jpg)
u32: filter on contenteth0
HTB
HTB
HTBHTB HTB
netemnetem netem
interface
root qdisc (type = HTB)
root class (type = HTB)
leaf qdiscs (type = netem)
leaf classes (type = HTB)
filters (type=u32)
otherip=10.0.5.*ip=10.0.4.*
latency=10ms
![Page 55: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/55.jpg)
Filtering with cBPF/eBPF
eth0
BPF
netemnetem
kernel
userspace
BPF_JMP...BPF_LD...BPF_RET...
if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation
clang... -march=bpf
uploadin the kernel:
- bpf()- Netlink
x86_64 codeJIT compilation
![Page 56: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/56.jpg)
eBPF maps
eth0
BPF
netemnetem
kernel
userspace
x86_64 code
eBPF map
tc
∘ Build statistics∘ Make them available to
the testing framework
![Page 57: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/57.jpg)
- Another Weave Scope plugin could handlemore filters: u32 or eBPF
Advanced tc plugin
Anotherplugin
ScopeProbe
ScopeApp
control
controleth0
u32,eBPF, ...
![Page 58: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/58.jpg)
The EndTry the demos yourself: https://github.com/kinvolk/demo
Weave Scope’s traffic control plugin: https://github.com/weaveworks-plugins/scope-traffic-control
The slides: https://goo.gl/knaz6m
![Page 59: Testing applications with traffic control in containers / Alban Crequy (Kinvolk)](https://reader031.vdocuments.net/reader031/viewer/2022020213/586f90661a28ab54768b799f/html5/thumbnails/59.jpg)
Questions?