Nagoya Institute of Technology, Japan

Implementation and Performance Analysis of STT Tunneling Protocol using vNIC Offloading (CVSW)

Ryota Kawashima and Hiroshi Matsuo

2

Outlines

Backgrounds Network Virtualization

Network Virtualization Overlays (NVO3)

Tunneling protocols VXLAN, NVGRE, and STT

Goals

Implementation CVSW framework

Performance Evaluation

Conclusions

3

Network Virtualization

Multi-tenant Datacenter Networks Each tenant can have its own virtual networks Each virtual network shares the physical network resources

Physical network

VM

VM

Tenant 1

Tenant 2

Tenant 3VM

VM VM

Virtual networks

VM

VM

VM

VM

4

Virtual NetworkTenantSystem

TenantSystem

Virtual NetworkTenantSystem

TenantSystem

NVO3 – Network Virtualization Overlays

L3 tunneling with multi-tenancy support RFC 7364, 7365

Physical network

Virtualization

NVE NVE

L3 tunnel

NVE : Network Virtualization Edge

5

NVE: Network Virtualization Edge

Tunnel end point Physical switches Virtual switches

Open vSwitch (OVS), NSX switch, Hyper-V virtual switch

NVE

TenantSystem

TenantSystem

Ove

rlay

Mod

ule

VNI

VNI

Logical ports

L3 tunnel

Encapsulates/Decapsulates

6

Tunneling Protocols

VXLAN (Virtual eXtensible LAN, RFC 7348) UDP based Linux kernel, OVS, VMware NSX, Cisco Nexus 1000V …

NVGRE (RFC draft) GRE based (no L4 protocol) Microsoft Hyper-V

Ethernet(Physical)

IP(Physical)

VXLANUDP FCSEthernet(Virtual)

Payload

Ethernet(Physical)

IP(Physical)

NVGRE FCSEthernet(Virtual)

Payload

Throughput is poor with software NVE

7

Yet Another Tunneling Protocol

STT (Stateless Transport Tunneling, RFC draft) Pseudo-TCP header

Exploiting TSO (TCP Segmentation Offload) feature Semantics of header fields are modified

VMware NSX

Ethernet(Physical)

IP(Physical)

STTPseudo-

TCP FCSEthernet(Virtual)

Payload

NIC

"This is a usual TCP packet !"

Protocol numberis 6 (TCP)

8

Is STT Best ?

Performance Much better than other tunneling protocols 2 times faster at maximum

Side effect...

Middleboxes can discard STT packets !

Firewalls, load balancers, ...

9

Our Goals

1. Implement various tunneling protocols

2. Evaluate performance of the protocols

3. Design a new high-performance protocol

Target of this paper

10

Implementation

Implemented tunneling protocols VXLAN, NVGRE, and STT

Platform Linux (CentOS 6.5) KVM CVSW framework

11

CVSW Framework

High-functional software component within vNIC

Simplifies tunneling protocol development

Open source (https://github.com/sdnnit/cvsw_net)

Virtualswitch

CVSW

VMUser space

Kernel space

Protocol stack

Physical serverOpenFlow controller

OpenFlow session

DC network

Match Action

OpenFlow+α

Flow table

・ OF Match/Action・ IP tunneling・MTU setting・ Offload setting

* R. Kawashima and H. Matsuo, "Virtual NIC Offloading Approach for Improving Performance of Virtual Networks", IEICE Transactions, vol.J97-B, no.4, pp. 639-647, 2014. (in Japanese)

12

Performance Evaluation

Iperf client

CVSW

VM

Iperf server

CVSW

VM

VXLAN, NVGRE, STT tunnel

TCP communication

Physical server 1 Physical server 2

Virtualswitch

Virtualswitch

40GbE

Environment

13

Performance Evaluation

Machine specifications

14

End-to-End Throughput (TCP)

15

Offloading Effects

GRO effect !

Offload Tx / Rx NIC / Kernel

TSO Tx NIC

GSO Tx Kernel

GRO Rx Kernel

16

New Tunneling Protocol ?

TSO does not improve performance

GRO is the key to high-performance protocol

Pseudo-TCP header is not required

L4 protocol should be Byte-stream or segment oriented

17

Conclusions

Tunneling protocols for NVO3 VXLAN and NVGRE have performance problem The performance of STT is excellent

Packet discarding problem

GRO is the key to high-performance communication

Future work Design and implement a new L4 protocol

GRO-aware Byte-stream oriented protocol