washington washington university in st louis endsystem support for network virtualization fred kuhns
DESCRIPTION
3 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 1/26/2016 Context: Virtual (Diversified) Networking substrate router virtual router substrate link virtual link virtual end-systemTRANSCRIPT
WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Endsystem Support for Network Virtualization
Fred Kuhns
2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Overview• Context• Endsystem networking model• Protocol instances: user or kernel space
– pros and cons– explore user space protocols– propose kernel level model
3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Context: Virtual (Diversified) Networking
substrate routervirtual router
substrate link
virtual link
virtual end-system
4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Simulates Star Topology for Substrate Links
…
ethernet switched LAN
VLANX1
Internetworking over a diversified networkEthernet example: • VLANs are used to provide the equivalent of a
virtualized “wire” connecting an endsystem to a specific access router.
• All vnets on an endsystem share common VLAN
• Use priority queuing (802.1P/Q) to isolate vnet traffic.
• Use admission control (static or dynamic) to provide bandwidth guarantees to vnet traffic.
• Substrate layer on endsystems enforce per VLAN and per vnet bandwidth constraints
• Each host to substrate router connection is assigned a distinct VLAN. So N hosts implies N VLANs on ethernet.
• Alternative is to define one VLAN tree for each protocol suite (i.e. vnet).
VLANX2 VLANXN
vNetX
VR1
5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
vNetX
VR1
vnetX traffic uses high priority queues
Ethernet Hubwith High and LowPriority TX queues
…
HighLow
HighLow
HighLow HighLow
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Substrate Link as a VLAN Tree
…
• One VLAN is used for all virtual net traffic to/from a substrate router.
VLANX
ethernet switched LAN
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Multiple Substrate Links
• Three VLANs are used for all virtual net traffic to/from a substrate router.
• Corresponds to 3 substrate links: 1.Low priority: default for best-effort traffic2.Medium priority for virtual nets with soft
performance requirements (average bandwidth)
3.High priority for isochronous or low-delay, interactive applications
VLANdgram
VLANmed
VLANhigh
ethernet switched LAN
…
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Multiple vNets per Host…
vlan 1 vlan 2 vlan 3
VLI VLI VLI
• Substrate link: serves to connect an endsystem to a substrate router. Virtualization of a physical cable or wire. A packet enters one end, exists the other and is opaque within. Simplex or Duplex?
• Substrate interface: (need better term?) endsystem abstraction representing a substrate link.
• Ethernet: <interface, VLAN, dest>. • Could be an IP tunnel• Not required to be point-to-point.
• Virtual link: represents the logical interconnection of adjacent network nodes for a given protocol suite.
• Point-to-point. Simplex or Duplex?• Virtual interface: endsystem abstraction representing one
end of a virtual link. Substrate defines mechanism for multiplexing onto common substrate link. For example a virtual link identifier (VLI) in a substrate header. Simplex or Duplex?
filter onethernet address and vlan membershipfor substrate router
ether addr/vlan
ether addr/vlan
ether addr/vlan
ethernet LAN
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Multiple next hop VRs?
VLANXA1
• Not a fundamental part of the model but it is consistent with the current model used for TCP/IP in endsystem.
• Allows us to implement TCP/IP as a virtual net protocol and not change the basic model vNetX
VR1
vNetX
VR2
vNetX
VR3
VLANXA2 VLANXA3
Host Aon vnetX
ethernet switched LAN
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
VLI VLI
IP
TCP/IP as an Example Protocol
…
destinationprefix
gatewayvirtual interface
substrateinterface
address
192.168.12.0/24 0.0.0.0 eth0 ARP
* 192.168.12.254vint0
(eth0, VLAN)VLI,dst
IP Route Tablevint0
(eth0 + VLANX)LL Info = SR1 addr + VLI
standard ethernetInterface
ethernet device
VLANX
direct connect
Substrate Interface:Ethernet interface. Destination address by ARP. Directly connected: destination IP address + ARP = enet addrGateway: (Gateway’s IP + ARP = enet addr) + VLAN
Virtual Interface:Directly connected: Not used, model only for internetworkingGateway: VLI assigned by substrate.
ethernet LAN
VLAN
VLI
Substrate RouterSR1
ethernetdest. addr
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
OS Kernel Block Diagram
configuration: registers, MMU (TLB, cache, VM) bus and peripheralsSystem Exception handlers
ethernet
Socket Interface
UDP RAW IP
IP routes
TCPnTCP2TCP1 …TCP module
clock handlerprocess accountingschedulingtime management
uart eth0timer
hardware dependent layer
HW interrupt/Exception
hardware independent layer
scheduler
SW int(AST)callout Q
TCPpoll
tasks
task management
openfiles
FS managementbuffercache
ops
File Interface ops
Device independent I/O
Inte
rrup
t Pro
cess
ing
AST
Pro
cess
ing
User Space (Applications)
Hardware
Basic I/O Interface
txqueue rxqueue
TC/AST
qdisc
device driver
OS ISR demux
callback
util
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
User or kernel Space protocols?• Each has pros and cons• User space protocols:
– easier to implement and debug– easier to introduce new protocols (not tightly dependent on socket layer
knowing about the new protocol)– easier to isolate and protect protocols and apps from each other (leverage
process model)
• kernel level protocols– easier to integrate into existing framework (simplifies support for system
interface functions like select/poll)– simplifies intra-protocol security and protection (since protocol runs within
trusted kernel)– simplifies kernel demultiplexing to correct protocol context (endpoint)– increased efficiency
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
User Space Protocol Implementation• Uncommon outside of high-performance community, they want
zero-copy and specialized demux keys.• Problems: asynchronous processing, life cycle, authentication and
demiultiplexing to endpoints– latency in delivering packets (i.e. acks) to user space– increased overhead in per packet processing before a drop/keep decision is
made– processing received acks– timeouts and retransmissions– establishing connections and security: snooping, masquerading– supporting select and poll– protocols where connection may outlive process (TCP’s TIMED_WAIT)– global routing and address resolution tables– global connection tables
• need to know what other ports are being used (locally)• accepting/rejecting new connections
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Assumptions• Assumptions:
– Applications using different VNs (or no VN) will need to communicate using the various IPC mechanisms
– We want to manage all aspects of Network I/O but not the use of other traditional resources (memory, files etc)
– CPU, memory and interface bandwidth controlled at the virtual net granularity
– intra-VN, implementers should have the mechanisms to support QoS and Security
– simple mechanism for adding new protocols/VNs
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
User Space Protocols
Chandramohan A. Thekkath , Thu D. Nguyen , Evelyn Moy , Edward D. Lazowska, Implementing network protocols at user level, IEEE/ACM Transactions on Networking (TON), v.1 n.5, p.554-565, Oct. 1993
Chris Maeda, Brian Bershad, Protocol Service Decomposition for High-Performance Networking, Proceedings of the 14th ACM Symposium on Operating Systems Principles. December 1993, pp. 244-255.
• Aled Edwards , Steve Muir, Experiences implementing a high performance TCP in user-space, Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication, p.196-205, 1995
• Kieran Mansley, Engineering a User-Level TCP for the CLAN Network, Proceedings of the ACM SIGCOMM workshop on Network-I/O convergence: experience, lessons, implications, Pages: 228 – 236, 2003
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
user-space protocols: Global Issues• Routing: Direct packets to/from correct endpoint/interface
– How is traffic demultiplexed and sent to the correct endpoint/process?• In-kernel filters
– Where are the routing tables and how are they maintained?• route fixed when connection established or located in shared memory
• Control: I use IPv4 as an example– Address resolution protocols/tables? – Other control protocols. For example ICMP, IGRP, others?– Where are the routing protocols implemented?
• Management:– Must manage a protocols namespace (for example, port numbers in IPv4).– Common programming technique, allow protocol instance to select local address part
• specify port = 0 and addr = 0 then implementation will assign correct values– Passive connect model?
• In IPv4 a server listens on a port (host:port:proto) for a connection request. To establish a connection a unique (to the endsystem) port number is assigned and new socket allocated.
– socket-oriented system calls must be supported. On UNIX must support non-blocking I/O with select and poll.
– Connection lifetime may outlast process.• For example TCP TIME_WAIT or simply waiting for a final ack or resending if no ack received.
• Security: we must provide sufficient mechanisms for protocol developers– implementations must be able to guard against masquerading and eavesdropping
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
User Space: Configurations• Given these global issues there are two likely
configurations:– all traffic passes through common protocol daemon in user
space– control daemon implements basic set of control functions while
user library implements majority of data path functions– prior work has shown the latter approach to be superior.
• Having all traffic pass through a common protocol daemon => at least one extra copy operation (kernel -> daemon -> user process)
• A better solution is for a daemon to insert relatively simple packet filters in kernel for established connections which directs packets to/filters packets from endpoints.
18WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
socket layer
connection filters
User-Space: Passive Open
vnetXcontrol daemon:
(namespace, lifecycle, connections)
vnetX: protocol library
application
ethernet
vnet demux
3. insert incoming andoutgoing filters forvnetX connection
1. connectionrequest (in)
4. new connection
0. listen/accept(passive open)
5. data, establishedconnections
compare against connection specific outgoing filter
use VLI to access incoming filters and use to demux to filter set and/or socket.
data copy
2. ack (out)
19WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
User-Space: Active Open
socket layer
connection filters
vnetXcontrol daemon:
(namespace, lifecycle, connections)
vnetX: protocol library
application
ethernet
vnet demux
3. insert incoming andoutgoing filters forvnetX connection
4. new connection
0. connect
5. data, establishedconnections
compare against connection specific outgoing filter
data copy
1. connectionrequest (out)
2. ack (in)
use VLI to access incoming filters and use to demux to filter set and/or socket.
20WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
socket layer
connection filters
User-Space: Datagram (Connectionless)
vnetX: protocol library
application
ethernet
vnet demux
1. insert incoming andoutgoing filters forvnetX connection
2. new connection(local address)
0. open(any)
3. data establishedconnections
compare against “connection” specific outgoing filter
use VLI to access incoming filters and use to demux to socket. In this case only the local part is used.
data copy
daemon fills in local address and binds to socket. No restrictions on destination
vnetXcontrol daemon:
(namespace, lifecycle, connections)
21WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
socket layer
connection filters
User-Space: Datagram (Connectionless)
vnetX: protocol library
application
ethernet
vnet demux
1. insert incoming andoutgoing filters forvnetX connection
2. new connection(local and remote)
0. open(local and remote addr)
3. data establishedconnections
compare against “connection” specific outgoing filter
use VLI to access incoming filters and use to demux to socket.
data copy
daemon fills in both local and destination addresses. Destination restricted
vnetXcontrol daemon:
(namespace, lifecycle, connections)
22WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
socket layer
connection filters
User-Space: App exits
vnetXcontrol daemon:
(namespace, lifecycle, connections)
vnetX: protocol library
application
ethernet
vnet demux
3. remove filters 1. connectionclose (out)
drop
2. ack (in/out)
TCP enters TIME_WAIT after close
23WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Extensible protocol frameworks in the kernel
• Herbert Bos, Bart Samwel, Safe Kernel Programming in the OKE, Proceedings of the fifth IEEE Conference on Open Architectures and Network Programming, June 2002
24WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
OKE• Context: For performance reasons it is useful to permit third parties to load optimized
modules into the kernel• Problem: Third party code is untrusted so loading into kernel will compromise system
security and reliability. Could use safe execution environment like java but incurs expensive runtime checks.
• Solution: create set of mechanisms and policies to permit non-root users to safely load untrusted application modules into kernel space with minimal impact on runtime performance.
– Safety: use a trusted compile to enforce policies (constraints). The constraints are designed to ensure the untrusted module will not adversely affect the kernel (core and loadable modules) or unrelated processes.
– User privileges: Vary enforced constraints based on user privileges (customizable language)– Termination: well defined termination boundaries to protect system state– Enforcement: Static and dynamic checks; language extensions– Ease of use: Familiar development environment using Cyclone (type safe, C extension) and
kernel module.• Contribution: definition of safe kernel programming environment that meets competing
needs:– performance– safety– ease of use– hosted in a commodity OS
25WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Considerations• Identified areas where modules may impact system
behavior1. program correctness: language restrictions for safety and
enforce coding conventions2. Memory access: static and dynamic enforcement of
memory access rules3. Kernel module access: static and dynamic enforcement
of kernel module (interface) access restrictions4. Resource usage: Bounded (deterministic or limited)
26WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Pushing protocols into the Kernel• Positives:
– All the issues associated with user-space protocol simply go away. Global tables and lifetime of the kernel
– Performance, efficiency, existing code base– Enhances intra-Protocol security– Simplifies integration with existing network I/O subsystems and
interfaces• Negatives:
– Isolation: More difficult to isolate system from protocol instances. Inter-protocol isolation difficult.
– Security: Proving trust/security more difficult– Implementation and debugging more difficult in kernel
27WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 05/03/23
Kernel-Space Protocols
…
ethetnet
TCPnTCP2TCP1 … UDP RAW IP
IP routes
TCP
eth device driver
HW interrupt/Exception
HW Interrupt
SW Interrupt
User Space (Applications)
Hardware
openfiles
FS managementbuffercache
opsFile Interface
I/O Interface
vnet Demux
VLAN
Application(s)
vnet Socket I/O Interfacevnet ops
vnet Protostate tables
/dev/protoX/dev/vnet
udp:porttcp:port rawIP…vnet:epvnet:ep
Socket InterfacePF_VNET PF_INET
eth0
route to interface
TCP/IPvnet Protostate tables …
Rework!