What is new in Neutron QoS?

Miguel Lavallemiguel@mlavalle.comIRC: mlavalle@freenode

Slawek Kaplonskiskaplons@redhat.comIRC: slaweq@freenode

2

AGENDA

● Introduction to Neutron QoS

● Existing QoS features in Neutron

● New features added in Queens release

● Planned New features

Neutron QoS - basic concept

Neutron QoS - data model

openstack network qos policy create Summit_Policy+-------------+--------------------------------------+| Field | Value |+-------------+--------------------------------------+| description | || id | 8a495fea-e617-45f6-82e6-a61f7860c5cc || is_default | False || name | Summit_Policy || project_id | 0783adde257d4a0280d3df1908b217c8 || rules | [] || shared | False || tags | [] |+-------------+--------------------------------------+

5

Neutron QoS - usageCreate QoS policy

openstack network qos rule create --type bandwidth-limit \--max-kbps 1024 \--max-burst-kbit 512 \Summit_Policy

+----------------+--------------------------------------+| Field | Value |+----------------+--------------------------------------+| direction | egress || id | f9bb1173-1bf0-4942-8932-a69b61ef5b52 || max_burst_kbps | 512 || max_kbps | 1024 || name | None || project_id | |+----------------+--------------------------------------+

6

Neutron QoS - usageCreate QoS rule in policy

openstack port set --qos-policy Summit_Policy \aa42976b-03d0-4fbb-90f8-fa7b88da8a2e

openstack port show aa42976b-03d0-4fbb-90f8-fa7b88da8a2e+-----------------------+-------------------------------------------------+| Field | Value |+-----------------------+-------------------------------------------------+| created_at | 2018-05-07T14:51:49Z || device_id | 80afc6fb-12a1-4bae-8cd4-6835aa819227 || id | aa42976b-03d0-4fbb-90f8-fa7b88da8a2e || mac_address | fa:16:3e:4e:e7:bc || network_id | c8714b0a-f960-4db3-bbf6-f6dddb5a6843 || project_id | 24f901074d52470da2c641e26c2753e1 || qos_policy_id | 8a495fea-e617-45f6-82e6-a61f7860c5cc || revision_number | 8 || status | ACTIVE || updated_at | 2018-05-09T20:36:11Z |+-----------------------+-------------------------------------------------+

7

Neutron QoS - usageAttach QoS policy to port (or network)

Open vSwitch SR-IOV Linux Bridge L3 agent

Bandwidth limit Egress / Ingress Egress (1) Egress / Ingress Egress / Ingress (3)

Minimum Bandwidth

- Egress (2) - -

DSCP Marking Egress - Egress -

8

Neutron QoS - available rule types

1. Max burst parameter is skipped because it's not supported by ip tool2. Best effort - scheduler is not aware about this minimum bandwidth requirement for ports3. Limit for Floating IP

openstack network qos rule create \--type bandwidth-limit \--max-kbps 512 \--max-burst-kbit 128 \--ingress \Summit_Policy

+----------------+--------------------------------------+| Field | Value |+----------------+--------------------------------------+| direction | ingress || id | f5fc67f8-83a2-40be-ad12-d15fbe29381f || max_burst_kbps | 128 || max_kbps | 512 || name | None || project_id | |+----------------+--------------------------------------+

9

Neutron QoS - bandwidth limit ruleRule API

sudo ovs-vsctl list qos_uuid : a4857dc7-e64f-4370-9cf2-408566871bf1external_ids : {id="tap9a566f82-0c"}other_config : {max-rate="512000"}queues : {0=671cc958-2328-4d77-9e36-91984582e5a5}type : linux-htb

sudo ovs-vsctl list queue_uuid : 671cc958-2328-4d77-9e36-91984582e5a5dscp : []external_ids : {id="tap9a566f82-0c", queue_type="0"}other_config : {burst="128000", max-rate="512000"}

sudo ovs-vsctl list port tap9a566f82-0c | grep qosqos : a4857dc7-e64f-4370-9cf2-408566871bf1

10

Neutron QoS - bandwidth limit ruleBackend implementation - Open vSwitch L2 agent - ingress traffic

sudo ovs-vsctl list interface tap9a566f82-0c | grep ingress_ingress_policing_burst: 256ingress_policing_rate: 1024

11

Neutron QoS - bandwidth limit ruleBackend implementation - Open vSwitch L2 agent - egress traffic

tc qdisc show dev tap2e939f9e-9eqdisc tbf 8002: root refcnt 2 rate 512Kbit burst 16Kb lat 50.0ms

12

Neutron QoS - bandwidth limit ruleBackend implementation - Linux bridge L2 agent - ingress traffic

tc filter show dev tap2e939f9e-9e parent ffff:filter protocol all pref 49 basic chain 0filter protocol all pref 49 basic chain 0 handle 0x1 police 0x1 rate 1024Kbit burst 32Kb mtu 64Kb action drop overhead 0b ref 1 bind 1

13

Neutron QoS - bandwidth limit ruleBackend implementation - Linux bridge L2 agent - egress traffic

ip link show enp8s0f03: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 90:e2:ba:5e:a6:40 brd ff:ff:ff:ff:ff:ffvf 0 MAC 00:00:00:00:00:00, tx rate 1000 (Mbps), max_tx_rate 1000Mbps,

spoof checking on, link-state autovf 1 MAC 56:92:c6:c6:e5:db, tx rate 2000 (Mbps), max_tx_rate 2000Mbps,

spoof checking on, link-state auto

14

Neutron QoS - bandwidth limit ruleBackend implementation - SR-IOV L2 agent - egress traffic

openstack network qos rule create \--type dscp-marking \--dscp-mark 16 \Summit_Policy

+------------+--------------------------------------+| Field | Value |+------------+--------------------------------------+| dscp_mark | 16 || id | e5256a9d-0ebd-4e45-acfe-bd2048d33d53 || name | None |+------------+--------------------------------------+

15

Neutron QoS - DSCP marking ruleRule API

sudo ovs-ofctl show br-int | grep tap9a566f82-0c 141(tap9a566f82-0c): addr:fe:16:3e:47:85:27

sudo ovs-ofctl dump-flows br-int | grep in_port=141 | grep tos

cookie=0x66a4b613766ef2e1, duration=70.889s, table=0, n_packets=0, n_bytes=0, idle_age=405, priority=65535,reg2=0,in_port=141 actions=mod_nw_tos:64,load:0x37->NXM_NX_REG2[0..5],resubmit(,0)

16

Neutron QoS - DSCP marking ruleBackend implementation - Open vSwitch L2 agent

iptables -t mangle -nL neutron-linuxbri-qos-o2e939fChain neutron-linuxbri-qos-o2e939f (1 references)target prot opt source destinationDSCP all -- 0.0.0.0/0 0.0.0.0/0 DSCP set 0x10

17

Neutron QoS - DSCP marking ruleBackend implementation - Linux bridge L2 agent

openstack network qos rule create \--type minimum-bandwidth \--min-kbps 512 \--egress \Summit_Policy+------------+--------------------------------------+| Field | Value |+------------+--------------------------------------+| direction | egress || id | 5e5969f2-66f8-4bed-9905-b31b33902189 || min_kbps | 512 || name | None |+------------+--------------------------------------+

18

Neutron QoS - Minimum bandwidthRule API

New rule types validation and reporting

20

Rules validation - old way

Report of available rule typesExample for deployment with openvswitch and sriovnicswitch mechanism

drivers

openstack network qos rule type list+-----------------+| Type |+-----------------+| bandwidth_limit | (- supported by both drivers)+-----------------+

21

Rules validation - old way - problems

● There was no real validation - it was only information for

user,

● Very limited range of rule types available in mixed

deployments,

● Some attributes of rules have big impact of backend

implementation (e.g. direction)

22

Rules validation - new way

Report of available rule typesExample for deployment with openvswitch and sriovnicswitch mechanism

drivers:

openstack network qos rule type list+-------------------+| Type |+-------------------+| bandwidth_limit | (- supported by both drivers)| dscp_marking | (- supported by openvswitch driver)| minimum_bandwidth | (- supported by sriovnicswitch driver)+-------------------+

23

Rules validation - new way

Report details of rule typeExample for deployment with openvswitch and sriovnicswitch mechanism

drivers

+----------------+------------------------------------------------------------------+ | Field | Value | +----------------+------------------------------------------------------------------+ | drivers | {'name': 'openvswitch', | | | 'supported_parameters': [{'parameter_values': [0, 8, .., 56], | | | 'parameter_type': 'choices', | | | 'parameter_name': 'dscp_mark'}]}] | | rule_type_name | dscp_marking | +----------------+------------------------------------------------------------------+

24

Rules validation - new way - validation

Validation if QoS policy can be applied to port (ports) during:

● Create/Update QoS rule,

● Create/Update port,

● Update network

Returns HTTP Conflict (409) error in case if rule(s) can’t be

applied for at least one port

25

Rules validation - new way - validation openstack network qos policy show DSCP_MARK_Policy+-------------+------------------------------------------+| Field | Value |+-------------+------------------------------------------+| name | DSCP_MARK_Policy || rules | [{u'type': u'dscp_marking'] |+-------------+------------------------------------------+

openstack network qos policy show Min_BW_Policy+-------------+------------------------------------------+| Field | Value |+-------------+------------------------------------------+| name | Min_BW_Policy || rules | [{u'type': u'minimum_bandwidth'] |+-------------+------------------------------------------+

openstack port show 5cac029f-dbac-4c04-ab81-a6ab8faefc43 -c binding_vif_type -c id+------------------+--------------------------------------+| Field | Value |+------------------+--------------------------------------+| binding_vif_type | ovs || id | 5cac029f-dbac-4c04-ab81-a6ab8faefc43 |+------------------+--------------------------------------+

26

Rules validation - new way - validation

openstack port set 5cac029f-dbac-4c04-ab81-a6ab8faefc43 --qos-policy Min_BW_PolicyConflictException: 409

openstack port show 5cac029f-dbac-4c04-ab81-a6ab8faefc43 -c qos_policy_id -c id+---------------+--------------------------------------+| Field | Value |+---------------+--------------------------------------+| id | 5cac029f-dbac-4c04-ab81-a6ab8faefc43 || qos_policy_id | None |+---------------+--------------------------------------+

Apply QoS policy with minimum bandwidth rule (NOT

supported by openvswitch driver) - Fails

27

Rules validation - new way - validation

openstack port set 5cac029f-dbac-4c04-ab81-a6ab8faefc43 --qos-policy DSCP_MARK_Policyopenstack port show 5cac029f-dbac-4c04-ab81-a6ab8faefc43 -c qos_policy_id -c id+---------------+--------------------------------------+| Field | Value |+---------------+--------------------------------------+| id | 5cac029f-dbac-4c04-ab81-a6ab8faefc43 || qos_policy_id | 4d8cf13e-2a34-43a2-a642-53b5d4dd898d |+---------------+--------------------------------------+

Apply QoS policy with DSCP Marking rule (supported by

openvswitch driver) - OK

New features in Queens release: bandwidth limit rules for floating IPs

29

Overview● Only bandwidth limit rules for floating IPs

● Both directions available: egress and ingress

● Rules are implemented as tc filters applied to:

○ qg device in qr name space for legacy and HA routers

○ rfp device in qr name space for DVR local routers

○ qg device in snat name space for DVR edge routers

● Filters are applied when floating ip is associated to a port

30

Create egress rule

$ openstack network qos rule create --type bandwidth-limit --max-kbps 3000 --max-burst-kbits 2400 --egress Summit_Policy+----------------+--------------------------------------+| Field | Value |+----------------+--------------------------------------+| direction | egress || id | 152e43f2-cb81-431b-a0a5-c8b7ac060016 || max_burst_kbps | 2400 || max_kbps | 3000 || name | None || project_id | |+----------------+--------------------------------------+

31

Create ingress rule

$ openstack network qos rule create --type bandwidth-limit --max-kbps 2100 --max-burst-kbits 1800 --ingress Summit_Policy+----------------+--------------------------------------+| Field | Value |+----------------+--------------------------------------+| direction | ingress || id | cd1ac042-2e78-4565-9186-960b2ee026ad || max_burst_kbps | 1800 || max_kbps | 2100 || name | None || project_id | |+----------------+--------------------------------------+

32

Create or update floating IP with QoS policy$ openstack floating ip create --qos-policy Summit_Policy public+---------------------+--------------------------------------+| Field | Value |+---------------------+--------------------------------------+| created_at | 2018-05-20T23:35:29Z || description | || dns_domain | || dns_name | || fixed_ip_address | None || floating_ip_address | 172.24.4.8 || floating_network_id | 90e59a26-8f24-4a93-b4d2-c496c2b56df5 || id | 715bf74c-4f09-4bf2-9293-9fa9cde08716 || name | 172.24.4.8 || port_id | None || project_id | d31cac1d60e34311bc47d557425fa24c || qos_policy_id | 67293480-f179-4434-b239-eb5ae743caab || revision_number | 1 || router_id | None || status | DOWN || subnet_id | None || tags | [] || updated_at | 2018-05-20T23:35:29Z |+---------------------+--------------------------------------+

33

After associating floating IP to VM port$ sudo ip netns exec qrouter-4aa23079-dfe1-4250-a788-94777b4bb13d tc qdisc show dev \ > qg-e0da4660-c9qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 63 direct_qlen 1000qdisc ingress ffff: parent ffff:fff1 ----------------

$ sudo ip netns exec qrouter-4aa23079-dfe1-4250-a788-94777b4bb13d tc -p -s -d filter show dev qg-e0da4660-c9filter parent 1: protocol ip pref 1 u32 filter parent 1: protocol ip pref 1 u32 fh 800: ht divisor 1 filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 match IP src 172.24.4.8/32 police 0x5 rate 3Mbit burst 300Kb mtu 2Kb action drop overhead 0b linklayer ethernet ref 1 bind 1

Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

34

After associating floating IP to VM port (cont.)

$ sudo ip netns exec qrouter-4aa23079-dfe1-4250-a788-94777b4bb13d tc -p -s -d filter show \ > dev qg-e0da4660-c9 parent ffff:fff1 prio 1filter parent ffff: protocol ip u32 filter parent ffff: protocol ip u32 fh 800: ht divisor 1 filter parent ffff: protocol ip u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 match IP dst 172.24.4.8/32 police 0x4 rate 2100Kbit burst 225Kb mtu 2Kb action drop overhead 0b linklayer ethernet ref 1 bind 1

Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

Planned new features: Strict minimum bandwidth support

36

What do we mean by strict minimum bandwidth support?

● Enforcement of minimum QoS policy minimum bandwidth

rules in the data plane

○ This can be done by Neutron alone

○ Egress minimum bandwidth rules are currently available

for SR-IOV back-end

○ Plans to implement for ingress and for other back-ends:

OVS, LB, etc.

37

What do we mean by strict minimum bandwidth support? (cont.)

● Avoid oversubscription of interfaces when scheduling VMs

and their ports in compute hosts

○ This will require cooperation between Neutron and

Nova

○ Neutron will influence Nova’s scheduling decision

○ Nova will influence Neutron’s port binding process

38

Limitation of first experimental release

● Ports will have to be created and associated to QoS policy

by user and then provided in the VM boot request

○ This is because currently, when booting an instance,

Nova creates its ports in the compute host, after the

scheduling decision has been made

○ Plans exist to move port creation by Nova to the

Conductor, i.e., before the scheduling decision

39

Components of strict minimum bandwidth support

● Neutron communicates to Nova the interfaces and

bandwidth inventory using the Placement API

● User sets in Neutron the minimum bandwidth requirements

of a port

● Neutron communicates to Nova the minimum bandwidth

requirements of a port

● Nova communicates to Neutron the selected host and the

backend within that host

40

Placement API concepts

Resource Provider

Source of resources like bandwidth

InventoriesTraits

Non-consumable qualitative aspects

of resources

Egress bandwidth

Ingress bandwidth VNIC typePhysnet

41

Placement data for each compute hostCompute RP

name = hostname

Agent RP(OVS agent)

uuid = agent_uuidPhysical NIC RP

uuid = uuid5(hostname:br0)traits: physnet_1, VNIC_TYPE_NORMALinventory: {ingress_kilobits_per_sec: 10000, egrees_kilobits_per_sec: 10000}

Physical NIC RPuuid = uuid5(hostname:br1)traits: physnet_2, VNIC_TYPE_NORMALinventory: {ingress_kilobits_per_sec: 10000, egrees_kilobits_per_sec: 10000} Agent RP

(SR-IOV agent)uuid = agent_uuid

Physical NIC RPuuid = uuid5(hostname:br0)traits: physnet_3, VNIC_TYPE_DIRECTinventory: {ingress_kilobits_per_sec: 10000, egrees_kilobits_per_sec: 10000}

Communicating to Nova interfaces and bandwidth inventories

ReSTNeutron Server Placement ServiceAgent RPC

Agent heartbeat ‘configurations’ with:- Bridge / interface mappings- Resource providers inventories- Resource providers traits

Config

openstack network qos rule create \--type minimum-bandwidth \--min-kbps 512 \--egress \Summit_Policy+------------+--------------------------------------+| Field | Value |+------------+--------------------------------------+| direction | egress || id | 5e5969f2-66f8-4bed-9905-b31b33902189 || min_kbps | 512 || name | None |+------------+--------------------------------------+

openstack port set --qos-policy Summit_Policy Summit_Port

43

User sets in Neutron the minimum bandwidth requirements of a port

Neutron communicates to Nova the minimum bandwidth requirements of a port

{ “port”: { “status”: “ACTIVE”, “name”: “Summit_Port”, ... “device_id”: “5e3898d7-11be-483e-9732-b2f5eccd2b2e”, “resource_request": { "resources": { "NET_BANDWIDTH_INGRESS_KILOBITS_PER_SECOND": 1000, "NET_BANDWIDTH_EGRESS_KILOBITS_PER_SECOND": 1000, } "required": [ "CUSTOM_PHYSNET_NET0", "CUSTOM_VNIC_TYPE_NORMAL” ] } }}

Nova communicates to Neutron the selected host and backend

{ “port”: { “binding:host_id”: “allinone”, “binding:profile”: { “allocation”: “rp uuid” } }}

Questions?