DPACC Management Aspects

Howard Huang, HuaweiPeng Yuan, Huawei

2

Bio Standard Engineer

• ETSI NFV• SNIA Object Drive TWG• OCI/Spec

Open Source Community Operation Manager• OpenStack (Tricircle, Kingbird, Dragonflow, …)• OPNFV (Parser, Multisite, DPACC, …)• OpenVswitch

Like Heavy Metal Music and father of twin girls !

3

D-PACC-MAN !

4

Overview

• Memory Lane• Detailed Work Proposal

5

Memory Lane : High Level Architecture

5

Acc DBInventory

DB

Catalogue DB

Nova Cinder Neutron

Acc DB Acc DB Acc DB

RabbitMQ

• Inventory DB : the state datastore that maintains overall acceleration resource state

• Catalogue DB : the config datastore that stores information such as version, performance, capability, compatibility and so forth.

Keystone

66

Memory Lane : Distributed Acc Management

Nova Cinder Neutron

Cat DB Cat DB Cat DB

Keystonedpacc

Inv DB

dpacc

Inv DB

dpacc

Inv DB

dpacc

Inv DB

7

Memory Lane : ETSI NFV Acceleration For DPACCVIM needs to be enhanced in three respects.

1) VI-Ha: Discovery of accelerator resource (Intel CCK, FPGA, GPU) in the Host.

2) Nf-Vi: Allocation and release of acceleration resources.

3)Or-Vi/Vi-Vnfm: External interface supporting VNFD.

"ACC":{ "acc_type":"IPSEC_3DES",    "acc_capability":  {"num":"800Mbps"} }

VNF

Virtio

HW Acc Devices

SAL

SW Acc DeviceSR-IOV Acc Driver

NFVI

Physical Device Driver

SR-IOV

APP

Crypto Interface DPI Interface Codec Interface

SAL (VirtIO Backend Driver)

VirtIO Front Driver

IPSec/GB/TLS/SRTP VNF1 Media

Intel CCK FPGA GPGPU

DPACC-MAN Agent

Management

MANO

NFVO

VNFM

VIM/OpenStack

Nova

Neutron

DPACC-MAN

VI-Ha

Nf-Vi

Vi-Vnfm

Or-Vi

8

Detailed Work Plan : OpenStack Rock

Nova-API

Queue

Nova-Compute

NovaDB

Nova-conductor

Nova-scheduler

libvirt

Openstack DashboardHorizon

HTTP(S)

Rock Agent

Queue

Rock

RockDB

HACC-API

hypervisorVM1 VM2 VM3 VM4

Acceleratorsacc1 acc2 acc3 acc4

Rock Conductor

AMQP

Nova

Rock

GlanceNeutron

A new project called Rock will be proposed in OpenStack to enhance Nova for accelerator management. Rock is responsible for lifecycle management of accelerators residing in a compute node or shared resource pool.HACC-API: “Standard” API for different accelerators.

9

OpenStack

Nova Neutron Rock

OpenStack Agent

Neutron Agent

Rock Agent

MANO 、 HEAT

Host OS

ACC1 、 System booting

2 、 Load driver3 、 Scan PCI for accelerator VF info

4 、 accelerator info

5 、 report accelerator info

6 、 create VM

8 、 Rock scheduling

Nova Agent

7 、 Create VM

9 、 Apply accelerator

10 、 config accelerator

11 、 Config accelerator

HACC API

VNFD:"ACC":{ "acc_type":"IPSEC_3DES",    "acc_capability":  {"num":"800Mbps"} }

Highlights:1) Accelerator QoS aware

scheduling for low latency APP in NFV.Scheduler guarantees that each VNF gets its acceleration ability and Load balance of acceleration resources among different VNFs.

2) Standard southbound APIs for different accelerators.

Detailed Work Plan : Rock Workflow

Copyright©2015 Huawei Technologies Co., Ltd. All Rights Reserved.The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.

11

Accelerator Management: Deployment View in Virtualizatoin (SRIOV+Virtio)

aaa VNFVNFC1

GuestLinux

VNFC2

Guest Linux

DPIAPI

LB

Guest Linux

Server

Host Linux

KVM/VMWare

HACC API

AAL

IPSecAPI

Algo

SRIOVCrypto

SRIOVGb

SRIOVDPI

AAL

SRIOVCrypto

ACC

MANO

(通过https访问Horizon)

Guest Linux

zzz VNFVNFC1

GuestLinux

VNFC2

Guest Linux

LB

Guest Linux

IPSec API

Server

Host Linux

KVM/VMWare

AAL

Media APi

Algo

SRIOVGb

AAL

SRIOVCrypto

ACC

IPSec加速API

network

DPDK/ODP

OpenStack Agent

Neutron Agent

Nova Compute

Rock Agent

OpenStack

Neutron Server

Nova Scheduler Rock

Horizon

HACC API

OpenStack Agent

Neutron Agent

Nova Compute Rock Agent

DPI模块 IPSec模块IPSec IPSecMedia

xxx VNF

VNFC1

GuestLinux

VNFC2

Guest Linux

DPIAPI

LB

Guest Linux

Server

Host Linux

KVM/VMWare

HACC API

AAL

IPSecAPI

Algo

VirtioCrypto

VirtioGb

VirtioDPI

AAL

VirtioCrypto

ACC

MANO

(通过https访问Horizon)

Guest Linux

yyy VNF

VNFC1

GuestLinux

VNF2

Guest Linux

LB

Guest Linux

IPSec API

Server

Host Linux

KVM/VMWare

AAL

Gb GEA加速API

Algo

VirtioGb

API

VirtioCrypto

ACC

IPSecAPI

network

DPDK/ODP

OpenStack AgentNeutron Agent

Nova Compute Rock Agent

OpenStack

Neutron Server

Nova Scheduler Rock

Horizon

HACC API

OpenStack AgentNeutron Agent

Nova Compute Rock Agent

Virtio Ctypto Backend Virtio Gb BackendVirtio DPI Backend Virtio Ctypto Backend Virtio Gb Backend

Virtio Vring Virtio Vring

DPI IPSec GEA模块IPSec IPSec

SRIOV Virtio

12

Rock API• Portability between different accelerator platform.

– Software-hardware decoupling– FPGA, GPU, Intel Cards…

• ESTI NFV ：– Req PER/C1: The compute domain shall support discovery of hardware (HW) /functional accelerators.– Req PER/C2: The compute domain shall support reservation of HW/functional accelerator resources.– Req PER/C3: The compute domain shall support binding/attaching to HW/functional accelerators.– Req PER/C4: The compute domain shall support unbinding/detaching from the HW/functional accelerator.– Req PER/C5: The compute domain shall support collection of the HW functional performance metrics.

Rock

Accelerator Drivers

Orchestrator (MANO, Task Scheduling)

Part 1: Overview & use cases

Part 2: VNF

Interfaces

Part 3: Switching aspects

Part 4: Management

aspects Future work

13

Management: Southbound API for Accelerator Driversacc* acc_discovery(); Find the accelerators in the host according to the dirvers.

FPGA/GPU/Hardware Engine

void acc_get_status(acc*); Get the acceleration description inlcuding capacity, usage, affinity, available VF/queues.

vacc* acc_create(acc*, VM*); * Apply a VF for accelerator.void acc_destroy(vacc*); * Delete VFvoid acc_attach (vacc*); * Attach VF to VM.void acc_detach(vacc); * Detatch VF 。void acc_get_affinity(vacc*, acc_affi*);acc_affi* acc_set_affinity(vacc*);

Get the Affinity.

acc_pilicy *acc_get_policy(vacc *);acc_pilicy *acc_set_policy(vacc *);

Get the policy such as priority and preempt 。FPGA dynamic loading…

FPGA VF/queue partition … Multiple VFs can be connected with a type of acceleration logic.

Example: Accelerator descriptions (Driver -> Rock):acc_discovery (){"acc_type":"IPSEC_3DES", "acc_capability":{ {"num":"10","unit":"G*PPS"}, {"num":"800","unit":"bps"}}

14

Management: Northbound API for Nova/MANOvoid acc_register(acc*, network*); * Register the accelerator in a network created by Neutrion.void acc_unregiste(acc*, network*); * Unregister the accelerator. 。acc* acc_find_physical(acc_config*); Find the best accelerator for the configure.

* For VM

Accelerator requirement in VNF template (MANO -> Rock): "VIU":  {"init_number": 0, "cpu":4,"memory":30720,"cpu_frequency":2600, "disk":35, "ACC":{ "acc_type":"IPSEC_3DES",                 "acc_capability":{                            {"num":"10","unit":"PPS"},                             {"num":"800","unit":"bps"}                   }                    },