Telco Technology Office Techies:François DuthilleulFred HerrmannTimo JokiahoIgnacio Verona
Towards vRAN / Edge / 5GRed Hat EMEA Telco Vertical
February 2019
ZA Telco Account Team Edgar KamupiraGavin McDougall
#TTO #vRAN #Edge #5G2
● Introduction ● Red Hat Development Model >● Red Hat Position in Telco Networking Industry, Network Function Virtualization (NFV) >● Highlights of The Work Done in RHOSP During NFV Journey >● Overview of vRAN models >● Red Hat @ The Edge >● 5G, High Level >● Some Technical Requirements Spanning over vRAN / Edge / 5G >● Seamless Support for Virtual Machines and Containers >● Red Hat Openstack Platform @ The Edge / NFV >● Red Hat OpenShift Platform @ The Edge / NFV >● Red Hat participation in vRAN / Edge / 5G Industry Work >● Red Hat Telco Customers & Ecosystem >
Contents
#TTO #vRAN #Edge #5G3
EMEA Telco Technology Office (TTO)
Telecom ExpertiseRAN, EPC, IMS, 4G, 5G, CPE,...
NFV Transformation
ETSI NFV/MEC,LF Networking, O-RAN, TIP, ...
NFV InfraSR-IOV, OVS-DPDK, EPA, RT-KVM, Security, Multisite, Upgrade, Service Assurance,...
TTO
#TTO #vRAN #Edge #5G4
Much of this is enabled by Open Source
Software Defined Networking (SDN)
Network Functions Virtualization
5G networksEdge / vRAN
IoT, M-to-M, Mobile Applications
IPTV, DVR & intelligent Content Delivery Networks (vCDNs)
Telecom Industry is Investing
Red Hat Development Model
#TTO #vRAN #Edge #5G66
PARTICIPATEINTEGRATE(community platforms) (supported products
platforms, and solutions)
INTEGRATE(community platforms)
STABILIZE(supported productsplatforms, and solutions)
STABILIZE
From Community to Enterprise: Upstream FirstRed Hat Development Model
#TTO #vRAN #Edge #5G7
Red Hat model is staying on mainline
Branch and rebase
and rebase and rebase ...
Mainline and Forking
Branching Strategies
#TTO #vRAN #Edge #5G8
Red Hat Open Advantage
38,968 or 17%
2,052 or <1%
24,662 or 11%
OpenStack — Contribution Otaca Release
www.stackalytics.com
79,455 or 66%
35 <1%
1,335 or <1%
Ceph— Contribution Since Inception
www.metrics.ceph.com
1,639 or 12.1%
<100 or <1%
Linux Kernel 4.3 — Contribution by changesets
https://lwn.net/Articles/654633<100 or <1%
169 or 24%
<25 or <1%
KVM— Contribution by Employer
KVM Forum 2015– Keynote (Paolo Bonzini) <25 or <1%
4,330 or 14%
66 or <1%
209 or <1%
Kubernetes — Contribution Since Inception
www.stackalytics.com
1,214 or 4%
8 or <0.1%
21 or <0.1%
Docker — Contribution Since Inception
www.stackalytics.com
516 or 4%
12 or <1%
0
OVS (Open vSwitch) — Contribution Since Inception
www.stackalytics.com
798 or 9%
382 or 4%
182 or 1%
OPNFV — Contribution Since Inception
www.stackalytics.com
(RedHat + Ansible) – 8,242 or 32%
0
10 or <1%
Ansible — Contribution Since Inception
www.stackalytics.com
2,384 or 7%
OpenDaylight (ODL) — Contribution Since Inception
http://spectrometer.opendaylight.org<100 or <1%
<100 or <1%
Red Hat Posit ion in Telecom Networking Industry, NFV
#TTO #vRAN #Edge #5G
Networkscore | edge
Internal IT, OSS/ BSS
SE
CU
RIT
Y -A
UTO
MA
TIO
N -
CLO
UD
NA
TIV
E
VNF
Cloud Services
vApp
cAppCNF cApp
PNF
MANO
integrat ion
applicat ions
platforms
management
consumer
B2Bcustomer
Developer
Operat ions
Gov IoT Video Partner/ 3rd Party Services/ SaaS SE Mobile
Apps Web
SDN
Red Hat Portfolio @ Telco
10
#TTO #vRAN #Edge #5G11
● Revolutionizing the traditional building practice of Network Equipment and enabling the interchangeability / freedom of choice of Network Functions
● Building the path towards 5G innovative Network Services by enhancing QoS, Bandwidth & Latency for IoT, M2M, Content Delivery (e.g. VoD)
Business Motivation for NFVNetwork Functions V irtualization (NFV) is a network architecture concept that proposes using IT virtualization related technologies, to virtualize entire classes of network node functions into building blocks that may be connected, or chained, together to create communication services.
#TTO #vRAN #Edge #5G
RH Focus:Trusted BaseSW Platform Provider
Red Hat works with partners
Red Hat works with partners
KVMHypervisor
Red Hat Enterprise Linux as Host
RHELGuest
RHELGuest
Red HatStorage
DPDKOpen vSwitch
ODL ML2libvirt
JBossOpenShift(tooling)
Open & AgileOpen, Flexible, Modular,
Programmable
Single Platform for multiple use cases beyond NFV
Carrier Class Architecture
Performance, Throughput, Scale, Optimised for NFV
High Availability & Security
12
Red Hat on ETSI NFV ISG Reference Diagram
OtherGuest
#TTO #vRAN #Edge #5G Cert ified Red Hat OpenStack Platform plugins: https:/ / access.redhat.com/ art icles/ 1535373
IaaS+
IaaS
TELEMETRY ORCHESTRATION
TELEMETRY SAHARA HEAT
DATAPROCESSING
COMPUTE
NOVA
NETWORKING
NEUTRON IRONICCINDER GLANCE SWIFT
STORAGE
BLOCK IMAGE OBJECT
BARE-METALPROVISIONING
HORIZON TRIPLEO
DASHBOARD
SHARED SERVICES
IDENTITY
KEYSTONE
DIRECTOR
DEPLOYMENTand
MANAGEMENT
MANILA
SHARED FILESYSTEM SECRETS
BARBICAN
LOAD BALANCING
OCTAVIA
QEMU-KVM Libvirt DPDK
R ed Hat E nterprise Linux
Red Hat Openstack Platform (RHEL Always as a Host)
13
#TTO #vRAN #Edge #5G14
Windows Windows WindowsLinuxLinux
SUPPORTED GUESTS
OpenStack
RHEL + KVM Ceph OVS
Storage Network
SERVERS
Virtualization Security Ecosystem Network Storage
KVM Network Stack
Device Drivers
LINUX KERNEL
Security Enhanced Linux (SELinux)
A typical OpenStack cloud is made up of at least 10 coreservices + plugins to interact with 3rd party systems
Co-engineered with RHEL
#TTO #vRAN #Edge #5G15
Red Hat is the only company which is one of the leading contributor on all upstream projects, which constitute a consistent, high quality, high performing
NFVI / VIM platform based on Openstack.
Openstack Platform is a collection of about 10 Open Source projects, all having their own release schedules, governance models etc. To be able to put together a consistent Openstack Platform, you
need to understand all of these projects in detail not just Openstack
Integrating it All Together
Work Done for OpenStack During NFV Journey
#TTO #vRAN #Edge #5G17
PERFORMANCE AVAILABILITY SECURITY MANAGEABILITY LIFECYCLE
Enhanced Platform Awareness (EPA) SR-
IOV, OVS/DPDK, vCPU, NUMA pinning, Huge
pages, RT-KVM ...
High AvailabilityFault Tolerant Design
Enterprise Hardened CodeInstance Availability
End-to-EndSELinux sVirt
Neutron Security GroupsBlock EncryptionSSL/TLS on APIs
LoggingPerformance Monitoring
Operational Visibility Policy and Compliance
OpenStack LifecycleUpdates/Patches
--> Absolutely Relevant and Necessary in vRAN / Edge / 5G Space as well
Now we are doing the same things to make OpenShift NFV Ready !!!
To Make OPENSTACK NFV Ready, Development During Last Years
#TTO #vRAN #Edge #5G18
virtio0
DPDK kernel
OVS-DPDK
OpenStack APIs
compute node regular NICs
virtio1 VF1DPDKkernel
VF0
regular NICs
DHCP+PXE
Base (VNFs management: ssh,
SNMP, logs)
eth0 eth0
bondedbonded
bonded
VNFc0 VNFc1
fabric0: user traffic(provider network)
fabric1: user traffic(provider network)
bonddataplanedataplane
PF0 PF1
SR-IOV Host/VNFs guests resources partitioning
NFV Deployments Today: OVS-DPDK and/or SR-IOV
#TTO #vRAN #Edge #5G19
GLOBAL COMPLIANCE DRIVEN RESOLUTION OF SECURITY REQUIREMENTS
Compliance Driven Approach• The global nature of our OSP
deployment places a diverse set of security requirements on our product.
• Our approach here has been to analyze the top compliance init iat ives globally and then build a plan to address them in a cohesive and focused manner.
Compliance Driven Approach
Technical Security Controls Resolution
Solving for the “Big Three”
Focus is on: FedRAMP, ANSSI and ETSI• The technical control groups are similar
across the big three.
• Solving for these should get us 90% of everyone else.Top Technical Control Groups
Addressed• Infrastructure and Virtualization S ecurity• E ncryption and K ey Management• Identity and Access Management• Threat and Vulnerability Management
OpenStack Security Strategy
#TTO #vRAN #Edge #5G20
● Ansible is a simple yet powerful and
agentless automation framework.
● Also for NFV! Modules available for
multiple VNFs
● Easy integration via API to current
tooling and processes
Enabling DevOps and GitOps through the
complete organization (infra, QA, AppDev)Specific features for
Network Device
Management, like:
● SSH CLI
● Netconf
● IPAM plugins
Automation IS A Critical Part of NFV
#TTO #vRAN #Edge #5G
Every 6 months we release a version of Red Hat OpenStack Plat form supported for 1 year
E very 18 months we produce a “long life” version, which customers can opt to have support for up to 5 years
Long Life Long Life Long Life
RHOSP 8Liberty
RHOSP 9Mitaka
RHOSP 10Newton
RHOSP 11Ocata
RHOSP 12Pike
RHOSP 13Queens
RHOSP 14Rocky
RHOSP 15Stein
RHOSP 16Train
3 years 3 years 3 years(+2 years) 1 year 1 year 3 years
(+2 years)* 1 year 1 year 3 years(+2 years)*
RELEASEDJune 27,
2018
RELEASEDNovember 13,
2018
ON DEV
21
RHOSP Lifecycle And Support
Radio Access Network (RAN) CLOUDIFICATION
#TTO #vRAN #Edge #5G23 NOTE: No voice circuits or related elements shown, only packet data related
2/3G RAN
4G RAN
Digital Mobile Networks Evolution, 2/3G+4G
#TTO #vRAN #Edge #5G
Distributed RAN
http://xhaul.eu/wp-content/uploads/2015/10/workshop-FSAN-09-2015-atlanta-orange.pdf
Full base station functionality distributed on each antenna sites -> expensive, no scalability, radio interference control challenges, etc ...
BaseBand Units @ Antenna Site BaseBand Units @ Central Office (Cloud)
Part of base station functionality (BaseBand Units, BBU) centralized (base station split).
X2: “fast” handover protocol between base stationsD-RoF: Digital Radio over Fibre
Target for NFV, virtualization or
containers
High Level Overview On vRAN (4G / LTE)
Centralized RAN
24
#TTO #vRAN #Edge #5G
Virtual Machinesor
Containers
vBBU
Industry is driving to standardize this interface to have freedom to acquire RRH and vBBU from different vendors(dis-aggregation)
CoreNetwork
S1 Interface
High Level View to Dis-Aggregation (LTE/4G)
Target for NFV (virtualization or containers), consistent with Core
Network NFV deployment
25
#TTO #vRAN #Edge #5G26
ALTIOSTAR Presentation @ Telenor, Oslo June 8th, 2016Cloud-RAN (vRAN) and Application Intelligence (AI)
ETSI PoC #34 - SDN Enabled Virtual EPC Gateway
#TTO #vRAN #Edge #5G27
https://www.itu.int/dms_pub/itu-t/opb/tut/T-TUT-HOME-2018-PDF-E.pdf
Mapping of CU and DU functions according to the split points:- 5G(a): high layer split (F1)- 5G(b): low layer split (FX)- 5G(c): cascaded split
Option 2(3GPP)
6 7 8
ITU-T: Evolving From 4G To Split Function In 5G
#TTO #vRAN #Edge #5G28
CU ● Distributed Cloud, Low Footprint
DU ● Realtime, PTP, HW Acceleration (FPGA, Smart NICs)
RAN Requirements on Telco Cloud
+
RAN Cloudification ● vBBU (4G), vCU/vDU (5G)
#TTO #vRAN #Edge #5G29
Why does Red Hat Care ?• Enables virtualization of the BBUs → NFV implementations
of pooled functions• S plit strategies have significant impact on key system
aspects that may affect NFV infrastructure, such as:• Latency Const raints, including real-time
requirements (R T kernels, network latency)• Synchronizat ion requirements• Throughput requirements (performance Mpps &
Gbps, host bypass mechanisms, acceleration)• NFV-PoP dist r ibut ion requirements (distribution
strategy & number of nodes per site / network tier)• Reliabil it y & Availabil it y, Service Assurance
and QoS related requirements• HW interface and HW assist requirements
vRAN, Why Should I Care?
Why do Operators Care ?• Enables CapEx savings – e.g. through site
acquisition cost and function pooling related gains as well as simpler electronics on antenna site
• Enables OpEx savings – e.g. through site maintenance, spare part logistics on E dge DC and power related gains on cell site
• Advanced interference control techniques such as eICIC (enhanced Inter-Cell Interference Coordination) will benefit from the parallelism and increased processing power at the centralized BBU
• Carrier aggregation and CoMP (Coordinated Multipoint) will leverage the operational and spectrum efficiencies of resource sharing at the centralized processing unit
#TTO #vRAN #Edge #5G30
https://www.redhat.com/cms/managed-files/ve-virtualized-radio-access-network-altiostar-partner-solution-brief-f11705-201803-en.pdf
ALTIOSTAR - Red Hat Partnership
#TTO #vRAN #Edge #5G31
https://multi-rat--network-terminal-function-virtualization-02.webself.net/file/si230266/download/2.%20vRAN%20ETSI%20workshop%20De%20Lannoy-fi4516387.pdf
ALTIOSTAR Trial @ Orange Poland
#TTO #vRAN #Edge #5G32
Intel HW
Red Hat Enterprise Linux
Red Hat OpenStack Platform
OpenShift
vSBC (P-CSCF)vS/PGW
vPCRFvBBU vMME vTAS
vI/S-CSCFISC
MwMb
SGi / Mb
Gx Rx
S1-MME
S1-U S11
VoLTE End-to-End ArchitectureMWC 2018
vRAN, VNFs on VMs, Network Function on Container (CNF)
#TTO #vRAN #Edge #5G33
Rakuten Cloud Native Mobile Network
Earlier Press Release:https://rakuten.today/blog/rakutens-upcoming-end-to-end-cloud-native-mobile-network.html
Great video, where Rakuten CTO Tareq Amin speaks:https://www.youtube.com/watch?v=adr8k0cVSTQ
Most recent Press Release on Feb 12th, where the partners are disclosed, including Red Hat:https://global.rakuten.com/corp/news/press/2019/0212_06.html
34
Red Hat @ The EDGE
#TTO #vRAN #Edge #5G35
Telecommunications Manufacturing
Transportation Health - Life Science
+Retail, Public Sector and others
But Here We Talk About Telecom Networking
Edge Is Not Specific to Any Industry
#TTO #vRAN #Edge #5G
vBBU
Wifi/DAS
Ethernet or Fiber Fronthaul
Virtualized RAN: Virtualized BBU (LTE/4G), CU/DU (5G)
eNodeB
5G RAN
LTE/4G
Fiber / Coax
CU
Functional Splits
ePDG
DU
RU
RU
Fronthaul Midhaul
Backhaul
DU
LTE RAN
Regional or Core DCFor LTE EPC or NG-
Core (5G)Edge Compute
Edge Compute
Edge Use Case - vRAN for LTE and 5G
36
#TTO #vRAN #Edge #5G
Form Factors
Deployment
Workloads
➢ Enable multiple deployments of Red Hat platforms at the Edge
➢ Adapt Red Hat platforms to all Edge form factors
➢ Enable Red Hat platforms to run all kinds of Edge workloads
Red Hat Edge Strategy as Software Platform Vendor
37
#TTO #vRAN #Edge #5G
Option 1: independent clustersFully Distributed OpenStack
RC#1
(RT)
RC#2
(RT)
RC#≤
NR(
RT)
Remote Server Pool
EN#1
(RT)
EN#2
(RT)
EN##
NE(
RT)
EN#3
(RT)
Central Server Pool
Fronthaul(Network)
Mid | BackhaulNetwork
Option 2: remote computeCentralized OpenStack
Remote Server Pool
EN#1
(RT)
EN#2
(RT)
EN##
NE(
RT)
EN#3
(RT)
Central Server Pool
Fronthaul(Network)
Mid | BackhaulNetwork
≤100us≤20km(CPRI,eCPRI)
≤2..5ms≤200km
(typ. Req)
Option 3: hybrid of 1 and 2
[SMALL] Remote Server Pool
EN#1
(RT)
EN#2
(RT)
EN##
NE(
RT)
EN#3
(RT)
Central Server Pool
Fronthaul(Network)
[LARGE] Remote Server Pool
Mid | BackhaulNetwork
RC#1
(RT)
RC#2
(RT)
RC#≤
NR(
RT)
RC#1
(RT)
RC#2
(RT)
RC#≤
NR(
RT)
RC#1
(RT)
RC#2
(RT)
RC#≤
NR(
RT)
OpenStack & OpenShift @ The Edge
38
RHOSPFullRHOSP
DCN
RHOSPFull
RHOSPFull
RHOSPFull
RHOSPFull
RHOSPDCN
5G, High Level
#TTO #vRAN #Edge #5G
Some Fundamental Use CasesARTIFICIAL
INTELLIGENCE
VIRTUAL RAN(vRAN)
ENTERPRISE EDGE
AR / VRMEDICINE
ENHANCED MOBILE BROADBAND
MASSIVE MACHINE TO MACHINE
CONNECTED DRONES
ULTRA RELIABLELOW LATENCY COMMS
Building ultra-reliable immersive experiences for people and objects when and where it matters
5G Will Transform Our Lives
40
#TTO #vRAN #Edge #5G
Enhanced Mobile Broadband
Massive Machine Type Communications
Ultra-Reliable Low Latency Communications
Differing Requirements for 5G Use Cases Enhanced Mobile Broadband:User Experienced Data Rate, Peak data rate, Spectral efficiency, Traffic density, Network energy Efficiency, MobilityMassive Machine Type Communications: Connection Density, Low Cost, Low Power ConsumptionUltra-Reliable Low Latency Communications:Latency, Reliability, Mobility
Three Fundamental Use Cases of 5G
41
#TTO #vRAN #Edge #5G © 2016 Affirmed Networks, Inc. All rights reserved.
Pre-Paid ConsumerPost-Paid Consumer
Fixed Wireless Access –Residential Fixed Wireless Access- Small Business
Consumer MVNO 1
Car ManufacturerFarm Conglomerate
Mining Company
Federal Government Devices
Smart City 1
Public Safety Video Surveillance Self Driving Cars
Mission Critical Application-Health Industrial Automation
Augmented Reality
Cloud Native Architecture allows for a low “minimum cost of entry” per slice –
Ability to cost effectively scale to thousands of slices of all
different sizes
LLC/URLLC
IoT/m-IoT
MBB/eMBB
Micro-Segmentation With Slicing
42
#TTO #vRAN #Edge #5G
● A given NF may be deployed closer to the edge depending on network slice requirements ⇒Importance of homogeneous NF execution environments
● Network Slice LCM (multi-site) and isolation (QoS, Resource Allocation) requirements
5G Network Slicing - Including Radio Interface
43 https://www.ericsson.com/assets/local/publications/white-papers/wp-5g-systems.pdf
#TTO #vRAN #Edge #5G
5G User Plane Function (UPF)
China Unicom @ OpenStack Summit 2018, Berlin
Same Function at Different Locations
44
#TTO #vRAN #Edge #5G
eNB
MME
SGW PGW
HLR
HSS
OCF
PCRF
CP CP
DP DP
Box / Device centricLTE/4G
5G RAN4G
RAN Localized GW or Central GW Data Plane
Control Plane – Mobility, Sessions & Service Management
AUSF AFPCF
5G - Cloud Based
OpenStack or KVM OpenStack
vBBU vMME vSGW vPGW
vPCRFvHSS/HLR
• CP-DP Separation• UPF is controlled by
AMF and SMF• Data plane extensibility
45
Mobile Packet Core Evolution Towards 5G
#TTO #vRAN #Edge #5G
API Management
Security
Cloud Native / Containers
AF Application FunctionAMF Access and Mobility Management FunctionAUSF Authentication Server FunctionDN Data NetworkFE Front EndNEF Network Exposure FunctionNRF Network Repository FunctionNSSF Network Slice Selection FunctionPCF Policy Control Function(R)AN (Radio) Access NetworkSEPP Security Edge Protection ProxySMF Session Management FunctionUDM Unified Data RepositoryUDR Unstructured Data Storage FunctionUE User EquipmentUPF User Plane Function
46
5G - Service Based Architecture (SBA)
Some Technical Requirements
#TTO #vRAN #Edge #5G
Latency
consumer ⇔ service
RTT ≈1ms ●
distance ≈100km
Bandwidth
funnel of 1M’s of 1Gb/s connections
Resilience
make edge sites autonomous, minimize failure domain to 1 site
Regulations
keep sensitive data on-site / within regulatory region
Distribution
need to deploy and perform the lifecycle
management of distributed systems
Scale
need to manage 10-100k’s of sites with 1-10’s of nodes
each
Edge - 5G Constraints
48
#TTO #vRAN #Edge #5G
RRC PDCP High-RLC
Low-RLC
High-MAC
Low-MAC
High-PHY
Low-PHY RF
RFRRC PDCP High-RLC
Low-RLC
High-MAC
Low-MAC
High-PHY
Low-PHY
EPC nGCEdge
UEs
“0” 1
R15 R15
DL
ULTotal RAN Latency (transmission + scheduling+ processing switching/queuing + fronthaul + midhaul + etc.)
Service specific targets:• eMBB: Enable 10-50ms RTT• URLLC: Enable 2-5ms RTT
MAC HARQ loop (3TTI-FH TWL)
2: PDCP-RLCLTE: 30 SCF.. 10ms5G FWA FH: 5-10ms
5: Split MACLTE: 6msSCF
5G FWA FH: 1-3ms
6: MAC-PHYLTE: 2msSCF
5G FWA FH: 100-500us
7-*: Split PHYLTE: .25-2ms5G FWA FH: 50us
8: BBU-RRH LTE: 250usSCF.. 100usCPRI
5G FWA FH: 50us
4: RLC-MACLTE: ?5G FWA FH: 1-3ms
3: Split RLCLTE: ? 5G FWA FH: 5-10ms
RAN targets [IMT2020]:• eMBB: ≤8ms RTT• URLLC: ≤2ms RTT• CP: ≤10ms3GPP
[CP time is for idle-act, multiple steps !]
Transport targets:• CPRI: 200us RTT• eCPRI: CoS “high” 100 us; CoS “med” 1ms• NGFI: TBD• 802.1CM: TBD (so far aligned with eCPRI)• SCF: 6ms (sub-ideal), 2ms (near ideal), 250us (Ideal)
Budgets split primarily to:• processing times, and• transport allocations• tradeoff, “flexible”…
Splits, Latency Budgets and Allocation Targets
49
#TTO #vRAN #Edge #5G
● DU, CU and vBBU – Containers attractive to deliver control functions and may be data plane functions in future
● 5G NG-Core - Separate Control and User Plane allows flexible deployment of functions○ => Orchestration models needed to place functions ○ => Granularity functions or services for flexibility – re-usability
● Support for Edge Computing and efficient UPF re-selection/redirection○ => Common orchestration of core and other functions○ => Reusability of components
● Support Network Slicing based on modular design and multi slice connectivity from UEs○ => Resource partitioning from air interface to edge to core
● Stateless functions – compute decoupled from storage● OTT Applications and offload
Red Hat is preparing the infrastructure so that these services can be delivered from it
vRAN and NG-Core - LTE+ and 5G
50
#TTO #vRAN #Edge #5G
● Real Time requirements○ Latency sensitive => 1ms processing budget at DU node
● Timing synchronization○ vRAN applications require host and VM time synchronization with <1us accuracy
● Accelerator support○ Data plane accelerators for Split RAN architecture => FPGA and/or Smart NIC
● High throughput○ Dedicated CPU and thread pinning and other performance tuning
5G Requirements
51
#TTO #vRAN #Edge #5G
● More cell sites and Edge compute => massive scale○ Distribution of compute => HA models○ Ex: 1.2M servers required for Edge compute and vRAN
● Massive Scale => Orchestration challenges○ Provisioning and life cycle management tools
● Hybrid Workloads○ Virtual Machines or Containers○ Any packaging model
● Storage Management○ At Edge and Core locations
5G Requirements
52
#TTO #vRAN #Edge #5G
● Massive Distribution of sites – Scale
● Real time requirements & Hybrid Container/VM Models
● Separate Control and User Plane allows flexible deployment of functions
● Support for Edge computing with efficient UPF re-selection/redirection
● Support Network Slicing based on modular design and multi slice connectivity from UEs
● Stateless functions – compute decoupled from storage
5G Requirements, Summary
53
Support for Both Virtual Machines and Containers
#TTO #vRAN #Edge #5G55
Red Hat OpenShift Container Platform
● NFV started with VMs and is now evolving towards cloud native Network Functions based on containers
● Kubernetes is the leading Container Management Platform
● Red Hat OpenShift Container Platform (OCP) is built around Kubernetes
Seamless Transition To Containers
#TTO #vRAN #Edge #5G56
with continued support of Virtual Machines
RED HAT OPENSTACK PLATFORM 13
OPENSHIFT CONTAINER PLATFORM 3.11
STANDARD HARDWARE
KVM Ironic (Bare Metal)
VMVM VM
App
App Networking
Networking
Kuryr*Octavia
App
App App App App App
Red Hat CloudForms
Red Hat Satellite
Red Hat OpenStack Platform Director
SHARED MANAGEMENT TOOLING
Red Hat Ansible Automation
Cinder
Containers Containers CinderIntegration
Compute Ceph
Seamless Transition To Containers
ObjectStorage
Registry
* Tech Preview
#TTO #vRAN #Edge #5G
NETWORKING ANSIBLE NEUTRON ML2 PLUG-IN (GA in OSP 14)
57
Provisioning Network is configured in the switch
Boot BM on Tenant Network
ML2 Plug-in Configures
Switch
BM is Provisioned
ML2 Plug-in Configures
Switch
Tenant Network is configured in the switch
BM is ready
New Generic ML2 plug-in for bare metal multi-tenant based on Ansible Networking
L2 Switch
BMNIC
BMNIC
Multi Tenant Bare Metal as a Service - Ironic
https://blogs.rdoproject.org/2018/09/networking-ansible/
#TTO #vRAN #Edge #5G
OpenShif t Clust er
Master Node
Infra Node
Worker NodeDeploy OpenShift
RHOSP with Ironic
12
3
Deploy OpenStack with Ironic
OpenShift on Bare MetalDeploy OpenShift on RHOSP Ironic-managed bare metal nodes
RHOSP Directorintegrates openshif t -ansible
58
#TTO #vRAN #Edge #5G59
Example of OpenShift Usage ...
#TTO #vRAN #Edge #5G60
http://images.engage.redhat.com/Web/RedHat/%7B17ba2edd-411d-4f72-a247-5c63416a6716%7D_OpenShift_OpenS tack_experiences_@_Proximus.pdf
Operators Are Experimenting ...
Red Hat OpenStack Platform @ The Edge / NFV
#TTO #vRAN #Edge #5G62
● RHOSP 13 is already fit for some vRAN deployments○ Hyperconverged deployments + Virtualized Control Plane○ Real Time KVM○ Basic FPGA support with PCI passthrough○ Several PoCs and Trials performed with vBBU VNF partners on RHOSP
Red Hat OpenStack Platform @ The Edge
#TTO #vRAN #Edge #5G63
Problem statement:● System calls and Interrupts introduce spikes
that impact the average scheduling latency
Solution:● New Real Time Compute Role that introduce deterministic and low average scheduling
latency● Max scheduling latency is 33us, 24 hours latency testing with RHEL 7.5 (cyclictest) ● PTP up to the VM (very accurate clock synchronization): required by 5G and Terrestrial
satellite stations
Red Hat Openstack / NFV @ The Edge, Realtime KVM
#TTO #vRAN #Edge #5G64
● RHOSP 13 is already fit for some some vRAN deployments○ Hyperconverged deployments + V irtualized Control Plane○ Real Time KVM○ Basic FPGA support with PCI passthrough○ Several PoCs and Trials performed with vBBU VNF partners on RHOSP
● RHOSP roadmap is addressing additional vRAN requirements such as○ Distributed Compute Nodes○ Enhanced FPGA support (Cyborg)○ Precision Time Protocol (PTP) support
Red Hat OpenStack Platform @ The Edge
#TTO #vRAN #Edge #5G
Stand-AloneLarge Clusters
Resource Node
Resource Node
Resource Node All-in-One
Small Clusters Distributed Nodes
Control Plane
Control Plane
Control Plane
Site A
Resource Node
Resource Node
Resource Node
Resource Node
Resource Node
Resource Node
Resource Node
Resource Node
Resource Node
Resource Node
Resource Node
...
Site E
Site F
Site G
Control Plane
Control Plane
Resource Node
Control Plane
Resource Node
typically 1 rack
Site B
Site I
OSP Director
RHOSP 15 Goal - Tech Preview
RHOSP @ The Edge - Deployment
65
#TTO #vRAN #Edge #5G
OSP 16 Goal (15 Tech Prev)
Characteristics:
● Distributed HCI nodes● Min of 3 HCI nodes per site● Multiple Ceph Clusters● Cells & Location awareness● Deployment of BM at scale
OSP Director
Site B
Control Plane
Control Plane
Resource Node
Control Plane
Resource Node
typically 1 rack
Site A
Site C Site D
spine & leaf
at scale
RHOSP @ The Edge - Distributed Compute Nodes
66
#TTO #vRAN #Edge #5G
● Life cycle management for accelerators● GPUs, FPGAs, ASICs, NPUs, … ● Vendor-Neutral, Hypervisor-Neutral● https://wiki.openstack.org/wiki/Cyborg
OpenStack - Cyborg Service
Key Services:● Nova Placement:
○ Compute Node = Resource Provider○ vCPUs, memory = Resource Classes○ Properties = Traits
● Cyborg: Devices contain Nested Resource Providers○ Accelerators: Resource Classes (E.g. ACCELERATOR_GPU)○ Properties: Traits
● User ask: 4 vCPUs, 12 GB RAM, FPGA with specific properties
67
#TTO #vRAN #Edge #5G68
https://zshisite.wordpress.com/2017/10/25/sync-your-cloud-with-ptp/
● Host: PTP supported in RHEL (linuxptp)○ Boundary and ordinary clocks○ PTP Hardware Clock on NIC
● Guest:○ PTP in Guest OS with hardware support
via PCI passthrough/SR-IOV
or○ vPTP support in KVM (kvm_ptp)
Precision Time Protocol - PTP
Red Hat OpenShift Platform @ The Edge / NFV
#TTO #vRAN #Edge #5G
● Fast Path for High-Throughput I/O○ SR-IOV○ DPDK support○ QoS○ Overlay/Non-Overlay
● Flexible Network Management○ Multiple Networks / Interfaces○ BYO IP address management○ Discontiguous IP address blocks
● Multicast● IPAM● Encrypted tunnels● Egress firewall● IPv6 / DHCPv6● Isolation / Multi-tenancy● ACL / Security Groups
70
● Kubernetes community is busy addressing basic NFV gaps○ Enhance platform awareness, multiple network interfaces,...○ Only recently created a new IoT/Edge working group
https://github.com/kubernetes/community/tree/master/wg-iot-edge
Critical Networking Feature For NFV Workloads
#TTO #vRAN #Edge #5G71
Feature(s): HugePages, CPU Manager, Device Manager
Description: We spoke about Device Manager here. CPU Manager Policy allows you to tell kube that your workload requires an affinity to a CPU core. Maybe your workload needs CPU cache affinity and can’t handle being bounced around to different CPU cores on the node via normal fair share scheduling on linux. HugePages allows you to request that your workload consume a specific amount of HugePages.
Performance Pods
How it Works: CPU manager is a flag on the kubelet that has the option of none or static. Static will cause guaranteed QoS pod access to exclusive CPU cores on the node. HugePages is a flag you set to true on the master and kubelet. The nodes will then be able to tell if any HugePages are available and workloads can request them via the pod definition.
ubelet device manager
CPU Manager Policy
# cat /etc/origin/node/node-config.yaml...kubeletArguments:...feature-gates:- CPUManager=truecpu-manager-policy:- staticcpu-manager-reconcile-period:- 5skube-reserved:- cpu=500m
Result:
# oc exec pod-name -- cat /sys/fs/cgroup/cpuset/cpuset.cpus2# oc exec pod-name -- grep ^Cpus_allowed_list /proc/1/statusCpus_allowed_list: 2
HugePages
# cat /etc/origin/node/node-config.yaml...kubeletArguments:...feature-gates:- HugePages=true
Pod spec:
resources:requests:cpu: 1memory: 256Mi
limits:cpu: 1memory: 256Mi
# cat /etc/origin/master/master-config.yaml...kubernetesMasterConfig:apiServerArguments:...feature-gates:- HugePages=true
Pod spec:
resources:limits:
hugepages-2Mi: 100Mi
Both the variable name and value are configurable.
OCP3.10
https://www.redhat.com/en/blog/paving-way-intelligent-and-performance-sensitive-applications-kubernetes-red-hat-openshift-container-platform-310
#TTO #vRAN #Edge #5G
Container Network Interface (CNI)
● Problem: Kubernetes only supports one network interface, “eth0”, but we need:○ Functional separation of control/data planes○ Link aggregation for network redundancy○ Different network protocol stacks, capabilities, SLAs○ Traffic isolation / Network segregation and security○ QoS
● Solution: Multus “meta plug-in” for Kubernetes CNI● Enables multiple network interfaces per pod, each assigned a
different CNI plug-in defined in pod spec○ Each with its configuration defined in CRD objects
● SR-IOV enablement
Podeth0 eth1
Pod with Multus
openshif t -sdn CNI plug-in
SR-IOVCNI plug-in
Multus
CNI
...
Kubernetes
OCP4.0
72
#TTO #vRAN #Edge #5G
● Method of packaging, deploying and managing applicat ions
● Encode human operat ional knowledge● Automat ically patch, upgrade,
recover, and tune apps and services● K ubernetes-native● Purpose-built for a specific application
or service
S DKLIFE CYCLE
MANAGE ME NTME TE R ING
ht tps:/ / github.com/ operator-framework
Kubernetes Operators
73
#TTO #vRAN #Edge #5G
APPLICATIONS AND SERVICESISV OperatorsCustom Operators (built w/ Operator SDK)
PLATFORM AND CLUSTER MANAGEMENTAutomated updates for Kubernetes, monitoring, security, registry and more
LINUX HOST Atomic, over-the-air updates for Red Hat CoreOS
ANY INFRASTRUCTURE
* coming soon
Full-Stack Automated Operations in OpenShift*
74
#TTO #vRAN #Edge #5G
60+ Cert if ied ISV Operators in Red Hat Early Access Program
ISV Operators
75
#TTO #vRAN #Edge #5G
Kubernetes Cluster APIs
76
● Brings declarative, Kubernetes-style APIs for cluster creation, configuration & management.
● Implemented using Kubernetes Operators to scale, upgrade, heal, etc. clusters in fully automated, provider-agnostic manner.
● Integrates with provider-specific Machine Controllers and Machine Config Daemons for cloud, virt. & bare metal environments.
● Manage your clusters consistently independently of where/how it’s deployed.
● New working group under Kubernetes Cluster Lifecycle S IG
https://kubernetes-sigs.github.io/cluster-api/
#TTO #vRAN #Edge #5G77
https://www.openstack.org/assets/presentation-media/5G-Cloud-Native-v4.pdfhttps://www.youtube.com/watch?v=pCkzJ bkQYLE
Cloud Native 5G Core - Affirmed
This is a public example of the work Red Hat does with its partners to push Cloud Nativeness into telco networks
#TTO #vRAN #Edge #5G
CONTAINER INFRASTRUCTURE AND ORCHESTRATIONContainer Applicat ion and Kubernetes orchestrat ion as provided by OpenShift are becoming the standard for new applicat ions.
VIRTUALIZED WORKLOADSVirtualized Workloads are not going anywhere fast! Business reasons (cost, t ime to market) and technical reasons (older/ different operat ing system)
CONVERGING INFRASTRUCTUREHow can we bring these two worlds closer together?
What About Existing Workloads?
78
#TTO #vRAN #Edge #5G
Technology enabling developer use of OpenShift Container Platform as a unified platform for building, modifying, and deploying applications residing in both containers and virtual
machines in a common, shared environment.
Add virtual machines to your OpenShift projects directly from the service catalog!
What Is Container Native Virtualization?
79
#TTO #vRAN #Edge #5G
Container
Virtual Machine
ENTERPRiSE CONTAINER HOST
Physical Machine
Container
Resultant virtual machines are able to run side by side directly on the same OpenShift nodes as application containers.
{Pods
Together At Last*
80
* Dev Preview
#TTO #vRAN #Edge #5G
● Integrates direct ly into exist ing Kubernetes clusters
● Uses a k8s-nat ive approach whenever possible
● Leverage Container Networking Interface (CNI),
Container Storage Interface (CSI), and other k8s-
nat ive integrat ions
Operat ing System
Kubernetes
Physical Machine
VM Pod Regular Pod
http://kubevirt.io
KubeVirt: The CNV Upstream
81
#TTO #vRAN #Edge #5G82
VF12
flannel, calico, (optional) istio, ..
25 Gbps interface
Web App pod
serviceeth0
VF11
vSwitch
Network Function pod
serviceVF10 virtio eth0
VF13
vSwitch
KubeVirt pod
Port FWDvirtioeth0 VF14
Virtual Machine
k8s VF14virtioservice
Network Plumbing for All Use Cases (WIP)...
Virtual Machine
pod, hosting one or more containers
Red Hat And Industry Work
#TTO #vRAN #Edge #5G84
Open Source Communit ies
Red Hat is Plat inum sponsor and key contributor to many upstream projects used in or around ONAP (Ansible, Drools, OpenDaylight, Kubernetes, OpenStack, Linux); Edge architecture contributor.
Red Hat is plat inum sponsor, Board and TSC member; act ive in mult iple projects, with driving role in Edge use case & requirements analysis and VCO 1.0/ 2.0 PoCs.
Red Hat is leading OpenStack contributor, act ively part icipat ing in Edge WG to identify gaps, address in relevant OpenStack projects.
Red Hat is contributor to many IoT-related projects, with focus on Linux, cloud platform / infrastructure, and messaging technologies.Red Hat is a leading Kubernetes
contributor, init iated and leads IoT/ Edge WG to identify gaps, address in relevant Kubernetes SIGs.
Red Hat is Board and TSC member; tech lead for the Kubernetes-Native Infrastructure Blueprint Family
IoT Use Cases
Telco Use Cases (e.g. VCO, vRAN)
Edge Community Involvement
#TTO #vRAN #Edge #5G85
Standards Development Organizat ions (SDO’s) and Industry Alliances
Red Hat is:• NFV ISG member• MEC ISG member• ZSM ISG member
Red Hat is:• virtual RAN fronthaul project member• Open RAN project member• Edge Computing project member
Red Hat is:• act ively contribut ing to ARM-enablement• sponsor of and lead contributor to Fedora IoT,
Linaro EDGE’s reference platform
Red Hat is:• ORAN Member since opened to vendors (Dec 2018) • Focus on WG6 (cloudificat ion & Virtualizat ion) WG• Was previously member of CMCC’s CRAN consort ium which was
merged with xRAN.org to form O-RAN
Edge Community Involvement
#TTO #vRAN #Edge #5G
https://www.o-ran.org/resources/
WP6: Cloudification and MANO enhancements (AT&T, Orange)
O-RAN Alliance Reference Architecture
86
#TTO #vRAN #Edge #5G87
Akraino Edge Stack is an open source software stack that improves the state of edge cloud infrastructure for carrier, provider, and IoT networks. https://www.akraino.org/
Akraino Edge Stack
#TTO #vRAN #Edge #5G
Red Hat is project lead of Akraino’s “Kubernetes-Native Infrastructure for Edge” (KNI-Edge) blueprint family. It’s blueprints
● implement the Kubernetes Cluster API for declarativecluster deployment, configuration, and lifecycle manage-ment in a provider-agnostic manner,
● leverage the Operator Framework for automated andsecure edge stack lifecycle management, and
● optimize for Kubernetes-native container workloads, butallow mixing in VM workloads via KubeVirt as needed.
Edge sites are deployed and upgraded in a rolling manner through a logically centralized CI/CD system (both software and configuration changes).
...
App1Op App2Op App2Op ...
baremetalprovider
OpenStackprovider
AWSprovider
...
Machine & Machine Config APIs / Operators
Cluster API / Operator
Akraino KNI-Edge Blueprint Family
88
#TTO #vRAN #Edge #5G
IoTGATEWAY
EDGE
CONNECTED “THINGS”
Edge analytics
Machine learning
Telemetry
Telemetry
Management
IoT INTEGRATION
HUB
Fedora Internet of Things Edition
IoT GATEWAY EDGE IoT DEVICE EDGE PROJECTS
PRODUCTS
• HW enablement (e.g. ARM)• OSTree (updates)• Ignit ion (enrollment)• Ansible (management)• Kubernetes (containerizat ion)
• Distributed Computing and Edge SKUs
• Containerized applicat ions• Centralized managementIoT messaging
89
IoT Gateway And Device Edge
Red Hat Telco Customers & Ecosystem
#TTO #vRAN #Edge #5G
SOME OF OUR TELCO CUSTOMERS
● Logos above are only a subset of the public Telco references (OpenStack & OpenShift)● It can cover NFV but also Internal IT or Public Clouds; some through Partners (NEP/SI)● Telcos are in prod with OpenStack; experimenting OpenShift for NFV● Some Telcos are in prod with OpenShift for non-NFV offers
...
91
#TTO #vRAN #Edge #5G
Management &Orchestration
(MANO)
Compute Storage Network• OSP Director• CloudForms• Satellite
vDDoS vRoutervCPE vCDNvEPC vMessagingvPCEF/RFvLB vADC vIMSvSBCClassifier
SMS
MM
SipM
essaging
Various NEP VNFs
Red Hat NFV Partner Ecosystem Representative
NFV Infrastructure (NFVI) + Virtualized Infrastructure Manager (VIM)
#TTO #vRAN #Edge #5G93 Red Hat Confidential
NEP Ecosystem
#TTO #vRAN #Edge #5G94
● Supportability - Partner supported in production for VNF and datapath (SR-IOV, OVS, OVS-DPDK)● Install and Inservice update - verify end-to-end install with RHOSP; verify “yum update” works flawlessly; ● Security - use the right drivers for secure VM-to-NIC, VM-to-VM communication● Tools are certification tests run by the partner, and then reviewed by Red Hat
○ Step one: architectural review for the VNF by PM/Partner team ○ Step two: Workflow guide https://access.redhat.com/documentation/en-us/red_hat_openstack_certification/1.0/html/red_hat_openstack_application_and_vnf_workflow_guide/
● Red Hat Openstack Platform application and VNF Policy Guide:https://access.redhat.com/documentation/en-us/red_hat_openstack_certification/1.0/html/red_hat_openstack_application_and_vnf_policy_guide/
Partner needs to recertify with each long lifecycle RHOSP release and major release of RHEL● Provide early content and ask partner to validate at least 1 month before GA through High Touch Beta● VNF lab upcoming - Automated certification/validation process with every release, partner encouraged to participate● Find certified or validated VNFs : Certified/Validated VNFs● RHEL Network Adapter support https://access.redhat.com/articles/1390483● Hardware certification: https://access.redhat.com/ecosystem/search/#/ecosystem/Red%20Hat%20OpenStack%20Platform?category=Server
Additional information for planning purposes● Network Functions Virtualization Planning and Prerequisites Guide: https://access.redhat.com/documentation/en-
us/red_hat_openstack_platform/12/html/network_functions_virtualization_planning_and_configuration_guide/
● Networking Guide for RHOSP13: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/networking_guide/
Value Of Certification / Validation / Integration
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews