TrellisAnOpen-Source,White-Box,SDNBasedLeaf-SpineFabric

CharlesChan,Ph.D.

Outline

•  WhatisTrellis?•  Trellisfeatures•  WhyTrellis?•  Recentactivities(in2018)•  Deployments&usecases•  Ecosystem&roadmap

WhatisTrellis?Production-readymulti-purposeleaf-spinefabricdesignedforNFV WhiteBox

Hardware

OpenSource Software

SDNBased

TrellisFullTopology

FieldOffice(2ndstage) CentralOffice(1ststage)

Spine

Leaf

Spine

PairedLeaves

Quagga

UpstreamRouter

AccessDevice

DHCPServerSinglehomedhosts

DualhomedhostsMetroNetwork

WayTooComplicated?

Trellissupportssimplerconfigurationstoo!

SingleLeaf

SingleLeafPair

SingleLeafPairwithUpstream

SingleStage

SingleStagewithLeafPairs&Upstream

TrellisinProduction

TrellisFeatures●  BridgingwithAccess&TrunkVLANs(withinarack)●  Routing(inter-rack)

○  IPv4&IPv6UnicastroutingwithMPLSSegment-Routing○  IPv4&IPv6Multicastrouting

●  Dual-homingforcompute-nodesandexternalrouters●  Multi-stagefabrics(2layersofspines)●  vRouter-entirefabricbehavesasasinglerouter

○  BGP(v4/v6)supportforexternalconnectivity○  Staticroutes,routeblackholing

●  DHCPL3relay(v4/v6)●  MPLSPseudowire●  Q-in-Qtermination●  T3(TrellisTroubleshootingTool),onos-diags

TrellisFeatures/Bridging&Routing

Bridging

Routing

PopMPLSLabel

PushMPLSLabel

TrellisFeatures/Multicast(1/3)

Single-homedsource,single-homedsink

TrellisFeatures/Multicast(2/3)

Single-homedsource,dual-homedsink

TrellisFeatures/Multicast(3/3)

Dual-homedsource,dual-homedsink

TrellisFeatures/Dual-Homing

TrellisFeatures/Dual-Homing/Failure(1/2)

Pairlinkisonlyusedtorecoverlocalfailure

TrellisFeatures/Dual-Homing/Failure(2/2)

TrellisFeatures/Multi-Stage

RetainMPLSLabel

PopMPLSLabel

PushMPLSLabel

TrellisFeatures/vRouter

BGPv4/v6session

Dataplanetraffic

FPMOF/P4RT

vRouter•  Control/dataseparation•  Entirefabricasabigrouter•  Alsosupportsstaticroutesandrouteblackholing

TrellisFeatures/DHCPL3Relay

TrellisFeatures/Pseudowire

TrellisFeatures/Q-in-QTermination

Double-taggedpacket

Untaggedpacket

Push/popdouble-tag

Workswithsingleswitchtoo

VariousASIC/VendorSupport●  BroadcomQumran,Tomahawk,Trident2switchesfromEdgeCore,QCT,

Delta,Inventec(WIP)●  PreliminarysupportP4-basedTofinoswitchesfromEdgeCore,Delta,

Inventec●  PreliminarysupportforMellanoxSpectrumswitches●  PreliminarysupportforCaviumXpliantswitches

Deployments&UseCases

•  TrellisinComcast-  UtilizesalmostallTrellisfeatures-  TrellisisinstalledandmonitoredbyKubernetes-  Integrateswithin-houseVNFs,logging,

telemetryandalarmsystems•  TrellisinCORD/SEBA

-  Utilizesbridging,routing,multicast,crossconnect,dualhoming

-  L2loadbalance(WIP)

WhyTrellis?

•  TrellisisdesignedforServiceProviders&NFV

•  SDNallowssimpler/easier/optimizedfeatures

•  SDN+Programmablepipelines->Newfeatures

•  Open-source->SPownership&customizability

RecentActivities(in2018)

ProductionReadiness

•  SupportComcastdesign/dev/QA/opsteams-issueanalysis;root-cause;

•  Recommendbest-practices/training

•  Designdiscussionsfornewfeatures&architecturalimprovements

•  Dailyscrum(-May‘18)•  Documentation

SupportComcast

Support Otherteams

•  Broadcom•  Nokia•  Harmonic

Deliver New

Features •  Pseudowiresforin-

bandcontrol•  RoutinginH-Agg

basedtopologies•  Multicast

improvements•  Dual-homing

improvements•  DHCPv4/v6Relay•  IPv6RouterAdv.•  ISSUarchitectural

discussions/progress•  Othersmallfeatures

ONOSStability&

Scale •  Focusonstabilityof

ONOSdistributedstores(9releasesofAtomixin4months)

•  Scaleinvestigationongoing

Tooling

•  T3-TrellisTroubleshootingTool

•  onos-diags-Diagnosticscollectiontool

•  Mininetscripts-Software-emulatedtestenvironment

QA&BugFixes

•  Developingautomatedfeaturetests(220newtestsinthe4months)

•  Extendingframeworkforhardwarebasedtests

•  180ticketsresolved

December2017-November2018

Stratum/P4Integration

Trellis

Stratum

ASIC

OFagent

ASIC

OpenFlow(OF-DPA)

ONOSNBAPI

ThinSwitch

Stratum

ASIC

ONOS

LocalCtrl

P4RuntimegNMI,gNOI

Hybrid

VNFs

VNF VNF

Server

Network Config

Pipeline Config

Controller

Switch

Ecosystem&Roadmap

DistributedDevOpsModel

•  Reportissuesfound;~75cases•  Createautomatedteststoreproduce•  Validatefixes/releases•  Helpwithissuesintheirsetup•  Designdiscussionsfornewfeatures•  Dailyscrum

•  Deliverfeatures;meetopsneeds•  Deliverstability,scale,perf,tools•  SupportComcastdesign/dev/QAteams–issue

analysis;root-cause;recommendbest-practices/training

•  Designdiscussionsfornewfeatures•  Dailyscrum•  Documentation

•  AnalyzeissuesreportedbyHarmonicintheirsetups

•  CodereviewpatchestheysubmittoONOS•  Designdiscussionsfornewfeatures•  BringingupONFpodhousedinSanJose

•  BringingupONFpodhostedbyFlex•  QAcollaboration

•  Reportissuesfoundinhw•  ValidateT2versionsofswitch

softwareEdgeCorebuildsforus

•  Designdiscussionsforapps/featurescreatedbyNokia

•  Codereviewsubmittedpatches(40changesets;4-5patchsets/change)

•  Validatesomefeatures;reportissues

andmore…

Roadmap•  Scale&Performanceimprovements•  DualhomingforAccessnodes(likeOLTs)•  InServiceSoftwareUpgrades(ISSU)

•  Stratum/P4integration•  BNGfeatures(e.g.PPPoEtermination,hierachicalQoS)•  5Guserplanefeatures

Next-GenSDNDemo

OF-DPA

Stratum Stratum

Stratum

End hosts GTP traffic

Field office

Base station (emulated)

OF-DPA OF-DPA

Stratum OF-DPA

Multicast video source

Central office

Stratum OF-DPA

Trellis (Segment routing, multicast, vRouter, etc)

In-band Network Telemetry (INT)

VNF Offloading Control (S/PGW)

ONOS

P4Runtime OpenFlow

Internet

Upstream BGP routers

Mixed P4/OpenFlow multi-vendor white-

box switches Broadcom, Barefoot, Edge-

Core, Inventec, Delta

P4 SmartNIC Netcope

INT collector

Telemetry collector Barefoot DeepInsight

INT+S/PGW

INT INT

INT

INT

INT

INT reports

fabric.p4 - L2/L3/MPLS - INT - GTP termination

P4 program

netcfg - Devices - Ports - DHCP relay

CharlesChancharles@opennetworking.org

ThankYou