Jeronimo BezerraFloridaInternationalUniversity

<jab@amlight.net>

Internet2GlobalSummitWashingtonDC,Apr26th 2017

Inter-domainSDNDataPlaneValidation:NextStepsatAmLight

MarcosSchwarzRede Nacional deEnsino ePesquisa

<marcos.schwarz@rnp.br>

Outline• IntroducingAmLight• TroubleshootingproductionSDNnetworks• TroubleshootingtheDataPlane• Inter-DomainTracingProtocol• FutureWork• IntroducingtheAmLightSDNTrace app• Inter-Domainscreenshots

2

AmLightisaDistributedAcademicExchangePoint• Production SDNInfrastructuresinceAug2014• ResponsiblefortheSouthAmericaacademicconnectivity• CarriesAcademicandNon-Academic/Commercialtraffic

– L2VPN,IPv4,IPv6,Multicast

• SupportsNetworkProgrammability/Slicing– OpenFlow 1.0– FlowSpaceFirewallforNetworkProgrammability/Slicing– OGFNetworkServiceInterface(NSI)enabled

• Currently,operatingwithmorethan1kflows(productionandexperimentation)• Website:www.sdn.amlight.net

WiththeSDNdeployment,everythingchanged….

3

Troubleshootingaproduction SDNnetwork

• Troubleshootingaproductionenvironmenthasdifferentrequirements– Itneedstobeagileandleastdisruptiveaspossible– Itmightneedhistoricalinformationandunderstandingoftrafficgoingthroughthenetwork– Toolshavetobehandy!

• Legacytroubleshootingtoolsarepartiallyusefulorcompletelyuseless– OAM(Operation,AdministrationandMaintenance)isnotsupportedbyOpenFlow(yet)– Ping,traceroute,SNMP,Wireshark/Tcpdump arenotmadeforOpenFlownetworks

• Deepknowledgeofthehardwareandsoftwareplatformisrequired:– Usageofthe”hidden”commandsbecomespartofyourroutine

4

TroubleshootingDataPlane?

• Insomecases,everythinglooksfine,buttrafficisnotflowing• Examplesofdataplaneblackholes:

– Aspecificlinecardorinterfacediscardingalltraffic• Duetoaninterfacememoryissue,flowsareinstalledbuttrafficisdiscarded

– InterfacedowninonesidebutupintheremoteandtheSDNappdoesn’tunderstandthat• Forinstance:10GLAN-PHY,Ethernetcircuitsand100Glonghaulcircuits• Inthiscase,dependingoftheside,theSDNappinstallsthecircuitspointingtotheaffectedlink,discardingalltraffic

– Aspecificinstalledflowentrycrashed• Duetoaninterfacememoryissue,onespecificflowisaffectedandtrafficisdiscarded• DependingofthenumberofOpenFlowswitchesandflowentries,findingtheproblemmightbeextremelytime-

consuming

• Inthesecases,in-bandtestsarerequired:– JustaveryfewSDNappstestin-bandperlink– NoSDNappstestin-bandperflow

5

DataPlaneMonitoring• Monitoringindividualflowsisimportantbutextremely

expensive– Beingproactivewithallflowsisdesiredbuttheintervalbetween

testsandnumberofflowsneedtobetakenintoconsideration– Usingareactiveapproachisthebestsuggestion

• Userswon’tbehappy,butyourswitcheswon’tcrash

• Approachestovalidateusers’flowsarebeingproposed:”SDNtraceroute:TracingSDNForwardingwithoutChangingNetworkBehavior””Multi-protocolNetworkTroubleshootingwithPathtrace protocol”

• AmLight’sdeveloped asolution to traceusers’flows:SDNTrace

6

ApplicationLayer

Forwarding Device

OESS ONOS/SDN-IP

OpenFlow 1.0

Forwarding Device

Forwarding Device

Forwarding Device

FlowSpace Firewall

OpenFlow 1.0

User AUser A User BUser B

Testbed

Monitoring User Flows: SDNTrace

But,waitaminute!Whataboutcircuitsthatspansmultipledomains?

Inter-domainDataPlaneTroubleshooting• Multi-domainvirtualcircuitsaresubjecttoproblemsineachdomaintheytransverse• Issuesonlinkspeeringtwodomainsareevenmoredifficulttodetectandtroubleshoot:

– MultipleNOCs,configurationinconsistencies,devicesmalfunction,unpredictedtopologychanges

• Thelegacyway:manuallyaddanIPtoeachswitchinthepath,pingeachswitchuntilyouisolatetheissue– WhataboutSDN?Youdon’teasilyaddanIPtoanOpenFlowswitch!– Lackofinter-domaintools,currenteffortsmainlyonintra-domain

• Recentexperience:– Twousers(BrazilandUK)– Fivedomainsinthepath,includingtwoOpenFlow-based(AmLightandInternet2)– 22days&45e-mailstorestoreasingleVLAN!

• Thenwedecidedtoworkonthisproblem… 8

• Aninter-domain SDNdataplanetroubleshootingsolutionwascreatedwiththefollowinginitialrequirements:• UsershouldnotneedtoknowthenetworktopologyorunderstandOpenFlow,justlikea

traditionaltraceroute• Eachdomaininthepathshouldbeabletohavedifferentprivacypolicies• Thesolutionshouldnotrequiretopologyortechnologychanges,justafewOpenFlowentries

• Optional:• Theinter-domaintraceprotocolshouldbeflexibleenoughtosupportdifferent solutions ofpath

trace

9

Requirementsforaninter-domaintroubleshootingsolution

• Step1:Contractestablishedbetweenneighbordomainswiththe"color"ofthepeeringswitches.

• Step2:EachControllerPushesthe”colored”flowstothepeeringswitches

10

Howdoesitwork?(1)– SimpleVersion

Peering configuration between A and B

IP address type remote label/color interface

a.b.c.d:443 tracepath domainA:switchA2 dl_src:111 p2

w.x.y.z:443 tracepath domainB:switchB1 dl_src:110 p1

Afteratracerequest:Step3:”ControllerA” usesthecontractandsendsaprobematchingtheneighbor’s switch”color”Step4:Switch”A2” matchestheprobewiththeinter-domainflowentryandforwarditto”B1”Step5:Switch”B1”matchesthe probe with the ”colored”flow and sends the probe to ”Controller B”Step7:ControllerBreports<”domain B”,”switchB1”,”port 1”>toControllerAStep8:ControllerAforwardsthereporttotheuser

StepN:DomainBcontinuestheintra-domaintracetilltheendofitsdomain.Ifthereisa"DomainC”forsuchusercircuit,processcontinuesinthenextdomain. 11

12

Howdoesitwork?(3)– FullVersion

AmLightSDNTrace

• DoesnotchangeuserflowentriesATALL• Lightweight(2-4flowentriesneededper

sw)• WorkswithOpenFlow1.0and1.3• BasedonRyu SDNframework• Traceflows fromdifferent SDNapplications• TraceUserFlows based onLayer 2,Layer 3

ormixoflayers• SupportsInter-domainTracing• Betacode:

http://github.com/amlight/SDNTrace13

14

SDNTrace running– Demo

15

Screenshots– TracefromRNP(left)toCLARA(right)

RNP CLARA

16

Screenshots– TracefromCLARA(right)toRNP(left)

RNP CLARA

Future• ExpandthesolutiontousetheNetworkServiceInterface(NSI)protocol

• NSIprovidesmodelsfordescribingnetworkservices andenablestheuseofsharedresourcesthroughsecureandreliable sessionsforcommunicationbetweendomains

• UsedbyGLIFAutoGOLE communityforinter-domainL2circuitsprovisioning• Supportsauthenticationandencryption• NSIcanbeusedtoenforcethe”peeringcontract” andtransportcommunicationbetween

controllers

• DeployatAMPATH,AmLight,SouthernLight,ANSPandRNPin2017

• Evolvetoamorecomplexsolutionwithalarms/triggers,etc.

17

Internet2GlobalSummitWashingtonDC,Apr26th 2017

AmLight’s DevelopmentTeam:• ANSP– AcademicNetworkofSaoPaulo:

• AntonioFrancisco• JorgeMarcos• Rogerio Motitsuki

• RNP– Rede Nacional deEnsino ePesquisa• MarcosSchwarz

• FIU– FloridaInternationalUniversity• Jeronimo Bezerra

ThankYou!

Questions?

Agarwal,Kanak,etal."SDNtraceroute:TracingSDNforwardingwithoutchangingnetworkbehavior."ProceedingsofthethirdworkshoponHottopicsinsoftwaredefinednetworking.ACM,2014.

http://groups.geni.net/geni/wiki/GEC24Agenda/EveningDemoSession#Multi-protocolNetworkTroubleshootingwithPathtraceprotocol

References