System Testing SDN- A Case Study on ONOS

Suibin Zhang, ON.LabJon Hall, Ciena

KnownUNKNOWNs

Known KNOWNs

UnknownUNKNOWNs

Design Considerations

Unit Test Considerations System Test Considerations

Quality Assurance on Software System

NASubjective Perception

Obj

ectiv

e Re

ality

✔

2#ONOSProject

3#ONOSProject

• Black/Grey Box Testing– Often mimic usage scenarios– Validity not influenced by internal design

• Product Verification/Qualification• Pre-Deployment Verification

System Test - Where Used

4#ONOSProject

System Test Environment

Testing SDN OS – Nontrivial

SDN Network Domain

Distributed

Controllers

SDNCTRL

1

SDNCTRL

2

SDNCTRL

n

APP APPAPP

Test Exec.

?

?

Legacy Network Domain

Dependencies

?

5#ONOSProject

• Technical– Accessing heterogeneous, evolving interfaces– Scaling test environment– Facilitate Debugging

• Arise from Community/Open Source – Run “anywhere”– Handling test dependencies– Contribution and merging tests

Challenges in Building System Tests

6#ONOSProject

A Case Study on ONOS

7#ONOSProject

• Functionality Suite– NB: Intents and Flow operations– SB: OF 1.0 &1.3, NETCONF, OVSDB, etc.

• High Availability Suite– Various scenarios of cluster failures

• Performance and Scale-out Suites– Intent, flow throughputs, latency as scaling-out– Topology discovery latency as scaling-out

• Longevity– Continuous operation under typical deployment scenarios

• Selected Use Case Suites– SDNIP App, Segment Routing App

ONOS System Test Suites

8#ONOSProject

Establish Test Handles Run-time Configuration Test Logic

Reset/Clean Test Env. Inject Run Params to

TestStation Start Test Cases

Jenkins Plugins Update Codes Publish Results

Onos System Test CI

9#ONOSProject

A Deeper Dive into TestON

Onos System Test CI

10#ONOSProject

TestON – Key Benefits• TestON

o SDN-focused Testing Frameworko Test case authoring, executiono Python-based

• Key Benefits:o Extensibility – Simplify case authoring by compartmentalizing

concernso Flexibility - Handle heterogeneous test env. OVS, HW, Quagga, etc.o Debuggability - Facilitate troubleshooting

11#ONOSProject

Test Case Files*.params*.py*.topo Dependency

Details on TestON FrameworkTestON Core Framework:

Logging, Utilities, …

Driver Files

Mininet Driver ONOS Rest Driver Quagga DriverONOS CLI Driver

Test Files.params File – XML file, used to pass variables into test, such as order of test cases.topo file – XML File, defines the components of the testDependency Folder – Config files or Mininet topology files used by the test.py file – Combine TestON and driver functions to define your test cases.

Test Cases are split into steps, each of which should have an assertion in them

• Driver Files• Can use REST, CLI via SSH, or other custom connection interfaces• Defined in a hierarchical structure• Creates a library of functions to be used in the test

12#ONOSProject

ONOS Intent Feature

• Set intent through NB REST interface• Validate intent and flows are correctly added• Test end-to-end connectivity in Mininet with the specific selectors

System Test Procedure

• Need a library to send packets with test-specified selector• Leveraging Python Scapy module

System Test Requirement

• High-level abstraction of setting end to end connectivity• Can be defined by “Selectors” of MAC, IP, PORT, MPLS label, etc.

Example: ONOS Intent Functionality Test

13#ONOSProject

Building a Scapy Driver(send/recv user-defined packets)

class ScapyCliDriver( Emulator ): def sendPacket( self, iface=None, packet=None, timeout=1 ): try:

sendCmd = 'srp( ' if packet: sendCmd += packet else:

... if iface: sendCmd += ", iface='{}'".format( iface ) sendCmd += ', timeout=' + str( timeout ) + ')' self.handle.sendline( sendCmd ) self.handle.expect( self.scapyPrompt ) except:

...

TestON |--drivers | |--common | |--api | |--cli | | |--emulator | | |--ScapyCliDriver.py |--tests

Example: ONOS’ Intent Functionality Test

14#ONOSProject

for hostName in [ “h1”,”h2” ]: # Create two host components with the scapy driver4 main.Scapy.createHostComponent( hostName )main.step( ”Login to scapy on these new components" )for host in [ main.h1, main.h2 ]: host.startHostCli() host.startScapy()

Using Scapy Library - in Mininet Host Handle

<Scapy> <host>192.168.1.101</host> <user>admin</user> <password></password> <type>ScapyCliDriver</type> <connect_order>6</connect_order></Scapy>

TestON |--drivers |--tests |--Example |--Example.py |--Example.params |--Example.topo

Example: ONOS’ Intent Functionality Test

15#ONOSProject

# Add connectivity from h1 to h2 (unidirectional)addResult = main.ONOS1.addPointIntent( ingress, egress, ethDst, ipDst) main.h2.startFilter() # Start packet filter on host2main.h1.buildEther( dst=main.h2.hostMac ) # build ethernet framemain.h1.buildIP( dst=main.h2.hostIp ) # Build IP framemain.h1.sendPacket() # Send built packetfinished = main.h2.checkFilter() # Check if host2 received any packets

Intent Test Case - Using Scapy Driver

TestON |--drivers |--tests |--Example |--Example.py |--Example.params |--Example.topo

Example: ONOS’ Intent Functionality Test

16#ONOSProject

• Parameterize Test Env & Run Var– Jenkins to inject at run time– Test cases deal with “pure” test

logic• Extensive Exception Handling in

Drivers– Take care of exceptions, timeout– Use “retry” method

• Test Dependencies– Run-time Activation– “Dockerize” Fixture (Future Work)

• Debug Utilities– Test pausing– Log collation– More (Future Work)

• Scale Test Network (Future Work)

Lessons Learned and Future Work

17#ONOSProject

For more Information• S3 Showcase: “SDN/ONOS System Test”• Wiki: https://wiki.onosproject.org/display/ONOS/System+Tests• Repo: https://gerrit.onosproject.org/#/admin/projects/OnosSystemTest• Github: https://github.com/opennetworkinglab/OnosSystemTest

Summary

Python-Based

Extensibility

DebuggabilityFlexibility

ONOS Test Suites:FunctionalityHigh AvailabilityPerformanceScale-out CapabilityLongevityUse Cases

ONOS System Test Highlights

Open Source

18#ONOSProjectJoin the journey @

onosproject.org

Software Defined Transformation of Service Provider Networks

Q&A