1

DEMYSTIFYING CONFIGURATION CHALLENGES IN NETWORK-BASED ISP SERVICES

Theophilus Benson*, Aditya Akella*, Aman Shaikh+

*University of Wisconsin, Madison+ ATT Labs Research

2

Adoption of Network-based Services

ISP Core

3

Adoption of Network-based Services

“By 2015, annual global IP traffic will reach 966 exabytes” [Cisco’11]

Customers adopting new applications ISPs upgrade and purchase new equipment Best effort ineffective for some applications

2010 2011 2012 2013 2014 20150E+00

2E+05

4E+05

6E+05

8E+05

1E+06

Time (in Years)

Siz

e o

f In

tern

et

Tra

ffic

(In

PB

)

4

Adoption of Network-based Services

Services are a crucial part of the Internet’s ecosystem

ISP Core

Host-based and network-based services “AT&T to spend $1 billion to ramp up enterprise services” Recover cost and improve application performance

5

Goals

Vision: Improve service integration/upgrades Simplify service management

Understand impediments Service configuration files Complexity of configuration

EdgeEdge

ISP Core

6

Configuration is a crucial component

Configuration determines Customer functionality ISP interactions

Configuration is complex Most time consuming task [Feamster ‘05 ]

Most error-prone > 50% customer problems due to configuration errors [Yankee Group ‘04]

OSPF

BGP

7

Configuring a Service

PECEPE CE

acz

ISP Core

Control Plane

Data plane

acz

acz

8

Contributions

Analyzed 2.5 years of configurations Show how complexity evolves over time

Worsens over time

Highlight the location of complexity Complexity exists at the edge

Identify the cause of complexity Due to provisioning of new customers

PEPE

ISP CoreCE

EdgeCustomersEdge

Customers

9

Contributions

Identified potential ways to mitigate complexity Showed the impact of design choices on complexity

Vendor

Co

mp

lex

ity

#1 #2

Routing DesignC

om

ple

xit

y#1 #2 #3

10

Outline

Motivation Background Models and Data-Set Understanding Complexity Mitigating Complexity Conclusion

11

Configuring the Provider’s Edge

Complexity is due to: Dependent commands

PECEPE CE

ISP Core

Ip vrf blueRd 23234:100223Route-target import 1000:1Route-target export 1000:1!Interface ethernet1Ip address 128.105.82.66/30Ip vrf forwarding blueServices-policy output policy1!Policy-map policy1Policy 100 20 confirm-action transmit

VRF

Interface

Policy-map

a

12

Configuring the Control-Plane Core

PECEPE CE

Router bgp 65000Neighbor 129.168.6.6Neighbor 129.168.2.2!

Ip vrf blueRd 23234:100223Route-target import 1000:1Route-target export 1000:1!Router bgp 65000Neighbor 129.168.2.1Neighbor 129.168.2.1

ISP Core

acz

acz

acz

Complexity is due to: Dependent commands Maintaining consistency

13

Configuring the Data-Plane Core

PECEPE CE

acz

acz

Interface gigethernet1Ip address 128.105.82.66/30!Router ospf 2Network 128.105.82.0/24!

Ip vrf blueRd 23234:100223Route-target import 1000:1Route-target export 1000:1!Interface gigethernet1Ip address 128.105.82.65/30!

ISP Corea

Complexity is due to: Dependent commands Maintaining consistency

acz

14

Models and Data-Set

Requirements for Data Models

Quantify complexity of configuration Capture dependencies between commands Capture consistency across devices

Use complexity metrics [Benson ‘09] Motivated by software engineering techniques

Abstract away low level details Abstract groups of commands stanzas

15

Ip vrf blueRd 23234:100223!Interface ethernet1Ip address 128.105.82.66Ip vrf forwarding blueServices-policy input policy2!

16

Data Models

Referential Graph [Benson ‘09] Syntactic dependencies operators must track Network graph of dependent stanzas Metric: size of graph Larger graph more dependencies

Templates [Benson ‘09] Clone detection used to capture uniformity

VRF

Interface

Policy-map

17

Data-Set

Service % of Routers ( PE + Core)

VPN 48%

VPLS 27%

VoIP 5%

DDoS Prev. 31%

Virtual Wire 25%

Diversity allows for a comprehensive study

Collected data from tier-1 ISP for 2.5 years 5 services: VPN, VPLS, VoIP, DDoS Prev., Virtual Wire Daily snapshots of router configuration files Metadata (per router): vendor, role and location

18

Understanding Complexity Provider Edge (PE) Complexity Control-Plane Core Complexity Data-Plane Core Complexity

PEPE

ISP CoreCE

19

Understanding Complexity Provider Edge (PE) Complexity Control-Plane Core Complexity Data-Plane Core Complexity

PEPEISP Core CE

20

PE Complexity over Time

Growth is due to worsening complexity New devices have less dependencies

Over time, configuration tasks become tricky

Dec 08

Dec 10

0E+0

2E+5

4E+5

6E+5

8E+5

1E+6

DDoS V-Wire VplsVPN VoIP

Time (in Months)

To

tal #

of

Re

f. L

ink

s

Dec 08

Dec 10

0

500

1000

1500

2000

2500

DDoS V-Wire VplsVPN VoIP

Time (in Months)

Ma

x P

E G

rap

h S

ize

21

Understanding PE Complexity Which stanza contributes to VPLS growth?

Edge EdgeISP Core

VRF

Interface

Policy-map

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Pe

rce

nta

ge

of

Sta

nza

s

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Pe

rce

nta

ge

of

Sta

nza

s

VRF

22

Understanding PE Complexity Which stanza contributes to VPN growth?

Complexity caused by customer provisioning

Edge EdgeISP Core

VRF

Interface

Policy-map

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Pe

rce

nta

ge

of

Sta

nza

s

Dec 08

Dec 10

0%20%40%60%80%

100%

Interface CoS VRF

Time (in Months)

Pe

rce

nta

ge

of

Sta

nza

s

23

Configuration Reuse over Time

Specialization leads to added complexity

• Reuse and specialization exists• Configuration overlap reduces over time

– Reduction due to specialized usage of service

0% 20% 40% 60% 80% 100%0

0.2

0.4

0.6

0.8

1

VPN 2008

Percentage of Reuse

CD

F o

f C

us

tom

ers

0% 20% 40% 60% 80% 100%0

0.2

0.4

0.6

0.8

1

VPN 2008 VPN 2010

Percentage of Reuse

CD

F o

f C

us

tom

ers

71%

62%88%

24

Understanding Complexity

Data-Plane Core: service-agnostic and simple Control-Plane Core: distinct across services

Growing number of adjacencies with PEs

PE is the most complex

PEPE

ISP CoreCE

25

Mitigating Complexity Vendor Selection

Cost

Fu

nct

ion

alit

y

Comple

xity

Time

Co

mp

lexi

ty

TimeC

om

ple

xity

26

How to Compare Vendors

Different vendors different languages Language impacts complexity Difference in structure of functionality

Comparing vendor languages Configurations representing same policy Same customer same policy on all PEs

PECEPE CE

ISP Core

Vendor1Vendor2

27

Vendor Selection

Graph for vendor1 is consistently larger Vendor1 requires more stanzas for same policies Operators need to track more dependencies

Choice of vendor can reduce PE complexity

1 2 3 4 5 6 70.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

Anonymized Customer ID

Ra

tio

of

Re

fere

nti

al g

rap

h(V

en

do

r1/V

en

do

r2)

1 2 3 4 5 6 70.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

Anonymized Customer ID

Ra

tio

of

Re

fere

nti

al g

rap

h(V

en

do

r1/V

en

do

r2)

28

Conclusion

Studied the factors that impede services Complexity grows over time

Modifications become time consuming

Most complexity lies in configuring customers Varying requirements and specialized configuration

Framework to systematically consider complexity Choice of vendor can reduce complexity

29

Thank You

Theophilus Benson (tbenson@cs.wisc.edu)

"Complex systems are built out of a myriad of simple components"