University of the Free StateRedesigning the campus network for innovation, VoIP, multicast and IPv6

Presented by Louis Marais (Deputy Director ICTS, UFS) and Andrew Alston (Owner, Alston Networks)

AfriNIC Meeting – November 2012Remote Presentation

The Original NetworkWhat we had:

• Approximately 19000 Wired Ports• 450+ CCTV Cameras• 500+ Switches• 400+ Wireless Access Points• /16 Worth of Legacy IP Space

• No Routing Protocols• No VLAN segmentation• No working multicast• No IPv6• No true IP Plan• No QoS Implementation (or way to implement it!)

The Original NetworkWhat we didn’t have:

The Original NetworkWhat we REALLY had:

• ONE Broadcast Domain• Overloaded ARP Tables• Switch CPU utilizations in excess of 90%• Regular broadcast storms• Spanning Tree / Topology Loop issues• A disaster waiting to happen….

Making a plan….• Step 1 – Figure out what the objectives are• Step 2 – Figure out how to get there• Step 3 – Obtain the required prerequisites• Step 4 – Plan the deployment• Step 5 – Implement (carefully)• Step 6 – Resolve any unforeseen issues• Step 7 – Skills Transfer

The Objectives….• Create a design that is scalable • Cater for the deployment of IPv6• Cater for the deployment of Multicast Technologies• Cater for IP Telephony (VoIP)• Allow for sufficient flexibility to support innovation• Get rid of the single broadcast domain• Don’t over complicate – it has to be manageable• Avoid a design that will require huge amounts of downtime during implementation• Ensure that NAT doesn’t exist on campus – it breaks too many things

The design we chose…• Build on service provider principles• Segment the network – Core / Distribution / Edge• OSPF for loopback distribution purposes – We wanted ISIS but it wasn’t supported• BGP for the majority of the table (IPv4)• OSPFv3 for all IPv6 – The vendor didn’t support v6 under BGP• No VLAN spanning between distributions• PIM for multicast routing – The vendor didn’t support MSDP

The problems and prerequisites• Rollout of routing protocols, multicast and IPv6 could not begin before the network

was properly segmented• Segmenting the network would normally result in large amounts of downtime due

to subnet mask issues.• The switches required additional licenses that had to be purchased to support

routing protocols

Why the segmentation problem?

• Devices in the various domains would have the same subnet mask

• Everything in each broadcast domain would still believe it was “directly connected” to everything in other broadcast domains

• Either insert more specific routes on every device (impractical), or change the subnet masks on each device as we went (impractical)

The Solution?

• Get more IP Space – Needed anyway for expansion

• Renumber the newly structured segments into the new IP space as we went

• Leave the original broadcast domain intact, simply reducing its size as we went

The planning exercise• Performed a comprehensive IP planning exercise, and figure out just how

much IP space we needed both today and going forward• Applied for and were granted an additional /15 worth of IP space from

AfriNIC• Put the new IP space through our IP plan and created a segmentation plan

for the network• Network divided into Core, 13 distributions, and the edge• Work out the size of the supernets required per distribution• Devise a VLAN plan that could be duplicated across the distributions in

a uniform fashion• Further divide the supernets routed to the distribution into VLAN

based “connected” subnets.

Further planning details• Each building on campus (100+ of them) received 6 VLAN’s

• Wired, Wireless, CCTV, Access Control, Building Management, VOIP• The distributions carried all VLAN’s for a number of buildings (In some

cases up to 200 VLAN’s)• A decision was taken NOT to route the subnets, ONLY the supernets per

distribution, let the distribution connected routes take care of the subnets.• Only place the core -> distribution point to points and the core/distribution

loopbacks into OSPF, let BGP carry the rest.• Connected route redistribution into the dynamic routing protocols was

avoided.

IP Planning – Supernet’s

IP Planning – Subnet’s

Management Tools

Management Tools (2)

Management Tools (3)

Management Tools (4)

Management Tools (5)

The Implementation…• Since VLAN 1 was already going to each distribution, leave it alone at the

start… we didn’t want downtime• Create a second VLAN for each distribution, and trunk it between the core

and the distribution.• Apply a /30 to each of these new point to point VLANs.• Create the required building VLAN’s on the distribution and assign the new

gateway IPs to each VLAN. All of this in the newly allocated AfriNIC space.• Assign the IPv6 addressing to the new VLAN’s at the same time.• Setup the OSPF / OSPFv3 / BGP between the core and the distribution and

ensure routing is working back to the core.• Trunk the building VLAN’s down to the edge switches from the distributions,

again, leaving VLAN 1 still trunked to the edge• NOTE: At this point there was still ZERO effect on production traffic – we had

NO downtime doing this.

Implementation Part 2…• Enable PIM Sparse Mode between the core and distributions on the point to

point VLAN’s.• Ensure the DHCP relays were setup correctly on the distribution switches• Create the DHCP Scopes for the new IPv4 space.• We decided to use RA(EUI-64) for IPv6, since DHCPv6 is not as supported on

all edge devices.• NOTE: We still had not touched production traffic, all of this was done as an

“overlay” network.

The moment of truth…• Come back in the dead of night… consume lots of red bull… take a deep

breath… and then….• Re-Tag the edge ports into the new correct VLAN’s for their respective

buildings / uses – LEAVING PRINTERS AND STATIC IP DEVICES IN VLAN 1• Force a switch port flap (shut down the port, wait 30 seconds, un shut the

port).• Edge devices see a disconnect and reconnect and request a new DHCP lease

as a result.• On bringing the port back the edge device requests a new DHCP lease, and

lands in the correct subnet. It also gets an IPv6 address through RA automatically.

• Total downtime on any network segment: 30 seconds.

The secondary moment of truth…• Once we knew that edge devices were getting IP and IPv6 addresses in the

correct subnets• Send some techs around to renumber static assigned devices (printers

and such things) into the new space and move them into the correct VLAN’s as we went

• Enable IPv6 on the proxy servers…• Enable IPv6 on the DNS servers…• Watch the traffic swing instantly to approx. 60% IPv6

The end result….• Smaller Broadcast domains

• Less broadcast traffic – an increase in network performance of *70%*• Switch CPU loads down from 90% to 3% as a result of smaller ARP

tables• A more manageable network – we know where IP addresses are in use, so

tracing things is easy• A more flexible network – work on one segment doesn’t affect the whole

network• Far less spanning tree / looping / L2 topology issues• IPv6 to every edge device – yes, it can be done

The end result (IPv6 Stats)

Mon 22 Oct 2012 00:05 Tue 23 Oct 2012 01:35 Wed 24 Oct 2012 03:05Thu 25 Oct 2012 04:35 Fri 26 Oct 2012 06:050%

10%20%30%40%50%60%70%80%90%

100%

TOTALIPv6

The costs / time involved• Planning took 3 weeks – the large majority of the time• Implementation was done largely at night over a 2 week period• No one on the network saw more than 30 seconds of downtime• The total cost to achieve a 70% performance increase and ensure the

network was future proof was less than $50,000.• The money spent included:

• Routing licenses for the core and distribution switches (the majority of the cost)

• Consultancy fees• Vast quantities of redbull and pizza over the 2 week

implementation period.

Next Steps… Where to from here• We want a more robust multicast network – this means changing the way

we do multicast.• In order to cater for multi-homing, better traffic analysis using netflow/ipfix,

and to allow for more flexibility we want a full BGP table in the core• We want to reduce the over-subscription between the edge and distribution

and distribution and core• MPLS Implementation across the border/core/distribution to allow for

EoMPLS cross connects between network segments and external entities.• To this end… a core and distribution upgrade is planned for early 2013.

The new core / distribution design• Replace the core switches with proper routers• Replace the distribution layer with 10G capable L3 switches that support the

required protocols• Implement a proper border router that is full table BGP capable• Implement MSDP from border to core to distribution.• Uplink each distribution with redundant links to separate core routers in a

virtual chassis configuration• Ensure that each distribution has at least 20G back to the core (and

potentially more)• Link the cores with redundant 100G technology.• Rollout MPLS across the border/core/distribution.• Support QoS where appropriate for VoIP implementation

The new core / distribution design

Lessons Learnt• Re-structuring a network for IPv6 / Multicast / The Future does not:

• Require loads of downtime• Require vast amounts of money

• Re-structuring a network for IPv6 / Multicast / The Future does require:• Some vision as to where you want to go• VERY careful planning• Careful documentation

THANK YOU!

Any questions?

Our Contact Details:Louis Marais: maraisl@ufs.ac.za

Andrew Alston: aa@alstonnetworks.net