isp & ixp design - middle east nog
TRANSCRIPT
![Page 1: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/1.jpg)
ISP & IXP Design Philip Smith MENOG 11
Amman 30th September – 9th October 2012
1
![Page 2: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/2.jpg)
ISP & IXP Network Design p PoP Topologies and Design p Backbone Design p Upstream Connectivity & Peering p Addressing p Routing Protocols p Out of Band Management p Operational Considerations p Internet Exchange Points
2
![Page 3: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/3.jpg)
Point of Presence Topologies
3
![Page 4: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/4.jpg)
PoP Topologies p Core routers – high speed trunk
connections p Distribution routers and Access routers –
high port density p Border routers – connections to other
providers p Service routers – hosting and servers p Some functions might be handled by a
single router
4
![Page 5: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/5.jpg)
PoP Design p Modular Design p Aggregation Services separated according
to n connection speed n customer service n contention ratio n security considerations
5
![Page 6: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/6.jpg)
Modular PoP Design
6
Backbone link to another PoP
Backbone link to another PoP
Business customer aggregation layer
for leased line circuit delivery Channelised circuits
Network Operations
Centre
Consumer Dial Access
Network Core
Consumer cable, xDSL and
wireless Access
for MetroE circuit delivery GigE fibre trunks
MetroE customer aggregation layer
ISP Services (DNS, Mail, News,
FTP, WWW)
Hosted Services & Datacentre
Other ISPs Web Cache
![Page 7: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/7.jpg)
Modular Routing Protocol Design p Modular IGP implementation
n IGP “area” per PoP n Core routers in backbone area (Area 0/L2) n Aggregation/summarisation where possible
into the core p Modular iBGP implementation
n BGP route reflector cluster n Core routers are the route-reflectors n Remaining routers are clients & peer with
route-reflectors only
7
![Page 8: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/8.jpg)
Point of Presence Design
8
![Page 9: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/9.jpg)
PoP Modules p Low Speed customer connections
n PSTN/ISDN dialup n Low bandwidth needs n Low revenue, large numbers
p Leased line customer connections n E1/T1 speed range n Delivery over channelised media n Medium bandwidth needs n Medium revenue, medium numbers
9
![Page 10: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/10.jpg)
PoP Modules p Broad Band customer connections
n xDSL, Cable and Wireless n High bandwidth needs n Low revenue, large numbers
p MetroE & Highband customer connections n Trunk onto GigE or 10GigE of 10Mbps and
higher n Channelised OC3/12 delivery of E3/T3 and
higher n High bandwidth needs n High revenue, low numbers
10
![Page 11: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/11.jpg)
PoP Modules p PoP Core
n Two dedicated routers n High Speed interconnect n Backbone Links ONLY n Do not touch them!
p Border Network n Dedicated border router to other ISPs n The ISP’s “front” door n Transparent web caching? n Two in backbone is minimum guarantee for
redundancy 11
![Page 12: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/12.jpg)
PoP Modules p ISP Services
n DNS (cache, secondary) n News (still relevant?) n Mail (POP3, Relay, Anti-virus/anti-spam) n WWW (server, proxy, cache)
p Hosted Services/DataCentres n Virtual Web, WWW (server, proxy, cache) n Information/Content Services n Electronic Commerce
12
![Page 13: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/13.jpg)
PoP Modules p Network Operations Centre
n Consider primary and backup locations n Network monitoring n Statistics and log gathering n Direct but secure access
p Out of Band Management Network n The ISP Network “Safety Belt”
13
![Page 14: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/14.jpg)
Low Speed Access Module
14
To Core Routers
Primary Rate T1/E1
PSTN lines to modem bank
PSTN lines to built-in modems
Access Servers
TACACS+/Radius proxy, DNS resolver,
Content
Web Cache
Access Network Gateway Routers
![Page 15: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/15.jpg)
Medium Speed Access Module
15
To Core Routers
Channelised T1/E1
64K and nx64K circuits
Mixture of channelised T1/E1, 56/64K and
nx64K circuits
Aggregation Edge
![Page 16: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/16.jpg)
High Speed Access Module
16
To Core Routers
Metro Ethernet
Channelised T3/E3
Channelised OC3/OC12
Aggregation Edge
![Page 17: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/17.jpg)
Broadband Access Module
17
To Core Routers
Telephone Network
The cable system
BRAS
SSG, DHCP, TACACS+ or Radius Servers/Proxies,
DNS resolver, Content
Web Cache
Access Network Gateway Routers
Cable RAS
DSLAM
IP, ATM
![Page 18: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/18.jpg)
ISP Services Module
18
DNS cache DNS
secondary POP3 Mail Relay NEWS
To core routers
WWW cache
Service Network Gateway Routers
![Page 19: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/19.jpg)
Hosted Services Module
19
Customer 7 Customer 3 Customer 4 Customer 5
Customer 6
To core routers Hosted Network Gateway Routers
Customer 2 Customer 1
vlan12 vlan11 vlan13 vlan14 vlan15 vlan16 vlan17
![Page 20: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/20.jpg)
Border Module
20
To core routers
Network Border Routers
To local IXP NB: router has no default route +
local AS routing table only ISP1 ISP2
![Page 21: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/21.jpg)
NOC Module
21
Primary DNS
To core routers Hosted Network Gateway Routers
SYSLOG server TACACS+
server Network Operations Centre Staff
Out of Band Management Network
2811/32async
NetFlow Analyser
Firewall
Billing, Database and Accounting
Systems
Corporate LAN Critical Services
Module
![Page 22: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/22.jpg)
Out of Band Network
22
Out of Band Management Network
Terminal server To the NOC
Out of Band Ethernet
NetFlow Collector
NetFlow enabled
routers
Router consoles
![Page 23: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/23.jpg)
Backbone Network Design
23
![Page 24: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/24.jpg)
Backbone Design p Routed Backbone p Switched Backbone
n ATM/Frame Relay core network n Now obsolete
p Point-to-point circuits n nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,
10GigE, OC192, OC768 p ATM/Frame Relay service from telco
n T3, OC3, OC12,… delivery n Easily upgradeable bandwidth (CIR) n Almost vanished in availability now 24
![Page 25: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/25.jpg)
Distributed Network Design p PoP design “standardised”
n operational scalability and simplicity p ISP essential services distributed around
backbone p NOC and “backup” NOC p Redundant backbone links
25
![Page 26: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/26.jpg)
Distributed Network Design
26
POP One
POP Two
POP Three
Customer connections
Customer connections
Customer connections
External connections
External connections Operations Centre
Backup Operations Centre
ISP Services
ISP Services
ISP Services
![Page 27: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/27.jpg)
Backbone Links p ATM/Frame Relay
n Virtually disappeared due to overhead, extra equipment, and shared with other customers of the telco
n MPLS has replaced ATM & FR as the telco favourite
p Leased Line/Circuit n Most popular with backbone providers n IP over Optics and Metro Ethernet very
common in many parts of the world
27
![Page 28: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/28.jpg)
Long Distance Backbone Links p These usually cost more p Important to plan for the future
n This means at least two years ahead n Stay in budget, stay realistic n Unplanned “emergency” upgrades will be
disruptive without redundancy in the network infrastructure
28
![Page 29: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/29.jpg)
Long Distance Backbone Links p Allow sufficient capacity on alternative
paths for failure situations n Sufficient can depend on the business strategy n Sufficient can be as little as 20% n Sufficient is usually over 50% as this offers
“business continuity” for customers in the case of link failure
n Some businesses choose 0% p Very short sighted, meaning they have no spare
capacity at all!!
29
![Page 30: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/30.jpg)
Long Distance Links
30
POP One
POP Two
POP Three
Long distance link
Alternative/Backup Path
![Page 31: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/31.jpg)
Metropolitan Area Backbone Links p Tend to be cheaper
n Circuit concentration n Choose from multiple suppliers
p Think big n More redundancy n Less impact of upgrades n Less impact of failures
31
![Page 32: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/32.jpg)
Metropolitan Area Backbone Links
32
POP One
POP Two
POP Three
Metropolitan Links
Metropolitan Links
Traditional Point to Point Links
![Page 33: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/33.jpg)
Upstream Connectivity and Peering
33
![Page 34: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/34.jpg)
Transits p Transit provider is another autonomous system
which is used to provide the local network with access to other networks n Might be local or regional only n But more usually the whole Internet
p Transit providers need to be chosen wisely: n Only one
p no redundancy n Too many
p more difficult to load balance p no economy of scale (costs more per Mbps) p hard to provide service quality
p Recommendation: at least two, no more than three
![Page 35: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/35.jpg)
Common Mistakes p ISPs sign up with too many transit providers
n Lots of small circuits (cost more per Mbps than larger ones)
n Transit rates per Mbps reduce with increasing transit bandwidth purchased
n Hard to implement reliable traffic engineering that doesn’t need daily fine tuning depending on customer activities
p No diversity n Chosen transit providers all reached over same satellite
or same submarine cable n Chosen transit providers have poor onward transit and
peering
![Page 36: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/36.jpg)
Peers p A peer is another autonomous system with which
the local network has agreed to exchange locally sourced routes and traffic
p Private peer n Private link between two providers for the purpose of
interconnecting p Public peer
n Internet Exchange Point, where providers meet and freely decide who they will interconnect with
p Recommendation: peer as much as possible!
![Page 37: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/37.jpg)
Common Mistakes p Mistaking a transit provider’s “Exchange”
business for a no-cost public peering point p Not working hard to get as much peering
as possible n Physically near a peering point (IXP) but not
present at it n (Transit is rarely cheaper than peering!!)
p Ignoring/avoiding competitors because they are competition n Even though potentially valuable peering
partner to give customers a better experience
![Page 38: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/38.jpg)
Private Interconnection p Two service providers agree to
interconnect their networks n They exchange prefixes they originate into the
routing system (usually their aggregated address blocks)
n They share the cost of the infrastructure to interconnect
p Typically each paying half the cost of the link (be it circuit, satellite, microwave, fibre,…)
p Connected to their respective peering routers
n Peering routers only carry domestic prefixes
38
![Page 39: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/39.jpg)
Private Interconnection
p PR = peering router n Runs iBGP (internal) and eBGP (with peer) n No default route n No “full BGP table” n Domestic prefixes only
p Peering router used for all private interconnects
PR PR
ISP1
ISP2
Upstream
Upstream
39
![Page 40: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/40.jpg)
Public Interconnection p Service provider participates in an
Internet Exchange Point n It exchanges prefixes it originates into the
routing system with the participants of the IXP n It chooses who to peer with at the IXP
p Bi-lateral peering (like private interconnect) p Multi-lateral peering (via IXP’s route server)
n It provides the router at the IXP and provides the connectivity from their PoP to the IXP
n The IXP router carries only domestic prefixes
40
![Page 41: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/41.jpg)
Public Interconnection
p ISP1-PR = peering router of our ISP n Runs iBGP (internal) and eBGP (with IXP peers) n No default route n No “full BGP table” n Domestic prefixes only
p Physically located at the IXP
ISP1-PR ISP1
Upstream
41
IXP
ISP2-PR ISP3-PR
ISP4-PR
ISP5-PR
ISP6-PR
![Page 42: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/42.jpg)
Public Interconnection p The ISP’s router IXP peering router needs careful
configuration: n It is remote from the domestic backbone n Should not originate any domestic prefixes n (As well as no default route, no full BGP table) n Filtering of BGP announcements from IXP peers (in and
out)
42
![Page 43: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/43.jpg)
Upstream/Transit Connection p Two scenarios:
n Transit provider is in the locality p Which means bandwidth is cheap, plentiful, easy to
provision, and easily upgraded n Transit provider is a long distance away
p Over undersea cable, satellite, long-haul cross country fibre, etc
p Both scenarios have different requirements which need to be considered
43
![Page 44: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/44.jpg)
Local Transit Provider
p BR = ISP’s Border Router n Runs iBGP (internal) and eBGP (with transit) n Either receives default route or the full BGP table from
upstream n BGP policies are implemented here (depending on
connectivity) n Packet filtering is implemented here (as required)
AR BR
Transit
ISP1
44
![Page 45: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/45.jpg)
Distant Transit Provider
p BR = ISP’s Border Router n Co-located in a co-lo centre (typical) or in the upstream provider’s
premises n Runs iBGP with rest of ISP1 backbone n Runs eBGP with transit provider router(s) n Implements BGP policies, packet filtering, etc n Does not originate any domestic prefixes
AR1
Transit ISP1
45
BR
AR2
![Page 46: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/46.jpg)
Distant Transit Provider p Positioning a router close to the Transit
Provider’s infrastructure is strongly encouraged: n Long haul circuits are expensive, so the router
allows the ISP to implement appropriate filtering first
n Moves the buffering problem away from the Transit provider
n Remote co-lo allows the ISP to choose another transit provider and migrate connections with minimum downtime
46
![Page 47: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/47.jpg)
Distant Transit Provider p Other points to consider:
n Does require remote hands support n (Remote hands would plug or unplug cables,
power cycle equipment, replace equipment, etc as instructed)
n Appropriate support contract from equipment vendor(s)
n Sensible to consider two routers and two long-haul links for redundancy
47
![Page 48: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/48.jpg)
Summary p Design considerations for:
n Private interconnects p Simple private peering
n Public interconnects p Router co-lo at an IXP
n Local transit provider p Simple upstream interconnect
n Long distance transit provider p Router remote co-lo at datacentre or Transit
premises
48
![Page 49: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/49.jpg)
Addressing
49
![Page 50: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/50.jpg)
Getting IPv4 & IPv6 address space
p Take part of upstream ISP’s PA space or
p Become a member of your Regional Internet Registry and get your own allocation n Require a plan for a year ahead n General policies are outlined in RFC2050, more
specific details are on the individual RIR website p There is no more IPv4 address space at IANA
n APNIC & RIPE NCC are now in their “final /8” IPv4 delegation policy phase
n Limited IPv4 available n IPv6 allocations are simple to get in most RIR regions
50
![Page 51: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/51.jpg)
What about RFC1918 addressing?
p RFC1918 defines IPv4 addresses reserved for private Internets n Not to be used on Internet backbones n http://www.ietf.org/rfc/rfc1918.txt
p Commonly used within end-user networks n NAT used to translate from private internal to public
external addressing n Allows the end-user network to migrate ISPs without a
major internal renumbering exercise p ISPs must filter RFC1918 addressing at their
network edge n http://www.cymru.com/Documents/bogon-
list.html 51
![Page 52: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/52.jpg)
What about RFC1918 addressing? p There is a long list of well known problems:
n http://www.rfc-editor.org/rfc/rfc6752.txt
p Including: n False belief it conserves address space n Adverse effects on Traceroute n Effects on Path MTU Discovery n Unexpected interactions with some NAT implementations n Interactions with edge anti-spoofing techniques n Peering using loopbacks n Adverse DNS Interaction n Serious Operational and Troubleshooting issues n Security Issues
p false sense of security, defeating existing security techniques
52
![Page 53: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/53.jpg)
What about RFC1918 addressing? p Infrastructure Security: not improved by using
private addressing n Still can be attacked from inside, or from customers, or
by reflection techniques from the outside p Troubleshooting: made an order of magnitude
harder n No Internet view from routers n Other ISPs cannot distinguish between down and broken
p Summary: n ALWAYS use globally routable IP addressing for ISP
Infrastructure
53
![Page 54: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/54.jpg)
Addressing Plans – ISP Infrastructure p Address block for router loop-back interfaces p Address block for infrastructure
n Per PoP or whole backbone n Summarise between sites if it makes sense n Allocate according to genuine requirements, not historic
classful boundaries p Similar allocation policies should be used for IPv6
as well n ISPs just get a substantially larger block (relatively) so
assignments within the backbone are easier to make
54
![Page 55: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/55.jpg)
Addressing Plans – Customer p Customers are assigned address space
according to need p Should not be reserved or assigned on a
per PoP basis n ISP iBGP carries customer nets n Aggregation not required and usually not
desirable
55
![Page 56: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/56.jpg)
Addressing Plans (contd) p Document infrastructure allocation
n Eases operation, debugging and management p Document customer allocation
n Contained in iBGP n Eases operation, debugging and management n Submit network object to RIR Database
56
![Page 57: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/57.jpg)
Routing Protocols
57
![Page 58: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/58.jpg)
Routing Protocols p IGP – Interior Gateway Protocol
n Carries infrastructure addresses, point-to-point links
n Examples are OSPF, ISIS,... p EGP – Exterior Gateway Protocol
n Carries customer prefixes and Internet routes n Current EGP is BGP version 4
p No connection between IGP and EGP
58
![Page 59: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/59.jpg)
Why Do We Need an IGP? p ISP backbone scaling
n Hierarchy n Modular infrastructure construction n Limiting scope of failure n Healing of infrastructure faults using dynamic
routing with fast convergence
59
![Page 60: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/60.jpg)
Why Do We Need an EGP? p Scaling to large network
n Hierarchy n Limit scope of failure
p Policy n Control reachability to prefixes n Merge separate organizations n Connect multiple IGPs
60
![Page 61: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/61.jpg)
Interior versus Exterior Routing Protocols p Interior
n Automatic neighbour discovery
n Generally trust your IGP routers
n Prefixes go to all IGP routers
n Binds routers in one AS together
p Exterior n Specifically configured
peers n Connecting with outside
networks n Set administrative
boundaries n Binds AS’s together
61
![Page 62: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/62.jpg)
Interior versus Exterior Routing Protocols p Interior
n Carries ISP infrastructure addresses only
n ISPs aim to keep the IGP small for efficiency and scalability
p Exterior n Carries customer
prefixes n Carries Internet
prefixes n EGPs are independent
of ISP network topology
62
![Page 63: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/63.jpg)
Hierarchy of Routing Protocols
63
BGP4
BGP4 and OSPF/ISIS
Other ISPs
Customers IXP
Static/BGP4
BGP4
![Page 64: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/64.jpg)
Routing Protocols: Choosing an IGP p OSPF and ISIS have very similar properties p Which to choose?
n Choose which is appropriate for your operators’ experience
n In most vendor releases, both OSPF and ISIS have sufficient “nerd knobs” to tweak the IGP’s behaviour
n OSPF runs on IP n ISIS runs on infrastructure, alongside IP n ISIS supports both IPv4 and IPv6 n OSPFv2 (IPv4) plus OSPFv3 (IPv6)
64
![Page 65: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/65.jpg)
Routing Protocols: IGP Recommendations p Keep the IGP routing table as small as possible
n If you can count the routers and the point-to-point links in the backbone, that total is the number of IGP entries you should see
p IGP details: n Should only have router loopbacks, backbone WAN
point-to-point link addresses, and network addresses of any LANs having an IGP running on them
n Strongly recommended to use inter-router authentication
n Use inter-area summarisation if possible
65
![Page 66: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/66.jpg)
Routing Protocols: More IGP recommendations p To fine tune IGP table size more, consider:
n Using “ip unnumbered” on customer point-to-point links – saves carrying that /30 in IGP
p (If customer point-to-point /30 is required for monitoring purposes, then put this in iBGP)
n Use contiguous addresses for backbone WAN links in each area – then summarise into backbone area
n Don’t summarise router loopback addresses – as iBGP needs those (for next-hop)
n Use iBGP for carrying anything which does not contribute to the IGP Routing process
66
![Page 67: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/67.jpg)
Routing Protocols: iBGP Recommendations p iBGP should carry everything which
doesn’t contribute to the IGP routing process n Internet routing table n Customer assigned addresses n Customer point-to-point links n Access network dynamic address pools,
passive LANs, etc
67
![Page 68: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/68.jpg)
Routing Protocols: More iBGP Recommendations p Scalable iBGP features:
n Use neighbour authentication n Use peer-groups to speed update process and
for configuration efficiency n Use communities for ease of filtering n Use route-reflector hierarchy
p Route reflector pair per PoP (overlaid clusters)
68
![Page 69: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/69.jpg)
Security
69
![Page 70: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/70.jpg)
Security p ISP Infrastructure security p ISP Network security p Security is not optional! p ISPs need to:
n Protect themselves n Help protect their customers from the Internet n Protect the Internet from their customers
p The following slides are general recommendations n Do more research on security before deploying any
network
70
![Page 71: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/71.jpg)
ISP Infrastructure Security p Router & Switch Security
n Use Secure Shell (SSH) for device access & management
p Do NOT use Telnet
n Device management access filters should only allow NOC and device-to-device access
p Do NOT allow external access n Use TACACS+ for user authentication and
authorisation p Do NOT create user accounts on routers/switches
71
![Page 72: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/72.jpg)
ISP Infrastructure Security p Remote access
n For Operations Engineers who need access while not in the NOC
n Create an SSH server host (this is all it does) p Or a Secure VPN access server
n Ops Engineers connect here, and then they can access the NOC and network devices
72
![Page 73: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/73.jpg)
ISP Infrastructure Security p Other network devices?
n These probably do not have sophisticated security techniques like routers or switches do
n Protect them at the LAN or point-to-point ingress (on router)
p Servers and Services? n Protect servers on the LAN interface on the router n Consider using iptables &c on the servers too
p SNMP n Apply access-list to the SNMP ports n Should only be accessible by management system, not
the world
73
![Page 74: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/74.jpg)
ISP Infrastructure Security p General Advice:
n Routers, Switches and other network devices should not be contactable from outside the AS
n Achieved by blocking typical management access protocols for the infrastructure address block at the network perimeter
p E.g. ssh, telnet, http, snmp,… n Use the ICSI Netalyser to check access levels:
p http://netalyzr.icsi.berkeley.edu
n Don’t block everything: BGP, traceroute and ICMP still need to work!
74
![Page 75: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/75.jpg)
ISP Network Security p Effective filtering
n Protect network borders from “traffic which should not be on the public Internet”, for example:
p LAN protocols (eg netbios) p Well known exploit ports (used by worms and
viruses) p Drop traffic arriving and going to private and non-
routable address space (IPv4 and IPv6) n Achieved by packet filters on border routers
p Remote trigger blackhole filtering
75
![Page 76: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/76.jpg)
ISP Network Security – RTBF p Remote trigger blackhole filtering
n ISP NOC injects prefixes which should not be accessible across the AS into the iBGP
n Prefixes have next hop pointing to a blackhole address n All iBGP speaking backbone routers configured to point
the blackhole address to the null interface n Traffic destined to these blackhole prefixes are dropped
by the first router they reach p Application:
n Any prefixes (including RFC1918) which should not have routability across the ISP backbone
76
![Page 77: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/77.jpg)
ISP Network Security – RTBF p Remote trigger blackhole filtering example:
n Origin router:
n iBGP speaking backbone router:
77
router bgp 64509 redistribute static route-map black-hole-trigger ! ip route 10.5.1.3 255.255.255.255 Null0 tag 66 ! route-map black-hole-trigger permit 10 match tag 66 set local-preference 1000 set community no-export set ip next-hop 192.0.2.1 !
ip route 192.0.2.1 255.255.255.255 null0
![Page 78: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/78.jpg)
ISP Network Security – RTBF p Resulting routing table entries:
78
gw1#sh ip bgp 10.5.1.3 BGP routing table entry for 10.5.1.3/32, version 64572219 Paths: (1 available, best #1, table Default-IP-Routing-Table) Not advertised to any peer Local 192.0.2.1 from 1.1.10.10 (1.1.10.10) Origin IGP, metric 0, localpref 1000, valid, internal, best Community: no-export
gw1#sh ip route 10.5.1.3 Routing entry for 10.5.1.3/32 Known via "bgp 64509", distance 200, metric 0, type internal Last update from 192.0.2.1 00:04:52 ago Routing Descriptor Blocks: * 192.0.2.1, from 1.1.10.10, 00:04:52 ago Route metric is 0, traffic share count is 1 AS Hops 0
![Page 79: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/79.jpg)
ISP Network Security – uRPF p Unicast Reverse Path Forwarding p Strongly recommended to be used on all
customer facing static interfaces n BCP 38 (tools.ietf.org/html/bcp38) n Blocks all unroutable source addresses the
customer may be using n Inexpensive way of filtering customer’s connection
(when compared with packet filters) p Can be used for multihomed connections too, but
extreme care required
79
![Page 80: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/80.jpg)
What is uRPF?
p Router compares source address of incoming packet with FIB entry n If FIB entry interface matches incoming interface, the
packet is forwarded n If FIB entry interface does not match incoming interface,
the packet is dropped 80
router
FIB: 172.16.1.0/24 fa0/0 192.168.1.0/24 se0/1
fa0/0 se0/1 src=172.16.1.1
src=192.168.1.1
![Page 81: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/81.jpg)
Security Summary p Implement RTBF
n Inside ISP backbone n Make it available to BGP customers too
p They can send you the prefix you need to block with a special community attached
p You match on that community, and set the next-hop to the null address
p Implement uRPF n For all static customers
p Use SSH for device access p Use TACACS+ for authentication
81
![Page 82: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/82.jpg)
Out of Band Management
82
![Page 83: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/83.jpg)
Out of Band Management p Not optional! p Allows access to network equipment in
times of failure p Ensures quality of service to customers
n Minimises downtime n Minimises repair time n Eases diagnostics and debugging
83
![Page 84: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/84.jpg)
Out of Band Management p OoB Example – Access server:
n modem attached to allow NOC dial in n console ports of all network equipment
connected to serial ports n LAN and/or WAN link connects to network
core, or via separate management link to NOC p Full remote control access under all
circumstances
84
![Page 85: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/85.jpg)
Out of Band Network
85 Ethernet
to the NOC
Router, switch and ISP server
consoles
(Optional) Out of band WAN link to other PoPs
Modem – access to PSTN for out of
band dialin
Equipment Rack Equipment Rack
![Page 86: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/86.jpg)
Out of Band Management p OoB Example – Statistics gathering:
n Routers are NetFlow and syslog enabled n Management data is congestion/failure
sensitive n Ensures management data integrity in case of
failure p Full remote information under all
circumstances
86
![Page 87: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/87.jpg)
Test Laboratory
87
![Page 88: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/88.jpg)
Test Laboratory p Designed to look like a typical PoP
n Operated like a typical PoP p Used to trial new services or new software
under realistic conditions p Allows discovery and fixing of potential
problems before they are introduced to the network
88
![Page 89: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/89.jpg)
Test Laboratory p Some ISPs dedicate equipment to the lab p Other ISPs “purchase ahead” so that
today’s lab equipment becomes tomorrow’s PoP equipment
p Other ISPs use lab equipment for “hot spares” in the event of hardware failure
89
![Page 90: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/90.jpg)
Test Laboratory p Can’t afford a test lab?
n Set aside one spare router and server to trial new services
n Never ever try out new hardware, software or services on the live network
p Every major ISP in the US and Europe has a test lab n It’s a serious consideration
90
![Page 91: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/91.jpg)
Operational Considerations
91
![Page 92: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/92.jpg)
Operational Considerations
92
Why design the world’s best network when you have not
thought about what operational good practices should be
implemented?
![Page 93: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/93.jpg)
Operational Considerations Maintenance p Never work on the live network, no matter how
trivial the modification may seem n Establish maintenance periods which your customers are
aware of p e.g. Tuesday 4-7am, Thursday 4-7am
p Never do maintenance on the last working day before the weekend n Unless you want to work all weekend cleaning up
p Never do maintenance on the first working day after the weekend n Unless you want to work all weekend preparing
93
![Page 94: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/94.jpg)
Operational Considerations Support p Differentiate between customer support
and the Network Operations Centre n Customer support fixes customer problems n NOC deals with and fixes backbone and
Internet related problems p Network Engineering team is last resort
n They design the next generation network, improve the routing design, implement new services, etc
n They do not and should not be doing support!
94
![Page 95: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/95.jpg)
Operational Considerations NOC Communications p NOC should know contact details for
equivalent NOCs in upstream providers and peers
95
![Page 96: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/96.jpg)
ISP Network Design Summary
96
![Page 97: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/97.jpg)
ISP Design Summary p KEEP IT SIMPLE & STUPID ! (KISS) p Simple is elegant is scalable p Use Redundancy, Security, and
Technology to make life easier for yourself p Above all, ensure quality of service for
your customers
97
![Page 98: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/98.jpg)
Why an Internet Exchange Point?
Saving money, improving QoS, encouraging a local Internet
economy
98
![Page 99: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/99.jpg)
Internet Exchange Point Why peer? p Consider a region with one ISP
n They provide internet connectivity to their customers n They have one or two international connections
p Internet grows, another ISP sets up in competition n They provide internet connectivity to their customers n They have one or two international connections
p How does traffic from customer of one ISP get to customer of the other ISP? n Via the international connections
99
![Page 100: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/100.jpg)
Internet Exchange Point Why peer? p Yes, International Connections…
n If satellite, RTT is around 550ms per hop n So local traffic takes over 1s round trip
p International bandwidth n Costs significantly more than domestic
bandwidth n Congested with local traffic n Wastes money, harms performance
100
![Page 101: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/101.jpg)
Internet Exchange Point Why peer? p Solution:
n Two competing ISPs peer with each other p Result:
n Both save money n Local traffic stays local n Better network performance, better QoS,… n More international bandwidth for expensive
international traffic n Everyone is happy
101
![Page 102: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/102.jpg)
Internet Exchange Point Why peer? p A third ISP enters the equation
n Becomes a significant player in the region n Local and international traffic goes over their
international connections p They agree to peer with the two other
ISPs n To save money n To keep local traffic local n To improve network performance, QoS,…
102
![Page 103: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/103.jpg)
Internet Exchange Point Why peer? p Peering means that the three ISPs have to
buy circuits between each other n Works for three ISPs, but adding a fourth or a
fifth means this does not scale p Solution:
n Internet Exchange Point
103
![Page 104: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/104.jpg)
Internet Exchange Point p Every participant has to buy just one
whole circuit n From their premises to the IXP
p Rather than N-1 half circuits to connect to the N-1 other ISPs n 5 ISPs have to buy 4 half circuits = 2 whole
circuits → already twice the cost of the IXP connection
104
![Page 105: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/105.jpg)
Internet Exchange Point p Solution
n Every ISP participates in the IXP n Cost is minimal – one local circuit covers all domestic
traffic n International circuits are used for just international
traffic – and backing up domestic links in case the IXP fails
p Result: n Local traffic stays local n QoS considerations for local traffic is not an issue n RTTs are typically sub 10ms n Customers enjoy the Internet experience n Local Internet economy grows rapidly
105
![Page 106: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/106.jpg)
Exchange Point Design
106
![Page 107: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/107.jpg)
IXP Design p Very simple concept:
n Ethernet switch is the interconnection media p IXP is one LAN
n Each ISP brings a router, connects it to the ethernet switch provided at the IXP
n Each ISP peers with other participants at the IXP using BGP
p Scaling this simple concept is the challenge for the larger IXPs
107
![Page 108: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/108.jpg)
Layer 2 Exchange
108
ISP 1 ISP 2 ISP 3
IXP Management Network
ISP 6 ISP 5 ISP 4
Ethernet Switch
IXP Services: Root & TLD DNS,
Routing Registry
Looking Glass, etc
![Page 109: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/109.jpg)
Layer 2 Exchange
109
ISP 1 ISP 2 ISP 3
IXP Management Network
ISP 6 ISP 5 ISP 4
Ethernet Switches
IXP Services: Root & TLD DNS,
Routing Registry
Looking Glass, etc
![Page 110: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/110.jpg)
Layer 2 Exchange p Two switches for redundancy p ISPs use dual routers for redundancy or
loadsharing p Offer services for the “common good”
n Internet portals and search engines n DNS Root & TLD, NTP servers n Routing Registry and Looking Glass
110
![Page 111: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/111.jpg)
Layer 2 Exchange p Requires neutral IXP management
n Usually funded equally by IXP participants n 24x7 cover, support, value add services
p Secure and neutral location p Configuration
n IPv4 /24 and IPv6 /64 for IXP LAN n ISPs require AS, basic IXP does not
111
![Page 112: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/112.jpg)
Layer 2 Exchange p Network Security Considerations
n LAN switch needs to be securely configured n Management routers require TACACS+
authentication, vty security n IXP services must be behind router(s) with
strong filters
112
![Page 113: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/113.jpg)
“Layer 3 IXP” p Layer 3 IXP is marketing concept used by
Transit ISPs p Real Internet Exchange Points are only
Layer 2
113
![Page 114: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/114.jpg)
IXP Design Considerations
114
![Page 115: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/115.jpg)
Exchange Point Design p The IXP Core is an Ethernet switch
n It must be a managed switch p Has superseded all other types of network
devices for an IXP n From the cheapest and smallest managed 12
or 24 port 10/100 switch n To the largest switches now handling high
densities of 10GE and 100GE interfaces
115
![Page 116: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/116.jpg)
Exchange Point Design p Each ISP participating in the IXP brings a
router to the IXP location p Router needs:
n One Ethernet port to connect to IXP switch n One WAN port to connect to the WAN media
leading back to the ISP backbone n To be able to run BGP
116
![Page 117: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/117.jpg)
Exchange Point Design p IXP switch located in one equipment rack
dedicated to IXP n Also includes other IXP operational equipment
p Routers from participant ISPs located in neighbouring/adjacent rack(s)
p Copper (UTP) connections made for 10Mbps, 100Mbps or 1Gbps connections
p Fibre used for 1Gbps, 10Gbps, 40Gbps or 100Gbps connections
117
![Page 118: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/118.jpg)
Peering p Each participant needs to run BGP
n They need their own AS number n Public ASN, NOT private ASN
p Each participant configures external BGP directly with the other participants in the IXP n Peering with all participants or
n Peering with a subset of participants
118
![Page 119: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/119.jpg)
Peering (more) p Mandatory Multi-Lateral Peering (MMLP)
n Each participant is forced to peer with every other participant as part of their IXP membership
n Has no history of success — the practice is strongly discouraged
p Multi-Lateral Peering (MLP) n Each participant peers with every other participant
(usually via a Route Server) p Bi-Lateral Peering
n Participants set up peering with each other according to their own requirements and business relationships
n This is the most common situation at IXPs today
119
![Page 120: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/120.jpg)
Routing p ISP border routers at the IXP must NOT be
configured with a default route or carry the full Internet routing table n Carrying default or full table means that this router and
the ISP network is open to abuse by non-peering IXP members
n Correct configuration is only to carry routes offered to IXP peers on the IXP peering router
p Note: Some ISPs offer transit across IX fabrics n They do so at their own risk – see above
120
![Page 121: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/121.jpg)
Routing (more) p ISP border routers at the IXP should not
be configured to carry the IXP LAN network within the IGP or iBGP n Use next-hop-self BGP concept
p Don’t generate ISP prefix aggregates on IXP peering router n If connection from backbone to IXP router goes
down, normal BGP failover will then be successful
121
![Page 122: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/122.jpg)
Address Space p Some IXPs use private addresses for the IX LAN
n Public address space means IXP network could be leaked to Internet which may be undesirable
n Because most ISPs filter RFC1918 address space, this avoids the problem
p Some IXPs use public addresses for the IX LAN n Address space available from the RIRs n IXP terms of participation often forbid the IX LAN to be
carried in the ISP member backbone
122
![Page 123: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/123.jpg)
Charging p IXPs should be run at minimal cost to participants p Examples:
n Datacentre hosts IX for free p Because ISP participants then use data centre for co-lo
services, and the datacentre benefits long term n IX operates cost recovery
p Each member pays a flat fee towards the cost of the switch, hosting, power & management
n Different pricing for different ports p One slot may handle 24 10GE ports p Or one slot may handle 96 1GE ports p 96 port 1GE card is tenth price of 24 port 10GE card p Relative port cost is passed on to participants
123
![Page 124: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/124.jpg)
Services Offered p Services offered should not compete with
member ISPs (basic IXP) n e.g. web hosting at an IXP is a bad idea unless
all members agree to it p IXP operations should make performance
and throughput statistics available to members n Use tools such as MRTG/Cacti to produce IX
throughput graphs for member (or public) information
124
![Page 125: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/125.jpg)
Services to Offer p ccTLD DNS
n the country IXP could host the country’s top level DNS n e.g. “SE.” TLD is hosted at Netnod IXes in Sweden n Offer back up of other country ccTLD DNS
p Root server n Anycast instances of I.root-servers.net, F.root-
servers.net etc are present at many IXes p Usenet News
n Usenet News is high volume n could save bandwidth to all IXP members
125
![Page 126: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/126.jpg)
Services to Offer p Route Collector
n Route collector shows the reachability information available at the exchange
p Looking Glass n One way of making the Route Collector routes
available for global view (e.g. www.traceroute.org)
n Public or members only access n Useful for members to check BGP filters n Useful for everyone to check route availability
at the IX 126
![Page 127: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/127.jpg)
Services to Offer p Route Server
n A Route Collector that also sends the prefixes it has collected to its peers
n Like a Route Collector, usually a router or Unix based system running BGP
n Does not forward packets n Useful for scaling eBGP sessions for larger IXPs n Participation needs to be optional
p And will be used by ISPs who have open peering policies
127
![Page 128: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/128.jpg)
Services to Offer p Content Redistribution/Caching
n For example, Akamised update distribution service
p Network Time Protocol n Locate a stratum 1 time source (GPS receiver,
atomic clock, etc) at IXP p Routing Registry
n Used to register the routing policy of the IXP membership
128
![Page 129: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/129.jpg)
What can go wrong? p High annual fees
n Should be cost recovery
p Charging for traffic between participants n Competes with commercial transit services
p Competing IXPs n Too expensive for ISPs to connect to all
p Too many rules & restrictions n Want all network operators to participate
p Mandatory Multi-Lateral Peering n Has no history of success
p Interconnected IXPs n Who pays for the interconnection?
p Etc… 129
![Page 130: ISP & IXP Design - Middle East NOG](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb34e32e268c58cd5b6c15/html5/thumbnails/130.jpg)
Conclusion p IXPs are technically very simple to set up p Little more than:
n An ethernet switch n Neutral secure reliable location n Consortium of members to operate it
p Political aspects can be more challenging: n Competition between ISP members n “ownership” or influence by outside parties
130