1 © 2006 cisco systems, inc. all rights reserved. poc vince fuller isp “peering”...

1© 2006 Cisco Systems, Inc. All rights reserved. POC Vince Fuller <[email protected]>

ISP “peering” Interconnection

What is it and why might I want it?

Vince Fuller [email protected]

Corporate Consulting Engineering (CE)

http://www.vaf.net/~vaf/prezos/bangkok.pdf

222© 2004 Cisco Systems, Inc. All rights reserved. POC Barry Raveendran Greene <[email protected]>

What is “peering”?

• Interconnection of Internet Service Provider (ISP) networks, usually at multiple, geographically-dispersed points, for traffic exchange between their customers

• Non-customer routes are not exchanged (usually)

• variations such as “partial transit”

• Equal exchange of value – no payments (usually)

• variations: “paid peering”, ratio-based settlements


Peering between two networks

ISP A

ISP B


Routing and traffic flow

• Each provider uses BGP to advertise list of destination for which it will carry traffic

• Each provider sends traffic for destinations learned from the other provider to that provider

• Route advertisements and traffic flow in opposite direction

• Filtering of received routes for sanity (using RIR-derived prefix lists) is recommended but not always practical

• But some obvious precautions to take, such as not accepting “bogon” routes and not accepting routes for your own infrastructure; your customer routes are a little more tricky


Traffic

Traffic

Routing and traffic flow

• ISP A advertises a list of its customers

• ISP B forwards traffic to ISP A’s customers over the interconnect

• Each must filter prefixes sent, may filter prefixes received

• Filtering may be difficult with large peers; consider uRPF (with VRF mode), bogon filters, iACLs, etc.

ISP A ISP B

Prefixes

Prefixes

Ingress FilterEgress Filter


Definition: “public peering”

• Each provider connects to third-party interconnection point (IXP)

• Co-located router at IXP

• Connection to Ethernet switch at IXP

• Fiber/circuit from provider POP to IXP

• Examples: AMS-IX, LINX, DE-CIX in Europe; PAIX/Equinix (SF, DC, LA, CHI, NYC, ATL) in U.S.; HKIX in Asia


Public peering - example

ISP A

ISP C

IXP co-lo

switch

RTR-A RTR-B

RTR-C

ISP B


Public peering - considerations

• Connection to public IXP does not guarantee routing or traffic exchange

• Agreements needed with other providers at IXP

• Separate BGP configuration for each

• IXP switch may or may not protect against “traffic dumping” by non-peers (security issue…)

• Large providers typically only appear at “major” (non-local/non-regional) IXPs

• IXP may also be used to sell transit services


Definition: “private peering”

• Two providers establish connection directly between their routers

• Historically, this was achieved by leasing point-to-point circuits

• Often now implemented by cross-connect within an IXP, providing easy upgrade path

• Dark fiber between providers also an option

• Typically only done when large traffic exchange known to exist between providers


Private peering - examples

ISP A

ISP C

IXP co-lo

switch

RTR-A RTR-B

RTR-C

ISP B

circuit or fiber

cross-connect


Private peering - considerations

• Requires more resources (one router port per peer) than public peering

• Somewhat more complicated configuration

• Offers greater control over traffic exchange

• Direct measurement of traffic volume

• Immune to “dumping” by third parties

• Easy to use CAR, filtering, other shaping, tracebacks, etc.

• N.B. that “private” doesn’t mean “private network”… still part of the public Internet

• And still have many of the same security issues…


Why do any of this? What are the benefits?

• Since no provider connects all Internet destinations, interconnections are needed

• Latency reductions by avoiding transit hops

• Cost reductions by avoiding transit provider fees


Adding new peers – before, lots of transit

You

Transit ISP

ISP A

ISP C

ISP B


Adding new peers – after, less transit/cost

You

Transit ISP

ISP A

ISP C

ISP B


What does it cost?

• Circuits or fiber to reach other provider(s)

• Interconnection point (IXP) fees

• Co-located equipment at IXP

• Line cards, dedicated routers in POP

• Staff to manage peering relationships, technical expertise to manage equipment, etc.

• Many costs scale according to # of peers


Some business considerations

• Can customers become “peers” also? May lead to service canniblization

• Good peering with another provider may make it easier for them to steal customers (works both ways…)


Peering agreements

• Agreements may be multi-lateral (MLPA, at some large, public IXPs) or bi-lateral)

• Usually must be in place before route or traffic exchange can occur

• Degree of formality varies

• Some providers will peer with a handshake (or email exchange/phone call)

• Some want something resembling a contract


Peering policies

• Different providers have different levels of willingness to peer with new entrants

• “open” – will generally establish peering with anyone at any number of IXPs; typical of content providers and small ISPs

• “selective” – formal agreement needed, may impose traffic ratio requirements, multiple IXPs across continent, possibly multiple continents; many mid-sized ISPs

• “restrictive” – reluctant to add new peers, ratios/settlements are issues, only connects to “true peers” of similar continental/intercontinental size/scope; “tier-1” ISPs


What’s a “good” peering policy?

• Business decision, based on budget, what level of service to offer to customers, etc.

• How expensive (in people, equipment, etc.) is it to collect the information needed to implement settlements?

• How expensive is it to simply over-provision the interconnects and the backbone network?

• How much leverage does a provider have with other providers?

• Answers vary by provider


Settlements – motivation

• Traffic exchange at interconnections is generally “shortest exit” (aka “hot potato”)

• Sender hands-off traffic at closest IXP to source

• Receiver carries on (often) longer path from IXP to destination

• Bulk of transport cost is on receiver

• If traffic exchange is not approximately balanced, provider receiving more traffic is bearing higher costs

• Leads to desire for cost recovery/sharing by provider which receives a greater share of traffic

• “Net Neutrality” is a new name for this old issue - it dates back to early days of Internet commercialization (late 1980s)


Shortest-exit traffic flow

ISP A

ISP B

user request

server response

10.22.0.0/16

172.17.0.0/16

10.22.9.101 10.22.4.54

10.22.0.0/1610.22.0.0/16

10.22.0.0/16

172.17.0.0/16

172.17.0.0/16 172.17.0.0/16

172.17.21.22


Settlements – implementation

• Total traffic rates sent and received is measured between two peering providers; typically, 95th percentile calculation is used

• Charge from net receiver to net sender based on this “traffic ratio”

• Traffic ratio may also be used in peering negotiations and agreements, e.g. ISP A will only peer with ISP B if the traffic ratio between them is less than 2.0


Settlements – non-cash alternative?

• Can the transport cost be shifted from the receiver to the sender through routing?

• “best exit” or “cold potato” routing attempts to do this

• Receiver advertises BGP Multi-Exit Discriminator in routes to sender

• Sender uses exit point which is closest to the receiver’s customer

• Nice idea, but doesn’t scale well in general case of multiple peering points (simple if only one peering point)

• Requires de-aggregation of routing information

• To avoid breaking CIDR, this can be done selectively but that introduces possible routing inconsistencies that can leak in unexpected ways - increased trust requirement

• Trust/verification that MEDs are being honored to offload receiver network (Netflow analysis helps here)


Best-exit routing and traffic flow

ISP A

ISP B

user request

server response

10.22.0.0/16

172.17.0.0/16

10.22.9.101

172.17.21.22

10.22.9.0/24 MED=5

10.22.4.54

10.22.0.0/16

10.22.4.0/24 MED=20 10.22.9.0/24 MED=1010.22.0.0/16

10.22.4.0/24 MED=10

10.22.9.0/24 MED=2010.22.0.0/16

10.22.4.0/24 MED=5

172.17.0.0/16

172.17.0.0/16 172.17.0.0/16


Peering capacity planning

• How to figure out with whom to peer?

• Where (to which other providers) is customer traffic going?

• Is there a better (shorter, lower latency, and/or less expensive) path to get to those destinations?

• Is there an easy way to obtain this information to establish better paths?


Capacity planning – how to

• Set up Netflow collection server (i.e. Cisco Netflow Collector, cflowd, flowd, …)

• Enable Netflow export on transit and peering edges of the network

• Use analysis tools (i.e. FlorScan, ASFlow, Arbor, …) to find next- and 2nd-hop ASNs with large flows

• Seek peering with those 2nd-hop ASNs to reduce hops (and possible transit cost) through next-hop ASNs

• Pointers to tools in “more reading” section


Capacity planning – how to (contd)

• OK, so you’ve figured out where traffic is going… how to best get it there?

• If public peering isn’t already in place with the target ASNs, consider establishing it

• If public peering exists but is running out of capacity, consider offloading to private peering, possibly by cross-connect within IXPs where a presence already exists


Tricks of the trade – negotiation

• What if the other guy doesn’t want to peer with you or you can’t otherwise get the set-up that seems best?

• Try various forms of persuasion…• Establish personal relationships with coordinators

• Socialize at ops conferences (APRICOT, RIPE, NANOG)

• What do you have that they need (consider services other than IP transit – co-lo, fiber, etc. in your market)?

• Do you have leverage in one market that you can trade?

• Small gifts don’t hurt (original BBN->Sprint interconect was facilitated by some Belgian chocolate…)

• See the “more reading” references for some ideas of what else to try


Finally… what about QoS?

• Today, “quality of service” on inter-provider peering generally means over-provisioning to keep everything uncongested

• There has been a fair amount of discussion of marking inter-provider traffic with appropriate QoS bits but not much real action

• Issues include:

• Policing the QoS settings - rate-limiting “high” QoS and re-marking received traffic

• Standardizing on the meaning of the QoS values

• Dealing with the added complexity

• Right now, over-provisioning seems to be working pretty well - that’s how inter-provider SLAs are generally met today

• And does it really make sense to build congested peering points? QoS does not create bandwidth, it just determines which traffic is dropped when there isn’t enough of it`


• Stateful inspection firewalls

• Flow inspection firewalls

• QoS policy enforcement

• Well planned IP addressing strategies

• Infrastructure ACLs (iACLs)

• Receive-path ACLs (rACLs), Control Plane Policing (CoPP)

• Selective Hardware-based Rate Limiting techniques

• BTSH/GTSM

What about security?

• Access Control Lists (ACLs)

• Unicast Reverse Path Forwarding (uRPF)

• Black Holes (including remotely-triggered)

• Sink Holes (including remotely)

• Shunts to packet clean-up boxes (including remotely)

• Rate-limiting (including remotely-triggered)

• Re-coloring packets

• MD5 for BGP/OSPF/ISIS

Peering creates very large new “edges” on your network; tools for mitigating security risk at the edge apply, including:


More reading – Netflow tools

Collection servers• Cisco Netflow Collector http://www.cisco.com/en/US/products/sw/netmgtsw/ps1964/index.html• cflowd http://www.caida.org/tools/measurement/cflowd/

• flow-tools http://www.splintered.net/sw/flow-tools/

• others…

Analysis tools:• Cisco Netflow Collector• ASFlow http://sourceforge.net/projects/asflow • FlowScan www.caida.org/tools/utilities/flowscan/• Arbor http://www.arbornetworks.com• others…

Much more information at: http://www.cisco.com/en/US/products/ps6601/products_white_paper0900aecd80406232.shtml


More reading – Bill Norton’s “white papers”

• “Interconnection Strategies for ISPs”http://www.equinix.com/pdf/whitepapers/ISPInterconnectionStrategies2.pdf

• “Internet Service Providers and Peering“http://www.equinix.com/pdf/whitepapers/PeeringWP.2.pdf

• "A Business Case for Peering“

http://www.equinix.com/pdf/whitepapers/Business_case.pdf

• "Evolution of the U.S. Peering Ecosystem v1.1“

http://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdf

1 © 2006 cisco systems, inc. all rights reserved. poc vince fuller isp “peering”...

Documents