tcp anycast - don’t believe the fud · proxy anycast • proxy traffic is easy to anycast! •...
TRANSCRIPT
![Page 1: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/1.jpg)
TCP Anycast - Don’t believe the FUD
Matt Levine (CacheNetworks), Barrett Lyon (BitGravity), Todd Underwood (Renesys)
Operational experience with TCP and Anycast.
1
![Page 2: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/2.jpg)
What’s (IPv4) Anycast?
• From a network perspective: nothing special.
• Just another route with multiple next-hops.
• Service(s) exist on each next-hop and respond from the anycast ip address.
2
![Page 3: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/3.jpg)
It’s the packets, stupid.
• The Perceived Problem: Anycast is not a stable environment for stateful protocols (notably TCP), especially “long lived” sessions.
• eg: High-Def Porn Downloads.
• Other presentations have made reference to existing deployments..that’s us!
• TCP Anycast not only works, it has been used in production for years.
3
![Page 4: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/4.jpg)
Anycast at CacheFly
• Deployed in 2002.
• Prefix announced on 3 continents.
• 3 POP’s in the US
• 5 “common carriers (transit)” + peering
• Effective BGP Communities from upstreams is key.
4
![Page 5: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/5.jpg)
Proxy Anycast
• Proxy traffic is easy to anycast!
• Customers are isolated on a VIP/virtual address.
• The virtual address lives over common carriers allowing even distribution of traffic.
• State is accomplished with custom hardware.
5
![Page 6: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/6.jpg)
Node Geography
• Anycast nodes that do not keep state must be geographical separated.
• Coasts and countries appear to provide enough padding for route instability.
• Nodes that are near by could possibly require state between each node if routes are unstable.
6
![Page 7: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/7.jpg)
IP Utilization
• ‘Anycast is wasteful’
• Really? How much IP Space do you need to advertise from 4 sites via unicast?
7
![Page 8: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/8.jpg)
Carriers and Peering
• For content players, having even peering and carriers is key.
• Having a European centric transit provider in the US without having the same routes in Europe could cause European traffic to home in the United States.
• Use quality global providers to keep traffic balanced.
8
![Page 9: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/9.jpg)
Carriers and Peering
• When peering...
• Keep in mind that a peer may isolate traffic to a specific anycast node.
• Try to peer with networks where it makes sense.
• Try to make sure your peers know what you’re doing, and/or have a good community set.
9
![Page 10: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/10.jpg)
Benefits of Anycast
• For content players, anycast can help with:
• Moving traffic without major impact or DNS lag.
• Provides buffers for major failures.
• Allows for simplistic traffic management, with a major (potential) performance upside.
• The ability to interface tools to traffic management.
10
![Page 11: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/11.jpg)
Data - May 9, 2006
• Renesys: Monitored changes in atomic-aggregator for a CacheFly anycast prefix.
• Keynote: Monitored availability and performance of 30k file.
• Revision3: Monitored behaviour of ‘long lived’ downloads of DiggNation videocast - Total of 7TB transfered.
11
![Page 12: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/12.jpg)
Renesys Data
• 130 BGP updates for May 9.
• Observed 34 distinct ‘POP Changes’ (monitoring atomic aggregator property on routes).
• 130 updates considered ‘quite stable prefix’.
12
![Page 13: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/13.jpg)
SJC “Event”
• Between 07:00 and 07:35 UTC - observed 98 updates, 20 aggregator changes.
• Unable to correlate these shifts with any traffic changes - most likely we don’t have a big enough sample size.
13
![Page 14: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/14.jpg)
Okay..so BGP seems ok..what about actual application stability, is TCP really stable??
14
![Page 15: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/15.jpg)
NO :( Let’s go shopping
Just Kidding
15
![Page 16: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/16.jpg)
(Short-Lived) Keynote Data
• 30k download sampled from 31 locations every 5 minutes. (or an average of 1 poll every 9.6 seconds)
• Compared against the ‘keynote business 40’
• Data Collected from May 9, 2006
16
![Page 17: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/17.jpg)
Green line: AnycastOrange: Keynote Business 40
17
![Page 18: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/18.jpg)
Analyzing Revision3 Data
• Monitored IPTV downloads for Revision3 Downloads for 24 hours (thanks, Jay).
• Methodology: Analyze packet captures - look for new TCP sessions not beginning with SYN.
• Compare that against global active connection table.
18
![Page 19: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/19.jpg)
Long-Lived DataTCP Session Length
0
200
400
600
800
1000
1200
140010
19
28
37
46
55
64
73
82
91
100
109
118
127
136
145
154
163
172
181
192
201
211
220
232
241
253
265
277
290
304
344
397
452
832
Minutes
# o
f S
ess
ion
s
Total Sessions: 683,204
Total Sessions > 10 minutes: 23,795
‘POP Switched’ Connections: 4
‘POP Switch’ failure rates:
Overall: 0.0006%
Long-Lived: 0.017%
19
![Page 20: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/20.jpg)
Anycast Gotchas
• Large-Scale Changes in provider policies can impact your traffic, and it’s up to you to figure out what changed.
• “Things that are bad” become worse, notably per-packet load balancing across provider or topological boundaries.
20
![Page 21: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/21.jpg)
Conclusions
• In our experience, stateful anycast is not inherently unstable, and failure/disconnect rates are inline with offering unicast services.
• This is counter-intuitive to some published data from previously published data.
• “Trust us, it works.” (tm)
• Widespread failures cause havok; however the internet doesn’t go crazy *that* often.
21
![Page 22: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/22.jpg)
Transitioning to IPv6
• We have a plan!
• The plan consists of being dead by the time customers demand v6.
22
![Page 23: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/23.jpg)
What you can do• Stop telling people anycast doesn’t work for
TCP if you haven’t tested it, it just makes us mad.
• If your application cannot handle TCP/IP failures gracefully, do not run anycast - in fact, don’t run it on the internet.
• Experiment
• Share your experience - we want to know if we’re crazy or not.
23
![Page 24: TCP Anycast - Don’t believe the FUD · Proxy Anycast • Proxy traffic is easy to anycast! • Customers are isolated on a VIP/virtual address. • The virtual address lives over](https://reader036.vdocuments.net/reader036/viewer/2022062602/5ede20dcad6a402d66696b15/html5/thumbnails/24.jpg)
Questions?
24