part iii: measuring inter- domain paths. march 8, 20042 packet forwarding path internet source...
TRANSCRIPT
Part III: Measuring Inter-domain Paths
March 8, 2004 2
Packet forwarding path
Internet
Source
Destination
IP traffic
Forwarding path - the path packets traverse through the Internet from a source to a destination
March 8, 2004 3
An inter-domain level view
Internet
Source
Destination
AS A
AS B
AS C
AS D
IP traffic
An IP forwarding path often span across multiple Autonomous Systems.
March 8, 2004 4
Why do we care?
Characterize end-to-end network paths
Diagnose routing anomalies Discover Internet topology
March 8, 2004 5
Why do we care?
Characterize end-to-end network paths Latency Capacity Link utilization Loss rate.
Diagnose routing anomalies Discover Internet topology
March 8, 2004 6
Varies link capacity
Internet
Source
Destination
March 8, 2004 7
Different loss rate
Internet
Source
Destination
March 8, 2004 8
Traffic engineering
Internet
Source
Destination
Customer service enhancement
March 8, 2004 9
Why do we care?
Characterize end-to-end network paths
Diagnose routing anomalies Forwarding loop, black holes, routing
changes, unexpected paths, main component of end-to-end latency.
Discover Internet topology
March 8, 2004 10
Forwarding loops
Internet
Source
Destination
March 8, 2004 11
Black holes
Internet
Source
Destination
March 8, 2004 12
Routing changes
Internet
Source
Destination
March 8, 2004 13
Unexpected routes
Internet
Source
Destination
March 8, 2004 14
Performance bottleneck
Internet
Source
Destination
March 8, 2004 15
Why do we care?
Characterize end-to-end network paths
Diagnose routing anomalies Discover Internet topology
Server placement
March 8, 2004 16
Internet topology
Internet
Client
Server
Client
Client
March 8, 2004 17
Server placement
Internet
Client
Server
Client
Client
Proxy
March 8, 2004 18
Key challenge
Need to understand how packets flow through the Internet without real-time access to proprietary routing data from each domain. Identify accurate packet forwarding
paths Characterize the performance metrics
of each hop along the paths
March 8, 2004 19
Identify forwarding path
Traceroute gives IP level forwarding path IP address of the router interfaces on
a forwarding path RTT statistics for each hop along the
way
March 8, 2004 20
Traceroute from UC Berkeley to www.cnn.com
1 169.229.62.1 2 169.229.59.225 3 128.32.255.169 4 128.32.0.249 5 128.32.0.66 6 209.247.159.109 7 * 8 64.159.1.46 9 209.247.9.17010 66.185.138.3311 * 12 66.185.136.1713 64.236.16.52
inr-daedalus-0.CS.Berkeley.EDUsoda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDUgigE6-0-0.inr-666-
doecev.Berkeley.EDUqsv-juniper--ucb-gw.calren2.netPOS1-
0.hsipaccess1.SanJose1.Level3.net??pos8-0.hsa2.Atlanta2.Level3.netpop2-atm-P0-2.atdn.net?pop1-atl-P4-0.atdn.netwww4.cnn.com
Traceroute output: (hop number, IP address, DNS name)
1 169.229.62.1 2 169.229.59.225 3 128.32.255.169 4 128.32.0.249 5 128.32.0.66 6 209.247.159.109 7 * 8 64.159.1.46 9 209.247.9.17010 66.185.138.3311 * 12 66.185.136.1713 64.236.16.52
inr-daedalus-0.CS.Berkeley.EDUsoda-cr-1-1-soda-br-6-2 vlan242.inr-202-doecev.Berkeley.EDUgigE6-0-0.inr-666-
doecev.Berkeley.EDUqsv-juniper--ucb-gw.calren2.netPOS1-
0.hsipaccess1.SanJose1.Level3.net??pos8-0.hsa2.Atlanta2.Level3.netpop2-atm-P0-2.atdn.net?pop1-atl-P4-0.atdn.netwww4.cnn.com
March 8, 2004 21
Traceroute from AT&T Research to www.cnn.com
traceroute to cnn.com (64.236.24.12), 30 hops max, 40 byte packets
1 oden (135.207.16.1) 1 ms 1 ms 1 ms 2 * * * 3 attlr-gate (192.20.225.1) 2 ms 2 ms 2 ms 4 12.119.155.157 (12.119.155.157) 3 ms 4 ms 4
ms 5 gbr6-p52.n54ny.ip.att.net (12.123.192.18) 4 ms
4 ms 4 ms 6 tbr2-p012401.n54ny.ip.att.net (12.122.11.29) 4
ms (ttl=249!) 5 ms (ttl=249!) 5 ms (ttl=249!) 7 ggr2-p390.n54ny.ip.att.net (12.123.3.62) 4 ms 5
ms 4 ms 8 att-gw.ny.aol.net (192.205.32.218) 4 ms 4 ms 4
ms 9 bb2-nye-P1-0.atdn.net (66.185.151.66) 4 ms 4
ms 4 ms10 bb2-vie-P8-0.atdn.net (66.185.152.201) 13 ms
(ttl=245!) 12 ms (ttl=245!) 12 ms (ttl=245!)11 bb1-vie-P11-0.atdn.net (66.185.152.206) 10 ms
10 ms 10 ms12 bb1-cha-P7-0.atdn.net (66.185.152.28) 20 ms
20 ms 20 ms13 bb1-atm-P6-0.atdn.net (66.185.152.182) 25 ms
25 ms 25 ms14 pop1-atl-P4-0.atdn.net (66.185.136.17) 25 ms
(ttl=243!) 24 ms (ttl=243!) 24 ms (ttl=243!)15 * * *
16 * * *17 * * *18 * * *19 * * *20 * * *21 * * *22 * * *23 * * *24 * * *25 * * *26 * * *27 * * *28 * * *29 * * *30 * * *
Who is responsible for the forwarding problem?
Destination unreachable!
March 8, 2004 22
Need to know Inter-domain level path
Internet
AT&T Research
www.cnn.com
AS A
AS B
AS C
AS D
Routing loop in AS C!
March 8, 2004 23
How to obtain AS level paths
BGP AS path Traceroute AS path
March 8, 2004 24
BGP AS path
AS AAS B
AS CPrefix d
Forwarding path: data traffic
Signaling path: control trafficd: path=[C]d: path=[BC]
Prefix AS pathd A B C… …
Is BGP AS path the answer? No!
March 8, 2004 25
BGP AS path is not the answer
Requires timely access to BGP data Signaling path may differ from
forwarding path Route aggregation and filtering Routing anomalies: e.g., deflections,
loops [Griffin2002] BGP misconfigurations: e.g., incorrect
AS prependingTwo paths may differ precisely when operators most need accurate data to diagnose a problem!
March 8, 2004 26
AS A AS B AS C AS D
Traceroute AS path
Obtain IP level path using traceroute Map IP addresses to ASes
Is traceroute AS path the answer? NO!
Source Destination
a b c d e
March 8, 2004 27
Example: UC Berkeley to CNN
1 169.229.62.1
2 169.229.59.225
3 128.32.255.169
4 128.32.0.249
5 128.32.0.66
6 209.247.159.109
7 *
8 64.159.1.46
9 209.247.9.170
10 66.185.138.33
11 *
12 66.185.136.17
13 64.236.16.52
Traceroute output: (hop number, IP)AS25
AS25
AS25
AS25
AS11423
AS3356
AS3356
AS3356
AS3356
AS1668
AS1668
AS1668
AS5662
Berkeley
CNN
Calren
Level3
GNN
March 8, 2004 28
Traceroute AS path is not the answer
Identifying ASes along forwarding path is surprisingly difficult! Internet route registry Origin AS in BGP routes
March 8, 2004 29
Internet route registry
Whois database E.g. NANOG traceroute, prtraceroute Out-of-date, incomplete
Address allocation to customers Acquisition, mergers, break-ups
March 8, 2004 30
Origin AS in BGP routes
Last AS in the AS path for each prefix
More accurate and complete than whois data
Prefix AS path
d A B C
… …
March 8, 2004 31
Limitations of BGP origin AS
Multiple Origin AS (MOAS) Infrastructure addresses may not
be advertised Addresses announced by someone
else
March 8, 2004 32
Limitations of BGP origin AS
Multiple Origin AS (MOAS) Multi-homing Misconfiguration Internet eXchange Points (IXPs)
Infrastructure addresses may not be advertised
Addresses announced by someone else
March 8, 2004 33
Limitations of BGP origin AS
Multiple Origin AS (MOAS) Infrastructure addresses may not
be advertised Does not require to be announced
publicly Security concerns
Addresses announced by someone else
March 8, 2004 34
Limitations of BGP origin AS
Multiple Origin AS (MOAS) Infrastructure addresses may not
be advertised Addresses announced by someone
else Static routed customers Shared equipments at boundary
between ASesNeed accurate IP-to-AS mapping!
March 8, 2004 35
Accurate AS-level traceroute
Combine BGP and traceroute data to find a better answer!
March 8, 2004 36
Assumptions
IP-to-AS mapping Mappings from BGP tables are mostly
correct. Change slowly
BGP paths and forwarding paths mostly match. 70% of the BGP path and traceroute
path match
March 8, 2004 37
BGP path and traceroute path could differ!
Inaccurate IP-to-AS mapping Traceroute problems Legitimate mismatches
March 8, 2004 38
BGP path and traceroute path could differ!
Inaccurate IP-to-AS mapping Internet eXchange Points (IXPs) Sibling ASes Unannounced infrastructure
addresses Traceroute problems Legitimate mismatches
March 8, 2004 39
Internet eXchange Points (IXPs)
Shared infrastructure connected to multiple service providers
Exchange BGP routes and data traffic May have its own AS number or
announced by participating ASes Dedicated BGP sessions between pairs
of participating ASes E.g., Mae-East, Mae-West, PAIX.
March 8, 2004 40
IXPs cause extra AS hop
Extra AS hop in traceroute path Large number of fan-in and fan-out
ASes Non-transit AS, small address
block, likely MOAS
March 8, 2004 41
IXPs cause extra AS hop
A
B
C
D
E
F
G
Traceroute AS path BGP AS path
B
C
F
G
A E
March 8, 2004 42
Sibling ASes
Single organization owns and manages multiple ASes
May share address space Large fan-in and fan-out for the
“sibling AS pair”
March 8, 2004 43
Sibling ASes cause extra AS hop
Large fan-in and fan-out for the “sibling AS pair”
Traceroute AS path BGP AS path
A
B
C
D
E
F
G
H
A
B
C
D
E
F
G
March 8, 2004 44
Unannounced infrastructure addresses
ASes do not necessarily announce infrastructure via BGP
Lead to “unmapped” addresses Sometimes fall into supernet
announced by AS’s provider or sibling
March 8, 2004 45
Unannounced infrastructure addresses
1. A,C
AS AAS B
AS C
2. A
3. B,A4. A,C,A
Extra AS hop in traceroute path
Missing AS hop in traceroute path
Substitute AS hop
AS loop in traceroute path
March 8, 2004 46
BGP path and traceroute path could differ!
Inaccurate IP-to-AS mapping Traceroute problems
Forwarding path changing during traceroute
Interface numbering at AS boundaries ICMP response refers to outgoing
interface Legitimate mismatches
March 8, 2004 47
Forwarding path changing during traceroute
AS A AS B AS C
AS A AS C
AS D AS E
AS D
AS hop B is substituted by AS D in the traceroute path
Route flaps between A B C and A D E
March 8, 2004 48
Interface numbering at AS boundaries
AS A AS B AS C
AS A AS C
Missing AS hop B in traceroute path
March 8, 2004 49
ICMP response refers to outgoing interface
AS B
AS A AS C
ICMPmessage
Extra AS hop B in traceroute path
March 8, 2004 50
BGP path and traceroute path could differ!
Inaccurate IP-to-AS mapping Traceroute problems Legitimate mismatches
Route aggregation and filtering Routing anomalies, e.g., deflections
March 8, 2004 51
Route aggregation/filtering
8.0.0.0/8 B C 8.0.0.0/8 C8.64.0.0/16 C D
AS B AS CAS A
Extended traceroute path due to filtering by AS B
March 8, 2004 52
Mismatch patterns and causes
Extra AS
Miss AS
AS Loop
Subst AS
Other
IXP X
Sibling ASes X X X X
Unannounced IP X X X X
Aggregation/ filtering X
Inter-AS interface X X
ICMP source address X X X X
Routing anomaly X X X X X
March 8, 2004 53
BGP and traceroute data collection
Initial mappings from origin AS of a large set of BGP tables
Traceroute pathsfrom multiple locations
•Compare•Look for known causes of mismatches
(e.g., IXP, sibling ASes)•Edit IP-to-AS mappings
(a single change explaining a large number of mismatches)
For each location:
Combine all locations:
Local BGP paths Traceroute AS pathsFor each location:
(Ignoring unstable paths)
March 8, 2004 54
Experimental methodology
200,000 destinations:d0, d1, d2, d3, d4, … d200,000
For each di
-Traceroute path-BGP path
March 8, 2004 55
Measurement setup
Eight vantage points Upstream providers: US-centric tier-1
ISPs Sweep all routable IP address
space About 200,000 IP addresses, 160,000
prefixes, 15,000 destination ASes
March 8, 2004 56
Eight vantage points
Organization Location Upstream provider
AT&T Research NJ, US UUNET, AT&T
UC Berkeley CA, US Qwest, Level3, Internet 2
PSG home network WA, US Sprint, Verio
Univ of Washington WA, US Verio, Cable&Wireless
ArosNet UT, US UUNET
Nortel ON, Canada AT&T Canada
Vineyard.NET MA, US UUNET, Sprint, Level3
Peak Web Hosting CA, US Level 3, Global Crossing, Teleglobe
Many thanks to people who let us collect data!
March 8, 2004 57
Preprocessing BGP paths
Discard prefixes with BGP paths containing Routing changes based on BGP
updates Private AS numbers (64512 - 65535) Empty AS paths (local destinations) AS loops from misconfiguration AS SET instead of AS sequence
Less than 1% prefixes affected
March 8, 2004 58
Preprocessing traceroute paths
Resolving incomplete traceroute paths Unresolved hops within a single AS map to
that AS Unmapped hops between ASes
Try match to neighboring AS using DNS, Whois Trim unresponsive (*) hops at the end
Compare with the beginning of local BGP paths MOAS at the end of paths
Assume multi-homing without BGP Validation using AT&T router
configurations More than 98% cases validated
March 8, 2004 59
Initial IP-to-AS Mapping
Whois Combined BGP tables
Resolving incomplete
s
Match 44.7% 73.2% 78.0%
Mismatch 29.4% 8.3% 9.0%
Ratio 1.5 8.8 9.0
March 8, 2004 60
Heuristics to improve mappings
Overall modification to mappings 10% IP-to-AS mappings modified 25 IXPs identified 28 pairs of sibling ASes found 1150 of the /24 prefixes shared
March 8, 2004 61
Heuristics to improve mappings
IXPs Sibling ASes
Unannounced address
space
Match 84.4% 85.9% 90.6%
Mismatch
8.7% 7.8% 3.5%
Ratio 9.7 11.0 26.0
March 8, 2004 62
Systematic optimization
Dynamic-programming and iterative improvement Initial IP-to-AS mapping derived from
BGP routing tables Identify a small number of
modifications that significantly improve the match rate.
95% match ratio, less than 3% changes, very robust
March 8, 2004 63
Optimization results
Mismatch ratio
Full initial Mapping 5.23%
Heuristically optimized mapping
3.08%
Omit 10% initial mapping 6.57%
Omit 4 probing sources 6.34%
Omit probing destinations (one probe per unique BGP path)
7.12%
March 8, 2004 64
Validation
Public data Whois/DNS data pch.net for known IXPs
Private data AS 7018
March 8, 2004 65
Validations – IXP heuristic
25 inferences: 19 confirmed Whois/DNS data confirm 18 of 25 inferences
AS5459 -- “London Internet Exchange” 198.32.176.0/24:
part of “Exchange Point Blocks”DNS name: sfba-unicast1-net.eng.paix.net
Known list from pch.net confirm 16 of 25 Missing 13 known IXPs due to
Limited number of measurement locations Mostly tier-1 US-centric providers
March 8, 2004 66
Validations – Sibling heuristic
28 inferences: all confirmed Whois for organization names (15 cases)
E.g., AS1299 and AS8233 are TeliaNet
MOAS origin ASes for several address blocks (13 cases) E.g., 148.231.0.0/16 has MOAS:
AS5677 and AS7132 (Pacific Bell Internet Services and SBC Internet Services)
March 8, 2004 67
Summary
Identify accurate AS level forwarding path improve infrastructure IP to AS
mappings Heuristics and Dynamic programming
optimization Match/mismatch ratio improvement: 8-
12 to 25-35 Reduction of incomplete paths: 18-22%
to 6-7%
March 8, 2004 68
Summary
Dependence on operational realities Most BGP routes are relatively stable Few private ASes, AS_SETs Public, routable infrastructure
addresses Routers respond with ICMP replieshttp://www.research.att.com/~jiawang/as_traceroute