trouble shooting cernet noc pengfei li. 2 contents trouble shooting tools trouble shooting bgp...
TRANSCRIPT
Trouble Shooting
CERNET NOCPengfei Li
2
Contents
Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting CEF Trouble Shooting IP Multicast Case Studies
Trouble Shooting Tools
Key of trouble shooting –Excellent NMS
4
Route Monitoring
5
Routing (BGP summary)
6
Routing Mornitoring
7
BGP Statistics (current status)
8
Looking-glass
9
BGP Monitoring (TEIN2-NORTH)
10
BGP Monitoring (TEIN2-SOUTH)
11
BGP Monitoring (TEIN2-JP)
12
AS Path Entries
13
Community Entries
14
IPv4 Prefix
15
IPv6 Prefix
16
Basic tools
Ping Traceroute telnet Show commands etc
Trouble Shooting BGP
18
BGP in Large Scale Networks
19
Avoid the Problem in theFirst Place Use simple configurations maintain a consistent policy througho
ut the AS Promote stable networks nail-down your routes use loopback in
terfaces Grow into your network use peer-groups and RRs for scalability
20
Agenda
Basic Tools Peer Establishment UPDATE Exchange Selection Algorithm Route Reflectors Route Flap Damping
21
BGP Troubleshooting Tools
show commands debug output Log messages
22
show Commandsrouter#show ip bgp ? A.B.C.D IP prefix <network>/<length>, e.g., 35.0.0.0/8 A.B.C.D Network in the BGP routing table to display cidr-only Display only routes with non-natural netmasks community Display routes matching the communities community-list Display routes matching the community-list dampened-paths Display paths suppressed due to dampening filter-list Display routes conforming to the filter-list flap-statistics Display flap statistics of routes inconsistent-as Display only routes with inconsistent origin ASs neighbors Detailed information on TCP and BGP neighbor connections paths Path information peer-group Display information on peer-groups quote-regexp Display routes matching the AS path "regular expression" regexp Display routes matching the AS path regular expression summary Summary of BGP neighbor status | Output modifiers <cr>
23
show Commands (Cont.)
router#show ip bgp neighbors x.x.x.x ? advertised-routes Display the routes advertised to a BGP neighbor dampened-routes Display the dampened routes received from neighbor flap-statistics Display flap statistics of the routes learned from
neighbor paths Display AS paths learned from neighbor received Display information received from a BGP neighbor received-routes Display the received routes from neighbor routes Display routes learned from neighbor | Output modifiers <cr>
24
router#show ip bgp BGP table version is 9, local router ID is 7.72.6.1 Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> 3.0.0.0 0.0.0.0 0 32768 i *> 5.0.0.0 0.0.0.0 0 32768 i *> 6.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i *> 7.0.0.0 0.0.0.0 0 32768 i *> 8.0.0.0/5 0.0.0.0 0 32768 i *> 17.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i *> 23.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i *> 35.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i
The BGP Table
25
The BGP Table (Cont.)
router#show ip bgp 6.0.0.0 BGP routing table entry for 6.0.0.0/8, version 2Paths: (2 available, best #1) Advertised to non peer-group peers: 7.25.14.4 7.72.6.3 7.75.7.1 2 6.72.6.2 from 6.72.6.2 (7.72.6.2) Origin IGP, metric 4294967294, localpref 100, valid, external, best 2 6.72.6.2 from 7.75.7.1 (7.75.7.1) Origin IGP, metric 4294967294, localpref 100, valid, internal
26
show ip bgp Summary
router#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1BGP table version is 9, main routing table version 98 network entries and 12 paths using 1176 bytes of memory3 BGP path attribute entries using 144 bytes of memory1 BGP AS-PATH entries using 24 bytes of memoryBGP activity 8/0 prefixes, 12/0 paths
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 6885 6882 9 0 0 4d18h 47.25.14.4 4 3 6882 6883 9 0 0 4d18h 07.72.6.3 4 1 6880 6886 9 0 0 4d18h 07.75.7.1 4 1 6884 6885 9 0 0 4d18h 4
27
show ip bgp neighborsrouter#show ip bgp neighbors 6.72.6.2BGP neighbor is 6.72.6.2, remote AS 2, external link Index 1, Offset 0, Mask 0x2 BGP version 4, remote router ID 7.72.6.2 BGP state = Established, table version = 9, up for 4d21h Last read 00:00:56, last send 00:00:48 Hold time 180, keepalive interval 60 seconds Neighbor NLRI negotiation: Configured for unicast routes only Peer negotiated unicast and multicast routes Exchanging unicast routes only Received route refresh capability from peer Minimum time between advertisement runs is 30 seconds Received 7044 messages, 0 notifications, 0 in queue Sent 7041 messages, 0 notifications, 0 in queue Prefix advertised 4, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Inbound path policy configured Route map for incoming advertisements is k Connections established 1; dropped 0 Last reset never Number of unicast/multicast prefixes received 4/0 External BGP neighbor may be up to 255 hops away.Connection state is ESTAB, I/O status: 1, unread input bytes: 0Local host: 3.72.6.1, Local port: 179Foreign host: 6.72.6.2, Foreign port: 11014
28
router#debug ip bgp ? A.B.C.D BGP neighbor address dampeningBGP dampening events BGP events keepalives BGP keepalives updates BGP updates <cr>
debug ip bgp
Remember—can be dangerous! Use only in the lab or If advised by the TAC
To make a little safer: logging buffered <size> no logging console
29
Session Establishment (debug ip bgp )
16:06:30: BGP: 7.72.6.1 sending OPEN, version 416:06:31: BGP: 7.72.6.1 OPEN rcvd, version 416:06:31: BGP: 7.72.6.1 rcv OPEN w/ OPTION parameter len: 1216:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 616:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 1, length 416:06:31: BGP: 7.72.6.1 OPEN has MP_EXT CAP for afi/safi: 1/116:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 216:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 128, length 0
16:06:31: BGP: 7.75.7.1 passive open16:06:31: BGP: 7.75.7.1 OPEN rcvd, version 416:06:31: BGP: 7.75.7.1 sending OPEN, version 416:06:31: BGP: 7.75.7.1 rcv OPEN w/ OPTION parameter len: 1216:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 616:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 1, length 416:06:31: BGP: 7.75.7.1 OPEN has MP_EXT CAP for afi/safi: 1/116:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 216:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 128, length 0
30
Session Establishment(debug ip bgp events)
17:31:39: BGP: 7.72.6.1 went from Idle to Active17:32:00: BGP: 7.72.6.1 went from Active to OpenSent17:32:00: BGP: 7.72.6.1 went from OpenSent to OpenConfirm17:32:00: BGP: 7.72.6.1 went from OpenConfirm to Established
17:31:59: BGP: 7.75.7.1 went from Idle to Active17:32:00: BGP: 7.75.7.1 went from Active to Idle17:32:00: BGP: 7.75.7.1 went from Idle to Connect17:32:00: BGP: 7.75.7.1 went from Connect to OpenSent17:32:00: BGP: 7.75.7.1 went from OpenSent to OpenConfirm17:32:00: BGP: 7.75.7.1 went from OpenConfirm to Established
31
Looking at the Updates
router#debug ip bgp updates? <1-199> Access list <1300-2699> Access list (expanded range) <cr>
router#debug ip bgp x.x.x.x updates? <1-199> Access list <1300-2699> Access list (expanded range) <cr>
Use an access-list to limit the output!Use an access-list to limit the output!
32
debug ip bgp Updates
BGP: 6.72.6.2 computing updates, neighbor version 0, table version 13, starting at 0.0.0.0BGP: 6.72.6.2 send UPDATE 3.0.0.0/8, next 3.72.6.1BGP: , metric 0, path 1BGP: 6.72.6.2 send UPDATE 5.0.0.0/8 (chgflags: 0x0), next 3.72.6.1BGP: 6.72.6.2 send UPDATE 7.0.0.0/8 (chgflags: 0x0), next 3.72.6.1BGP: 6.72.6.2 1 updates enqueued (average=56, maximum=56)BGP: 6.72.6.2 update run completed, ran for 0ms, neighbor version 0, start version 13, throttled to 13, check point net 0.0.0.0
Peer Address Prefix Being Advertised NEXT_HOP
33
debug ip bgp Updates (Cont.)
BGP: 6.72.6.2 rcv UPDATE w/ attr: nexthop 6.72.6.2, origin i, metric 294, path 2 1BGP: 6.72.6.2 rcv UPDATE about 3.0.0.0/8 -- DENIED due to: as-path contains our own AS;BGP: 6.72.6.2 rcv UPDATE about 7.0.0.0/8 -- DENIED due to: as-path
contains our own AS;
BGP: 6.72.6.2 rcv UPDATE w/ attr: nexthop 6.72.6.2, origin i, metric 494, path 2BGP: 6.72.6.2 rcv UPDATE about 6.0.0.0/8BGP: 6.72.6.2 rcv UPDATE about 17.0.0.0/8BGP: 6.72.6.2 rcv UPDATE about 23.0.0.0/8BGP: 6.72.6.2 rcv UPDATE about 35.0.0.0/8
Prefixes in the Same UPDATE
Attributes Apply to
All Prefixes
Peer Address
34
Logging Neighbor Changes
Generate a log message whenever a BGP neighbor changes state, also indicate reason for reset
Syntax (router subcommand): [no] bgp log-neighbor-changes
Typical log messages: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up %BGP-5-ADJCHANGE: neighbor x.x.x.x Down-Remote AS
changed
35
show ip bgp neighbors x.x.x.x
router#show ip bgp neighbors 7.75.7.1BGP neighbor is 7.75.7.1, remote AS 2, external link...Received 194 messages, 1 notifications, 0 in queue Sent 194 messages, 0 notifications, 0 in queue Prefix advertised 0, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Connections established 7; dropped 7 Last reset 00:04:11, due to BGP Notification received, hold time expired Number of unicast/multicast prefixes received 0/0 External BGP neighbor may be up to 255 hops away. No active TCP connection
36
show ip bgp neighbors x.x.x.x cont. "BGP protocol initialization" "No memory" "BGP Notification received" "BGP Notification sent" "User reset" "Peer timeout” "Password change” "Error during connection collision" "Peer closed the session" "Peer over prefix limit" "Interface flap" "Router ID changed” "Neighbor deleted" "Member added to peergroup" "Admin. shutdown" "Remote AS changed" "Capability changed" "RR client config change” "Soft reconfig change" "Local AS change"
37
Common Problems
Sessions are not establishedNo IP reachabilityIncorrect configuration
Peering addressesOPEN parameters
38
Can’t Establish Session - Symptoms
The peering session is not established! State may change between active,
idle and connect
routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1 BGP table version is 4, main routing table version 4 6 network entries and 6 paths using 774 bytes of memory 2 BGP path attribute entries using 96 bytes of memory 1 BGP AS-PATH entries using 24 bytes of memory BGP activity 6/0 prefixes, 6/0 paths
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 0 0 0 0 0 never Idle7.25.14.4 4 3 4 5 4 0 0 00:01:43 07.72.6.3 4 1 0 0 0 0 0 never Active7.75.7.1 4 1 7 5 4 0 0 00:01:55 3
39
The BGP Finite State Machine has 5 states:
Idle: no resources are allocated Connect: waiting for TCP session to be establish
ed; no connection is established actively, but one may be accepted
Active: actively trying to establish a TCP connection
OpenSent: waiting for OPEN message from peer OpenConfirm: waiting for confirmation of the O
PEN message (KEEPALIVE or NOTIFICATION)Established: UPDATES may now be exchanged…t
his is the normal operational state of BGP.
40
Can’t Establish Session— Troubleshooting I
Is the remote-as assigned correctly?
router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 7.72.6.3 remote-as 1
Local AS eBGP Peer
iBGP Peer
41
Can’t Establish Session— Troubleshooting I (Cont.)
Verify IP connectivity check the routing table use ping/trace to verify two way reachability inspect for ACLs in the path to the neighbor
routerA#show ip route 7.72.6.3Routing entry for 7.72.6.3/32 Known via "ospf 123”, distance 110, metric 87, type intra area Last update from 27.27.27.254 on POS5/0, 00:09:33 ago Routing Descriptor Blocks: * 27.27.27.254, from 7.72.6.3, 00:09:33 ago, via POS5/0 Route metric is 87, traffic share count is 1
routerA#ping 7.72.6.3Sending 5, 100-byte ICMP Echos to 7.72.6.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 28/30/32 ms
42
Can’t Establish Session— Troubleshooting I (Cont.)
routerA#debug ip bgpBGP debugging is on10:51:02: BGP: 7.72.6.3 open active, delay 6864ms10:51:09: BGP: 7.72.6.3 open active, local address 27.27.27.25310:51:09: BGP: 7.72.6.3 open failed: Connection refused by remote host
• Is the remote router configured for BGP?What IP address is the remote router configured to receive?
router bgp 1 no synchronization bgp log-neighbor-changes neighbor 7.72.6.1 remote-as 1
43
The TCP session is alwaysThe TCP session is alwayssourced from thesourced from the closestclosest IP IP address to the destination!address to the destination!
Can’t Establish Session— Troubleshooting I (Cont.)Can’t Establish Session— Troubleshooting I (Cont.)
Configuration:Router Arouter bgp 1neighbor 27.27.27.254 remote-as 1Router Crouter bgp 1neighbor 27.27.27.253 remote-as 1
A C
27.27.27.254
27.27.27.253
If redundant paths exist, If redundant paths exist, useuse loopback interfacesloopback interfaces to to
establish the session.establish the session.
44
Can’t Establish Session— Troubleshooting I (Cont.)
Solution: make sure both routers source the
information from the appropriate interface
routerA#debug ip tcp transactions11:19:48: BGP: 7.72.6.3 open active, delay 9916ms11:19:53: TCP: sending RST, seq 0, ack 309812912111:19:53: TCP: sent RST to 7.7.7.6:11719 from 7.72.6.1:179
router bgp 1 neighbor 7.72.6.3 remote-as 1 neighbor 7.72.6.3 update-source Loopback0
Information sourcedInformation sourcedfrom the IP address in from the IP address in interface Loopback0interface Loopback0
45
Can’t Establish Session—Symptoms
The eBGP session is still having trouble!
routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1BGP table version is 4, main routing table version 46 network entries and 6 paths using 774 bytes of memory2 BGP path attribute entries using 96 bytes of memory1 BGP AS-PATH entries using 24 bytes of memoryBGP activity 6/0 prefixes, 6/0 paths
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 0 0 0 0 0 never Idle7.25.14.4 4 3 385 385 4 0 0 06:22:17 07.72.6.3 4 1 42 49 4 0 0 00:00:15 07.75.7.1 4 1 388 385 4 0 0 06:22:30 3
46
routerA#configure terminal Enter configuration commands, one per line. End with CNTL/Z.routerA(config)#ip route 6.72.6.2 255.255.255.255 1.1.1.5
routerA#ping 6.72.6.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 6.72.6.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Can’t Establish Session - Troubleshooting II
Verify IP connectivity check the routing table use ping/trace to verify two way
reachabilityrouterA#show ip route 6.72.6.2%Network not in table
47
Can’t Establish Session— Troubleshooting II (Cont.)
Peering with a loopback interface Advantages
Interface is always up Multiple physical paths may
exist to reach it Disadvantages
Physical link failure may take longer to detect
48
Can’t Establish Session— Troubleshooting II (Cont.)
The debug output indicates the neighbor’s configured peering address
routerA#debug ip bgprouterA#debug ip tcp transactions13:25:30: TCP: sending RST, seq 0, ack 203010066913:25:30: TCP: sent RST to 6.72.6.2:11041 from 3.72.6.1:179
Neighbor is tryingNeighbor is tryingto peer with this IPto peer with this IP
addressaddress
router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 6.72.6.2 update-source Loopback1
49
Can’t Establish Session— Troubleshooting II (Cont.)
Hint: by default, eBGP peers should be directly connected in this case, the peering address
doesn’t match a connected interface in the local router
13:33:30: TCP: sending RST, seq 0, ack 251012964513:33:30: TCP: sent RST to 6.72.6.2:11045 from 3.72.6.1:179
50
Can’t Establish Session— Troubleshooting II (Cont.)
routerA#show ip bgp neighbors 6.72.6.2BGP neighbor is 6.72.6.2, remote AS 2, external link Index 1, Offset 0, Mask 0x2 BGP version 4, remote router ID 0.0.0.0 BGP state = Idle, table version = 0 Last read 00:00:06, last send never Hold time 180, keepalive interval 60 seconds Neighbor NLRI negotiation: Configured for unicast routes only Minimum time between advertisement runs is 30 seconds Received 0 messages, 0 notifications, 0 in queue Sent 0 messages, 0 notifications, 0 in queue Prefix advertised 0, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Connections established 0; dropped 0 Last reset never Number of unicast/multicast prefixes received 0/0 External BGP neighbor not directly connected. No active TCP connection
51
Can’t Establish Session— Troubleshooting II (Cont.)
At this point, the session should come up
router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 6.72.6.2 ebgp-multihop 255 neighbor 6.72.6.2 update-source Loopback1
52
Can’t Establish Session— Symptoms
Still having trouble! Connectivity issues have already been checked and corrected.
routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1… Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 10 26 0 0 0 never Active
router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 6.72.6.2 ebgp-multihop 255 neighbor 6.72.6.2 update-source Loopback1
53
Can’t Establish Session— Troubleshooting II (Cont.)
If an error is detected, a notification is sent and the session is closed In this case the remote router had
a bad configuration
14:06:37: BGP: 6.72.6.2 open active, local address 3.72.6.114:06:37: BGP: 6.72.6.2 went from Active to OpenSent14:06:37: BGP: 6.72.6.2 sending OPEN, version 414:06:37: BGP: 6.72.6.2 received NOTIFICATION 2/2
(peer in wrong AS) 2 bytes 000114:06:37: BGP: 6.72.6.2 remote close, state CLOSEWAIT14:06:37: BGP: service reset requests14:06:37: BGP: 6.72.6.2 went from OpenSent to Idle14:06:37: BGP: 6.72.6.2 closing
54
UPDATE Exchange
Once the session has been established, UPDATEs are exchanged all the locally known routes only the bestpath is advertised
Incremental UPDATE messages are exchanged afterwards
55
Propagation Decisions
bestpath received from eBGP peer advertise to all peers
bestpath received from iBGP peer advertise only to eBGP peers a full iBGP mesh must exist
56
Common Problems
Missing routes No iBGP full mesh Filters: routes are not received/sent
Slow convergence
57
Missing Routes—Troubleshooting Steps
Determine which filters are applied
to the BGP session show ip bgp neighbors x.x.x.x Look at the configuration
Examine the route and pick out the relevant attributes show ip bgp x.x.x.x
58
Missing Routes—Troubleshooting Steps (Cont.)
Compare the route against the filters
If no match is found Use route-refresh or soft-
reconfiguration Filter the updates through an ACL to
determine where the problem is
59
Missing Routes—Symptoms
Missing 4.0.0.0/8 in 7.75.7.1 (routerA) not received from 7.72.6.3 (routerB)
routerB#sh ip bgp nei 7.75.7.1 advertised-routes | include 4.0.0.0*> 4.0.0.0 0.0.0.0 0 32768 i
routerB shows that the route was advertised to routerA!
60
Missing Routes—Troubleshooting
routerA#show access-lists 10Standard IP access list 10 permit 4.0.0.0
routerA#debug ip bgp 7.72.6.3 updates 10BGP updates debugging is on for access list 10 for neighbor 7.72.6.3
routerA#clear ip bgp 7.72.6.3 in01:22:41: BGP: 7.72.6.3 rcv UPDATE w/ attr: nexthop 7.72.6.3, origin i, metric 0, path 201:22:41: BGP: 7.72.6.3 rcv UPDATE about 4.0.0.0/8 -- DENIED due
to: distribute/prefix-list;
61
Missing Routes—Troubleshooting (Cont.)
router bgp 1 no synchronization bgp log-neighbor-changes neighbor 7.72.6.3 remote-as 2 neighbor 7.72.6.3 ebgp-multihop 255 neighbor 7.72.6.3 update-source Loopback0 neighbor 7.72.6.3 prefix-list filter in!ip prefix-list filter seq 5 deny 4.0.0.0/8ip prefix-list filter seq 10 permit 0.0.0.0/0 le 32
62
Common Problems
Inconsistent decision/policy MED External paths Communities By default, communities are not
propagated neighbor x.x.x.x send-community
63
Inconsistent Decision—Symptom I
The bestpath changes every time the peering is reset.
routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 40Paths: (3 available, best #3, advertised over IBGP, EBGP) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external, best
64
Inconsistent Decision—Symptom I (Cont.)
routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 2Paths: (3 available, best #3, advertised over EBGP) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal, best
Same paths, but different result!
65
Inconsistent Decision—Symptom I (Cont.)
Different result…again!!
routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 12Paths: (3 available, best #3, advertised over EBGP) 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal, best
66
Deterministic MED
By default, the prefixes are compared in order of arrival it may result in inconsistent decisions use bgp deterministic-med the bestpath is recalculated as soon
as the command is entered enable in all the routers in the AS
67
Deterministic MED—Operation
The paths are ordered by peer-AS The bestpath for each group
is selected The overall bestpath results from
comparing the winners in each group
68
Deterministic MED—Result
The bestpath will always be the same!
routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 15Paths: (3 available, best #1, advertised over EBGP) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal, best 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external
69
Inconsistent Decision—Symptom II
The bestpath changes every time the peering is reset
routerA#show ip bgp 7.0.0.0BGP routing table entry for 7.0.0.0/8, version 15Paths: (2 available, best #2) Not advertised to any peer 2 1.1.1.5 from 1.1.1.5 (1.1.1.1) Origin IGP, metric 0, localpref 100, valid, external 2 21.21.21.254 from 21.21.21.254 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, external, best
70
Inconsistent Decision—Symptom II (Cont.)
The “oldest” external is the bestpath. All other attributes are the same Stability enhancement!
routerA#show ip bgp 7.0.0.0 BGP routing table entry for 7.0.0.0/8, version 17Paths: (2 available, best #2) Not advertised to any peer 2 21.21.21.254 from 21.21.21.254 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, external 2 1.1.1.5 from 1.1.1.5 (1.1.1.1) Origin IGP, metric 0, localpref 100, valid, external, best
71
Routes Flapping BGP learned routes change periodically res
ulting in high cpu BGP prefixes bounces between a suboptim
al path through a different provider and a recursive route
Recursive routing occurs when the next hop address of a network is an address of that network itself
72
Routes Flapping (Contd..)
73
Routes Flapping (Contd..) The DMZ address space between AS1
and AS3 (161.108.0.0/16) also belongs to AS3. The address 161.108.0.3 is the next hop address advertised by AS3-EBGP toward AS1-IBGP.
Router AS1-IBGP is NOT advertising the DMZ address space into AS1 via the IGP. This in combination with not having next-hop-self configured
74
Routes Flapping (Contd..)
S2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 0 Paths: (1 available, no best path) 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal
75
Routes Flapping (Contd..)
S2-R3-2500#show ip bgp 161.108.0.0 BGP routing table entry for 161.108.0.0/16, version 0 Paths: (1 available, no best path) 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal
76
Routes Flapping (Contd..)
After EBGP session comes upS2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 4 Paths: (2 available, best #1, advertised over IBGP) 2 3 131.108.50.1 from 131.108.50.1 (10.1.1.1) Origin IGP, localpref 100, valid, external, best 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal S2-R3-2500#
77
Routes Flapping (Contd..)
The next time that BGP scans the routing table, we have a route to thenext hop 161.108.0.3 via AS2 S2-R3-2500# BGP: scanning routing tables RT: del 161.108.0.0 via 131.108.50.1, bgp metric [20/0] RT: delete network route to 161.108.0.0 RT: add 161.108.0.0/16 via 161.108.0.3, bgp metric [200/0] RT: del 192.168.1.0 via 131.108.50.1, bgp metric [20/0] RT: delete network route to 192.168.1.0 RT: add 192.168.1.0/24 via 161.108.0.3, bgp metric [200/0]
78
Routes Flapping (Contd..)
These routes, once installed This inconsistency is discovered the next time that BGP computes an update for his AS2 neighbor, are recursive.
S2-R3-2500# BGP: 131.108.50.1 computing updates, neighbor version 2, table
version 10, starting at 0.0.0.0 RT: recursion error routing 161.108.0.3 - probable routing loop RT: recursion error routing 161.108.0.3 - probable routing loop BGP: 131.108.50.1 update run completed, ran for 8ms, neighbor
version 2, start version 10, throttled to 10, check point net 0.0.0.0
79
Routes Flapping (Contd..)
These unreachable routes will be marked appropiately the next time BGP scans the routing table and will be removed.
S2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 10 Paths: (2 available, best #2, advertised over IBGP, EBGP) 2 3 131.108.50.1 from 131.108.50.1 (10.1.1.1) Origin IGP, localpref 100, valid, external 3 161.108.0.3 from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal, best
80
Routes Flapping (Contd..)
We can see that the routes are changed and the routes via the next hop address of 161.108.0.3 are removed the next time that BGP scans:
BGP: scanning routing tables RT: del 161.108.0.0 via 161.108.0.3, bgp metric [200/0] RT: delete network route to 161.108.0.0 RT: add 161.108.0.0/16 via 131.108.50.1, bgp metric [20/0] RT: del 192.168.1.0 via 161.108.0.3, bgp metric [200/0] RT: delete network route to 192.168.1.0 RT: add 192.168.1.0/24 via 131.108.50.1, bgp metric [20/0] S2-R3-2500#
81
Routes Flapping (Contd..)
S2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 12 Paths: (2 available, best #1, advertised over IBGP, EBGP) 2 3 131.108.50.1 from 131.108.50.1 (10.1.1.1) Origin IGP, localpref 100, valid, external, best 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal
82
BGP Case Studies
© 2002, Cisco Systems, Inc. All rights reserved. 82
Case 1 : Peer Establishment
Missing RoutesInconsistent Route Selection
Loops and Convergence IssuesAdd CERNET CASE
84
Peer Establishment
Routers establish a TCP session Port 179—permit in ACLs IP connectivity (route from IGP)
OPEN messages are exchanged Peering addresses must match the
TCP session Local AS configuration parameters
85
Common Problems
Sessions are not established No IP reachability Incorrect configuration
Peers are flapping Layer 2 problems
86
Peer Establishment—Diagram
R2#sh run | begin bgp router bgp 1 bgp log-neighbor-changes Neighbor 1.1.1.1 remote-as 1 Neighbor 3.3.3.3 remote-as 2
AS 1
AS 2
R1R1
IBGPEBGP
1.1.1.1 2.2.2.2
3.3.3.3?
?
R2R2
R3R3
87
R2#show ip bgp summary
BGP router identifier 2.2.2.2, local AS number 1
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State
1.1.1.1 4 1 0 0 0 0 0 never Active
3.3.3.3 4 2 0 0 0 0 0 never Idle
Peer Establishment—Symptoms
Both peers are having problems State may change between active,
idle and connect
88
Peer Establishment
Is the remote-as assigned correctly? Verify with your diagram or other
documentation!
R2#router bgp 1 neighbor 1.1.1.1 remote-as 1 neighbor 3.3.3.3 remote-as 2
Local AS
eBGP Peer
iBGP Peer
89
Peer Establishment—IBGP
Assume that IP connectivity has been checked Check TCP to find out what connections we are accepting
R2#show tcp brief allTCB Local Address Foreign Address (state)005F2934 *.179 3.3.3.3.* LISTEN0063F3D4 *.179 1.1.1.1.* LISTEN
We are listening for TCP connections for port 179 for the configured peering addresses only!
R2#debug ip tcp transactionsTCP special event debugging is onR2#TCP: sending RST, seq 0, ack 2500483296TCP: sent RST to 4.4.4.4:26385 from 2.2.2.2:179
Remote is trying to open the session from 4.4.4.4 address …
90
Peer Establishment—IBGPWhat about us ?
R2#debug ip bgp BGP debugging is onR2#BGP: 1.1.1.1 open active, local address 4.4.4.5BGP: 1.1.1.1 open failed: Connection refused by remote host
We are trying to open the session from 4.4.4.5 address…
R2#sh ip route 1.1.1.1Routing entry for 1.1.1.1/32 Known via "static", distance 1, metric 0 (connected) * directly connected, via Serial1 Route metric is 0, traffic share count is 1
R2#show ip interface brief | include Serial1Serial1 4.4.4.5 YES manual up up
91
Peer Establishment—IBGP
Source address is the outgoing interface towards the destination but peering in this case is using loopback interfaces!
Force both routers to source from thecorrect interface
Use “update-source” to specify the loopback when loopback peering
R2#router bgp 1 neighbor 1.1.1.1 remote-as 1 neighbor 1.1.1.1 update-source Loopback0 neighbor 3.3.3.3 remote-as 2 neighbor 3.3.3.3 update-source Loopback0
92
Peer Establishment—Symptoms
R1 is established now The EBGP session is still having
trouble!
R2# sh ip bgp summaryBGP router identifier 2.2.2.2, local AS number 1BGP table version is 1, main routing table version 1Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 7 7 1 0 0 00:00:24 33.3.3.3 4 2 0 0 0 0 0 never Idle
93
R2#ping 3.3.3.3Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms
Peer Establishment—EBGP Trying to load-balance over multiple links to the eBGP
peer Verify IP connectivity
Check the routing table Use ping/trace to verify two way
reachability
Routing Towards Destination Correct, but…
94
Peer Establishment—EBGPR2# ping ipTarget IP address: 3.3.3.3Repeat count [5]: Datagram size [100]: Timeout in seconds [2]: Extended commands [n]: ySource address or interface: 2.2.2.2Type of service [0]: Set DF bit in IP header? [no]: Validate reply data? [no]: Data pattern [0xABCD]: Loose, Strict, Record, Timestamp, Verbose[none]: Sweep range of sizes [n]: Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:.....Success rate is 0 percent (0/5)
No Route Back from our Peer!
95
Peer Establishment—EBGP
R2#sh ip bgp neigh 3.3.3.3BGP neighbor is 3.3.3.3, remote AS 2, external link BGP version 4, remote router ID 0.0.0.0 BGP state = Idle Last read 00:00:04, hold time is 180, keepalive interval is 60 seconds Received 0 messages, 0 notifications, 0 in queue Sent 0 messages, 0 notifications, 0 in queue Route refresh request: received 0, sent 0 Default minimum time between advertisement runs is 30 seconds For address family: IPv4 Unicast BGP table version 1, neighbor version 0 Index 2, Offset 0, Mask 0x4 0 accepted prefixes consume 0 bytes Prefix advertised 0, suppressed 0, withdrawn 0 Connections established 0; dropped 0 Last reset never External BGP neighbor not directly connected. No active TCP connection
Neighbor Added Route but Still Having Problems?…
96
Peer Establishment—EBGP
eBGP peers are normally directly connected By default, TTL is set to 1 for eBGP peers If not directly connected, specify ebgp-multihop
At this point, the session should come up
router bgp 1 neighbor 3.3.3.3 remote-as 2 neighbor 3.3.3.3 ebgp-multihop 255 neighbor 3.3.3.3 update-source Loopback0
97
Peer Establishment—EBGP
Still having trouble! Connectivity issues have already
been checked and corrected
R2#show ip bgp summary BGP router identifier 2.2.2.2, local AS number 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd3.3.3.3 4 2 10 26 0 0 0 never Active
98
Peer Establishment—EBGP
If an error is detected, a notification is sent and the sessionis closed
R3 is configured incorrectly Has “neighbor 2.2.2.2 remote-as 10” Should have “neighbor 2.2.2.2 remote-as 1”
After R3 makes this correction the session comes up
R2#debug ip bgp events14:06:37: BGP: 3.3.3.3 open active, local address 2.2.2.214:06:37: BGP: 3.3.3.3 went from Active to OpenSent14:06:37: BGP: 3.3.3.3 sending OPEN, version 414:06:37: BGP: 3.3.3.3 received NOTIFICATION 2/2
(peer in wrong AS) 2 bytes 000114:06:37: BGP: 3.3.3.3 remote close, state CLOSEWAIT14:06:37: BGP: service reset requests14:06:37: BGP: 3.3.3.3 went from OpenSent to Idle14:06:37: BGP: 3.3.3.3 closing
Case 2 : Load Balancing
Parallel Links eBGP Multipath
Multi-homed
100
Command
neighbor {ip-address | peer-group-name} ebgp-multihop [ttl]
To accept and attempt BGP connections to external peers residing on networks that are not directly connected, use the neighbor ebgp-multihop command in router configuration mode.
101
Command
neighbor {ip-address | peer-group-name} update-source interface-type
To have the Cisco IOS software allow Border Gateway Protocol (BGP) sessions to use a specific operational interface for TCP connections, use the neighbor update-source command in router configuration mode.
To restore the interface assignment to the closest interface, which is called the best local address, use the no form of this command.
102
Command maximum-paths maximum-number
To control the maximum number of parallel routes an IP routing protocol can support, use the maximum-paths command in address family or router configuration mode. To restore the default value (1), use the no form of this command.
103
Load Balancing
BGP is not designed to load-balance traffic BGP chooses and installs one “best” route “Attempting” to balance traffic comes in two parts
Inbound trafficOutbound traffic
Load balancing is possible in some topologiesA pair of eBGP peers connected via multiple linksTwo connections from one router to the same AS
But not othersMulti-homed to more than one provider
104100200
A Loopback 02.2.2.2/32
Single Path Router A: interface loopback 0 ip address 1.1.1.1 255.255.255.255 ! router bgp 100 neighbor 2.2.2.2 remote-as 200 neighbor 2.2.2.2 update-source loopback0 neighbor 2.2.2.2 ebgp-multi-hop ! ip route 2.2.2.2 255.255.255.255 serial 0 ip route 2.2.2.2 255.255.255.255 serial 1 !
B
RA must do a recursive lookup for 2.2.2.2RA has two equal cost paths to 2.2.2.2
RA will load balance traffic over these two links Both inbound and outbound traffic will be balanced
105
eBGP Multipath Support
Router peers with multiple routers in the same neighbor AS Install multiple routes in IP routing table
Use ‘maximum-paths’ command Routes must be identical in terms of LOCAL_PREF, AS_PATH, MED, etc…
Outbound traffic will be split over these two links BGP still advertises one best path Next-hop is set to self (use loopback interface)
100 200
106
AS 100
AS 200
AS 300
AA
BB CC
DD
Multi-Homed AS
Very common topology for many customers Customer wants to split traffic between AS 100 and AS 300 Misconception: “I’ll make half of my routes preferred via AS
100 and the other half through AS 300. Then I’ll have load-balancing!!”
107
Multi-Homed AS This does not provide load balancing, just “prefix splitting” Traffic may be balanced perfectly until traffic patterns
change Load balancing is now over :( Some customers use this method but they are forced to
change their policies to accommodate for changes in traffic patterns
For outbound balancing use Weight LOCAL_PREF (recommended)
For inbound balancing use Conditional-advertisement AS_PATH prepending (may not work) MEDs (may not work) Communities and LOCAL_PREF (recommended)
Case 3 : BGP community
community-list route-map
109
110
Demand
The backup path of KOREN access TEIN2 backbone via CERNET
Backup link
111
Demand (Contd..) The backup path of TEIN2
Beijing POP and TEIN2 Singapore POP via CERNET-KOREN connection
Backup link
112
The detail demands CERNET advertises all the routers from TEIN2 Peking POP (containi
ng community 24489:65200 ) except for the router containing community 24489:65500 to KOREN, advertising such routers to KOREN, CERNET does the as-path-prepending (adding one “4538” ) in order to avoid KOREN selecting the path from CERNET to TEIN2-NORTH priorityr;
CERNET receives all the routers ( containing community 24489:65200 ) transmitting from TEIN2 Singapore POP from KOREN, adds community 4538:9270, and advertises those routers to TEIN2 Peking POP;
CERNET receives all the routers ( containing community 9270:65155 ) transmitting from KOREN internal R&E network from KOREN, adds the community 4538:9270, and advertises those routers to the TEIN2 Peking POP;
When the routers containing the community 4538:9270 are received from TEIN2 Peking POP, and the routers containing the community 4538:9270 are advertised to TEIN2 Peking POP, any restricting and adjusting should not be done, including the IPv4 and IPv6 router.
113
Configuration
KORENTEIN2
AS9270
.85 .86
AS24489
.25
.26 CERNET
202.179.241/302001:254:1:7::2/64
202.112.61/302001:250:0:30::1/64
AS4538
router bgp 4538 bgp router-id 202.112.60.247 no bgp fast-external-fallover bgp log-neighbor-changes bgp deterministic-med neighbor 202.112.61.86 remote-as 9270 neighbor 202.112.61.86 description KOREN neighbor 202.112.61.86 activate neighbor 202.112.61.86 send-community neighbor 202.112.61.86 soft-reconfiguration inbound neighbor 202.112.61.86 route-map KORENtoTEIN2 in neighbor 202.112.61.86 route-map TEIN2toKOREN out neighbor 202.112.61.86 activate neighbor 202.112.61.86 soft-reconfiguration inbound neighbor 202.112.61.86 filter-list 12 out neighbor 202.179.241.25 remote-as 24489 neighbor 202.179.241.25 description TEIN2 neighbor 202.179.241.25 password 7 02040D550C0A0A774F410615 neighbor 202.179.241.25 activate neighbor 202.179.241.25 send-community neighbor 202.179.241.25 soft-reconfiguration inbound neighbor 202.179.241.25 distribute-list 26 in neighbor 202.179.241.25 route-map tein2-in in neighbor 202.179.241.25 route-map CERNETtoTEIN2 out
114
Configuration (Contd..)
KORENTEIN2
AS9270
.85 .86
AS24489
.25
.26 CERNET202.179.241/30
2001:254:1:7::2/64202.112.61/30
2001:250:0:30::1/64
AS4538
route-map CERNETtoTEIN2 permit 10 match community CERNETtoTEIN2!route-map CERNETtoTEIN2 permit 20 match ip address prefix-list CERNET2 set community 4538:65155 additive
route-map TEIN2toKOREN permit 10 match community TEIN2bjpop set as-path prepend 4538!route-map TEIN2toKOREN permit 20 match as-path 12 match community TEIN2bjpop1!route-map TEIN2toKOREN permit 30 match ip address prefix-list CERNET2!
route-map KORENtoTEIN2 permit 10 match community KORENtoTEIN2 set local-preference 330 set as-path prepend 4538 set community 4538:9270 additive!route-map KORENtoTEIN2 permit 20 match ip address prefix-list KOREN-filter set local-preference 330 set as-path prepend 4538 set community 4538:9270 additive
route-map tein2-in permit 10 set local-preference 330 set community 4538:24489 additive!
115
Configuration (Contd..)
KORENTEIN2
AS9270
.85 .86
AS24489
.25
.26 CERNET
202.179.241/302001:254:1:7::2/64
202.112.61/302001:250:0:30::1/64
AS4538
ip community-list standard TEIN2bjpop deny 24489:65500ip community-list standard TEIN2bjpop permit 24489:65200ip community-list standard TEIN2bjpop1 deny 24489:65500ip community-list standard KORENtoTEIN2 permit 24489:65200ip community-list standard KORENtoTEIN2 permit 9270:65155ip community-list standard CERNETtoTEIN2 permit 4538:9270 9270:65155ip community-list standard CERNETtoTEIN2 permit 4538:9270 24489:65200
116
Show commandsbj-bgw-r0a#sh ip bgp sumBGP router identifier 202.112.60.247, local AS number 4538BGP table version is 61059, main routing table version 61059219479 network entries using 22167379 bytes of memory353680 path entries using 16976640 bytes of memory53580 BGP path attribute entries using 3000592 bytes of memory74 BGP rrinfo entries using 1776 bytes of memory41271 BGP AS-PATH entries using 1169092 bytes of memory2465 BGP community entries using 173974 bytes of memory2 BGP extended community entries using 48 bytes of memory11515 BGP route-map cache entries using 368480 bytes of memory776 BGP filter-list cache entries using 9312 bytes of memoryBGP using 43867293 total bytes of memory249339 received paths for inbound soft reconfigurationBGP activity 228135/1064 prefixes, 768702/399806 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd193.62.157.29 4 786 0 0 0 0 0 never Idle202.112.60.246 4 4538 576 161 61059 0 0 02:37:48 16791202.112.60.251 4 4538 4252 4407 61059 0 0 02:37:53 47349202.112.61.14 4 23911 89332 192 61059 0 0 02:38:39 4202.112.61.46 4 9264 775 344 61059 0 0 02:37:40 132202.112.61.86 4 9270 2294 913 61055 0 0 02:38:50 10038202.112.61.94 4 9505 215 186 61059 0 0 02:37:41 366202.179.241.25 4 24489 3896 420 61055 0 0 02:38:44 9589203.181.194.125 4 7660 4626 1275 61055 0 0 02:38:19 10036203.181.194.126 4 7660 9799 1116 61055 0 0 02:37:16 10031203.222.180.225 4 1239 44795 215 61059 0 0 02:37:30 3bj-bgw-r0a#
Bgp session end
Trouble Shooting OSPF
119
Agenda
LSA Overview Troubleshooting Commands Common Issues
120
LSA OVERVIEWLSA OVERVIEW
© 2004 Cisco Systems, Inc. All rights reserved.
RST-33019721_05_2004_c2 120120120
121
LSA Type Review
Type LSA
1 Router
2 Network
3 Summary Network
4 Summary ASBR
5 External
6 Group Membership
7
9–11
NSSA
External Attributes
Opaque
8
122
Router LSA Details Router LSA (Type 1)
Describes the state and cost of the router’s links to the area
All of the router’s links in an area must be described in a single LSA
Flooded throughout the particular area and no more
Router indicates whether it is an ASBR, ABR, or end point of virtual link
123
192.1.4.0/24
192.1.1.4
192.1.1.3
Area 0
R3#show ip ospf database router 3.3.3.3Router Link States (Area 1)
LS age = 0 Always 0 at originationOptions = (E-bit)LS type = 1 This is a router LSALink State ID = 3.3.3.3 Router ID of R3Advertising Router = 3.3.3.3 Router ID of R3It is an area border router bit B = 1 # links = 2
Link ID = 192.1.1.4 IP address of the DRLink Data = 192.1.1.3 Interface address of this router
Type = 2 This is a transit network # TOS metrics = 0
metric = 1 Cost to reach the interfaceLink ID = 192.1.4.0 IP network numberLink Data = 255.255.255.0 Subnet mask of the interfaceType = 3 Stub network# TOS metrics = 0metric = 2
DR
R4
2R3
1
Router LSA of R3 for Area 1
124
Router Link States (Area 0)
LS age = 0Options = (E-bit)LS type = 1Link State ID = 3.3.3.3Advertising Router = 3.3.3.3It is an area border router bit B = 1 # links = 2
Link ID = 6.6.6.6 Router id of the neighborLink Data = 18.10.0.5 IP interface address of the router Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8Link ID = 18.10.0.4 IP subnet address
Link Data = 255.255.255.252 Subnet mask Type = 3 This is a stub link # TOS metrics = 0 metric = 8
Area 0
DR
1
6.6.6.6
Router LSA of R3 for Area 0 (Cont.)
R6
192.1.1.3
R3
R4192.1.1.4
192.1.4.0/24
8
18.10.0.5/30
2
125
Router LSA DetailsType Description Link ID Link Data
11Point-to-PointPoint-to-Point
NumberedNumberedNeighbors’Neighbors’
RIDRIDInterface IPInterface IP
AddressAddress
11Point-to-PointPoint-to-PointUnnumberedUnnumbered
Neighbors’Neighbors’RIDRID
MIB-II MIB-II IfindexIfindexValueValue
22 TransitTransitIP AddressIP Addressof the DRof the DR
Interface IPInterface IPAddressAddress
33 StubStub IP NetworkIP NetworkNumberNumber Subnet MaskSubnet Mask
44 Virtual LinkVirtual Link Neighbors’Neighbors’RIDRID
Interface IPInterface IPAddressAddress
126
Network LSA
Network LSA (Type 2) Generated for every transit broadcast
and NBMA network Describes all the routers attached to
the network Only the designated router originates
this LSA Flooded throughout the area and no
more
127
R3#show ip ospf database network 192.1.1.4Network Link States (Area 1)
LS age = 948Options = (E-bit)LS type = 2Link State ID = 192.1.1.4 IP interface address of DRAdvertising Router = 4.4.4.4 RID of DRNetwork Mask = 255.255.255.0
Attached Router = 4.4.4.4Attached Router = 3.3.3.3 RID of attached routers FULL with the DRAttached Router = 2.2.2.2Attached Router = 1.1.1.1
Network LSA for 192.1.1.0
192.1.1.3
DR
R4
192.1.4.0/24
18.10.0.4/32
6.6.6.6
R3
192.1.1.4
1.1.1.1
2.2.2.2
R1
R2
/241
2
8
R6
4.4.4.4
Area 0
128
Summary LSA Describes the destination outside the area
but still in the AS Summary is created for each IP subnets in
one area and is flooded out in all other areas
Originated by an ABR Only intra-area routes are advertised into
the backbone Type 4 is the information about the ASBR
129
R4#show ip ospf database summary 192.1.2.0Summary Net Link States (Area 0)
LS age = 1514Options = (E-bit)LS type = 3Link State ID = 192.1.2.0 IP network numberAdvertising Router = 4.4.4.4 RID of ABRNetwork Mask = 255.255.255.0metric = 4
Type 3 Details
192.1.1.3
Area 0
192.1.4.0/24
18.10.0.4
6.6.6.6
R3
192.1.1.4
1.1.1.1
2.2.2.2
R2
/24
1
2
8
8192.1.2.0/24
ABR 192.1.2.0/24 metric 4
O IA
DR
R4R1
3
R6
ABR
130
R4#show ip ospf database asbr-summary 7.7.7.7Summary ASB Link States (Area 0)
LS age = 1548Options = (E-bit)LS type = 4Link State ID = 7.7.7.7 RID of ASBRAdvertising Router = 4.4.4.4 RID of ABRNetwork Mask = 0.0.0.0Metric = 16
Type 4 Details
18.10.0.4
R3
1.1.1.1
2.2.2.2
/24 Area 01
8
192.1.4.0/24
R2
8
192.1.2.0/24
DR
R4R1
3
140.10.0.0
External Route
R7
RID ASBR 7.7.7.7
8
R3
ABRType 4 Summary
2
6.6.6.6
R6
192.1.1.3
192.1.1.4
(4.4.4.4)
131
External LSA
External LSA (Type 5) Defines routes to destination external
to the AS Default route is also sent as external Two types of external LSA:
E1: Consider the total cost up to the external destination
E2: Considers only the cost of the outgoing interface to the external destination
132
R4#show ip ospf database external 140.10.0.0LS age = 1156Options = (E-bit)LS type = 5Link State ID = 140.10.0.0 IP network numberAdvertising Router = 7.7.7.7 Router ID of R7Network Mask = 255.255.0.0Metric Type: 2 bit E = 1 -> O E2 (Default)metric = 20 The metric is 20 in all redistributed E2 routesForwarding address = 0.0.0.0 Traffic should be forwarded to the ASBR
1.1.1.1
Area 0
192.1.4.0/24R2
192.1.2.0/24
DR
R4R1
R7
RID ASBR 7.7.7.7
8
R3 2 18.10.0.4
6.6.6.6192.1.1.3
192.1.1.4
2.2.2.2
/24
1
8
3140.10.0.0
External Route
External Type 5R6
Type 5 Details
133
NSSA External LSA NSSA External LSA (Type 7) RFC1587
NSSA was created to inject external routes from stub area into OSPF domain
Redistribution in NSSA creates Type 7 LSA Generated by the NSSA ASBR Type 7 can only exist in NSSA area NSSA ABR does the translation from 7–5
134
TROUBLESHOOTING COMMANDSTROUBLESHOOTING COMMANDS
© 2004 Cisco Systems, Inc. All rights reserved.
RST-33019721_05_2004_c2 134134134
135
R3#show ip ospf Routing Process "ospf 1" with ID 3.3.3.3 and Domain ID 0.0.0.1 Supports only single TOS(TOS0) routes Supports opaque LSA It is an area border router SPF schedule delay 5 secs, Hold time between two SPFs 10 secs Minimum LSA interval 5 secs. Minimum LSA arrival 1 secs Number of external LSA 1. Checksum Sum 0x3B57 Number of opaque AS LSA 0. Checksum Sum 0x0 Number of DCbitless external and opaque AS LSA 0 Number of DoNotAge external and opaque AS LSA 0 Number of areas in this router is 2. 2 normal 0 stub 0 nssa External flood list length 0 Area BACKBONE(0) Number of interfaces in this area is 2 Area has no authentication SPF algorithm executed 2773 times Area ranges are Number of LSA 17. Checksum Sum 0x686B5 Number of opaque link LSA 0. Checksum Sum 0x0 Number of DCbitless LSA 0 Number of indication LSA 0 Number of DoNotAge LSA 9 Flood list length 0
Show IP OSPF
136
. . .Area 1 Number of interfaces in this area is 2 Area has no authentication SPF algorithm executed 22 times Area ranges are Number of LSA 19. Checksum Sum 0x8FE73 Number of DCbitless LSA 0 Number of indication LSA 0 Number of DoNotAge LSA 0 Flood list length 0
Show IP OSPF (Cont.)
137
R3#show ip ospf database OSPF Router with ID (3.3.3.3) (Process ID 1) Router Link States (Area 0) Link ID ADV Router Age Seq# Checksum Link count3.3.3.3 3.3.3.3 106 0x80000009 0xC3F1 3 . . . Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum18.10.0.0 7.7.7.7 3 (DNA) 0x80000008 0x3DC2 18.10.0.0 8.8.8.8 1396 0x80000004 0x27D8 . . . Router Link States (Area 1)
Link ID ADV Router Age Seq# Checksum Link count1.1.1.1 1.1.1.1 671 0x80000016 0xE6CD 2. . .
Show IP OSPF Database
138
R3#show ip ospf database database-summary
OSPF Router with ID (3.3.3.3) (Process ID 1)
Area 0 database summary LSA Type Count Delete Maxage Router 6 0 0 Network 4 0 0 Summary Net 10 0 0 Summary ASBR 0 0 0 Type-7 Ext 0 0 0 Opaque Link 0 0 0 Opaque Area 0 0 0 Subtotal 20 0 0
Area 1 database summary LSA Type Count Delete Maxage Router 4 0 0 Network 1 0 0 Summary Net 10 0 0 Summary ASBR 4 0 0 . . .
Show IP OSPF Database Database-Summary
139
R3#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface1.1.1.1 1 FULL/DROTHER 00:00:33 192.1.1.1 FastEthernet0/02.2.2.2 1 FULL/DROTHER 00:00:32 192.1.1.2 FastEthernet0/04.4.4.4 1 FULL/DR 00:00:39 192.1.1.4 FastEthernet0/06.6.6.6 1 FULL/ - 00:00:38 18.10.0.6 Serial0/0R3#
Show IP OSPF Neighbor
192.1.1.3
DR
R4
192.1.4.0/24
18.10.0.4
6.6.6.6
R3
192.1.1.4
1.1.1.1
2.2.2.2
R1
R2
/241
2
8
R6
4.4.4.4
Area 0
140
R3#config terminalR3(config)#router ospf 1R3(config-router)#log-adjacency-changes%OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on FastEthernet0/0 from 2WAY to DOWN, Neighbor Down: Interface down or detached%OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on OSPF_VL0 from FULL to DOWN, Neighbor Down: Interface down or detached%LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet0/0 from LOADING to FULL, Loading Done%OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on OSPF_VL0 from LOADING to FULL, Loading Done
OSPF Log-Adjacency-Changes Default as of 12.1.3 and 12.0.12S
141
R3#show ip ospf neighbor detailNeighbor 1.1.1.1, interface address 192.1.1.1 In the area 1 via interface FastEthernet0/0 Neighbor priority is 1, State is 2WAY, 2 state changes DR is 192.1.1.4 BDR is 192.1.1.2 Options is 0x2 Dead timer due in 00:00:39 Neighbor is up for 00:06:30 Index 0/0, retransmission queue length 0, number of retransmission 0 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 0, maximum is 0 Last retransmission scan time is 0 msec, maximum is 0 msec Neighbor 2.2.2.2, interface address 192.1.1.2 In the area 1 via interface FastEthernet0/0 Neighbor priority is 1, State is FULL, 6 state changes DR is 192.1.1.4 BDR is 192.1.1.2 Options is 0x42 Dead timer due in 00:00:38 Neighbor is up for 00:06:31 Index 2/2, retransmission queue length 0, number of retransmission 0 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 0, maximum is 0 Last retransmission scan time is 0 msec, maximum is 0 msec
Show IP OSPF Neighbor Detail
142
R3#show ip ospf interfaceFastEthernet0/0 is up, line protocol is up Internet Address 192.1.1.3/24, Area 1 Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 1 Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 4.4.4.4, Interface address 192.1.1.4 Backup Designated router (ID) 2.2.2.2, Interface address 192.1.1.2 Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:03 Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 0, maximum is 5 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 3, Adjacent neighbor count is 2 Adjacent with neighbor 2.2.2.2 (Backup Designated Router) Adjacent with neighbor 4.4.4.4 (Designated Router) Suppress hello for 0 neighbor(s)
Show IP OSPF Interface
143
R3#show ip ospf virtual-linksVirtual Link OSPF_VL0 to router 4.4.4.4 is up Run as demand circuit DoNotAge LSA allowed. Transit area 1, via interface FastEthernet0/0, Cost of using 1 Transmit Delay is 1 sec, State POINT_TO_POINT, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:09 Adjacency State FULL (Hello suppressed) Index 1/3, retransmission queue length 0, number of retransmission 1 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 1, maximum is 1 Last retransmission scan time is 0 msec, maximum is 0 msecR3#
Show IP OSPF Virtual-Links
144
R3#sh ip ospf stat Area 0: SPF algorithm executed 42 times Area 1: SPF algorithm executed 38 times
SPF calculation timeDelta T Intra D-Intra Summ D-Summ Ext D-Ext Total Reason00:22:00 0 0 0 0 0 0 0 R, N, SN, 00:21:44 0 0 4 0 0 0 4 R, SN, X00:21:34 0 0 4 0 0 0 4 R, SN, X00:21:24 0 0 0 4 0 0 4 R, SN, X00:21:14 0 0 0 0 0 0 0 R, 00:21:04 0 0 0 0 0 0 0 R, N, SN, 00:20:54 0 0 0 0 0 0 0 X00:20:44 0 0 4 0 0 0 4 R, SN, X00:20:34 0 0 0 0 0 0 0 X00:00:17 4 0 0 0 0 0 4 R, N, SN, SA, X. . .R=Router LSA; N=NetworkLSA; SN=Summary Network LSA; SA=Summary ASBR LSA; X=External LSA
Show IP OSPF Stat Requires enable mode
145
R3#show ip ospf borders-routersOSPF Process 1 internal Routing Table
Codes: i - Intra-area route, I - Inter-area route
i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 0, SPF 42i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 1, SPF 38i 8.8.8.8 [10] via 18.10.0.6, Serial0/0, ABR/ASBR, Area 0, SPF 42i 7.7.7.7 [17] via 192.1.1.4, FastEthernet0/0, ABR/ASBR, Area 0, SPF 42
Show IP OSPF Borders
146
Other Show CommandsR3#show ip ospf database self-originate OSPF Router with ID (3.3.3.3) (Process ID 1)
Router Link States (Area 0)Link ID ADV Router Age Seq# Checksum Link
count3.3.3.3 3.3.3.3 1520 0x80000015 0xABFD 2 Summary Net Link States (Area 0)Link ID ADV Router Age Seq# Checksum192.1.1.0 3.3.3.3 1520 0x80000006 0x4E1A 192.1.2.0 3.3.3.3 1521 0x80000006 0x6103 . . . Router Link States (Area 1)Link ID ADV Router Age Seq# Checksum Link
count3.3.3.3 3.3.3.3 1536 0x80000028 0x612D 2
147
Other Show Commands (Cont.)
R3#show ip ospf database adv-router 7.7.7.7OSPF Router with ID (3.3.3.3) (Process ID 1) Router Link States (Area 0)Link ID ADV Router Age Seq# Checksum Link
count7.7.7.7 7.7.7.7 871(DNA) 0x8000000D 0x8FE2 2 Summary Net Link States (Area 0)Link ID ADV Router Age Seq# Checksum20.10.0.0 7.7.7.7 871 (DNA) 0x8000000A 0x39C4 Type-5 AS External Link StatesLink ID ADV Router Age Seq# Checksum Tag140.100.0.0 7.7.7.7 1944 0x80000004 0x3759 0
148
COMMON ISSUESCOMMON ISSUES
148148148
149
Common Issues
Adjacency is not coming up OSPF neighbor stuck in ? state Information is in the DB but not in the
RT SPF running constantly Neighbor flapping (Frame Relay) NSSA ABR not translating Type 7 LSA Demand circuit problems
150
Adjacency Is Not Coming Up
Useful commands for this problem Show IP OSPF neighbor Show IP OSPF interface Debug IP OSPF adjacency
151
R3#show ip ospf neighborR3#
R3#show ip ospf interface serial 2Serial2 is down, line protocol is DOWN Internet Address 18.10.0.3/30, Area 0 Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT, Cost: 64 Transmit Delay is 1 sec, State DOWN, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Layer 2 is down
Adjacency Is Not Coming Up
152
R3#show ip ospf neighborR3#
R3#show ip ospf interface serial 2Serial2 is up, line protocol is up OSPF not enabled on this interface
In 12.0:
R3#show ip ospf interface serial 2R3#
Tip: Check for the wrong network statement re-enter the network statement
Adjacency Is Not Coming Up
OSPF not enabled on the interface
153
R3#show ip ospf neighborR3#
R3#show ip ospf interface e0Ethernet0 is up, line protocol is up Internet Address 192.1.1.3/24, Area 1 Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 10 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 192.1.1.4, Interface address 192.1.1.3 No backup designated router on this network Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 No Hellos (Passive interface)
Adjacency Is Not Coming Up
Interface is defined as passive
154
Adjacency Is Not Coming Up
Mismatched subnet mask
R3#show ip ospf neighborR3#
R3#debug ip ospf adjOSPF adjacency events debugging is onR3#OSPF: Mismatched hello parameters from 192.1.1.4Dead R 40 C 40, Hello R 10 C 10 Mask R 255.255.255.192 C 255.255.255.0
155
R3#show ip ospf neighborR3#
R3#debug ip ospf adjOSPF adjacency events debugging is onR3#OSPF: Mismatched hello parameters from 192.1.1.4Dead R 40 C 40, Hello R 15 C 10 Mask R 255.255.255.0 C 255.255.255.0
R4(config-if)#interface ethernet 0R4(config-if)#no ip ospf hello-interval 15
Tip: Default is 10 second on LAN
Adjacency Is Not Coming Up Mismatched hello/dead interval
156
R3#show ip ospf neighborR3#
R3#debug ip ospf adjOSPF adjacency events debugging is onR3#OSPF: Rcv pkt from 192.1.1.4, Ethernet0 : Mismatch Authentication Key - Clear Text
Tip: Watch for the “space” at the end of the Authentication key
Adjacency Is Not Coming Up
Mismatched authentication key
157
OSPF: Rcv pkt from 192.1.1.4, Ethernet0, area 0.0.0.1 mismatch area 0.0.0.2 in the header
Neighbor is in area 2 but we are not:
%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID, from backbone area must be virtual-link but not found from 192.1.1.4, Ethernet0
R4#show ip ospf neighborR4#
R4#debug ip ospf adjOSPF adjacency events debugging is on
Adjacency Is Not Coming Up
Mismatched area ID
158
OSPF: Hello from 18.10.0.2 with mismatched Stub/Transit area option bit
Adjacency Is Not Coming Up
Mismatched transit/stub/NSSA option
R7#show ip ospf neighborR7#
R7#debug ip ospf adjOSPF adjacency events debugging is on
7.7.7.7 8.8.8.8
R8R7area 2
area 2 nssa
18.10.0.218.10.0.1
159
OptionsNormal area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492E bit is 1, Allow externals, option: 0x2(HEX) = 00000010(Bin)Stub area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1866 opt 0x0 flag 0x3 len 372E bit is 0, no external allowed, options: 0x0 = 00000000NSSA: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x118 opt 0x8 flag 0x3 len 372N/P bit is on, options: 0x8 = 00001000DC: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1A1E opt 0x20 flag 0x3 len 392DC bit is negotiated, options: 0x20 = 00100000
* *DC EA N/P MC EO
160
OSPF Neighbor Stuck in ? State
Useful commands for this problem Show IP OSPF neighbor Debug IP OSPF adjacency
161
RID =6.6.6.6
N = 3.3.3.3
Hello
Stuck in ATTEMPT
3.3.3.3 6.6.6.6
RID =3.3.3.3
Hello
R3 R6
NBMA
162
Stuck in ATTEMPT
Reasons: Our hellos are getting lost in NBMA cloud Neighbor hellos are getting lost in NBMA
cloud We received neighbor’s hello but rejects
it for some reason Misconfigured neighbor statement Broken Unicast
163
RID =2.2.2.2
Hello
Stuck in INIT 2.2.2.2
RID =1.1.1.1
Hello
R2
RID =1.1.1.1
Hello
N =2.2.2.2
RID =2.2.2.2
Hello
1.1.1.1
R1
164
Stuck in INIT
Reasons: One side is blocking the hello packet
with access-list One side is translating (NAT) ospf hello One side multicast capabilities is broken
(Layer 2) Dialer map or frame-relay map is missing
keyword ‘broadcast’
165
Stuck in 2-WAY 1.1.1.1 2.2.2.2
RID =1.1.1.1 P=0
Hello
RID =1.1.1.1 P=0
Hello
N =2.2.2.2
R2R1
RID =2.2.2.2 P=0
Hello
N =1.1.1.1
166
Stuck in 2-WAY Reasons:
This is normal in broadcast network types This is to reduce the amount of flooding on the
wire Problem can happen if all the router are
configured with priority equal to ‘0’
In a situation where you have high and low end boxes on the same segment the configure low end routers with priority 0 so they don’t participate in DR election
167
Stuck in EXSTART/EXCHANGE
3.3.3.3 6.6.6.6
RID =3.3.3.3
HelloR6R3
RID =6.6.6.6
Hello
N =3.3.3.3
MTU = 1500 flag = 0x7
DBD
Seq = 1E55
MTU = 1500 flag = 0x7
DBD
Seq = 22AB
168
Stuck in EXSTART/EXCHANGE Useful in debugging, defines I, M and MS bits
OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492
Flag 0x7--> 111 means I(Initial) = 1, M = 1(More), MS = 1(Master) Flag 0x6 --> 110 not possible Flag 0x5 --> 101 not possible Flag 0x4 --> 100 not possible Flag 0x3 --> 011 means master has more data to send Flag 0x2 --> 010 means slave has more data to send Flag 0x1 --> 001 means master has no more data left to send Flag 0x0 --> 000 means slave has no more data left to send
0 00 0 I
M MS0
169
Stuck in EXSTART/EXCHANGE Reasons:
MTU mismatch Note: If Cisco IOS is < 12.0.3 neighbor will show stuck in EXCHANGE
Neighbor RID is same as ours.Note: If Cisco IOS is > 12.0.7, it displays msg: %OSPF-3-DUP_RTRID & OSPF neighbor list will be empty
Unicast is brokena. Wrong VC/DLCi mapping in frame/ATM environment in highly redundant networkb. MTU problem, can’t ping across with more than certain length packet c. Access-list blocking unicast; after 2-way OSPF send unicast packet except p2p linksd. NAT is translating unicast packet
Between PRI and BRI/dialer and network type is p2p
170
Stuck in LOADING3.3.3.3 6.6.6.6
R6R3
LS Type
LS Req
Link State ID
Advertising Router
# LSAs
LS Update
LSAs. . .
171
Stuck in LOADING Reasons:
LS request is being made and neighbor is sending bad packet or mem corrupt
a. Do show IP OSPF bad to see bad lsa b. Show log will show
OSPF-4-BADLSATYPE msg LS request is being made and neighbor is
ignoring the request
MTU mismatch problem (RFC 1583 and 2178 compatibility issue)
172
Stuck in LOADING3.3.3.3 6.6.6.6
R6R3
LS Type
LS Req
Link State ID
Advertising Router
# LSAs
LS Update
LSAs
. . .
MTU = 2048
IOS 11.3.10T
RFC 1583
MTU = 4470
IOS 12.0.7T
RFC 2178
Too Big!
173
Information Is in the DBbut Not in the RT
Useful commands for this problem Show IP OSPF interface <interface> Show IP OSPF database <x> Where ‘x’ can be router, network,
summary, summary-asbr, external, nssa
174
R3#show ip ospf interface serial 0Serial0 is up, line protocol is up Internet Address 18.0.0.5/30, Area 0 Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT, Cost: 64
R6#show ip ospf interface serial 0Serial0 is up, line protocol is up Internet Address 18.0.0.6/30, Area 0 Process ID 1, Router ID 6.6.6.6, Network Type BROADCAST, Cost: 64
Mismatched Network Types
Area 0
R3
3.3.3.3
R6
6.6.6.6
18.10.0.4/30
R3#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0
175
R618.10.0.4/30
Area 0
R3
3.3.3.3 6.6.6.6
R3#show ip ospf database router 3.3.3.3. . .
Link ID = 6.6.6.6 Router id of the neighborLink Data = 18.10.0.5 IP interface address Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8
. . .R3#show ip ospf database router 6.6.6.6. . . Link ID = 18.10.0.6 IP address of the DR Link Data = 18.10.0.6 Interface address Type = 2 This is a transit link # TOS metrics = 0 metric = 8
Mismatched Network Types (Cont.)
176
Point-to-Point Numbered and Unnumbered Links
R3#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0
R3#show interface serial0 Serial0 is up, line protocol is up Hardware is HD64570 Interface is unnumbered.Using address of Ethernet1 (192.1.4.1)
R6#show interface serial0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 18.10.0.6/30
Area 0
R3
3.3.3.3
R6
6.6.6.6
18.10.0.4/30
192.1.4.1
177
R618.10.0.4/30
Area 0
R3
3.3.3.36.6.6.6
R3#show ip ospf database router 3.3.3.3. . .
Link ID = 6.6.6.6 Router id of the neighborLink Data = 0.0.0.5 MIBII IfIndex Value Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8
. . .R3#show ip ospf database router 6.6.6.6. . . Link ID = 3.3.3.3 Router id of the neighbor Link Data = 18.10.0.6 IP interface address Type = 1 This is a transit link # TOS metrics = 0 metric = 8
192.1.4.1
Point-to-Point Numbered and Unnumbered Links (Cont.)
178
R3#show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 19.10.0.5/24
R6#show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 18.10.0.6/30
Different Mask or IP Subneton p2p Links
Area 0
R3
3.3.3.3
R6
6.6.6.6
19.10.0.5/30
R3#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0
18.10.0.6/30
179
R3#show ip ospf database router 3.3.3.3. . .
Link ID = 6.6.6.6 Router id of the neighborLink Data = 19.10.0.5 Interface address Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8
. . .R3#show ip ospf database router 6.6.6.6. . . Link ID = 3.3.3.3 Router id of the neighbor Link Data = 18.10.0.6 Interface address Type = 1 This is a point-to-point link # TOS metrics = 0 metric = 8
R6
Area 0
R3
3.3.3.3 6.6.6.6
Different Mask or IP Subneton p2p Links (Cont.)
180
Address Flipped on Dual Links
R3#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.10.0.6 Serial0 6.6.6.6 1 FULL/ - 00:00:33 18.10.0.22 Serial1
R3#show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 18.10.0.21/30
R3
3.3.3.3
R6
6.6.6.6
18.10.0.20/30
Area 0
18.10.0.4/30S0
S1
181
Forwarding Address Problem
R1#show ip ospf database external 140.10.0.0. . .Link State ID = 140.10.0.0Advertising Router = 5.5.5.5Network Mask = 255.255.0.0. . .Forwarding address = 18.10.0.10
R1#show ip route 18.10.0.10Routing entry for 18.10.0.8/29 Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 10. . .. . .
R5R4 R7R1
18.10.0.1/30
18.10.0.9/29
18.10.0.5/30RIP
Area 1Area 0
140.10.0.0
External Route
182
Forwarding Address Problem
R5:
router ospf 1 network 18.10.0.0 0.0.0.255 area 1 redistribute rip subnets redistribute connected subnets ! router rip network 10.0.0.0
R4:
router ospf 1 area 1 range 18.10.0.0 255.255.255.240
R5 R7
18.10.0.1/30
18.10.0.5/30Area 1
Area 0140.10.0.0
External Route
R4R1 18.10.0.9/29
RIP
183
ABR2ABR1 R2R1Area 1
Area 0 Area 0
Discontigous Backbone
R1 and R2 are not be able to see each other Summary LSA for Inter-area routes must not
be generated into the backbone The solution is to create virtual link between
ABR1 and ABR2
Summary LSA
Summary LSA
Summary LSA
Summary LSA
Area 0
184
Distribute-List in Blocking the Routes
R4#show ip route 18.10.0.9 % Subnet not in table
R4:
router ospf 1 network 18.10.0.0 0.0.0.255 area 1 distribute-list 1 in ! access-list 1 permit 18.10.0.0 0.0.0.3
R5R4
R7
18.10.0.1/30
18.10.0.9/29
185
SPF Running Constantly
Useful commands for this problem Show IP OSPF stat Show IP OSPF database Show IP OSPF database database-sum
186
SPF Running Constantly
Reasons: LSA flaps due to:
Duplicate RID/IP address Constant link flapping in an area
187
R3#sh ip ospf stat Area 0: SPF algorithm executed 42 times Area 1: SPF algorithm executed 38 times
SPF calculation timeDelta T Intra D-Intra Summ D-Summ Ext D-Ext Total Reason00:22:00 0 0 0 0 0 0 0 R, N, SN, 00:21:44 0 0 4 0 0 0 4 R, SN, X00:21:34 0 0 4 0 0 0 4 R, SN, X00:21:24 0 0 0 4 0 0 4 R, SN, X00:21:14 0 0 0 0 0 0 0 R, 00:21:04 0 0 0 0 0 0 0 R, N, SN, 00:20:54 0 0 0 0 0 0 0 X00:20:44 0 0 4 0 0 0 4 R, SN, X00:20:34 0 0 0 0 0 0 0 X00:00:17 4 0 0 0 0 0 4 R, N, SN, SA, X. . .R=Router LSA; N=NetworkLSA; SN=Summary Network LSA; SA=Summary ASBR LSA; X=External LSA
SPF Running Constantly Requires enable mode
188
R3#deb ip ospf mon
OSPF: schedule SPF in area 1
Change in LS ID 1.1.1.1, LSA type R,
OSPF: schedule SPF: spf_time 0ms wait_interval 861421816s
OSPF: begin SPF at 0x33585480ms, process time 752ms
Spf_time 0ms, wait_interval 861421816s
OSPF: end SPF at 0x33585488ms, total elapsed time 8ms
Intra: 4ms, inter: 0ms, external: 0ms
SPF Running Constantly
189
R3#show ip ospf database
OSPF Router with ID (3.3.3.3) (Process ID 1)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
3.3.3.3 3.3.3.3 106 0x80000009 0xC3F1 3
. . .
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
18.10.0.0 7.7.7.7 3 (DNA) 0x80000008 0x3DC2
18.10.0.0 8.8.8.8 1396 0x80000004 0x27D8
. . .
Router Link States (Area 1)
Link ID ADV Router Age Seq# Checksum Link count
1.1.1.1 1.1.1.1 2 0x80000016 0xE6CD 2
. . .
SPF Running Constantly
190
R3#show ip ospf database database-summary
OSPF Router with ID (3.3.3.3) (Process ID 1)
Area 0 database summary LSA Type Count Delete Maxage Router 124 0 0 Network 4 0 0 Summary Net 10 0 0 Summary ASBR 0 0 0 Type-7 Ext 0 0 0 Opaque Link 0 0 0 Opaque Area 0 0 0 Subtotal 138 0 0
Area 1 database summary LSA Type Count Delete Maxage Router 4 0 0 Network 1 0 0 Summary Net 10 0 0 Summary ASBR 4 0 0 . . .
SPF Running Constantly
191
Neighbor Flapping (Frame Relay)
Useful commands for this problem Debug ip ospf adj OSPF log-adjacency-change Show IP OSPF neighbors detail Show interface
192
Area 0
router ospf 1network 10.10.10.10 0.0.0.0 area 0area 0
RID: 7.7.7.7
Type 7–5 Conversion
NSSA ABR Not Translating Type 7 LSA
Only NSSA ABR with the highest RID does the conversion
No Type 7/5 Translation
Area 1NSSA
Type 7
RID: 8.8.8.8
NSSAABR
NSSAASBR
193
NSSA ABR Not Translating Type 7 LSA
Only NSSA ABR with the highest RID does the conversion
Area 0NSSAABRrouter ospf 1
network 10.10.10.10 0.0.0.0 area 0area 0
RID: 8.8.8.8
NSSAASBR
Type 7
Type 7–5 Conversion
RID: 7.7.7.7
Area 1NSSA
194
Demand Circuit Problems
The DC is bringing up the link: There is a change in OSPF topology debug IP OSPF monitor is helpful in this case Network type on DC is defined broadcast There is a router in the network that is incapable
to understand DC bit The DC is configured over async interface
(need to configure a dialer interface as a solution)
OSPF (DC)
18.10.0.1/30 18.10.1.0/24
195
Demand Circuit Problems
DC is bringing up the link (cont.) PPP host route is also owned by RIP, when PPP
host route disappears, the database is change Solution 1: no peer neighbor-route Solution 2: Block /32 route getting into OSPF with
route-map Solution 3: Use different majornet for RIP
RIP—OSPF Redistribution
OSPF (DC)
18.10.0.1/30 18.10.1.0/24
RIP
196
Summary
What we learned? Overview of OSPF LSAs Different troubleshooting commands
and what to look for in those commands while troubleshooting?
Common issues in OSPF networks; e.g adjacency problems, CPU hogs and SPF problems, NSSA and DC problems etc and how to correct those problems
© 2002, Cisco Systems, Inc. All rights reserved. 197
OSPF case studies
Case 1
Why Does the show ip ospf neighbor Command Reveal Neighbors Stuck in Two-Way
State?
199
Introduction
This case explains why the show ip ospf neighbor command shows neighbors stuck in a two-way state. It also provides configuration tips.
200
How OSPF Forms Its Neighbors
In this topology, all routers are running Open Shortest Path First (OSPF) over the Ethernet network
201
The sample output of the show ip ospf neighbor command
R7# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 170.170.3.4 1 2WAY/DROTHER 00:00:34 170.170.3.4 Ethernet0 170.170.
3.3 1 2WAY/DROTHER 00:00:34 170.170.3.3 Ethernet0 170.170.3.8 1 FULL/DR 00:00:32 170.170.3.8 Ethernet0 170.170.3.2 1 FULL/BDR 00:00:39 170.170.3.2 Ethernet0 R8# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 170.170.3.4 1 FULL/DROTHER 00:00:37 170.170.3.4 Ethernet0 170.170.3.3 1 FULL/DROTHER 00:00:37 170.170.3.3 Ethernet0 170.170.3.7 1 FULL/DROTHER 00:00:38 170.170.3.7 Ethernet0 170.170.3.2 1 FULL/BDR 00:00:32 170.170.3.2 Ethernet0
202
• Whenever a router sees itself in a neighbor hello packet, it confirms bidirectional communication and transitions the neighbor state to two-way. At this point, the routers perform DR and BDR election. Once DR and BDR are elected, a router attempts to form a full adjacency with a neighbor if one of the two routers is the DR or BDR. OSPF routers become fully adjacent with routers with which they have successfully completed the database synchronization process. This is the process by which OSPF routers exchange link-state information to populate their databases with the same information. Again, this database synchronization process is only executed between two routers if one of the two routers is the DR or BDR.
203
Why Do Routers Only Form Full Adjacencies with the DR or BDR?
204
Why Do Routers Only Form Full Adjacencies with the DR or BDR? (cont.)
Sometimes it is desirable for a router to be configured so that it is not eligible to become the DR or BDR. You can do this by setting the OSPF priority to zero with the ip ospf priority priority# interface subcommand. If two OSPF neighbors both have their OSPF interface priority set to zero, they establish two-way adjacency instead of full adjacency.
205
example.(topology)
206
example. The topology below provides an example.
There are three routers connected via Frame Relay. The Frame Relay interfaces are defined as broadcast, but only the router with a connection back to the main network is eligible to be the DR. The other two routers have their interface priorities set to zero, so they are not eligible to become the DR or BDR. Although they do become neighbors, they only reach two-way state.
207
example. The neighbor table for this topology looks
like this: DRother1# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 170.170.9.5 1 FULL/DR 00:00:30 170.170.9.5 Serial0.5 170.170.10.8 0 2WAY/DROTHER 00:00:38 170.170.9.8 Serial0.5 DRother1#
Notice that, in the figure above, the DRother1 router establishes a two-way adjacency with the DRother2 router.
Case 2
OSPF routers do not form neighbor relationships due to authentication type
mismatch
209
OSPF Errors, Warnings, and Log Messages
Receiving "Mismatch Authentication type."
210
Core Issue Before exchanging routing information, routers
running Open Shortest Path First (OSPF) form neighbor relationships with other OSPF routers on the same segment. This is done by exchanging hello packets. The hello packets contain various parameters, one of which is related to authentication. This specifies the authentication type and authentication information for the originating interface. Authentication is useful for preventing malicious or incorrect routing information from getting introduced into the routing table. OSPF supports two types of authentication: plain text and Message Digest 5 (MD5).
211
To resolve this issue, perform these steps:
1. Identify the adjacency state with the neighbor by issuing the show ip ospf neighbor command from privileged EXEC mode.
2. Find the authentication type configured under an interface by issuing the show ip ospf interface command from privileged EXEC mode.
212
To resolve this issue, perform these steps(cont.): If the authentication type does not match between
two routers on the same segment, you will see a message similar to this:
OSPF: Rcv pkt from x.x.x.x, Ethernet1/0 : Mismatch Authentication type. Input packet specified type 2, we use type 1
This occurs when you issue the debug ip ospf adj command from privileged EXEC mode.
This message indicates this information: Type 0 indicates that no authentication is enabled. Type 1 is for plain text authentication to be enabled. Type 2 means that MD5 authentication is enabled.
213
To resolve this issue, perform these steps: 3. Make sure that all the routers connected to the sa
me segment use the same authentication type. This is done by issuing the area authentication command in router configuration mode or the ip ospf authentication command in interface configuration mode.
The area authentication command configures all the interfaces under a particular area to use the specified authentication type.
The ip ospf authentication command can be used to individually configure the authentication type for an interface or override the type configured under an interface with the area authentication command.
214
Trouble shooting IS-IS
© 2002, Cisco Systems, Inc. All rights reserved. 214
215
Troubleshooting Agenda
Adjacencies LSP Flooding and Contents SPF Computation Monitoring Performance Advanced/New Features
216
Router-B--------------
interface Loopback0 ip address 192.168.1.1
255.255.255.255 ! Interface Serial0 ip address 192.168.120.10
255.255.255.0 ip router isis ! interface Serial1 ip address 192.168.222.1
255.255.255.0 ip router isis ! router isis passive-interface Loopback0 net 49.0001.1921.6800.1001.00
Router-A--------------
interface Loopback0 ip address 192.168.1.5
255.255.255.255 ! interface Serial0 ip address 192.168.120.5
255.255.255.0 ip router isis ! router isis passive-interface Loopback0 net 49.0001.1921.6800.1005.00 is-type level-1
Basic Configuration
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
217
Router-C--------------
interface Loopback0 ip address 192.168.2.2
255.255.255.255 ! interface Serial0 ip address 192.168.111.2
255.255.255.0 ip router isis isis circuit-type level-1 ! interface Serial1 ip address 192.168.222.2
255.255.255.0 ip router isis isis circuit-type level-2 ! router isis passive-interface Loopback0 net 49.0002.1921.6800.2002.00
Router-D--------------
interface Loopback0 ip address 192.168.2.4
255.255.255.255 ! interface Serial1 ip address 192.168.111.4
255.255.255.0 ip router isis ! router isis passive-interface Loopback0 net 49.0002.1921.6800.2004.00 is-type level-1
Basic Configuration
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
218218218© 2004 Cisco Systems, Inc. All rights reserved.
ADJACENCIES
RST-33029803_05_2004_c2
219
Check that CLNS Is Running OK?
General CLNS information at a glance: Lists number of CLNS-enabled interfaces Lists NET configured on router Lists the mode that is running (IP or OSI) Lists IS-type
show clns show clns protocol
220
Rtr-B#show clnsGlobal CLNS Information: 2 Interfaces Enabled for CLNS NET: 49.0001.1921.6800.1001.00 Configuration Timer: 60, Default Holding Timer: 300, Packet Lifetime 64 ERPDU's requested on locally generated packets Running IS-IS in IP-only mode (CLNS forwarding not allowed)
show clns
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
221
Rtr-B#show clns protocol
IS-IS Router: <Null Tag> System Id: 1921.6800.1001.00 IS-Type: level-1-2 Manual area address(es): 49.0001 Routing for area address(es): 49.0001 Interfaces supported by IS-IS: Serial1 - IP Serial0 - IP Redistribute: static (on by default) Distance for L2 CLNS routes: 110
show clns protocol
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
222
Are the Hellos Sent?
Interface up/line protocol up Are bits exchanged: sh cdp neigh
detail Show clns neighbor Debug isis adj-packet
223
Rtr-B#show ip int briefInterface IP-Address OK? Method Status ProtocolLoopback0 192.168.1.1 YES NVRAM up up
Serial0 192.168.222.1 YES manual up up
Serial1 192.168.120.10 YES manual up up
Check Interface Status
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
224
Rtr-B#show cdp neighborCapability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater
Device ID Local Intrfce Holdtme Capability Platform Port IDRtr-C Ser 0 125 R 2500 Ser 1Rtr-A Ser 1 148 R 2500 Ser 0
Check CDP Neighbor Connectivity
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
225
Are the Adjacencies Up?
Entry in “show clns neighbor”? Protocol is IS-IS (not ES-IS)? State is up (should not show init)? IP addresses on both sides match? MTU size OK?
226
Rtr-B#show clns neighbors
System Id Interface SNPA State Holdtime Type ProtocolRtr-C Se0 *HDLC* Up 23 L2 IS-IS1921.6800.1005 Se1 *HDLC* Up 21 L1 IS-IS
Check CLNS Neighbors
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
227
Rtr-B#sh clns nei det
System Id Interface SNPA State Holdtime Type ProtocolRtr-C Se0 *HDLC* Up 23 L2 IS-IS Area Address(es): 49.0002 IP Address(es): 192.168.222.2* Uptime: 03:09:191921.6800.1005 Se1 *HDLC* Up 27 L1 IS-IS Area Address(es): 49.0001 IP Address(es): 192.168.120.5* Uptime: 03:50:01
Check CLNS Neighbor Detail
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
228
Rtr-B#show clns neighbors
System Id Interface SNPA State Holdtime Type ProtocolRtr-C Se0 *HDLC* Up 23 L2 IS-IS1921.6800.1005 Se1 *HDLC* Up 21 L1 ES-IS
If Protocol Field Shows ES-IS?
Misconfigured IP interface subnet Cisco IOS® validates source IP
address of neighbor before bringing up adjacency
No “ip router isis” command on interface
Common Causes:
229
Mixing IS-IS IP and IS-IS OSI
Can cause protocol field to show ES-IS!
Mode is per router, not per interface Using passive-interface forces IP mode
All routers in an area must agree IP-only, CLNS-only or integrated (dual)
Can only mix areas, not inside areas L2 routing not guaranteed to work
230
Rtr-B#show clns int serial1Serial1 is up, line protocol is up Checksums enabled, MTU 1500, Encapsulation HDLC ERPDUs enabled, min. interval 10 msec. CLNS fast switching enabled CLNS SSE switching disabled DEC compatibility mode OFF for this interface Next ESH/ISH in 47 seconds Routing Protocol: IS-IS Circuit Type: level-1-2 Interface number 0x2, local circuit ID 0x101 Level-1 Metric: 10, Priority: 64, Circuit ID: Rtr-A.00 Number of active level-1 adjacencies: 1 Level-2 Metric: 10, Priority: 64, Circuit ID: Rtr-B.01 Number of active level-2 adjacencies: 0 Next IS-IS Hello in 6 seconds
show clns interface
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
231
Rtr-B#show clns int serial0Serial0 is up, line protocol is up Checksums enabled, MTU 1500, Encapsulation HDLC ERPDUs enabled, min. interval 10 msec. CLNS fast switching enabled CLNS SSE switching disabled DEC compatibility mode OFF for this interface Next ESH/ISH in 30 seconds Routing Protocol: IS-IS Circuit Type: level-1-2 Interface number 0x1, local circuit ID 0x100 Level-1 Metric: 10, Priority: 64, Circuit ID: Rtr-C.01 Number of active level-1 adjacencies: 0 Level-2 Metric: 10, Priority: 64, Circuit ID: Rtr-B.00 Number of active level-2 adjacencies: 1 Next IS-IS Hello in 6 seconds
show clns interface
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
232
Rtr-B#debug isis adj-packetsIS-IS Adjacency related packets debugging is onRtr-B#05:45:21: ISIS-Adj: rcvd state UP, old state UP, new state UP05:45:21: ISIS-Adj: Action = ACCEPT05:45:24: ISIS-Adj: Sending serial IIH on Serial0, length 149905:45:26: ISIS-Adj: Rec serial IIH from *HDLC* (Serial1), cir type L1, cir id 00, length 149905:45:26: ISIS-Adj: rcvd state UP, old state UP, new state UP05:45:26: ISIS-Adj: Action = ACCEPT05:45:26: ISIS-Adj: Sending serial IIH on Serial1, length 149905:45:31: ISIS-Adj: Rec serial IIH from *HDLC* (Serial0), cir type L1L2, cir id 01, length 1499
debug isis adj-packets
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
233
Are the LSPs Flooded?
show isis database Compare sequence numbers Compare checksum or content debug isis update-packets debug isis snp-packets show clns traffic
LSP FLOODING AND CONTENTS
235
Rtr-B#show isis database
IS-IS Level-1 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000020 0x0C24 674 1/0/01921.6800.1005.00-00 0x00000023 0x909E 830 0/0/01921.6800.1005.01-00 0x00000017 0xC896 841 0/0/0IS-IS Level-2 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000024 0x7D98 748 0/0/0Rtr-C.00-00 0x00000028 0x1E01 1128 0/0/0
show isis database
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
236
Rtr-B#debug isis update-packetsIS-IS Update related packet debugging is on
Rtr-B#05:45:21: ISIS-Upd: Rec L2 LSP 1921.6800.2002.00-00, seq 24E, ht 1199,05:45:21 : ISIS-Upd: from SNPA *HDLC* (Serial0)05:45:21 : ISIS-Upd: LSP newer than database copy05:45:21 : ISIS-Upd: No change05:45:21 : ISIS-Upd: Refreshing L2 1921.6800.1001.00-0005:45:21 : ISIS-Upd: Sending L2 LSP 1921.6800.1001.00-00, seq 25E, ht 1199 on Serial005:45:21 : ISIS-Upd: Rec L2 LSP 1921.6800.2002.00-00, seq 24F, ht 1199,05:45:21 : ISIS-Upd: from SNPA *HDLC* (Serial0)05:45:21 : ISIS-Upd: LSP newer than database copy05:45:21 : ISIS-Upd: No change05:45:21 : ISIS-Upd: Refreshing L2 1921.6800.1001.00-0005:45:21 : ISIS-Upd: Sending L2 LSP 1921.6800.1001.00-00, seq 25F, ht 1199 on Serial0
debug isis update-packets
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
237
Rtr-B#debug isis snp-packetsIS-IS CSNP/PSNP packets debugging is onRtr-B#07:51:59: ISIS-Snp: Build L2 PSNP entry for 1921.6800.2002.00-00, seq 3507:51:59: ISIS-Snp: Sending L2 PSNP on Serial007:53:50: ISIS-Snp: Rec L1 PSNP from 1921.6800.1005 (Serial1)07:53:50: ISIS-Snp: PSNP entry 1921.6800.1001.00-00, seq 31, ht 119707:53:50: ISIS-Snp: Same entry 1921.6800.1001.00-00, seq 3107:54:26: ISIS-Snp: Build L1 PSNP entry for 1921.6800.1005.00-00, seq 2F07:54:26: ISIS-Snp: Sending L1 PSNP on Serial107:55:18: ISIS-Snp: Rec L2 PSNP from 1921.6800.2002 (Serial0)07:55:18: ISIS-Snp: PSNP entry 1921.6800.1001.00-00, seq 32, ht 119707:55:18: ISIS-Snp: Same entry 1921.6800.1001.00-00, seq 32
debug isis snp-packets
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
238
Does the LSP Contain ALL Info? LSPs are identified by:
LSP identifier (8 bytes—sysID, n-sel, frag.) Sequence number (higher means newer
LSP) Remaining lifetime (expiration purges LSP) Checksum (if corrupt discard—sender r-tx’s)
show isis database detail! Correct IP prefixes and metrics present? debug isis local-updates
239
Rtr-B#sh isis dat det 1921.6800.1001.00-00
IS-IS Level-1 LSP Rtr-B.00-00LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000001 0x20D5 1015 1/0/0 Area Address: 49.0001 NLPID: 0xCC Hostname: Rtr-B IP Address: 192.168.1.1 Metric: 0 IP 192.168.1.1/32 Metric: 10 IP 192.168.222.0/24 Metric: 10 IP 192.168.120.0/24 Metric: 10000 IS-Extended 1921.6800.1005.00IS-IS Level-2 LSP Rtr-B.00-00LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000263 0xAD32 1184 0/0/0 Area Address: 49.0001 NLPID: 0xCC Hostname: Rtr-B IP Address: 192.168.1.1 Metric: 10000 IS-Extended Rtr-C.00 Metric: 10 IP 192.168.120.0/24 Metric: 0 IP 192.168.1.1/32 Metric: 20 IP 192.168.1.5/32 Metric: 10 IP 192.168.222.0/24
show isis database detail
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
240
tr-B#sh debugCLNS: IS-IS local updates debugging is onRtr-B#conf tEnter configuration commands, one per line. End with CNTL/Z.Rtr-B(config)#int serial 1Rtr-B(config-if)#shutRtr-B(config-if)#07:59:26: ISIS-Loc: IP route adjust (level-1-2/Serial1/192.168.120.10)07:59:26 : ISIS-Upd: Building L1 LSP07:59:26 : ISIS-Upd: Building L2 LSP07:59:26 : ISIS-Upd: Building L2 LSP07:59:26 : ISIS-Loc: IP route adjust (level-1-2/Serial1/192.168.120.10)07:59:26 : ISIS-Upd: Building L1 LSP07:59:26 : ISIS-Upd: Building L2 LSP07:59:26 : ISIS-Loc: L2 non-summarized metric change (4294967295->20), 192.168.1.507:59:26 : ISIS-Upd: Building L2 LSP
debug isis local-updates
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
241
Rtr-B#show isis lsp-log Level 1 LSP log When Count Interface Triggers01:50:44 1 CONFIG01:50:35 1 Loopback0 IPUP01:50:28 1 Serial0 IPUP01:50:20 1 Serial1 IPUP01:50:20 1 Serial1 NEWADJ01:50:18 1 ATTACHFLAG01:36:49 1 Loopback0 CONFIG
Level 2 LSP log When Count Interface Triggers01:50:46 1 CONFIG01:50:36 1 Loopback0 IPUP01:50:30 2 Serial0 NEWADJ IPUP01:50:22 1 Serial1 IPUP01:50:10 1 IPIA01:48:21 1 Serial0 DELADJ01:48:16 1 Serial0 NEWADJ01:36:51 1 Loopback0 CONFIG
show isis lsp-log
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
242
LSP Checksum
Depending on Layer 2 CRC is not enough, corruption happens in routers and switches
Compute checksum of received LSP and check against checksum inside LSP
If corrupt, silently discard LSP Sender will always retransmit the LSP
Detection of LSP Corruption During Flooding
243
Does SPF Calculate Correct Routes?
show isis topology debug isis spf-events debug isis spf-statisticsAlso: show ip route isis debug ip routing debug clns routing
SPF COMPUTATION
245
Rtr-B#show isis topology
IS-IS paths to level-1 routersSystem Id Metric Next-Hop Interface SNPARtr-B --1921.6800.1005 10 1921.6800.1005 Se1 *HDLC*
IS-IS paths to level-2 routersSystem Id Metric Next-Hop Interface SNPARtr-B --Rtr-C 10 Rtr-C Se0 *HDLC*
show isis topology
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
246
Rtr-B#debug isis spf-eventsIS-IS SPF events debugging is onRtr-B#conf tEnter configuration commands, one per line. End with CNTL/Z.Rtr-B(config)#int serial 0Rtr-B(config-if)#shutRtr-B(config-if)#07:46:22: ISIS-Spf: L2 LSP 1 (1921.6800.2002.00-00) flagged for recalculationfrom 35F69EE07:46:22: ISIS-Spf: L1 LSP 2 (1921.6800.1001.00-00) flagged for recalculation from360CF3E07:46:22: %LINK-5-CHANGED: Interface Serial0, changed state to administratively down07:46:22: ISIS-Spf: Calculating routes for L2 LSP 1 (1921.6800.2002.00-00)07:46:22: ISIS-Spf: Add 192.168.111.0/255.255.255.0 to IP RIB, metric 2007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Add 192.168.2.2/255.255.255.255 to IP RIB, metric 1007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Add 192.168.2.4/255.255.255.255 to IP RIB, metric 2007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Add 192.168.222.0/255.255.255.0 to IP RIB, metric 2007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Aging L2 LSP 1 (1921.6800.2002.00-00), version 51607:46:22: ISIS-Spf: Calculating routes for L1 LSP 2 (1921.6800.1001.00-00)07:46:22: ISIS-Spf: Aging L1 LSP 2 (1921.6800.1001.00-00), version 144
debug isis spf-events
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
247
Rtr-B#debug isis spf-statisticsIS-IS SPF Timing and Statistics Data debugging is onRtr-B#conf tEnter configuration commands, one per line. End with CNTL/Z.Rtr-B(config)#int serial 0Rtr-B(config-if)#shutRtr-B(config-if)#no shutRtr-B(config-if)#07:49:37: %LINK-3-UPDOWN: Interface Serial0, changed state to up07:49:37: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-C (Serial0) Up, new adjacency07:49:37: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to up07:49:37: ISIS-Spf: Compute L2 SPT07:49:37: ISIS-Spf: Complete L2 SPT,07:49:37: ISIS-Spf: Compute time 0.008/0.008, 2/0 nodes, 1/0 links, 0 suspendsRtr-B(config-if)#07:49:37: ISIS-Spf: Compute L1 SPT07:49:37: ISIS-Spf: Complete L1 SPT,07:49:37: ISIS-Spf: Compute time 0.004/0.004, 2/0 nodes, 1/0 links, 0 suspends
debug isis spf-statistics
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
248
Is the Change Noticed?
show isis spf-log debug isis spf-trigger show isis lsp-log on generating
router show clns traffic to watch for PRCs
249
Rtr-B#show isis spf-log
Level 1 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:52 0 1 1 Rtr-B.00-00 NEWLSP02:16:42 0 1 1 Rtr-B.00-00 TLVCODE02:16:32 0 1 2 Rtr-B.00-00 NEWADJ TLVCONTENT02:16:22 8 3 4 Rtr-B.00-00 ATTACHFLAG LSPHEADER
TLVCON TENT02:02:57 4 3 1 Rtr-B.00-00 TLVCONTENT02:01:52 8 3 1 PERIODIC01:46:52 8 3 1 PERIODIC01:31:53 8 3 1 PERIODIC01:16:52 8 3 1 PERIODIC01:01:52 8 3 1 PERIODIC00:46:52 8 3 1 PERIODIC00:31:51 8 3 1 PERIODIC00:16:51 8 3 1 PERIODIC00:01:50 64 3 1 PERIODIC
show isis spf-log
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
250
Level 2 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:54 0 1 1 Rtr-B.00-00 NEWLSP02:16:44 0 1 1 Rtr-B.00-00 TLVCODE02:16:34 8 2 3 Rtr-B.00-00 NEWADJ NEWLSP TLVCONTENT02:14:29 8 2 3 Rtr-B.00-00 NEWADJ TLVCONTENT02:14:23 4 2 1 Rtr-C.00-00 TLVCODE02:13:56 8 2 1 Rtr-C.00-00 TLVCONTENT02:02:59 4 2 1 Rtr-B.00-00 TLVCONTENT02:01:54 4 2 1 PERIODIC01:46:54 4 2 1 PERIODIC01:31:54 4 2 1 PERIODIC01:16:54 4 2 1 PERIODIC01:01:54 4 2 1 PERIODIC00:46:53 4 2 1 PERIODIC00:31:53 4 2 1 PERIODIC00:16:53 4 2 1 PERIODIC00:01:53 60 2 1 PERIODIC
show isis spf-log
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
251
Rtr-B#debug isis spf-triggersIS-IS SPF triggering events debugging is onRtr-B#07:32:10: ISIS-Spf: L1 SPF needed, periodic SPF, from 0x356C8DC07:32:10: ISIS-Spf: L2 SPF needed, periodic SPF, from 0x356C8DCRtr-B#conf tRtr-B(config)#int serial0Rtr-B(config-if)#isis metric 15Rtr-B(config-if)# ^Z07:38:27: ISIS-Spf: L1 SPF needed, new metric, from 0x3560762Rtr-B(config)#int serial0Rtr-B(config-if)#shutRtr-B(config-if)# ^Z07:39:23: ISIS-Spf: L2, 1921.6800.1001.00-00 TLV contents changed, code 0x207:39:28: ISIS-Spf: L1 SPF needed, L2 attach changed, from 0x357CF3607:39:28: ISIS-Spf: L1, LSP fields changed 1921.6800.1001.00-00
debug isis spf-triggers
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
252
Monitoring Performance
show proc cpu IS-IS adjacency process(es) Send and receives hellos Manages adjacency database DIS election
Should be low CPU usage (< 1%)
MONITORING PERFORMANCE
254
Monitoring Performance
IS-IS update process(es) SPF computation and flooding
CPU usage should fluctuate Don’t worry unless constant > 20% Distributed (CEF) switching should
help
255
Rtr-B# sh proc cpuCPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 15760 89776 175 0.00% 0.00% 0.00% 0 Load Meter 2 108116 6965 15522 0.08% 0.00% 0.00% 0 Exec 3 1205472 83405 14453 0.00% 0.25% 0.25% 0 Check heaps . . . 47 70580 121119 582 0.00% 0.00% 0.00% 0 CLNS Input 48 14184 79119 179 0.00% 0.00% 0.00% 0 ES-IS Routing 49 205748 206607 995 0.00% 0.00% 0.00% 0 ISIS Adj 50 34596 52216 662 0.00% 0.00% 0.00% 0 ISIS Upd
show proc cpu
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
256
show clns traffic LSPs sourced indicates stability of IS LSP retransmissions should stay low PRCs can not be checked elsewhere LSP checksum errors are a bad sign Update queue should not stay full Update queue should not drop much
257
Rtr-B#show clns trafficCLNS: Time since last clear: neverCLNS & ESIS Output: 669, Input: 4773CLNS Local: 0, Forward: 0CLNS Discards: Hdr Syntax: 0, Checksum: 0, Lifetime: 0, Output cngstn: 0 No Route: 0, Discard Route: 0, Dst Unreachable 0, Encaps. Failed: 0 NLP Unknown: 0, Not an IS: 0CLNS Options: Packets 0, total 0 , bad 0, GQOS 0, cngstn exprncd 0CLNS Segments: Segmented: 0, Failed: 0CLNS Broadcasts: sent: 0, rcvd: 0Echos: Rcvd 0 requests, 0 replies Sent 0 requests, 0 repliesESIS(sent/rcvd): ESHs: 0/0, ISHs: 669/660, RDs: 0/0, QCF: 0/0ISO-IGRP: Querys (sent/rcvd): 0/0 Updates (sent/rcvd): 0/0ISO-IGRP: Router Hellos: (sent/rcvd): 0/0ISO-IGRP Syntax Errors: 0
IS-IS: Time since last clear: neverIS-IS: Level-1 Hellos (sent/rcvd): 282/0
show clns traffic
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
258
IS-IS: Level-2 Hellos (sent/rcvd): 285/0IS-IS: PTP Hellos (sent/rcvd): 420/415IS-IS: Level-1 LSPs sourced (new/refresh): 8/2IS-IS: Level-2 LSPs sourced (new/refresh): 9/1IS-IS: Level-1 LSPs flooded (sent/rcvd): 5/8IS-IS: Level-2 LSPs flooded (sent/rcvd): 7/8IS-IS: LSP Retransmissions: 0IS-IS: Level-1 CSNPs (sent/rcvd): 1/1IS-IS: Level-2 CSNPs (sent/rcvd): 2/2IS-IS: Level-1 PSNPs (sent/rcvd): 7/4IS-IS: Level-2 PSNPs (sent/rcvd): 7/5IS-IS: Level-1 DR Elections: 1IS-IS: Level-2 DR Elections: 1IS-IS: Level-1 SPF Calculations: 7IS-IS: Level-2 SPF Calculations: 9IS-IS: Level-1 Partial Route Calculations: 1IS-IS: Level-2 Partial Route Calculations: 5IS-IS: LSP checksum errors received: 0IS-IS: Update process queue depth: 0/200IS-IS: Update process packets dropped: 0
show clns traffic
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
259
show isis spf-log
How often are SPFs run? Doesn’t really matter how often Frequent SPFs can indicate a problem In a stable network—only periodic SPF
runs Who triggered the SPF?!
Check the LSPID of first trigger LSP Helps to find the source of the problem
260
Rtr-B#show isis spf-log
Level 1 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:52 0 1 1 Rtr-B.00-00 NEWLSP02:16:42 0 1 1 Rtr-B.00-00 TLVCODE02:16:32 0 1 2 Rtr-B.00-00 NEWADJ TLVCONTENT02:16:22 8 3 4 Rtr-B.00-00 ATTACHFLAG LSPHEADER
TLVCON TENT02:02:57 4 3 1 Rtr-B.00-00 TLVCONTENT02:01:52 8 3 1 PERIODIC01:46:52 8 3 1 PERIODIC01:31:53 8 3 1 PERIODIC01:16:52 8 3 1 PERIODIC01:01:52 8 3 1 PERIODIC00:46:52 8 3 1 PERIODIC00:31:51 8 3 1 PERIODIC00:16:51 8 3 1 PERIODIC00:01:50 64 3 1 PERIODIC
show isis spf-log
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
261
show isis spf-log
Normally in the region of msecs 400 node network takes around 50msecs If closer to seconds then maybe an issue (Could mean LSP re-transmissions over
p2p links) If so, can change lsp-retransmit-timer
How Much CPU Time Is Used per SPF?
262
Level 2 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:54 0 1 1 Rtr-B.00-00 NEWLSP02:16:44 0 1 1 Rtr-B.00-00 TLVCODE02:16:34 8 2 3 Rtr-B.00-00 NEWADJ NEWLSP TLVCONTENT02:14:29 8 2 3 Rtr-B.00-00 NEWADJ TLVCONTENT02:14:23 4 2 1 Rtr-C.00-00 TLVCODE02:13:56 8 2 1 Rtr-C.00-00 TLVCONTENT02:02:59 4 2 1 Rtr-B.00-00 TLVCONTENT02:01:54 4 2 1 PERIODIC01:46:54 4 2 1 PERIODIC01:31:54 4 2 1 PERIODIC01:16:54 4 2 1 PERIODIC01:01:54 4 2 1 PERIODIC00:46:53 4 2 1 PERIODIC00:31:53 4 2 1 PERIODIC00:16:53 4 2 1 PERIODIC00:01:53 60 2 1 PERIODIC
show isis spf-log
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
263
show isis lsp-log
How often do we generate a new LSP?
One router can influence whole net Why did it generate a new LSP? Flapping adjacency is shown by
interface show clns neighbour detail
Look at the “uptime value”
264
Rtr-B#show isis lsp-log
Level 1 LSP log When Count Interface Triggers01:50:44 1 CONFIG01:50:35 1 Loopback0 IPUP01:50:28 1 Serial0 IPUP01:50:20 1 Serial1 IPUP01:50:20 1 Serial1 NEWADJ01:50:18 1 ATTACHFLAG01:36:49 1 Loopback0 CONFIG Level 2 LSP log When Count Interface Triggers01:50:46 1 CONFIG01:50:36 1 Loopback0 IPUP01:50:30 2 Serial0 NEWADJ IPUP01:50:22 1 Serial1 IPUP01:50:10 1 IPIA01:48:21 1 Serial0 DELADJ01:48:16 1 Serial0 NEWADJ01:36:51 1 Loopback0 CONFIG
show isis lsp-log
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
265
Traffic Engineering
Support for MPLS-TE: Traffic Engineering with Multi Protocol La
bel Switching IS-IS allows MPLS-TE to flood resourc
e, policy and reservation information about links inside LSPs
New information carried in sub-TLVs IS-IS Metrics were extended for TE
ADVANCED/NEW FEATURES
267
Traffic Engineering: IS-IS Metrics
The interface metric was increased from 6 bits wide to 24 bits wide
The total path metric was increased from 10 bits wide to ~32 bits wide
Can configure the old or new metrics Default is old style metrics However—must configure wide
metrics to use TE
268
Traffic Engineering: IS-IS Metric Styles
Narrow Use old style of TLVs with narrow metric
Wide Use new style of TLVs to carry wider metric
Transition Used when migrating from old to new format Router will accept both old and new style
metrics Caveat: Should only be used in transitioning
Current Metric Styles That May Be Present:
269
Rtr-C#sh isis dat det
IS-IS Level-1 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-C.00-00 * 0x00000080 0x2237 958 1/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-C IP Address: 192.168.2.2 Metric: 0 IP 192.168.2.2/32 Metric: 10 IP 192.168.111.0/24 Metric: 10 IP 192.168.222.0/24 Metric: 10000 IS-Extended Rtr-D.00Rtr-D.00-00 0x00000076 0xB426 1137 0/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-D IP Address: 192.168.2.4 Metric: 0 IP 192.168.2.4/32 Metric: 10 IP 192.168.111.0/24 Metric: 10000 IS-Extended Rtr-C.00
Traffic Engineering Support
sh isis database detail Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
270
IS-IS Level-2 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 0x0000007D 0x0BAD 670 0/0/0 Area Address: 49.0001 NLPID: 0xCC Hostname: Rtr-B IP Address: 192.168.1.1 Metric: 6250000 IS-Extended Rtr-C.00 Metric: 10 IP 192.168.120.0/24 Metric: 0 IP 192.168.1.1/32 Metric: 10 IP 192.168.1.5/32 Metric: 10 IP 192.168.222.0/24Rtr-C.00-00 * 0x00000080 0xA130 1141 0/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-C IP Address: 192.168.2.2 Metric: 10000 IS-Extended Rtr-B.00 Metric: 10 IP 192.168.111.0/24 Metric: 0 IP 192.168.2.2/32 Metric: 10 IP 192.168.2.4/32 Metric: 10 IP 192.168.222.0/24
Traffic Engineering Support
sh isis database detail Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
271
hr3.iad.00-00 0x00001FE3 0x5FEE 34751 0/0/0 Area Address: 39.840f.8011.3824.0000.9000.0820 NLPID: 0xCC Hostname: hr3.iad Router ID: 204.152.166.8 IP Address: 204.152.166.8 Metric: 5 IS-Extended hr3.iad.03 Affinity: 0x00000000 Interface IP Address: 209.143.220.163 Physical BW: 100000000 bits/sec Reservable BW: 100000000 bits/sec BW Unreserved[0]: 100000000 bits/sec, BW Unreserved[1]: 100000000 bits/sec BW Unreserved[2]: 100000000 bits/sec, BW Unreserved[3]: 100000000 bits/sec BW Unreserved[4]: 100000000 bits/sec, BW Unreserved[5]: 100000000 bits/sec BW Unreserved[6]: 100000000 bits/sec, BW Unreserved[7]: 100000000 bits/sec Metric: 4 IS-Extended cr2.iad3.00 Affinity: 0x00000000 Interface IP Address: 206.132.253.62 Neighbor IP Address: 206.132.253.61 Physical BW: 155000000 bits/sec Reservable BW: 155000000 bits/sec BW Unreserved[0]: 155000000 bits/sec, BW Unreserved[1]: 155000000 bits/sec BW Unreserved[2]: 155000000 bits/sec, BW Unreserved[3]: 155000000 bits/sec BW Unreserved[4]: 155000000 bits/sec, BW Unreserved[5]: 155000000 bits/sec BW Unreserved[6]: 155000000 bits/sec, BW Unreserved[7]: 155000000 bits/sec
Traffic Engineering Support
sh isis database verbose (for TE BW info)
272
System ID: wr1.sfo1.00Router ID: 206.132.110.69Link Count: 9 Link[1] Neighbor System ID: wr2.sfo1.00 (P2P link) Interface IP address: 206.132.110.73 Neighbor IP Address: 206.132.110.74 Admin. Weight: 1 Physical BW: 2488000000 bits/sec Reservable BW: 2480000000 bits/sec BW unreserved[0]: 2480000000 bits/sec, BW unreserved[1]: 2420891904 bits/sec BW unreserved[2]: 2420771840 bits/sec, BW unreserved[3]: 2356474880 bits/sec BW unreserved[4]: 2313820928 bits/sec, BW unreserved[5]: 2313820928 bits/sec BW unreserved[6]: 2313820928 bits/sec, BW unreserved[7]: 2313820928 bits/sec Affinity Bits: 0x00000000
Traffic Engineering Support
sh isis mpls traffic-eng advertisementsAlso to View Bandwidth Information Use:
273
sh isis mpls traffic-eng tunnel
Rtr-B# show isis mpls traffic-eng tunnel Station Id Tunnel Name Bandwidth Nexthop Metric ModeRtr-C.00 Tunnel1022 3333 2.2.2.2 -3 Relative Tunnel1021 10000 2.2.2.2 11 AbsoluteRtr-D.00 Tunnel1031 10000 3.3.3.3 -1 Relative Tunnel1032 10000 3.3.3.3
Traffic Engineering Support
To View Tunnel Information:
274
Rtr-B#show isis mpls traffic-eng adjacency-log IS-IS RRR logWhen Neighbor ID IP Address Interface Status Level04:52:52 0000.0024.0004.02 0.0.0.0 Et0/2 Up level-104:52:50 0000.0026.0001.00 170.1.1.2 PO1/0/0 Up level-104:52:37 0000.0024.0004.02 0.0.0.0 Et0/2 Up level-1
Traffic Engineering Support
sh isis mpls traffic-eng adjacency-log
To View Adjacency Information:
275
Rtr-B#debug isis update-packetsSep 2 19:03:19.591: ISIS-Update: Rec L1 LSP 3333.3333.3333.00-00, seq B5D, ht 1199,Sep 2 19:03:19.591: ISIS-Update: from SNPA *HDLC* (Serial2/2)Sep 2 19:03:19.591: ISIS-Update: LSP newer than database copySep 2 19:03:19.591: ISIS-Update: TLV code mismatch (87, 80)Sep 2 19:03:19.591: ISIS-Update: TLV contents different, code 87Sep 2 19:03:19.591: ISIS-Update: TLV code mismatch (16, 2)Sep 2 19:03:19.591: ISIS-Update: Full SPF required
Metric Style Issues Metric style issues An IS is configured for WIDE metrics and receives a TLV
with NARROW metrics: Mismatch TLVs 0x80 (TLV 128) and 0x87 (TLV 135) TLV 128 is IP Internal Reachability (Narrow) TLV 135 is Extended IP Reachability TLV (Wide)
276
Route Leaking
Feature to enable redistributing Level 2 IP routes into Level 1 areas
Enables Level 1-only routers to pick the best path to exit the area
Enables shortest-exit and MED for BGP Redistribution can be controlled via distr
ibute-lists IP-only feature (CLNS still uses stub)
277
Rtr-D#sh ip ro isisi ia 1.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 2.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 3.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 4.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 5.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 6.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 7.0.0.0/8 [115/74] via 192.168.111.2, Serial1 192.168.2.0/32 is subnetted, 2 subnetsi L1 192.168.2.2 [115/10] via 192.168.111.2, Serial1i L1 192.168.222.0/24 [115/20] via 192.168.111.2, Serial1i*L1 0.0.0.0/0 [115/10] via 192.168.111.2, Serial1
Routes highlighted are shown as IS-IS inter-area Routes originated on Rtr-B and leaked from Rtr-C
Verifying Route Leaking
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
Leaking from Rtr-C to Rtr-D:
278
Rtr-D#sh isis dat det
IS-IS Level-1 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-C.00-00 0x000000FB 0x8E85 620 1/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-C IP Address: 192.168.2.2 Metric: 0 IP 192.168.2.2/32 Metric: 10 IP 192.168.111.0/24 Metric: 10 IP 192.168.222.0/24 Metric: 10000 IS-Extended Rtr-D.00 Metric: 74 IP-Interarea 1.0.0.0/8 Metric: 74 IP-Interarea 2.0.0.0/8 Metric: 74 IP-Interarea 3.0.0.0/8 Metric: 74 IP-Interarea 4.0.0.0/8 Metric: 74 IP-Interarea 5.0.0.0/8 Metric: 74 IP-Interarea 6.0.0.0/8 Metric: 74 IP-Interarea 7.0.0.0/8
Verifying Route Leaking
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
Leaking from Rtr-C to Rtr-D:
279
HMAC-MD5 Authentication Supported platforms:
12000, 10720, 10000, 7500, 7200 Supported across all packet types:
Hellos, LSPs, CSNPs, PSNPs Can authenticate under the IS-IS process
(for LSP/CSNP/PSNP) and/or on the interface(for hellos)
Can authenticate at Level 1 and/or Level 2
280
!key chain cisco key 100 key-string systems!interface Serial1/0 ip address 10.1.1.1 255.255.255.252 ip router isis isis authentication mode md5 isis authentication key-chain cisco!router isis net 49.0000.0101.0101.0101.00 authentication mode md5 authentication key-chain cisco
HMAC-MD5 Authentication
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
Typical Configuration:
281
Rtr-A#sh key chain
Key-chain cisco:
key 100 -- text "systems"
accept lifetime (always valid) - (always valid) [valid now]
send lifetime (always valid) - (always valid) [valid now]
Rtr-A#sh clns nei
System Id Interface SNPA State Holdtime Type Protocol
Rtr-B Se2/2 *HDLC* Up 28 L1L2 IS-IS
HMAC-MD5 Authentication
Use “sh clns neighbor” to verify adjacency
Use “sh key chain” to view key chain information
282
Rtr-A#debug isis authentication informationSep 23 15:41:45.040: ISIS-AuthInfo: IIH no change, use the same hmac valueSep 23 15:41:54.880: ISIS-AuthInfo: IIH no change, use the same hmac valueSep 23 15:42:04.620: ISIS-AuthInfo: IIH no change, use the same hmac value
HMAC-MD5 Authentication Use “debug isis authentication information” Debug below shows normal operation as same
MD5 info is re-used if there is no change in IIH
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
283
Rtr-B#debug isis authentication informationSep 23 15:42:28.950: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:29.830: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:30.106: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:30.422: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:30.458: ISIS-AuthInfo: No auth TLV found in received packet
HMAC-MD5 Authentication Use “debug isis authentication information” Debug below shows that local router received
a packet that does not contain authentication data
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
284
Rtr-A#sh isis datIS-IS Level-1 Link State DatabaseLSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-A.00-00 * 0x00000002 0xDFE8 1195 0/0/0
IS-IS Level-2 Link State DatabaseLSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-A.00-00 * 0x00000002 0x5E9A 1195 0/0/0
HMAC-MD5 Authentication Neighbor does not use authentication for the
IS-IS instance No LSPs, CSNPs or PSNPs sent with
authentication No LSPs, CNPs or PSNPs received by
neighbor
285
Rtr-A#sh ip route
192.168.60.0/30 is subnetted, 1 subnetsC 192.168.60.4 is directly connected, FastEthernet0/0 192.168.110.0/32 is subnetted, 1 subnetsC 192.168.110.1 is directly connected, Loopback0 192.168.20.0/30 is subnetted, 1 subnetsC 192.168.20.12 is directly connected, Serial2/2 10.0.0.0/22 is subnetted, 1 subnetsC 10.200.96.0 is directly connected, Ethernet5/0
HMAC-MD5 Authentication
No neighbor routes installed in local RIB
286
Rtr-A#Sep 23 17:16:36.191: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-B (Serial2/2) Down, hold time expired
Rtr-A#sh clns neiSystem Id Interface SNPA State Holdtime Type Protocol
HMAC-MD5 Authentication Neighbor does not use authentication for the
interface No hellos sent with authentication No hellos received by neighbor Neighborship is taken down—hold time
expired
287
Rtr-B#Sep 23 17:16:43.102: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-A (Serial0/2) Down, neighbor forgot us
Rtr-B#sh clns neiSystem Id Interface SNPA State Holdtime Type ProtocolRtr-A Se0/2 *HDLC* Init 27 L1L2 IS-IS
HMAC-MD5 Authentication
Neighbor does not use authentication for the interface Neighborship taken down Non-authenticating router in Init state
288
Rtr-A#sh ip ro 192.168.60.0/30 is subnetted, 1 subnetsC 192.168.60.4 is directly connected, FastEthernet0/0 192.168.110.0/32 is subnetted, 1 subnetsC 192.168.110.1 is directly connected, Loopback0 192.168.20.0/30 is subnetted, 1 subnetsC 192.168.20.12 is directly connected, Serial2/2 10.0.0.0/22 is subnetted, 1 subnetsC 10.200.96.0 is directly connected, Ethernet5/0
HMAC-MD5 Authentication
Rtr-B#sh ip ro 192.168.30.0/30 is subnetted, 1 subnetsC 192.168.30.4 is directly connected, Serial0/0 192.168.150.0/30 is subnetted, 1 subnetsC 192.168.150.4 is directly connected, Ethernet3/2 192.168.20.0/30 is subnetted, 1 subnetsC 192.168.20.12 is directly connected, Serial0/2 10.0.0.0/22 is subnetted, 1 subnetsC 10.200.96.0 is directly connected, Ethernet3/0 192.168.220.0/32 is subnetted, 1 subnetsC 192.168.220.1 is directly connected, Loopback0
No Routes in RIB on Both Neighbors
289
Multi-Topology (MT) Support for IS-IS Now IPv4 and IPv6 topologies are
separate No need for IPv4 and IPv6 to be congruent
Lifts previous restrictions Much easier deployment of IPv6
Avoids previous possible complications Supported from 12.0(26)S, 12.2(15)T Draft-ietf-isis-wg-multi-topology-xx.txt
290
MT for IPv6: Hello Processing
Basic hello processing and checking is the same MT membership is advertised in IIH packets
MT ID #0 for IPv4 MT ID #2 for IPv6
Must have at least one common set of topology types to form an adjacency
LAN nodes will always establish an adjacency regardless of MT for reliable flooding and sync
Maintaining MT Adjacencies
291
MT for IPv6: LSP Generation and Flooding
The LSP flooding mechanism is unchanged for multi-topology integrated IS-IS
For LANs—DIS, CSNP and PSNP functions are unchanged by MT extension
Use standard show and debug commands to troubleshoot
292
MT for IPv6: SPF and RIBs Each topology (IPv4 and IPv6) runs its own SPF During IPv6 SPF we examine TLV 222 for MT ID
#2 to build the Shortest Path Tree Two-Way Connectivity Check (TWCC) follows TWCC ensures bidirectional reachability
For leaf nodes we examine TLV 237 for MT ID #2 and install the IPv6 prefixes in the IPv6 RIB
293
MT for IPv6: Memory and Performance
Additional memory will be required However—will not have major impact on
overall system requirements Performance impact will be additional
SPF runtime to compute L1 and L2 IPv6 topology However IPv6 SPF not run back-to-back
with IPv4—but interleaved so no CPU hogging
294
MT for IPv6: Restrictions Not compatible with previous implementation
Not compatible with single SPF IPv6 However transition mode possible to migrate
from current IPv6 to MT IPv6 Advertises both IPv6 and MT IPv6 TLVs
If IPv4 and IPv6 are configured on the same interface—they must be running the same level
Must use wide metrics
295
Multi-Topology IS-IS Example
IPv4-IPv6 Enable Router
Area A
Area BArea C
Area D
IPv4-Only Enable Router
The Multi-Topology Software Will Create Two Topologies Inside Area IPv4 and IPv6
IPv4-Only Routers Will Be Excluded from the IPv6 Topology
296
Area B
MT for IPv6: Basic Configuration
The optional keyword TRANSITION may be used for transitioning existing IS-IS IPv6 single SPF mode to MT IS-IS
Wide metric is mandated for Multi-Topology to work
Router1# interface ethernet-1 ip address 10.1.1.1 255.255.255.0 ipv6 address 2001:0001::45c/64 ip router isis ipv6 router isis isis ipv6 metric 20
interface ethernet-2 ip address 10.2.1.1 255.255.255.0 ipv6 address 2001:0002::45a/64 ip router isis ipv6 router isis isis ipv6 metric 20
router isisnet 49.0000.0100.0000.0000.0500metric-style wide!address-family ipv6multi-topology exit-address-family
LAN1: 2001:0001::45c/64
LAN2: 2001:0002::45a/64
Ethernet-1
Ethernet-2
Router1
297
Cisco IOS Multi-Topology IS-IS Display
Router# show clns neighbors detailSystem Id Interface SNPA State Holdtime Type Protocol2653 Se0/1 *HDLC* Up 25 L1L2 M-ISIS
Area Address(es): 49.0000.01IP Address(es): 192.168.0.6*IPv6 Address(es): FE80::204:C1FF:FEDB:2FA0Uptime: 00:01:22Topology: IPv4, IPv6
2652# show isis database detailIS-IS Level-2 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL2651.00-00 0x0000000F 0x0161 1066 0/0/0
Area Address: 49.0000.01Topology: IPv4 (0x0) IPv6 (0x2)NLPID: 0xCC 0x8EHostname: 2651IP Address: 192.168.0.2IPv6 Address: 3FFF:FFFF:2::1Metric: 10 IS-Extended 2652.00Metric: 10 IS-Extended 2653.01Metric: 10 IS (MT-IPv6) 2653.01Metric: 10 IP 192.168.0.0/30Metric: 20 IP 192.168.0.4/30Metric: 10 IP 192.168.1.0/24Metric: 20 IPv6 (MT-IPv6) 3FFF:FFFF:1::/64Metric: 10 IPv6 (MT-IPv6) 3FFF:FFFF:2::/64
MT IS-IS
298
Local RIB
The Local RIB contains entries for all possible routes to destinations
From this RIB, the best route is chosen to be installed in the ‘IP routing table’
Local RIB is also necessary for prefix prioritization
Local RIB can significantly reduceconvergence time
299
Rtr-B#sh isis rib ? A.B.C.D Network prefix redistribution ISIS IP redistribution RIB information | Output modifiers <cr>
Rtr-B#sh isis rib 192.168.40.14 IPv4 local RIB for IS-IS process Routes under majornet 192.168.40.0/24:192.168.40.4/30 [115/L1/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[3/51] [115/L2/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[4/166] 192.168.40.16/30 [115/L1/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[3/51] [115/L2/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[4/166]
Viewing Local RIB Information
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
300
Rtr-B#debug isis rib ? global IS-IS IP routes in global RIB local IS-IS IP routes in local RIB redistribution IS-IS IP redistribution RIB
Rtr-B#debug isis rib local IS-IS IPv4 local RIB debugging is onRtr-B#*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.4/30: Looking for RT *Mar 15 01:18:34.882: ISIS-LR: RT exists*Mar 15 01:18:34.882: ISIS-LR: path exists: [115/40/20] via 192.168.40.5(Se0/0) from 192.168.40.17 tg 0 LSP[3/(51->52)]*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: Looking for RT *Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: Create new RT*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: create new path: [115/40/20] via 192.168.40.5(Se0/0) from 192.168.40.5 tag 0 LSP[3/52]*Mar 15 01:18:34.882: ISIS-LR: Enqueued to updateQ[2] for 192.168.40.12/30*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.16/30: Looking for RT *Mar 15 01:18:34.882: ISIS-LR: RT exists*Mar 15 01:18:34.882: ISIS-LR: path exists: [115/40/20] via 192.168.40.5(Se0/0) from 192.168.40.17 tg 0 LSP[3/(51->52)]*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: not aged out in LSP ix 3 same ver(52)*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.16/30: not aged out in LSP ix 3 same ver(52*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.4/30: not aged out in LSP ix 3 same ver(52)
Debugging Local RIB Information
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
301
Rtr-B#debug isis rib ? global IS-IS IP routes in global RIB local IS-IS IP routes in local RIB redistribution IS-IS IP redistribution RIB
Rtr-B#debug isis rib globalIS-IS IPv4 global RIB debugging is onRtr-B# *Mar 15 01:22:06.117: ISIS-GR: ------ Start updateQ, from 60773650*Mar 15 01:22:06.117: ISIS-GR: 192.168.210.1 255.255.255.255: updateQ[1] entry*Mar 15 01:22:06.117: ISIS-GR: del rdb *Mar 15 01:22:06.117: ISIS-GR: 192.168.250.12 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.117: ISIS-GR: del rdb *Mar 15 01:22:06.117: ISIS-GR: 192.168.250.4 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.121: ISIS-GR: del rdb *Mar 15 01:22:06.121: ISIS-GR: 192.168.30.16 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.121: ISIS-GR: del rdb *Mar 15 01:22:06.121: ISIS-GR: 192.168.20.4 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.121: ISIS-GR: del rdb *Mar 15 01:22:06.121: ISIS-GR: ------ End updateQ, 0 suspends*Mar 15 01:22:06.121: ISIS-GR: ------ Start updateQ, from 60773DE0*Mar 15 01:22:06.121: ISIS-GR: 192.168.40.12 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.125: ISIS-GR: del rdb *Mar 15 01:22:06.125: ISIS-GR: ------ End updateQ, 0 suspends
Debugging Global RIB Information
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
302
Incremental SPF (i-SPF)
Modified Dijkstra algorithm We keep the unchanged part of the tree We rebuild only the affected parts of
the tree Re-attach the affected parts of the tree
to the unchanged part of the tree Fairly complex algorithm
Incremental SPF
303
Incremental SPF (i-SPF) When new LSPs are received, each router will
check what has changed in the LSP Based on the changed information, the SPT is
“modified” in order to reflect the changes Start the computation from the node that
received the change i-SPF is essentially used to prepare lists for
running standard Dijkstra and computing next-hops
No need to run SPF if a link that was not in the SPT is reported down
304
S0
S1
S2S3
D
C-G Link Is Down; C-G Link Was Not Used in SPT Anyway, Therefore There Is No Need to Run SPF
Incremental SPF (i-SPF)
Cost: 0, NH: --A
Cost: 3, NH: BB
Cost: 11, NH: BE
Cost: 13, NH: DG
Cost: 6, NH: D, BC Cost: 3, NH: D
Cost: 8, NH: D, BF
305
S0
S1
S2S3
F Reports a New Neighbor; the SPT Need Only To Be Extended Behind F; There Is No Need for Router A to Recompute the Whole SPTRouter A Will Compute SPF from Node F
H
Incremental SPF (i-SPF)
Cost: 0, NH: --A
Cost: 3, NH: BB
Cost: 11, NH: BE
Cost: 13, NH: DG
Cost: 6, NH: D, BC Cost: 3, NH: DD
Cost: 8, NH: D, BF
306
Incremental SPF (i-SPF): Configuration When router is powered on—we always run a
full SPF to build the tree To configure i-SPF under router isis:
ispf [level-1/level-2/level-1-2] [1-600] Must specify levels The last parameter (1–600) is optional and
corresponds to the duration (in seconds) of full SPF runs after reload
By default, full SPF is run for the first 60 seconds after reload—even if i-SPF is configured
Integrated into 12.0(24)S
307
Rtr-A#debug isis spf-statistics*Mar 15 01:03:44.044: ISIS-Stats: Compute L1 SPT*Mar 15 01:03:44.044: ISIS-Stats: Starting incremental SPF for level-1*Mar 15 01:03:44.048: ISIS-Stats: SPF only compute time 0.000*Mar 15 01:03:44.048: ISIS-Stats: IPv4 RIB only compute time 0.004*Mar 15 01:03:44.048: ISIS-Stats: Complete L1 SPT, Compute time 0.004, 2 nodes, 0 links on SPT, 0 suspends*Mar 15 01:03:44.048: ISIS-Stats: I-SPF NewLSP: 0.000 (3 nodes) - Reattach: 0.000 - WalkTENT: 0.000*Mar 15 01:03:44.048: ISIS-Stats: I-SPF WalkParents: 0.000 (0 nodes) - Total time: 0.000
debug isis spf-statistics
L1 Routers
Area 49.0001
Rtr-A
Rtr-B S0
S1
S0
L1L2 Routers
Area 49.0002
Rtr-D
Rtr-C S0
S1
S1
308
Rtr-A#sh isis spf-log detail
level 1 SPF log
i-SPF triggered 2 times, first at 01:03:38.548 Mar 15 1993 by TLVCONTENT wait enforced 5.500, next wait interval 5.500 SPT node total/processed/time: 4/2/0.000 RIB prefix processed/time: 0/0.000 1/0.000 3/0.004
Full SPF triggered 2 times, first at 00:51:17.877 Mar 15 1993 by NEWADJ wait enforced 5.500, next wait interval 5.500 SPT node total/processed/time: 2/2/0.000 RIB prefix processed/time: 0/0.000 0/0.000 2/0.004
Viewing i-SPF
Area 49.0001
Rtr-A
Rtr-B
S0
S1
S0
L1 Router
L1L2 Routers
Area 49.0002
Rtr-C
Rtr-D
S1
S0
S1
L1 Router
309
Non-Stop Forwarding Allows uninterrupted data forwarding during a swi
tchover to a standby RP Routing protocol mechanisms are also applicab
le to a restart in the RP Forwarding Information Base (FIB) is maintained a
nd updated once the routing protocolsreconverge
May be used for planned and unplanned events Two modes of NSF available:
Cisco and IETF
310
NSF: Routing Protocol Requirements Switchover MUST be completed before dead/hold
timer expires Peers will reset the adjacency and reroute the
traffic after that time FIB MUST remain unchanged during switchover
Current routes marked as “dirty” during restart; “cleaned” once convergence is complete
Adjacencies MUST NOT be reset when switchover is complete Protocol state is not maintained
Peers of restarting router SHOULD also beNSF-aware
311
NSF Configuration Commands
nsf [cisco/ietf ] Enables isis nsf, OFF by default
nsf interval xxx Minimal time interval between two restart (default =5mi
n) nsf holdtime [manual <seconds> | adjacency]
IETF version only Time NSF will wait for the LSP database to synchronize befo
re generating and flooding its own LSP with the overload-bit set
To Configure NSF under “router isis”:
312
NSF Configuration and Debug Commands nsf interface wait xxx
Cisco version only (default=10 sec, range [1–60] ) Time an NSF restart will wait for all interfaces with ISIS adjacen
cies to come up before completing the restart Show commands
show isis nsf nsf capability and restart information
show clns neighbor detail peer’s nsf capability show isis database detail LSP information
Debug commands debug isis nsf [cisco | detail | ietf] debug isis adj-packets
313
IS-IS NSF “show commands”
NSF Cisco mode—check that it is enabled
chi-hr1#show isis nsf
NSF is ENABLED, mode 'cisco'
RP is ACTIVE, standby ready, bulk sync completeNSF interval timer expired (NSF restart enabled)Checkpointing enabled, no errorsLocal state: ACTIVE, Peer state: STANDBY HOT, Mode: SSO
We maintain the IS-IS Neighbor and Database Information on the STANDBY [G]RPGRP-Slot9#show clns neighbor
System Id Interface SNPA State Holdtime Type Protocolchi-ar1 PO2/0 *HDLC* Stby 30 L2 IS-IS
314
chi-hr1#sh isis nsf
NSF is ENABLED, mode 'ietf'NSF pdb state: InactiveNSF L2 active interfaces: 0NSF L2 active LSPs: 0NSF interfaces awaiting L2 CSNP: 0Awaiting L2 LSPs:NSF T3 remaining: 0 secondsInterface: Ethernet0 NSF L2 Restart state: Running NSF L2 Restart retransmissions: 0 Maximum L2 NSF Restart retransmissions: 3Interface: POS2/0 NSF L2 Restart state: Running NSF p2p Restart retransmissions: 0 Maximum L2 NSF Restart retransmissions: 3
IS-IS NSF “show commands”
NSF IETF mode—check that it is enabled
315
IS-IS NSF “show commands”
chi-hr1#show clns neighbor detailSystem Id Interface SNPA State Holdtime Type Protocolchi-ar1 PO2/0 *HDLC* Up 21 L2 IS-IS Area Address(es): 10 IP Address(es): 172.1.1.21* Uptime: 01:52:02 NSF capable
316
IS-IS NSF “debug commands”
Rtr-A#debug isis nsf? cisco Include only Cisco NSF information detail Include detailed information ietf Include only IETF NSF information
Other debug information also useful with NSF: debug isis update-packets debug isis snp-packets debug isis adj-packets
317
IS-IS NSF “debug isis nsf ietf”
*Aug 12 17:32:45.507 PDT: ISIS-NSF: Inserting p2p NSF REQ on POS3/0 IIH
*Aug 12 17:32:45.519 PDT: ISIS-NSF: POS3/0 level-2 state progression: state=Restarting/event=RA Rcvd/newstate=RA Seen
*Aug 12 17:32:45.519 PDT: ISIS-NSF: ISIS NSF restart ACK option received on p2p itf from 2002.0020.0001 (POS3/0)
*Aug 12 17:32:45.559 PDT: ISIS-NSF: POS3/0 level-2 state progression: state=RA Seen/event=CSNP Rcvd/newstate=Running
318
.Aug 23 14:10:18.926 PDT: ISIS-Update: Sending L2 P2P sync CSNP on PO2/0
.Aug 23 14:10:18.926 PDT: ISIS-Update: Sending L2 CSNP on POS2/0
.Aug 23 14:10:18.926 PDT: ISIS-SNP: Rec L2 PSNP from 2002.0020.0001 (POS2/0)
.Aug 23 14:10:18.926 PDT: ISIS-SNP: PSNP entry 2002.0020.0003.FE-FE, seq 0, ht 1199
.Aug 23 14:10:20.962 PDT: ISIS-Update: Sending L2 LSP 2002.0020.0003.00-00, seq F7, ht 1199 on POS2/0
.Aug 23 14:10:21.790 PDT: ISIS-Adj: Rec serial IIH from *HDLC* (POS2/0), cir type L2, cir id 00, length 4469
.Aug 23 14:10:21.790 PDT: ISIS-Adj: rcvd state UP, old state UP, new state UP
• debug isis update-packets
• debug isis snp-packets
• debug isis adj-packets
IS-IS NSF: Other Useful Debug Commands
Trouble Shooting CEF
320
Vocabulary
CEFCEF: Cisco Express Forwarding FIBFIB: Forwarding Information Base
collection of data used to make switching decision
• RIBRIB: Routing Information Basetopology information that the router learns from routing protocols
321
CEF...What is it?
Advanced layer-3, IPIP switching method; scaleable, distributed, high performance
CEF differs from current fast switchingfast switching in the way the router maintains its forwarding table
CEF switching is required for most of the new IP QoS features
322
Features
Load balancing Per destination (the default) and per
packet over equal/unequal cost links for as many paths as known in the routing topology
Traffic statistics Byte and packet counts at a
granularity of per-prefix, per-neighbor etc..
323
Features (cont..)
Media independence CEF currently supports Packet over Sonet,
ATM/AAL5, Frame Relay, Ethernet, FDDI, HDLC, PPP and tunnels.
Tunneling: Generic Route Encapsulation (GRE).
Subinterface support: allowing for the flexibility of per subinterface configurations e.g. MTU.
324
Today’s Problems and CEF’s Solutions
Type of traffic - EfficiencyToday’s traffic is short lived. Cache on demand (fast switching) requires constant cache create/invalidate (aging). FIB has a forwarding entry for every routable prefix.
Dynamic networks - PerformanceThe FIB table mirrors the routing table exactly - thus fast convergence
325
Today’s Problems and CEF’s Solutions (cont.)
Scalability Core routers carry high number of prefixe
s (mostly recursive) which need to be resolved each time a packet has to be process switched (cache on demand). In CEF, the recursive route is resolved when the FIB table is built. dCEF is also available.
Better use of memory and CPU.
326
Possible problem The throughput of the router is less th
an expected. The throughput of an interface is less
than expected. Packets are getting dropped when CE
F/dCEF is enabled Packets are not CEF switched but
fast switched instead.
327
Possible problem(cont.) Packets are not dCEF switched but fast swit
ched instead Packets are not dCEF switched but CEF swit
ched instead. Load-sharing is not working The route processor runs out of memory wh
en dCEF is enabled CEF disables itself on a linecard due to lack
of memory
The throughput of the router is less than expected.
Possible problem :CEF/dCEF is not enabled globally on the rou
ter.
329
Solution Step 1 Check is CEF/dCEF is enabled globally on ro
uter. Use global command ip cef for central processor s
witching or ip cef distributed if you want distributed CEF switching. When CEF/dCEF is enabled globally, CEF/dCEF is automatically enabled on the interfaces.
If show ip summary command gives an output that means CEF is enabled. If not, you get the following output for show ip summary:
router#show ip summary %CEF not running
The throughput of an interface is less than expected.
Possible problem : CEF/dCEF is not enabled on this interface.
331
Solution Step 1 Check if CEF/dCEF is enabled on the interfa
ce. Use interface command ip route-cache cef for central processor switching or ip route-cache distributed if you want distributed CEF switching.
The result of show cef interface x/x and show ip interface x/x must report one of the following:
IP CEF switching enabled IP Normal CEF switching turbo vector IP Distributed CEF switching enabled IP Distributed CEF switching turbo vector
Below is a example on a router:
332
Examplerouter#show cef interface FastEthernet0/1 FastEthernet0/1 is up (if_number 3) Corresponding hwidb fast_if_number 3 Corresponding hwidb firstsw->if_number 3 Internet address is 10.254.0.200/24 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set IP policy routing is disabled Hardware idb is FastEthernet0/1 Fast switching type 1, interface type 18 IP CEF switching enabled IP Feature Fast switching turbo vector IP Feature CEF switching turbo vector Input fast flags 0x0, Output fast flags 0x0 ifindex 2(2) Slot 0 Slot unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500
Packets are getting dropped when CEF/dCEF is enabled
Possible problem : The CEF entries and the adjacencies
table are not completely built.
334
Solution Step 1 Verify whether the router has completely built the CEF
entries and adjacencies table. If it is not the case the router may switch the packet using an alternative switching method. This is why the packets are 50% successful. To verify this, issue the command show ip cache. There should be an IP cache entry built for this prefix.
router#show ip cache IP routing cache 0 entries, 0 bytes 3 adds, 3 invalidates, 0 refcounts The above result of the command shows that the router has
been doing cache switching for a moment: 3 adds, 3 invalidates. The result of the ping is 50% successful as the number of packets that are punted is rate limited.
335
Solution(cont.) router #ping ip Target IP address: 12.0.0.1Repeat count [5]: 20 Datagram size [100]: Timeout in seconds [2]: Extended commands [n]: Sweep range of sizes [n]: Type escape sequence to abort.Sending 20, 100-byte ICMP Echoes to 12.0.0.1, timeout is 2 seconds:!.!.!.!.!.!.!.!.!.!.Success rate is 50 percent (10/20), round-trip min/avg/max = 1/1/1 msAn adjacency on interface POS8/0/0 is still incomplete. router#show adjacencyProtocol Interface AddressIP POS8/0/0 point2point(3)IP Serial10/0/0:28 point2point(3)IP POS8/0/0 point2point(2) (incomplete)
Packets are getting dropped when CEF/dCEF is enabled
Possible problem : Part of the adjacencies is “incomplete adjacency” or “drop adjacency”.
337
Solution Step 1: Check the features you have enabled; som
e platforms only support dCEF with a limited number of features. If you enable a feature not supported by dCEF a drop adjacency is created and the packets are dropped. Issue the command show cef drop and look for packets being dropped. Below you will find the output of the command when a feature is not supported.
router#show cef drop CEF Drop Statistics Slot Encap_fail Unresolved Unsupported No_r
oute No_adj RP 0 0 2 0 0
338
Solution(cont.) Step 2 If your hardware supports CEF and dCEF, in case of pa
cket drops you may see only “incomplete adjacency”. Regardless of the platform, here is the method to follow to isolate the problem.
Use show ip route to determine which inbound interface the traffic should be switched through. Issue show ip cef <prefix> and look for the interface adjacencies:
router#show ip cef 10.254.0.200 10.254.0.200/32, version 18, cached adjacency 10.254.0.200 0 packets, 0 bytes via 10.254.0.200, FastEthernet0/1, 0 dependencies next hop 10.254.0.200, FastEthernet0/1 valid cached adjacency
339
Solution(cont.)router#show adjacency FastEthernet 0/1 detailProtocol Interface AddressIP FastEthernet0/1 10.254.0.200(5) 0 packets, 0 bytes 00055FAF2C06 00055FAC18010800 ARP 03:45:10 IP FastEthernet0/1 server(5) (10.254.0.1) 0 packets, 0 bytes 00508B121E46 00055FAC18010800 ARP 03:54:27
Packets are getting dropped when CEF/dCEF is enabled
Possible problem : None of the above solutions are applying
– try to debug the issue.
341
Solution Step 1: Check if the packets are dropped or punt fo
r unexpected reason. The following counters are reporting the punt and dropped packets:
show cef not-cef-switched (punt reason counters)
show cef drop (drop reason counters) The above is particularly useful in a controlled envi
ronment. You send a number of packets. If all the packets are dropped or punt you it gives you the reason for this unexpected behavior.
342
Solution(cont.) Step 2: If the traffic to a destination is limite
d, you can try the following debug commands:
debug ip cef drop <acl> debug ip cef receive <acl> access-list <acl> permit <destination> The drop and receive debug information wil
l provide more information on why packets are being dropped or punted by CEF/dCEF.
Packets are getting dropped when CEF/dCEF is enabled
Possible problem :None of the above solutions are applying –
Call the TAC
344
Solutionstep 1: As always, for a CEF/dCEF unexpected behavior, collect: show tech cef Step 2: If you cannot get a destination and if none of the above solutions ar
e applying to your case, you will have to collect a list of information to help the Cisco TAC solving your problem.
show ip route <destination> show ip cef <destination> internal show adjacency <output interface > show ip interface brief show cef interface exec all show ip cef <destination> internal exec all show adjacency <output interface> detail exec all show cef interface show ip arp (or analogous link info, e.g. show frame-relay map)To get to a destination, there has to be a route, a forwarding entry, and a val
id adjacency. The data collected above will show which element is missing.
Packets are not CEF switched but fast switched instead.
Possible problem: CEF does not support a configured
feature.
346
Solution Step 1 While CEF supports a large range of f
eatures, however some features are not supported. If a feature is not supported packets are forwarded to the next slower level of switching - fast switching.
Check the following document to know exactly what is supported:
IOS features supported by CEF and dCEF matrix
347
Solution(cont.) Step 2 Check the configuration of the router to make
certain CEF is activated. Also use the following commands:
show cef interface x/y, show ip cef <prefix> , show adjacency <interface> detail Ultimately issue show cef not-switch and look for p
ackets that are not CEF switched. If CEF is not able to forward packets to a destination,
the CEF table is built anyway but the adjacency is called a “punt”. If just a part of the packets cannot be CEF switched a cache adjacency is created.
348
Solution(cont.)Router #show ip cef 172.20.1.2172.20.1.0/30, version 4, attached, connected, per-destination sharing0 packets, 0 bytes via Serial4/0, 1 dependency valid punt adjacencyAs the result of punt adjacency the router will report CEF packets passed on to next switching
layer.router#show cef interface serial4/0 Serial4/0 is up (if_number 13) Corresponding hwidb fast_if_number 13 Corresponding hwidb firstsw->if_number 13 Internet address is 172.20.1.1/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set IP policy routing is disabled Interface is marked as point to point interface Packets switched to this interface are dropped to the next slow path
349
Solution(cont.)Step3 Look under interface/global configuration
for features that you are not sure to be compatible with CEF. Log messages from router, then remove and put back one by one each of these features.
When removing a not CEF switched feature, the log will report:
%FIB-5-NOPUNTINTF: CEF resuming switching packets to <interface>
When adding a feature which cannot be CEF switched, the log will report:
%FIB-4-PUNTINTF: CEF punting packets switched to <interface> to next slower path
Packets are not dCEF switched but fast switched instead
Possible problem: Neither dCEF nor CEF support the feature conf
igured.
351
Solution Step 1 While dCEF supports a large range of
features, some features are not supported. In this case the packets are forwarded to the next slower level of switching. If dCEF cannot process the packet, the router tries to use CEF; if the packet cannot be processed with CEF switched it will be fast switched.
Check the following document to know exactly what is supported:
IOS features supported by CEF and dCEF matrix
352
Solution(cont.) Step 2 Even if neither dCEF nor CEF ar
e enabled to forward the packet to a destination, the CEF table is built anyway but the adjacency is called a “punt”. See table 4 for further troubleshooting steps.
Packets are not dCEF switched but CEF switched instead.
Possible problem: dCEF does not support the feature configure
d.
354
Solution Step 1 While dCEF supports a large range of featur
es, However some features or hardware are not supported. In this case the packets are forwarded to the next higher level of switching, which is centralized. In this case the feature configured is supported by CEF.
Check the following documents to know exactly what is supported:
- IOS features supported by CEF and dCEF matrix - IOS hardware support of CEF and dCEF matrix
355
Solution(cont.) Step 2 Make certain dCEF is configured on the rout
ing using the following commands: show cef interface x/y, show ip cef <prefix>, show adjacency <interface> detail Ultimately issue show cef not-switch and look for
packets that are not dCEF switched. Use the show interface <interface> stats comma
nd to figure out if the packets are dCEF switched.
356
ExampleRouter# show interface FastEthernet0/0/0 statsFastEthernet0/0/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 3736 383776 280261 387230212 Route cache 3800 390105 380276 487270282 Distributed cache 0 0 0 0 Total 7536 773881 660537 874500494In this case none of the packets going through the interface are
dCEF switched. All the counters for “Distributed cache” row are still zero.
Load-sharing is not working
Possible problem: The traffic includes only a small
number of flows.
358
Solution Step1 Check the load-sharing mode. Load-sharing
during CEF/dCEF switching has two modes: Per-destination (original, tunnel, universal) Per-packet. If you do not want to enable per-packet load shari
ng, enable the per-destination tunnel algorithm that is designed to give a good traffic load-sharing when the number of flows is small. Issue the global following command:
ip cef load-sharing algorithm tunnel
Load-sharing is not working
Possible problem: The network topology is subject to
traffic polarization
360
Solution Step1 Per destination load sharing is based upon a hash function
to perform the balancing. This hash function is subject to polarization, canceling the benefits of load sharing – some routers, under specific traffic distribution patterns, assign all the sessions to the same link regardless of other available paths. The next hop will perform the same hash algorithm and will assign all the traffic to a single link.
Check the load-sharing mode. Load-sharing during CEF/dCEF switching has two modes:
Per-destination (old algorithm, new algorithm, tunnel) Per-packet. Issue the following global command on all the routers in the net
work: ip cef load-sharing algorithm universal This modifies the hash algorithm on each router to correct the eff
ects of polarization. You can also enable per packet load sharing.
Load-sharing is not working
Possible problem: Only one next-hop is accessible to go to
a destination.
362
SolutionStep1 Make certain there are multiple valid next-hops t
o the destination. If the data streams are destined to different hosts, then we should have load sharing. If not, check to see if all the possible parallel paths are up and their next-hop are reachable. You can issue the following commands:
show ip route <prefix> and look for more than 1 next-hop.
show ip cef <prefix> and look for more that 1 next-hop.show ip cef <prefix> internalshow ip cef exact-route <source address> <destination
address>show adjacency detail on those next hop interfaces.
The route processor runs out of memory when dCEF is enabled
Possible problem: CEF messages are queued in
the route processor.
364
Solution Step1 Check the memory available on the route proces
sor. dCEF is downloading CEF tables to all of the linecards of the router if distributed CEF is turned on. During this process the route processor may queue a lot of CEF messages, and run out of memory.
The following commands can be issued on the route processor to track the problem:
show ip cef summary show memory summary show process memory The depth of the queue of IPC information elements (x
drs) packed into IPC messages sent from the route processor to the line card is reported in the output of the following command:
365
Examplerouter#show cef linecard detailCEF linecard slot number 0, status up Sequence number 46, Maximum sequence number expected 118, Seq Ep
och 1 Send failed 0, Out Of Sequence 0, drops 0 Linecard CEF reset 0, reloaded 1 1208 elements packed in 944 messages(50328 bytes) sent 0 elements cleared linecard disabled - failed a reload 0/0/0 xdr elements in LowQ/MediumQ/HighQ 24/9/25 peak elements on LowQ/MediumQ/HighQ Input packets 0, bytes 0 Output packets 0, bytes 0, drops 0 CEF Table statistics: Table name Version Prefix-xdr Status Default-table 106 117 Active, sync
366
Solution(cont.) Step2 Tuning the windowing mechanism u
sed for the route processor/linecard communication will help fixing the problem. Increasing this may reduce the queuing, and thus memory used on the route processor.
With the following command you will change the default value (100 Kbytes) of the window:
ip cef linecard ipc memory <Kbytes>
CEF disables itself on a linecard due to lack of memory
Possible problem: The linecard has a memory short
age.
368
Solution Step1 Check the memory available on the linecards. d
CEF downloads CEF tables to all of the linecards of the router if distributed CEF is turned on, even if you aren't CEF switching on that interface. Because of this, all linecards must have the memory necessary to carry all the routes.
The following commands can be issued on each linecards to track the problem:
show ip cef summary show memory summary show process memory show cef linecard On a 7500, enter the VIP card with if-con <x> console.
CEF Case Study
370
Case Study(topology)
FDDIDual Ring
IGRP-555 BGP-AS-333
BGP AS-111Local Preference 2000
R1
R3 R2
12.0.0.1
12.0.0.2 12.0.0.3
11.0.0.2
11.0.0.1
13.0.0.3
13.0.0.1
Nets99, 192, 195,197
BGP AS-222Local Preference 100
371
R1’s Output
R1#show ip routeCodes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, * - candidate default U - per-user static route Gateway of last resort is not set B 2.0.0.0/8 [200/0] via 13.0.0.1, 18:17:13B 197.12.11.0/24 [200/0] via 11.0.0.1, 18:17:50B 99.0.0.0/8 [200/0] via 11.0.0.1, 18:17:50B 197.12.12.0/24 [200/0] via 11.0.0.1, 18:17:51B 195.31.54.0/24 [200/0] via 11.0.0.1, 18:17:51I 11.0.0.0/8 [100/1110] via 12.0.0.2, 00:00:49, Fddi1/1/0B 195.27.191.0/24 [200/0] via 11.0.0.1, 18:17:51B 197.28.99.0/24 [200/0] via 11.0.0.1, 18:17:51C 12.0.0.0/8 is directly connected, Fddi1/1/0B 192.12.1.0/24 [200/0] via 11.0.0.1, 18:17:51B 192.33.1.0/24 [200/0] via 11.0.0.1, 18:17:51B 195.181.32.0/24 [200/0] via 11.0.0.1, 18:17:51I 13.0.0.0/8 [100/252] via 12.0.0.3, 00:00:00, Fddi1/1/0B 197.21.251.0/24 [200/0] via 11.0.0.1, 18:17:51B 197.111.82.0/24 [200/0] via 11.0.0.1, 18:17:51B 192.18.2.0/24 [200/0] via 11.0.0.1, 18:17:51
372
R1’s Output (cont.)
R1#show ip cefPrefix Next Hop Interface0.0.0.0/32 receive2.0.0.0/8 12.0.0.3 Fddi1/1/011.0.0.0/8 12.0.0.2 Fddi1/1/012.0.0.0/8 attached Fddi1/1/012.0.0.0/32 receive12.0.0.1/32 receive12.0.0.2/32 12.0.0.2 Fddi1/1/012.0.0.3/32 12.0.0.3 Fddi1/1/012.255.255.255/32 receive13.0.0.0/8 12.0.0.3 Fddi1/1/099.0.0.0/8 12.0.0.2 Fddi1/1/0192.12.1.0/24 12.0.0.2 Fddi1/1/0192.18.2.0/24 12.0.0.2 Fddi1/1/0192.33.1.0/24 12.0.0.2 Fddi1/1/0195.27.191.0/24 12.0.0.2 Fddi1/1/0195.31.54.0/24 12.0.0.2 Fddi1/1/0195.181.32.0/24 12.0.0.2 Fddi1/1/0197.12.11.0/24 12.0.0.2 Fddi1/1/0197.12.12.0/24 12.0.0.2 Fddi1/1/0197.21.251.0/24 12.0.0.2 Fddi1/1/0197.28.99.0/24 12.0.0.2 Fddi1/1/0197.111.82.0/24 12.0.0.2 Fddi1/1/0224.0.0.0/4 receive255.255.255.255/32 receive
373
R1’s Output (cont.)
R1#show adj detailProtocol Interface AddressIP Fddi1/1/0 12.0.0.2(31) 0 packets, 0 bytes 5000000C0B 1C8000603E617C28AAAA030000000800 ARP 01:33:09 IP Fddi1/1/0 12.0.0.3(9) 0 packets, 0 bytes 5000603E28 B22800603E617C28AAAA030000000800 ARP 01:32:43
R1#show ip cef 195.181.32.0 d195.181.32.0/24, version 17, cached adjacency 12.0.0.20 packets, 0 bytes via 11.0.0.1, 0 dependencies, recursive next hop 12.0.0.2, Fddi1/1/0 via 11.0.0.0/8 valid cached adjacency
R1#show ip cef 11.0.0.1 detail11.0.0.0/8, version 9, cached adjacency 12.0.0.20 packets, 0 bytes via 12.0.0.2, Fddi1/1/0, 12 dependencies next hop 12.0.0.2, Fddi1/1/0 valid cached adjacency
CEF Case Study
TroubleshootingTroubleshooting
375
Useful Information
Verify CEF/DCEF entryVerify adjacency entryCheck interface switching modeCheck all features that CEF/DCEF supports
376
Useful Information (cont.)
Use “show” and “debug” commands
show cef dropshow cef not-cef-switchshow cef [linecard] [interface]
Verify with other switching modes
377
Show Outputscenter#show cef dropCEF Drop StatisticsSlot Encap_fail Unresolved Unsupported No_route No_adj ChksumErrRP 4 0 0 0 0 02 0 0 0 0 0 03 0 0 0 2452 0 04 0 0 0 0 0 05 0 0 0 0 0 0
center#show cef not-cef-switchedCEF Packets passed on to next switching layerSlot No_adj No_encap Unsupp'ted Redirect Receive Bad_ttl Options AccessRP 0 0 41462 0 7922 0 0 02 0 0 0 0 0 0 0 03 0 0 357 0 1161 0 0 04 0 0 0 0 23 3250 0 05 0 0 216504 0 0 0 0 0
378
center#show cef int f9/0/0Fddi9/0/0 is up (if_number 26) Internet address is 190.1.0.2/16 ICMP redirects are always sent Per packet loadbalancing is disabled Inbound access list is 1 Outbound access list is 3 Hardware idb is Fddi9/0/0 Fast switching type 2, interface type 8 IP Distributed CEF switching enabled Fast flags 0x5. ifindex 25(25) Slot 9 Slot unit 0 VC -1 Hardware transmit queue ptr 0x48001B80 (0x48001B80) Transmit limit accumulator 0x48001B82 (0x48001B82) IP MTU 4352
center#show cef linecard 9CEF linecard slot number 9, status up, sync Linecard CEF version number 246 Sequence number 1364, Maximum sequence number expected 1388 Send failed 0, Out Of Sequence 0 Linecard CEF reset 1, reloaded 1 190/10761/178 prefix/adjacency/interface elements queued 11022 elements packed in 1365 messages(235456 bytes) sent 0/0 xdr elements in LowQ/HighQ Input packets 7050739, bytes 2838181523 Output packets 7042682, bytes 2799387214, drops 0
Show Outputs (cont.)
379
Show Outputs (cont.)
VIP-Slot5#show ip cef sum IP Distributed CEF with switching (Table Version 223) 223 routes, 0 unresolved routes (0 old, 0 new) 223 leaves, 168 nodes, 199464 bytes, 724 inserts, 501 invalidations 0 load sharing elements, 0 bytes, 0 references 4 CEF resets, 0 revisions of existing leaves refcounts: 36926 leaf, 36706 node VIP-Slot5#show ip cacheIP radix m-way trie cache 0 entries, 0 tables 0 unique encapsulations, 0 total bytes used by cache 7 adds, 7 invalidates, 46024768 switched, 36807 misses misses: 0 frags, 0 runts, 0 chksm, 0 access 0 badiplen Prefix/Length Age Interface MAC Header IP Flow Cache, 0 active, 512 alloced, 0 opened 0 closed: 0 tcp-fin, 0 idle-time, 0 prefix-inval 0 create failures, 0 switched
VIP-Slot5#show VIP memdMEMD statistics : 0 local-switched misses: 0 tqls, 0 bhs, 0 txaccs, 0 deferred
380
Debug Commands
• debug ip cef [drop] | [receive] | [events] | [prefix-ipc] | [table] | [ipc] | [interface-ipc]
• debug ip [packet] [error]
• debug adjacency
381
center#debug ip cef table
Debug Outputs
Nov 13 12:31:21.075: CEF-IP: Receive address 99.0.0.0/32 already existsNov 13 12:31:21.075: CEF-IP: Receive address 99.255.255.255/32 already existsNov 13 12:31:21.075: CEF-Table: Event up, 2.0.0.0/8Nov 13 12:31:21.075: CEF-Table: Event up, 197.12.11.0/24Nov 13 12:31:21.087: CEF-Table: Event up, 197.21.251.0/24Nov 13 12:31:21.087: CEF-Table: Event up, 192.18.2.0/24Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 1 Seq 0Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 1 Seq 0Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 4 Seq 0 Nov 13 12:31:36.067: CEF-Table: attempting to resolve 197.12.12.0/24Nov 13 12:31:36.067: CEF-IP: resolved 197.12.12.0/24 via 99.0.0.1 to 99.0.0.1 Loopback1Nov 13 12:31:36.067: CEF-Table: attempting to resolve 197.12.11.0/24Nov 13 12:31:36.067: CEF-IP: resolved 197.12.11.0/24 via 99.0.0.1 to 99.0.0.1 Loopback1Nov 13 12:31:57.695: CEF-IP: Checking dependencies of 12.0.0.0/8Nov 13 12:31:57.695: CEF-Table: Adjacency-prefix 12.0.0.1/32 add request -- succeeded Nov 13 12:33:03.079: CEF-Table: Flushing entry for 195.27.191.0/24Nov 13 12:33:03.079: CEF-Table: Flushing entry for 195.31.54.0/24Nov 13 12:33:03.079: CEF-Table: Flushing entry for 195.181.32.0/24Nov 13 12:33:03.079: CEF-Table: Flushing entry for 197.12.11.0/24
382
Debug Output (cont.)
VIP-Slot9#debug ip packIP-DCEF: Try to DCEF switch 196.2.3.1 from Fddi0/0IP-DCEF: DCEF switched 196.2.3.1 to FastEthernet8/1/0IP-DCEF: Try to DCEF switch 196.2.3.1 from Fddi0/0IP-DCEF: DCEF switched 196.2.3.1 to FastEthernet8/1/0
VIP-Slot9#debug ip cef dropIP-CEF: No route found for 34.1.1.1IP-CEF: No route found for 34.1.1.1
border2#debug adjacency ADJ: add 16.2.0.2 (Ethernet0/0) via ARP for 01:21:15ADJ: add 16.2.0.1 (Ethernet0/0) via ARP for 01:20:37ADJ: add 62.0.0.1 (Ethernet0/1) via ARP for 01:20:32ADJ: add 66.0.0.1 (Ethernet0/2) via ARP for 01:22:13ADJ: add 161.2.1.50 (Fddi1/0) via ARP for 03:45:59ADJ: add 180.2.0.2 (FastEthernet8/0/0) via ARP for 01:22:13ADJ: add 200.2.0.2 (Fddi10/0/0) via ARP for 01:23:07ADJ: add 16.2.146.97 (Ethernet0/0) via ARP for 01:23:07ADJ: add 0.0.0.0 (ATM 5/0/0.100) via ATM-PVC for 00:00:00ADJ: add 0.0.0.0 (ATM 5/0/0.103) via ATM-PVC for 00:00:00ADJ: add 0.0.0.0 (Serial4/0/0) via FIB for 00:02:59ADJ: add 0.0.0.0 (Serial4/0/1) via FIB for 00:03:00
Troubleshooting IP Multicast
0981_03F8_c3NW98_US_112
384
Troubleshooting ToolsTroubleshooting Tools Basic Troubleshooting Advanced
Troubleshooting Case studies
Agenda
385
“show ip igmp group” command “show ip igmp interface” command “show ip pim neighbor” command “show ip pim interface” command “show ip rpf ” command “show ip mroute” commands
Troubleshooter’s “Hand” Tools
386
Troubleshooter’s “Hand” Tools
“show ip pim rp” command “show ip pim rp map” comman
d
Special Sparse Mode Tools
387
show ip igmp group
Shows: Currently joined multicast groups.
Troubleshooting usage: Verify that a receiver has actually joined
the target group If not, use “show ip igmp interface” to check for
proper igmp version, querier, timers, etc. Use “debug ip igmp” to verify that proper igmp
host-router exchange is happening Watch for IGMP v1-v2 interoperability problems
388
show ip igmp group
R4#show ip igmp groupsIGMP Connected Group MembershipGroup Address Interface Uptime Expires Last Reporter224.1.1.1 Ethernet1 3d16h 00:01:59 172.16.7.2224.0.1.40 Ethernet0 4d15h never 172.16.6.2
389
show ip igmp interface
Shows: Key IGMP timers, status, etc.
Troubleshooting usage: Verify that correct IGMP version is running Verify that timers are set properly Verify that correct router is IGMP Querier
If not, use “debug ip igmp” to determine what’s wrong
390
show ip igmp interface
R4#show ip igmp interfaceEthernet1 is up, line protocol is up Internet address is 172.16.7.1, subnet mask is 255.255.255.0 IGMP is enabled on interface Current IGMP version is 2 CGMP is disabled on interface IGMP query interval is 60 seconds IGMP querier timeout is 120 seconds IGMP max query response time is 10 seconds Inbound IGMP access group is not set Multicast routing is enabled on interface Multicast TTL threshold is 0 Multicast designated router (DR) is 172.16.7.1 (this system) IGMP querying router is 172.16.7.1 (this system) No multicast groups joined
391
show ip pim neighbor
Shows: PIM Neighbor Adjacencies
Troubleshooting usage: Verify that all neighbors are up and using
proper mode If not, check router configs and/or interface status Use “debug ip pim” to observe PIM Query msg
exchange
392
show ip pim neighbor
R6#show ip pim neighborPIM Neighbor TableNeighbor Address Interface Uptime Expires Mode172.16.10.2 Serial0 4d15h 00:01:19 Dense172.16.11.2 Serial1 4d15h 00:01:00 Dense172.16.9.1 Ethernet0 4d15h 00:01:00 Dense
393
show ip pim interface
Shows: PIM Interface information.
Mode, Neighbor Count, DR
Troubleshooting usage: Verify correct PIM mode is configured
on interface(s) If not, check router configs
Verify Designated Router is correct If not, check router configs Especially critical for Sparse Mode!
394
show ip pim interface
R6#show ip pim interfaceAddress Interface Mode Nbr Query DR Count Intvl172.16.10.1 Serial0 Dense 1 30 0.0.0.0172.16.11.1 Serial1 Dense 1 30 0.0.0.0172.16.9.2 Ethernet0 Dense 1 30 172.16.9.2
395
show ip rpf
Shows: RPF interface information for source
Troubleshooting usage: Verify that RPF information is correct
If not, check unicast routing data for correctness Ping or Trace “source” to verify unicast route is
working. (Fix any unicast routing problems first!) May need to use DVMRP routes or Static Mroutes
to fix unicast-multicast incongruency
396
show ip rpf
R4#show ip rpf 172.16.8.1RPF information for Source1 (172.16.8.1) RPF interface: Ethernet0 RPF neighbor: R3 (172.16.6.1) RPF route/mask: 172.16.8.0/255.255.255.0 RPF type: unicast
R4#sh ip rpf 172.16.12.2RPF information for Source2 (172.16.12.2) RPF interface: Tunnel0 RPF neighbor: R6 (172.16.11.1) RPF route/mask: 172.16.12.0/255.255.255.0 RPF type: DVMRP
397
show ip mroute commands
“show ip mroute sum” “show ip mroute count” “show ip mroute active” “show ip mroute”
398
show ip mroute summary
Shows: Multicast state at a glance
Active groups Active senders in the group. (If SPT joined)
Troubleshooting usage: Verify multicast group(s) are active.
If not, check for group state at RP. (Sparse mode) Work your way from a known source to a receiver
or the RP to find where things stop Verify senders are active. (If SPT joined)
If not, check state in 1st-hop router Verify sender is really sending
399
show ip mroute summary
dallas-gw>show ip mroute summaryIP Multicast Routing TableFlags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPTTimers: Uptime/ExpiresInterface state: Interface, Next-Hop, State/Mode
(*, 224.0.255.255), 6d23h/00:02:59, RP 171.69.10.13, flags: SJC (171.68.37.121/32, 224.0.255.255), 6d23h/00:02:55, flags: CT (171.69.58.88/32, 224.0.255.255), 6d23h/00:02:58, flags: CT (171.69.60.189/32, 224.0.255.255), 6d23h/00:02:55, flags: CT (171.69.128.115/32, 224.0.255.255), 3d01h/00:02:58, flags: CJT (171.69.199.49/32, 224.0.255.255), 6d23h/00:02:57, flags: CT (171.70.247.82/32, 224.0.255.255), 01:55:33/00:02:58, flags: CJT...
400
show ip mroute count
Shows: Multicast traffic flow rates, drops, etc.
Group traffic summary Sender rates, packet counts, drops, etc.
Troubleshooting usage: Verify multicast traffic is being received
If not, work your way from source to receiver tofind where things stop
Verify multicast traffic is being forwarded If not, why? “oif null”, “rpf-failure”
401
show ip mroute countdallas-gw>show ip mroute 224.0.255.255 countIP Multicast Statistics - Group count: 7, Average sources per group: 3.28Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per secondOther counts: Total received/RPF failed/Other drops(OIF-null, rate-limit etc)
Group: 224.0.255.255, Source count: 6, Group pkt count: 538169 RP-tree: Forwarding: 0/0/0/0, Other: 0/0/0 Source: 171.68.37.121/32, Forwarding: 120484/0/94/0, Other: 120535/0/51 Source: 171.69.58.88/32, Forwarding: 120281/0/84/0, Other: 120283/2/0 Source: 171.69.60.189/32, Forwarding: 120445/0/95/0, Other: 120447/2/0 Source: 171.69.128.115/32, Forwarding: 53018/0/93/0, Other: 53018/0/0 Source: 171.69.199.49/32, Forwarding: 120414/1/92/0, Other: 120415/1/0 Source: 171.70.247.82/32, Forwarding: 3527/1/78/0, Other: 3527/0/0
402
show ip mroute active
Shows: Sources with traffic rates above threshold
Aggregate RP Tree and (S, G) rates shown Rates in Kbps (1 sec, 1 min, 5 min avgs.)
Troubleshooting usage: Determine which sources/groups are active Determine the traffic rate of each source
Note: Must have switched to Shortest-Path tree Verify “target” group multicast traffic is
being received If not, work your way from source to receiver
403
show ip mroute activebarrnet-gw>show ip mroute activeActive IP Multicast Sources - sending >= 4 kbps
Group: 224.2.156.43, *cisco: Bloomington IPTV Beacon Source: 172.17.67.43 (bloom-iptv.cisco.com) Rate: 6 pps/63 kbps(1sec), 65 kbps(last 19 secs), 37 kbps(life avg)
Group: 224.2.154.118, Radio Bandit Source: 192.36.125.68 (falcon.pilsnet.sunet.se) Rate: 11 pps/30 kbps(1sec), 30 kbps(last 33 secs), 23 kbps(life avg) Group: 224.2.246.13, UO Presents KWAX Classical Radio Source: 128.223.83.204 (d83-204.uoregon.edu) Rate: 24 pps/69 kbps(1sec), 72 kbps(last 2 secs), 70 kbps(life avg) Group: 224.2.180.115, ANL TelePresence Microscopy Site Source: 146.139.72.5 (aem005.amc.anl.gov) Rate: 1 pps/5 kbps(1sec), 9 kbps(last 52 secs), 12 kbps(life avg)...
404
show ip mroute
Shows: Detailed multicast state in the router
Troubleshooting usage: Verify Incoming Interface is correct
If not, check unicast routing table (May need touse DVMRP routes or Static mroutes)
Verify Outgoing Interface(s) are correct If interface incorrectly “Pruned”, check state in
downstream router? May need to “debug ip pim <group>” to
determine problem
405
show ip mroutebarrnet-gw>show ip mrouteIP Multicast Routing TableFlags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPTTimers: Uptime/ExpiresInterface state: Interface, Next-Hop, State/Mode(*, 224.2.130.100), 00:18:53/00:02:59, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Fddi1/0, Forward/Dense, 00:09:20/00:02:38 Hssi3/0, Forward/Dense, 00:18:53/00:00:00(208.197.169.209/32, 224.2.130.100), 00:18:53/00:02:27, flags: T Incoming interface: Hssi3/0, RPF nbr 131.119.26.9 Outgoing interface list: Fddi1/0, Forward/Dense, 00:16:16/00:02:38(*, 239.100.111.224), 05:35:08/00:02:58, RP 171.69.10.13, flags: DP Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Null
406
show ip pim rp map
Shows: RP assignments by multicast group range
Troubleshooting usage: Verify that configured (static or Auto-RP)
RP’s are correct If not, check local router config
and/or network Auto-RP configuration
407
show ip pim rp map
dallas-gw>show ip pim rp mapPIM Group-to-RP Mappings
Group(s) 224.0.0.0/4, uptime: 6d21h, expires: 00:02:56 RP 171.69.10.13 (sj-eng-mbone.cisco.com) Info source: 192.31.7.37 (barrnet-gw.cisco.com)
408
show ip pim rp
Shows: RP’s by active group
Troubleshooting usage: Verify that the RP for the target multicast
group is correct If not, check RP mapping, local router RP conf
ig and/or network Auto-RP
409
show ip pim rp
dallas-gw>sh ip pim rpGroup: 224.2.127.253, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.1.127.255, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.2.127.254, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.0.255.255, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.2.0.1, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34
410
Troubleshooter’s “Power” Tools
“mtrace” and “mstat” commands
“mrinfo” command “show ip mpacket” command
411
mtrace and mstat commands
Based on Unix “mtrace” command Split into two separate commands Both use the same mechanism
draft-ietf-idmr-traceroute-ipm-xx.txt
412
mtrace
Shows: Multicast path from source to receiver
Similar to unicast “trace” command Trace path between any two points in network TTL Thresholds and Delay shown at each node
Troubleshooting usage: Find where multicast traffic flow stops
Focus on router where flow stops Verify path multicast traffic is following
Identify sub-optimal paths
413
mstat
Shows: Multicast path in pseudo graphic format
Trace path between any two points in network Drops/Duplicates shown at each node TTLs and Delay shown at each node
Troubleshooting usage: Locate congestion point in the flow
Focus on router with high drop/duplicate count Duplicates indicated as “negative” drops
414
Multicast Dist. Tree
Mtrace Packet
src dest
mtrace re
quest
Addsmtrace
data
Addsmtrace
data
Addsmtrace
data
Addsmtrace
data
Addsmtrace
data
mtrace response
Unix Workstationor
Cisco Router
Last-hopRouter
First-hopRouter
Mtrace Packet Flow
mtrace/mstat—How it Works
Note: Mtrace packets use specialIGMP packets with IGMP Type codes of 0x1E and 0x1F.
415
mtrace/mstat—How it Works
Each hop adds data to packet Query arrival time Incoming Interface Outgoing Interface Prev. Hop Router address Input packet count Output packet count Total packets for this Source/Group Routing Protocol TTL Threshold Fowarding/Error Code
416
mtrace
dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43Type escape sequence to abort.Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43From source (?) to destination (bwilliam-ss5.cisco.com)Querying full reverse path... 0 bwilliam-ss5 (171.68.37.121)-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms-5 bloom-iptv-svr (172.17.67.43)dallas-gw>
417
mstatdallas-gw>mstat bloom-iptv-svr bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 547 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 547 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop -409 ms -11/168 = --% 16 pps 0/67 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop 379 ms -9/170 = --% 17 pps -3/67 = --% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 28 ms -3/195 = --% 19 pps 0/70 = 0% 7 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 196 19 pps 70 7 pps171.68.37.121 171.68.86.194 Receiver Query Source
350981_03F8_c3NW98_US_112
418
mstatdallas-gw>mstat bloom-iptv-svr bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 399 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 399 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop 119 ms 77/694 = 11% 69 pps 0/65 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop -150 ms 395/609 = 65% 60 pps 44/65 = 68% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 30 ms -8/39 = --% 3 pps -1/21 = --% 2 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 39 3 pps 22 2 pps171.68.37.121 171.68.86.194 Receiver Query Source
360981_03F8_c3NW98_US_112
419
mrinfo
Shows: Multicast neighbor router information
Indicates router’s capabilities and code version Multicast interface information
TTL-Thresholds, Metric, Protocol, Status
Troubleshooting usage: Verify multicast neighbors.
Confirm bi-directional neighbor adjacency exists Verify Tunnels are up in both directions
420
mrinfodallas-gw>mrinfo paloalto-mbone1.bbnplanet.netTranslating " paloalto-mbone1.bbnplanet.net "...domain server (171.68.10.70) [OK]131.119.0.197 (paloalto-mbone1.bbnplanet.net) [version cisco 11.2] [flags: PMSA]: 131.119.0.197 -> 131.119.0.201 (paloalto-cr1.bbnplanet.net) [1/0/pim] 131.119.244.244 -> 0.0.0.0 [1/32/pim/querier] 131.119.0.197 -> 204.162.119.8 (hydra.precept.com) [1/32/tunnel/querier] 192.42.110.249 -> 192.9.9.71 (mbone.Sun.COM) [1/32/tunnel] 192.42.110.249 -> 204.123.13.69 (chocolate.research.digital.com) [1/32/tunnel] 192.42.110.249 -> 36.253.0.11 (alpo.Stanford.EDU) [1/32/tunnel] 131.119.0.197 -> 0.0.0.0 [1/64/tunnel/pim/querier/leaf] 131.119.0.197 -> 0.0.0.0 [1/32/tunnel/pim/querier/leaf] 192.42.110.249 -> 204.94.211.39 (sgi-too.SGI.COM) [4/64/tunnel/querier] 192.42.110.249 -> 192.216.174.1 [1/32/tunnel/querier/down/leaf] 192.42.110.249 -> 198.94.216.2 [1/32/tunnel/querier/down/leaf] 192.42.110.249 -> 204.161.60.33 (berkeley.faslab.com) [1/32/tunnel/querier] 131.119.0.197 -> 204.154.181.12 [1/32/tunnel/querier/down/leaf]...
421
show ip mpacket
Used to view multicast packet headers Command syntax
show ip mpacket <source> <group> [detail] You can view:
{source, group} traffic pairs IP ident and ttl Inter-packet delay
Configure multicast header capture first “ip multicast cache-headers” config cmd Captures multicast headers in 1024 entry ring buffer
422
show ip mpacketdino-cisco-fr#show ip mpacket 224.2.231.173IP Multicast Header Cache - entry count: 29, next index: 30Key: id/ttl timestamp (name) source group
D782/117 206416.908 (all-purpose-gunk.near.net) 199.94.220.184 224.2.231.1737302/113 206417.172 (speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.1736CB2/114 206417.412 (wayback.uoregon.edu) 128.223.156.117 224.2.231.173D786/117 206417.868 (all-purpose-gunk.near.net) 199.94.220.184 224.2.231.173E2E9/123 206418.488 (dino-ss20.cisco.com) 171.69.58.81 224.2.231.1731CA7/127 206418.544 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CAA/127 206418.584 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CAC/127 206418.624 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CAF/127 206418.664 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CB0/127 206418.704 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CB2/127 206418.744 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1732BBB/114 206418.840 (crevenia.parc.xerox.com) 13.2.116.11 224.2.231.1733D1D/123 206419.380 (dalvarez-ss20.cisco.com) 171.69.60.189 224.2.231.1732BC0/114 206419.672 (crevenia.parc.xerox.com) 13.2.116.11 224.2.231.1737303/113 206419.888 (speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.1737304/113 206420.140 (speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.1732C7E/123 206420.360 (lwei-ss20.cisco.com) 171.69.58.88 224.2.231.173
423
Troubleshooting Tools Basic TroubleshootingBasic Troubleshooting Advanced
Troubleshooting Case studies
Agenda
424
Basic Troubleshooting
Is each piece working correctly?
Troubleshooting Table
SourceSource NetworkNetwork ReceiversReceivers
StateState
Packet Flow Packet Flow ?? ?? ??
?? ??NANA
425
Check Source Packet Flow
Check interface counters on source Check source TTL > 1
Verify TTL setting in application Confirm on upstream router
show ip traffic Increasing “Invalid hop count”
426
Check Source Packet Flow
Check 1st-Hop router for traffic flow show ip mroute count show ip mroute active show ip mpacket
Don’t forget to turn on: ip multicast cache-headers
debug ip mpacket Use with caution!! “detail” or ACL for granularity
427
Check Network State
Most complex piece Depends of protocol, mode, etc. Check initial state creation Check for pruning and timer
expiration during session
428
Network State
show/debug ip mroute commands watch oilist for null entries
show/debug ip pim commands show/debug ip dvmrp commands show ip rpf mtrace command
429
PIM SM Troubleshooting
show ip pim rp [<group>] indicates RP for the group
show ip pim rp mapping indicates RP for the group
debug ip pim auto-rp
430
Check Network Packet Flow
show ip mroute count show ip mroute active show ip mpacket
Turn on “ip multicast cache-headers” first
debug ip mpacket Be Careful with this one!
mstat
431
Check Receiver State
show ip igmp interface show ip igmp group debug ip igmp IGMPv1 vs. IGMPv2
432
Check Receiver Packet Flow
Check receiver interface stats Is the stack installed and configured
properly? Is the application installed and
configured properly? Watch for duplicates
Performance implication
433
Troubleshooting Tools Basic Troubleshooting Advanced Advanced
TroubleshootingTroubleshooting Case studies
Agenda
434
Advanced Troubleshooting
Troubleshooting Network Troubleshooting Network StateState
Troubleshooting PIM-DVMRP Troubleshooting ATM P2MP
VCs
435
Troubleshooting Network State
Use “show ip mroute <group>” Specify target group to limit output. DM: Trace state from source to receiver. SM: Trace state from receiver to RP then from source
to RP Use “debug ip pim <group>”
Be sure to specify target group to limit debug output! Use with caution!!
Use “debug ip mroute <group>” Be sure to specify target group to limit debug output! Use with caution!!
436
Mroute Flags
“S”—Sparse Mode Appears only on the (*, G) entries
“D”—Dense Mode Appears only on the (*, G) entries
“P”—Pruned Sparse mode: oilist is null Dense mode: all interfaces in oilist = Pruned
437
Mroute Flags (Cont.)
“C”—Connected A rcvr for this group is directly connected
to this router. “L”—Local
The router itself is a member of this group and is receiving group traffic.
“T”—Shortest-path Tree (SPT) flag Set on (S,G) entry when packets are being
successfully received via the SPT
438
Mroute Flags (Sparse Mode Only)
“R”—RP bit Only appears on (S, G) entries (S, G) state is associated with Shared Tree RPF interface points up Shared Tree to RP Used to prune unwanted (S, G) traffic from
the Shared Tree after SPT switchover “F”—Register Flag
Appears on (S, G) entries Set on (*, G) if any (S, G) “F” flags set Router is directly connected to a source Register messages must be sent to RP
439
Mroute Flags (Sparse Mode Only)
“J” —Join SPT Set once/sec when SPT-Threshold exceeded Switch to SPT for next (S, G) packet rcvd’d
(*, G) Entry
“J” —SPT Joined SPT Joined due to SPT-Threshold exceeded. Switch back to Shared Tree if traffic rate falls below SPT-Threshold. (Checked once/min)
(S, G) Entry
440
Network Mroute State Examples
PIM DM PIM SM
Joining Registering SPT-Switchover
441
PIM DM
11 “rtr-a” initially floods (S, G) traffic out all interfaces in “oilist”.
11 Multicast Packets(128.9.160.43, 224.2.127.254)
S0
rtr-a
rtr-b
S1
E1
S0
S3
442
(*, 224.2.127.254), 00:00:10/00:00:00, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Serial1, Forward/Dense, 00:00:10/00:00:00 Serial3, Forward/Dense, 00:00:10/00:00:00
(128.9.160.43/32, 224.2.127.254), 00:00:10/00:02:49, flags: T Incoming interface: Serial0, RPF nbr 198.92.1.129 Outgoing interface list: Serial1, Forward/Dense, 00:00:10/00:00:00 Serial3, Forward/Dense, 00:00:10/00:00:00
Initial “Flooding” Statein “rtr-a”
PIM DMS0
rtr-a
rtr-b
Multicast Packets(128.9.160.43, 224.2.127.254)
S1
E1
S0
S3
443
PIM DM
11 “rtr-a” initially floods (S, G) traffic out all interfaces in “oilist”
Multicast Packets(128.9.160.43, 224.2.127.254)
22 “rtr-b” is a leaf node w/o receivers. Sends Prune for (S,G)
Prune22
33 “rtr-a” Prunes interface for (S,G)
S0
rtr-a
rtr-b
S1
E1
S0
33
XX
S3
444
(*, 224.2.127.254), 00:00:12/00:00:00, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Serial1, Forward/Dense, 00:00:12/00:00:00 Serial3, Forward/Dense, 00:00:12/00:00:00
(128.9.160.43/32, 224.2.127.254), 00:00:12/00:02:48, flags: T Incoming interface: Serial0, RPF nbr 198.92.1.129 Outgoing interface list: Serial1, Forward/Dense, 00:00:12/00:00:00 Serial3, Prune/Dense, 00:00:04/00:02:56
PIM DM
Multicast Packets(128.9.160.43, 224.2.127.254)
S0
rtr-a
rtr-b
S1
E1
S0
S3
State in “rtr-a”after Pruning
445
(*, 224.2.127.254), 00:00:12/00:00:00, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Serial0, Forward/Dense, 00:00:12/00:00:00
(128.9.160.43/32, 224.2.127.254), 00:00:12/00:02:48, flags: PT Incoming interface: Serial0, RPF nbr 198.92.1.129 Outgoing interface list: Null
PIM DM
Multicast Packets(128.9.160.43, 224.2.127.254)
S0
rtr-a
rtr-b
S1
E1
S0
S3
State in “rtr-b”after Pruning
446
• “Rcvr A” wishes to receive group G traffic. Sends IGMP Join for G11
IGMP Join11
• “rtr-b” creates (*,G) state; sends (*,G) PIM Join towards RP22
PIM Join22
• “rtr-a” creates (*,G) state; sends (*,G) PIM Join towards RP33
PIM Join33
Shared Tree44
• Shared tree is built all the way back to the RP44
PIM SM Joining
E0S0 rtr-a
rtr-b
S1
E0E1
Rcvr A
Shared Tree
To RP (10.1.5.1)
10.1.2.2
10.1.2.110.1.4.2
447
PIM SM Joining
rtr-b>sh ip mroute
(*, 224.1.1.1), 00:00:05/00:02:54, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.2.1 Outgoing interface list: Ethernet1, Forward/Sparse, 00:00:05/00:02:52
State in “rtr-b” after Joining (*, 224.1.1.1)
E0S0 rtr-a
rtr-b
S1
E0E1
To RP (10.1.5.1)
Rcvr A
10.1.2.2
10.1.2.110.1.4.2
Shared Tree
448
rtr-a>sh ip mroute
(*, 224.1.1.1), 00:00:05/00:02:54, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.4.1 Outgoing interface list: Ethernet0, Forward/Sparse, 00:00:05/00:02:54
State in “rtr-a” after Joining (*, 224.1.1.1)
PIM SM Joining
E0S0 rtr-a
rtr-b
S1
E0E1
To RP (10.1.5.1)
Rcvr A
10.1.2.2
10.1.2.110.1.4.2
Shared Tree
449
rtr-c>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:00:03/00:02:56, RP 171.68.28.140, flags: S Incoming interface: Null, RPF nbr 0.0.0.0, Outgoing interface list: Serial0, Forward/Sparse, 00:03:14/00:02:59 Serial1, Forward/Sparse, 00:03:14/00:02:59
State in “RP” before Registering(with receivers on Shared Tree)
PIM SM Registering
rtr-a
RP
rtr-crtr-b
Shared Tree
S3
S0 S1
171.68.28.139
450
rtr-a>sh ip mroute 224.1.1.1
No such group.
State in “rtr-a” before Registering(with receivers on Shared Tree)
rtr-a
RP
rtr-crtr-b
Shared Tree
PIM SM Registering
E0
S0
451
Source 171.68.37.121
• “Source” begins sending group G traffic11
11
(171.68.37.121, 224.1.1.1) Mcast Packets
• “rtr-a” encapsulates packets in Registers; unicasts to RP22
Register Msgs22
• “rtr-c” (RP) de-encapsulates packets; forwards down Shared tree33
33 (*, 224.1.1.1)Mcast Traffic
rtr-a
RP
rtr-crtr-b
Shared Tree
PIM SM Registering
452
rtr-a>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:00:03/00:02:56, RP 171.68.28.140, flags: SP Incoming interface: Serial0, RPF nbr 171.68.28.191, Outgoing interface list: Null
(171.68.37.121/32, 224.1.1.1), 00:00:03/00:02:56, flags: FPT Incoming interface: Ethernet0, RPF nbr 0.0.0.0, Registering Outgoing interface list: Null
State in “rtr-a” while Registering
(171.68.37.121, 224.1.1.1) Mcast Packets
Register Msgs
Source 171.68.37.121
rtr-a
RP
Shared Tree
rtr-crtr-b
(*, 224.1.1.1)Mcast Traffic
PIM SM Registering
E0
S0
70
453
rtr-a
• RP sends (S,G) Join toward Source; builds SPT44
Join 44Join rtr-b
• “Source” begins sending group G traffic 11
• “rtr-a” encapsulates packets in Registers; unicasts to RP22
• RP (“rtr-c”) de-encapsulates packets; forwards down Shared tree33
RP
rtr-c
Shared Tree
(*, 224.1.1.1)Mcast Traffic
(171.68.37.121, 224.1.1.1) Mcast Packets
Register Msgs
Source 171.68.37.121
PIM SM Registering
454
• RP begins receiving (S,G) traffic down SPT 55
55
• RP sends “Register-Stop” to “rtr-a”66
Register-Stop66
rtr-a
RP
rtr-crtr-b
Shared Tree
(*, 224.1.1.1)Mcast Traffic
Source 171.68.37.121
(171.68.37.121, 224.1.1.1) Mcast Packets
Register Msgs
PIM SM Registering
455
• “rtr-a” stops encapsulating traffic in Register Messages77
• (S,G) Traffic now flowing down a single path (SPT) to RP88
88
rtr-a
RP
rtr-crtr-b
Shared Tree
(*, 224.1.1.1)Mcast Traffic
Source 171.68.37.121
(171.68.37.121, 224.1.1.1) Mcast Packets
• RP begins receiving (S,G) traffic down SPT55
• RP sends “Register-Stop” to “rtr-a”66
PIM SM Registering
456
rtr-a>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: SP Incoming interface: Serial0, RPF nbr 171.68.28.191, Outgoing interface list: Null
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: FT Incoming interface: Ethernet0, RPF nbr 0.0.0.0 Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32
State in “rtr-a” after Registering
rtr-a
RP
rtr-crtr-b
Shared Tree
(*, 224.1.1.1)Mcast Traffic
Source 171.68.37.121
(171.68.37.121, 224.1.1.1) Mcast Packets
PIM SM Registering
E0
S0
457
rtr-b>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: SP Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Null
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32
State in “rtr-b” after “rtr-a” Registers(with receivers on Shared Tree)
rtr-a
RP
rtr-crtr-b
Shared Tree
(*, 224.1.1.1)Mcast Traffic
Source 171.68.37.121
(171.68.37.121, 224.1.1.1) Mcast Packets
PIM SM Registering
S0 S1
171.68.28.190
458
rtr-c>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:09:21/00:02:38, RP 171.68.28.140, flags: S Incoming interface: Null, RPF nbr 0.0.0.0, Outgoing interface list: Serial0, Forward/Sparse, 00:09:21/00:02:38 Serial1, Forward/Sparse, 00:03:14/00:02:46
(171.68.37.121, 224.1.1.1, 00:01:15/00:02:46, flags: T Incoming interface: Serial3, RPF nbr 171.68.28.139, Outgoing interface list: Serial0, Forward/Sparse, 00:00:49/00:02:11 Serial1, Forward/Sparse, 00:00:49/00:02:11
State in “RP” after “rtr-a” Registers(with receivers on Shared Tree)
rtr-a
RP
rtr-crtr-b
Shared Tree
(*, 224.1.1.1)Mcast Traffic
Source 171.68.37.121
(171.68.37.121, 224.1.1.1) Mcast Packets
PIM SM Registering
S3
S0 S1
171.68.28.139
459
PIM SM SPT-Switchover Review
Once each second Compute new (*, G) traffic rate If threshold exceeded, set “J” flag in (*,
G) For each (Si , G) packet received:
If “J” flag set in (*, G) Clear “J” flag in (*,G) Join SPT for (Si , G) Mark (Si , G) entry with “J” flag
SPT-Switchover Mechanism
460
PIM SM SPT-Switchover
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.5.1, Outgoing interface list: Serial1, Forward/Sparse, 00:01:43/00:02:11
State in “rtr-c” before switch
Rcvr A
E0S0 rtr-a
rtr-b
S1
E0E1
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
461
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.4.1, Outgoing interface list: Ethernet0, Forward/Sparse, 00:01:43/00:02:11
State in “rtr-a” before switch
PIM SM SPT-Switchover
E0S0 rtr-a
rtr-b
S1
E0E1
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
462
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.4.2, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11
State in “rtr-b” before switch
PIM SM SPT-Switchover
Rcvr A
E0S0 rtr-a
rtr-b
S1
E0E1
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
463
PIM SM SPT-Switchover
E0S0 rtr-a
S1
E010.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
Group “G” rate exceeds SPT Threshold at “rtr-b”;11
Group “G” rate > Threshold11
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.4.2, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11
Set J Flag in (*, G) and wait for next (Si,G) packet22
22
J
rtr-bE1
Rcvr A
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
464
PIM SM SPT-Switchover
(Si,G) packet arrives down Shared tree.33
E0S0 rtr-a
S1
E010.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SCJ Incoming interface: Ethernet0, RPF nbr 10.1.4.2, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11
33
Clear J Flag in (*,G) and44 55 Send (Si,G) Join towards Si .
44
rtr-bE1
Rcvr A
Group “G” rate exceeds SPT Threshold at “rtr-b”;11
Set J Flag in (*, G) and wait for next (Si,G) packet.22
SC
(Si,G) Join55
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
820981_03F8_c3NW98_US_112
465
PIM SM SPT-Switchover
Send (Si,G)RP-bit Prune toward RP to prune traffic from RPT66
(Si,G)RP-bit Prune 66
E0S0 rtr-a
rtr-b
S1
E0E1
Rcvr A
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
SPT & RPT diverge; “rtr-a” forwards (Si,G) Join toward Si77
(Si,G) Join77
“rtr-a” forwards (Si,G)RP-bit Prune toward RP88
(Si,G)RP-bit
Prune
88
(Si, G) traffic begins flowing down SPT tree99
(Si,G) Traffic99
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
To RP (10.1.5.1)rtr-c
10.1.4.1
S1S0
466
PIM SM SPT-Switchover
E0S0 rtr-a
rtr-b
S1
E0E1
Rcvr A
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
Send (Si,G)RP-bit Prune toward RP to prune traffic from RPT66
SPT & RPT diverge; “rtr-a” forwards (Si,G) Join toward Si77
“rtr-a” forwards (Si,G)RP-bit Prune toward RP88
(Si, G) traffic begins flowing down SPT tree99
• (Si, G) traffic ceases flowing down Shared tree1010
1010To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
467
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.4.1, Outgoing interface list: Ethernet0, Forward/Sparse, 00:01:43/00:02:11
(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: T Incoming interface: Serial1, RPF nbr 10.1.9.2 Outgoing interface list: Ethernet0, Forward/Sparse, 00:13:25/00:02:30
State in “rtr-a” after switch
Rcvr A
PIM SM SPT-Switchover
E0S0 rtr-a
rtr-b
S1
E0E1
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
468
PIM SM SPT-Switchover
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.2.1, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11
(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: SCJT Incoming interface: Ethernet0, RPF nbr 10.1.2.1 Outgoing interface list: Ethernet1, Forward/Sparse, 00:13:28/00:02:53
State in “rtr-b” after switch
Rcvr A
E0S0 rtr-a
rtr-b
S1
E0E1
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
J Flag indicates(S, G) created by
exceeding theSPT-threshold
SCJT
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
469
PIM SM SPT-Switchover
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.5.1, Outgoing interface list: Serial1, Forward/Sparse, 00:01:43/00:02:11
(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: PR Incoming interface: Serial0, RPF nbr 10.1.5.1 Outgoing interface list: Null
State in “rtr-c” after switch
Rcvr A
E0S0 rtr-a
rtr-b
S1
E0E1
10.1.2.2
10.1.2.110.1.4.2
To Source “Si”
(Si, G) Traffic FlowShared (RPT) Tree
SPT Tree
To RP (10.1.5.1)
10.1.4.1
S1S0
rtr-c
470
Advanced Troubleshooting
Troubleshooting Network State
Troubleshooting PIM-DVMRPTroubleshooting PIM-DVMRP Troubleshooting ATM P2MP
VCs
471
PIM-DVMRP Troubleshooting
pim-dvmrp-gw:
interface tunnel0ip unnumbered ethernet0ip pim dense-modetunnel mode dvmrptunnel source ethernet0tunnel destination 135.1.22.98
interface ethernet0ip addr 135.1.3.102 255.255.255.0ip pim dense-mode
interface ethernet1ip addr 135.1.2.102 255.255.255.0ip pim dense-mode
Example Network
Site
ISP
pim-dvmrp-gw
Tunnel0
Ethernet0
mrouted
Ethernet1
135.1.2.100
472
PIM-DVMRP Troubleshooting
Verifying the DVMRP tunnelVerifying the DVMRP tunnel Verifying DVMRP route exchange
473
Verifying the DVMRP Tunnel
Using the “show interface” Commandpim-dvmrp-gw> show int tunnel 0Tunnel0 is up, line protocol is up Hardware is Tunnel Interface is unnumbered. Using address of Ethernet0 (135.1.3.102) MTU 1500 bytes, BW 9 Kbit, DLY 500000 usec, rely 255/255, load 1/255 Encapsulation TUNNEL, loopback not set, keepalive set (10 sec) Tunnel source 135.1.3.102 (Ethernet0), destination 135.1.22.98 Tunnel protocol/transport IP/IP (DVMRP), key disabled, sequencing disabled Checksumming of packets disabled, fast tunneling enabled Last input 00:00:05, output 00:00:08, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0 (size/max/drops); Total output drops: 0 .
.
.
474
Verifying the DVMRP Tunnel
pim-dvmrp-gw>mrinfo135.1.3.102 [version cisco 11.2] [flags: PMA]: 135.1.3.102 -> 0.0.0.0 [1/0/pim/querier/leaf] 135.1.2.102 -> 135.1.2.2 [1/0/pim/querier] 135.1.2.102 -> 135.1.2.3 [1/0/pim/querier] 135.1.3.102 -> 135.1.22.98 [1/0/tunnel/querier]
pim-dvmrp-gw>mrinfo 135.1.22.98135.1.22.98 [version mrouted 3.8] [flags: GPM]: 172.21.32.98 -> 172.21.32.191 [1/1] 172.21.32.98 -> 172.21.32.1 [1/1] 135.1.22.98 -> 135.1.22.102 [1/1/querier] 135.1.22.98 -> 135.1.3.102 [1/1/tunnel]
Using the “mrinfo” Command
Both Ends SeeEach Other
475
PIM-DVMRP Troubleshooting
Verifying the DVMRP tunnel Verifying DVMRP route exchangeVerifying DVMRP route exchange
476
Verifying DVMRP Route Exchange
Using “show ip dvmrp route”pim-dvmrp-gw# show ip dvmrp routeDVMRP Routing Table - 8 entries
130.1.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
135.1.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
135.1.22.0/24 [0/2] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
171.69.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.27.0/24 [0/3] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.32.0/24 [0/2] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.33.0/24 [0/3] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.120.0/24 [0/3] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
477
Verifying DVMRP Route Exchange
Using “debug ip dvmrp”pim-dvmrp-gw# debug ip dvmrp DVMRP debugging is onpim-dvmrp-gw#Mar 20 11:39:36.335: DVMRP: Aging routes, 0 entries expiredMar 20 11:39:41.271: DVMRP: Received Probe on Tunnel0 from 135.1.22.98Mar 20 11:39:45.335: DVMRP: Building Report for Tunnel0 224.0.0.4Mar 20 11:39:45.335: DVMRP: Send Report on Tunnel0 to 135.1.22.98Mar 20 11:39:45.335: DVMRP: 2 unicast, 8 DVMRP routes advertisedMar 20 11:39:47.335: DVMRP: Aging routes, 0 entries expiredMar 20 11:39:51.371: DVMRP: Received Probe on Tunnel0 from 135.1.22.98Mar 20 11:39:52.379: DVMRP: Received Report on Tunnel0 from 135.1.22.98
478
Verifying DVMRP Route Exchange
pim-dvmrp-gw# debug ip dvmrp detailDVMRP debugging is onMar 20 11:42:45.337: DVMRP: Building Report for Tunnel0 224.0.0.4Mar 20 11:42:45.337: DVMRP: Report 130.1.0.0/16, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 135.1.0.0/16, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 135.1.22.0/24, metric 34, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 171.69.0.0/16, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.27.0/24, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.32.0/24, metric 34, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.33.0/24, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.120.0/24, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 135.1.2.0/24, metric 1Mar 20 11:42:45.337: DVMRP: Report 135.1.3.0/24, metric 1Mar 20 11:42:45.337: DVMRP: Send Report on Tunnel0 to 135.1.22.98Mar 20 11:42:45.337: DVMRP: 2 unicast, 8 DVMRP routes advertised
Checking DVMRP Routes Being Advertised
479
Verifying DVMRP Route Exchange
pim-dvmrp-gw# debug ip dvmrp detailDVMRP debugging is on... : DVMRP: Received Report on Tunnel0 from 135.1.22.98... : DVMRP: Origin 130.1.0.0/16, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 135.1.0.0/16, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 171.69.0.0/16, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 135.1.2.0/24, metric 34, metric-offset 1, infinity... : DVMRP: Origin 135.1.3.0/24, metric 34, metric-offset 1, infinity... : DVMRP: Origin 135.1.22.0/24, metric 1, metric-offset 1, distance 0... : DVMRP: Origin 172.21.27.0/24, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 172.21.32.0/24, metric 1, metric-offset 1, distance 0... : DVMRP: Origin 172.21.33.0/24, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 172.21.120.0/24, metric 2, metric-offset 1, distance 0
Checking DVMRP Routes Being Received
480
Advanced Troubleshooting
Troubleshooting Network State
Troubleshooting PIM-DVMRP Troubleshooting ATM P2MP Troubleshooting ATM P2MP
VCsVCs
481
Multicast over ATM P2MP VCs
ATMATMNBMANBMACloudCloud
One p2mp VC/group performs multicast replication instead of the router
B’cast p2mp VC used when # Groups > max p2mp VC count
Use PIM Sparse mode p2mp VCs map group m
embership Fast Switched!!
A
DD
BB CC
Note: Only p2mp m’cast VCs for Router A shown for clarity.
482
M’cast P2MP VC Troubleshootingrtr-a> show ip pim vc IP Multicast ATM VC Status ATM0/0 VC count is 5, max is 5 Group VCD Interface Leaf Count Rate 224.0.1.40 21 ATM0/0 2 0 pps 224.2.2.2 26 ATM0/0 1 0 pps 224.1.1.1 28 ATM0/0 1 0 pps 224.4.4.4 32 ATM0/0 2 0 pps 224.5.5.5 35 ATM0/0 1 0 pps
483
rtr-a> show atm vc AAL / Peak Avg. Burst Interface VCD VPI VCI Type Encapsulation Kbps Kbps Cells StatusATM0/0 1 0 5 PVC AAL5-SAAL 155000 155000 96 ACT ATM0/0 2 0 16 PVC AAL5-ILMI 155000 155000 96 ACT ATM0/0 3 0 124 MSVC-3 AAL5-SNAP 155000 155000 96 ACT ATM0/0 4 0 125 MSVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 5 0 126 MSVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 6 0 127 MSVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 9 0 130 SVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 10 0 131 SVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 11 0 132 MSVC-3 AAL5-SNAP 155000 155000 96 ACT ATM0/0 12 0 133 MSVC-1 AAL5-SNAP 155000 155000 96 ACT ATM0/0 13 0 134 SVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 14 0 135 MSVC-2 AAL5-SNAP 155000 155000 96 ACT ATM0/0 15 0 136 MSVC-2 AAL5-SNAP 155000 155000 96 ACT
Root P2MP VC with 3 Leaf RoutersRoot P2MP VC with 3 Leaf Routers
P2MP VC for which we are a LeafP2MP VC for which we are a Leaf
M’cast P2MP VC Troubleshooting
484
show ip mroute 224.1.1.1 IP Multicast Routing Table Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT Timers: Uptime/Expires Interface state: Interface, Next-Hop or VCD, State/Mode (*, 224.1.1.1), 00:03:57/00:02:54, RP 130.4.101.1, flags: SJ Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: ATM0/0, VCD 3, Forward/Sparse, 00:03:57/00:02:53
ATM P2MP VC information for GroupATM P2MP VC information for Group
M’cast P2MP VC Troubleshooting
485
rtr-a> show atm vc 3
ATM0/0: VCD: 3, VPI: 0, VCI: 124, etype:0x0, AAL5 - LLC/SNAP, Flags: 0x650PeakRate: 155000, Average Rate: 155000, Burst Cells: 96, VCmode: 0xE000OAM DISABLED, InARP DISABLEDInPkts: 0, OutPkts: 12, InBytes: 0, OutBytes: 496InPRoc: 0, OutPRoc: 0, Broadcasts: 12InFast: 0, OutFast: 0, InAS: 0, OutAS: 0OAM F5 cells sent: 0, OAM cells received: 0Status: ACTIVE, TTL: 2, VC owner: IP Multicast (224.1.1.1)interface = ATM0/0, call locally initiated, call reference = 2vcnum = 11, vpi = 0, vci = 132, state = Active aal5snap vc, multipoint callRetry count: Current = 0, Max = 10timer currently inactive, timer value = 00:00:00Leaf Atm Nsap address: 47.0091810000000002BA08E101.444444444444.02Leaf Atm Nsap address: 47.0091810000000002BA08E101.333333333333.02Leaf Atm Nsap address: 47.0091810000000002BA08E101.222222222222.02
P2MP VC Opened by Group 224.1.1.1 P2MP VC Opened by Group 224.1.1.1
NSAP Addresses of Leaf Nodes NSAP Addresses of Leaf Nodes
M’cast P2MP VC Troubleshooting
486
Troubleshooting Tools Basic Troubleshooting Advanced
Troubleshooting Case studiesCase studies
Agenda
487
Troubleshooting Cheat Sheet Check IGMP membership on PIM DR on Last
Hop LAN Check RP address in (*,G) entry on the DR Check RPF interface to RP in (*,G) entry Repeat above check for (*,G) state on routers
along the shared tree, up to RP. Repeat above check for (S,G) state on routers
along the source tree, up to Source DR.
488
PIM Timers The secret to understanding PIM is to watch
the timers. 3 minutes (3:30) is the “magic” number. Interface expiration timers are updated every
minute so if the expire timer goes below 2:00 the route is not being used.
Entry expiration timers are updated when data is forwarded so if the timer drops below 2:59, the source has stopped sending.
489
Source Tree
In IOS a (*,G) entry is always created whenever a (S,G) entry is created.
The Source-tree may overlap the Shared-tree in which case the (*,G) entry will be non-NULL.
The Source-tree may be independent of the Shared-tree in which case the (*,G) entry will be NULL.
490
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32
•(S,G) forwarding entry
NOTE: These uptimes indicate the receiver has always been present
PIM SM Source Tree
491
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32
•(S,G) forwarding entry
Receivers have stopped joining
PIM SM Source Tree
492
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32
•(S,G) forwarding entry
Data is not flowing
PIM SM Source Tree
493
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32
•(*,G) forwarding entry
All Sources for this group will be forwarded out the olist
PIM SM Shared Tree
494
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32
•(*,G) forwarding entry
This always points to the RP
PIM SM Shared Tree
495
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32
•(*,G) forwarding entry
This is the next-hop to the RP from “sh ip RPF”
PIM SM Shared Tree
496
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32
•(*,G) forwarding entry
The entry has been up for this long. Note the uptime of the olist
PIM SM Shared tree
497
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32
•(*,G) forwarding entry
All receivers for the entry may have left
PIM SM Shared tree
498
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32
•(*,G) forwarding entry
A sparse-mode group must have an RP
PIM SM Shared tree
499
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: RP Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: NULL
•(S,G,R) forwarding entry
Points toward the RP!!!!
PIM Shared Tree
Last-hop router is sending (s,g,r) prunes
Note R-flag
mtrace
• Shows:– Multicast path from source to receiver.
» Similar to unicast “trace” command» Trace path between any two points in network» TTL Thresholds & Delay shown at each node
• Troubleshooting Usage:– Find where multicast traffic flow stops.
» Focus on router where flow stops
– Verify path multicast traffic is following.» Identify sub-optimal paths.
501
Multicast Dist. Tree
Mtrace Packet
src dest
mtrace re
quest
Addsmtrace
data
Addsmtrace
data
Addsmtrace
data
Addsmtrace
data
Addsmtrace
data
mtrace response
Unix Workstationor
Cisco Router
Last-hopRouter
First-hopRouter
Mtrace Packet Flow
mtrace/mstat—How it works
Note: Mtrace packets use specialIGMP packets with IGMP Type codes of 0x1E and 0x1F.
mtrace
dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43Type escape sequence to abort.Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43From source (?) to destination (bwilliam-ss5.cisco.com)Querying full reverse path... 0 bwilliam-ss5 (171.68.37.121)-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms-5 bloom-iptv-svr (172.17.67.43)dallas-gw>
dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43Type escape sequence to abort.Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43From source (?) to destination (bwilliam-ss5.cisco.com)Querying full reverse path... 0 bwilliam-ss5 (171.68.37.121)-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms-5 bloom-iptv-svr (172.17.67.43)dallas-gw>
mstat
• Shows:– Multicast path in pseudo graphic format.
» Trace path between any two points in network» Drops/Duplicates shown at each node» TTLs & Delay shown at each node
• Troubleshooting Usage:– Locate congestion point in the flow.
» Focus on router with high drop/duplicate count» Duplicates indicated as “negative” drops
mstatdallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 547 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 547 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop -409 ms -11/168 = --% 16 pps 0/67 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop 379 ms -9/170 = --% 17 pps -3/67 = --% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 28 ms -3/195 = --% 19 pps 0/70 = 0% 7 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 196 19 pps 70 7 pps171.68.37.121 171.68.86.194 Receiver Query Source
dallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 547 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 547 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop -409 ms -11/168 = --% 16 pps 0/67 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop 379 ms -9/170 = --% 17 pps -3/67 = --% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 28 ms -3/195 = --% 19 pps 0/70 = 0% 7 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 196 19 pps 70 7 pps171.68.37.121 171.68.86.194 Receiver Query Source
mstatdallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 399 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 399 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop 119 ms 77/694 = 11% 69 pps 0/65 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop -150 ms 395/609 = 65% 60 pps 44/65 = 68% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 30 ms -8/39 = --% 3 pps -1/21 = --% 2 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 39 3 pps 22 2 pps171.68.37.121 171.68.86.194 Receiver Query Source
dallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 399 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 399 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop 119 ms 77/694 = 11% 69 pps 0/65 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop -150 ms 395/609 = 65% 60 pps 44/65 = 68% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 30 ms -8/39 = --% 3 pps -1/21 = --% 2 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 39 3 pps 22 2 pps171.68.37.121 171.68.86.194 Receiver Query Source
506
Debugging Auto-RP Operation
Understand the Auto-RP mechanisms This is the fundamental debugging tool for problems with Auto-This is the fundamental debugging tool for problems with Auto-
RP!!!RP!!! Verify Group-to-RP Mapping Caches
First on the Mapping Agents Other routers will learn Group-to-RP mapping info from these routers
If not correct, use debug commands to see what’s wrong Make sure all MA’s have consistent Group-to-RP information
If not, watch for TTL Scoping problems Then on other routers
If info doesn’t match MA, there is a problem distributing the information
Use show and debug commands to find where the break is
507
Debugging Auto-RP Operation Insure Auto-RP group state is correct
Should normally be in Dense mode Watch out for mixed DM and SM conditions
Can occur when Static RP’s are also defined Always ‘deny’ Auto-RP groups on Static RP
configurations Use ‘Accept-RP’ filters on all routers as insurance
Watch out for DM problems in NBMA networks
(See Module 7 for details)
508
Debugging BSR Operation Understand the BSR mechanisms
This is the fundamental debugging tool for problems with This is the fundamental debugging tool for problems with BSR!!!BSR!!!
Verify Group-to-RP Mapping Caches First on the BSR
Other routers will learn Group-to-RP mapping info from this router
If not correct, use debug commands to see what’s wrong
Then on other routers If info doesn’t match BSR, there is a problem distributing
the information Use show and debug commands to find where the break
is
Case Studies
510
THANKS