multicast troubleshooting

IBRKIPM-2264

Multicast Troubleshooting

Follow us on Twitter for real time updates of the event:

@ciscoliveeurope, #CLEUR

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKIPM-2264 2

Housekeeping

We value your feedback- don't forget to complete your online session evaluations after each session & the Overall Conference Evaluation which will be available online from Thursday

Visit the World of Solutions and Meet the Engineer

Visit the Cisco Store to purchase your recommended readings

Please switch off your mobile phones

After the event don’t forget to visit Cisco Live Virtual: www.ciscolivevirtual.com

Follow us on Twitter for real time updates of the event: @ciscoliveeurope, #CLEUR

http://www.ciscolivevirtual.com/

http://www.ciscolivevirtual.com/

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKIPM-2264 3 3

Housekeeping Items

Maintaining a Multicast Network

Quick Review

Troubleshooting Multicast

Agenda

3


Housekeeping

Cell Phones

“Intermediate” Class

-Assumes Multicast Operational Experience

Slide Format

4


Housekeeping

New Case to Solve Case Closed One slide Case

Geek - o - meter Geek-o-meter*

* Mr Multicast himself, Beau Williamson

5

Maintaining a Multicast Network


Maintaining a Multicast Network Challenges

Multicast forwarding state is dynamic

Best Effort Delivery

No Congestion control

-Requires External Monitoring

-Applications may have feedback mechanism (e.g. Tibco, PGM)

“High” on the network food chain

7


Router

Cards

Interfaces

Routing

Packet

Maintaining a Multicast Network Challenges

Multicast Infrastructure Overlay

Supporting Multicast Configs

PIM Neighbors

Multicast Distribution Trees

Knowledge of RP (root of shared tree)

Knowledge of Source (root of source tree)

Quick Review


Quick Review PIM-SM Categories

Any Source Multicast (ASM)

-Original (Classic) PIM-SM

-Supports both Shared and Source Trees

Source Specific Multicast (SSM)

-a.k.a. Single Source Multicast

-Supports only Source Trees

No need for RPs, RP Failover, etc.

Bidirectional PIM (Bidir)

-Supports only Shared Trees


PIM SM

RPF

Source Tree

{S,G}

Quick Review Frequently Used Terms



RP - Rendezvous Point

MDT - Multicast Distribution Tree

FHR - First Hop Router

LHR - Last Hop Router

IR - Intermediate Router

IGMP - Internet Group Management Protocol http://en.wikipedia.org/wiki/Internet_Group_Management_Protocol

http://en.wikipedia.org/wiki/Internet_Group_Management_Protocol



(*,G) = *,G

= Shared Tree

= RP tree

(S,G) = S,G

= Source Tree

= Shortest Path Tree

Q: What is the Root of the (*,G) Tree?

Q: What is the Root of the (S,G) Tree?

A: RP

A: Source



IIF = Incoming Interface

OIL = Outgoing interface List

Q: What does the IIF of the (*,G) Tree point towards?

Q: What does the IIF of the (S,G) Tree point towards

A: RP

A: Source

Interface towards the

root of the tree


Quick Review Sending a Join

Assuming Router A

• Knows the RP

• Has the RP in its RIB via an IGP with B

• Has a PIM neighbor with Router B

IP of the RP? 224.0.0.13?

(All PIM Routers) IP of Router B

(10.4.5.4)

10.4.5.0 .5 .4

PIM Join

A B RP

Q: When Router A sends a PIM (*,G) Join out, what will be the Destination IP?


Message Type Destination

0 = Hello Multicast to ALL-PIM-ROUTERS

1 = Register Unicast to RP

2 = Register-Stop Unicast to source of Register packet

3 = Join/Prune Multicast to ALL-PIM-ROUTERS

4 = Bootstrap Multicast to ALL-PIM-ROUTERS

5 = Assert Multicast to ALL-PIM-ROUTERS

The PIM header common to all PIM messages is:

PIM Ver

PIM Version number is 2.

Some PIM Types are:


http://www.ietf.org/rfc/rfc4601.txt



10.4.5.0 .5 .4

PIM Join

A B RP

DIP 224.0.0.13

Q: When Router A sends a PIM (*,G) Join out, what will be the Destination IP?

IP of the RP? 224.0.0.13?

(All PIM Routers) IP of Router B

(10.4.5.4)



10.4.5.0 .5 .4

A B

RP

C

.1

PIM Join DIP 224.0.0.13

Q: If the destination IP address of the Join is 224.0.0.13 (all PIM Routers), then

how do we keep Router B and Router C from BOTH acting on that join?



The address of upstream neighbor that is the

target of the message

10.4.5.0 .5 .4

A B

RP

C

.1




Quick Review: Sending a Join

Destination IP

224.0.0.13

RP

2.2.2.2

IP address of B

10.4.5.4

10.4.5.0 .5 .4

A B

RP

C

.1



Geek - o - meter


FHR

Quick Review Registering a Source***

RP = 2.2.2.2

E0

Register Stop to the FHR

Mcast data Mcast packets now only out the (S,G)

tree

Mcast data

Mcast Data

Mcast packet is encapsulated into unicast

PIM packet:

Destination IP is the RP.

PIM header type is Register Mcast data Dest IP=RP

RP sends PIM (S,G) Join Back Towards Source

if there is an active shared tree for that G

E0 added to the OIL

for (S,G)

Mcast packets now get sent out

twice

1.With Register

2.Out (S,G)

Mcast data

Mcast data Once the RP sees the packets come in

on the (S,G), it sends a unicast Register

Stop

***When a shared MDT is already

active for that group on the RP

Geek - o - meter

PIM


Designated Router (DR):

A shared-media LAN like Ethernet may have multiple PIM-SM

routers connected to it. A single one of these routers, the

DR, will act on behalf of directly connected hosts with respect

to the PIM-SM protocol. A single DR is elected per interface

(LAN or otherwise) using a simple election process.

What IS a Designated Router?

RE: “Election” of DR ---

Default behavior is for highest IP address on the

LAN to become the DR unless the DR priority

option is configured

Vlan 110 .3

.2

RP

Quick Review: Designated Routers (DRs)



Quick Review: Designated Routers (DRs)

The DR_Priority Option allows a network administrator to give

preference to a particular router in the DR election process by

giving it a numerically larger DR Priority. …. The default priority is 1

What the DR Priority Option?

R4(config-if)# ip pim dr-priority ?

<0-4294967294> DR priority, preference given to larger value


Vlan 110 .3

.2

RP



Troubleshooting Backwards With Multicast

Topologies Overview

ASM Troubleshooting

SSM Specific Troubleshooting

“In solving a problem of this sort, the grand thing is to be able to reason backward. That is a very useful accomplishment, and a very easy one, but people do not practice it much.”

Sherlock Holmes


Multicast – Troubleshooting ASM

Traffic

Troubleshooting

RP FHR

LHR


Multicast – Troubleshooting SSM

Traffic

Troubleshooting

FHR

LHR


Topologies Overview

10.1.2.0

10.2.3.0

10.2.4.0 10.4.5.0

3/1 0/1

0/2

0/3

3/3

2/2 2/4

3/5

3/4

3/5

6500-1 L0: 1.1.1.1

CRS L0: 2.2.2.2

N7K L0: 3.3.3.3

6500-2 L0: 4.4.4.4

4900M L0: 5.5.5.5

10.3.5.0

R1 R2

R3

R4

R5


R1 R2

R3

R4

R5

Topologies Overview

Source(s)

Receiver(s)

Reference Router

ASM Troubleshooting


ASM Troubleshooting

R1 R2

R3

R4

(*,G) from LHR towards the RP 1

LHR Cutting over to SPT (S,G)

3

RP FHR

LHR

FHR registers source with RP

RP sends PIM (S,G) Join Towards Source

2


ASM Troubleshooting: Shared Tree


RP

R2

R3

R4

LHR receives

IGMP Membership Report

1a

LHR (R5) sends a PIM (*,G)

To R4

1b

R4 (an IR) sends a PIM (*,G)

To R2 (RP)

1c

LHR



R2

R3

R4

LHR receives


1a

LHR

LHR receives


1a

RP

LHR Receives IGMP Membership Report 1a


LHR#show ip mroute Is completely empty

LHR#show ip igmp group Is completely empty

LHR#show ip igmp interface Is completely empty

LHR#show ip interface gig1/1 No multicast reserved groups joined


LHR Receives


1a

Problem:

LHR not seeing an IGMP Membership

Report from Receiver


R3

R4

R2

RP

Receiver:

100.1.1.100

1/1

LHR


LHR#show run interface gig1/1 interface GigabitEthernet1/1

no switchport

ip address 100.1.1.1 255.255.255.0

ip pim sparse-mode


LHR Receives


1a

Problem:

LHR not seeing an IGMP Membership


Issue:

PIM not enabled on interface of

Receiver

Other Things to Look For:

Layer 1 issues, Layer 2 issues, Access-lists.

R3

R4

R2

RP

Receiver:

100.1.1.100

LHR


1/1



Receiver:

100.1.1.100

LHR Receives


1a

GigabitEthernet1/1 is up, line protocol is up

Internet address is 100.1.1.1/24

….

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13

R3

R4

R2

RP

Receiver:

100.1.1.100

LHR


LHR#show ip interface gig1/1

1/1

Note Difference

Before enabling pim sparse-mode on the

interface, the result of show ip interface

gig1/1 resulted in showing that no multicast

reserved groups were joined.

Now…



http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xml#multicast-addresses-1

Address(s) Description

224.0.0.0 Base Address

224.0.0.1 All Systems on this Subnet

224.0.0.2 All Routers on this Subnet

….

224.0.0.4 DVMRP Routers [RFC1075]

224.0.0.5 OSPFIGP All Routers [RFC2328]

224.0.0.6 OSPFIGP Designated Routers [RFC2328]

….

224.0.0.9 RIP2 Routers [RFC1723]

224.0.0.10 IGRP Routers

….

224.0.0.13 All PIM Routers

….

224.0.0.18 VRRP

…

224.0.0.22 IGMP

http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xml











no switchport

ip address 100.1.1.1 255.255.255.0

ip pim sparse-mode


LHR receives


1a

LHR#show ip igmp groups

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter

239.1.1.1 GigabitEthernet1/1 00:18:35 00:02:31 100.1.1.100

R3

R4

R2

RP

Receiver:

100.1.1.100

1/1

LHR




10.4.5.0

10.3.5.0

R2

R3

R4 LHR (R5) sends a PIM (*,G)

To R4

1b

LHR RP

Who?

Where?

RPF Nbr?


To R4

1b

RP

LHR Sends PIM (*,G) Join 1b


LHR#show ip mroute (*, 239.1.1.1), 00:01:23/00:02:48, RP 0.0.0.0, flags: SJC

Incoming interface: Null, RPF nbr 0.0.0.0

Outgoing interface list:

GigabitEthernet1/1, Forward/Sparse, 00:01:23/00:02:48, H

LHR#show ip pim rp 239.1.1.1 Group: 239.1.1.1, RP: 0.0.0.0


Problem:

LHR doesn‘t know who the RP is for

this group.

RP

Who?

Where?

RPF Nbr?

?

?


RP 10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


LHR(config)#ip pim rp-address 2.2.2.2 ?

<1-99> Access-list reference for group

<1300-1999> Access-list reference for group (expanded range)

WORD IP Named Standard Access list

override Overrides dynamically learnt RP mappings

<cr>

LHR(config)#ip pim rp-address 2.2.2.2 override


• Static

• Auto-rp

• BSR

Note:

RP information can be shared 3 ways

RP

Who?

Where?

RPF Nbr?


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


RP

Who?

Where?

RPF Nbr?

LHR#sh ip pim rp 239.1.1.1

Group: 239.1.1.1, RP: 2.2.2.2, uptime 00:00:39, expires never

LHR#sh ip pim rp mapping 239.1.1.1

PIM Group-to-RP Mappings

Group(s): 224.0.0.0/4, Static-Override

RP: 2.2.2.2 (?)



RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


RP

Who?

Where?

RPF Nbr?

LHR#show ip mroute (*, 239.1.1.1), 00:08:18/00:02:46, RP 2.2.2.2, flags: SJC




LHR#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2) failed, no route exists

LHR#show ip route 2.2.2.2 % Network not in table


Problem:

LHR doesn’t know where the RP is for this

group.

Issue:

RP address not in RIB of LHR – routing

problem


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


RP

Who?

Where?

RPF Nbr?


LHR#show ip route 2.2.2.2 2.0.0.0/32 is subnetted, 1 subnets

O 2.2.2.2 [110/42] via 10.4.5.4, Gig 1/5

[110/42] via 10.3.5.3, Gig 1/4

Note:

Routing problem fixed. LHR now has

2 equal cost paths to the RP.


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


RP

Who?

Where?

RPF Nbr?


LHR#show ip mroute

(*, 239.1.1.1), 00:08:18/00:02:46, RP 2.2.2.2, flags: SJC




Problem:

LHR doesn’t know who the RPF

neighbor is for the shared tree for this

group.


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


RP

Who?

Where?

RPF Nbr?

LHR#show ip rpf 2.2.2.2

RPF information for ? (2.2.2.2) failed, no route exists


LHR#show ip pim neighbor No PIM neighbors listed

Problem:

LHR doesn’t know who the RPF neighbor

is for the shared tree for this group.


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


LHR#show ip pim neighbor

PIM Neighbor Table

Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,

P - Proxy Capable, S - State Refresh Capable

Neighbor Interface Uptime/Expires Ver DR

Address

Prio/Mode

10.3.5.3 GigabitEthernet1/4 00:17:57/00:01:21 v2 1 /

10.4.5.4 GigabitEthernet1/5 00:17:56/00:01:31 v2 1 / S P


Note:

“ip pim sparse-mode” added to

Gig1/4 and Gig1/5 on LHR.


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR



IP Multicast Best Practices for Enterprise Customers

“By default, if ECMP paths are available, the RPF for multicast traffic will be based on the highest IP address. This method is referred to as

the highest PIM neighbor behavior and is consistent with RFC 2362 for PIM Sparse mode but also applies to PIM-Bidir and PIM-SSM.”

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6552/ps6592/whitepaper_c11-474791.html

Question:

Why was 10.4.5.4 chosen as the RPF neighbor?

Answer:

If ECMP are available, the RPF for the

multicast traffic will be based on highest

IP address for the RPF neighbor

LHR#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2) RPF interface: GigabitEthernet1/5 RPF neighbor: ? (10.4.5.4) RPF route/mask: 2.2.2.2/32 RPF type: unicast (ospf 1)


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR


RP

Who?

Where?

RPF Nbr?


LHR (R5) sends a PIM

(*,G) To R4

1b

Note:

LHR now has all the information it

needs to send a PIM (*,G) join to its

RPF neighbor.

LHR#show ip mroute

(*, 239.1.1.1), 00:36:39/00:02:56, RP 2.2.2.2, flags: SJC

Incoming interface: Gig 1/5, RPF nbr 10.4.5.4


Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H

*,G

IIF

OIL

RPF nbr

RP flags


RP

2.2.2.2

10.4.5.0

2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

1/1

R4

LHR



RP

R2

R3

R4


To R2 (RP)

1c

LHR

RP

Who?

Where?

RPF Nbr?


To R2 (RP)

1c

IR Sends PIM (*,G) Join 1c


LHR#show ip mroute 239.1.1.1

(*, 239.1.1.1), 00:36:39/00:02:56, RP 2.2.2.2, flags: SJC

Incoming interface: GigabitEthernet1/5, RPF nbr 10.4.5.4


Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H

LHR#telnet 10.4.5.4

Trying 10.4.5.4 ... Open

R4#show ip mroute 239.1.1.1

(*, 239.1.1.1), 00:48:49/00:02:33, RP 2.2.2.2, flags: S

Incoming interface: Gig 3/3, RPF nbr 10.2.4.2, RPF-MFD


Gig 3/5, Forward/Sparse, 00:47:16/00:02:33, H


RP

Who?

Where?

RPF Nbr?

RP = 2.2.2.2 10.2.4.0

10.4.5.0

0/3

3/3 3/5

1/5

R2

R3

1/1

R4

LHR

Note:

Telnetting backwards towards the

root is a good way to troubleshoot.



R4#telnet 10.2.4.2

Trying 10.2.4.2... Open

R2#show mrib route 239.1.1.1

(*,239.1.1.1) RPF nbr: 2.2.2.2 Flags: C

Up: 00:52:12

Incoming Interface List

Decapstunnel0 Flags: A, Up: 00:52:12

Outgoing Interface List

GigabitEthernet0/0/0/3 Flags: F NS, Up: 00:52:12


R4 (an IR) sent a PIM (*,G)

To R2 (RP)

1c

Note:

R4 had all the information it needed

to send a PIM (*,G) join to its RPF

neighbor. RP = 2.2.2.2

10.2.3.0

10.2.4.0 10.4.5.0

0/2

0/3

3/3

2/2 2/4

3/5

1/4

1/5

10.3.5.0

R2

R3

R4

LHR


1/1




RP

R2

R3

R4

Receiver sends


1a


To R4

1b


To R2 (RP)

1c

LHR

(*,G) Between RP and LHR


R1 R2

ASM Troubleshooting Source Tree***

FHR receives

Multicast Data

2a

RP FHR

2b



RP builds S,G Back Towards Source

2

2c


***When a shared MDT is already active for that group on the RP

Geek - o - meter


R1 R2

ASM Troubleshooting Source Tree

FHR receives

Multicast Data

2a

RP FHR

FHR receives Multicast Data

2a

FHR Receives Multicast Data 2a


10.1.2.0

3/1 0/1 R1 R2

S: 200.1.1.100

G: 239.1.1.1

FHR

R1#show ip mroute Empty

R1#show run interface gig3/2 interface GigabitEthernet3/2

no switchport

ip address 200.1.1.1 255.255.255.0

3/2


Problem:

FHR not seeing multicast traffic

Issue:

PIM not enabled on interface of Source

FHR receives

Multicast Data

2a

RP




ip pim sparse-mode


R1#show ip mroute (*, 239.1.1.1), 00:02:17/00:00:42, RP 0.0.0.0, flags: SP


Outgoing interface list: Null

Note:

We see that a (*,G) for that group has been

built.

10.1.2.0

3/1 0/1 R1 R2

S: 200.1.1.100

G: 239.1.1.1

FHR

3/21

RP





R1 R2

RP FHR

2b


2b


FHR Registers Source with RP 2b





Note:

We see that a (*,G) for that group has been built.

Question:

Why is there no (S,G)?

10.1.2.0

3/1 0/1 R1 R2

S: 200.1.1.100

G: 239.1.1.1

FHR

3/21

RP



Geek - o - meter


3/21

R1#show ip mroute 239.1.1.1 count

IP Multicast Statistics

2 routes using 1230 bytes of memory

2 groups, 0.00 average sources per group

Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second

Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)

Group: 239.1.1.1, Source count: 0, Packets forwarded: 0, Packets received: 6027

RP-tree: Forwarding: 0/0/0/0, Other: 6027/6026/1 RPF failed


FHR

10.1.2.0

3/1 0/1 R1 R2

RP


S: 200.1.1.100

G: 239.1.1.1

Note:

We see that a (*,G) for that group has been built.

Question:

Why is there no (S,G)?

Geek - o - meter


3/21





FHR

10.1.2.0

3/1 0/1 R1 R2

RP

Problem:

FHR does not know who the RP is for

this group.

RP

Who?

Where?

RPF Nbr?

R1#sh ip rpf 2.2.2.2

RPF information for ? (2.2.2.2) failed, no route exists


S: 200.1.1.100

G: 239.1.1.1


RP

Who?

Where?

RPF Nbr?

3/21

R1#sh ip pim rp mapping 239.1.1.1

PIM Group-to-RP Mappings

Group(s): 224.0.0.0/4, Static-Override

RP: 2.2.2.2 (?)

ip pim rp-address 2.2.2.2 override





FHR

10.1.2.0

3/1 0/1 R1 R2

RP = 2.2.2.2

• Static

• Auto-rp

• BSR

Note:

RP information can be shared 3 ways


S: 200.1.1.100

G: 239.1.1.1


RP

Who?

Where?

RPF Nbr?

R1#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2) failed, no route exists

2.0.0.0/32 is subnetted, 1 subnets

O 2.2.2.2 [110/2] via 10.1.2.2, 01:11:17, Gig3/1

R1#show ip route 2.2.2.2

R1#show ip pim neighbor No PIM neighbors


Problem:

FHR doesn’t know who the RPF

neighbor is for the shared tree for this

group.

3/21

FHR

10.1.2.0

3/1 0/1 R1 R2

S: 200.1.1.100

G: 239.1.1.1

RP = 2.2.2.2



R1#show ip mroute

(*, 239.1.1.1), 00:02:45/stopped, RP 2.2.2.2, flags: SPF

Incoming interface: Gig3/1, RPF nbr 10.1.2.2, RPF-MFD


(200.1.1.100, 239.1.1.1), 00:01:34/00:02:59, flags: PFT

Incoming interface: Gig3/21, RPF nbr 0.0.0.0, Registering (data-header), Partial-SC



Note:

“ip pim sparse-mode” added to Gig3/1 on

FHR (R1) 3/21

FHR

10.1.2.0

3/1 0/1 R1 R2

RP = 2.2.2.2

S: 200.1.1.100

G: 239.1.1.1



R1 R2


RP FHR

2c


2c


Geek - o - meter


3/21

R1#show ip mroute


Incoming interface: Gig3/1, RPF nbr 10.1.2.2, RPF-MFD


(200.1.1.100, 239.1.1.1), 00:01:34/00:02:59, flags: PFT

Incoming interface: Gig3/21, RPF nbr 0.0.0.0, Registering (data-header), Partial-SC



FHR

10.1.2.0

3/1 0/1 R1 R2

RP

Problem:

(S,G) stuck in “registering” on FHR (R1)

S: 200.1.1.100

G: 239.1.1.1


Geek - o - meter


R1#show proc cpu

(note: 1 mcast stream is being sent at Tx rate of 20Mbps)

CPU utilization for five seconds: 74%/31%; one minute: 73%; five

minutes: 41%

PID 5Sec 1Min 5Min TTY Process

156 42.31% 41.93% 23.57% 0 IP Input


Note:

Not only is the (S,G) stuck in registering, but

the CPU on FHR is also higher than normal. 10.1.2.0

3/1 0/1 R1 R2

RP FHR

Question:

Is the high CPU related to our (S,G) that is

stuck in registering? S: 200.1.1.100

G: 239.1.1.1


Geek - o - meter


R1#sh int summary | exclude ^

* GigabitEthernet3/1 0 0 0 0 0 0 20079000 2385 0

* GigabitEthernet3/21 24 96 0 0 19614000 2393 0 0 0

There is a space after ^

RXBS: rx rate (bits/sec) TXBS: tx rate (bits/sec)

RXPS: rx rate (pkts/sec) TXPS: tx rate (pkts/sec)


10.1.2.0

3/1 0/1 R1 R2

RP

Answer:

We actually have only 2 interfaces that are

up/up on R1. So yes, the high CPU is

related to the multicast traffic stream

Question:

Is the high CPU related to our (S,G) that is

stuck in registering?

FHR

S: 200.1.1.100

G: 239.1.1.1


Geek - o - meter


R2#show pim rpf 200.1.1.100

* 200.1.1.100/32 [-1/-1]

via Null with rpf neighbor 0.0.0.0

R2#show route 200.1.1.100

% Network not in table


FHR

10.1.2.0

3/1 0/1 R1 R2

RP

Question:

Why is the FHR having higher CPU?

Clue:

OIL on FHR for (S,G) is null

Note:

S,G tree will get built backwards from the RP.

So troubleshoot backwards from RP.

FHR

S: 200.1.1.100

G: 239.1.1.1


Geek - o - meter


FHR

ASM Troubleshooting Source Tree***

RP

E0

Register Stop to the FHR

Mcast data Mcast packet is encapsulated into

unicast PIM packet:

Destination IP is the RP.

PIM header type is Register

Mcast data Dest IP=RP

RP sends PIM (S,G) Join Back Towards

Source if there is an active shared tree

for that G

Mcast data Once the RP sees the packets

come in on the (S,G), it sends a

unicast Register Stop

***When a shared MDT is already

active for that group on the RP

Question:

Why is the FHR having higher CPU?

Geek - o - meter


Source

Who?

Where?

RPF Nbr?

R2#show pim rpf 200.1.1.100

* 200.1.1.100/32 [-1/-1]

via Null with rpf neighbor 0.0.0.0

R2#show route 200.1.1.100

% Network not in table


FHR

10.1.2.0

3/1 0/1 R1 R2

RP

Problem:

RP doesn’t know where the Source

is for this group.

Issue:

Source address not in RIB of RP –

routing problem

FHR

S: 200.1.1.100

G: 239.1.1.1



Source

Who?

Where?

RPF Nbr?

R2#show mrib route

(200.1.1.100,239.1.1.1) RPF nbr: 10.1.2.1 Flags: L

Up: 00:50:36


GigabitEthernet0/1 Flags: A, Up: 00:11:56


GigabitEthernet0/3 Flags: F NS, Up: 00:49:00


R2#show pim rpf 200.1.1.100 * 200.1.1.100/32 [110/2]

via Gig /0/1 with rpf neighbor 10.1.2.1

Note:

Fix routing problem. RP now has

Source in its RIB.

FHR

10.1.2.0

3/1 0/1 R1 R2

RP FHR

S: 200.1.1.100

G: 239.1.1.1



10.1.2.0

3/1 0/1 R1 R2

RP

R1#show ip mroute


Incoming interface: GigabitEthernet3/1, RPF nbr 10.1.2.2, RPF-MFD


(200.1.1.100, 239.1.1.1), 00:10:47/00:03:20, flags: FT





Note:

Going back to FHR (R1) with this issue

fixed we now see that the (S,G) is no

longer stuck in registering and the CPU

is no longer high.

FHR

S: 200.1.1.100

G: 239.1.1.1



R1 R2

CPU utilization for five seconds: 0%/0%

R1#show proc cpu


Note:

Going back to FHR (R1) with this issue

fixed we now see that the (S,G) is no

longer stuck in registering and the CPU

is no longer high.

10.1.2.0

3/1 0/1

RP FHR

R1 R2

S: 200.1.1.100

G: 239.1.1.1



ASM Troubleshooting Source Tree Wireshark Sniffing*

10.1.2.0

3/1 0/1 S: 200.1.1.100

G: 239.1.1.1

RP FHR

*Not a complete Wireshark capture

** Wireshark reporting inner source/dest

R1 R2

RP Sending Register Stop

PIM Unicast Register w/ encapsulated

mcast data frame**

R2 sending a PIM

Join for (S,G)

Geek - o - meter



PIM Unicast Register w/ Encapsulated mcast Data Frame**


** Wireshark reporting inner source/dest

Source IP=10.1.2.1

Dest IP=2.2.2.2

Source IP=10.1.2.100

Dest IP=239.1.1.1

PIM

Register

Geek - o - meter



R2 Sending a PIM Join for (S,G)


Source IP=10.1.2.2

Dest IP=224.0.0.13

Group 239.1.1.1

Source 200.1.1.100

RPF Nbr

Geek - o - meter

PIM

Join/Prune



RP Sending Register Stop


Source IP=2.2.2.2

Dest IP=10.1.2.1

Group 239.1.1.1

PIM

Register-Stop

Source 200.1.1.100

Geek - o - meter


Case Study Source Registration Doesn‘t Stop

Potential Reasons for “Black Hole”

1. Source not known in IGP at RP or some IR on the way back to the source (this is the situation we just covered)

2. Usual suspects for any kind of packet loss (physical, firewall, ACL, etc)

3. Missing PIM Neighbor Relationship

4. Multicast Boundary Configured for this Multicast Group

S: 200.1.1.100

G: 239.1.1.1

RP FHR Mcast data

RP Wants to Join (S,G)

Registering of Source to the RP


10.1.2.0

3/1 0/1 R1 R2

S: 200.1.1.100

G: 239.1.1.1

RP

(S,G) Between RP and FHR


RP building S,G Back Towards Source

2


FHR

“The default behavior of PIM-SM is that routers with directly connected members will join the shortest path tree as soon as they detect a new multicast source.”

PIM-SM Frequently Forgotten Fact


R1 R2

R3

R4


3

RP FHR

LHR

ASM Troubleshooting Joining the SPT



(PIM (S,G) Joins towards FHR for source

3

10.4.5.0

10.3.5.0

R1 R2

R3

R4

LHR Builds Shortest Path Tree (S,G) Back Towards Source

1. If SPT Threshold is not infinity (default 0), when the mcast stream comes into the LHR it will try to cutover to the shortest path tree

2. LHR Now must answer 3 questions before sending (S,G) Join

Source

Who?

Where?

RPF Nbr?

LHR

FHR RP

LHR Cuts over to the SPT


3


MRT(0): RPF lookup for 200.1.1.100[0.0.0.0] (200.1.1.100) returned Gig3/2 10.1.5.1

MRT(0): (200.1.1.100,239.1.1.1), RPF change from /0.0.0.0 to Gig3/2/10.1.5.1

MRT(0): Reset the z-flag for (200.1.1.100, 239.1.1.1)

MRT(0): Create (200.1.1.100,239.1.1.1), RPF Gig3/2/10.1.5.1

MRT(0): WAVL Insert interface: Gig3/11 in (200.1.1.100,239.1.1.1) Successful

MRT(0): set min mtu for (200.1.1.100, 239.1.1.1) 18010->1500

MRT(0): Add Gig3/11/239.1.1.1 to the olist of (200.1.1.100, 239.1.1.1), Forward state - MAC built

PIM(0): Insert (200.1.1.100,239.1.1.1) join in nbr 10.1.5.1's queue

MRT(0): Set the J-flag for (200.1.1.100, 239.1.1.1)

PIM(0): Building Join/Prune packet for nbr 10.1.5.1

PIM(0): Adding v2 (200.1.1.100/32, 239.1.1.1), S-bit Join

PIM(0): Send v2 join/prune to 10.1.5.1 (Gig3/2)

S: 200.1.1.100

G: 239.1.1.1

3/1 0/1 0/3 3/3

R1 R2 R4

3/5

3/2

FHR

LHR


RP

Geek - o - meter

RPF lookup for Source changes from

/0.0.0.0 to Gig3/2/10.1.5.1

Create the (S,G)

Send the PIM (S,G) Join

LHR Cuts over to the SPT 3


R1 R2

R3

R4


3

RP FHR

LHR

(S,G) Between LHR and FHR


LHR Cuts over to the SPT 3


Vlan 110 .3

.2

RP

(*,G) and (S,G) from LHR DR

The LH DR is responsible for sending the PIM (*,G) Join towards the RP

The LH DR will also be responsible for cutting over to the (S,G) tree

R3

R4

Last Hop Designated Router Responsibilities

ASM Designated Routers Receivers


Multicast DR

R1

R3

R2

R4

.3

.2

Mcast Data 1

FH Designated Router registers Source to the RP

2

The FH DR is responsible for registering the Source/Group with the RP

First Hop Designated Router Responsibilities

RP

ASM Designated Routers Sources

Geek - o - meter


Multicast DR

10.2.3.0

10.2.4.0

R1

R3

R2

R4

.3

.2

Mcast Data 1


2

Question:

Who will be the RPF neighbor chosen by R2 for this PIM (S,G) Join? (assuming equal cost paths to the source)

RP


Geek - o - meter


Multicast DR

10.2.3.0

10.2.4.0

R1

R3

R2

R4

.3

.2

Mcast Data 1

Question:

Who will be the RPF neighbor chosen by

R2 for this PIM (S,G) Join? (assuming equal cost paths to the source)

RP

IP Multicast Best Practices for Enterprise Customers

“By default, if ECMP paths are available, the RPF for multicast traffic will be based on the highest IP address. This

method is referred to as the highest PIM neighbor behavior and is consistent with RFC 2362 for PIM Sparse mode but also

applies to PIM-Bidir and PIM-SSM.”

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6552/ps6592/whitepaper_c11-474791.html

PIM (S,G) Join

3

10.2.3.3 (R3)

10.2.4.4 (R4)

Question: Which one is higher?


2


Geek - o - meter


R4 is router passing the traffic


(110.1.1.100/32, 239.1.1.1/32), uptime: 00:03:23, ip pim

Incoming interface: Vlan110, RPF nbr: 110.1.1.100

Outgoing interface list: (count: 0)


(*, 239.1.1.1), 00:01:01/stopped, RP 2.2.2.2, flags: SP

Incoming interface: Gig 3/3, RPF nbr 10.2.4.2, RPF-MFD


(110.1.1.100, 239.1.1.1), 00:01:01/00:03:20, flags: T

Incoming interface: Vlan110, RPF nbr 0.0.0.0, RPF-MFD


Gig 3/3, Forward/Sparse, 00:01:01/00:03:28, H Notice lack of “F” (register flag) on R4

Multicast DR

10.2.3.0

10.2.4.0

R1

R3

R2

R4

.3

.2

Mcast Data


Geek - o - meter


R1 R2

R3

R4



3

RP FHR

LHR

ASM Troubleshooting


RP sends PIM (S,G) Join Towards Source

2


R1 R2

R3

R4

(S,G) from LHR towards Source

FHR

LHR




R1 R2

R3

R4


FHR

LHR

LHR receives

IGMPv3 Membership Report

ssm1

LHR (R5) sends a PIM (S,G)

To R4

ssm2

R4 (IR) send (S,G) Join to R2

ssm3

R2 (IR) send (S,G) Join to R1(FHR)

ssm4



R1 R2

R3

R4

FHR

LHR

LHR receives


ssm1

LHR Receives IGMPv3 Membership Report ssm1

LHR receives


ssm1



R1 R2

R3

R4

Receiver:

100.1.1.100

LHR#show ip mroute Shows nothing but the autorp group 224.0.1.40

LHR#show ip igmp group Is completely empty except for 224.0.1.40

G 1/1

LHR#show ip pim interface gig1/1

Address Interface Ver/ Nbr Query DR DR

Mode Count Intvl Prior

100.1.1.1 Gig1/1 v2/S 0 30 1 100.1.1.1

LHR

LHR receives


ssm1 Problem:

LHR not seeing IGMPv3 Membership Report

from Receiver




Destination IP

224.0.0.22

IGMP v3

http://www.rfc-editor.org/rfc/rfc3376.txt

R1 R2

R3

R4

G 1/1

LHR

LHR receives


ssm1 Note:

Sniffer trace of IGMPv3 Membership report

from receiver

Problem:

LHR not seeing

IGMPv3 Membership






no switchport

ip address 100.1.1.1 255.255.255.0

ip pim sparse-mode

LHR#show ip igmp interface gig3/11

GigabitEthernet3/11 is up, line protocol is up

Internet address is 100.1.1.1/24

IGMP is enabled on interface

Current IGMP host version is 2

Current IGMP router version is 2

<snip>

R1 R2

R3

R4

G 1/1

LHR

Problem:

LHR not seeing IGMPv3

Membership Report

from Receiver

LHR receives


ssm1

Issue:

IGMPv3 not enabled on interface






LHR#show ip igmp group g3/11

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter

232.1.1.1 Gig1/1 17:53:07 stopped 100.1.1.100

Note:

“ip igmp version 3” interface command

added under gig3/11 on the LHR

R1 R2

R3

R4

G 1/1

LHR

LHR receives


ssm1




R1 R2

R3

R4

FHR

LHR


To R4

ssm2


To R4

ssm2

LHR Sends (S,G) Join ssm2



LHR#show ip mroute

(*, 232.1.1.1), 17:51:47/00:02:58, RP 2.2.2.2, flags: SJC




LHR#show ip mroute ssm Is completely empty

??

Flags:

D - Dense,

S - Sparse,

B - Bidir Group,

s - SSM Group

Mode of Group

ASM

SSM

R1 R2

R3

R4

G 1/1

LHR

Problem:

LHR is configured for this group

to be an ASM group, not an SSM

group.



Mode of Group

ASM

SSM


LHR(config)#ip pim ssm ?

default Use 232/8 group range for SSM

range ACL for group range to be used for SSM

LHR(config)#ip pim ssm default

-- http://www.iana.org/assignments/multicast-addresses/

“232.0.0.0-232.255.255.255 (232/8) Source-Specific Multicast Block”

R1 R2

R3

R4

G 1/1

LHR

Note:

Some platforms (XR) have the

232/8 group as SSM by default.

In IOS it must be configured.

If SSM Source

Who?

Where?

RPF Nbr?




R1 R2

R3

R4

LHR

Question:

Assuming equal cost paths to the source from the

LHR thru R3 and R4, which router will the LHR use

as its RPF neighbor?

10.4.5.0

2/4

3/5

1/5

10.3.5.0

14




LHR#show ip mroute

(200.1.1.100, 232.1.1.1), 00:01:24/00:01:59, flags: sTI




s - SSM Group

T - SPT-bit set

I - Received Source Specific Host Report

PIM (S,G)

Join Sent

R1 R2

R3

R4

G1/1

LHR

Mode of Group

ASM

SSM

If SSM Source

Where?

RPF Nbr?

Note:

With this group now an SSM group, the

LHR asks itself where is the Source and

who is the RPF nbr for this source.

Then it sends the (S,G) Join. 10.4.5.0

2/4

3/5

1/5

10.3.5.0

1/4




R1 R2

R3

R4

FHR

LHR


ssm3


ssm3

IR (R4) Sends (S,G) Join ssm3



R4#show ip mroute

(*, 232.1.1.1), 18:13:34/stopped, RP 2.2.2.2, flags: SP

Incoming interface: GigabitEthernet3/3, RPF nbr 10.2.4.2, Partial-SC


(200.1.1.100, 232.1.1.1), 00:00:12/00:03:17, flags:



GigabitEthernet3/5, Forward/Sparse, 00:00:12/00:03:17

??

(S,G) Join

Received

R1 R2

R3

R4

LHR

Mode of Group

ASM

SSM

Problem:

IR is configured for this group to be

an ASM group, not an SSM group. 10.4.5.0

3/5 3/3 10.2.4.0




R4#show ip mroute

(200.1.1.100, 232.1.1.1), 00:05:41/00:02:44, flags: sT




R1 R2

R3

R4

G1/1

LHR

Mode of Group

ASM

SSM

If SSM Source

Where?

RPF Nbr?

Note:

With this group now an SSM group, the

IR (R4) asks itself where is the Source

and who is the RPF nbr for this source.

Then it sends the (S,G) Join.

10.4.5.0 3/5 3/3

10.2.4.0




R1 R2

R3

R4

FHR

LHR


ssm4


ssm4




R2#show pim group-map

IP PIM Group Mapping Table

(* indicates group mappings being used)

(+ indicates BSR group mappings active in MRIB)

Group Range Proto Client Groups RP address Info

224.0.1.39/32* DM perm 0 0.0.0.0

224.0.1.40/32* DM perm 1 0.0.0.0

224.0.0.0/24* NO perm 0 0.0.0.0

232.0.0.0/8* SSM config 1 0.0.0.0

224.0.0.0/4* SM config 0 2.2.2.2 RPF: De0,2.2.2.2

224.0.0.0/4 SM static 0 0.0.0.0 RPF: Null,0.0.0.0

R1 R2

R3

R4

G1/1

LHR

Note:

XR has a very useful command

“show pim group-map”




R2#show mrib ipv4 route 232.1.1.1

(200.1.1.100,232.1.1.1) RPF nbr: 10.1.2.1 Flags:

Up: 01:22:22


GigabitEthernet0/1/0/1 Flags: A, Up: 01:22:22


GigabitEthernet0/1/0/3 Flags: F NS, Up: 01:22:22

R2 sends

PIM (S,G) Join

R1 R2

R3

R4

LHR

Mode of Group

ASM

SSM

If SSM Source

Where?

RPF Nbr?

Note:

IR (R2) asks itself where is the Source

and who is the RPF nbr for this source.

Then it sends the (S,G) Join.

0/1

10.1.2.0

0/3


G1/1




(200.1.1.100, 232.1.1.1), 00:00:28/00:03:01, flags: sT




R1#show ip mroute 232.1.1.1 count

Group: 232.1.1.1, Source count: 1, Pkts forwarded: 0, Pkts received: 0

Source: 200.1.1.100/32, Forwarding: 0/0/0/0, Other: 0/0/0

Note:

(S,G) MDT built all the way from the

LHR to the FHR. But note how no

traffic has yet been sent from the

source.

R2 sends

PIM (S,G) Join

R1 R2

R3

R4

LHR

0/3

3/1

10.1.2.0

G1/1




R1 R2

R3

R4


FHR

LHR

LHR receives


ssm1


To R4

ssm2


ssm3


ssm4

Questions

BRKIPM- 2264

Recommended Reading


Please complete your Session Survey

Don't forget to complete your online session evaluations after each session.

Complete 4 session evaluations & the Overall Conference Evaluation

(available from Thursday) to receive your Cisco Live T-shirt

Surveys can be found on the Attendee Website at www.ciscolivelondon.com/onsite

which can also be accessed through the screens at the Communication Stations

Or use the Cisco Live Mobile App to complete the

surveys from your phone, download the app at

www.ciscolivelondon.com/connect/mobile/app.html

We value your feedback

http://m.cisco.com/mat/cleu12/

1. Scan the QR code

(Go to http://tinyurl.com/qrmelist for QR code reader

software, alternatively type in the access URL above)

2. Download the app or access the mobile site

3. Log in to complete and submit the evaluations

http://www.ciscolivelondon.com/onsite

http://www.ciscolivelondon.com/connect/mobile/app.html

http://tinyurl.com/qrmelist


Thank you.

multicast troubleshooting

Documents