leveraging route explorer to solve common eigrp … · eigrp a bandwidth-efficient routing protocol...

Leveraging Route Explorer to

Solve Common EIGRP Problems

EIGRP Route Analytics Technical Brief

2013 Packet Design, Inc. of 18

Table of Contents

Introduction ................................................................................................. 3

Technical Background ................................................................................. 3

EIGRP Operation ................................................................................................. 3

Route Explorer for EIGRP Networks ................................................................... 4

Troubleshooting EIGRP with Route Explorer .............................................. 5

Categories of Common EIGRP Problems ............................................................. 5

Neighbor Relationship Problems .......................................................................... 5

Unidirectional links .......................................................................................... 6

Mismatched interface masks and AS numbers .................................................. 7

Stuck-in-Active ................................................................................................ 8

Route Advertisement Problems ............................................................................ 8

Discontiguous Networks .................................................................................. 9

Unexpected Routes and Metrics ..................................................................... 10

Route Installation Problems ............................................................................... 14

Duplicate Router IDs ...................................................................................... 14

Route Flapping Problems ................................................................................... 15

Conclusion ..................................................................................................18



Introduction

Cisco’s EIGRP routing protocol is an enhanced distance-vector protocol used in many large enterprise networks due to its scalability and quick convergence time. As with networking in general, troubleshooting EIGRP problems can be an intricate and time-consuming affair, especially due to the large amount of manual data collection that is required to assemble the basis for an intelligent analysis. The new technology of route analytics greatly increases the efficiency of routing problem diagnosis and resolution, by providing network-wide, real-time and historical routing analysis. Packet Design’s Route Explorer is the leading route analysis solution, utilized by dozens of global Fortune 500 class enterprises, government and military agencies, educational institutions and the world’s largest Service Providers. Route Explorer is also the only route analytics solution that supports the EIGRP protocol, providing network-wide, real-time and historical monitoring, analysis and troubleshooting of complex EIGRP routing issues. This technical brief illustrates how Route Explorer can be used to speed detection and resolution of some common EIGRP issues.

Technical Background

This section gives a brief overview of how EIGRP and Route Explorer work.

EIGRP Operation

EIGRP implements a number of enhancements over earlier distance-vector protocols that make EIGRP a bandwidth-efficient routing protocol useful for deployment in large networks:

• Incremental route updates based on topology changes that are not locally resolvable

• Updates are only sent to relevant neighbor routers rather than to all routers

• Finite time windows within which protocol messages must be received

When a router advertises a route, it includes the total calculated metric for the entire path to that destination network. Each router establishes a feasible distance for each route by calculating the lowest total metric based on the reported distances (RD) it has received from its neighbors and the metric to reach each of the advertising routers. The neighbor router that is calculated to have the lowest feasible distance to a route is established as the successor for that route and is placed in the routing table. As route advertisements are received for particular networks, their RD is compared to the feasible distance of the successor. If (and only if) a route is advertised with an RD lower than the successor’s feasible distance, the advertising router is recorded in the routing table as a feasible successor.

When the network is in steady state and the successors for each network are known, each network listed in the routing table is said to be in the passive state. The list of feasible successors for a particular route will be reassessed locally if there are any changes to the cost of the links, a change of state or if update, query or reply packets are received. It could be that a feasible distance changes, or that a feasible successor takes over from the existing successor. Provided that a new successor is found, this is advertised via updates while remaining in passive state. In this case, if a topology change occurs, the router can quickly find an alternate route without having to recompute the route.



However, if the network changes, and a feasible successor is not found for a given route, then the local router goes into active state, and queries its neighbors for routing information to the desired network. The local router sets a Reply Status flag to track all the queries to its neighbor routers. If a neighbor has feasible successors, it will recalculate its own local distance to the network and send this back as an RD. If a neighbor does not have a feasible successor, then it will itself move into active state and query its neighbors.

Route Explorer for EIGRP Networks

Route Explorer’s EIGRP routing analysis software discovers and monitors multi-domain EIGRP networks from a single appliance. The network topology map (see Figure 1) shows all EIGRP ASes clearly and distinctly using different colors for each area. Route prefixes are listed by the AS they belong to, making it easy to trouble-shoot problems. Additional information about the network such as the model of each router and the IOS version running on it is also displayed.

Figure 1: Full topology view of a multi-AS EIGRP network

A complete prefix advertisement history from the EIGRP update packets is maintained for the entire network, providing an audit trail that includes prefix type, AS of origin, metrics, etc. These events are then resolved by Route Explorer’s patent-pending algorithms into the link state events that caused the EIGRP updates. Stepping through the event history greatly aids the network engineer in performing forensic analysis.

Tuning of link metrics or simulating down links and routers for impact analysis simplifies network planning and allows preventive actions to be taken. Alerts and reports can be generated on flapping routes or prefixes, excessive network churn, critical path changes and other events that indicate potential service outages.



For a more thorough explanation of how Route Explorer works in EIGRP networks, see the Route Explorer EIGRP Tech Note at

http://www.packetdesign.com/documents/EIGRPTechNoteRev3.pdf.

Troubleshooting EIGRP with Route Explorer

Categories of Common EIGRP Problems

EIGRP problems fall into a number of categories, for which Route Explorer provides significant troubleshooting and forensic intelligence, leading to greater IT efficiency and responsiveness. The following topics are covered in this technical brief:

• Neighbor relationship problems

• Route advertisement problems

• Route installation problems

• Route flapping problems

Neighbor Relationship Problems

Whenever an EIGRP router resets a neighbor relationship, it is noted in the router’s log with the reason for the reset. However, detecting the source of the problems by log analysis can be difficult and time consuming. Route Explorer can dramatically cut down the time needed to identify and troubleshoot neighbor relationship problems by providing flapping link reports, topology visualizations and detailed analyses. Figure 2 shows a sample flapping link report.



Figure 2: Flapping Link report

In addition, Route Explorer can be configured to send threshold-based SNMP traps or Syslog messages on lost, established or flapping adjacencies to upstream management systems.

Following are some of the EIGRP neighbor relationship problems that can be detected and diagnosed by Route Explorer:

Unidirectional links

A problem that can easily evade detection by standard network operation processes is a one-way neighbor relationship. A one-way neighbor relationship is a unidirectional connection between neighbors, perhaps caused by Layer 2 link errors such as CRC errors, or a misconfigured access list. The difficulty in detecting such as problem is that it requires validation from both neighbors to verify that there is in fact an issue. Route Explorer provides standard reports and alerts on adjacency and link changes, and also provides a real-time and historical, network-wide topology view that color-codes uni-directional links for easy identification, as shown in figure 3.



Figure 3: An EIGRP network topology view in Route Explorer, displaying a uni-directional link

Mismatched interface masks and AS numbers

Simple router misconfigurations such as changing the interface subnet mask or the AS number on an EIGRP neighbor can cause adjacencies to be lost. Route Explorer can be used to alert on lost adjacencies through SNMP traps or Syslog messages, or network operators can monitor the topology view for red color-coded links, indicating lost adjacencies. When a lost adjacency is detected, the operator can right-click on one of the routers in the topology view, then click the “neighbors” button to display all its known neighbors, as shown in Figure 4. Any recently lost neighbor will still be listed, with its last known interface IP address and AS membership. This information can be used to investigate the neighbor router and if that interface address still exists or has been misconfigured in some fashion.

Color-coded display of uni-

directional link



Figure 4: Neighbor list for an EIGRP router

Stuck-in-Active

One of the most trying and difficult to diagnose errors for EIGRP administrators is a condition known as Stuck-in-Active. Stuck-in-Actives occur when a query for an active prefix is left unresponded to by a neighbor, due typically to a link problem. The lack of response triggers the Stuck-in-Active state on the chain of routers involved in the active query path to the unresponsive neighbor. Since EIGRP resets Stuck-in-Active adjacencies after three minutes, in the worst cases a cascading series of downed adjacencies can trigger a large number of active queries for prefixes lost when the adjacencies were reset. Route Explorer offers a number of ways to prevent, detect and resolve Stuck-in-Actives that are covered in a separate technical brief, entitled: “Preventing and Diagnosing EIGRP Stuck-in-Active Issues with Route Explorer”, which can be found along with other white papers at Packet Design’s website at:

http://www.packetdesign.com/technology/wp.htm

Route Advertisement Problems

It is not uncommon for network engineers to encounter situations where route advertisements are not working as they would anticipate, such as when EIGRP is not advertising certain routes or where routes are being advertised in an unexpected manner. Following are two examples of



route advertisement problems where Route Explorer’s network-wide analysis can be very helpful in troubleshooting the issue.

Discontiguous Networks

Route advertisement problems can sometimes occur when there are discontiguous networks (defined as two subnets of a major network separated by another major network)--for example, when two related networks, 192.121.3.13/28 and 192.121.13.21/28 are separated by 10.10.3.0/24. By definition, manual route summarization on an interface, or auto-summarization across a major network boundary restrains route advertisements. In this case, when the two discontiguous networks are advertised across the major network boundary, they will both be summarized into 192.121.3.0/24. Since both sides of the major network boundary already possess a route for 192.121.3.0/24, the advertisement will be rejected and the discontiguous networks will be unreachable across the major network boundary.

Up until now it has been difficult for network administrators to easily find the summarization points in an EIGRP network, making it difficult to troubleshoot this type of route advertisement problem. Route Explorer can easily show all manual or auto-summarization points by allowing the network engineer to display a list of all prefixes in an entire network or in a particular AS, as shown in Figure 5. Each prefix shows the router that advertised the prefix, along with the metrics associated with the prefix, and the EIGRP prefix type (internal, external, static, static external, loopback, dial-up, manual or auto-summarized). By sorting the prefix column, then finding the prefix that is not being advertised, it is easy to find all instances of the prefixes that have been summarized.

Figure 5: Prefix list for an entire, multi-AS EIGRP network, sorted to easily show that prefix 245.132.224.0/19 is being summarized at router Loke.

Route Explorer can also provide a filtered view of prefixes as shown in Figure 6, which displays the prefix list filtered by auto summary and manual summary EIGRP Prefix Types. Figure 6 also shows that by clicking on the router in question, the topology display highlights and flashes the selected router for easy identification on the topology map. Route Explorer allows network engineers to easily view prefixes filtered by:



• Protocol (EIGRP or Static)

• Router (user specified)

• Prefix (user specified)

• EIGRP Prefix Type

• Static Next-Hop Type

• Regular Expressions

Figure 6: A filtered list of prefixes easily finds an auto-summary that may be blocking the prefix.

Unexpected Routes and Metrics

EIGRP can sometimes advertise unexpected routes to its neighbors. One case where this can occur is when static routes are configured to an interface rather than referencing next-hop IP addresses, which EIGRP interprets to mean that they should be redistributed since they belong to a routed link. In this case, Route Explorer’s prefix list tool can be filtered to show all instances of prefixes with a static next hop to an interface (as seen in Figure 7), and the routers that are advertising these links. Once the redistribution problem is corrected, Route Explorer’s topology view can be used to validate that the static routes are no longer being advertised by selecting an adjacent router, and examining its prefix list to see if the routes in question are still seen (and thus being advertised)

Highlighted router flashes yellow in the topology view



Figure 7: Route Explorer can show prefixes with an interface static nexthop

Yet another route advertisement issue is when unexpected route metrics are being advertised. Router misconfigurations such as inadvertently created offset lists can cause suboptimal routing by altering metrics that affect EIGRP route selection. With Route Explorer it is easy to identify metrics for any route from the prefix list display. In addition, Route Explorer allows network engineers to perform what-if analyses on link metrics, where a change in a link’s metric can be simulated in Route Explorer’s topology view mode, and its effect on a route can be easily visualized. An illustration of a changed metric analysis is shown in Figures 8 - 10



Figure 8 shows how a route can be modeled in the network by selecting any router as the route source, then selecting any other router as the route destination. A green, bi-directional highlighted path shows the route that packets will take in the production network in either direction.

Route Explorer’s route modeling and failure/change scenario analysis capabilities help network engineers assess critical routes that service a high number or sensitive types of flows. Engineers can easily determine if the expected and optimal routing is in effect, and what the impact of routing changes or link/router failures would be on the reachability for that critical route. Multiple failures can be simulated allowing predictive analysis of the network’s redundancy and resiliency with no impact on users. Route Explorer’s route modeling also provides visualization of single vs. Equal Cost Multi-Path (ECMP) routes.



Figure 9 shows how a link can be selected and its metrics changed.

A number of what-if scenarios can be modeled on the as-running network topology in Route Explorer, including:

• Changed metrics

• Link state changes (up/down)

• New routers and links

• Router status changes (up/down)



Figure 10 shows the results of the changed metric—a different path between the selected route source and route destination.

Since it reflects the “as-running” network’s routing, Route Explorer displays what routing changes would occur as a result of any metric change on the production network. Combined with the prefix list, this modeling capability can be an invaluable tool in assessing the state of metrics in the network, and troubleshooting the effect of intentional or unintentional metric changes.

Route Installation Problems

Another condition that network administrators encounter is routes that are not installed in the routing table as expected. One major cause for route installation errors is route summarization, which has been covered in the route advertisement section. Another cause for route installation errors is duplicate router IDs in a network.

Duplicate Router IDs

EIGRP determines router ID by selecting the highest address of the router’s loopback interfaces, or if there are no loopback interfaces, the highest address of the router’s external interfaces. Due to misconfigurations, two routers in the network can end up with the same router ID, which can block route installation when one of the two routers sharing the same ID is the originating router for an external route. In this case, the other router will not install the route, since it



assumes that it is in fact the originator of the EIGRP route. As shown in Figure 11, Route Explorer provides a network-wide EIGRP topology error list, making it easy to monitor an EIGRP network for duplicate ID and other EIGRP topology errors, such as:

• Router ID unroutable

• Duplicate IP address on two separate routers

• External Prefix unreachable due to potential redistribution errors

• Prefixes with invalid delay value of zero

Figure 11: The EIGRP topology error list shows a duplicate router ID issue, and the two routers are highlighted in yellow in the topology view by double-clicking on the error line.

Route Flapping Problems

Persistent route flapping is a serious problem that can cause application performance degradation. A major challenge in troubleshooting route flaps is a lack of easily obtained, comprehensive and historical information on EIGRP events. Routers have very limited logging space to keep historical information on EIGRP events—typically only 500 lines, which may cover only a few hundred milliseconds of EIGRP events. Since route flaps generate a significant volume of EIGRP events, depending on router logs is insufficient. The alternative, turning on debug mode, can have ill effects on production networks, so it is not casually utilized. Also, since each router’s log is independent, it can take a lot of work to assemble a comprehensive view of the network’s events.



By contrast, Route Explorer keeps a full history of EIGRP events and a historically synchronized topology of the entire network, for greatly improved forensics and real-time troubleshooting. Route Explorer provides a number of reports that can be proactively monitored and that provide a home base for troubleshooting emerging problems such as route flaps. One such report, shown in Figure 12, is the “Network Churn” report, that can be run on a configurable timeframe, and that shows a summary of all network change events on a per router basis, including router, link and prefix events. A high incidence of prefix events may warrant investigation to see if there is a persistent route flap occurring. For example, router “Huko” had 125 prefix events in the 48 hours between March 18th and the end of March 19th. An advantage of utilizing Route Explorer is that it introduces low overhead on the network, since it relies primarily on passive listening to EIGRP updates.

Figure 12: The Network stability report shows network churn events such as router churn, link flaps, and prefix flaps



The network engineer can then launch the History Navigator tool, shown in Figure 13, and utilize a variety of analysis capabilities:

• RIB Browser: Shows the state of the network-wide Routing Information Base, includingdowned links and prefixes

• RIB Comparison: Shows a before and after comparison of the network-wide RIB,highlighting the changes between any two points in time (see Figure 14)

• Events Analysis: Shows a list of all events in a defined timeframe, which can be selectedby choosing start and stop times with a cursor on the histogram

Figure 13: Route Explorer’s History Navigator tool provides a comprehensive history of routing events for forensic analysis, troubleshooting and network planning



Figure 14: The RIB Comparison shows a before and after analysis of the network-wide or AS-wide RIB, including downed links or downed prefixes (shown above)

Conclusion

EIGRP route analysis greatly enhances network administrators’ visibility, preventive and forensic analysis capabilities, leading to increased IT efficiency, responsiveness and productivity. Using a combination of Route Explorer’s capabilities with other router administration tools, network administrators and engineers can prevent many common EIGRP errors and more rapidly detect and diagnose the cause of complex routing problems when they happen. The result is a reduction of costly network downtime, and freeing of resources to focus on proactive service availability improvements.

To learn more about Packet Design and Route Explorer, please:

• Email us at [email protected]

• Visit Packet Design’s web site at http://www.packetdesign.com

• Call us at 408.490.1000

Corporate Headquarters Packet Design Inc.

2455 Augustine Drive

Santa Clara, CA 95054

Phone: 408.490.1000 Fax: 408.562.0080

http://www.packetdesign.com

leveraging route explorer to solve common eigrp … · eigrp a bandwidth-efficient routing protocol...

Documents