7750 series troubleshooting manual

Upload: eddiemark

Post on 08-Feb-2018

283 views

Category:

Documents


2 download

TRANSCRIPT

  • 7/22/2019 7750 Series Troubleshooting Manual

    1/130

    Alcatel

    This document contains Confidential Information of Alcatel.

    31NAN0090

    Issue Version 2.0, Aug. 5th, 2004

    7750 SR Series Troubleshooting Guide

    Application Note

    IPD Support & Services

    Abstract:This document provides detailed information on diagnosing faults in R2.0 of the 7750 SR

  • 7/22/2019 7750 Series Troubleshooting Manual

    2/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    2

    Table of contents

    1. INTRODUCTION ...............................................................................................................................................................5

    1.1. INTENDED AUDIENCE FOR THIS GUIDE .........................................................................................................................51.2. HOW THIS DOCUMENT IS ORGANIZED...........................................................................................................................51.3. WHERE TO BEGIN?........................................................................................................................................................61.4. RELATED DOCUMENTS .................................................................................................................................................6

    2. TROUBLESHOOTING PROCESS...................................................................................................................................7

    2.1. ESTABLISHING A BASELINE ..........................................................................................................................................72.2. CHARACTERIZE THE PROBLEM......................................................................................................................................82.3. IDENTIFY THE ROOT CAUSE ..........................................................................................................................................92.4. PLAN YOUR ACTIONS &RESOLVE THE PROBLEM........................................................................................................102.5. VERIFY SOLUTIONS.....................................................................................................................................................11

    3. TROUBLE SHOOTING TOOLS ....................................................................................................................................12

    3.1. EVENT LOGS ...............................................................................................................................................................123.1.1. Event logging overview .......... .......... ........... .......... ........... .......... ........... .......... ........... .......... .......... .......... ........... .. 12

    3.1.1.1 Event Sources................................................................................................................................................................ 133.1.1.2 Event Control ................................................................................................................................................................ 143.1.1.3 Log manager.................................................................................................................................................................. 173.1.1.4 Event Filter Policies ...................................................................................................................................................... 173.1.1.5 Log Destinations............................................................................................................................................................ 19

    3.1.2. List of show commands for event logging ........... ........... .......... ........... .......... ........... .......... ........... ......... ........... .... 223.2. SERVICE MIRRORING...................................................................................................................................................23

    3.2.1. Service mirroring overview...................................................................................................................................233.2.2. Mirror implementation .......... ........... .......... ........... .......... ........... .......... ........... .......... ........... ............. ........... ......... 24

    3.2.2.1 Mirror Source and Destinations..................................................................................................................................... 253.2.2.2 Mirroring performance.................................................................................................................................................. 27

    3.2.3. Mirroring configuration ........... .......... ........... .......... ........... .......... ........... .......... ........... ........... .......... ........... ......... 273.2.3.1 Mirror configuration process overview ......................................................................................................................... 293.2.3.2 Mirror configuration components.................................................................................................................................. 293.2.3.3 Basic mirror configuration Example.............................................................................................................................. 303.2.3.4 Mirror configuration Notes............................................................................................................................................ 333.2.3.5 List of CLI commands to configure Mirroring parameters............................................................................................ 34

    3.3. OA&MCOMMANDS FOR TROUBLESHOOTING .............................................................................................................363.3.1. LSP Diagnostics .......... ........... .......... ........... .......... ........... ........... .......... ........... .......... ......... ........... .......... ........... .. 363.3.2. SDP Diagnostics ...................................................................................................................................................363.3.3. Service Diagnostics...............................................................................................................................................373.3.4. VPLS MAC Diagnostics ........................................................................................................................................383.3.5. OAM Command Summary.....................................................................................................................................40

    4. HARDWARE OPERATIONAL STATUS......................................................................................................................42

    4.1. 7750SR-12HARDWARE OVERVIEW...........................................................................................................................424.2. VERIFYING ROUTER BOOT SEQUENCE ........................................................................................................................454.3. VERIFYING MANAGEMENT CONNECTION OPERATIONAL STATUS...............................................................................45

    4.3.1. Console Port Management Connection.................................................................................................................454.3.2. Telnet Management Connection............................................................................................................................46

    4.4. VERIFYING CHASSIS OPERATIONAL STATUS............................................................................................................... 464.4.1. Chassis Configurations .........................................................................................................................................464.4.2. Things to Check - Power Supply ...........................................................................................................................484.4.3. Things to Check - Fans..........................................................................................................................................49

    4.5. VERIFYING SF/CPMOPERATIONAL STATUS ..............................................................................................................504.5.1. Minimum Configuration .......... .......... ........... ........... .......... ........... .......... ........... .......... ........... ........... ........... ......... 504.5.2. SF/CPM LED Status..............................................................................................................................................504.5.3. CLI commands for SF/CPM troubleshooting........................................................................................................514.5.4. CLI commands for SF/CPM health check.............................................................................................................53

    4.6. VERIFYING IOMOPERATIONAL STATUS ....................................................................................................................574.7. VERIFYING MDAOPERATIONAL STATUS ...................................................................................................................58

  • 7/22/2019 7750 Series Troubleshooting Manual

    3/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    3

    5. SYSTEM LEVEL CONFIGURATION VERIFICATION............................................................................................60

    5.1. SUMMARY OF SYSTEM CONFIGURATION VERIFICATION...............................................................................................605.2. SYSTEM INITIALIZATION TROUBLESHOOTING .............................................................................................................61

    5.2.1. Boot Option File configuration ......... ........... .......... ........... .......... ........... ........... .......... ........... ........... ........... ......... 615.2.2. Troubleshooting notes on BOF configuration.......................................................................................................635.2.3. Commands to check config file contents ...............................................................................................................63

    5.3. VERIFY SYSTEM MANAGEMENT CONFIGURATION .......................................................................................................675.3.1. Display system information ........... .......... ........... .......... ........... .......... ........... ........... .......... ............. .......... ........... .. 685.3.2. Verify Synchronization and Redundancy...............................................................................................................695.3.3. Verify timing configuration ...................................................................................................................................705.3.4. Verify SNTP configuration....................................................................................................................................72

    5.4. SECURITY ACCESS CONFIGURATION ...........................................................................................................................725.4.1. Authentication, Authorization and Accounting .......... ......... ......... .......... ......... ......... .......... ......... ................. ......... 725.4.2. How AAA is configured .......... ........... .......... ........... .......... ........... .......... ........... .......... ........... .......... ........... .......... .745.4.3. Security Configuration Components .....................................................................................................................76

    5.4.3.1 Configuring Management access filters......................................................................................................................... 765.4.3.2 Configuring Password management parameters............................................................................................................ 775.4.3.3 Configuring profiles ...................................................................................................................................................... 785.4.3.4 Configuring User access parameters.............................................................................................................................. 795.4.3.5 Configuring RADIUS Authentication ........................................................................................................................... 805.4.3.6 Configuring RADIUS Authorization............................................................................................................................. 815.4.3.7 Configuring VSA when RADIUS Authorization is enabled ......................................................................................... 825.4.3.8 Configuring RADIUS Accounting ................................................................................................................................ 855.4.3.9 Enabling TACACS+ Authentication ............................................................................................................................. 865.4.3.10 Configuring TACACS+ Authorization.......................................................................................................................... 875.4.3.11 Configuring TACACS+ Accounting ............................................................................................................................. 875.4.3.12 Enabling SSH ................................................................................................................................................................ 885.4.3.13 Configuring Login controls ........................................................................................................................................... 88

    5.4.4. SNMP security configuration................................................................................................................................895.4.4.1 SNMP overview............................................................................................................................................................ 895.4.4.2 Which SNMP version to use.......................................................................................................................................... 925.4.4.3 SNMP security configuration components.................................................................................................................... 935.4.4.4 Commands displaying SNMP security configuration.................................................................................................... 94

    5.4.5. User Access failure troubleshooting .....................................................................................................................945.5. VERIFY EVENT &ACCOUNTING LOGS CONFIGURATION .............................................................................................95

    5.5.1. Accounting logging Overview ........... ........... .......... ........... .......... ........... .......... ........... .......... ......... .......... ........... .. 955.5.2. Verifying the logging configurations.....................................................................................................................98

    6. COMMON TROUBLESHOOTING SCENARIOS .....................................................................................................100

    6.1. LAYER 1&LAYER 2PROBLEMS ...............................................................................................................................1006.1.1. How to show Layer 1 & Layer 2 alarms ............. .......... ........... .......... ........... .......... ........... .......... .......... ........... .. 1006.1.2. Verify cards, MDAs and ports configuration ...................................................................................................... 1006.1.3. How to show or clear statistics on a port or a LAG or a SAP.............. ........... .......... ........... .......... ........... ......... .1016.1.4. How to show or modify the operational status of a port ......... ......... ......... .......... ......... ......... .......... ............... ..... 1026.1.5. How to loop ports .......... .......... ........... ........... .......... ........... .......... ........... .......... ........... ......... .......... ........... ......... 102

    6.2. OSPFPROBLEMS......................................................................................................................................................1046.2.1. Commands common to any OSPF troubleshooting.............................................................................................1046.2.2. OSPF not come up...............................................................................................................................................106

    6.3. BGPPROBLEMS........................................................................................................................................................111

    6.3.1. Commands common to any BGP troubleshooting...............................................................................................1116.3.2. BGP peer session not established........... ........... .......... ........... ........... .......... ........... .......... .......... ........... .......... .... 1126.3.3. BGP load balancing issue .......... ........... .......... ........... ........... .......... ........... .......... ........... ........ .......... ........... ....... 115

    6.4. PREFIX-LIST (ACCESS-LIST)IN THE ROUTE POLICY .................................................................................................. 1176.5. BLACK HOLING PROBLEMS .......................................................................................................................................1206.6. LDPNOT ESTABLISHED ............................................................................................................................................1216.7. CPUUTILIZATION HIGH SCENARIO ..........................................................................................................................1226.8. TROUBLESHOOTING IES(INTERNET ENHANCED SERVICE)SERVICES ....................................................................... 1236.9. NETWORK MONITORING ...........................................................................................................................................125

    7. MISCELLANEOUS........................................................................................................................................................128

  • 7/22/2019 7750 Series Troubleshooting Manual

    4/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    4

    TABLESTable 1: Event Severity Levels .................................................................................................................. 14

    Table 2: Valid Filter Policy Operators.......................................................................................................... 18

    Table 3: 7750 SR OS to Syslog Severity Level Mappings .......... ........... .......... ........... .......... ........... .......... .. 22

    Table 4: CLI Commands to Configure Mirroring Parameters ........... .......... ........... ........... .......... ........... ...... 35Table 5: Chassis Front View Features .......................................................................................................... 43

    Table 6: Chassis Rear View Features ........................................................................................................... 44

    Table 7: Console Configuration Parameter Values.............. ........... ........... .......... ........... .......... ........... ......... 46

    Table 8: 7750 SR-12 Hardware Component Operating Requirements........... .......... ........... .......... ........... .... 47

    Table 9: 7750 SR-12 AC Power Supply LED Descriptions ......................................................................... 48

    Table 10: SF/CPM Field Descriptions .......................................................................................................... 51

    Table 11: Index of system configuration verification tasks .......... ........... .......... ........... .......... ........... ........... 60

    Table 12: Configuring Authentication .......................................................................................................... 74

    Table 13: Configuring Authorization......... ........... .......... ........... .......... ........... .......... ........... ........... .......... .... 75

    Table 14: Configuring Accounting ............................................................................................................... 76Table 15: Accounting Record Name and Collection Periods ................................................................ 97

    FIGURES:Figure 1: Event Logging Block Diagram.................. .......... ........... .......... ........... .......... ........... ............ ......... 12

    Figure 2: show log application command output......... ........... .......... ........... .......... ........... ........... ........... ...... 14

    Figure 3: Service Mirroring ......................................................................................................................... 24

    Figure 4: Local mirroring Example .............................................................................................................. 28

    Figure 5: Remote mirroring Example ........................................................................................................... 29

    Figure 6: Service mirror configuration and implementation flow .......... .......... ........... .......... ........... .......... .. 29

    Figure 7: Local Service Mirroring Configuration.............. .......... ........... .......... ........... .......... ........... ............ 31

    Figure 8: Remote Service Mirroring Configuration............. .......... ........... .......... ........... .......... ........... .......... 32

    Figure 9: 7750 SR-12 Chassis Front View .................................................................................................. 43

    Figure 10: 7750 SR-12 Chassis Rear View......... ........... .......... ........... ........... .......... ........... .......... ........... .... 44

    Figure 11: Management Console Port Connection...................................................................................... 45

    Figure 12: Telnet Management Port Connection.......... .......... ........... .......... ........... .......... ........... ............ .... 46

    Figure 13: 7750 SR-12 AC Power Supply LEDs........ ........... ........... .......... ........... .......... ........... .......... ....... 48

    Figure 14: SF/CPM Front Panel .................................................................................................................. 50Figure 15: SNMPv1 and SNMPv2c Configuration and Implementation Flow .................................... 93

    Figure 16: SNMP Configuration Components ......................................................................................... 93

    Figure 17: Alarm relationships on the 5620 SAM GUI .......... ........... .......... ........... .......... ........... .......... ..... 127

  • 7/22/2019 7750 Series Troubleshooting Manual

    5/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    5

    1. Introduction

    1.1. Intended Audience for this Guide

    This document has been written to address the needs of network administrators and networksupport personnel who are on the front-line of diagnosing issues with the Alcatel 7750 SR.Typically, this includes network operations groups within customer organizations, Alcatel 2ndLine

    Support, various Technical Assistance Center (TAC) staff, sales engineers and pre-sales engineers.

    This guide requires knowledge of IP networking technology.

    1.2. How This Document is organized

    This Guide provides an overview of the troubleshooting process and provides a convenient

    description of all the troubleshooting tools that are available on the Alcatel 7750 SR. The Guidethen breaks down troubleshooting by the major hardware components of the router in addition to

    providing guidance to troubleshooting system level, router level and service level configurationissues.

    Troubleshooting Process provides a systematic approach to troubleshooting router problemsthat is based on the categorization of the symptoms of the trouble, the collection of descriptive

    information related to the problem, the analysis of the information to identify potential causesand the resolution through a systemic application of corrective actions.

    Troubleshooting Tools describes the tools and utilities that are used to configure, monitor andtroubleshoot the Alcatel 7750 SR.

    Hardware Operational Status

    describes how to verify the operational status and validate theconfiguration of the hardware components of the Alcatel 7750 SR:

    o SF/CPM

    o IOM

    o MDA

    System Level Configuration Verification describes how to verify the proper configuration ofsystem components such as the Boot Option File, the System Management settings, the router

    security settings and the system settings for the hardware components of the Alcatel 7750 SR.

    Common Troubleshooting Scenarios provides information on troubleshooting problems that

    commonly occur at layer 1& layer 2 (such as IOM, MDA or port level), router level (such asOSPF, BGP or route policy), and other specific scenarios.

  • 7/22/2019 7750 Series Troubleshooting Manual

    6/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    6

    1.3. Where to Begin?

    There are many and various methodologies that are followed to troubleshoot problems, be that aproblem in a network, in a computer, in an application, or even in a car. All methodologies will

    invariably have the same or at least similar actions and goals, these being to identify, characterizeand finally resolve the problem.

    After having established a baseline, the 1st

    step in troubleshooting any node is to start in the "EventLogs" - where the alarms are logged. The Event logs maybe stored locally on the node or remotely

    on a server or on the Alcatel 5620 SAM. Collect all the symptoms you can for the problem nodeas the more information you have to work from, the easier it is to isolate the cause and figure out

    how to resolve the problem. Other information you will probably want to collect includes

    hardware, software and nodal configuration information, equipment and service operating statistics

    and service specific configuration data.

    More detail is on the troubleshooting process is provided in section 2 Troubleshooting Process.

    This guide is based on the hardware and software introduced in the Alcatel 7750 SR R2.0.

    1.4. Related Documents

    Please refer to the following for further information on the Alcatel 7750 SR:

    5620 SRM r1.2 New Feature Training (Service Assurance)- 07NPT0067.E_(Service

    Assurance)_v1.1.ppt

    Alcatel 5620 Service Router Manager R2.0 User Guide- 5620SRM20_UG.pdf

    Alcatel 7750 SR-12 Installation Guide- 7750_SR-12_Installation_Guide_Rev-02.pdf

    Alcatel 7750 SR OS System Guide- 7750_SR_OS_System_Guide_2.0.pdf

    Alcatel 7750 SR OS Services Guide- 7750_SR_OS_Services_Guide_2.0.pdf

    Alcatel 5620 SAM Service Aware Manager R2.0 General Information Book

    Note: The Alcatel 5620 SRM is now known as the Alcatel 5620 SAM

  • 7/22/2019 7750 Series Troubleshooting Manual

    7/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    7

    2. Troubleshooting Process

    Troubleshooting and problem solving is basically the same thing. In either case, there is theacknowledgment that something in the network, be that a component of the network or a servicewithin the network, is not operating within expected operating parameters. The problem can result

    in a total or catastrophic failure in the network, or the problem can manifest itself intermittently, or

    then again, the problem might have resulted in degradation of how the service is performing.

    There are many accepted methodologies for troubleshooting a problem and they all must naturally

    start with the identification that a problem exists. This implies a certain level of understanding ofthe designed state and behavior of a network and the services that are using that network as well as

    an identification of a symptom that the desired behavior is no longer there. This identification can

    come in the form of an alarm received from a network component, through the analysis of networkcapacity and performance data or even from a call from a customer reporting a problem with their

    service.

    The basis for effective troubleshooting is in having a well understood baseline for the network andservices, a detailed knowledge of the elements of the network, from transport to routing, a

    thorough understanding of the services and how they operate, and finally, a degree of expertise inthe use of troubleshooting tools that are available in the network elements and the network

    management systems. These elements are discussed in more details in the following sections of

    this guide.

    2.1. Establishing a Baseline

    Having a thorough knowledge of your network and how it functions under normal conditions isessential if you want to be efficient in troubleshooting problems as it allows for rapid and easyidentification that a fault exists in your network. It is therefore essential that a sound baseline of

    your network and services be established and rigorously maintained since a network is never astatic environment. Customer churn, new service introductions, new service points of presence areadded, links fail, etc

    How detailed should that baseline be? That depends on how much time and money you want to

    invest in establishing the baseline, on the level of expertise and degree of experience your

    operations staff has and on how good the fault management capabilities are in your networkmanagement system. Establishing a baseline typically includes:

    Creating Network Configuration Documentation

    Create End-System Networking Configuration Documentation

    Periodically backing up router running configurations

    Storing the backups at a safe, off-site location

    Documenting service descriptions and service SLAs

    Collecting and understanding statistics on traffic flows, router and trunk utilization levels

    Document customer profiles, customer contact numbers

  • 7/22/2019 7750 Series Troubleshooting Manual

    8/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    8

    Document the General Troubleshooting Process

    Maintaining a detailed history of problems, their symptoms, how the root cause was identified andhow the problem was resolved is also a powerful tool towards efficient troubleshooting. Your

    problem tracking system should maintain a history of network and service problems and their

    resolution and include details such as:

    Problem symptoms

    Associated alarms and network event messages

    Network conditions, such as link failures, congestion, packet discards

    Type, version and configuration of hardware and software for the affected networkelements

    Description of service impacts

    Results of any corrective actions

    Problem resolution

    2.2. Characterize the Problem

    A computer network, such as the Internet, is considered to be a well defined system whose state

    and expected behavior can be well defined and documented. The goal in troubleshooting well

    defined systems is to return the system to the as-designed behavior state. The first step in

    returning the system to its design intent is to fully characterize the problem state.

    Part of characterizing problems is differentiating between total failures and problems that result ina degradation in performance. For a customer that has a single DS3 link into the network, a failure

    of the access router results in a total failure for that customer. A core router operating above 80%average utilization will start to discard packets which will result in a degradation of performance

    for at least certain applications running through that router. Performance degradations will exhibit

    greatly different symptoms from total failures and may not generate alarms or significant networkevents.

    Multiple problems can and often will happen at the same time and can manifest same, related orcompletely different symptoms. It is therefore critical when identifying symptoms that as many

    characterizing parameters be collected from the network as possible including:

    Alarm files

    Error logs

    Network statistics

    Network analyzer traces

    Core dumps

  • 7/22/2019 7750 Series Troubleshooting Manual

    9/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    9

    Serial line traces

    Stack dumps

    Output of various show commands in CLI (current configuration)

    Accounting logs

    Customer trouble reports

    The more detailed the documented symptoms, the easier it is to identify the root cause of the

    problem. It is important to remember that in many cases the individual or the team that is

    recording the problem symptoms may not be the same people who will be finding the root causeand resolving the problem, therefore close attention to detail in recording the problem symptoms is

    crucial to rapid problem resolution.

    Alarms can be viewed directly from the 7750 SR node alarm file or through the use of the fault

    management features available in the 5620 SAM. The 5620 SAM converts SNMP traps fromnetwork routers to events and alarms which can be easily correlated against the appropriatemanaged equipment and configured services and policies.

    Some questions to answer and conditions to investigate when characterizing the problem are:

    Is it an intermittent problem, or is the problem static in nature?

    If the problem is intermittent, how often has it happened, is there a pattern?

    What alarms or network events are associated with the problem?

    Can you identify any congestion in routers or network links?

    Identify and record any changes that have taken place since the network was lastfunctioning properly.

    2.3. Identify the Root Cause

    As mentioned, a particular symptom can be the result of more than one network problem.Successfully troubleshooting a problem state therefore involves the identification of the root cause

    of each and every individual cause of the problem state. It is entirely possible to fix the problem

    by trying a variety of actions, such as resetting a network link, rebooting a router, reseating an IOmodule, in the general case the intended solution will be arrived at more rapidly by following a

    systematic approach to troubleshooting. A systematic approach to identifying the root cause of the

    problem includes the following elements:

    Once the symptoms have been identified and thoroughly documented, first try to identify ifthey have anything in common and focus on the common stuff first and work out fromthere.

  • 7/22/2019 7750 Series Troubleshooting Manual

    10/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    10

    Alarms available through the 5620 SAM contain vendor-specific and X.733 standardizedprobable cause that can be very useful in identifying the root cause.

    Statistics on alarms available from the 5620 SAM tell you how often an alarm has beenraised based on specified scenarios that can be helpful in identifying the root cause of a

    problem.

    If the symptoms are present in different areas of the network try to identify what iscommon across these areas.

    Work on one problem at a time, fix that problem, then move on to the next.

    Divide the problem space into natural segments and try to isolate the problem to one of thesegments. One way of segmenting the network is:

    o LAN switching (edge access).

    o LAN routing (distribution, core).

    o Metropolitan-area networks.

    o WAN (national backbone).

    o Partner services (extranet).

    o Remote access services.

    Try to determine the precise network state that existed before the problem appeared.

    Identify which specific functions are not working properly and focus on those.

    Extrapolate from the network alarms and network events what conditions could result in

    the observed symptoms. Test for these to see if the problem can be reproduced.

    2.4. Plan your actions & Resolve the Problem

    The actions you take will depend on the type of problem that you are trying to resolve. Criticalproblems that are affecting a wide range of services for a large number of gold service level

    customers require a different tact from minor problems affecting a small number of best-effort

    service customers. The former situation will by necessity require drastic and immediate actions to

    restore service while the latter can afford to take a little more time to ensure that the actions willnot put any other services at risk. The key is to balance the risk of creating further service

    interruptions while attempting to restore service in the shortest possible timeframe. Whatever

    corrective action is planned, you should:

    Reproduce the symptom

    Document each step of the corrective action

    Test the corrective action

  • 7/22/2019 7750 Series Troubleshooting Manual

    11/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    11

    Use CLI to verify behavior changes for each step

    The next step after testing your hypothesis and verifying that the corrective action is going tocorrect the problem and not introduce any new symptoms is to apply the corrective action to the

    live network. When doing so, it is recommended to resolve the easiest problem, in terms of risk,

    effort and time, first.

    2.5. Verify Solutions

    After having taken corrective action to resolve the problem it is important to verify that thechanges have not introduced new symptoms and that the original problem has been completed

    corrected. If new symptoms are detected or if the problem has only been mitigated, you need to

    start the troubleshooting process again.

  • 7/22/2019 7750 Series Troubleshooting Manual

    12/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    12

    3. Trouble shooting tools

    3.1. Event logs

    Event logs are the means of recording system generated events for later analysis. Should there be afault within a 7750 SR system, event logs are the means for troubleshooting. Events are messagesgenerated by the system by applications or processes within the 7750 SR.

    3.1.1. Event logging overview

    7750 SR OS supports event logging. Event logging controls the generation, dissemination andrecording of system events for monitoring status and troubleshooting faults within the system. The

    logging:

    Provides you with logging information for monitoring and troubleshooting.

    Allows you to select the types of logging information to be recorded. Allows you to assign a severity to the log messages.

    Allows you to select the source and destination of logging information.

    Event logs are the means of recording system generated events for later analysis. Events are

    messages generated by the system by applications or processes within the 7750 SR.

    Figure 1depicts a function block diagram of event logging.

    Figure 1: Event Logging Block Diagram

  • 7/22/2019 7750 Series Troubleshooting Manual

    13/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    13

    3.1.1.1 Event Sources

    The event sources are the main categories of events that feed the log manager. The 7750 SR

    groups events into four major categories.

    Security events - Events that pertain to attempts to breach system security. The securityevent source is all events that affect attempts to breach system security such as failed login

    attempts, attempts to access MIB tables to which the user is not granted access or attemptsto enter a branch of the CLI to which access has not been granted. Security events are

    generated by the SECURITY application.

    Change events - Events that pertain to the configuration and operation of the node. Thechange activity event source is all events that directly affect the configuration or operation

    of the node. Change events are generated by the USER application.

    Debug-trace events - Debug and trace messages that have been enabled for applications orprocesses. The debug event source is all debugging and trace messages that have been

    enabled on the system. Debug events are generated by the DEBUG application.

    Main events - Events that pertain to 7750 SR OS applications that are not assigned to otherevent categories/sources.

    Examples of applications within 7750 SR OS include IP, MPLS, OSPF, CLI, services, etc. Figure

    2displays the show log applicationscommand output which displays all applications.

  • 7/22/2019 7750 Series Troubleshooting Manual

    14/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    14

    Figure 2: show log application command output

    3.1.1.2 Event Control

    Event control pre-processes the events generated by applications before the event is passed into themain event stream. Event control assigns the severity for each application event and whether theevent should be generated or suppressed. The severity numbers and severity names supported in

    7750 SR OS conform to ITU standards M.3100 X.733 & X.21 and are listed in Table 1.

    Table 1: Event Severity Levels

  • 7/22/2019 7750 Series Troubleshooting Manual

    15/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    15

    Events that are suppressed by event control will not generate any event log entries as it never

    reaches the log manager. Event control maintains a count of the number of events generated

    (logged) and dropped (suppressed) for each application event. The severity of an application eventcan be configured in event control.

    Application events contain an event number and description that explains why the event is

    generated. The event number is unique within an application, but the number can be duplicated inother applications.

    The following example, generated by querying event control for application events, displays apartial list of event numbers and names.

  • 7/22/2019 7750 Series Troubleshooting Manual

    16/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    16

  • 7/22/2019 7750 Series Troubleshooting Manual

    17/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    17

    3.1.1.3 Log manager

    Events that are forwarded by event control are sent to the log manager. The log manager managesthe event logs in the system and the relationships between the log sources, event logs and log

    destinations, and log filter policies.

    An event log has the following properties:

    A unique log ID

    The log ID is a short, numeric identifier for the event log.

    One or more log sources

    The source stream or streams to be sent to log destination can be specified. The source

    must be identified before the destination can be specified. The events can be from the main

    event stream, events in the security event stream, events in the user activity stream, or alldebug-trace messages in the debug stream.

    One event log destination

    A log can only have a single destination. The destination for the log ID destination can be

    one of console, session, syslog, snmp-trap-group, memory, or a file on the local file system.

    An optional event filter policy

    An event filter policy defines whether to forward or drop an event or trap based on match criteria.

    3.1.1.4 Event Filter Policies

    The log manager uses event filter policies to allow fine control over which events are forwarded ordropped based on various criteria. Filter policies have a default action. The default actions are to

    either:

    Forward

    Drop

    Filter policies also include a number of filter policy entries that are identified with an entry ID anddefine specific match criteria and a forward or drop action for the match criteria.

    Each entry contains a combination of matching criteria that define the application, event number,

    severity, and subject conditions. The entrys action determines how the packets should be treated ifthey have met the match criteria.

    Entries are evaluated in order from the lowest to the highest entry ID. The first matching event issubject to the forward or drop action for that entry.

  • 7/22/2019 7750 Series Troubleshooting Manual

    18/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    18

    Valid operators are displayed in Table 2:

    Table 2: Valid Filter Policy Operators

    A match criteria entry can include combinations of: Equal to or not equal to a given system application.

    Equal to, not equal to, less than, less than or equal to, greater than or greater than or equalto an event number within the application.

    Equal to, not equal to, less than, less than or equal to, greater than or greater than or equalto a severity level.

    Equal to or not equal to an event subject string.

    The following example shows the event filter policies configured on a 7750 SR.

  • 7/22/2019 7750 Series Troubleshooting Manual

    19/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    19

    3.1.1.5 Log Destinations

    An event log within 7750 SR OS associates the event sources with logging destination. 7750 SROS supports the following log destinations:

    Console

    Session

    Memory logs

    Log files

    SNMP trap group

    Syslog

    Only a single log destination can be associated with an event log or with an accounting log. An

    event log can be associated with multiple event sources, but it can only have a single logdestination.

    A file destination is the only type of log destination that can be configured for an accounting log.

    Console

    Sending events to a console destination means the message will be sent to all active consolesessions. If there are no active console sessions, the event log entries are dropped. The console

    device can be used as an event log destination.

    Session

    A session destination is a temporary log destination which directs entries to the active consolesession for the duration of the console session. When the session is terminated, the event log is

    removed. Event logs with a session destination are not stored in the configuration file. Event logscan direct log entries to the session destination.

  • 7/22/2019 7750 Series Troubleshooting Manual

    20/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    20

    Memory Logs

    A memory log is a circular buffer. When the log is full, the oldest entry in the log is replaced withthe new entry. When a memory log is created, the specific number of entries it can hold can bespecified, otherwise it will assume a default value. An event log can send entries to a memory log

    destination.

    Default System Log

    Log 99 is a pre-configured memory-based log which logs from the main event source (not security,

    debug/trace, etc.). Log 99 exists by default.

    The following example displays the log 99 configuration.

    Log Files

    Log files are stored on the compact flash devices (specifically cf1 or cf2) in the 7750 SR file

    system.

    A log file is identified with a single log file ID, but a log file will generally be composed of anumber individual files in the file system. A log file is configured with a rollover parameter which

    determines how long in minutes an individual file which is a component of the log file should be

    written to before a new file is created for the log file ID.

    The retention time for a log file specifies the amount of time the file should be retained on the

    system based on the creation date and time of the file. The retention time is used as a factor to

    determine which files should be deleted first if the file system device nears 100% usage.

    One log file can only be attached to one log ID.

    When a log file is created, only the compact flash device for the log file is specified. Log files are

    created in specific subdirectories with standardized names depending on the type of information

    stored in the log file.

    Event log files are always created in the \log directory on the specified compact flash device.

  • 7/22/2019 7750 Series Troubleshooting Manual

    21/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    21

    SNMP Trap Group

    An event log can be configured to send events to SNMP trap receivers by specifying an SNMPtrap group destination.

    An SNMP trap group can have multiple trap-receivers with different trap destinations. Each trap

    receiver can have different operational parameters.

    A trap destination has the following properties:

    The IP address of the trap receiver.

    The UDP port used to send the SNMP trap. SNMP version (v1, v2c, or v3) used to format the SNMP notification.

    SNMP community name for SNMPv1 and SNMPv2c receivers.

    Security name and level for SNMPv3 trap receivers.

    For SNMP traps that will be sent out-of-band through the Management Ethernet port on the SF/

    CPM, the source IP address of the trap is the IP interface address defined on the Management

    Ethernet port. For SNMP traps that will be sent in-band, the source IP address of the trap is thesystem IP address of the 7750 SR.

    Each trap destination of a trap group receives the identical sequence of events as defined by the logID and the associated sources and log filter applied.

    Syslog

    An event log can be configured to send events to one syslog destination. Syslog destinations havethe following properties:

    Syslog server IP address.

    The UDP port used to send the syslog message. The Syslog Facility Code (0 - 23) (default 23 - local7). The Syslog Severity Threshold (0 - 7) - events exceeding the configured level will be

    sent.

    Because syslog uses eight severity levels whereas the 7750 SR OS uses six internal severity levels,

    the 7750 SR OS severity levels are mapped to syslog severities. Table 3displays the 7750 SR OS

    severity level mappings to syslog severities.

  • 7/22/2019 7750 Series Troubleshooting Manual

    22/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    22

    Table 3: 7750 SR OS to Syslog Severity Level Mappings

    3.1.2. List of show commands for event logging

    Information to view show commands

    Displays a list of all application namesthat can be used in event-control and

    filter commands.

    show log applications

    Displays event control settings for eventsincluding whether the event is suppressedor generated and the severity level for the

    event.

    show log event-control [application [event-name |event-number]]

    Displays event file log information. show log file-id [file-id]

    Displays event log filter policyinformation.

    show log filter-id [filter-id]

    Show log collector statistics for the main,

    security, change and debug logcollectors.

    show log log-collector

    Displays an event log summary withsettings and statistics or the contents of a

    specific log file, SNMP log, or memory

    log.

    show log log-id [log-id] [severity severity-level]

    [application application] [sequencefrom-seq [toseq]][count number] [subject subject] [ascending |

    descending]

    configure log log-id [log-id]

  • 7/22/2019 7750 Series Troubleshooting Manual

    23/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    23

    log-id# info detail

    Displays SNMP trap group configuration

    information.show log snmp-trap-group [log-id]

    Displays syslog event log destinationsummary information or detailed

    information on a specific syslog

    destination.

    show log syslog [syslog-id]

    3.2. Service mirroring

    3.2.1. Service mirroring overview

    When troubleshooting complex operational problems, customer packets can be examined as theytraverse the network. One way to accomplish this is with an overlay of network analyzers

    established at multiple PoPs, together with skilled technicians to operate them to decode the dataprovided. This method of traffic mirroring often requires setting up complex filters in multiple

    switches and/or routers. These, at best, are only able to mirror from one port to another on the

    same device.

    Alcatels Service Mirroring extends and integrates these capabilities into the network and provides

    significant operational benefits. Each 7750 SR can mirror packets from a specific service to any

    destination point in the network, regardless of interface type or speed.

    Alcatels 7750 SR routers support service-based mirroring. While some Layer 3 switches and

    routers can mirror on a per-port basis within the device, Alcatel 7750 SR routers can mirror on ann-to-1 unidirectional service basis and re-encapsulate the mirrored data for transport through the

    core network to another location, using either IP or MPLS tunneling as required Figure 3).

    Original packets are forwarded while a copy is sent out the mirrored port to the mirroring(destination) port. Service mirroring allows an operator to see the actual traffic on a customers

    service with a sniffer sitting in a central location. In many cases, this reduces the need for a

    separate, costly overlay sniffer network.

    The mirrored frame size that is to be transmitted to the mirror destination can be explicitly

    configured by using slicing features. This enables mirroring only the parts needed for analysis. Forexample, only the headers can be copied for analysis, protecting the integrity and security of

    customer data, or conversely, copying the full packet, including customer data.

    Service mirroring is supported on any interface type and on mixed interface types. For example, a

    service that uses only Ethernet service interfaces can be mirrored to a SONET/SDH network port,

    transported across the core network and delivered on either Ethernet or SONET/SDH egress ports

    at the location where service analysis is performed. The packet traffic is uninterrupted and packetsflow normally through the mirrored port.

  • 7/22/2019 7750 Series Troubleshooting Manual

    24/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    24

    Figure 3: Service Mirroring

    3.2.2. Mirror implementation

    Mirroring can be implemented on ingress or egress service access points (SAPs) or ingress andegress network interfaces. The Flexible Fast Path processing complexes preserve the ingress

    packet throughout the forwarding and mirroring process, making incremental packet changes on aseparate copy.

    Alcatels implementation of packet mirroring is based on two assumptions:

    Ingress and egress packets are mirrored as they appear on the wire. This is important for

    troubleshooting encapsulation and protocol issues.

    o When mirroring at ingress, the Flexible Fast Path network processor array (NPA) sendsan exact copy of the original ingress packet to the mirror destination while normal

    forwarding proceeds on the original packet.

    o When mirroring is at egress, the NPA performs normal packet handling on the egresspacket, encapsulating it for the destination interface. A copy of the forwarded packet

    (as seen on the wire) is forwarded to the mirror destination.

    Mirroring must support tunnel destinations.

    o Remote destinations are reached by encapsulating the ingress or egress packet within anSDP, like the traffic for distributed VPN connectivity services. At the remote

    destination, the tunnel encapsulation is removed and the packet is forwarded out a local

    SAP.

  • 7/22/2019 7750 Series Troubleshooting Manual

    25/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    25

    3.2.2.1 Mirror Source and Destinations

    Mirror sources and destinations have the following characteristics:

    They can be on the same 7750 SR router (local) or on two different routers (remote).

    Mirror destinations can terminate on egress virtual ports which allow multiple mirrordestinations to send to the same packet decode device, delimited by IEEE 802.1Q (referred

    to as dot1q) tags. This is helpful when troubleshooting a multi-port issue within thenetwork.

    When multiple mirror destinations terminate on the same egress port, the individual dot1q

    tags can provide a DTE/DCE separation between the mirror sources.

    Packets ingressing a port can have a mirror destination separate from packets egressinganother or the same port (the ports can be on separate nodes).

    A total of 255 mirror destinations are supported (local and/or remote), on a per chassisbasis.

    The mirror egress port (local or remote) can be PoS or Ethernet. If an Ethernet frame is mirrored toa PoS port, the frame is translated to PPP/BCP encapsulation. If a PoS frame is mirrored to an

    Ethernet port, the frame is translated to PPPoE encapsulation. This allows the use of PoS or

    Ethernet packet decode devices.

    Local and Remote Mirroring

    Mirrored frames can be copied and sent to a specific local destination or service on the 7750 router(local mirroring) or copies can be encapsulated and sent to a different 7750 SR router (remotemirroring). This functionality allows network operators to centralize not only network analyzer

    (sniffer) resources, but also the technical staff who operate them.

    The 7750 SR allows multiple concurrent mirroring sessions so traffic from more than one ingressmirror source can be mirrored to the same or different egress mirror destinations.

    Remote mirroring uses a service distribution path (SDP) which acts as a logical way of directing

    traffic from one SR-Series router to another through a uni-directional (one-way) service tunnel.

    The SDP terminates at the far-end 7750 SR which directs packets to the correct destination on that

    device.

    The SDP configuration from the mirrored device to a far-end 7750 SR requires a return path SDP

    from the far-end 7750 SR back to the mirrored router. Each device must have an SDP defined forevery remote router to which it wants to provide mirroring services. SDPs must be created first,

    before services can be configured.

  • 7/22/2019 7750 Series Troubleshooting Manual

    26/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    26

    Encapsulation Translation

    Service mirroring can also map frames from a monitored service to another endpoint using adifferent encapsulation type at the mirror destination. For example, a service using PPP over

    Packet over SONET/SDH can have its traffic mirrored to an Ethernet port destination with anEthernet-attached analyzer. The 7750 SR router translates the PPP header into a PPPoE header so

    the Ethernet-attached analyzer can properly decode the frames.

    The automatic translation of PPP or Ethernet frames into PPPoE or BCP encapsulations can bemanually disabled. The type of translation depends on the type of the destination SDP or SAP

    defined for the mirror destination. Translation is important to allow PoS packet-decoding devices

    to receive Ethernet frames or Ethernet packet-decoding devices to receive PPP frames.

    When translating an Ethernet frame for transmission to a SONET/SDH SAP or SDP, the Ethernet

    frame gets encapsulated in a PPP/BCP frame format. When translating a SONET/SDH PPP frame

    for transmission to an Ethernet SAP or SDP, the PPP frame gets encapsulated in a PPPoE frameformat.

    Slicing

    A further service mirroring refinement is slicing which copies a specified packet size of eachframe. This is useful to monitor network usage without having to copy the actual data. Slicing

    enables mirroring larger frames than the destination packet decode equipment can handle. It also

    allows conservation of mirroring resources by limiting the size of the stream of packet through the

    7750 SR and the core network.

    When a mirror slice-sizeis defined, a threshold that truncates a mirrored frame to a specific sizeis created. For example, if the value of 256 bytes is defined, up to the first 256 bytes of the frame

    are transmitted to the mirror destination. The original frame is not affected by the truncation.Mirrored frames, most likely, will grow larger as encapsulations are added when packets are

    transmitted through the network core or out the mirror destination SAP to the packet/protocol

    decode equipment.

    The transmission of a sliced or non-sliced frame is also dependent on the mirror destination SDP

    path MTU and/or the mirror destination SAP physical MTU. Packets that require a larger MTUthan the mirroring destination supports are discarded if the defined slice size does not truncate the

    packet to an acceptable size.

  • 7/22/2019 7750 Series Troubleshooting Manual

    27/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    27

    3.2.2.2 Mirroring performance

    Replication of mirrored packets can, typically, affect performance and should be used carefully.

    Alcatel 7750 SR routers minimize the impact of mirroring on performance by taking advantage of

    its distributed Flexible Fast Path technology. Flexible Fast Path forwarding allows efficient mirrorservice scaling and, at the same time, allows a large amount of data to be mirrored with minimal

    performance impact. When a mirror destination is configured, the packet slice option can truncate

    mirrored packets to the destination, which minimizes replication and tunneling overhead. Themirroring architecture also supports mirror rate limiting both at the ingress and egress Flexible Fast

    Path NPA. This rate limiting is accomplished through a shaping queue and is settable according to

    the maximum amount of mirroring desired.

    Mirroring can be performed based on the following criteria:

    Port

    SAP MAC filter

    IP filter

    Ingress label

    3.2.3. Mirroring configuration

    Configuring mirroring is similar to creating a uni-directional service. Mirroring requires theconfiguration of:

    Mirror source - the traffic on a specific point(s) to mirror.

    Mirror destination - the location to send the mirrored traffic, where the sniffer will belocated.

    Figure 4depicts a local mirror service configured on SR A.

    Port 2/1/2 is specified as the source. Mirrored traffic ingressing and egressing this port willbe sent to port 2/1/3.

    SAP 2/1/3 is specified as the destination. The sniffer is physically connected to this port.Mirrored traffic ingressing and egressing port 2/1/2 is sent here. SAP, encapsulation

    requirements, packet slicing, and mirror classification parameters are configured. SDPs arenot used in local mirroring.

  • 7/22/2019 7750 Series Troubleshooting Manual

    28/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    28

    Figure 4: Local mirroring Example

    Figure 5depicts a remote mirror service configured as SR B as the mirror source and SR A as the

    mirror destination. Mirrored traffic ingressing and egressing port 5/2/1 (the source) on SR B ishandled the following ways:

    Port 5/2/1 is specified as the mirror source port. Parameters are defined to select specifictraffic ingressing and egressing this port.

    Destination parameters are defined to specify where the mirrored traffic will be sent. In thiscase, mirrored traffic will be sent to a SAP configured as part of the mirror service on port

    3/1/3 on SR A (the mirror destination).

    SR A decodes the service ID and sends the traffic out of port 3/1/3.

    The sniffer is physically connected to this port (3/1/3). SAP, encapsulation requirements,packet slicing, and mirror classification parameters are configured in the destination

    parameters.

  • 7/22/2019 7750 Series Troubleshooting Manual

    29/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    29

    Figure 5: Remote mirroring Example

    3.2.3.1 Mirror configuration process overview

    Figure 6displays the process to provision basic mirroring parameters.

    Figure 6: Service mirror configuration and implementation flow

    3.2.3.2 Mirror configuration components

    The example below demonstrates the major components to configure service mirroring.

  • 7/22/2019 7750 Series Troubleshooting Manual

    30/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    30

    Mirror destination Sets up a service which allows the mirrored packets to be directed locally

    or over the core of the network and have a far end 7750 SR decode the mirror encapsulation. The

    service ID must match in the mirror-destination and the mirror-source context.

    SAP (mirror destination) Creates a service access point (SAP), which defines the port and

    encapsulation parameters to which the mirrored source packets are sent. The sniffer is physically

    connected to this port.

    SDP For remote mirrored service. Binds an existing (mirror) service distribution path (SDP)

    to the mirror destination service ID to transport the source mirrored traffic to the destination.

    Remote source For remote mirrored services. Specifies the remote (source) SR allowed to

    mirror traffic to this device for mirror service egress.

    Mirror source Configures packet mirroring match criteria for a mirror destination service. The

    same mirror destination service ID and the mirror source service ID must be configured.

    Port A packet mirroring option which defines ingress and/or egress traffic monitoring by port.

    SAP (mirror source) A packet mirroring option which defines ingress and/or egress trafficmonitoring by SAP defined by the port-id:encap-val or portid.channel-

    id:encap-val.

    IP filter A packet mirroring option which specifies that packets matching the IP filter are

    mirrored to a mirror destination.

    MAC filter A packet mirroring option which specifies that packets matching the MAC filterare mirrored to a mirror destination.

    Ingress label A packet mirroring option which defines packets with a specific MPLS label to amirror destination.

    3.2.3.3 Basic mirror configuration Example

    Local Service mirroring configuration

    Each local mirrored service (within the same router) requires the following configurations:

    1. Specify mirror destination (SAP, SDP).

    2. Specify mirror source (port, SAP, SDP, IP filter, MAC filter, ingress label).

    Note that the mirror source and mirror destination components must be configured under the sameservice ID context.

  • 7/22/2019 7750 Series Troubleshooting Manual

    31/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    31

    Figure 7: Local Service Mirroring Configuration

    The following example displays a sample configuration for Figure 7 of a local mirrored service

    where the source and destinations are on the same SR (SR1).

    SRA>config>mirror# info----------------------------------------------

    mirror-dest 103 createsap 2/1/3:0 create

    egressqos 1

    exitexit

    no shutdownexit----------------------------------------------SRA>config>mirror#

    The following displays the mirror source configuration:

    SRA>debug>mirror-source# show debug mirrordebug

    mirror-source 103port 2/1/2 egress ingressno shutdown

    exitexitSR1>debug>mirror-source# exit

    Remote Service mirroring configuration

    Each remote mirrored service (across the network core) requires the following configurations:

    1. Define the remote destination (SDP)2. Identify the remote source(the device allowed to mirror traffic to this device)

  • 7/22/2019 7750 Series Troubleshooting Manual

    32/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    32

    3. Specify the mirror destination(SAP)

    4. Specify mirror source(port, SAP, SDP, IP filter, MAC filter, ingress label)

    Note that the mirror source and mirror destination components must be configured under the sameservice ID context.

    Figure 8: Remote Service Mirroring Configuration

    The following example displays a sample configuration of a remote mirrored service for Figure 8where the source is a port on SRB and the destination is a SAP on SRA.

    SRB>config>mirror# info----------------------------------------------

    mirror-dest 1000 createsdp 2 egr-svc-label 7000no shutdown

    exit----------------------------------------------SRB>config>mirror# exit allSRB# show debugdebug

    mirror-source 1000

    port 5/2/1 egress ingressno shutdown

    exitexitSRB#

    SRA>config>mirror# info----------------------------------------------

    mirror-dest 1000 createremote-source

    far-end 10.10.10.104 ing-svc-label 7000exitsap 3/1/3:0 create

    egressqos 1

    exitexitno shutdown

    exit----------------------------------------------SRA>config>mirror#

  • 7/22/2019 7750 Series Troubleshooting Manual

    33/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    33

    3.2.3.4 Mirror configuration Notes

    This section describes limitations or notes regarding mirroring configuration.

    Up to 255 mirroring service IDs may be created within a single system.

    A mirrored source can only have one destination.

    The destination mirroring service IDs and service parameters are persistent between router

    (re)boots and are included in the configuration saves.

    The source packet mirroring enabling criteria defined in debug mirror mirror-source

    commands are not preserved in configuration saves.

    Physical layer problems such as collisions, jabbers, etc., are not mirrored. Typically, onlycomplete packets are mirrored. An exception to this is that packets with CRC errors are

    mirrored. Complete stats are available on the interface for these physical layer problems.

    SONET ports or channels in access mode and with frame-relay encapsulation types cannotbe mirrored.

    Either LAG ports or LAG port members can be mirrored. If a LAG port member is beingmirrored, then the LAG port cannot be mirrored and vice-versa.

    Clear channel ports (TDM or SONET) that are being mirrored cannot be channelized untilthe mirroring is disabled.

    Encap type on an access port/channel can not be changed to frame-relay if it is beingmirrored.

    Starting and shutting down mirroring:

    Mirror destinations:

    The default state for a mirror destination service ID is shutdown. You must issue ano shutdown command to enable the feature.

    When a mirror destination service ID is shutdown, mirrored packets associated with theservice ID are not accepted from its mirror source or remote source 7750 SR router.The associated mirror source is put into an operationally down mode. Mirrored packets

    are not transmitted out the SAP or SDP. Each mirrored packet is silently discarded. If

    the mirror destination is a SAP, the SAPs discard counters are incremented.

    Issuing the shutdown command causes the mirror destination service or its mirrorsource to be put into an administratively down state. Mirror destination service IDsmust

    be shut down first in order to delete a service ID, SAP, or SDP association from the

    system.

    Mirror sources:

    The default state for a mirror source for a given mirror-dest service ID is no

    shutdown. You must enter a shutdown command to deactivate (disable) mirroringfrom that mirror-source.

    Mirror sources do not need to be shutdown to remove them from the system. When amirror source is shutdown, mirroring is terminated for all sources defined locally for

    the mirror destination service ID.

  • 7/22/2019 7750 Series Troubleshooting Manual

    34/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    34

    3.2.3.5 List of CLI commands to configure Mirroring parameters

    Table 4lists all the configuration commands to configure 7750 SR mirroring parameters,

    indicating the configuration level at which each command is implemented with a short command

    description. The command list is organized in the following task-oriented manner:

    Configure mirror destination parameters

    Configure mirror source parameters Configure an SDP

  • 7/22/2019 7750 Series Troubleshooting Manual

    35/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    35

    Table 4: CLI Commands to Configure Mirroring Parameters

  • 7/22/2019 7750 Series Troubleshooting Manual

    36/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    36

    Show command

    show mirror mirror-dest [service- id] Displays mirror configuration and operationinformation.

    3.3. OA&M commands for troubleshooting

    Proper delivery of services requires a number of operations occur properly and at different levelsin the service delivery model. For example, operations such as the association of packets to a

    service, VC-labels to a service and each service to a service tunnel must be performed properly inthe forwarding plane for the service to function properly. In order to verify that a service is

    operational, a set of in-band, packet-based OAM tools is required, with the ability to test each of

    the individual packet operations.

    For in-band testing, the OAM packets closely resemble customer packets to effectively test the

    customers forwarding path, but they are distinguishable from customer packets so they are kept

    within the service providers network and not forwarded to the customer.

    The 7750 SR OS suite of OAM diagnostics supplement the basic IP ping and traceroute operations

    with diagnostics specialized for the different levels in the service delivery model. There are

    diagnostics for MPLS LSPs, SDPs, Services and VPLS MACs within a service.

    3.3.1. LSP Diagnostics

    The 7750 SR OS LSP diagnostics are implementations of LSP ping and LSP traceroute based onInternet Draft draft-ietf-mpls-lsp-ping-02.txt. LSP ping, as described in the draft, provides a

    mechanism to detect data plane failures in MPLS LSPs. LSP ping and LSP traceroute are modeledafter the ICMP echo request/reply used by ping and traceroute to detect and localize faults in IP

    networks.

    For a given FEC, LSP ping verifies whether the packet reaches the egress label edge router (LER),

    while in LSP traceroute mode, the packet is sent to the control plane of each transit label switched

    router (LSR) which performs various checks to see if it is actually a transit LSR for the path.

    3.3.2. SDP Diagnostics

    The 7750 SR OS SDP diagnostics are SDP Ping and SDP MTU Path Discovery.

    SDP Ping

    SDP Ping performs in-band uni-directional or round-trip connectivity tests on SDPs. The SDP

    Ping OAM packets are sent in-band, in the tunnel encapsulation, so it will follow the same path astraffic within the service. The SDP Ping response can be received out-of-band in the control plane,

    or in-band using the data plane for a round-trip test.

    For a unidirectional test, SDP Ping tests:

  • 7/22/2019 7750 Series Troubleshooting Manual

    37/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    37

    Egress SDP ID encapsulation

    Ability to reach the far-end IP address of the SDP ID within the SDP encapsulation

    Path MTU to the far-end IP address over the SDP ID Forwarding class mapping between the near-end SDP ID encapsulation and the far-end

    tunnel termination

    For a round-trip test, SDP Ping uses a local egress SDP ID and an expected remote SDP ID. SinceSDPs are unidirectional tunnels, the remote SDP ID must be specified and must exist as a

    configured SDP ID on the far-end 7750 SR. SDP round trip testing is an extension of SDP

    connectivity testing with the additional ability to test:

    Remote SDP ID encapsulation

    Potential service round trip time Round trip path MTU

    Round trip forwarding class mapping

    SDP MTU Path Discovery

    In a large network, network devices can support a variety of packet sizes that are transmittedacross its interfaces. This capability is referred to as the Maximum Transmission Unit (MTU) of

    network interfaces. It is important to understand the MTU of the entire path end-to-end whenprovisioning services, especially for virtual leased line (VLL) services where the service must

    support the ability to transmit the largest customer packet.

    The Path MTU Discovery tool provides a powerful tool that enables service provider to get the

    exact MTU supported between the service ingress and service termination points (accurate to onebyte).

    3.3.3. Service Diagnostics

    Alcatels Service Ping feature provides end-to-end connectivity testing for an individual service.

    Service Ping operates at a higher level than the SDP diagnostics in that it verifies an individualservice and not the collection of services carried within an SDP.

    Service Ping is initiated from a 7750 SR router to verify round-trip connectivity and delay to the

    far-end of the service. Alcatels implementation functions for both GRE and MPLS tunnels andtests the following from edge-to-edge:

    Tunnel connectivity VC label mapping verification

    Service existence

    Service provisioned parameter verification Round trip path verification

    Service dynamic configuration verification

  • 7/22/2019 7750 Series Troubleshooting Manual

    38/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    38

    3.3.4. VPLS MAC Diagnostics

    While the LSP ping, SDP ping and Service ping tools enable transport tunnel testing and verifywhether the correct transport tunnel is used, they do not provide the means to test the learning and

    forwarding functions on a per-VPLS-service basis.

    It is conceivable, that while tunnels are operational and correctly bound to a service, an incorrect

    Forwarding Information Base (FIB) table for a service could cause connectivity issues in theservice and not be detected by the ping tools. Alcatel has developed VPLS OAM functionality tospecifically test all the critical functions on a per-service basis. These tools are based primarily on

    the IETF document draft-stokes-vkompella-ppvpn-hvpls-oam-00.txt.

    The 7750 SR VPLS OAM tools include:

    MAC Ping Provides the ability to trace end-to-end switching of specified MAC addresses.

    MAC ping provides an end-to-end test to identify the egress customer-facing port where acustomer MAC was learned. MAC ping can also be used with a broadcast MAC address to identify

    all egress points of a service for the specified broadcast MAC.

    MAC Trace Provides the ability to trace a specified MAC address hop-by-hop until the last

    node in the service domain.

    MAC Populate Allows specified MAC addresses to be injected in the VPLS service domain.

    This triggers learning of the injected MAC address by all participating nodes in the service. This

    tool is generally followed by MAC ping or MAC trace to verify if correct learning occurred.

    MAC Purge Allows MAC addresses to be flushed from all nodes in a service domain.

    MAC Ping

    For a MAC ping test, the destination MAC address (unicast or multicast) to be tested must bespecified. A MAC ping packet can be sent through the control plane or the data plane. When sent

    by the control plane, the ping packet goes directly to the destination IP in a UDP/IP OAM packet.

    If it is sent by the data plane, the ping packet goes out with the data plane format.

    In the control plane, a MAC ping is forwarded along the flooding domain if no MAC address

    bindings exist. If MAC address bindings exist, then the packet is forwarded along those paths (if

    they are active). Finally, a response is generated only when there is an egress SAP binding to thatMAC address. A control plane request is responded to via a control reply only.

    In the data plane, a MAC ping is sent with a VC label TTL of 255. This packet traverses each hopusing forwarding plane information for next hop, VC label, etc. The VC label is swapped at each

    service-aware hop, and the VC TTL is decremented. If the VC TTL is decremented to 0, the packet

    is passed up to the management plane for processing. If the packet reaches an egress node, andwould be forwarded out a customer facing port, it is identified by the OAM label below the VC

    label and passed to the management plane.

    MAC pings are flooded when they are unknown at an intermediate node. They are responded toonly by the egress nodes that have mappings for that MAC address.

  • 7/22/2019 7750 Series Troubleshooting Manual

    39/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    39

    MAC Trace

    A MAC trace functions like an LSP trace with some variations. Operations in a MAC trace aretriggered when the VC TTL is decremented to 0.

    Like a MAC ping, a MAC trace can be sent either by the control plane or the data plane.

    For MAC trace requests sent by the control plane, the destination IP address is determined fromthe control plane mapping for the destination MAC. If the destination MAC is known to be at aspecific remote site, then the far-end IP address of that SDP is used. If the destination MAC is not

    known, then the packet is sent unicast, to all SDPs in the service with the appropriate squelching.

    A control plane MAC traceroute request is sent via UDP/IP. The destination UDP port is the LSP

    ping port. The source UDP port is whatever the system gives (note that this source UDP port is

    really the demultiplexor that identifies the particular instance that sent the request, when

    correlating the reply). The source IP address is the system IP of the sender.

    When a traceroute request is sent via the data plane, the data plane format is used. The reply can be

    via the data plane or the control plane.

    A data plane MAC traceroute request includes the tunnel encapsulation, the VC label, and the

    OAM, followed by an Ethernet DLC, a UDP and IP header. If the mapping for the MAC address isknown at the sender, then the data plane request is sent down the known SDP with the appropriate

    tunnel encapsulation and VC label. If it is not known, then it is sent down every SDP (with the

    appropriate tunnel encapsulation per SDP and appropriate egress VC label per SDP binding).

    The tunnel encapsulation TTL is set to 255. The VC label TTL is initially set to the min-ttl (default

    is 1). The OAM label TTL is set to 2. The destination IP address is the all-routers multicast

    address. The source IP address is the system IP of the sender.

    The destination UDP port is the LSP ping port. The source UDP port is whatever the system gives

    (note that this source UDP port is really the demultiplexor that identifies the particular instancethat sent the request, when correlating the reply).

    The Reply Mode is either 3 (i.e., reply via the control plane) or 4 (i.e., reply via the data plane),depending on the reply-control option. By default, the data plane request is sent with Reply Mode

    3 (control plane reply).

    The Ethernet DLC header source MAC address is set to either the system MAC address (if nosource MAC is specified) or to the specified source MAC. The destination MAC address is set to

    the specified destination MAC. The ethertype is set to IP.

    MAC Populate

    MAC Populate is used to send a message through the flooding domain to learn a MAC address asif a customer packet with that source MAC address had flooded the domain from that ingress point

    in the service. This allows the provider to craft a learning history and engineer packets in aparticular way to test forwarding plane correctness.

  • 7/22/2019 7750 Series Troubleshooting Manual

    40/130

    Alcatel 31NAN0090 7750 Troubleshooting Guide

    40

    The MAC populate request is sent with a VC TTL of 1, which means that it is received at the

    forwarding plane at the first hop and passed directly up to the management plane. The packet is

    then responded to by populating the MAC address in the forwarding plane, like a conventionallearn although the MAC will be an OAM-type MAC in the FIB to distinguish it from customer

    MACs addresses.

    This packet is then taken by the control plane and flooded out the flooding domain (squelchingappropriately, the sender and other paths that would be squelched in a typical flood).

    This controlled population of the FIB is very important to manage the expected results of an OAMtest.

    The same functions are available by sending the OAM packet as a UDP/IP OAM packet. It is thenforwarded to each hop and the management plane has to do the flooding.

    Options for MAC Populate are to force the MAC in the table to type OAM (in case it already

    existed as dynamic or static or an OAM induced learning with some other binding), to prevent newdynamic learning to over-write the existing OAM MAC entry, to allow customer packets with this

    MAC to either ingress or egress the network, while still using the OAM MAC entry.

    Finally, an option to flood the MAC Populate request causes each upstream node to learn the MAC

    (i.e., populate the local FIB with an OAM MAC entry), and to flood the request along the data

    plane using the flooding domain.

    An age can be provided to age a particular OAM MAC after a different interval than other MACs

    in a FIB.

    MAC Purge

    MAC Purge is used to clear the FIBs of any learned information for a particular MAC address.This allows one to do a controlled OAM test without learning induced by customer packets. Inaddition to clearing the FIB of a particular MAC address, the purge can also indicate to the control

    plane not to allow further learning from customer packets. This allows the FIB to be clean, and be

    populated only via a MAC Populate.

    MAC Purge follows the same flooding mechanism as the MAC Populate.

    A UDP/IP version of this command is also available that does not follow the forwarding notion of

    the flooding domain, but the control plane notion of it.

    3.3.5. OAM Command Summary

    LSP diagnostic commands

    oam lsp-ping In-band LSP ping utility to verify LSP connectivity

    oam lsp-trace In-band LSP traceroute command to determine the hop-by-hop pathfor an LSP.

  • 7/22/2019 7750 Series Troubleshooting Manual

    41/130

    31NAN0090 7750 Troubleshooting Guide Alcatel

    41

    SDP diagnostic commands

    oam sdp-mtu Performs in-band MTU Path tests on an SDP to determine the largestpath-mtu supported on an SDP.

    oam sdp-ping Tests an SDP for in-band uni-directional or round trip connectivitywith a round trip time estimate.