troubleshooting lans with wirespeed packet capture and

Application Note

WEBSITE: www.jdsu.com/test

Troubleshooting LANs with Wirespeed Packet Capture and Expert Analysis

IntroductionThis application note is one in a series of local area network (LAN) troubleshooting papers from JDSU Communications Test and Measurement. Troubleshooting LAN issues covers a wide array of network problems and diagnostic scenarios that may include:• evaluatingnetworkutilizationoverthecourseofabusinessdaybylink,virtualLAN(VLAN),orsubnet• detectingexcessivebroadcastormulti-casetraffic• finding“bandwidthhogs”• understandingwhatprotocolsarepresentonthenetwork(anddeterminingwhethertheyshouldbe)• identifyingthe“top-talkers”onthelink—theIPdevicesthatareconsumingthemostcapacity• experiencingapplicationperformanceissues(slowwebserverresponsetimeorintermittentunavail-

abilityofane-mailserver).

Beforenetworktroubleshootingcanbegin,onemusthaveaclearunderstandingofnetworktestaccess.Testing tools used for network analysis and troubleshooting scenarios must be able to monitor the network trafficbeingtested.Themostcommonmeansformonitoringanetworkisusingthebuilt-inportmirroringcapabilitiesofanetworkdevice,suchastheswitch/routerortoinstallaspecial“tap”devicebetweenthedevicesbeinganalyzed,suchasthosebetweenanapplicationserveranddatabaseserver.Figures1and2showeachtestaccessmodeforanalyzingtrafficbetweentwoservers.

Figure 1: Test access via port mirroring

ApplicationServer

Database Server

Ethernet Switch

Port 1 Port 3

Port 2

JDSU T-BERD/MTS-4000 with ESAM

Port 3“mirrored”

to Port 2

Application Note: Troubleshooting LANs with Wirespeed Packet Capture and Expert Analysis 2

IntheportmirroringtestaccessmodeshowninFigure1,theT-BERD®/MTS-4000EnterpriseServicesApplicationModule(ESAM)isconnecteddirectlytoaspareswitchport(10M/100M/1000M)thatiscon-figuredtocopyalltraffictoandfromPort3(databasetraffic)toPort2(thetestaccessport).Becauseamirrorportcancopytrafficfrombothdirectionsouttoasingleport,itwilldropframesiffull-duplexlinkutilizationexceeds50percent.Aggregatingtapsperformsimilarlyastheyfunnelbothdirectionsoftrafficoutofasingleport.Likewise,ifthefull-duplexlinkutilizationexceeds50percentitalsodropsframes.Someaggregatingtapshaveinternalbuffersthatallowthemtocompensateforburstsabove50percent;however,thiscanresultinincorrecttimestampswhenperformingcaptures.Asageneralrule,portmirroringisthepreferredapproachastherearegenerallyspareswitchportsandnointerruptiontoproductiontraffic.The following table provides a summary of each test access mode:

Item Port Mirroring Aggregating Network Tap

Disruptive to network operation No. A port mirror command Yes. Must install network taps during off-hours or as part of the does not interrupt normal production traffic. production installation on critical network links.Handle full line rate traffic Handles up to 50-percent traffic utilization Handles traffic up to 50-percent utilization before dropping before dropping packets. Port mirroring may packets. Aggregating taps with buffers can compensate for not be able to “keep up” on busy network links. bursts above 50 percent. A good network tap will not drop any production traffic, but may drop duplex traffic (on heavily loaded links) because the duplex traffic is combined into a single test access port.Pass Layer 1 and Layer 2 Errors No. Depends on the tap. Some pass errors, some do not.Require network device Yes. Console access to the network device is No.administrative privileges required to enable port mirroring. Cost Usually free, because most switches have a Reputable 1000Base-T taps can cost $1000+. spare port.

ThefollowingsectionssummarizeapracticalapproachtoconductingnetworkanalysisusingtheJDSUESAMfortheT-BERD/MTS-4000.

Figure 2: Test access with a network tap

ApplicationServer

Database Server

Port 1 Port 2


Network tapcopies tra�cto analysis port

Network Tap


Network Analysis Workflow

Thereisnosinglemethodforanalyzingnetworkissues,andyettherearesomebestpracticesthatexpertsuseintheirday-to-daytroubleshootingactivities.Figure3showstheJDSU-proposednetworkanalysisworkflow.

Figure 3: Best practices Network Analysis workflow

Asecondapplicationnoteinthisseries,TroubleshootingLANswithNetworkStatisticsAnalysis,coversthedetailsofthefirstthreenetworktroubleshootingsteps(NetworkBandwidthBaselining,IdentifyingTopTalkersandProtocols,andTrend/DiagnoseApplicationIssues).

This application note covers these two steps:1. Filter and Capture Network Traffic:Atanyofthestepslistedabove,itisimportanttoconductcon-

text-sensitivepacketcaptures.Forexample,theabilitytoidentifyanInternetProtocol(IP)TopTalkerandsimplyselectitasacapturefilter.TheJDSUESAMprovidesasimplecapturefilteruserinterfaceforthemorecommonfilterscenariosandalsoprovidesadvanceddeeppacketinspection(DPI)filtersthat can search within the payload of packets.

2. Network Packet Analysis:TheJDSUESAMconductsanalysisofthecapturefilesdirectlyusingthepopularopensourcesoftwareWireshark.Theabilityofthetesttooltoperformexpertanalysisanddiagnosecommonnetworkproblemswithinthepacketfileisalsoessential.

The following sections describe the details of packet filtering, capture, and analysis bothwith theWiresharktoolandtheJDSUpacketcaptureexperttestfeatureJ-Mentor.

NetworkBandwidthBaselining

IdentifyTop Talkers

and Protocols

Filter andCapture

Network Tra�c

ConductNetwork

Packet Analysis

Trend/DiagnoseApplication

Issues


Filtering and Packet Capture

ThepacketcapturedevicemustbeabletocapturepacketsatfullGigabitEthernet(GigE)linespeed.Wireshark-basedcapturesusethenetworkinterfacecard(NIC)ofthepersonalcomputer(PC)toperformthepacketcaptures.Theaverage-performingPCwilldroppacketsevenatlineratesof100Mbps.Onlyverypowerful(andexpensive)workstationsorserverswithhighperformanceNICscancaptureatGigabitEthernet(GigE)wirespeed.WhilePCswithWiresharkaresufficientforcasualnetwork“sniffing,”havingatesttoolthatcancaptureallframes(regardlessofsize)atGigEspeedisessentialforperformingaccurateproblemanalysisanddiagnosis.TheESAMcancaptureupto1GBofpacketsatfulllineratesupto1GigEandstorethemnativelyinindus-try-standardpacketcapture(pcap)format.WiresharkisusedtodisplayanddecodethepacketcapturesdirectlyontheESAMuserinterface.Figure4showsanexampleofWiresharkrunningontheESAMuserinterface.

Figure 4: Wireshark running directly on the JDSU ESAM User Interface

Often,trafficvolumeisveryhighandrequiresfilteringbeforepacketcapture,commonlyreferredtoasapre-capturefilter.Themostcommonformofpre-capturefilteris:IP Address (or Address Pair) AND Protocol (forexample,hypertexttransferprotocol[HTTP]andsimplemailtransferprotocol[SMTP]).AnIPAddressPairiscommonlyreferredtoasanIPConversation.Figure5isanexampleofaSmall–MediumBusiness(SMB)enterprisewith100usersaccessingacorporatedatabase.


Figure 5: SMB with Corporate Database Server

Althoughsimplified,Figure5illustrates100localusersaccessingthedatabase,mail,andwebserverswithinthecorporatedatacenter.Additionally,remotebranchofficesandtelecommuterscanaccesstheseresourcesviavirtualprivatenetwork(VPN)accessandtheInternet.Asacasestudy,letusintroduceahypotheticalscenariowhereJoeexperiencespoorperformancewhenaccessingthedatabaseserver.Becausethedatabaseserver linkiscritical, thenetworkteamatJoe’scompanyhasadedicatednetworktapplacedinlinewiththedatabaseserver.Alltraffictoandfromthedatabaseserveriscopiedtothepacketanalyzerthat,inthiscase,istheJDSUESAM.Withoutpre-capturefilters,theJDSUESAMcapturesallofthenetworktrafficfromallusers,whichisnotapracticalsolution.Figure6illustratestheconfigurationofapre-capturefilterusingtheESAM.

Figure 6: Configure Pre-capture Filter for IP = 10.10.65.16

Notethatinadditiontospecifyingthe10.10.65.16IPaddress,bothdirectionsofcommunicationwillbecaptured.Oftenasingleserverwillhostmultipleapplications.LookingbackatFigure5,imaginethatallthreefunctions(database,web,ande-mailservers)nowresideononephysicalserver.SettingthefiltertoJoe’sIPaddresswouldcapturethetrafficbetweenJoeandallthreeapplications.

LaptopJoe’s IP = 10.10.65.16

Web Server

Layer 2 Switch

Router

Firewall


Network Tap

Internet

Mail Server

DB Server


ThesolutiontothisproblemistospecifyanIPaddressandprotocolfilter,asillustratedinFigure7.Notethat“TNS”isthetransportlayerprotocoloverwhichOraclestructuredquerylanguage(SQL)databasetransactions are carried.

Figure 7: Configure Pre-capture Filter for IP = 10.10.65.16 and TNS

Formorecomplexsituations,IPandProtocol-basedfiltersmaybeinadequateforcapturingthedesiredtraffic.Frequently,adatabaseserverisconnectedtoafront-endwebserverorapplicationserverandtheusertraffictothedatabaseservercannotbedetectedbyIPaddress.ThedatabaseserverseesoneIPaddress(forexample,fortheapplicationserver)andalloftheusertrafficiscontainedwithinapooloftransmissioncontrolprotocol(TCP)connections.Thiscanbethoughtofas“trunk”communicationbetweentheappli-cationserverandthedatabaseserver(allusertrafficfromapplicationserverIPtothedatabaseserverIP).Inthisscenario,DPItechniquesaretheappropriatemeanstotroubleshootthisproblem.DPIexaminespayloadwithinthepacketandtheJDSUESAMisequippedwithadvancedDPItechnology.Forthisexample,theESAMmustfirstbeconfiguredwithacustomDPIfilter.AssumethatJoe’sOracleuserIDisjoe_knap_2301.ThisIDcanusuallybefoundinthepayloadofthepacketssenttothedatabaseserver.ThecomplexDPIbasedfilterwillsearchwithinthepacketpayloadtodetectandcaptureonlythepacketswithJoe’sID.Figure8illustratestheeaseofconfiguringthistypeofcomplexDPI-basedfilterontheESAM.

Figure 8: Configuring a DPI Filter for “joe_knap_2301”

NotethatnoIPorportsettingsareconfiguredforthisaddressastheyarenotrelevant;theESAMwillconductDPItodetectJoe’sdatabasecommunicationsandonlycapturethosepackets.AfterlaunchingWiresharkdirectlyontheESAMuserinterface,theusercantroubleshootthedatabaseissue.


Expert Analysis

Wiresharkpacketcapturesprovideawealthofinformation,butitisverydifficultfortheaverageusertodiagnosenetworkproblems.ItrequiressignificantnetworkexpertiseisrequiredtocombthroughthepacketcapturefileanddetectissuessuchasTCPretransmissions,InternetControlMessageProtocol(ICMP)events,andothers.Expertpacketanalysisisessentialtoanalyzinganddiagnosingproblemsinpoorlyperformingnetworksandapplications.TheJDSUESAMprovidesJ-Mentor,whichisapacketcaptureexpertthatcananalyzepacketcapturefilesanddiagnosecommonnetworkproblems.Inthisexample,a2MBfiledownloadtookover60secondsandthecauseofthispoorthroughputmustbediagnosed.Figure9showsscreenshotsofthiscapturefileafteropeninginWiresharkandJDSUJ-Mentor.

Figure 9: Wireshark Decodes versus J-Mentor Diagnosis of FTP Download

AstheWiresharkscreenontheleftillustrates,usersrequireexpertisetonavigatethroughthepacketdecodesandadvancedanalysismenuoptions.However,theJ-Mentorscreenontherightillustratestheeasewithwhichnetworktechnicianscanquicklyisolatetheproblemtotheproblemnetworklayer.

Inthisexample,thenetworkissueoccurredatLayer4(TCP)andbyclickingontheDetailsbutton,Figure10ispresented.

Figure 10: Drilling into Layer 4 (TCP) Issues


The results show that a total of 24 retransmissions occurred. Next the user clicks on the TCPRetransmissionstabtofurtherisolatetheproblemasFigure11illustrates.

Figure 11: Isolating TCP Retransmissions to Source IP

Asthefigureshows,theFTPclient(206.191.62.221)caused18oftheretransmissions.Thisprovidessimplediagnostic information to the network troubleshooter and points to the problem source for further isolation andtroubleshooting(determineifthehostIPhashalf-duplexportissues,badcabling,orarelatedproblem.)Half-duplexportissuesalsoremainacauseofconsiderableheadaches.Mostnetworkslistportsettings:J-MentorautomaticallydetectsthesemessagesandprovidesalistofsourceMACaddressesthatlisthalf-duplexsettingsduringthepacketcapturetimeinterval,seeFigure12.

Figure 12: Isolating Source MAC using the Half-Duplex Setting


It iscommontosearchfor“bandwidthhogs”as thesourceofpotential issues inpoorlyperformingnetworks.Therefore,asFigure13shows,J-MentorprovidesalistingoftheTopTalkersdetectedwithinthepacketcapturefilealongwiththenumberofbytesandframesforeachtalker.

Figure 13: Simple Display of IP Top Talkers


ConclusionTroubleshootingLANproblemseffectivelyrequiresapacketcapturetoolcapableofcapturingpacketsatfullGigElinespeed.Theaverage-performingPCwilldroppacketsevenatlineratesof100Mbps.WhilePCswithWiresharkaresufficientforcasualnetworksniffing,havingatesttoolthatcancaptureallframes(regardlessofsize)atGigEspeedisessentialforperformingaccurateproblemanalysisanddiagnosis.TheJDSUESAMfortheT-BERD/MTS-4000cancaptureupto1GBofnetworkpacketsandstorethemnativelyinindustry-standardpcapformat.TheESAMprovidesasimplecapturefilteruserinterfaceforthemorecommonfilterscenariosandalsoprovidesadvancedDPIfiltersthatcansearchwithinthepayloadofpackets.WiresharkrunsnativelyontheESAMdisplayandJ-Mentorprovidesexpertpacketcapturediag-nosticstoprovidebest-practicetroubleshootingforthelessexperiencednetworktroubleshooter.TheESAMprovidesaworkflow-basedinterfacethat“walks”theuserthroughthebestpracticesapproachtowardsolvingamultitudeofnetworkproblems.Figure14istheJDSUT-BERD/MTS-4000platformwithESAMinterface(andanoptionalfiberscope),andFigure15showstheworkflow-baseduserinterface.

Figure 14: JDSU T-BERD/MTS-4000 platform with the ESAM


Figure 15: Workflow-based user interface of the ESAM

TheJDSUESAMfortheT-BERD/MTS-4000providescomprehensiveLANtestingcapabilitieswiththesefeatures:• Layer1-7protocolcaptureandexpertanalysis• networkconnectivity• networkdiscovery• afullrangeofphysicalmediatests• aworkflow-baseduserinterface• amodularplatformwithmanyoptions: –VoiceoverIP(VoIP)phoneemulation –opticalpowermeter/visualfaultlocator –fiberinspectionprobewithautomatedpass/fail –Wirelessfidelity(WiFi)testing –OTDRmodules

Throughitsworkflow-basedintuitiveuserinterface,theESAMprovidesphysicalmediatestsincludingspeed-certification of electrical Ethernet cabling, network connectivity tests, discovery, wirespeeddeep-packetstatistics,andwirespeedprotocolcaptureandexpertanalysisusingunique,in-depthJDSUJ-Mentorcapabilities.Inaddition,theESAMispartofthemodularJDSUT-BERD/MTS-4000platformallowingadditionaloptionsthatincludeVoIPemulation,WiFitesting,IPvideotesting,opticalpowermeters(OPMs),visualfaultlocators(VFLs),digitalfiberinspectionprobes,andopticaltimedomainreflectometers(OTDRs).Testconnectivitycanbeobtainedeitherelectricallyviaa10/100/1000RJ45Ethernetjackorviaasmallform-factorpluggable(SFP)foropticalEthernet.


Productspecificationsanddescriptionsinthisdocumentsubjecttochangewithoutnotice.©2010JDSUniphaseCorporation301681420000710LANWPCEA.AN.TFS.TM.AEJuly2010

Test & Measurement Regional Sales

NORTH AMERICATEL: 1 866 228 3762FAX: +1 301 353 9216

LATIN AMERICATEL: +1 954 688 5660FAX: +1 954 345 4668

ASIA PACIFICTEL: +852 2892 0990FAX: +852 2892 0770

EMEATEL: +49 7121 86 2222FAX: +49 7121 86 1222

WEBSITE: www.jdsu.com/esam

troubleshooting lans with wirespeed packet capture and

Documents