http://sdu.ictp.it/lowbandwidth/ diagnostic steps les cottrell – slac presented at the...
Post on 21-Dec-2015
223 views
TRANSCRIPT
![Page 1: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/1.jpg)
http://sdu.ictp.it/lowbandwidth/
Diagnostic Steps
Les Cottrell – SLACPresented at the Optimization Technologies for Low-Bandwidth Networks, ICTP
Workshop, Trieste, Italy, 9-20 October 2006 http://www.slac.stanford.edu/grp/scs/net/talk06/diagnostics.ppt
Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP
![Page 2: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/2.jpg)
Slide: 2Les Cottrell, SLAC
Get ready
Bring up terminal window so can try some commands Bring up the presentation so can click on links:
www.slac.stanford.edu/grp/scs/net/talk06/diagnostics.ppt
![Page 3: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/3.jpg)
Slide: 3Les Cottrell, SLAC
AimGoal: provide a practical guide to debugging common
problems Why is diagnosis difficult yet important? Local host Ping, Traceroute, PingRoute Looking at time series Locating bottlenecks Correlation of problems with routes More tools and problems Where is a node Who do you tell, what do you say? Case studies and More Information
![Page 4: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/4.jpg)
Slide: 4Les Cottrell, SLAC
Why is diagnosis difficult?
Internet's evolution as a composition of independently developed and deployed protocols, technologies, and core applications
Diversity, highly unpredictable, hard to find “invariants” Rapid evolution & change, no equilibrium so far
Findings may be out of date Measurement/diagnosis not high on vendors list of priorities
Resources/skill focus on more interesting an profitable issues Tools lacking or inadequate Implementations are flaky & not fully tested with new releases
![Page 5: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/5.jpg)
Slide: 5Les Cottrell, SLAC
Add to that … Distributed systems are very hard
A distributed system is one in which I can't get my work done because a computer I've never heard of has failed. Butler Lampson
Network is deliberately transparent The bottlenecks can be in any of the following components:
the applications the OS the disks, NICs, bus, memory, etc. on sender or receiver the network switches and routers, and so on
Problems may not be logical Most problems are operator errors, configurations, bugs
When building distributed systems, we often observe unexpectedly low performance
the reasons for which are usually not obvious Just when you think you’ve cracked it, in steps security
Firewall, NAT boxes etc. Block pings, traceroute looks like port scan, diagnostic tool ports are
blocked … ISPs worried about providing access to core, making results public, &
privacy issues
![Page 6: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/6.jpg)
Slide: 6Les Cottrell, SLAC
Sources of problems
Host “errors” TCP buffers, heavy utilization …
Duplex mismatch (Ethernet) Misconfigured router/switches
Including routing errors, especially for backup paths
Bad equipment, wiring/fiber problem Congestion
![Page 7: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/7.jpg)
Slide: 7Les Cottrell, SLAC
Fire: Local Host
Usual Unix tools (uname-a, top, vmstat, iostat …) Is the host overloaded, do you have a gateway (route), name
server (nslookup/dig), which interface are you using (mii-tool (needs root), gives duplex & speed = common error source)21cottrell@pinger:~>sudo mii-tool eth0
– eth0: 100 Mbit, full duplex, link ok
Net: ifconfig –a (look at errors), netstat –a | more
Is server running (if you know port)? >telnet localhost 2811
Trying 127.0.0.1220 aftpexp04.bnl.gov GridFTP Server 1.12 GSSAPI type
Globus/GSI wu-2.6.2 (gcc32dbg, 1069715860-42) ready.^]telnet> quit
![Page 8: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/8.jpg)
Slide: 8Les Cottrell, SLAC
Ping Ping 1. to localhost, 2. ping to gateway (use route or traceroute to find
gateway), 3. ping to well known host 4. & to relevant remote host
Use IP address to avoid nameserver problems Look for connectivity, loss, RTT, jitter, dups May need to run for a long time to see some pathologies
(e.g. bursty loss due to DSL loss of sync) Try flood pings if suspect rate limited Use synack or sting if ICMP blocked
www-iepm.slac.stanford.edu/tools/synack/
![Page 9: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/9.jpg)
Slide: 9Les Cottrell, SLAC
Ping example
syrup:/home$ ping -c 6 -s 64 thumper.bellcore.com PING thumper.bellcore.com (128.96.41.1): 64 data bytes 72 bytes from 128.96.41.1: icmp_seq=0 ttl=240 time=641.8 ms 72 bytes from 128.96.41.1: icmp_seq=2 ttl=240 time=1072.7 ms 72 bytes from 128.96.41.1: icmp_seq=3 ttl=240 time=1447.4 ms 72 bytes from 128.96.41.1: icmp_seq=4 ttl=240 time=758.5 ms 72 bytes from 128.96.41.1: icmp_seq=5 ttl=240 time=482.1 ms --- thumper.bellcore.com ping statistics --- 6 packets transmitted, 5
packets received, 16% packet loss round-trip min/avg/max = 482.1/880.5/1447.4 ms
Repeat count Packet size Remote host
RTT
Missing seq #
Summary
![Page 10: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/10.jpg)
Slide: 10Les Cottrell, SLAC
Try the following Ping Examplesping cepheid.physics.utoronto.ca
From mcl-gpb.gw.utoronto.ca … Destination Host Unreachable
ping rolandlap.ph.unimelb.edu.auFrom rtr4-000037.unimelb.edu.au … Packet filtered
ping www.ncit.edu.npping: unknown host www.ncit.edu.np
ping inpe-gw-sp.cptec.inpe.brFrom 150.163.200.100 icmp_seq=0 Time to live exceeded
ping www.ug.edu.gh34 packets transmitted, 0 received, 100% packet loss, time 33068ms
synack -p 80 -k 5 www.ug.edu.gh5 packets transmitted, 5 packets received, 0.00 percent packet lossround-trip (ms) min/avg/max = 182.052/182.701/183.151 (std = 0.578) (median = 183.095) (interquartile range = 1.039) (25 percentile = 182.085) (75 percentile = 183.124)
![Page 11: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/11.jpg)
Slide: 11Les Cottrell, SLAC
3rd party ping Find servers:
http://www.slac.stanford.edu/comp/net/wan-mon/traceroute-srv.html Glasgow University*# Scotland. ICTP +*, Trieste, Italy. IHEP + Beijing, China.
Modify URL to request a ping for hosts with +pinger.ictp.it/cgi-bin/traceroute.pl?
function=ping&target=brunsvigia.tenet.ac.zaping from 134.79.18.163 (www.slac.stanford.edu) to
196.21.99.222 (brunsvigia.tenet.ac.za) for 140.105.16.64
– PING 196.21.99.222: 56 data bytes– 64 bytes from brunsvigia.tenet.ac.za (196.21.99.222): icmp_seq=0. time=370. ms – 64 bytes from brunsvigia.tenet.ac.za (196.21.99.222): icmp_seq=1. time=1911. ms – 64 bytes from brunsvigia.tenet.ac.za (196.21.99.222): icmp_seq=2. time=911. ms 64 bytes
from brunsvigia.tenet.ac.za (196.21.99.222): icmp_seq=3. time=385. ms – 64 bytes from brunsvigia.tenet.ac.za (196.21.99.222): icmp_seq=4. time=366. ms – ----196.21.99.222 PING Statistics---- 5 packets transmitted, 5 packets received, 0% packet
loss round-trip (ms) min/avg/max = 366/788/1911
![Page 12: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/12.jpg)
Slide: 12Les Cottrell, SLAC
RTT from California to world
Longitude (degrees)
300ms
300ms
RTT (ms.)
Fre
quen
cy
RT
T (
ms)
Source = Palo Alto CA, W. Coast
E. C
oast
US
W. C
oast
US
Eur
ope
& S
. Am
eric
a
Europe
0.3*0.6c
Bra
zil
E. C
oast
Data from CAIDA Skitter project
![Page 13: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/13.jpg)
Slide: 13Les Cottrell, SLAC
Traceroute Traceroute to remote host
Is the route direct, over commercial congested nets Reverse traceroute from remote host to you or 3rd
party www.slac.stanford.edu/comp/net/wan-mon/traceroute-srv.ht
ml www.tracert.com/
CAIDA Mouse sensitivemap
![Page 14: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/14.jpg)
Slide: 14Les Cottrell, SLAC
Traceroute
UDP/ICMP tool to show route packets take from local to remote host
17cottrell@flora06:~>traceroute -q 1 -m 20 lhr.comsats.net.pktraceroute to lhr.comsats.net.pk (210.56.16.10), 20 hops max, 40 byte packets 1 RTR-CORE1.SLAC.Stanford.EDU (134.79.19.2) 0.642 ms 2 RTR-MSFC-DMZ.SLAC.Stanford.EDU (134.79.135.21) 0.616 ms 3 ESNET-A-GATEWAY.SLAC.Stanford.EDU (192.68.191.66) 0.716 ms 4 snv-slac.es.net (134.55.208.30) 1.377 ms 5 nyc-snv.es.net (134.55.205.22) 75.536 ms 6 nynap-nyc.es.net (134.55.208.146) 80.629 ms 7 gin-nyy-bbl.teleglobe.net (192.157.69.33) 154.742 ms 8 if-1-0-1.bb5.NewYork.Teleglobe.net (207.45.223.5) 137.403 ms 9 if-12-0-0.bb6.NewYork.Teleglobe.net (207.45.221.72) 135.850 ms10 207.45.205.18 (207.45.205.18) 128.648 ms11 210.56.31.94 (210.56.31.94) 762.150 ms12 islamabad-gw2.comsats.net.pk (210.56.8.4) 751.851 ms13 * 14 lhr.comsats.net.pk (210.56.16.10) 827.301 ms
Probes/hop Max hops Remote host
No response:Lost packet or router
ignores
Long delaysatellite
location
![Page 15: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/15.jpg)
Slide: 15Les Cottrell, SLAC
Traceroute server results Example: www.slac.stanford.edu/cgi-bin/nph-traceroute.pl
Securitywarning
Traceroute
Relatedinfo
Enter IP address or name
![Page 16: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/16.jpg)
Slide: 16Les Cottrell, SLAC
Graphical Traceroute http://visualroute.visualware.com/
![Page 17: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/17.jpg)
Slide: 17Les Cottrell, SLAC
Pingroute Ping routers along route, e.g. a tool to install that helps:
www.slac.stanford.edu/comp/net/fpingroute.pl or www.slac.stanford.edu/comp/net/pingroute.pl if fping N/A
15cottrell@noric04:~>fpingroute.plfpingroute.pl does a traceroute to the selected host. For each of the hops along the route it then uses fping to ping each node (in parallel) 'count' times. Output includes traceroute information, RTTs, losses for 100 and 'size‘ byte pings.Version=0.21, 8/24/04Usage: fpingroute.pl [Opts] host where host is the remote host's IP address or name e.g. www.slac.stanford.edu Opts: [-c count default=10] [-s size default=1400] [-i initial default=1]Example: fpingroute.pl -i 3 -c 10 -s 1400 www.triumf.ca
![Page 18: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/18.jpg)
Slide: 18Les Cottrell, SLAC
Pingroute example May help tell where losses start Will need many pings if losses small
Routers may not
respond
Start of losses?
But?
Start ofsustained
losses
![Page 19: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/19.jpg)
Slide: 19Les Cottrell, SLAC
Look at time series Look at history plots (PingER, IEPM-BW, ISPs, own
border router etc.), when did problem start, how big an effect is it? Assumes you know “proximity” of paths for which there are
archived active measurements to the path that you are interested in
Also that relevant measurements existwww-iepm.slac.stanford.edu/pinger/ amp.nlanr.net/ unfortunately no longer fundedISPs plots:
(www.slac.stanford.edu/comp/net/wan-mon/netmon.html for a a place to start looking)
– Abilene: http://stryper.uits.iu.edu/abilene/ – GEANT: http://stats.geant.net/usagemap/usagemap– RIPE: http://www.ripe.net/projects/ttm/Plots/ – ESnet: http://measurement.es.net/ (OWAMP)
Collaboration between Internet2/ESnet/Geant to provide access to router measurements holds promise
Look at traceroute histories (see later)
![Page 20: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/20.jpg)
Slide: 20Les Cottrell, SLAC
Example time series
Look for change in measured value Note
time Correlate Italy disconnected
![Page 21: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/21.jpg)
Slide: 21Les Cottrell, SLAC
Find location of a bottleneck Look at hops along the path
Pingroute (see earlier) If possible look at utilizations or active probes launched from there Pathneck http://www.cs.cmu.edu/~hnn/pathneck/
Uses trains of packets to probe hops along route, looking at dispersion induced by queuing
Pipechar (son of pathchar, pchar) http://www.dsd.lbl.gov/OldProjects/NCSSend packets of varying sizes to each router along pathLook at RTT as a function of packet sizeFrom slope deduce “bandwidth”Diferentiate to find capacity at each hopHowever pipechar has uncertain supportPacket size variation limited to 1-MTU (~1500) Bytes, so on fast links
timing is difficult, with the result that estimates may not be reliable (OK for slow links)
![Page 22: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/22.jpg)
Slide: 22Les Cottrell, SLAC
Divide & Conquer
Abilene has hosts at major PoPs running bwctl So make measurements from end to middle to ID loss
of performance http://e2epi.internet2.edu/pipes/ami/bwctl/
![Page 23: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/23.jpg)
Slide: 23Les Cottrell, SLAC
Correlate with routes (traceanal)
![Page 24: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/24.jpg)
Slide: 24Les Cottrell, SLAC
Visualizing traceroutes www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/
slac_wan_bw_tests.html, => traceroutes One compact page per day One row per host, one column per hour One character per traceroute to indicate pathology or change (usually
period(.) = no change) Identify unique routes with a number
Be able to inspect the route associated with a route number Provide for analysis of long term route evolutions
Route # at start of day, gives idea of route stability
Multiple route changes (due to GEANT), later restored to original route
Period (.) means no change
![Page 25: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/25.jpg)
Slide: 25Les Cottrell, SLAC
Changes in network topology (BGP) can result in dramatic changes in performance
Snapshot of traceroute summary table
Samples of traceroute trees generated from the table
ABwE measurement one/minute for 24 hours Thurs Oct 9 9:00am to Fri Oct 10 9:01am
Drop in performance(From original path: SLAC-CENIC-Caltech to SLAC-Esnet-LosNettos (100Mbps) -Caltech )
Back to original path
Changes detected by IEPM-Iperf and AbWE
Esnet-LosNettos segment in the path(100 Mbits/s)
Hour
Rem
ote
host
Dynamic BW capacity (DBC)
Cross-traffic (XT)
Available BW = (DBC-XT)
Mbit
s/s
Notes:1. Caltech misrouted via Los-Nettos 100Mbps commercial net 14:00-17:002. ESnet/GEANT working on routes from 2:00 to 14:003. A previous occurrence went un-noticed for 2 months4. Next step is to auto detect and notify
Los-Nettos (100Mbps)
![Page 26: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/26.jpg)
Slide: 26Les Cottrell, SLAC
Moving towards application Try user application (mem to mem & disk to disk)
GridFTP, bbcp, bbftp … Iperf or thrulay (also provides RTT) to test TCP or UDP
throughput (injects traffic, +server) dast.nlanr.net/Projects/Iperf/ www.internet2.edu/~shalunov/thrulay/
Available bandwidth: Pathload:
www-static.cc.gatech.edu/fac/Constantinos.Dovrolis/pathload.html Pathchirp: www.spin.rice.edu/Software/pathChirp/ bing …
NDT What are the interface speeds? What is the bottleneck? Is there a duplex mismatch? Are buffers set right (both ends)?
Bottleneck
Min spacingAt bottleneckSpacing preserved
On higher speed links
![Page 27: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/27.jpg)
Slide: 27Les Cottrell, SLAC
http://e2epi.internet2.edu/ndt/NDT example (Rich Carlson)
![Page 28: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/28.jpg)
Slide: 28Les Cottrell, SLAC
Other tools Ntop
Summarizes libpcap (sniffer) infor
Internet2 Detective: Tests connectivity to I2, bandwidth, multicast, IPv6
Can run as Java applethttp://detective.internet2.edu/
NLANR Internet Advisor Ethereal, tcpdump, snoop for masochists Passive tools:
Netflow for characterizing network, spotting abnormalities, e.g. www.itec.oar.net/abilene-netflow
www.slac.stanford.edu/comp/net/slac-netflow/html/SLAC-netflow.html
SNMP based tools
![Page 29: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/29.jpg)
Slide: 29Les Cottrell, SLAC
And then … Wireless
Avoid peer-to-peer/ad-hoc connectionsDisable connecting to ad-hoc (set infrastructure only)Disable bridgingHow to do it varies by OS (XP, OSX, Linux)
Ad hoc can still interfere if on same channel Tools to locate an access point (e.g. Yellow-Jacket) Vendors have management tools to enable APs to detect rogue APs
NAT boxes may block or not support application Private addresses:
10.0.0.0 - 10.255.255.255 a single class A net172.16.0.0 - 172.31.255.255 16 contiguous class Bs192.168.0.0 – 192.168.255.255 256 contiguous class Cs
![Page 30: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/30.jpg)
Slide: 30Les Cottrell, SLAC
“Where is” a host? Beware some of information following is ephemeral, in general use
heuristics with Google Google “Internet country codes” for TLDs
Host may not be in TLD country, especially developing regions often use proxies elsewhere
Location may be encoded in router name ipls=Indianapolis, snv=Sunnyvale …
Name server lookup to find hostname given IP address47cottrell@netflow:~>nslookup 210.56.16.10Server: localhostAddress: 127.0.0.1Name: lhr.comsats.net.pkAddress: 210.56.16.10
Use a whois server, e.g. www.networksolutions.com/cgi-bin/whois/whois (Americas & Africa)www.ripe.net/cgi-bin/whois (Europe)www.apnic.net/ (Asia)May identify site name, address, contact, etc, not all domains are in
databases (e.g. will not find comsats.net.pk)
![Page 31: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/31.jpg)
Slide: 31Les Cottrell, SLAC
“Where is” a host – cont.
Find the Autonomous System (AS) administering Form giving AS for domain name
http://www.fixedorbit.com/search.htmGives AS number, name adjacent AS’s web page for AS
Given an AS find out more about it:Use http://bgp.potaroo.net/cidr/ go to bottom and enter AS into
form:– Gives ISP name, web page, phone number, email, hours etc.
Review list of AS's ordered by Upstream AS Adjacencywww.telstra.net/ops/bgp/bgp-as-upsstm.txtTells what AS is upstream of an ISP
![Page 32: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/32.jpg)
Slide: 32Les Cottrell, SLAC
“Where is” a host - cont.
May be able to get latitude & longitude: http://www.hostip.info/index.html http://www.ip2location.com/
But it is a subscriber service ($$$, but …), however it is probably best for developing regions
Google:www.geoiptool.com/http://www.geoiptool.com/
Triangulate pings from landmarks (in development) http://www.slac.stanford.edu/comp/net/wan-mon/tulip/
Need more landmarks, send email [email protected]
http://www.cs.cornell.edu/~bwong/octant/ # for US only
![Page 33: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/33.jpg)
Slide: 33Les Cottrell, SLAC
Who you gonna tell?
Local network support people Internet Service Provider (ISP) usually done by local networker
Usually will know immediate one, e.g. [email protected] Use puck.nether.net/netops/nocs.cgi to find ISP Use www.telstra.net/ops/bgp/bgp-as-upsstm.txt to find upstream ISPs
Well managed sites and ISPs maintain a list of email addresses such as abuse@ or postmaster@, that one can send email to, for example to complain about spam etc. This follows an Internet recommendation (RFC 2142). Some less helpful sites do not provide such services, for more on these,
see RFC-ignorant.org
![Page 34: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/34.jpg)
Slide: 34Les Cottrell, SLAC
What ya gonna tell ‘em? Describe problem with details
What is affected?Application, host OS (uname –a), NIC (ifconfig, route)
How is it affected?Non responsiveness, unable to contact remote hostSlow performance (see Brian’s talk), packet loss
When did it start?
Send ping output between hosts Send traceroute forward & reverse – if possible
Maybe use –I (ICMP option)
NDT Identify when it started If complex think about creating web page with details
Top, vmstat, pingroute, pipechar, application output (GridFTP, iperf)…
![Page 35: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/35.jpg)
Slide: 35Les Cottrell, SLAC
Web page examples: Case studies
http://www.slac.stanford.edu/grp/scs/net/case/html/ http://e2epi.internet2.edu/case-studies/
![Page 36: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/36.jpg)
Slide: 36Les Cottrell, SLAC
More Information Tutorial on monitoring
www.slac.stanford.edu/comp/net/wan-mon/tutorial.html RFC 2151 on Internet tools
www.freesoft.org/CIE/RFC/Orig/rfc2151.txt Network monitoring tools
www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html www.caida.org/tools/taxonomy/
Network Performance Tools: an I2 Cookbook e2epi.internet2.edu/network-perf-wk/tools-cookbook.pdf
Network Monitoring sites www.slac.stanford.edu/comp/net/wan-mon/netmon.html
How to Accelerate Your Internet, ISBN: 0-9778093-1-5, Ed. Flickenger R.
![Page 37: Http://sdu.ictp.it/lowbandwidth/ Diagnostic Steps Les Cottrell – SLAC Presented at the Optimization Technologies for Low-Bandwidth Networks, ICTP Workshop,](https://reader036.vdocuments.net/reader036/viewer/2022062313/56649d595503460f94a399d4/html5/thumbnails/37.jpg)
Slide: 37Les Cottrell, SLAC
Local Host - LISA
Localhost Information Service Agent LISA is a Java Web Start application which provides: Integration with MonALISA Complete Monitoring of the System (Load, CPU, Memory, Disk,
Disk IO, Paging, Processes, Network Traffic and Connectivity...). History and instantaneous Filters to trigger actions when predefined conditions are detected. A user Friendly GUI to present the monitoring information. Optimization modules for distributed applications. It is a lightweight application that can be easily deployed on any
system. Modules for End to End network measurements ( e.g. IPERF). See monalisa.caltech.edu/dev_lisa.html