Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
1
Introduction to the PPNCG Networking for the PPARC Community
Introduction to the PPNCG UK Network TopologiesExternal Connectivity – Europe & USAstronomy & Astrophysics Sites Grid Network Monitoring PingER – World wide MonitoringQoS – a micro Introduction
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
2
Introduction to the PPNCGMembership includes HEP and Astronomy users
Dave Terrett , Bob Bentley, Ralph Spencer
RemitEnsure the community has the required networking facilities
Monitor end-to-end performance
Investigate new network applications / technologies
Provide advice on kit / facilities
Active Network MonitoringPPNCG ping, ftp and traceping
ICFA monitoring
Report problems to UKERNA
Regular meetings with UKERNA invitedRecognised as a subject group in JNUG and JISCLinks to several Grid Projects
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
3
SuperJANET4: Backbone and Access links
Worldcom supplied the transmissionUKERNA layer the IP service
Core PoP IP router at Worldcom
Backbone Access Router at MANs
Access Links:Large MAN 2.5 Gbit -> 10-20 Gbit
Medium MAN 622 Mbit -> 2.5 Gbit
4 node DWDM development netDeployment Status:
Backbone Oct 00 Routers Nov 00
All sites Mar 01
Proved to be Stable
Constant growth of traffic
Upgrade Backbone to 10Gbit Jun 02
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
4
SuperJANET4: ping rtt Core routers Jun 01
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
5
SuperJANET4: ping rtt Site nodes
Lancaster Glasgow
MAN / LAN Issues
Bristol Cambridge
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
6
London MAN Upgrade
UDP Throughput
Mbit/s
UDP Packet loss
%
UDPmon Tests
Manchester – London
MAN was 155 Mbit ATM
1st Oct Time interval in Weeks Richard HJ
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
7
Previous External Connectivity
Europe:TEN-155155Mbit Access link
US:6 * 155 Mbit links
Peer in Hudson St. 622 Mbit to Esnet622 Mbit to Abilene.
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
8
ICFAMON Plot from RAL to CERN for 19th Oct to 1st Nov 2001
Europe – Access links (1)
UK Access link 155 Mbit ATMSustained rate 130 MbitContract to end of Nov 01Bad news for users !
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
9
Traceping Oxford to CERN for 31st October 2001
Europe – Access links (2)
loss around ten155-gw.ja.net router
J. Macallister
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
10
New External Connectivity
6 * 155 Mbit links2.5Gbit line installedIP commodity peer in London Research traffic over 2.5G bitPeer in Hudson St. 622 Mbit to Esnet622 Mbit to Abilene.
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
11
Connectivity to Europe : GeantStart mid November 2001UKERNA switched off TEN-155 3 Dec 2001
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
12
ICFAMON Plot from DL to CERN for 18th Feb to 3rd Mar 2002
Connectivity to Europe
UK Dante Access link 2.5 Gbit POS
Remember 19th Oct to 1st Nov 2001Access link over loaded
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
13
Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US
7 day periods 1 hour averages
Weekend-Before Weekday-After
Weekday-Before
14 Jan 2002 (800Mbit/s)peak 86% of total 930 Mbit
17 Jan 2002Peering altered 22 Jan
22 Jan 2002 Weed day peak 175 Mbit/s
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
14
Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US7 Dec 2001 (900kbit/s) 29 Jan 2002 (175kbit/s)peak is 88% of total BW 930 Mbit10 minute averages 10 minute averages
Last 7 days 1 hour averages
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
16
ICFAMON Plot from DL to Anglo-Australian Observatory for 11th Apr to 24th Apr 2002
Connectivity to Australia
Packet loss reasonablertt improves ~420 ms to ~300 msVariations ~100ms
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
17
ICFAMON Plots for 11th Apr to 24th Apr 2002 DL to NOAO, Arizona DL to Goddard GSFC NASA
Connectivity to US
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
18
ICFAMON Plot from DL to The Joint Astronomy Centrefor 11th Apr to 24th Apr 2002
Connectivity to Hawaii
Packet loss goodrtt ~210 msVariations – queuing
traceroute:Cross SuperJANET4 to NY OKCross Abilene to Seattle OKEnters uhnetStops after 2-3 routers
No connectivity to La Palmatraceroute ends in iac.es network Tenerife ?
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
19
Grid Network Monitoring
Several tools in test – plugged into a coherent structure: PingER, RIPE one way times, iperf, UDPmon, rTPL, GridFTP, and
NWS prediction engine continuous tests for last few months to selected sites:
DL Man RL UCL CERN Lyon Bologna SARA NBI SLAC … The aims of monitoring for the Grid:
to inform Grid applications, via the middleware, of the current status of the network – input for resource broker and scheduling
to identify fault conditions in the operation of the Grid to understand the instantaneous, day-to-day, and month-by-month
behaviour of the network – provide advice on configuration etc.
Network information published in LDAP schema Will be used by UK GridPP and e-science centresAstroGrid ?
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
20
Local NetworkMonitoring
Store & Analysisof Data (Access)
Access to current and historic dataand metrics via the Web, i.e. WP7NM Pages, access to metric forecasts
Backend LDAP script to fetch metricsMonitor process to push metrics
localLDAPServer
Grid Application access viaLDAP Schema to- monitoring metrics; - location of monitoring data.
PingER(RIPE TTB)
IperfERUDPmon
rTPLNWS
etc
LDAPSchema
Grid AppsGridFTP
Network Monitoring Architecture
Robin Tasker
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
21
Network Monitoring Components
Ping Netmon UDPmon iPerf Ripe
Cronscript
plot
Table
LDAP
raw
control Cronscript
controlCronscript
plot
Table
LDAP
raw plot
Table
LDAP
raw
WEB Display AnalysisGrid BrokerPredictions
Web I/f
Scheduler
Tool
Clients
LDAP
raw
LDAP
raw
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
22
Ping & UDP throughput MAN-RAL From 20 Oct 01
PingER rtt (ms)
dl – RAL
1000 byte packet
Forecast
UDPmon Zero packet loss!
UDPmon throughput Mbit/s
man – RAL
300 * 1400 byte frames
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
23
Ping & UDP throughput MAN-CERNFrom 20 Oct 01
PingER rtt (ms)
dl – cern
1000 byte packet
Forecast
UDPmon throughput Mbit/s
man – cern
300 * 1400 byte frames
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
24
iperf TCP & UDP throughput MAN-SARA From 20 Oct 01
Iperf TCP throughput Mbit/s
ucl – sara
262144 byte buffer
Forecast
UDPmon throughput Mbit/s
man – sara
300 * 1400 byte frames
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
25
iperf & Pinger UK-Bologna From 20 Oct 01
Iperf throughput
ucl – Bologna
262144 byte buffer
Forecast in green
PingER rtt (ms)
dl – Bologna
1000 byte packet
Forecast
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
26Geant Enabled Routing Stable
Iperf ThroughputMbit/sUCL – SARA262144 byte
buffer
UDPmon Loss
ThroughputMbit/sMAN – SARA
iperf throughput UCL-SARA From 1 Nov 01 – Geant Operational
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
27
PingER deploymentLes Cottrell Measurements from
34 monitors in 14 countries Over 600 remote hosts Over 72 countries Over 3300 monitor-remote site pairs Measurements go back to Jan-95 Reports on RTT, loss, reachability, jitter, reorders, duplicates …
Countries monitored Contain 78% of world population 99% of online users of Internet
Lightweight (100bps/host pair) Very useful for inter-regional and poor links, need more intensive for high
performance & Grid sites
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
28
Losses: World by region, Jan ‘02 Packet loss <1%=good, <2.5%=acceptable, < 5%=poor, >5%=bad
Russia, S
America bad Balkans,
M East, Africa, S Asia, Caucasus poor
Monitored Region \ Monitor Country
BR (1)
CA (2)
DK (1)
DE (1)
HU (1)
IT (3)
JP (2)
RU (2)
CH (1)
UK (3)
US (16) Avg Region
Avg -(H
Avg NA + WEU + JP Pairs
COM 0.2 0.3 0.3 0.2 COM 0.27 23
Canada 1.8 1.6 0.3 0.5 9.0 0.3 1.4 21.7 0.7 0.7 0.5 3.5 Canada 0.74 126
US 0.4 2.6 0.2 0.3 8.0 0.1 1.4 13.8 0.3 1.3 0.9 2.7 US 0.88 2149
C America 0.9 0.9 C America 0.89 19
Australasia 0.8 1.8 1.3 Australasia 1.30 18
E Asia 1.2 3.5 1.0 1.1 9.0 0.9 2.0 5.2 1.5 1.4 1.5 2.6 E Asia 1.61 215
Europe 0.4 5.6 0.3 0.5 5.4 0.4 1.3 15.5 1.1 1.0 1.0 2.9 Europe 1.38 852
NET 1.7 6.2 1.0 1.3 8.0 1.6 3.6 21.9 0.7 0.8 0.9 4.3 NET 2.00 85
FSU- 4.5 0.5 9.8 0.5 1.6 11.2 4.3 1.2 2.0 4.0 FSU- 2.09 48
Balkans 3.8 3.8 Balkans 3.83 109
Mid East 4.6 1.4 3.0 8.5 2.8 3.2 11.8 2.0 2.5 2.1 4.2 Mid East 2.70 57
Africa 5.8 1.5 12.0 1.2 4.2 11.9 2.0 1.9 2.5 4.8 Africa 2.72 45
Baltics 5.3 0.8 2.3 7.7 2.2 3.5 10.8 4.8 2.1 3.9 4.3 Baltics 3.12 67
S Asia 1.6 7.3 0.1 3.1 9.2 3.0 3.9 17.9 1.5 3.1 3.0 4.9 S Asia 3.12 97
Caucasus 3.2 3.2 Caucasus 3.22 19
S America 24.1 11.3 0.6 0.9 6.7 12.9 7.7 23.0 9.3 1.1 6.6 9.5 S America 6.30 203
Russia 35.9 24.1 22.2 13.4 23.8 21.7 13.6 0.7 8.7 24.1 12.7 18.3 Russia 17.57 91
Avg 7.5 6.9 2.8 2.4 9.8 3.7 3.9 13.8 3.1 3.2 2.8 4.4 Avg 3.16
Pairs 64 144 54 67 70 203 190 114 209 192 1990 Pairs
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
29
Quality improvement seen
from SLAC & NASA
NASA results courtesy of Andy Germain, NASA, GSFC
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
30
Iperf mem-mem vs file copy disk to disk
0 400
100
Iperf TCP Mbits/s
File copy dis k-to- dis k
Fast Ethernet
OC3
Disklimited
Over 60Mbits/s iperf >> file copy
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
31
QoS: Terms and Concepts
Identifying frames – marking / setting IP precedence bitsSorting frames into queuesSelecting which frame to sendAction taken when a queue is full
Sort Fail
DiscardTest
Identify & ClassifyPolice
Dequeue
Configurable Queues
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
32
QoS: What Next?
Dante propose the following services: IP Premium (EF)
(AF) difficult to define in a way that suits most NRNs Best Efforts Scavenger “Less than best efforts”
UKERNA ran a Think Tank to Study QoS requirements in the UK MB-NG Network development project to test MPLS and QoS SuperJANET is expected to offer similar services to Dante
Applications need end to end QoS – so we need to cross: LAN SuperJANET4 Dante Remote NRN Remote LAN
Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester
33
More Information Some URLsPPNCG Home page with Stop Press:
http://ppncg.rl.ac.uk/
PPNCG Page for monitoring Astronomy & Astrophysics Sites http://icfamon.dl.ac.uk/ppncg/astronomy.html
and e-mail:[email protected]
DataGrid WP7 Networking: http://www.gridpp.ac.uk/wp7/index.html
IEPM PingER home site:http://www-iepm.slac.stanford.edu/
IEPM-BW site:http://www-iepm.slac.stanford.edu/bw