Network Connectivity and Projects – a Perspective from the UK

Richard Hughes-Jones PPNCG

SuperJANET4 StatusConnectivity to EuropeConnectivity to the USGrid Networking Projects

SuperJANET4: Backbone and Access links

Worldcom supplied the transmissionUKERNA layer the IP service

Core PoP IP router at WorldcomBackbone Access Router at MANs

Access Links:Large MAN 2.5 Gbit -> 10-20 GbitMedium MAN 622 Mbit -> 2.5 Gbit

4 node DWDM development netDeployment Status:

Backbone Oct 00 Routers Nov 00All sites Mar 01Proved to be StableConstant growth of traffic

SuperJANET4: ping rtt Core routers Jun 01

SuperJANET4: ping rtt Core routers

Lancaster Glasgow

MAN / LAN Issues

Bristol Cambridge

London MAN Upgrade

UDP ThroughputMbit/s

UDP Packet loss%

UDPmon TestsManchester – London

MAN was 155 Mbit ATM

1st Oct Time interval in Weeks Richard HJ

ICFAMON Plot from RAL to CERN for 19th Oct to 1st Nov 2001

Connectivity to Europe – Access links (1)

UK Access link 155 Mbit ATMSustained rate 130 MbitContract to end of Nov 01Bad news for users !

Traceping Oxford to CERN for 31st October 2001 Connectivity to Europe – Access links (2)

loss around ten155-gw.ja.net router

J. Macallister

Connectivity to Europe : GeantStart mid November 2001UKERNA switched off TEN-155 3 Dec 2001Monitoring – see later

Connectivity to US

6 * 155 Mbit links

US link about to go to 2.5 G bitIP commodity attached in London Research traffic over 2.5G bitPeer in Hudson St. 622 Mbit to Esnet622 Mbit to Abilene.

Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US17 Jul 2000 (300kbit/s) 7 Dec 2001 (900kbit/s)

peak is 88% of total BW 930 Mbit10 minute averages 10 minute averages

Last 7 days 1 hour averages Last 7 days 1 hour averages

Monitoring: US Traffic - TrendsJan 99 – Nov 02

UKERNA – US Traffic Daily totals Mbytes

Weekend

Xmas 99

Weekday

Xmas 01

Peak load ~88 %

J. Macallister

Grid Network Projects

DataGridGridPP

Extending DataGrid to BaBar, D0, CDF,… in UKDataTAG

Foci: Network resarch, Grid inter-operability MB-NG

UK funded 2.5Gbit Development NetworkOptical Switching

Network MonitoringSeveral tools in test – plugged into a coherent

structure: PingER, RIPE one way times, iperf, UDPmon, rTPL, GridFTP,

and NWS prediction engine continuous tests for last few months to selected sites:

DL Man RL UCL CERN Lyon Bologna SARA NBI SLAC … The aims of monitoring for the Grid:

to inform Grid applications, via the middleware, of the current status of the network – input for resource broker and scheduling

to identify fault conditions in the operation of the Grid to understand the instantaneous, day-to-day, and month-by-month

behaviour of the network – provide advice on configuration etc.Report written on LDAP scheme for publishing the

network informationWill be used by UK e-science sites and non HEP WPs

UDPmon Measuring Throughput with UDP

Send a burst of UDP frames spaced at regular intervalsVary the frame size and the frame transmit spacingRecord

The time to send and the time to receive the frames The number received, the number lost, number out of order The received inter-packet spacing

Zero stats

OK done

●●●

Get remote statistics

Send statistics

Send data frames atregular intervals ●●●

Time to send

Time to receive

Inter-packet time

Local NetworkMonitoring

Store & Analysisof Data (Access)

Access to current and historic dataand metrics via the Web, i.e. WP7NM Pages, access to metric forecasts

Backend LDAP script to fetch metricsMonitor process to push metrics

localLDAPServer Grid Application access via

LDAP Schema to- monitoring metrics; - location of monitoring data.

PingER(RIPE TTB)

IperfERUDPmon

rTPLNWS

etc

LDAPSchemaGrid Apps

GridFTP

Network Monitoring Architecture

Robin Tasker

Network Monitoring Components

Ping Netmon UDPmon iPerf Ripe

Cronscript

plot

Table

LDAP

raw

control Cronscript

controlCronscript

plot

Table

LDAP

raw plot

Table

LDAP

raw

WEB Display AnalysisGrid BrokerPredictions

Web I/f

Scheduler

Tool

Clients

LDAP

raw

LDAP

raw

Ping & UDP throughput MAN-RAL From 20 Oct 01

PingER rtt (ms) dl – RAL 1000 byte packetForecast

UDPmon Zero packet loss!UDPmon throughput

Mbit/s man – RAL300 * 1400 byte frames

Ping & UDP throughput MAN-CERNFrom 20 Oct 01

PingER rtt (ms) dl – cern 1000 byte packetForecast

UDPmon throughput Mbit/s

man – cern300 * 1400 byte frames

iperf & UDP throughput MAN-CERN From 20 Oct 01

Iperf throughput ucl – cern 262144 byte bufferForecast

UDPmon throughput Mbit/s

man – cern300 * 1400 byte frames

iperf & UDP throughput MAN-SARA From 20 Oct 01

Iperf throughput ucl – sara 262144 byte bufferForecast

UDPmon throughput Mbit/s

man – sara300 * 1400 byte frames

iperf & Pinger UK-Bologna From 20 Oct 01

Iperf throughput ucl – Bologna 262144 byte bufferForecast in green

PingER rtt (ms) dl – Bologna1000 byte packetForecast

iperf throughput UCL-SARA From 1 Nov 01 – Geant Operational

Iperf throughput ucl – sara 262144 byte buffer

Geant Enabled Routing Stable

Brief outline of work plan

High Performance High Throughput: Understanding the end system HW, best way to monitor traffic and protocol packets, Type and effect of background traffic, Throughput vs rtt, throughput vs window-size, Throughput using multiple TCP streams, and effect on the Network GridFTP and BBftp at Gigabit, Effect of different TCP stacks, Use of non-TCP protocols, and effect on the Network Show and tell demos

Bandwidth Reservation and QoS Identification of traffic classes and middleware components

requiring QoS Study of traffic differentiation and traffic engineering techniques demonstration of QoS & Bandwith Reservation

The EU DataTAG project

EU Transatlantic Gigabit project.Status: approved for 4 M ECU start on 1 Dec 2001Partners: CERN/PPARC/INFN/UvA. IN2P3 sub-

contractorThe main foci are:Grid Network Research including:

Provisioning (CERN) Investigations of high performance data transport (PPARC) End-to-end inter-domain QoS + BW / network resource

reservation Bulk data transfer and monitoring (UvA)

Interoperability between Grids in Europe and the US PPDG, GriPhyN, DTF, iVDGL (USA)PPDG, GriPhyN, DTF, iVDGL (USA)

DataTAG project

Abilene

ESNET

MREN

STAR-TAP

UKSuperJANET4

ITGARR-

B

GEANT

NewYork

Olivier Martin

NLSURFne

t

2.5 Gbit lambda between CERN and StarlightPOS 2nd half 2002 – WDM later

STAR-LIGHT

CERN

2.5 Gbps SDH “Lambda”100/1000 Mbps Ethernet

Amsterdam

GSR

GSRGSR

GSR

GSR

λ Broker

Application

CERN

StarLight

Italy

UK

Optical Switching - DataTAG

7600

IwireAbileneESnet

MB – NG SuperJANET Testbed

MB - NGSuperJANET4ProductionNetwork

Leeds

RAL /UKERNA

RAL /UKERNA

SJ4 DevC-PoP

Reading

SJ4 DevC-PoP

Warrington

Manc MCC

ULCC UCL

SJ4 DevC-PoPLondon

Gigabit Ethernet2.5 Gbit POS10 Gbit POS

E-science core projectMB - NG

Project to investigate and pilot: end-to-end traffic engineering and management over multiple

administrative domains – MPLS in core diffserv at the edges. Managed bandwidth and Quality-of-Service provision. (Robin Tasker) High performance high bandwidth data transfers. (Richard HJ) Demonstrate end-to-end network services to CERN using Dante EU-

DataGrid and the US DataTAG. Status: approved with requested funds start on 1 Dec 2001 Partners:CISCO, CLRC, Manchester, UCL, UKERNA plus

Lancaster and Southampton (IPv6) A technical meeting held – draft spec. of the HW required Would like to use real Grid traffic – as demonstrator:

CDF UCL-RAL BaBar Man-RAL

And Finally in case it helps:

GridPP and DataGrid have been working on packaging Globus v2 as a .rpm kit.Makes installation simpler.Available from DataGrid WP6 web: marianne.in2p3.fr/datagrid/testbed1/globus

Conclusion & Status

SuperJANET4 stable working well Access links to Campus becoming very inportantNetwork Links and peering to US good overallUS-UK bandwidth upgrade overdue Geant access to Europe OK (at the moment !)Considerable UK interest in the Internet2 tests Grid: Lots of interesting work to do …