r. hughes-jones esnet meeting kyoto july 2000 networking for hep in the uk uintroduction to the...

R. Hughes-Jones ESnet Meeting Kyoto July 2000

Networking for HEP in the UK

Introduction to the PPNCG Network TopologiesMonitoring ActivitiesPeering to the US and CanadaGrid Work SuperJANET4


Introduction to the PPNCGMembership includes HEP and Astronomy users

–Ensure the community has the required networking facilities–Monitor end-to-end performance–Investigate new network applications / technologies–Provide advice on kit / facilities– URL icfamon.rl.ac.uk/ppncg/title.html

Acts as an Interface between HEP and UKERNA–Collaboration : Multicast, H.323, QoS tests, SuperJANET4

Active Network Monitoring–Uses the ICFA monitoring tools and the PPNCG traceping and ftp –Monitor MANs / SuperJANET & the networks to CERN, DESY, US–Report problems to UKERNA


SuperJANET Connections in the UK

Warwick

Lancaster

TrinityAfan

Merthyr

CoventryTechnical College

Northern College

Leeds

Bristol

Londo

n

Manch

ester CW

C

O-BEN

R-BEN

Nott-BEN

Le-BEN

Ncl-BEN

Ed-BEN

C-BEN

ULCC-BEN

Cf-BEN

Be-BEN

Gla-BEN

Ex-BEN

Bham-BEN

Bris-BEN

155 Mbps

34 Mbps

CWC

Soton-BEN

Rdg-BEN

BW

EM

AN

MidMAN

LMAN

EMMAN

NorMAN

YHMAN

SWAN

SWMAN

FaTMAN

AbMANClydeNET

HomertonHinxton Hall

ITE Monkswood

BAS

Anglia Poly

Cambridge

RAL

Brookes

Oxford

Portsmouth

BEN

PoP Site

MAP

Birkbeck

UCLIOE

SOAS RV

CLSHTM

Southbank

LondonInstitute

LondonGuildhall

Imperial

Westminster

Kingston

Roehampton

LBS

RCA

Birmingham

Westhill

Newman

Coventry

UCEBWolverhampton

AstonQMW

UEL

Telehouse

Goldsmiths

ULCC

Swansea

Carmarthen

Lampeter

Soton

Nottingham

Leeds

Greenwich

Ravensbourne

St Andrews College

Glasgow

Exeter

Plymouth

Dundee

Cardiff

Bristol

Belfast

St Mary

Stranmill

BGS

Bath

Bath College

HEFCE

UWE

RobertGordon

AberdeenSt Andrews

Abertay

ULCC

NewportWCM

DSwansea Institute

Glamorgan

St Davids

Cardiff Institute

GlasgowCaledonia

Strathclyde

RSAMD

GSA

Paisley

EaStMAN

Edinburgh

Napier Queen Margaret

Stirling Heriot

Watt

ECA

Sunderland

Northumbria

DurhamUNN

Teesside

Bishop GrossetesteNott

Trent

Derby

Huddersfield

Sheffield

Linc & Humb

Sheffield Hallam

Leeds Metropolitan

Hull

Scarborough

Bradford

York

MCC-BEN

MCC

NEWI

Bangor

G-MINGSalford

Manchester Metropolitan

RNCM

UMIST

Liverpool

LJMU

NNW

Central Lancaster

Daresbury PO

Lab

ITE Bangor

Keele

Bolton Inst.

Liverpool Hope

Staffordshire

Edge Hill

Sussex Brighton

Edinburgh

UHI

Chichester

Yale

ColegLlysfasi

Deeside

WCoH

Swansea CollegeBridgen

d

Gorseinon

Aberdare

Neath

Pembroke

Llandrillo

Menai

Aberystwyth

HEAnet

Ulster

Harlech

Powys

Meirion-Dwyfor

Ceredigion

MST Radar

Falmouth

Dartington

BGSExeter

St Mark & John

BucksUKERN

A

C&G

Seeseparate diagram

See separate diagram

Bournemouth

Royal Holloway

CWC

Surrey

Reading

Cranfield

UEA

Essex

U-Net

Leicester

UC Northampton

UCL

CWC

Mid KentCollege

Stephenson

ChichesterCollege

Soton City

BCFTCS

LCC

City CollegeManchester

Halton

Newcastle

Newcastle

CarlisleCollege

Cleveland CollegeNew College Durham

Southport

SandwellWorcester Coll.


4 * 155 Mbit links to SuperJANET

2 * 155 Mbit transatlantic links 3rd link in use 18 Jul 00 4th planned by End of 00

SuperJANET Connections in London


UK to Europe TEN-155 April 2000

Peering with CANARIE madein London, uses 10 Mbit of theDante US line for all Europe


Europe and ESnet in the US


Monitoring: US Traffic

UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US10 Nov 1999 17 Jul 200010 minute averages 10 minute averages

Last 7 days 1 hour averages Last 7 days 1 hour averages


Monitoring: US Traffic - Trends

Traffic Kbit/s. Blue Traffic from US Maroon Traffic to US10 Nov 1999 9 Jul 2000

10 minute averages

UKERNA – US Traffic Daily totals Mbytes

Weekend

Xmas 99

From US To US

Weekday


Monitoring: ping plots UK - US Ping average round trip time & Lost packet % for 100 byte packets icfamon.dl.ac.uk – ns2.slac.stanford.edu

13 Oct 99 27 Jun 00


Monitoring: traceping UK - US Mar 99

John Macallister PPNCG Oxford


Monitoring: traceping UK - US Oct 99


Monitoring: traceping UK - US Jul 2000


Monitoring: ping plots UK - CERN & DESYPing average round trip time & Lost packet % for 100 byte packetsicfamon.dl.ac.uk – rsplus.cern.ch icfamon.dl.ac.uk – x4u2.desy.de

27 Jun 00 27 Jun 00


Peering with ESnetUsed CAR and WRED to give priority to Esnet traffic into JANET.Rtt and packet loss for 100 byte packets.Tests DL to SLACStart 21 Feb 00

CAR+WREDenabled

Config changes toCAR and WRED

Further configchanges

High Priority

Normal Priority

Robin Tasker PPNCG DL


Peering with ESnet and Abilene rtt and packet loss for 100 byte packetsPlot start 16 May 00

ESnetHigh PriorityDL - SLAC

AbileneHigh PriorityDL - Stanford

Further configchanges

Change in bandwidthrestrictions from Abileneinto JANET


The Grid

Government has discussed 165M pounds to develop Computing Grid Infrastructure.Across Research Councils Initiative – not just HEP Setting up management structure to co-ordinate PPARC activities.Have run Globus workshop so US developers could inform the community.Several sites now working with Globus toolkit.Web site www.datagrid.org.uk


The Grid

Keen Interest in the EU DataGrid (CERN) middleware proposal.HEP experiments will need world Grid access.Grid Networking interests include:

–High throughput, reliable, very large data transfers–QoS issues – diffserv, RSVP and packet marking–Monitoring, Metrics, and prediction accuracy–Modelling

Physics Applications


The Grid: PAW with Globus

Core ServicesMetacomputing

Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf

High-level Services and Tools

DUROC globusrunMPI Nimrod/GMPI-IO CC++

GlobusView Testbed Status

GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit

Linux GASS Server

Sun GASS Server

SGI GASS Server

CampusLAN

Zebra

HBOOK


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP


Directory Service

GRAMGlobus

Security Interface

Heartbeat Monitor

Nexus

Gloperf




GASS

Local Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDPLocal Services

LSF

Condor MPI

NQEEasy

TCP

SolarisLinuxSGI

UDP

Toolkit

PAWNext Steps:•Working on “root”•Brokers•Data Location•Security + Global farms

BaBar Tau Analysis

Andrew McNab Manchester


ATM MAN 7 routers ~0.8ms / router Router+link 0.4 s/byte DL-Liverpool ~10 ms rtt - agreement

The Grid:Measurements on SuperJANET

64 bytes pc-SJ-DL req-resp

0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)


0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)


0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)1500 bytes pc-SJ-DL req-resp

0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)


0

100

200

300

400

500

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

Sigma ~1.2 ms

y = 2.2589x + 5643.6

y = 2.1519x + 5533.9

y = 0.107x + 109.71

0

2000

4000

6000

8000

10000

12000

0 500 1000 1500 2000 2500 3000

Message length bytes

La

ten

cy u

s

0

50

100

150

200

250

300

350

400

450

500

La

ten

cy f

or

PC

=P

C u

s

PC-SJIII-DL PCdt PC-SJIII-DL PCPC=PC

Latency Mancester - DL

Richard Hughes-Jones PPNCG Manchester


The Grid: Latency on SuperJANET

64 bytes pc-SJIII-sun req-resp

0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)


0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)


0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency usN

(t)


0100200300400500600700

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)


0

100

200

300

400

500

600

10000 15000 20000 25000 30000 35000 40000 45000

Latency us

N(t

)

64 bytes pc-MC req-resp

0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500

N(t

) g

w-m

c


0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

N(t

) g

w-m

c


0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500

N(t

) g

w-m

c


0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500

3000

N(t

) g

w-m

c1024 bytes pc-MC req-resp

0

100

200

300

400

0 5000 10000 15000 20000 25000 30000

Latency us

N(t

)

gw

-me

tro

0

500

1000

1500

2000

2500N

(t)

gw

-mc

Pink: LAN Blue: MAN-SJ CORE Pink: Man-RAL MB Blue: Man-RAL ~5 s nice exponential ~ 2 ms 4.5 -> 6.7 ms


The Grid: Throughput on SuperJANET

Manc – UCL 10 Mbit Manc – Glasgow

0

2

4

6

8

10

12

0 200 400 600 800 1000 1200 1400

Transmit Time/packet us

Re

cv W

ire

ra

te M

bit

s/s

100 bytes 200 bytes400 bytes 800 bytes600 bytes 1000 bytes1200 bytes 1400 bytes

0

5

10

15

20

25

30

35

40

45

50

0 200 400 600 800 1000 1200 1400


Re

cv W

ire

Ra

te M

bit

/s

100 bytes 200 bytes400 bytes 600 bytes800 bytes 1000 bytes1200 bytes 1400 bytes

Manc – RAL 24 Mbit link Manc – RAL 2 Mbit Managed BW

0

5

10

15

20

25

30

35

0 500 1000 1500 2000 2500


Rec

v W

ire

Rat

e in

clud

ing

ATM

Mbi

t/s

100 bytes 200 bytes

400 bytes 600 bytes

800 bytes 1000 bytes


0

0.5

1

1.5

2

2.5

0 500 1000 1500 2000 2500 3000Transmit Time/packet us

Re

cv W

ire

Rat

e in

clu

din

g A

TM

Mb

it/s

100 bytes 200 bytes

400 bytes 600 bytes




The Grid: Multi-stream ftp Oxford to SLACSingle stream 8 ftp streams ~ 0.8 single stream

John Macallister PPNCG Oxford


The Grid: Multi-stream ftp SLAC to Oxford


The Grid: Multi-stream ftp SLAC to Oxford

25 min


SuperJANET4

SuperJANET4 Topology based on Worldcom Trident NetworkBackbone initially SDH based

2.5 Gbit (STM-16)Upgrade to 10 Gbit in 2001,

20 Gbit in 2002Migration from SDH to DWDMTechnology review in 2003


SuperJANET4: Backbone and Access links

Worldcom supply the transmissionUKERNA layer the IP

–Core PoP IP router at Worldcom–Backbone Access Router at MANs

Access Links:–Large MAN 2.5 Gbit -> 10-20 Gbit–Medium MAN 622 Mbit -> 2.5 Gbit

4 node DWDM development netDeployment:

–Backbone delivery Oct 00–Routers + acceptance Nov 00–Manc-ULCC traffic end Dec 00–All sites Mar 01


Conclusion & Status

Network Links to US and Europe are OK (at the moment !)Peering with Abilene, ESnet and CANARIE OK, interesting effects with CAR – WRED !Considerable UK interest in the Internet2 tests Grid: active in several areas including networksSuperJANET4 rollout is very promising

r. hughes-jones esnet meeting kyoto july 2000 networking for hep in the uk uintroduction to the...

Documents

us slide

esnet traffic

us weekday slide

traceping uk us

us maroon traffic

ukerna slide

esnet high priority

europe slide