ASCAC Presentation

James F. LeightonESnet Project Manager

Lawrence Berkeley National Laboratory

May 3, 2001

AnESnet Project

Overview

ASCAC Presentation

ESnet: In Support of DOE Science

• Mission:• Provide, interoperable, highly capable and reliable

communications infrastructure and leading-edge network services that support DOE’s missions

• Vision:• Provide seamless and ubiquitous access, via shared

collaborative information and computational environments, to thefacilities, data, and colleagues needed to accomplish their goals.

• Role: • A component of the Office of Science infrastructure critical to the

success of its research programs.

ASCAC Presentation

ESnet: What Is It?

• A nation-wide high-performance “agency mission” network— chartered to provide advanced network services to

support scientific research in DOE• Centrally funded by DOE/Office of Science

— operated and managed by project staff at LBNL• An immense (and very successful) cooperative effort• Was rated both “outstanding” and “extremely cost effective”

at the last formal Program Review• Has an extensive structure of domestic (commercial and

R&E) and international interconnects• Has a growing Advanced Technology and Research

program

ASCAC Presentation

ESnet: What Is The User Base?

• 10,000-100,000 researchers in U.S. use ESnet (guestimate)• Mostly Office of Science programs: HEP, NP, FES, BES, BER, MICS• Traffic also carried for DP and others• Involves essentially all U.S. national labs• Hundreds of universities• Hundreds of foreign institutions• Large and small collaborations (from a few to almost 2000 members)

CHARACTERISTICS OF USER BASE• Many (unaware) casual users• Variety and breadth in science• Size and geographic distribution of research community• Many data intensive and computationally intensive tasks supported

ASCAC Presentation

ESnet:An Enabler for DOE Science

• We've seen a dramatic change in the way science is done• Increase in scientific productivity - much shorter turn around time for

disseminating, assimilating and testing new ideas• Innumerable meetings via remote conference technologies• This DOE community has been a leader in use of the network for science• Network planning and deployment based on program requirements and

technological opportunities - push/ pull

ASCAC Presentation

ESnet:Network Usage

• Basic services - email, file transfer, remote login, distributed file systems• Teleconferencing - integral part of work flow, planning, coordination• Remote access to unique facilities

– experiments– supercomputers– databases– installed codes

• Collaboratories– Traditional approach - visit and/or relocate– New approach - virtual laboratories, remote participation

• Distributed computing - Grid Computing

ASCAC Presentation

ESnet:Research Life Cycle Support

• Planning and coordination• Design of experiments by distributed teams• Engineering/analysis• Online documentation• Remote Participation• Distributed code development• Distributed data analysis and

visualization

ASCAC Presentation

ESnet: A Very Brief History

• One line of ancestry can be traced back to ‘74 dial-up access to CTRCC• MFEnet, HEPnet (I.e. DECnet) & ARPAnet development continued over

’75-’85 timeframe• Memo signed in Oct, 1986 to create multi-program “ESnet”• 1987-1990: The “sociology of DOE-wide networking” barrier broken• Dec ’89: ESnet DECnet turned on• Feb ’90: Log shows security problem with hacker• Mar ’90: All T1 trunks and routers for “ESnet (1)” on-line• Feb ’92: ESnet (2) RFP released

— Jun ’94: 3 T3s turned on (LLNL-LANL, LANL-FNAL, FNAL-PPPL)— Aug ’94: Sprint master contract signed for fast-packet service

• 1995-1998: T3, OC3, and OC12 ATM connections brought on-line• Nov ’95: Decision to move ESnet and NERSC to LBNL announced• Jun ’99: ESnet (3) RFP released

— Dec ’99 Contract with Qwest signed— And a major transition is underway

ASCAC Presentation

ESnet: It’s the Sociology that’s Hard

ProgramOffices

DOE

OSC

PROGRAMS

OASCR

MICS

ESSC

ESCC

WG&TFWG&TF

WG&TFWG&TF

ESCRC

ESnetProject

• The ESnet project enjoys an excellent working relationship with both it’s technical and program “user”communities

The ESnet Steering Committee (ESSC) deals with requirements and priorities as established by DOE representative Program Principal Investigators.The ESnet Coordinating Committee (ESCC) deals with associated site and technical issues.

ASCAC Presentation

ESnet: The FY01 Budget

• The ESnet project is centrally funded by the MICS office in DOE/SC

$6.42M Base budget6.92 Communications1.19 International.35 DCS

1.00 Testbed&Research.89 Equipment

-----$16.77M FY01 MICS Funding

Additional funds are collected for special project/programsupport (mostly pass-through) now at roughly $2.5M/yr

ASCAC Presentation

ESnet: Staffing

Career ContractEngineering Services Group: 6.0Information & Services Group: 4.0 Technical Services Group: 10.0 1.0Operations : 2.0Admin Support: 1.0Management : 2.0

--- ---23.0 3.0

ASCAC Presentation

ESnet: StaffingESnet Monthly TrafficCompared to Budget

TER

AB

YTES

/Mon

th

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

45.00

Jan.

90

May

. 90

Sep.

90

Jan.

91

May

.91

Sep.

91

Jan.

92

May

. 92

Sep.

92

Jan.

93

May

.93

Sep.

93

Jan.

94

May

. 94

Sept

. 94

Jan.

95

May

. 95

Sep.

95

Jan,

96

May

, 96

Sep,

96

Jan,

97

May

, 97

Sep,

97

Jan,

98

May

, 98

Sep,

98

Jan,

99

May

, 99

Sep,

99

Jan,

00

May

, 00

Sep,

00

Jan,

01

E S net B udget

0

5000

10000

15000

20000

25000

FY90 FY91 FY92 FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01(P lan)

FY02(P lan)

Fiscal Y ear

$ in

$10

00E S net B udget

0

5000

10000

15000

20000

25000

FY90 FY91 FY92 FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01(P lan)

FY02(P lan)

Fiscal Y ear

$ in

$10

00

ASCAC Presentation

ESnet: Qwest Contract Overview

• Procurement for “Umbrella Services Contract”—Includes three major components

• advanced services and technology for production network• high-performance test-bed• research collaboration

—Multi-year contract (3+2+2 years), $50M+—Will overlap with existing Sprint contract

• nearly 2 year overlap possible, but not expected• includes very competitive ATM pricing

—Contract signature Dec ‘99• Transition planning & implementation underway• First sites went on-line in August, 2000• Approximately 90% done with initial transition

ESnet (Qwest) BACKBONE Apr 2001

ESnet (ESnet (QwestQwest) BACKBONE ) BACKBONE Apr 2001Apr 2001

NTON OC12OC48 ATMOC12 ATMOC3 ATMT3 ATMT1-T3 ATMT3T1-T3T1<T1Sprint Connected

6 Apr, 2001

JAPANJAPAN

TWC

TWC

TWC

JGIJGIJGI

SNLLSNLLSNLL

LBNLLBNL

SLACSLAC

Fix/MaeWest

Fix/MaeWest

Fix/MaeWest

Fix/MaeWest

YUCCAMT

YUCCAMT

YUCCAMT

BECHTEL

BECHTEL

BECHTEL

LLNLLLNL

SNV HUBSNV HUBSNV HUB

SAICSAICSAICGAGAGA

PNNLPNNLPNNLLIGOLIGOLIGOLIGO

INEELINEELINEELINEEL

LANLLANLLANL

SNLASNLASNLAALBHUBALBHUBALBHUBALBHUB Allied

SignalAlliedSignalAlliedSignalAlliedSignal

PANTEX

PANTEX

PANTEX

PANTEX

ARMARMARM

AlliedSignalAlliedSignalAlliedSignalAlliedSignal

NOAANOAANOAANOAAOSTIOSTIOSTIOSTI

ORAUORAUORAUORAUOROOROOROORO

SRSSRSSRSSRS

ORNLORNLORNL

JLABJLABJLAB

MAE-EastMAE-EastMAE-EastMAE-East

PPPLPPPLPPPLGTNGTNGTN

ANL-DCINEEL-DCANL-DCINEEL-DCANL-DCINEEL-DC

MITMITMIT

ANLANLANL

BNLBNLBNL

FNALFNALFNAL

AMESAMESAMES

CHI NAPCHI NAPCHI NAPCHI NAP

CHI HUBCHI HUBCHI HUBCHI HUB

NY-NAPNY-NAPNY-NAP

NevisColumbia

Yale

NevisColumbia

Yale

NevisColumbia

Yale

PAIX-WPAIX-WPAIX-W

RUSSIARUSSIA

QWESTQWESTATMATM

QWESTQWESTATMATM

ATLATLATLATL

4xLAB-DC4xLAB-DC4xLAB-DC

Brandeis

Brandeis

Brandeis

ITALYITALY

GERMANYGERMANY

UKUK

DANTEDANTE

NY H

UB

NY H

UB

NY H

UB

CHINACHINA

RUSSIARUSSIA

NLNLISRAELISRAEL

SINGAPORE

TAIWANTAIWANCANADACANADA NORDUNETNORDUNET

FRANCECERN

SEA HUBSEA HUBSEA HUB

SNVSNVSNV

SNVSNVSNV

SNV

SNV

SNV

SNVSNVSNV

ORN HUBORN HUBORN HUB

ORNORNORN

ORNORNORN

ORN

ORN

ORN

ALB

ALB

DC HUBDC HUBDC HUBNERSCNERSC

NY

NY

NY

KEKKEK

ORNORNORN

TECHnet

ESnetESnetDirect IP NeighborsDirect IP Neighbors

EARLY 2001EARLY 2001

JAPAN

VerioVerio

RussiaRussia

AbileneCalRENUCBUCnet

AbileneCalRENUCBUCnet

ITALYITALY

CHI-NAP60 PeersCHI-NAP60 Peers

MAE-East51+8 PeersMAE-East51+8 Peers

GermanyGermany

GWUNetwork VAvBNS

GWUNetwork VAvBNS

CHINACHINA KEKKEKRUSSIARUSSIA

NLNLISRAELISRAEL

SINGAPORE

TAIWANTAIWAN

NY-NAP38 PeersNY-NAP38 Peers

CANADACANADA NORDUNETNORDUNETFRANCE

AbileneSOXAbileneSOX

CalRENSAICSDSCvBNS

CalRENSAICSDSCvBNS

FIX/MAE-West40+6 PeersFIX/MAE-West40+6 Peers

MIX-WestAbileneDRENEI-AXExodusIBM GlobalICMLevel3Lightening InetNRENQwestSprintLinkTeleglobeVerio

MIX-WestAbileneDRENEI-AXExodusIBM GlobalICMLevel3Lightening InetNRENQwestSprintLinkTeleglobeVerio

LBL

LBL

GACGAC

LANLLANL

FIX/MAE-WFIX/MAE-W

CHI-NAPCHI-NAPCHI-NAPSTAR-TAPSTAR-TAPSTAR-TAP

ATLATLATL

MAE-EMAE-EDC-HubDC-Hub

PPPLPPPLNY-NAPNY-NAP

1 Feb, 2001

UKUK

DanteDante

NY-HUBNY-HUB

AbileneAbilene

SNV

SNV

CHICHI

CERN

ASCAC Presentation

ESnet: Collaboration Services

• Began in ’90-91 with HEP use of “excess” bandwidth on T1 trunks

• Now grown to the point that a “typical” month includes over 1200 conferences at ~2 Hours average, supporting roughly 100 conference-days of collaboration per month!

• DCS Resources now include:• Accord ISDN MCU (48 ports)• Latitude Audio Bridge (76 ports)• A web based reservation system (DCS)

• A commercial replacement for the DCS scheduler is being evaluated

• H323 (IP Video) support and interoperation with H320 (ISDN Video) now being researched and tested for system integration

All Conferences - March, 2001Number of Conferences 1279Conference Hours 2440Audio Bridge Port Hrs 5137MCU Port Hours 4155Room Hours 5529

ASCAC Presentation

ESnet: Security Considerations

• ESnet supports unclassified activity only• ESnet security responsibility falls under the auspices of the LBNL CPPM• ESnet has it’s own Project Security Officer

— point-of-contact for ESnet Project to LBNL CPPM• ESnet has a defined AUP• ESnet’s security responsibilities:

— cover ESnet resources — end at the site demarc!

• ESnet’s security requirements cover:— network utilization— component access— user services— office environment

THE ESNET

UNCLASSIFIED-SECURITY PLAN

1 Feb, 2001

ASCAC Presentation

ESnet: Research & Adv. Technology• Multiprotocol Routers

— IPv4, DECnet, OSI, X.25 at one point

• ATM— Industry catalyst for carrier deployment— First production nation-wide WAN deployment

• Advanced Protocols-one of earliest adopters of:— CIDR— BGP-4— MBGP— IPv6:

• QoS— Queue management research— Policing (CAR)— VoIP (application)

• Testbed (planned)

• PKI/Directory (proposed)

ASCAC Presentation

ESnet: Future Initiatives as Drivers

• Nano-technology• Computational materials science

network• LHC• RHIC detectors come fully

on line• Burning plasma experiment• SciDAC• SNS• Post-genome• Network research initiatives

ASCAC Presentation

ESnet: Critical Issues for the Future

• Keep ahead of the curve on domestic connectivity• Closely monitor quality of connections to universities• Continue to improve International links• Deployment and

support ofadvanced services

ASCAC Presentation

Summary

• The ESnet project thrives in an environment that— is experiencing phenomenal growth in usage— has an extremely rapid pace and broad spectrum of technology— Has simultaneous user demand for performance & reliability— is extraordinarily dependent upon wide-scale collaboration and

interaction with “peers”

• The program is highly service oriented and enjoys the trust, confidence, and support of both its technical and end user community

• ESnet is a highly successful, effective, and cooperative effort in meeting the networking and data communications requirements of the agency science research community.

ASCAC Presentation

The ESnet Project

Thank YouFor

Listening

ASCAC Presentation

The “Transition”

ASCAC Presentation

ESnet: Internal Measurements

Include monthly, by site, mrtg

We maintain an extensive set of performance and traffic measurements

ASCAC Presentation

ESnet: Outside Measurements

Loss seen from Esnet to groups of Sites

ITU limit for lossITU limit for loss

ASCAC Presentation

ESnet: Other Services

• NTP• PGP Key Server

— over 1 million keys now— roughly 10K updates/day

• Multi-cast Backbone – now native mode on backbone• Secondary DNS• Certificate Authority, Certificate Server• Directory Services• Meeting and conference networking support

What Makes It “Fun”? (1/2)

• Moving 35+ sites without a break in service

• Economics1) Service overlap, e.g. overlap between hubs2) Termination charges3) Bridging between vendors

• Trying to forecast install dates for planning

• Every site seems to have its unique set of issues—LBL – city of Berkeley wants a franchise—ANL – required an environmental impact study—NY – power shortage in collocation bldg—NERSC – new site, operational date before circuit can be

installed

What Makes It “Fun”? (2/2)

• Hardware differences— Juniper vrs Cisco routers— Cisco vrs FORE ATM switches

• Topology differences— Hub routers vrs full BGP mesh

• Interoperability (ESnet <> ESnet3)— Tested CLIP, OSPF, BGP, MBGP, MSDP, PIM-SM

• Non-persistent bridge sites— Sites bridging between ESnet and ESnet3 will change

during the transition• But – we’re nearly there!

SNV

ALB

ORN

NYCCHI

LANL

SNLA

BNL(TELEHOUSE)

OC48-ATMOC12-ATMOC3-ATM

OC48-SONET

OC3-SONET

T3-ATM

T3

MIT

CHI-NAPPPPL

ORNLATL

SRS

ANLFNAL

AMES

PANTEX

JLAB

GTN

ASIG

T1

DC

(NY-NAP)

SNLL

LLNL

LBNL

NERSC

(FIX-W)

(PB-NAP)

(MAE-W)

OC12-SONET

SLAC

(MAE-E/ATM)

(GA)

JGI

(PNNL)

GA

(SDSC)

YUCCA-MT

(BECHTEL)

ESNET3 INITIAL CONFIGURATIONTop Level View – Qwest Access

SEA

(SAIC)

INEEL

(site) non-Qwest

ANL DCINEEL DC

4X-DC

KEK

QWESTATM

ESnet

ATM

*

***

****

* * ****

*

*

*

**

*

*

*

#

*

*

#

*

***

Installed-finalInstalled-temp#

**

20 Apr, 2001

* LLNL DC**

**

**

**

PAIX-W

*

#

#

*

**

PAIX-W* *

*

*

#

#

#

#

###

#

(NREL) *

*

#

ASCAC Presentation

ESnet: Staffing Analysis

Career ContractEngineering Services Group: 6.0Network engineering, 3rd level problem resolution (24x7), measurements & analysis, new protocols & technology system integration, documentationInformation & Services Group: 4.0Security, DCS development, testing, operation (8x5), grid servicesTechnical Services Group: 10.0 1.02nd line problem resolution, installation coordination, security, desktop & server operation & system administration, Operations: 2.0Monitoring & 1st line problem resolution (24x7)Admin Support: 1.0Management: 2.0

--- ---23.0 3.0