OptIPuter

Larry Smarr, PITom DeFanti, Jason Leigh, Mark Ellisman, Phil Papadopoulos, Co-PIs

Maxine Brown, Project Manager

Knowing the User’s Bandwidth Requirements

DSL GigE LAN

C

A

B

A -> Need full Internet routing

B -> Need VPN services on/and full Internet routing

C -> Need very fat pipes, limited multiple Virtual Organizations

Source: Cees de Laat, UvA

Number of users

Bandwidth consumed

The OptIPuter is a Distributed “Infostructure” forData-Intensive Scientific Research and Collaboration

“A global economy designed to waste transistors, power, and silicon area – and conserve bandwidth above all – is breaking apart and reorganizing itself to waste bandwidth and conserve power, silicon area, and transistors.”— George Gilder, Telecosm (2000)

The OptIPuter: A Philosophy• Bandwidth is getting cheaper, faster than storage.• Storage is getting cheaper, faster than computing.• The exponentials are crossing – we are moving from a processor-

centric world, to one centered on optical bandwidth, where the networks will be faster than the computational resources they connect.

The OptIPuter: A Paradigm• Supercomputers maximize computing, and minimize bandwidth use.• The OptIPuter maximizes bandwidth use: A 32-node IA-64 Linux

cluster assumes 1GigE per node (32 Gbps I/O). When clusters are optically connected at 40 Gbps (for example), bandwidth is >100% the throughput!

A Useful and Usable Tool for Data-IntensiveApplication Drivers: BioScience and GeoScience

The OptIPuter project has two application drivers where scientists are generating multi-gigabytes of 3D volumetric data objects that reside on distributed archives that they want to correlate, analyze and visualize.

NIH Biomedical Informatics Research NetworkInitially a multi-scale brain imaging federated repository, but to be expanded to other organs of the body.

NSF EarthScopeThe acquisition, processing and scientific interpretation of satellite-derived remote sensing, near-real-time environmental, and active source data.

http://ncmir.ucsd.edu/gallery.html

siovizcenter.ucsd.edu/library/gallery/shoot1/index.shtml

A Radical New Architecture forScientific Cyber “Infostructure”

The OptIPuter is a “virtual” parallel computer in which the individual “processors” are widely distributed clusters; the “memory” is in the form of large distributed data repositories; “peripherals” are very-large scientific instruments, visualization displays and/or sensor arrays; and the “motherboard” uses standard IP delivered over multiple dedicated lambdas.

Think of the OptIPuter as a giant graphics card, connected to a giant disk system, via a system bus that happens to be an extremely high-speed optical network. One major design goal is to provide scientists with advanced interactive querying and visualization tools, to enable them to explore massive amounts of previously uncorrelated data in near real time.

Computing ● Data ● Visualization ● Networking

A 21st Century Amplified Work Environment: The Continuum

Passive stereo display AccessGrid Digital white board

Tiled display

Cluster Visualization

San Diego OptIPuter Design:LambdaGrid Enabled by Chiaro Router (Spring 2003)

switch switch

switchswitch

Medical Imaging and Microscopy

Chemistry, Engineering, Arts

San Diego Supercomputer Center

Scripps Institution of Oceanography

ChiaroEnstara

SL 6509

16-processor cluster

8-processor cluster

ONI DWDM

SL OPSW

NL 15454

16

8

2

8

8

8

2xOC-192

128x128 MEMSOptical Switch

OMNInet

810Gb

10Gb

Layer2/3 Switch/Router

EVL 6509

16-processor cluster

16

16

EVL OPSW

64x64 MEMSOptical Switch

16

ONI DWDM

OMNInet

8

8

Layer2/3 Switch/Router 16

UIC/EVL LAN

StarLightStarLight LAN

NetherLight/UvA

NL 15454

NL OPSWNL Force10Layer2/3 Switch/Router

8

2

32-processor cluster

32

32

Fiber Use in Pairs

GE ElectronicallySwitched

GE OpticallySwitched

64x64 MEMSOptical Switch

OptIPuter ConnectionsEVL/StarLight/NetherLight/UvA

Summer 2003 Plan

SL Force10

6x10GE24x1GE

N

2x10GE80x1GE

2

2x10GE48x1GE

What is a Lambda?

• A lambda, in networking, is a fully dedicated wavelength of light in an optical network, typically used today for 1-10Gbps.

• Lambdas are circuit-based technology, but can carry packet-based information.

• We are now mostly working with 1Gb dedicated layer2 circuits that act like lambdas

Switching Lambdas make the OptIPuter Possible

• Visualization getting very good on PCs• Networking getting very fast on PCs (10GigE)• Disk getting cheaper and faster• PCIX bus, 64-bit architectures available• Cost of bandwidth is plummeting• Optical switching is coming• We don’t have to scale to reach everybody in

their homes

Why Optical Switching?

• No need to look at every packet when transferring a terabyte of information– 1% the cost of routing– 10% the cost of switching– 64x64 10Gb:

• $100,000 O-O-O switched• $1,000,000 O-E-O switched• $10,000,000 O-E-O Routed

• Spend the savings on computing and collaboration systems instead!

• Replaces patch panels; allows rapid reconfiguration of 1 and 10Gb experiments

Large-Scale International Application Development

  Guaranteed Latency

GuaranteedScheduling

Guaranteed Bandwidth

Access GridUSA, Canada, The Netherlands, UK, Italy, Germany, Russia, Australia, China, Korea, Thailand, Taiwan, Japan, Brazil

●    

BABARUSA and internationally

    ●

The D0 ExperimentUSA, CERN, Germany, France, Japan and other worldwide collaborators

    ●

GiDVN: Global Internet Digital Video NetworkCCIRN DVWG worldwide membership

● ●  

Hubble Space TelescopeUSA, France and others worldwide

    ●

SC GlobalUSA and International

●    

Sloan Digital Sky Survey (SDSS)USA, France and worldwide

    ●

Virtual Room Videoconferencing System (VRVS)CERN, Switzerland; Caltech, USA; Others

●    

vlbiGridUSA, The Netherlands, Finland, UK and worldwide

    ●

Large-Scale International Middleware and Toolkit Development

  Guaranteed Latency

GuaranteedScheduling

Guaranteed Bandwidth

EU DataGridCERN, France, Italy, The Netherlands, UK, Czech Republic, Finland, Germany, Hungary, Spain, Sweden (in cooperation with US grid projects, notably GriPhyN, PPDG and iVGL

● ● ●

EU DataTAGUSA and Europe

    ●

Globally Interconnected Object Databases (GIOD)USA and CERN

    ●

GlobusUSA, Sweden, others internationally

● ● ●

MONARC for LHC ExperimentsCERN, Switzerland; Caltech, USA; Others

    ●

UK e-Science ProgrammeUK and USA

● ● ●

StarLight :Perhaps the World’s Largest 1GigE and 10GigE Exchange

Abbott Hall, Northwestern University’sChicago downtown campus

StarLight is an experimental optical infrastructure and proving ground for network services optimized for high-performance applications.

Operational since summer 2001, StarLight is a 1GigE and 10GigE switch/router facility for high-performance access to participating networks and is becoming a true optical switching facility for wavelengths.

StarLight’s Dutch and Canadian Partners

Kees Neggers, SURFnet Bill St. Arnaud, CA*net4

TransPAC/APAN

Euro-LinkSurfNET

Euro-LinkCERN

GÉANT

CA*net4

Abilene

Euro-Link Funded 10Gb LinkEuro-Link Co-Funded 10Gb LinkTransPAC Co-Funded 10Gb LinkPartner Matching 10Gb LinkGÉANT Matching 2.5Gb Links

Euro-Link/Trans-Light andTransPAC/Trans-Light SupplementFunded, Co-Funded and Matching Links

Hard Problems

• Internet is not designed for single large-scale users—TCP is not usable for long fat applications

• Circuits are not “scalable”• All intelligence has to be on the edge• Tuning compute, data, visualization,

networking using clusters to get order of magnitude improvement

• Security at 10Gb line speed

The OptIPuter Project will

• Investigate, procure and install optical and electronic switches.• Explore protocol stack development (for VLAN/optical management,

control and data planes). • Make progress in removing hierarchical protocol layers, moving toward

network transparency.• Optimize network-attached computational PC clusters for large-scale data

mining and visualization.• Facilitate measuring and monitoring 1-10Gb networks in experimental and

production configurations.• Compare strategies for grids moving large data over moderately

congested links vs. dedicated links.• Investigate the dynamic provisioning of light paths by high-performance

applications on optical networks.• Explore mechanisms to communicate and deliver Class of Service in

high-speed networks.• Debate the security advantages of switched lambdas

Questions?