SoCal Infrastructure

OptIPuter Southern CaliforniaNetwork Infrastructure

Philip PapadopoulosOptIPuter Co-PI

University of California, San Diego

Program Director, Grids and clusters

San Diego Supercomputer Center

January 2004

SoCal Infrastructure

Fiber to CRCA

Fiber to6th College

4

4

44 BondedGigE Linksover fiber pairs

4

4

4

4

DedicatedFiber pair

4

Dell 5224

Dell 5224

Year 1 Mod-0, UCSD

SoCal Infrastructure

Building an Experimental Apparatus

• Mod-0 Optiputer Ethernet (Packet) Based– Focused as an Immediately-usable High-bandwidth Distributed Platform– Multiple Sites on Campus ( a Few Fiber Miles )– Next-generation Highly-scalable Optical Chiaro Router at Center of Network

• Hardware Balancing Act– Experiments Really Require Large Data Generators and Consumers– Science Drivers Require Significant Bandwidth to Storage– OptIPuter Predicated on Price/performance curves of > 1GE networks

• System Issues– How does one Build and Manage a Reconfigurable Distributed Instrument?

SoCal Infrastructure

Year 2 – Mod-0, UCSD

10

1

4

102

SoCal Infrastructure

Southern Cal Metro Extension Year 2

JenksLab

CRCA VizPlanned

6th College VizPlanned

10

4

4

1

4

10

1

4

Dedicated Fiber pair

2

SoCal Infrastructure

Aggregates

• Year 1 (Network Build)– Chiaro Router Purchased, Installed, Working (Feb)– 5 sites on Campus. Each with 4 GigE Uplinks to Chiaro– Private Fiber, UCSD-only. – ~40 Individual nodes, Most Shared with Other Projects– Endpoint resource poor. Network Rich

• Year 2 (Endpoint Enhancements)– Chiaro Router – Additional Line Cards, IPV6, Starting 10GigE Deployment– 8 Sites on Campus– h 3 Metro Sites– Multiple Virtual Routers for Connection to Campus, CENIC HPR, others– > 200 Nodes. Most are Donated (Sun and IBM) . Most Dedicated to OptIPuter– Infiniband Test Network on 16 nodes + Direct IB Switch to GigE– Enough Resource to Support Data-intensive Activity, – Slightly network poor.

• Year 3 + (Balanced Expansion Driven by Research Requirements)– Expand 10GigE deployments– Bring Network, Endpoint, and DWDM (Mod-1) Forward Together– Aggregate at Least a Terabit (both Network and Endpoints) by Year 5

SoCal Infrastructure

Web Information on the SD OptIPuter

• http://web.optiputer.net

• Need folks to start using resources and feedback how things can work better

• Intention is give full control of resources to experiments

• Experiments themselves should be of defined timeframe

– This is a shared instrument

• The resource endpoints are experimental

– NO BACKUPS

– DO NOT EXPECT 7/24 (We don’t have the staff)

– THINGS WILL BREAK

•

SoCal Infrastructure

High-Level Program Bullets

• UCSD will complete deployment of a 150+ node distributed test bed that consists of compute, storage, visualization, and instrument endpoints. Management policies will be put in place so that experiments can be assigned physical hardware and then have specialized (experiment-specific) software loaded on assigned nodes. This forms the Core Southern California OptIPuter Test bed

• In year 2, we won an IBM SURS Grant which allowed us to define a larger storewidth evaluation platform than in the original program plan. This cluster, deployed in January 2004 is 48 nodes with 6 spindles per node. This enables middleware and applications to understand how lambdagrids enable applications.

• As part of the SURS grant a small IB test fabric was purchased. The Topspin switch includes 4 gigE uplinks to allow investigation of IB to Ethernet communication.