developments in the louisiana state grid (loni)
DESCRIPTION
Developments in the Louisiana State Grid (LONI). Dick Greenwood Louisiana Tech University. DOSAR III Workshop at The University of Oklahoma September 21-22, 2006. LONI Background. In September 2004, the Louisiana State has committed $40M for a state-wide optical network. 40Gb/sec bandwith - PowerPoint PPT PresentationTRANSCRIPT
Louisiana State Grid
Dick Greenwood
Developments in the Louisiana State Grid (LONI)
Dick Greenwood
Louisiana Tech University
DOSAR III Workshop at
The University of Oklahoma
September 21-22, 2006
21SEP06 2Louisiana State Grid
Dick Greenwood
LONI Background
• In September 2004, the Louisiana State has committed $40M for a state-wide optical network.
• 40Gb/sec bandwith• Spanning 6 Universities
and 2 Health Centers:– LSU– Latech– UL-Lafayette– Tulane– UNO– Southern University– LSU Health Centers in
• New Orleans• Shereveport
21SEP06 3Louisiana State Grid
Dick Greenwood
2001 Plan
Louisiana State Grid
Dick Greenwood
Not an Ordinary Optical Network
21SEP06 5Louisiana State Grid
Dick Greenwood
Systems DeployedIBM p5 Systems– 1 February 2006 – LaTech [bluedawg] AIX v5.3● 17 users submitted 307 jobs– 15 March 2006 – Tulane [ducky] AIX v5.2● 43 users submitted 2946 jobs– 7 August 2006 – ULL [zeke] AIX v5.3● 9 users submitted 286 jobs– 8 September 2006 – UNO [neptune] AIX v5.3● 4 users submitted 10 jobs– ~October 2006 – SUBR [lacumba] AIX v5.3● 89 total LONI users
21SEP06 6Louisiana State Grid
Dick Greenwood
LONI Software Stack
• 100% TeraGrid compatible • …• Globus Toolkit• Condor• Virtual Data Toolkit (VDT)• … • The complete SW stack still to
be finalized
21SEP06 7Louisiana State Grid
Dick Greenwood
New Dell Linux Clusters To Be Delivered
• “Remote” Systems– 132 nodes, four-way Intel “Woodcrest”2.33 GHz, 4 GB RAM [4.921 TF; 528 GB RAM]– Shipment expected week of 25 September– First deployment will be LSU's system
• Next deployment likely LaTech
• Central System– 720 nodes, eight-way Intel “Cloverto(w)n”2.33 GHz,4 GB RAM [53.683 TF; 2.88 TB RAM]– Shipment expected end of November or earlyDecember
21SEP06 8Louisiana State Grid
Dick Greenwood
Linux ClustersWoodcrest and Cloverto(w)n– Woodcrest is dual-core on single silicon with ashared 4 MB cache– Cloverto(w)n is two dual-core modules in a
singlemodule (may change to four cores on singlesilicon)– Both based on 65 nm technology initially then45 nm– Both are capable of retiring four floating pointoperations per clock cycle, rather than two [IBM'sPOWER5 also does four floats per cycle]
21SEP06 9Louisiana State Grid
Dick Greenwood
Dell Cluster DescriptionEnvironmentals:– 208 V / 310 amperes; 64.5 KW; 18 tons of
cooling(7,868 cfm)– 6 racks total (4 node,1 control, 1 storage)– Rack Dimensions:78.7"Hx23.94"Wx39.93"D– Each rack has 4 PDUs;4 L6-30 208v connects(total of 24 L6-30circuits)
21SEP06 10Louisiana State Grid
Dick Greenwood
StorageCurrently very tight:– /home is 25 GB and /scratch is 280 GB– this limits usability– all served via Network File System (NFS) which isnot high performance● Future:– When central Linux cluster comes, it will include:● 14.0 TB raw at each “remote” site in one rack● 225 TB raw at the central site– Will provide central /home storage as well as global/scratch space– Using Lustre filesystem supported by ClusterFilesystems, Inc.
21SEP06 11Louisiana State Grid
Dick Greenwood
Award of NSF MRI: PetaShare1 x Overland Storage NEO Series 8000 tape storage system – 400 TB
Capacity: $ 153,558Specifications : Model 500, 12 Native Fibre LTO-3 Tape Drives, 500
Tape Slots, Redundant Power, Remote Access & Control, Touch Screen Front Panel (Quote includes
shipping and assembly)Deployment site: Louisiana State University5 x Overland Storage REO Series 9000 disk storage system - 44 TB
capacity each: 5 x $96,144 = $480,720Specifications : 44TB Raw SATA, 38 TB usable RAID 5, Protecton
OS, Remote Access and Control, Rackmount (Quote includes shipping and assembly)Deployment sites: Louisiana State University, Louisiana Tech
University, Tulane University, University of Louisiana at Lafayette, Tulane University, University of New
Orleans,Total requested equipment cost: $634,278
21SEP06 12Louisiana State Grid
Dick Greenwood
Tevfik’s Schedule for PetaShareYear 1: The required equipment for the development of PetaShare will be purchased,
installed at each participatingsite, calibrated, and will be integrated with the existing equipment at these sites. In
parallel to this process,we will start developing the key technologies necessary for this instrumentation: data-
aware storage, data-awareschedulers, and remote data analysis and visualization.
Year 2: Technologies developed during the first year will be integrated. Transparent
storage selection, dataarchival, cataloging and remote access techniques will be developed on top of this
integrated system. A userfriendly and uniform interface will be developed for interaction.
Year 3: The first prototypes will be deployed at the participating sites. Testing and
further development will beperformed. Application groups will start using the new instrumentation actively.
Year 4: The developed system will be ported to other platforms. It will be made publicly available to the
community. Commercialization potential will be investigated.
We are currently in the process of negotiating with different vendors. The purchase of the equipment may happen
sometime in late Decemebr or early January.
21SEP06 13Louisiana State Grid
Dick Greenwood
THEEND
21SEP06 14Louisiana State Grid
Dick Greenwood
EXTRA SLIDES
21SEP06 15Louisiana State Grid
Dick Greenwood
21SEP06 16Louisiana State Grid
Dick Greenwood
Coastal Modeling• Hurricane Track
Prediction• Storm Surge
Modeling• Predicting Wind
Effects• Coastal Erosion
Modeling• Emergency
Response
21SEP06 17Louisiana State Grid
Dick Greenwood
SCOOP Project• SCOOP Portal
17
21SEP06 18Louisiana State Grid
Dick Greenwood
Other Applications• Numerical Relativity• Petroleum Engineering• Computational Fluid Dynamics (CFD)• High Energy Physics• Bioinformatics
21SEP06 19Louisiana State Grid
Dick Greenwood
RamahRamahLONI DWDM
LandryLandryLONI DWDM
AlexandriaAlexandriaLONI POP
ULLULLLONI POPOTM POP
CoushattaCoushattaLONI DWDM
DerryDerryLONI DWDM
55Km55Km
80Km80Km
89Km89Km
LSULSU
HammondHammondLONI POP
UNOUNOLONI POP
TulaneTulaneLONI POP
LSU HSC NOLSU HSC NOLONI POPOTM POP
86Km86Km 48Km48Km
54Km54Km
75Km75Km 20Km20Km
70Km70KmSouthernSouthern
DWDM LoopDWDM Loop
MonroeMonroeLONI POP
La TechLa TechLONI POP
82Km82Km
55Km55Km
82Km82Km
72Km72Km
61Km61Km
82Km82Km
Northern DWDM LoopNorthern DWDM Loop
SouthernSouthernLONI POP
MindenMindenLONI DWDM
38Km38Km
LSU HSC ShreveportLSU HSC ShreveportLONI POPOTM POP
NLR BRNLR BRLONI POP
LSULSU
Jackson, MSJackson, MSLONI DWDM
FranklinFranklinLONI DWDM
SchrieverSchrieverLONI DWDM
Port BarrePort BarreLONI DWDM
13Km13Km
102Km102Km
42Km42Km
LaPlaceLaPlaceLONI DWDM
65Km65Km
CrowleyCrowleyLONI DWDM
Lake CharlesLake CharlesLONI POP
65Km65Km
69Km69Km
11Km11Km
13Km13Km
39Km39Km
LPBLPB11Km11Km
LSULSU
SeminarySeminaryLONI DWDM
GreensburgGreensburgLONI DWDM
TylertownTylertownLONI DWDM
MendenhallMendenhallLONI DWDM
JacksonJacksonLONI DWDM
TallulahTallulahLONI DWDM
StartStartLONI DWDM
EdwardsEdwardsLONI DWDM
RoanokeRoanokeLONI DWDM
35Km35Km38Km38Km
46Km46Km
35Km35Km
38Km38Km
24Km24Km
23Km23Km 64Km64Km 56Km56Km 48Km48Km
LONI Topology – POPs & DWDMs
21SEP06 20Louisiana State Grid
Dick Greenwood
RamahRamahLONI DWDM
LandryLandryLONI DWDM
AlexandriaAlexandriaLONI POP
ULLULLLONI POPOTM POP
CoushattaCoushattaLONI DWDM
DerryDerryLONI DWDM
55Km55Km
80Km80Km
89Km89Km
LSULSU
HammondHammondLONI POP
UNOUNOLONI POP
TulaneTulaneLONI POP
LSU HSC NOLSU HSC NOLONI POPOTM POP
86Km86Km 48Km48Km
54Km54Km
75Km75Km 20Km20Km
70Km70KmSouthernSouthern
DWDM LoopDWDM Loop
MonroeMonroeLONI POP
La TechLa TechLONI POP
82Km82Km
55Km55Km
82Km82Km
72Km72Km
61Km61Km
82Km82Km
Northern DWDM LoopNorthern DWDM Loop
SouthernSouthernLONI POP
MindenMindenLONI DWDM
38Km38Km
LSU HSC ShreveportLSU HSC ShreveportLONI POPOTM POP
NLR BRNLR BRLONI POP
LSULSU
Jackson, MSJackson, MSLONI DWDM
FranklinFranklinLONI DWDM
SchrieverSchrieverLONI DWDM
Port BarrePort BarreLONI DWDM
13Km13Km
102Km102Km
42Km42Km
LaPlaceLaPlaceLONI DWDM
65Km65Km
CrowleyCrowleyLONI DWDM
Lake CharlesLake CharlesLONI POP
65Km65Km
69Km69Km
11Km11Km
13Km13Km
39Km39Km
LPBLPB11Km11Km
LSULSU
SeminarySeminaryLONI DWDM
GreensburgGreensburgLONI DWDM
TylertownTylertownLONI DWDM
MendenhallMendenhallLONI DWDM
JacksonJacksonLONI DWDM
TallulahTallulahLONI DWDM
StartStartLONI DWDM
EdwardsEdwardsLONI DWDM
RoanokeRoanokeLONI DWDM
35Km35Km38Km38Km
46Km46Km
35Km35Km
38Km38Km
24Km24Km
23Km23Km 64Km64Km 56Km56Km 48Km48Km
LONI Topology - Status
21SEP06 21Louisiana State Grid
Dick Greenwood
LONI Computing Power
• At each 6 universities:– IBM p5-575 system(112 power proc, 224 GB
RAM)• At LSU:
– SuperMike (1024 Xeon proc, 1 TB RAM)– SuperHelix (256 Xeon proc, 256 GB
RAM)– MiniMike (32 Xeon proc, 32 GB RAM)– SGI Prism (32 proc, 128 GB RAM)– Nemeaux (64 Mac processors)– LSU Campus Grid with ~2000 processors
(soon)• At UL Lafayette
– LITE (384 proc, 384 GB RAM) • At Latech
– CAPS (30 processors)• And several others..