tops “technology for optical pixel streaming” paul wielinga division manager high performance...
TRANSCRIPT
TOPS “Technology for Optical Pixel Streaming”
Paul Wielinga
Division Manager High Performance NetworkingSARA Computing and Networking Services
Courtesy EVL
iGrid2005 - GLVF OptIPuter panel
Outline
• About SARA• Rationale for adapting OptIPuter concept• TOPS• Demonstration US117b at iGrid2005• (Preliminary) Results• Other demo’s SARA is involved in
iGrid2005 - GLVF OptIPuter panel
SARA Computing and Networking Services
• SARA: National e-Science Support Center• Home of
Dutch Supercomputing infrastructure SURFnet NOC SURFnet6 major core node NetherLight (sister site of StarLight) Amsterdam LightHouse
• OptIPuter affiliate international partner
• LHC Tier-1 site
• Proposed LOFAR storage site
• SARA is independent and not-for-profit
iGrid2005 - GLVF OptIPuter panel
Teraflops, Terabytes, Terabits, Terapixels @SARA
CAVESGI Origin 3800
STK tape library
Tiled-panel display
IBM Cluster 1600
Linux Nationaal Rekencluster
Netherlight SURFnet 6
SGI Altix 3700
iGrid2005 - GLVF OptIPuter panel
Why we adopt OptIPuter
• Science pull:Petabytes of complex data explored and analyzed by
nationally and internationally dispersed scientists, in many teams
SARA’s mission as e-Science Support Center
Life Sciences
~PBytes/sec
LHC Tier-1 site (with NIKHEF)
Brain Activity Analysis Univ Utrecht
LOFAR
iGrid2005 - GLVF OptIPuter panel
Why we adopt OptIPuter (2)
• Technology push: SURFnet6 – Hybrid Optical
and Packet Switching Network 5 Photonic RIngs Light Path Provisioning for all
universities and research institutions in NL
Dordrecht1
Breda1
Tilburg1
DenHaag
NLR
BT
BT NLR
BT
Zutphen1
Lelystad1
Subnetwork 4:Purple
Subnetwork 3:Red
Subnetwork 1:Green
Subnetwork 2:Dark blue
Subnetwork 5:Grey
Emmeloord
Zwolle1
Venlo1
Enschede1
Groningen1
LeeuwardenHarlingen
Den Helder
Alkmaar1
Haarlem1
Leiden1
Assen1
Beilen1
Meppel1
Emmen1
Arnhem
Apeldoorn1
Bergen-op-ZoomZierikzee
Middelburg
Vlissingen Krabbendijke
Breukelen1
Ede
Heerlen2Geleen1
DLO
Schiphol-Rijk
Wageningen1 Nijmegen1
Hilversum1
Hoogeveen1
Lelystad2
Amsterdam1
Dwingeloo1
Amsterdam2
Den Bosch1
Utrecht1
Beilen1
Nieuwegein1Rotterdam1
Delft1
Heerlen1
Heerlen1
Maastricht1
Eindhoven1
Maasbracht1
Rotterdam4
3XLSOP
IBG1 & IBG2Middenmeer1
iGrid2005 - GLVF OptIPuter panel
TOPSTechnology for Optical Pixel Streaming
• Use OptIPuter concept to split visualization from display Render images from very large data sets on central
visualization cluster Use high speed (lambda) networks to stream pixels to ultra-
high resolution tiled panel displays
• Adapt the idea of video streaming: constant stream of pixels from renderer to display
• Use lossy protocols for long distance connectivity: High performance TCP hard to achieve, UDP performance trivial
• Light weight application• Possibly incorporated in SAGE (EVL and UCSD)?
iGrid2005 - GLVF OptIPuter panel
DEMO US117b – GLVF TOPS
• Very large 2D and 3D datasets located at SARA in Amsterdam
• Render cluster (29 nodes) at SARA• 20 Gbps connectivity between cluster and
NetherLight, via University of Amsterdam switch
• 2 * 10 Gbps lambda between Amsterdam and San Diego (SURFnet, CaveWave, WANPHY Amsterdam-SD)
• UCSD LambdaVision display at iGrid: 100 Megapixel Tiled Display
• 1 Frame = 100 Mpixel*24 bits = 2.4 Gbits• Uncompressed and compressed mode• Streaming pixels from Amsterdam to San
Diego
iGrid2005 - GLVF OptIPuter panel
Preliminary results
• First demoslot, 1 10 Gbps available Sustained bandwidth used between rendering and display
9.2 Gbps
• Second demoslot, 2 * 10 Gbps Sustained bandwidth 5,3 + 6,2 = 11,5 Gbps
iGrid2005 - GLVF OptIPuter panel
Final results
• Third demoslot, 2 times 10 Gbps available Sustained bandwidth used between rendering and display
18 Gbps Peak bandwidth of 19.5 Gbps!
• New world record of transatlantic bandwidth usage by one single application visualizing scientific content
iGrid2005 - GLVF OptIPuter panel
Preliminary results
• Limitation was MTU size – not whole path was jumbo-frame enabled
• Display cluster had 100% cpu load due to small MTU size• Large UDP packet loss (upto 50%)• Noticable difference in latency (visible) between
CaveWave (with 4 switches) and direct WANPHY connection
• Later today: new trial with jumbo frames enabled Expect higher framerate and bandwidth usage of 15 Gbps
or more
iGrid2005 - GLVF OptIPuter panel
iGrid2005 demonstrations
• US117b: GLVF – Unreliable Stream of Images (TOPS)
• SARA participation in other demo’s: US117a: SAGE (EVL) computing support in Amsterdam NL102: Dead Cat (UvA) – Visualization support
(Amsterdam) US106: 10 Gb line-speed security (NORTEL) – Networking
and computing support (Amsterdam)
iGrid2005 - GLVF OptIPuter panel
Credits
• Bram Stolk (SARA)• Jeroen Akershoek (SARA)• Ronald van der Pol (SARA)• Pieter de Boer (SARA)• Hanno Pet (SARA)• University of Amsterdam (Paola Grosso et al.)• SURFnet (Dennis Paus et al.)• Jason Leigh, Luc Renambot, Lance Long and colleagues
at EVL and UCSD• iGrid2005 networking and LambdaVision crews
Thank You