Download - Unconventional Networking
Unconventional Unconventional NetworkingNetworkingMakoto BentzMakoto Bentz
October 13, 2010October 13, 2010
CS 6410CS 6410
PapersPapers
U-Net: A User-Level Network Interface for Parallel U-Net: A User-Level Network Interface for Parallel and Distributed Computing,and Distributed Computing, Von Eicken, Basu, Buch and Werner Vogels. 15th Von Eicken, Basu, Buch and Werner Vogels. 15th SOSP, December 1995.SOSP, December 1995.
Active Messages: A Mechanism for Integrated CoActive Messages: A Mechanism for Integrated Communication and Controlmmunication and Control, Thorsten von Eicken, David E. Culler, Seth Copen , Thorsten von Eicken, David E. Culler, Seth Copen Goldstein, and Klaus Erik Schauser. Goldstein, and Klaus Erik Schauser. In Proceedings In Proceedings of the 19th Annual International Symposium on of the 19th Annual International Symposium on Computer ArchitectureComputer Architecture, 1992., 1992.
Evaluation of the Virtual Interface Architecture Evaluation of the Virtual Interface Architecture (VIA)(VIA). Xin Liu June 8, 1999 Department of . Xin Liu June 8, 1999 Department of Computer Science and Engineering University of Computer Science and Engineering University of California, San DiegoCalifornia, San Diego
OverviewOverview
Active Messages:Active Messages: Instructions between Instructions between multiprocessorsmultiprocessors
U-Net:U-Net: Networking beyond the kernel, Networking beyond the kernel, separation into the User Spaceseparation into the User Space
VIA:VIA: Fast Messaging on virtualized standard Fast Messaging on virtualized standard network interfaces (VIA)network interfaces (VIA)
Parallel versus Parallel versus DistributedDistributed
CPUCPUCPUCPU
CacheCacheCacheCache
MemorMemoryy
MemorMemoryy
CPUCPUCPUCPU
CacheCacheCacheCache
............
Parallel Parallel ComputingComputing
Multiple Multiple processors processors connected in a connected in a computercomputer
Tightly coupledTightly coupled
Shared memoryShared memory
Parallel versus Parallel versus DistributedDistributed
CPUCPUCPUCPU
CacheCacheCacheCache......
MemorMemoryy
MemorMemoryy
CPUCPUCPUCPU
CacheCacheCacheCache
MemorMemoryy
MemorMemoryy
Distributed Distributed ComputingComputing
Multiple Multiple autonomous autonomous computers computers connected by connected by networknetwork
Loosely coupledLoosely coupled
Distributed memoryDistributed memory
NetworkNetwork
Active MessagesActive Messages
Written at by von Eicken, Written at by von Eicken, Cohen and at University of Cohen and at University of California Berkeley, in David California Berkeley, in David E. Culler’s research groupE. Culler’s research group
Other projects by group Other projects by group include Split-C, Castle and include Split-C, Castle and NinjaNinja
NinjaNinja
Active Messages: Active Messages: AuthorsAuthors
Seth Copen Cohen is Seth Copen Cohen is now a professor at CMUnow a professor at CMU
Programmable matterProgrammable matter
Klaus Erik Schauser is Klaus Erik Schauser is now a professor at now a professor at UCSBUCSB
Parallel computingParallel computing
Thorston von EickenThorston von Eicken
Advised by David E. Culler (MIT, teaches at Advised by David E. Culler (MIT, teaches at U of C Berkeley)U of C Berkeley)
Former Cornell Assistant ProfessorFormer Cornell Assistant Professor
Active Messages becomes his Ph.D. thesisActive Messages becomes his Ph.D. thesis
Now teaches at UCSBNow teaches at UCSB
Worked at ExpertCity.com, worked on Worked at ExpertCity.com, worked on GoToMyPC and GoToMeetingGoToMyPC and GoToMeeting
Now works at RightScale, a cloud Now works at RightScale, a cloud computing companycomputing company
Active Messages: Active Messages: MotivationMotivation
Message passing machines Message passing machines have reasonable hardware have reasonable hardware costs but suffer fromcosts but suffer from
Poor overlap between Poor overlap between communication and communication and computationcomputation
High overhead for messagesHigh overhead for messages
And this is not due to And this is not due to hardwarehardware
Com
m.
Com
m.
Pro
c.
Pro
c.
WastedWasted
Tim
eTim
e
task
task
task
task
task
task
Active Active Messages:HardwareMessages:Hardware
nCube-2nCube-2
1.9 Gflops/s1.9 Gflops/s
Hypercube layoutHypercube layout
Thinking Machines Thinking Machines Connection Machines Connection Machines (CM)-5(CM)-5
Non-standard von Non-standard von Neumann architectureNeumann architecture
Fat-tree network MIMDFat-tree network MIMD
NSA FROSTBURG aNSA FROSTBURG aCM-5 machineCM-5 machine
Active Messages:DesignActive Messages:Design
Get around the comm/proc delayGet around the comm/proc delay
Active messages are messaging objects that Active messages are messaging objects that can perform its own processingcan perform its own processing
Each message carries a userspace handler or Each message carries a userspace handler or codecode
The message contains the arguments to pass The message contains the arguments to pass to the handlerto the handler
Single Program, Multiple Data
Active Messages:DesignActive Messages:Design
Program pre-allocates receiving structures, Program pre-allocates receiving structures, eliminating bufferingeliminating buffering
The network itself acts as a buffer for small The network itself acts as a buffer for small messagesmessages
Creates low overhead on both sidesCreates low overhead on both sides
Message handlers are not allowed to blockMessage handlers are not allowed to block
Active Active Messaging:TestingMessaging:Testing
nCube/2nCube/2
11us send + 15us receive (nearly an order of 11us send + 15us receive (nearly an order of magnitude reduction)magnitude reduction)
Almost near minimal instructionsAlmost near minimal instructions
CM-5CM-5
1.6us send + 1.7us receive1.6us send + 1.7us receive
Active Messaging:Split-CActive Messaging:Split-C
Spit-C PUT and GET Spit-C PUT and GET perform split-phase perform split-phase copies of memory copies of memory blocks between nodesblocks between nodes
For high performance, For high performance, should overlap should overlap communication and communication and computationcomputation
Active Messaging:Split-CActive Messaging:Split-C
Active Messaging:Message Active Messaging:Message Driven ArchitecturesDriven Architectures
Message Driven Architectures Support Message Driven Architectures Support languages with support for dynamic parallelismlanguages with support for dynamic parallelism
Active Message handlers execute immediately Active Message handlers execute immediately upon message arrival, cannot suspend or blockupon message arrival, cannot suspend or block
Active Active Messaging:Further Messaging:Further Work: TAMWork: TAM
Threaded Abstract Machine is a fine-grained Threaded Abstract Machine is a fine-grained parallel execution model based on Active parallel execution model based on Active MessageMessage
No testing resultsNo testing results
Active Messaging: Active Messaging: DiscussionDiscussion
This paper becomes von Eicken’s thesisThis paper becomes von Eicken’s thesis
I personally disliked the style of this paper, but I personally disliked the style of this paper, but liked the contentliked the content
Too much in one article, works much better Too much in one article, works much better as a thesisas a thesis
U-Net: AuthorsU-Net: Authors
Assistant Professor von EickenAssistant Professor von Eicken
Anindya Basu (advised by von Anindya Basu (advised by von Eicken)Eicken)
Vineet Buch (M.S. from Cornell)Vineet Buch (M.S. from Cornell)
co-founds like.com (now part of co-founds like.com (now part of Google)Google)
worked with BlueRun (VC)worked with BlueRun (VC)
Werner VogelsWerner Vogels
former Researcher at Cornellformer Researcher at Cornell
now Amazon CTOnow Amazon CTO
U-Net: MotivationU-Net: Motivation
Networks are now so fast the biggest delay Networks are now so fast the biggest delay is kernel processing of the message stacksis kernel processing of the message stacks
Developing in kernel space limits Developing in kernel space limits development of new message send/receive development of new message send/receive interfacesinterfaces
U-Net: MotivationU-Net: Motivation
Removing the kernel from Removing the kernel from the critical path would the critical path would
allow specializationallow specialization
U-Net: MotivationU-Net: Motivation
Previously, Kernel’sPreviously, Kernel’sResponsibility, andResponsibility, and
opaque to theopaque to theapplicationapplication
Want toWant tomake thismake thisapplicationapplication
specificspecific
Would like to be able to integrate Would like to be able to integrate application specific behavior to application specific behavior to
optimize performanceoptimize performance
U-Net: DesignU-Net: Design
U-Net must be able toU-Net must be able to
Multiplex the network between processesMultiplex the network between processes
Provide isolation and protection between Provide isolation and protection between processesprocesses
Manage resources without kernel pathsManage resources without kernel paths
Allow the development of an interfaceAllow the development of an interface
U-Net: Design ConcernsU-Net: Design Concerns
Focus on low overheads to optimize small Focus on low overheads to optimize small message transmission latency and bandwidthmessage transmission latency and bandwidth
Emphasis on protocol design and integrationEmphasis on protocol design and integration
To do both on standard off-the-shelf hardwareTo do both on standard off-the-shelf hardware
U-Net: Design: Low U-Net: Design: Low OverheadOverhead
U-Net components map toU-Net components map to
Real hardware in the NIReal hardware in the NI
-and/or--and/or-
Memory locations interpreted by the OSMemory locations interpreted by the OS
Zero-Copy tries to limit intermediate Zero-Copy tries to limit intermediate buffering between NI and User-Levelbuffering between NI and User-Level
U-Net: Design: ProtocolsU-Net: Design: Protocols
Tags specific to the network substrate used for Tags specific to the network substrate used for addressingaddressing
ATM (VCI)ATM (VCI)
Ethernet (Physical addressing)Ethernet (Physical addressing)
Endpoints, communication channels and Endpoints, communication channels and queues are only accessible by ownersqueues are only accessible by owners
U-Net: Design: HardwareU-Net: Design: Hardware
Implemented on Implemented on SPARCstationSPARCstation
Fore SBA-100 Fore SBA-100 interfacesinterfaces
Fore SBA-200 Fore SBA-200 interfacesinterfaces
Could use on-board Could use on-board processor on SBA-200, processor on SBA-200, custom firmware due to custom firmware due to poor bundled firmwarepoor bundled firmware
U-Net: Testing ResultsU-Net: Testing Results
U-Net: Testing ResultsU-Net: Testing Results
U-Net: DiscussionU-Net: Discussion
Feasible technologyFeasible technology
Good performance on off the shelf Good performance on off the shelf hardware with existing standardshardware with existing standards
““This encouraging result should, however, This encouraging result should, however, not obscure the fact that not obscure the fact that significant significant additional system resourcesadditional system resources, such as , such as parallel process schedulers and parallel file parallel process schedulers and parallel file systems, systems, still need to be developedstill need to be developed before the cluster of workstations can be before the cluster of workstations can be viewed as a unified resource.”viewed as a unified resource.”
FM/VIAFM/VIA
Virtual Interface Architecture standardizes Virtual Interface Architecture standardizes System Area Networks (Microsoft, Intel, System Area Networks (Microsoft, Intel, Compaq)Compaq)
VIA however, has no simple programming VIA however, has no simple programming interfaceinterface
Fast Messages provided layering for small Fast Messages provided layering for small messagesmessages
FM/VIA :: Testing ResultsFM/VIA :: Testing Results
FM/VIA :: TestingFM/VIA :: Testing
Effects of U-NetEffects of U-Net
FM/VIA was not useful because it did not meet FM/VIA was not useful because it did not meet the requirement of fast end pointsthe requirement of fast end points
FM/VIA goes on to influence InfiniBand, iWarpFM/VIA goes on to influence InfiniBand, iWarp
Zero-copyZero-copy