winter 2006 engr 9861 – high performance computer architecture march 2006 interconnection networks
TRANSCRIPT
![Page 1: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/1.jpg)
Winter 2006
ENGR 9861 – High Performance Computer Architecture
March 2006Interconnection Networks
![Page 2: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/2.jpg)
Winter 2006
Introduction When considering interconnection networks
for parallel computation there are many shared concepts with LANs (Local Area Networks) and WANs (Wide Area Networks).
Interconnection Networks for parallel computing is a wide and interesting field. There are many areas of theoretical and practical research with regards to this topic.
“Parallel Computer Architecture”;Ch10, Culler, Singh.
![Page 3: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/3.jpg)
Winter 2006
Introduction There are different ways at looking at this
topic: Firstly, the interconnection structure often
has mathematical properties that often reflect the communication patterns of important algorithms (a regular structure).
Secondly, the design of the physical link between asynchronous elements is a huge area of research.
Thirdly, competition between shared resources within a network is also a large area of research.
![Page 4: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/4.jpg)
Winter 2006
Basic Definitions Ch 10.1
Communication
Assist
![Page 5: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/5.jpg)
Winter 2006
Basic Definitions Some terms:
CA: Communications Assist NI: Network Interface Mem: Memory P: Processor
Communication Requirements for this generic view: The interconnection network will need to provide
network transactions that support the programming model.
Latency should be minimized. Adequate concurrent transactions must be
supported.
![Page 6: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/6.jpg)
Winter 2006
Basic Definitions Physical Protocol: Converts analog
signals into digital ones. Link Protocol: is responsible for
grouping symbols into packets. Node Level Protocol: is responsible
for attaching information so that the target CA can accomplish the transfer.
![Page 7: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/7.jpg)
Winter 2006
Basic Definitions We can look at an IN as a graph
that contains vertices (processing hosts or switch elements) and channels between vertices.
Channels have the following properties: Width w (in bits) Signaling Rate f = 1 / T
![Page 8: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/8.jpg)
Winter 2006
Basic Definitions Channel Bandwidth b = w * f The amount of data transferred
across a link in one cycle is called a physical unit or phit.
Switches connect input channels to output channels. The number of channels connected is called the switch degree.
![Page 9: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/9.jpg)
Winter 2006
Network Components The following components make up a
network: Topology: The structure of the network
graph. 2D grid, 3D cube, irregular etc. A direct network has a host (processing
element) connected to each switch. An indirect network will have hosts
connected to a subset of available switches. The hosts will then be on the edge of the network graph.
![Page 10: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/10.jpg)
Winter 2006
Network Components Routing Algorithm: The path that messages
make through the network is called a route. Procedure describing which route each message takes is called the routing algorithm.
Switching Strategy: How a message travels its route.
Circuit Switching: The same route is used until the entire message is transferred by establishing an end-to-end connection. The route can be reversed as well.
Packet Switching: Message is broken into packets with its own routing information. Each packet can be individually routed. Requires routing tag overhead.
![Page 11: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/11.jpg)
Winter 2006
Network Components Flow Control Mechanism: Controls
when a message or parts of it, move along its route. Flow control becomes a necessity when a network resource has to be utilized by multiple messages at the same time.
Flow Control Options: Stalled in place. Buffered. Re-routed. Discarded.
![Page 12: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/12.jpg)
Winter 2006
Network Components The largest unit of information that can be
accepted or rejected by the nodes in a network is called a flit.
How big can a flit be? ANS: It can be as big as the entire message or
packet. It can be as small as a phit. Some other terms that can be used when
talking of networks for parallel processing are: Diameter: the maximum length of the shortest
path between two nodes through a network. Routing Distance: number of links between
source and destination. Average Distance: average of routing distance.
![Page 13: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/13.jpg)
Winter 2006
Packet Formatting
![Page 14: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/14.jpg)
Winter 2006
Packet Format Header: contains routing and
control info so that the switches can interpret what to do when the packet arrives.
Payload: The information contained within the packet.
Trailer: Usually contains an error checking code.
![Page 15: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/15.jpg)
Winter 2006
Packet Format In parallel processing networks, like LANs,
WANs and the Internet we have issues of Encapsulation and Fragmentation. Encapsulation involves carrying info
from a higher level of abstraction within the current layer.
Fragmentation involves splitting up the higher level information into a sequence of messages.
![Page 16: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/16.jpg)
Winter 2006
Communication Performance There are four components that
affect the time to transfer n bits from source to destination:
TimeS-D(n) = Overhead + Routing Delay + Channel Occupancy + Contention Delay
![Page 17: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/17.jpg)
Winter 2006
Performance Overhead: comes from getting the message in and
out of the network (ie can be caused by the CA) Channel Occupancy: gives us a lower bound on
latency. Channel occupancy could be simply viewed as the time taken for the message to get from source to destination on a direct link.
The CA takes time to process the communication request. Each channel traveled by the packet encounters delay. The destination CA takes some time to process the
packet.
![Page 18: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/18.jpg)
Winter 2006
Performance Overall the occupancy of the channel
can be determined by: (n + ne) / b n = number if bits in payload ne = number of bits in header and trailer b = channel bandwidth
We can also look at the effective bandwidth:
n / ( n + ne)
![Page 19: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/19.jpg)
Winter 2006
Performance Routing Delay: Each channel in the
route incurs a little delay that builds up (we will consider the time taken for node to switch interface to be part of the routing delay). Causes of routing delay:
Routing distance (h): number of channels used in route.
Switching delay (Δ): time taken for a switch to select the proper output port.
h depends on network topology, routing algorithm used and specific nodes involved in the transaction.
![Page 20: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/20.jpg)
Winter 2006
Unloaded Latency (based on switching
strategy)
Store and Forward Routing (packet switched): The entire packet is received by the switch before
forwarded to the next channel. Latency:
Number of bitsin packet, including headerand trailer.
![Page 21: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/21.jpg)
Winter 2006
Unloaded Latency (based on switching
strategy)
Circuit switched Once the route is setup it is maintained.
We therefore only encounter the switching latency when the route is setup.
![Page 22: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/22.jpg)
Winter 2006
Unloaded Latency (based on switching
strategy)
In the case of circuit switching we can note the following: As the message size increases, the
amount of latency caused by route setup per hop (h * Δ), and hence the topology becomes insignificant.
How can we reduce the latency in the case of store and forward packet switching?
![Page 23: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/23.jpg)
Winter 2006
Unloaded Latency (based on switching
strategy)
Solution: Fragment the message packet into smaller
packets. The smaller packets flow through in a pipelined fashion. The unloaded latency becomes:
Size of fragments. Same form as before.
![Page 24: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/24.jpg)
Winter 2006
The previous example is commonly used in the internet and larger networks.
In the case of parallel processing cut-through routing can be used:
Once a few phits are received by the switch the rest of packet is routed to the output.
Unloaded Latency (based on switching
strategy)
This valuewill be differentfrom the circuit switched case.
![Page 25: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/25.jpg)
Winter 2006
Unloaded Latency (Based on Switching
Strategy)
![Page 26: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/26.jpg)
Winter 2006
Contention As in traditional networking, contention
will occur then two incoming messages need to be routed to the same output at the same time.
In store and forward routing the switch will buffer an entire packet. If there is contention, one packet will get switched to the output and one will get blocked until the next switching cycle.
![Page 27: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/27.jpg)
Winter 2006
Contention In circuit switching, usually some type
of probe is sent from the source node to the destination. If there is contention, the probe will be resent after some time later.
Cut through routing can handle contention in two ways: Virtual cut-through, route one of the packets
into a buffer then route in the next switch cycle. This has the same penalty as store and forward routing under contention.
![Page 28: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/28.jpg)
Winter 2006
Contention Wormhole , only a few flits are
buffered from the header of the packet, then the tail portion is maintained. Similar to holding the circuit open from the sender’s point of view.
![Page 29: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/29.jpg)
Winter 2006
Bandwidth By just looking at the bandwidth from the
viewpoint of a single node, the channel has a channel bandwidth that is higher than the bandwidth that the node can send useful data on.
beff =
![Page 30: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/30.jpg)
Winter 2006
Bandwidth Taking into account routing delay
at the switch (Δ) we have the following expression:
beff =
w is included here in case
the channel width in more
than a bit. Note that n will
be the size in phits.
![Page 31: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/31.jpg)
Winter 2006
Bandwidth These expressions are useful for looking
at the bandwidth available to one node. What if we want a measure of the
overall bandwidth in a network? Most common measure is the bisection
bandwidth: The sum of the bandwidths of the minimum set of
channels that, if removed, partition the network into two equal unconnected sets of nodes.
This has a nice property when considering a uniform communication pattern, what is it?
![Page 32: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/32.jpg)
Winter 2006
Bandwidth ANSWER:
Half of the messages are expected to cross the bisection in each direction.
With this in mind what is wrong with this notion of global or “aggregate” bandwidth available? ANSWER:
If communication is localized, then the bisection bandwidth will give a lower value for communication time.
![Page 33: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/33.jpg)
Winter 2006
Total Bandwidth and Average Link Utilization
Total Bandwidth = C * b (bytes/sec) = C * w (bits / cycle) = C (phits / cycle)
Assuming each of N hosts issue a packet every M cycles with average routing distance h. Then each packet occupies h channels for l = n / w cycles.
The total load on the network is(N * h * l ) / M) phits/cycle
![Page 34: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/34.jpg)
Winter 2006
Total Bandwidth and Average Link Utilization
The average link utilization is (<1):
This is discussed on P. 762 of the text.
![Page 35: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/35.jpg)
Winter 2006
Bandwidth The number of links or channels
per node (C /N) is the total communication bandwidth (phits/cycle/node).
This is consumed in direct proportion to the message size and to the routing distance.
![Page 36: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/36.jpg)
Winter 2006
Factors That Limit ρ Before we look at why the link
utilization is less than one (in some cases much less than one) let us consider the various properties on the network.
The number of links per network node is a property of topology.
![Page 37: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/37.jpg)
Winter 2006
Factors That Limit ρ Average routing distance depends on:
The topology Routing algorithm Program communication pattern Mapping of program onto machine
Often good communication locality will provide a small h, random communication will give the average routing distance and a bad pattern will cross the entire diameter.
![Page 38: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/38.jpg)
Winter 2006
Factors That Limit ρ Factors:
Communication may not be balanced over all links.
Even if it is balanced, the routing algorithm may not support the communication pattern of the program.
Contention for other networking resources may arise.
These factors affect the saturation point of the network.
![Page 39: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/39.jpg)
Winter 2006
What assumption is being
made here?
![Page 40: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/40.jpg)
Winter 2006
Topology of INs
Before we discuss some different types of interconnection network topologies we want to consider the following: The number of host nodes that is
connected to the network will be defined to be N.
Characteristics of each topology will be discussed as a function of N.
![Page 41: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/41.jpg)
Winter 2006
Fully Connected Network This type of network connects all inputs
to all outputs. It can be considered a single big switch.
The diameter of such a network is: 1. The degree in N. Unfortunately, if there is a hardware
failure in such a network the entire network goes down, or at least full connectivity is lost.
![Page 42: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/42.jpg)
Winter 2006
Fully Connected Network A bus is an example of a fully
connected network. Its cost scales with O(N) Bandwidth:
Total Bandwidth = O(1) Bisection Bandwidth = O(1)
Bandwidth scaling is worse than O(1) as clock rate reduces with #ports due to RC
![Page 43: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/43.jpg)
Winter 2006
Fully Connected Networks A crossbar switch is another example. Bandwidth is O(N) Cost is O(N2), why?
As more inputs/outputs are added, the total number of cross points grow by N2.
The scalability of fully connected networks is bad for large host sizes. Usually smaller components of the network (like a basic switching element) may be fully connected.
![Page 44: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/44.jpg)
Winter 2006
Linear Arrays Linear Array
Assume we have N (0 ..N-1) nodes assembled in a linear fashion.
Assume each node is connected with a bi-directional link.
What is the diameter? ANS: N – 1
Average routing distance ~ 2/3 N. The bisection is one link.
![Page 45: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/45.jpg)
Winter 2006
Linear Arrays The route from node A to node B can be
described by the operation B-A. This result is termed as the relative
address. Provides a log N – bit number with
positive numbering being away from node 0.
This arrangement provides no fault tolerance.
![Page 46: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/46.jpg)
Winter 2006
Ring Bi-directional Links Easily constructed by connecting the
ends of a linear array together. Degree: 2 The diameter is N/2 The bisection of the network is 2 The average routing distance is N/3 Note there are two relative addresses
because we can travel in either direction. Also provides better fault tolerance.
![Page 47: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/47.jpg)
Winter 2006
Ring Unidirectional Links
If we have a ring the can only transmit in one direction we have the following properties: Diameter N – 1 Average Distance is N/2 Relative Address ( B – A ) mod N Bisection width: 1
![Page 48: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/48.jpg)
Winter 2006
![Page 49: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/49.jpg)
Winter 2006
Higher Dimension Meshes and Tori A d-dimensional array consists of the
following elements { kd-1 x kd-2 … x k0 }
Where k is a vector of elements. If 0<= ij <= kj-1 for 0<= j <= d-1 we
can use a vector to locate any node in the mesh, i.e., the coordinates of a node are comprised of <id-1, id-2, .. i0>
![Page 50: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/50.jpg)
Winter 2006
Higher Dimension Meshes and Tori
Assuming the length along each dimension is equal. N = kd
The degree of each node varies between 2d and d. Nodes on the inside have the maximum degree and nodes on the corners have the smallest.
For example for d = 3, nodes on the corners have 3 links or channels, and nodes on the inside have 6 channels. What about tori?
![Page 51: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/51.jpg)
Winter 2006
Higher Dimension Meshes and Tori
These arrays are called d-dimensional k-ary arrays.
To extend to a torus, the edges are simply connected to the opposite side.
Usually these types of structures are direct networks, meaning that every node contains a processing element.
The network will scale by increasing k.
![Page 52: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/52.jpg)
Winter 2006
Higher Dimension Arrays and Tori We can form a relative address by
simply performing vector subtraction (unidirectional case): R = (bd-1 - ad-1 , bd-2 - ad-2 , … b0 - a0)
Actual routing can be performed in any order.
The diameter is simply d*(k-1) If k is even, the bisection of a d-
dimensional k-ary structure will be kd-1. If k is odd it maybe a little larger.
![Page 53: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/53.jpg)
Winter 2006
Higher Dimension Arrays and Tori
The average distance is the average distance in each dimension.
Therefore average h = d * 2/3 * k roughly.
Spatially these networks scale in size to whatever dimension we have. Volume in d=3 and planar space in d=2. Assuming shortest possible wiring.
![Page 54: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/54.jpg)
Winter 2006
Higher Dimension Arrays and Tori
![Page 55: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/55.jpg)
Winter 2006
Higher Dimension Arrays and Tori
![Page 56: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/56.jpg)
Winter 2006
Trees With meshes, the average routing distance
grows with logdN . A binary tree has a degree of 3 (three
connections per node) Usually trees are used in indirect networks.
Indirect Case: Addressing to the leaves can be taken as a log2 N bit
vector. This gives the path from the root to the host. 0 = left, 1 = right.
The diameter is 2 * log2 N Average routing distance is almost as large as the
diameter of the network.
![Page 57: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/57.jpg)
Winter 2006
Trees Relative addressing can be
accomplished by doing the bit-wise XOR operation. For example to get the relative
address from A to B. R = A XOR B. The position of the most significant 1 is how many levels we go up. Then we use the lower bits of B to get to B. We may not have to go all the way to the root!
![Page 58: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/58.jpg)
Winter 2006
Trees
A = 0001 B = 0101
A XOR B = 0100
![Page 59: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/59.jpg)
Winter 2006
Trees We can have trees of higher order
called k-ary trees. We can also have fat trees.
More bandwidth is assigned to more important links as we go towards the root.
A big problem with trees is that the root is composed of one link, therefore the bisection is one link.
![Page 60: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/60.jpg)
Winter 2006
Butterflies The construction of a butterfly is
similar to that of a tree. We have many roots in a butterfly. In addition, many parallel
algorithms communicate in a butterfly structure, ex: Fast Fourier Transform and Batcher odd-even merge sort.
![Page 61: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/61.jpg)
Winter 2006
Butterflies As a building block we start with 2 x 2
switch elements. The basic building block is setup so that
addressing can occur. A bit of 0 causes a straight edge to be followed. While a 1 will cause a crossover to occur.
In the case of a unidirectional indirect butterfly with N hosts, the bisection is N/2 links.
![Page 62: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/62.jpg)
Winter 2006
Butterflies
![Page 63: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/63.jpg)
Winter 2006
Butterflies When considering scalability,
butterflies can be better than meshes and trees in the there are a total of N log2N (in the case of the previous figure) links with packets crossing log2N links on average
Therefore on average there shouldn’t be any collisions.
How many links are in the bisection?
![Page 64: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/64.jpg)
Winter 2006
Butterflies
![Page 65: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/65.jpg)
Winter 2006
Hypercubes If we take the original butterfly and
collapse each straight column into a single log2N switch. This is close to a hypercube arrangement.
We can cross dimensions of the hypercube to get from source to destination.
Each node of a hypercube can embed a lower dimension mesh.
Text P778.
![Page 66: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/66.jpg)
Winter 2006
Hypercubes
From: http://linux.cs.sonoma.edu/~ravi/ces516sp04/Lectures/feb18.ppt
![Page 67: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/67.jpg)
Winter 2006
Some Example Architectures
![Page 68: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/68.jpg)
Winter 2006
![Page 69: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/69.jpg)
Winter 2006
Routing Routing from source to destination is
of primary importance to parallel computing.
We have already seen some examples of how a relative address is formed. In the case of a d=3 cube the relative address will give the shortest path in all three dimensions.
![Page 70: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/70.jpg)
Winter 2006
Routing The routing algorithm decides at
each switch element, which output port to place the packet onto.
3 ways to determine output port based on packet header: Arithmetic Source based port select Table Lookup
![Page 71: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/71.jpg)
Winter 2006
Routing Arithmetic
2D Mesh: Each relative address contains the length
to be traveled in both the x and y directions [Δx, Δy]
At switch i,j we perform the following routing:
![Page 72: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/72.jpg)
Winter 2006
Routing The switch will look at the routing info in
the packet and modify the distance in the appropriate direction. Dimension order routing takes each dimension in turn.
Source based routing can also be used in which the source node assigns switch port numbers to the header. Simple from the switch side. May have variable header size and maybe
large.
![Page 73: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/73.jpg)
Winter 2006
Routing Table Driven Routing:
Similar the the internet and WANs. Switches will have a table of
information use for routing. The header contains an index that is used in the table to select the proper output port.
Tables must be updated. Switch specific messages. The table must be established in the first
place.
![Page 74: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/74.jpg)
Winter 2006
Routing Deterministic Routing
Route is determined solely on the source and destination. The status of the network is not considered.
Dimension ordered routing is such an example.
In the case of a 2D mesh how else can we route a packet?
ANS: If one dimension was blocked, we could switch to the other dimension etc…
![Page 75: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/75.jpg)
Winter 2006
Routing Adaptive:
The route of the packet is determined by source and destination, but may be influenced by network conditions.
In the previous case we could zig-zag across the mesh if both dimensions on the edges were congested.
![Page 76: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/76.jpg)
Winter 2006
Deadlock Deadlock: occurs when a packet
waits for an event that cannot occur. Live lock: Occurs when the routing
of a packet never arrives at its destination.
Indefinite Postponement: Occurs when the packet waits for an event that never happens.
![Page 77: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/77.jpg)
Winter 2006
Deadlock Example
![Page 78: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/78.jpg)
Winter 2006
Virtual Channels One way of avoiding such deadlock is to
implement virtual channels. Virtual channels are used in wormhole
routing and involve each physical channel to have multiple buffers.
Assume we have 2 virtual channels in the prevoius example. Say packets at a node higher than their destination are placed in the high channel and the opposite for lower destinations.
![Page 79: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/79.jpg)
Winter 2006
Virtual Channels
![Page 80: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/80.jpg)
Winter 2006
Adaptive Routing
![Page 81: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/81.jpg)
Winter 2006
Other Topics In Interconnection Networks
Turn-Model Routing Switch Design Channel Buffers Flow Control
There are differences between LANs and interconnection networks for parallel processing.
Global Communications.
![Page 82: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/82.jpg)
Winter 2006
SGI Origin Network Named SPIDER (we’ll see why when we
look at it’s stricture), supports 1.56 GB/s total bandwidth in both directions.
Each switch contains 6 pairs of unidirectional links.
Two nodes are connected to each switch leaving 4 links to connect to other switches.
Routing is table based. So that Message priority is supported.
![Page 83: Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks](https://reader034.vdocuments.net/reader034/viewer/2022042718/56649e625503460f94b5d6b6/html5/thumbnails/83.jpg)
Winter 2006
SGI Origin Network