indirect networks or dynamic networks
DESCRIPTION
Interconnection Networks. Indirect Networks or Dynamic Networks. Guihai Chen …with major presentation contribution from José Flich, UPV (and Cell BE EIB slides by Tom Ainsworth, USC). Questions in mind. Difference between Static/direct and Dynamic/indirect Networks - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/1.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
Indirect Networks orDynamic Networks
Guihai Chen
…with major presentation contribution from José Flich, UPV(and Cell BE EIB slides by Tom Ainsworth, USC)
Interconnection Networks
![Page 2: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/2.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
2
Questions in mind
Difference between Static/direct and Dynamic/indirect Networks
Why Multi-Stage Interconnection Networks
Large Switch and Small Switch
How to design non-blocking MINs
![Page 3: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/3.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
3
Outline
Network Topology
Preliminaries and Evolution
Centralized Switched (Indirect) Networks
From non-blocking crossbar to blocking MINs
From blocking MINs to non-blocking MINs
![Page 4: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/4.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
4
Network Topology
Preliminaries and Evolution• One switch suffices to connect a small number of devices
– Number of switch ports limited by VLSI technology, power consumption, packaging, and other such cost constraints
• A fabric of interconnected switches (i.e., switch fabric or network fabric) is needed when the number of devices is much larger
– The topology must make a path(s) available for every pair of devices—property of connectedness or full access (What paths?)
• Topology defines the connection structure across all components– Bisection bandwidth: the minimum bandwidth of all links crossing
a network split into two roughly equal halves
– Full bisection bandwidth: › Network BWBisection = Injection (or Reception) BWBisection= N/2
– Bisection bandwidth mainly affects performance• Topology is constrained primarily by local chip/board pin-outs;
secondarily, (if at all) by global bisection bandwidth
![Page 5: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/5.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
5
Network Topology
Preliminaries and Evolution• Several tens of topologies proposed, but less than a dozen used• 1970s and 1980s
– Topologies were proposed to reduce hop count• 1990s
– Pipelined transmission and switching techniques
– Packet latency became decoupled from hop count• 2000s
– Topology still important (especially OCNs, SANs, P2P Overlays, DCNs) when N is high
– Topology impacts performance and has a major impact on cost
![Page 6: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/6.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
6
Network Topology
Centralized Switched (Indirect) Networks• Crossbar network
– Crosspoint switch complexity increases quadratically with the number of crossbar input/output ports, N, i.e., grows as O(N2)
– Has the property of being non-blocking
7
6
5
4
3
2
1
0
76543210
7
6
5
4
3
2
1
0
76543210
![Page 7: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/7.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
7
Network Topology
From Crossbar to MINs
![Page 8: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/8.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
8
Network Topology
Centralized Switched (Indirect) Networks• Multistage interconnection networks (MINs)
– Crossbar split into several stages consisting of smaller crossbars
– Complexity grows as O(N × log N), where N is # of end nodes
– Inter-stage connections represented by a set of permutation functions
Omega topology, perfect-shuffle exchange
7
6
5
4
3
2
1
0
7
6
5
4
3
2
1
0
![Page 9: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/9.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
9
Network Topology
Appendix • Shuffle function
– N= 0…N-1
– f(i)= 2i, when I <N/2
– f(i)=(2i+1), mod N when i ≥ N/2
– Often used as a connection pattern
– unshuffle function, also often used
perfect-shuffle
7
6
5
4
3
2
1
0
unshuffle
7
6
5
4
3
2
1
0
![Page 10: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/10.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
10
Network Topology
Centralized Switched (Indirect) Networks
16 port, 4 stage Omega network
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
![Page 11: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/11.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
11
Network Topology
Centralized Switched (Indirect) Networks
16 port, 4 stage Baseline network
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
![Page 12: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/12.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
12
Network Topology
Centralized Switched (Indirect) Networks
16 port, 4 stage Butterfly network
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
![Page 13: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/13.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
13
Network Topology-Correction to Butterfly
Centralized Switched (Indirect) Networks
16 port, 4 stage Butterfly network
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
![Page 14: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/14.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
14
Network Topology
Centralized Switched (Indirect) Networks
16 port, 4 stage Cube network
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
00000001
00100011
01000101
01100111
10001001
10101011
11001101
11101111
![Page 15: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/15.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
15
Network Topology
Centralized Switched (Indirect) Networks• Multistage interconnection networks (MINs)
– MINs interconnect N input/output ports using k x k switches
› logkN switch stages, each with N/k switches
› N/k(logkN) total number of switches
– Example: Compute the switch and link costs of interconnecting 4096 nodes using a crossbar relative to a MIN, assuming that switch cost grows quadratically with the number of input/output ports (k). Consider the following values of k:
› MIN with 2 x 2 switches› MIN with 4 x 4 switches› MIN with 16 x 16 switches
![Page 16: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/16.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
16
Network Topology
Centralized Switched (Indirect) Networks• Example: compute the relative switch and link costs, N = 4096
relative_cost(2 × 2)switches
= 40962 / (22 × 4096/2 × log2 4096) = 170
relative_cost(4 × 4)switches
= 40962 / (42 × 4096/4 × log4 4096) = 170
relative_cost(16 × 16)switches
= 40962 / (162 × 4096/16 × log16
4096) = 85
relative_cost(2 × 2)links
= 8192 / (4096 × (log2 4096 + 1)) = 2/13 = 0.1538
relative_cost(4 × 4)links
= 8192 / (4096 × (log4 4096 + 1)) = 2/7 = 0.2857
relative_cost(16 × 16)links
= 8192 / (4096 × (log16
4096 + 1)) = 2/4 = 0.5
cost(crossbar)switches
= 40962
cost(crossbar)links
= 8192
![Page 17: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/17.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
17
Network Topology
Relative link cost
0
0.5
1
1.5
2
2 16 128 1024 81922
32512
8192
k
N
0-0.5 0.5-1 1-1.5 1.5-2
050
100150200250300350
2 16 128 1024 81922
32512
8192
k
N
Relative switch cost
Centralized Switched (Indirect) Networks• Relative switch and link costs for various values of k and N
(crossbar relative to a MIN)
![Page 18: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/18.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
18
Network Topology
From blocking to non-blocking again
![Page 19: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/19.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
19
Network Topology
Centralized Switched (Indirect) Networks• Reduction in MIN switch cost comes at the price of performance
– Network has the property of being blocking
– Contention is more likely to occur on network links› Paths from different sources to different destinations share one or
more links
blocking topology
X
non-blocking topology
7
6
5
4
3
2
1
0
76543210
7
6
5
4
3
2
1
0
7
6
5
4
3
2
1
0
![Page 20: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/20.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
20
Network Topology
Centralized Switched (Indirect) Networks• How to reduce blocking in MINs? Provide alternative paths!
– Use larger switches (can equate to using more switches)› Clos network: minimally three stages (non-blocking)
» A larger switch in the middle of two other switch stages provides enough alternative paths to avoid all conflicts
– Use more switches
› Add logkN - 1 stages, mirroring the original topology
» Rearrangeably non-blocking» Allows for non-conflicting paths» Doubles network hop count (distance), d» Centralized control can rearrange established paths
› Benes topology: 2(log2N) - 1 stages (rearrangeably non-blocking)
» Recursively applies the three-stage Clos network concept to the middle-stage set of switches to reduce all switches to 2 x 2
![Page 21: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/21.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
21
Network Topology
Centralized Switched (Indirect) Networks
16 port Crossbar network
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
![Page 22: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/22.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
22
Network Topology
Centralized Switched (Indirect) Networks
16 port, 3-stage Clos network
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
![Page 23: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/23.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
23
Network Topology
Centralized Switched (Indirect) Networks
16 port, 5-stage Clos network
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
![Page 24: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/24.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
24
Network Topology
Centralized Switched (Indirect) Networks
16 port, 7 stage Clos network = Benes topology
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
![Page 25: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/25.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
25
Network Topology
Centralized Switched (Indirect) Networks
Alternative paths from 0 to 1. 16 port, 7 stage Clos network = Benes topology
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
![Page 26: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/26.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
26
Network Topology
Centralized Switched (Indirect) Networks
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
Alternative paths from 4 to 0. 16 port, 7 stage Clos network = Benes topology
![Page 27: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/27.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
27
Network Topology
Centralized Switched (Indirect) Networks
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
Contention free, paths 0 to 1 and 4 to 1. 16 port, 7 stage Clos network = Benes topology
![Page 28: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/28.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
28
Network Topology
• Bidirectional MINs• Increase modularity• Reduce hop count, d• Fat tree network
– Nodes at tree leaves– Switches at tree
vertices– Total link bandwidth
is constant across all tree levels, with full bisection bandwidth
– Equivalent to folded Benes topology
– Preferred topology in many SANs
Folded Clos = Folded Benes = Fat tree network
Centralized Switched (Indirect) Networks
7
6
5
4
3
2
1
0
15
14
13
12
11
10
9
8
NetworkBisection
![Page 29: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/29.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
29
Network Topology
Myrinet-2000 Clos Network for 128 Hosts
• Backplane of the M3-E128 Switch• M3-SW16-8F fiber line card (8 ports)
http://myri.com
![Page 30: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/30.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
30
Network Topology
Myrinet-2000 Clos Network for 128 Hosts
• “Network in a Box” • 16 fiber line cards connected to the M3-E128 Switch backplane
http://myri.com
![Page 31: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/31.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
31
Network Topology
Myrinet-2000 Clos Network Extended to 512 Hosts
http://myri.com
![Page 32: Indirect Networks or Dynamic Networks](https://reader036.vdocuments.net/reader036/viewer/2022081513/56814598550346895db28a61/html5/thumbnails/32.jpg)
Sh
an
gh
ai Jia
o T
on
g U
niv
ersity
20
12
Assignment 2-2
Chose one of the following exercises:
• Calculate how many permutations nxn Omega network could support.
• Prove that folded Clos network, folded Bens network, and Fat tree network are isomorphic to each other.
SJTU@Fall 2012Parallel Processing, Low-Diameter ArchitecturesSlide 32