switches and indirect networks computer architecture amano, hideharu textbook pp. 92~13 0
TRANSCRIPT
![Page 1: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/1.jpg)
Switches and indirect networks
Computer Architecture
AMANO, Hideharu
Textbook pp. 92~13 0
![Page 2: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/2.jpg)
Switch connected parallel machines Simple extension of bus connected
machines PU-Memory connection: UMA Node-node connection: NUMA, NORA Snoop is impossible Directory based methods or compiler assisted
methods are used for UMA/NUMA How to build large scale systems
![Page 3: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/3.jpg)
Switch connected UMA
Switch
Local Memory
CPU
Interface
Main Memory
. . . .
… .
Local Memory is sometimes dispensable
![Page 4: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/4.jpg)
Switch connected UMABlocking
Switch
Local Memory
CPU
Interface
Main Memoryn
. . . .
1 … .
0
1
n
0
Shared Memory
![Page 5: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/5.jpg)
Switch connected UMAInterleaving
Switch
Local Memory
CPU
Interface
Main Memoryn
. . . .
1 … .0
Shared Memory
… .… .
Size: Double word or Cache Line
![Page 6: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/6.jpg)
Switch connected UMA with circular connection
Switch
CPU
Interface
Main Memory
. . . .
… .
Main memory is used as a home memoryInterleave is often difficult
![Page 7: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/7.jpg)
Switch connected NUMA
Switching Fabrics
…
Symmetric Multi-Processor
Switching Fabrics sometimes become hierarchical structure→ Fat TreeDirectory based Cache coherent methods are used→ CC-NUMATypical recent high performance server: SUN’s or IBM’s
![Page 8: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/8.jpg)
Switch based network
Single stage Crossbar
Multi-stage Symmetric: Multistage Interconnection Network Asymmetric: Fat-tree, base-m n-cube → Direct
interconnection network
![Page 9: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/9.jpg)
Crossbar switch
n
m
Cross point: smallswitching element
The number ofcross points:nxm
Extension of the buses
![Page 10: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/10.jpg)
Non-blocking property
n
m
For differentdestination,conflict free
![Page 11: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/11.jpg)
Head Of Line (HOL) conflict
n
m
X
Arbiter is required for each bus
The buffer is required
The number of crosspoint is not dominant.
![Page 12: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/12.jpg)
Input buffer switch
Crossbar
Input buffer
One of conflicting packets is selected.Others are stored Into the input buffer.
![Page 13: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/13.jpg)
Merit/demerit of Crossbars
Non-blocking property Simple structure/Control The hardware for cross-points usually do not
limit the system (Fallacy of crossbars) Extension is difficult by the pin-limitation of L
SIs If pins can be used, a large crossbar can be const
ructed → Earth simulator
![Page 14: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/14.jpg)
Earth Simulator (2002,NEC)V
ect
or
Pro
cess
or
Vect
or
Pro
cess
or
…
Vect
or
Pro
cess
or
0 1 7
Shared Memory16GB
Vect
or
Pro
cess
or
Vect
or
Pro
cess
or
…
Vect
or
Pro
cess
or
0 1 7
Shared Memory16GB
Vect
or
Pro
cess
or
Vect
or
Pro
cess
or
…
Vect
or
Pro
cess
or
0 1 7
Shared Memory16GB
….
639 Inputs crossbar (16GB/s x 2)
Node 0 Node 1 Node 639
Peak performance40TFLOPS
![Page 15: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/15.jpg)
MIN ( Multistage Interconnection Network) Multistage connected switching elements form a large switch.
Symmetric Smaller number of cross-points, high
degree of expandability Bandwidth is often degraded Latency is stretched
![Page 16: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/16.jpg)
Classification of MIN
Blocking network : Conflict may occur for destination is different :NlogN type standard MIN,πnetwork,
Re-arrangeable : Conflict free scheduling is possible : Benes network 、 Clos network( rearrangeable configuration )
Non-blocking : Conflict free without scheduling : Clos network (non-blocking configuration) 、 Batcher-Banyan network
![Page 17: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/17.jpg)
Properties of MIN
Throughput for random communication Permutation capability Partition capability F ault torelance Routing
![Page 18: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/18.jpg)
Blocking Networks
Standard NlogN networks Omega network Generalized Cube Baseline
Pass through ratio (throughput) is the same. Π network
![Page 19: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/19.jpg)
Omega network
The number of switching element (2x2 , in this case ) is 1/2 N x LogN
000001
010
011
100101
110111
000001
010
011
100101
110111
![Page 20: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/20.jpg)
Perfect Shuffle
Rotate to left
000001010011100101110111
000010100110001011101111
Inverse ShuffleRotate to right
![Page 21: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/21.jpg)
Destination Routing
000001
010
011
100101
110111
000001
010
011
100101
110111
Check the destination tag from MSBIf 0 use upper link, else use lower link.
1→ 3
0
1
1
5→6
1
1 0
![Page 22: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/22.jpg)
Blocking Property
For different destination, multiple paths conflict
000001
010
011
100101
110111
000001
010
011
100101
110111
X
0→04→2
![Page 23: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/23.jpg)
For using large switching elements ( Delta network )
In the current art of technology, 8x8 (4x4) crossbars are advantageous.
00011011
20213031
00011011
20213031
0123
0123
1
2
Shuffle connection is also used.
![Page 24: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/24.jpg)
Omega network
The same connection is used for all stages. destination routing A lot of useful permutations are available. Problems on partitioning and expandability.
![Page 25: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/25.jpg)
Generalized Cube
000001
010
011
100101
110111
000001
010
011
100101
110111
000100
100
110
100
101
Links labeled with 1bit distance are connectedto the same switching element.
000
010
000
001
![Page 26: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/26.jpg)
Routing in Generalized Cube
000001
010
011
100101
110111
000001
010
011
100101
110111
The source label and destination label is compared (Ex-Or ):Same(0) : Straight Different (1) : Exchange
001→011010
01 0
![Page 27: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/27.jpg)
Partitioning
000001
010
011
100101
110111
000001
010
011
100101
110111
The communication in the upper half never disturbs the lower half.
![Page 28: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/28.jpg)
Expandability
A size of network can be used as an element of larger size networks
![Page 29: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/29.jpg)
Generalized Cube
Destination routing cannot be applied. The routing tag is generated by exclusive or
of source label and destination label. Partitioning Expandability
![Page 30: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/30.jpg)
Baseline Network
000001
010
011
100101
110111
000001
010
011
100101
110111
The area of shuffling is changed.
001
100
010
001
3bit shuffle 2bit shuffle
![Page 31: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/31.jpg)
Destination Routing in Baseline network
000001
010
011
100101
110111
000001
010
011
100101
110111
Just like Omega network
1
1
0
![Page 32: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/32.jpg)
Partitioning in Baseline
000001
010
011
100101
110111
000001
010
011
100101
110111
![Page 33: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/33.jpg)
Baseline network
Providing both benefits of Omega and Generalized Cube Destination Routing Partitioning Expandability
Used in NEC’s Cenju
![Page 34: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/34.jpg)
Π network
Tandem connection of two Omega networks
000001
010
011
100101
110111
000001
010
011
100101
110111
![Page 35: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/35.jpg)
Bit reversal permutation(Used in FFT)
Conflicts occur in Omega network.
000001
010
011
100101
110111
000001
010
011
100101
110111
0426
1537
0123
4567
![Page 36: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/36.jpg)
Bit reversal permutation in Π network000001
010
011
100101
110111
000001
010
011
100101
110111
0426
1537
0527
1436
The first Omega : Upper input has priority.The next Omega : Destination Routing Conflict free
![Page 37: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/37.jpg)
Permutation capacity
All possible permutation is conflict free = Rearrangeable networks
Tree tandem connection of Omega network is rearrangeable.
The tandem connection of Omega and Inverse Omega (Baseline and Inverse Baseline) is rearrangeable. Benes network
![Page 38: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/38.jpg)
Benes Network
Note that the center of stage is shared. The rearrangeable network with the s
mallest hardware requirement.
000001
010
011
100101
110111
000001
010
011
100101
110111
![Page 39: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/39.jpg)
Non-blocking network
Clos network m>n1+n2-1 : Non-blocking m>=n2 : Rearrangeable Else: Blocking
![Page 40: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/40.jpg)
Clos network
... ...
n1xm r1xr2 mxn2
r1 m r2
m=n1+n2-1 : Non-blockingm=n2 : Rearrangeablem<n2 : Blocking
The number of intermediatestage dominates the permutationcapability.
3-stage
![Page 41: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/41.jpg)
Batcher network
5704
2136
5740
1263
0457
6321
0123
4567
Bitonic sorting network
![Page 42: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/42.jpg)
Batcher-Banyan
5704
2136
5740
1263
0457
6321
0123
4567
Sorted input is conflict free in the banyan network
OmegaBaseline
![Page 43: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/43.jpg)
Banyan networks
Only a path is provided between source and destination. The number of intermediate stages is flexible. Approach from graph theory SW-Banyan , CC-Banyan , Barrel Shifter
Irregular structure is allowed.
![Page 44: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/44.jpg)
Batcher-banyan
If there are multiple packets to the same destination, the conflict free condition is broken→ The other packets may conflict. The extension of banyan network is required.
The number of stages is large.→ Large pass through time The structure of sorting network is simple.
![Page 45: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/45.jpg)
Classification of MINs
Omega
Baseline
Generalized Cube
π
Benes
Clos
BatcherBanyan
Banyan
Blocking
Rearrageble
Nonblocking
![Page 46: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/46.jpg)
Fault tolerant MINs
Multiple paths Redundant structure is required. On-the-fly fault recovery is difficult. Improving chip yield.
![Page 47: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/47.jpg)
Extra Stage Cube (ESC)
An extra stage + Bypass mechanism
000001
010
011
100101
110111
000001
010
011
100101
110111
X
If there is a fault on stages or links, another path is used.
![Page 48: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/48.jpg)
The buffer in switching element
Conflicting packets are stored into buffers.
000001
010
011
100101
110111
000001
010
011
100101
110111
![Page 49: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/49.jpg)
Hot spot contention
Buffer is saturated in the figure of t ree状 ( Tree Saturation)
000001
010
011
100101
110111
Hot spot
![Page 50: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/50.jpg)
Relaxing the hot spot contention Wormhole routing with Virtual channels →
Direct network Message Combining
Multiple packets are combining to a packet inside a switching element (IBM RP3)
Implementation is difficult (Implemented in SNAIL)
![Page 51: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/51.jpg)
Other issues in MINs
MIN with cache control mechanism Directory on MIN Cache Controller on MIN
MINs with U-turn path → Fat tree
![Page 52: Switches and indirect networks Computer Architecture AMANO, Hideharu Textbook pp. 92~13 0](https://reader036.vdocuments.net/reader036/viewer/2022062802/56649ece5503460f94bda8f2/html5/thumbnails/52.jpg)
Exercise
Every path between source and destination is determined with the destination routing in Omega network. Prove (or explain) the above theory in Omega network with 8-input/output.