anshul kumar, cse iitd ece729 : advanced computer architecture lecture 27, 28: interconnection...

Anshul Kumar, CSE IITD

ECE729 : Advanced Computer ECE729 : Advanced Computer ArchitectureArchitecture

ECE729 : Advanced Computer ECE729 : Advanced Computer ArchitectureArchitecture

Lecture 27, 28: Interconnection Mechanisms

In Multiprocessors

29th, 31st March, 2010

Anshul Kumar, CSE IITD slide 2

Connecting Processors and MemoriesConnecting Processors and MemoriesConnecting Processors and MemoriesConnecting Processors and Memories

• Shared Buses

• Interconnection Networks– Static Networks

– Dynamic Networks

P P P P

M M M

Interconnection Network

M

M M M

P P P P

M M M

Interconnection Network

M

M M M

Global Interconnection Network

M M M


Shared BusShared BusShared BusShared Buseach processor sees this picture:

processing

bus access

timentransactiobustimeprocessing

timentransactiobusnutilizatiobus

prob of a processor using the bus = prob of a processor not using the bus = 1 – prob of none of the n processors using the bus = (1 – )n

prob of at least one processor using the bus = 1 – (1 – )n

achieved BW on a relative scale = 1 – (1 – )n

required BW = n available BW = 1


Effect of re-submitted requestsEffect of re-submitted requestsEffect of re-submitted requestsEffect of re-submitted requests

A

W

1-

Probability of a request being accepted PA

Probability of a request not being accepted 1 - PA

prob = qA

prob = qW

Active state

Probability of making a bus requestProbability of not making a bus request

Wait state

Probability of making a bus request 1Probability of not making a bus request 0



A W

(1-PA )1- + PA 1-PA

PA

1 also11

111

1

raterequestactual

111

a

aA

na

AA

A

A

A

wA

AWA

A

AA

AA

aPanBW

PP

P

P

P

qqa

qqP

P

PP

Pq

prob = qA prob = qW

Shared Bus : BW per proc

-0.100

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

BW required (req probability)

BW a

chie

ved

n = 2

n = 3

n = 4

n = 2

n = 3

n = 4

withoutresubmission

withresubmission

Shared Bus : utilization

-0.200

0.000

0.200

0.400

0.600

0.800

1.000

1.200

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

req probability

utili

zatio

n

n = 2

n = 3

n = 4

n = 2

n = 3

n = 4

withoutresubmission

withresubmission


Waiting timeWaiting timeWaiting timeWaiting time

busa

abus

A

A

A

AAbus

iA

iAbus

Ai

Ai

busw

Ai

A

th

bus

TTP

P

P

PPTPiPT

PPTiT

PP

)(ii

T i

1

)1(1

1 )1(

)1( time waitingof valueExpected

)1( thisofy probabilit

attempt 1on accepted and times rejected isrequest if

timewaiting

21

1


Buses vs. NetworksBuses vs. NetworksBuses vs. NetworksBuses vs. Networks

Buses• Shared media• Lower Cost• Lower throughput• Scalability poor

Networks• Switched paths• Higher cost• Higher throughput• Scalability better


Interconnection NetworksInterconnection NetworksInterconnection NetworksInterconnection Networks• Topology : who is connected to whom

• Direct / Indirect : where is switching done

• Static / Dynamic : when is switching done

• Circuit switching / packet switching : how are connections established

• Store & forward / worm hole routing : how is the path determined

• Centralized / distributed : how is switching controlled

• Synchronous/asynchronous : mode of operation


PM

Direct and Indirect NetworksDirect and Indirect NetworksDirect and Indirect NetworksDirect and Indirect Networks

PMS

PMS

SMP

SMP

PM

PM

PM

SW

ITC

H

DIRECTINDIRECT

node node

node node

link

linklink link

node

node

node

node

link

link

link

link


Static and Dynamic NetworksStatic and Dynamic NetworksStatic and Dynamic NetworksStatic and Dynamic Networks

Static Networks• fixed point to point

connections• usually direct• each node pair may

not have a direct

connection• routing through nodes

Dynamic Networks• connections

established as per need• usually indirect• path can be

established between

any pair of nodes• routing through

switches


Static Network TopologiesStatic Network TopologiesStatic Network TopologiesStatic Network Topologies

Linear

Star

2D-Mesh

Tree

Non-uniform connectivity


Static Networks Topologies- contd.Static Networks Topologies- contd.Static Networks Topologies- contd.Static Networks Topologies- contd.

Ring

Fully ConnectedTorus

Uniform connectivity


Illiac IV Mesh NetworkIlliac IV Mesh NetworkIlliac IV Mesh NetworkIlliac IV Mesh Network

0 1 2

3 4 5

6 7 8

01

2

3

45

6

7

8

neighbors of node r :(r 1) mod 9 and(r 3) mod 9 Chordal Ring


Fat Tree NetworkFat Tree NetworkFat Tree NetworkFat Tree Network


Dynamic NetworksDynamic NetworksDynamic NetworksDynamic Networks

k kcross -bar

switch

building block for multi-stagedynamic networks

2 2switch

straight exchange upperbroadcast

lowerbroadcast

simplestcross-bar


Baseline NetworkBaseline NetworkBaseline NetworkBaseline Network

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

blocking can occur

A

B

C

A

B

C


Benes NetworkBenes NetworkBenes NetworkBenes Network

non-blocking


Network on Chip (NoC)Network on Chip (NoC)Network on Chip (NoC)Network on Chip (NoC)Processing node

LinkRouter


Switching MechanismSwitching MechanismSwitching MechanismSwitching Mechanism

• Circuit Switching (connection oriented communication)– A circuit is established between the source and

the destination

• Packet Switching (connectionless communication)– Information is divided into packets and each

packet is sent independently from node to node


Routing in NetworksRouting in NetworksRouting in NetworksRouting in Networks

nodeincomingmessage

outgoingmessage

header payload/datastore & forward

routing

worm holerouting

time

BW

H

BW

l

BW

l

BW

Hnlatency

BW

l

BW

Hnlatency


Routing in presence of congestionRouting in presence of congestionRouting in presence of congestionRouting in presence of congestion

• Worm hole routing– When message header is blocked, many links

get blocked with the message

• Solution: cut-through routing– When message header is blocked, tail is

allowed to move, compressing the message into a single node


Routing OptionsRouting OptionsRouting OptionsRouting Options

• Deterministic routing: always same path followed

• Adaptive routing: best path selected to minimize congestion

• Source based routing: message specifies path to destination

• Destination based routing: message specifies only destination address


Some Performance ParametersSome Performance ParametersSome Performance ParametersSome Performance Parameters

time

sender

receiver

time of flight

overhead

overhead

Tx time=bytes/BW

Tx time=bytes/BW

transport latency

total latency


Other ParametersOther ParametersOther ParametersOther Parameters

• Throughput Bandwidth (no credit for header)

• Bisection bandwidth = BW across a bisection

• Node degree

• Network Diameter

• Cost

• Fault Tolerance


Multidimensional Grid/MeshMultidimensional Grid/MeshMultidimensional Grid/MeshMultidimensional Grid/Mesh

Size

=k k …. k (n times)

= k n

Diameter

= (k-1) n without end around

connections

= k n /2 with end around

connections

k-ary n-cube

for (Binary) Hypercube : k = 2


Grid/Mesh Performance - 1Grid/Mesh Performance - 1Grid/Mesh Performance - 1Grid/Mesh Performance - 1

cycle ain req message of prob is

dimension one along

hops of no. av. is

dimensions ofnumber is

rate arrival Message

r

k

n

knr

d

d

kd


Grid/Mesh Performance - 2Grid/Mesh Performance - 2Grid/Mesh Performance - 2Grid/Mesh Performance - 2

2Occupancy Server

2 rate Service

sd

s

Tkr

T

n


Switch PerformanceSwitch PerformanceSwitch PerformanceSwitch Performance

k mcross -bar

switch

mm

mm

m

mm

E(i)i

rrCq(i)ki

T

r

i

i

ii

ikii

k

11

)1(

portsoutput of num

portoutput specific a including patterns address offraction

requests ofout accepted requests of no. expected

)1( ports on requests ussimultaneo of prob

timeservice same requires packet)(or

mesageeach that assumed isit Here

cycle service one duringport

input an at request of probLet


Switch Performance – contd.Switch Performance – contd.Switch Performance – contd.Switch Performance – contd.

kk

k

i

iki

ik

k

i

ikii

k

k

i

ikii

kik

i

ikii

k

k

i

ikii

ki

k

i

m

rmmrr

m

mmm

rrm

mCmrrCm

rrCmm

mrrCm

rrCmm

m

iqiE

1)1(1

)1(1

)1(

)1(1

)1(

)1(1

1

)()( scale) relative(on BW Expected

00

00

0

0


Switch Performance – contd.Switch Performance – contd.Switch Performance – contd.Switch Performance – contd.

waiting?of because Delays

conflicts? to due submission-re request of Effect

requests of acceptance of prob

) that (assuming as wellas than less is this

conflicts) port output of (because BW Expected

conflicts) port output no werethere (if BW Expected

bandwidth Requested

kr

BWP

rr km

m

rmm

m

r k

A

k

1

1



link ofBW

1 time waitingaverage

'

'1

1'

) and states graph with Markov (using

' raterequestactual

lHtimecycleT

TP

P

kr

BWP

m

rmmBW

rPr

rr

qq

qqrr

A

A

A

k

A

wA

wA


BufferingBufferingBufferingBuffering

Where are buffers located?

• Buffering before switching (k buffers, one at each input port)

• Buffering after switching (m buffers, one at each output port)

• Virtual output buffering (at each of the k inputs there are m buffers, one for each output)


BufferingBufferingBufferingBuffering

k mcross -bar

switch

k mcross -bar

switch

k mcross -bar

switch

B

C, AC

BA

B

CAC

B, A

CA

B

anshul kumar, cse iitd ece729 : advanced computer architecture lecture 27, 28: interconnection...

Documents

cse iitdanshul kumar

cse iitdslide

qwanshul kumar

cse iitdece729

cse iitdchart5000

pa pa prob

bus requestprobability

pa probability