structure of computer systems course 10 interconnection systems
TRANSCRIPT
Structure of Computer Structure of Computer SystemsSystems
Course 10 Course 10
Interconnection systemsInterconnection systems
Interconnection systemsInterconnection systems
Purpose:Purpose: connect different components of a computer system connect different components of a computer system
(CPU, memory, interfaces, peripheral devices) - (CPU, memory, interfaces, peripheral devices) - busesbuses
interconnect multiple computer systems – networksinterconnect multiple computer systems – networks
why?why? exchange of data and instruction codesexchange of data and instruction codes synchronization and coordination of actionssynchronization and coordination of actions events signalingevents signaling
Interconnection systemsInterconnection systems
EvolutionEvolution Inside of a computer system:Inside of a computer system:
• first 3 generation of computers – dedicated connections first 3 generation of computers – dedicated connections between computer modulesbetween computer modules
• microprocessors (4microprocessors (4thth generation) – system bus generation) – system bus• high performance processors – multiple buses with different high performance processors – multiple buses with different
speeds and destinationsspeeds and destinations• multi-core processors – network-on-chipmulti-core processors – network-on-chip
Between computer systems:Between computer systems:• first generations – dedicated point-to-point serial connectionsfirst generations – dedicated point-to-point serial connections• the 80’s – network communication and Internetthe 80’s – network communication and Internet• last years – very high speed interconnection systems for Grids last years – very high speed interconnection systems for Grids
and clusters (e.g. InfiniBand)and clusters (e.g. InfiniBand)
Interconnection systemsInterconnection systems
Design decisionsDesign decisions general purpose or dedicated connectionsgeneral purpose or dedicated connections serial or parallelserial or parallel synchronous or asynchronoussynchronous or asynchronous speedspeed dimension/distance (in circuit, on board, on dimension/distance (in circuit, on board, on
system, inter-system)system, inter-system) single or multi-mastersingle or multi-master
Interconnection systemsInterconnection systems
Interconnection system examplesInterconnection system examples General purpose, parallel, asynchronous, single and General purpose, parallel, asynchronous, single and
multi-master bus (classical system bus) multi-master bus (classical system bus) General purpose, parallel, synchronous busGeneral purpose, parallel, synchronous bus Transactional parallel busesTransactional parallel buses Specialized, parallel busesSpecialized, parallel buses Serial, point-to-point, and multipoint asynchronous Serial, point-to-point, and multipoint asynchronous
busesbuses Serial, synchronous buses Serial, synchronous buses Peripheral serial buses Peripheral serial buses
General purpose, parallel, General purpose, parallel, asynchronous bus (classical bus)asynchronous bus (classical bus)
purpose – one interconnection environment for all purpose – one interconnection environment for all the components of a computerthe components of a computer
features:features: parallel bus – transfer is made on multiple parallel lines parallel bus – transfer is made on multiple parallel lines
(signals)(signals) asynchronous – the bus in not controlled by clock asynchronous – the bus in not controlled by clock
signal; signals travel on the bus with a limited speed signal; signals travel on the bus with a limited speed causing delays causing delays
single master – only one module (the CPU) can initiate single master – only one module (the CPU) can initiate transfers on the bustransfers on the bus
multi-master – multiple modules can initiate transfers multi-master – multiple modules can initiate transfers on the buson the bus
General purpose, parallel, General purpose, parallel, asynchronous bus (classical bus)asynchronous bus (classical bus)
signals (sub-buses):signals (sub-buses): address signals Aaddress signals A00- A- Ann – used for specifying the – used for specifying the
location of the transfer (memory location or I/O location of the transfer (memory location or I/O register/portregister/port
• 22nn – the maximum addressing space allowed by the bus – the maximum addressing space allowed by the bus• selecting an optimal n: selecting an optimal n:
too small – limit the addressing spacetoo small – limit the addressing space too big – wasted space on the boardtoo big – wasted space on the board
data bus Ddata bus D00-D-Dmm – used for transferring data or – used for transferring data or
instruction codesinstruction codes• m - the maximum width of the data, which can be transferred m - the maximum width of the data, which can be transferred
in a bus cycle; in accordance with the CPU structure (e.g. 8, in a bus cycle; in accordance with the CPU structure (e.g. 8, 16, 32 or 64 bits)16, 32 or 64 bits)
General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus)bus (classical bus)
signals (cont.)signals (cont.) control and command signals – used to control the traffic on the control and command signals – used to control the traffic on the
bus (examples from ISAx86)bus (examples from ISAx86)• command signals – determine the type of the transfer cyclecommand signals – determine the type of the transfer cycle
MRDC\ (memory read command), MWTC\, IORC\, IOWC\, INTA\MRDC\ (memory read command), MWTC\, IORC\, IOWC\, INTA\• control signals – enable and disable data and address amplifiers, control signals – enable and disable data and address amplifiers,
validate transfers, reset the systemvalidate transfers, reset the system DEN (data enable), ALE (address latch enable), Ready, RST (reset)DEN (data enable), ALE (address latch enable), Ready, RST (reset)
• interrupt signals – used for signaling eventsinterrupt signals – used for signaling events IRQIRQ00-IRQ-IRQ77 (interrupt request) (interrupt request)
• bus arbitration signals – in multi-master buses, used for deciding bus arbitration signals – in multi-master buses, used for deciding who has the control of the buswho has the control of the bus
BRQ, BGT or HOLD, HOLDABRQ, BGT or HOLD, HOLDA• clock signals – used for synchronization or for generating other clock signals – used for synchronization or for generating other
useful frequenciesuseful frequencies CLK, BCLK (bus clock), PCLK (peripheral clock)CLK, BCLK (bus clock), PCLK (peripheral clock)
power signalspower signals• GND, Vcc, +12V, -12VGND, Vcc, +12V, -12V
General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus)bus (classical bus)
a single bus configuration with: a single bus configuration with: CPU(s), memory CPU(s), memory modules, Input/Output interfaces and devicesmodules, Input/Output interfaces and devices
CPU Memory Memory
System bus
Address bus
Data bus
Control bus
I/O int. I/O int.
I/O dev. I/O dev.
General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus) - time diagramsbus (classical bus) - time diagrams
Memory Read Cycle
A0-An
MRDC
MWTC
D0-Dm
valid address
valid data
tcycle
taccess
Ready
General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus) - time diagramsbus (classical bus) - time diagrams
A0-An
Memory Write Cycle
MRDC
MWTC
D0-Dm
valid address
valid data
tcycle
taccess
Ready
General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus)bus (classical bus)
Advantages:Advantages: simple operation (easy to understand and debug)simple operation (easy to understand and debug) simple design of bus modulessimple design of bus modules no dimensional limitations (asynchronous mode)no dimensional limitations (asynchronous mode) single communication environment for all the single communication environment for all the
components of a computer components of a computer
Drawbacks:Drawbacks: low speed – limited to the slowest modulelow speed – limited to the slowest module limited number of modules connected on the bus (10-limited number of modules connected on the bus (10-
16 – see fan-out of a TTL circuit)16 – see fan-out of a TTL circuit)
General purpose, parallel, asynchronous General purpose, parallel, asynchronous bus (classical bus)bus (classical bus)
Examples of general purpose, parallel, asynchronous Examples of general purpose, parallel, asynchronous buses:buses: 8086 bus8086 bus ISA (ISA (Industry Standard ArchitectureIndustry Standard Architecture), EISA (extended ISA)), EISA (extended ISA) S-100 S-100
EISA connectors and Interface board
General purpose, parallel, General purpose, parallel, synchronous bussynchronous bus
Why ? (Purpose): increase the speed through a Why ? (Purpose): increase the speed through a better control of timingbetter control of timing
How ? (Principles)How ? (Principles) every signal on the bus is related (synchronized) with every signal on the bus is related (synchronized) with
the clock signalthe clock signal modules may anticipate next steps (does not have to modules may anticipate next steps (does not have to
wait until a signal arrives to the module, as in wait until a signal arrives to the module, as in asynchronous mode)asynchronous mode)
modules on the bus must have some intelligencemodules on the bus must have some intelligence Examples:Examples:
PCIPCI P6 (Pentium Pro) busP6 (Pentium Pro) bus
General purpose, parallel, General purpose, parallel, synchronous bussynchronous bus
Block read cycle (PCI bus)Block read cycle (PCI bus) request for a block of data (first period)request for a block of data (first period) memory generates data from consecutive addressesmemory generates data from consecutive addresses
General purpose, parallel, General purpose, parallel, synchronous bussynchronous bus
Advantages:Advantages: higher transfer speedhigher transfer speed small average access timesmall average access time promotes block transfers (good for cache line transfers)promotes block transfers (good for cache line transfers)
Disadvantages:Disadvantages: dimension of the bus is limited by the clock frequency dimension of the bus is limited by the clock frequency
• if the bus is too long, clock signal is not synchronized with itself if the bus is too long, clock signal is not synchronized with itself at the two ends of the bus (the speed of the signal is limited)at the two ends of the bus (the speed of the signal is limited)
more complex design of modules connected on the busmore complex design of modules connected on the bus harder debugging processharder debugging process
Transactional, parallel, Transactional, parallel, synchronous busessynchronous buses
Why ? (Purpose): increase the speed of the bus Why ? (Purpose): increase the speed of the bus How ?(Principles):How ?(Principles):
pipeline implementation of a transfer on the buspipeline implementation of a transfer on the bus use transaction (set of operations) instead of transfer use transaction (set of operations) instead of transfer
cyclescycles a transaction divided into stages that use different a transaction divided into stages that use different
signal groups and therefore can be executed in a signal groups and therefore can be executed in a pipeline mannerpipeline manner
Example: Example: P6 (Pentium pro) busP6 (Pentium pro) bus
Transactional, parallel, Transactional, parallel, synchronous busessynchronous buses
Example: the P6 busExample: the P6 bus Phases:Phases:
• ArbitrationArbitration – decides which master has access on the bus– decides which master has access on the bus
• Transfer requestTransfer request – specifies the request (read or write, – specifies the request (read or write, start address, number of bytes)start address, number of bytes)
• SnoopingSnooping – detect and solve cache inconsistencies– detect and solve cache inconsistencies
• Error Error – – detect and solve transmission errors (ECC – error detect and solve transmission errors (ECC – error correction code on data and parity on address and command correction code on data and parity on address and command signals)signals)
• Response Response – specifies the type of the answer (now, – specifies the type of the answer (now, delayed, refused)delayed, refused)
• TransferTransfer – data transfer in accordance with the request– data transfer in accordance with the request
Transactional, parallel, Transactional, parallel, synchronous busessynchronous buses
P6 bus (cont.)P6 bus (cont.) 1 2 3 4 5 6 7 8 9 1
0 11
12
13
14
15
16
BCLK
Arbitration
Request
Error
Snooping
Response
Transfer
Concurrent transactions on the P6 bus
Specialized, parallel busesSpecialized, parallel buses buses specialized for a group of peripheral devices (e.g. buses specialized for a group of peripheral devices (e.g.
HDD, DVD, etc.)HDD, DVD, etc.) Examples: IDE, SCASI (read “scazi”), ATAExamples: IDE, SCASI (read “scazi”), ATA SCASI details:SCASI details:
assures communication between an initiator (computer) and a assures communication between an initiator (computer) and a target (peripheral device)target (peripheral device)
protocol steps:protocol steps:• initiator sends a command (command descriptor block) to the targetinitiator sends a command (command descriptor block) to the target• target respond with a status code (success, error or busy)target respond with a status code (success, error or busy)• target returns a Check condition and the initiator respond with SCI target returns a Check condition and the initiator respond with SCI
Request sense commandRequest sense command there are about 60 command types grouped in 4 categories: there are about 60 command types grouped in 4 categories:
• non-data, read, write and bidirectionalnon-data, read, write and bidirectional SCASI and ATA have serial versions tooSCASI and ATA have serial versions too
Multi-master parallel busesMulti-master parallel buses
Issue:Issue: the bus is a shared resource; only one master can control the the bus is a shared resource; only one master can control the
bus at a given momentbus at a given moment how to establish who has the control of the bushow to establish who has the control of the bus
Solutions:Solutions: centralized controlcentralized control
• the central CPU is controlling the access on the bus – the central CPU is controlling the access on the bus – example: DMA transfer (HOLD, HOLDA handshaking mechanism)example: DMA transfer (HOLD, HOLDA handshaking mechanism)
• bus arbiter circuit bus arbiter circuit example: I8289 – bus arbiter (BRQ, BGT, CBRQ)example: I8289 – bus arbiter (BRQ, BGT, CBRQ)
distributed controldistributed control• every master has an arbitration component:every master has an arbitration component:
serial linkserial link token basestoken bases
Serial busesSerial buses Serial bus v.s. parallel busSerial bus v.s. parallel bus
less signals (lines)less signals (lines) longer transmission distanceslonger transmission distances cheaper implementation (e.g. less wires, less space on cheaper implementation (e.g. less wires, less space on
the PCB - printed circuit board, less pins on the circuits)the PCB - printed circuit board, less pins on the circuits) speed:speed:
• old view – lower speed than parallel connectionold view – lower speed than parallel connection• new view – higher speed than parallel connectionnew view – higher speed than parallel connection• explanation - its easier to increase more than 10 times the explanation - its easier to increase more than 10 times the
transmission frequency on serial bus than on a parallel one (see transmission frequency on serial bus than on a parallel one (see electro-magnetic interferences in case of long parallel lines)electro-magnetic interferences in case of long parallel lines)
• consequence – most of the parallel buses are replaced with consequence – most of the parallel buses are replaced with serial ones:serial ones:
serial ATA and SCASIserial ATA and SCASI network-on-chipnetwork-on-chip serial system buses – e.g. I2C for microcontrollersserial system buses – e.g. I2C for microcontrollers
Serial busesSerial buses
Design decisions:Design decisions: synchronous or asynchronoussynchronous or asynchronous point-to-point or multipointpoint-to-point or multipoint character-based or message-basecharacter-based or message-base information coding: voltage levels, differential information coding: voltage levels, differential
voltages, light impulses, radio wavesvoltages, light impulses, radio waves flow control: hardware, software, protocol-flow control: hardware, software, protocol-
basedbased error detection and correctionerror detection and correction
Serial buses – Serial buses – Synchronous transmissionSynchronous transmission
synchronous – an extra clock signal synchronous – an extra clock signal controls the transmissioncontrols the transmission
Clk
Data
0 1 1 0 1 0 0 0
Shift reg.Data signal
Clock signal
Shift reg.
Sender ReceiverGND
Serial buses – Serial buses – Synchronous transmissionSynchronous transmission
FeaturesFeatures easy to implement, no other synchronization easy to implement, no other synchronization
mechanisms are neededmechanisms are needed requires an extra signal (clock), inefficient use requires an extra signal (clock), inefficient use
of wiresof wires hard to synchronize sender and receiver on hard to synchronize sender and receiver on
long distances long distances
Serial buses – Serial buses – Synchronous transmissionSynchronous transmission
Example: I2C protocol (read: eye to see) Example: I2C protocol (read: eye to see) multi-point, serial, synchronous busmulti-point, serial, synchronous bus used in microcontroller systems to connect external components: used in microcontroller systems to connect external components:
memory circuits, analog-digital convertermemory circuits, analog-digital converter master-slave protocol (1 master controls the traffic on the bus)master-slave protocol (1 master controls the traffic on the bus) uses two lines: SCL- clock and SDA - datauses two lines: SCL- clock and SDA - data
address
Serial buses Serial buses Asynchronous transmissionAsynchronous transmission
FeaturesFeatures no clock signalno clock signal synchronization made through the specific structure of synchronization made through the specific structure of
the transmitted data the transmitted data the sender and the transmitter must use the same the sender and the transmitter must use the same
protocol that specifies:protocol that specifies:• transmission frequency transmission frequency • number of bits/character or bytes/messagenumber of bits/character or bytes/message• coding of logical 0 and1coding of logical 0 and1• data-flow control mechanismsdata-flow control mechanisms• error detection methoderror detection method
Serial buses Serial buses Asynchronous transmissionAsynchronous transmission
best known protocol (standard): RS232 or V24best known protocol (standard): RS232 or V24 Specifications of RS232 protocol:Specifications of RS232 protocol:
point-to-point bidirectional transmission on characterspoint-to-point bidirectional transmission on characters standard frequencies: standard frequencies: 300,600, 1200 ...9600 ...Bauds300,600, 1200 ...9600 ...Bauds bits/character: 6,7, 8 bits bits/character: 6,7, 8 bits 1 START bit = 0 and 1or 2 STOP bits = 11 START bit = 0 and 1or 2 STOP bits = 1 error detection – optional parity bit, even or odderror detection – optional parity bit, even or odd flow-control protocols:flow-control protocols:
• software (XON/XOFF) – with ASCII codes for starting (XON) and software (XON/XOFF) – with ASCII codes for starting (XON) and stopping (XOFF) the transmissionstopping (XOFF) the transmission
• hardware – with 2 pairs of signals: RTS-CTS or DSR-DTRhardware – with 2 pairs of signals: RTS-CTS or DSR-DTR
Serial buses – Serial buses – Asynchronous transmissionAsynchronous transmission
Specifications of RS232 protocol (cont.):Specifications of RS232 protocol (cont.): signals:signals:
• RXD – receive dataRXD – receive data
• TXD – transmit dataTXD – transmit data
• GND – ground (voltage reference)GND – ground (voltage reference)
• RTS – request to sendRTS – request to send
• CTS – clear to sendCTS – clear to send
• DSR - data set readyDSR - data set ready
• DTR – data terminal readyDTR – data terminal ready
max. transmission distance: 100mmax. transmission distance: 100m data format: data format: Start (1 bit = 1), data (6-8 bits), Parity (1 bit), Stop (1-Start (1 bit = 1), data (6-8 bits), Parity (1 bit), Stop (1-
2 bits=1)2 bits=1)
RXDTXDGNDRTSCTSDSRDTR
RXDTXDGNDRTSCTSDSRDTR
Sender Receiver
Start data bits (6-8 bits) Parity Stop (1-2 bits)
Serial buses – Serial buses – Asynchronous transmissionAsynchronous transmission
Specifications of the RS485 protocol:Specifications of the RS485 protocol: multi-point, serial, asynchronous transmission on multi-point, serial, asynchronous transmission on
characterscharacters• transceivers (receiver-transmitter circuit) with three-state transceivers (receiver-transmitter circuit) with three-state
capabilitycapability transmission on two twisted wires (A and B) transmission on two twisted wires (A and B) bit coding: differential voltagebit coding: differential voltage
Multipoint interconnectionsMultipoint interconnections
RingRing busbus treetree matrixmatrix hyper-cubehyper-cube switch fabricsswitch fabrics
Multipoint interconnectionsMultipoint interconnections
Crossbar switchCrossbar switch multiple connections between multiple componentsmultiple connections between multiple components
• multi-bus access of CPUs to memory modulesmulti-bus access of CPUs to memory modules
Multipoint interconnectionsMultipoint interconnections
Issues:Issues: reduce the number of connectionsreduce the number of connections reduce the number of interfacesreduce the number of interfaces reduce communication delays (data latency)reduce communication delays (data latency) reduce the number of hops (nodes) involved reduce the number of hops (nodes) involved
in a transferin a transfer increase the bandwidthincrease the bandwidth
Multipoint interconnectionsMultipoint interconnections Implementations:Implementations:
latest Intel processorslatest Intel processors (Sand Bridge) (Sand Bridge) • internal ring between cache memoriesinternal ring between cache memories• QPI – QuickPath InterconnectQPI – QuickPath Interconnect – connection between CPUs – connection between CPUs
InfiniBandInfiniBand • a switched fabric communications link used in high-a switched fabric communications link used in high-
performance computing and enterprise data centersperformance computing and enterprise data centers• features:features:
high throughput, high throughput, low latency, low latency, quality of service and failover, quality of service and failover, scalable. scalable.
• The InfiniBand architecture specification defines a The InfiniBand architecture specification defines a connection between processor nodes and high performance connection between processor nodes and high performance I/O nodes such as storage devices. Infiniband host bus I/O nodes such as storage devices. Infiniband host bus adapters and network switches are manufactured by adapters and network switches are manufactured by Mellanox and Intel Mellanox and Intel
Multipoint interconnectionsMultipoint interconnections Wishbone Bus:Wishbone Bus:
an open source hardware computer an open source hardware computer bus intended to let the parts of an bus intended to let the parts of an integrated circuit communicate with integrated circuit communicate with each other. each other.
the aim is to allow the connection of the aim is to allow the connection of differing cores to each other inside differing cores to each other inside of a chip. of a chip.
is used by many designs in the is used by many designs in the OpenCores project.OpenCores project.