cse 502 graduate computer architecture lec 15 – midterm review

159
CSE 502 Graduate Computer Architecture Lec 15 – MidTerm Review Larry Wittie Computer Science, StonyBrook University http://www.cs.sunysb.edu/~cse502 and ~lw Slides adapted from David Patterson, UC-Berkeley cs252- s06 11/3/2009 1 CSE502-F09, Lec 15 - MTRevu

Upload: hermione-mendez

Post on 02-Jan-2016

30 views

Category:

Documents


4 download

DESCRIPTION

CSE 502 Graduate Computer Architecture Lec 15 – MidTerm Review. Larry Wittie Computer Science, StonyBrook University http://www.cs.sunysb.edu/~cse502 and ~lw Slides adapted from David Patterson, UC-Berkeley cs252-s06. Review: Some Basic Unit Definitions. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

CSE 502 Graduate Computer

Architecture

Lec 15 – MidTerm Review

Larry WittieComputer Science, StonyBrook University

http://www.cs.sunysb.edu/~cse502 and ~lw

Slides adapted from David Patterson, UC-Berkeley cs252-s06

11/3/2009 1CSE502-F09, Lec 15 - MTRevu

Page 2: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

Review: Some Basic Unit Definitions• Kilobyte (KB) – 210 (1,024) or 103 (1,000 or “thousand”) Bytes (a 500-page book)

• Megabyte (MB) – 220 (1,048,576) or 106 (“million”) Bytes (1 wall of 1000 books)

• Gigabyte (GB) – 230 (1,073,741,824) or 109 (“billion”) Bytes (a 1000-wall library)

• Terabyte (TB) – 240 (1.100 x 1012) or 1012 (“trillion”) Bytes (1000 big libraries)

• Petabyte (PB) – 250 (1.126 x 1015) or 1015 (“quadrillion”) Bytes (½ hr satellite data)

• Exabyte – 260 (1.153 x 1018) or 1018 (“quintillion”) Bytes (40 days: 1 satellite’s data)

Remember that 8 bits = 1 Byte• millisec (ms) – 10-3 (“a thousandth of a”) second light goes 300 kilometers

icrosec (s) – 10-6 (“a millionth of a”) second light goes 300 meters

• nanosec (ns) – 10-9 (“a billionth of a”) second light goes 30 cm, 1 foot

• picosec (ps) – 10-12 (“a trillionth of a”) secondlight goes 300 m, 6 hairs

• femtosec (fs) – 10-15 (“one quadrillionth”) second light goes 300 nm, 1 cell

• attosec – 10-18 (“one quintillionth of a”) second light goes 0.3 nm, 1 atom

Page 3: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

CSE 502 Graduate Computer

Architecture

Lec 1-2 - Introduction

Larry WittieComputer Science, StonyBrook University

http://www.cs.sunysb.edu/~cse502 and ~lw

Slides adapted from David Patterson, UC-Berkeley cs252-s06

9/1,3,8/2009 3CSE502-F09, Lec 01-3 - intro

Page 4: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 4

1

10

100

1000

10000

1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006

Performance (vs. VAX-11/780)

25%/year

52%/year

??%/year

Crossroads: Uniprocessor Performance

• VAX : 25%/year 1978 to 1986• RISC + x86: 52%/year 1986 to 2002• RISC + x86: ??%/year 2002 to 2006

From Hennessy and Patterson, Computer Architecture: A Quantitative Approach, 4th edition, October, 2006

Page 5: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 5

1) Taking Advantage of Parallelism

• Increasing throughput of server computer via multiple processors or multiple disks

• Detailed HW design– Carry lookahead adders uses parallelism to speed up computing

sums from linear to logarithmic in number of bits per operand

– Multiple memory banks searched in parallel in set-associative caches

• Pipelining: overlap instruction execution to reduce the total time to complete an instruction sequence.

– Not every instruction depends on immediate predecessor executing instructions completely/partially in parallel possible

– Classic 5-stage pipeline: 1) Instruction Fetch (Ifetch), 2) Register Read (Reg), 3) Execute (ALU), 4) Data Memory Access (Dmem), 5) Register Write (Reg)

Page 6: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 6

Pipelined Instruction Execution Is Faster

Instr.

Order

Time (clock cycles)

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 6 Cycle 7Cycle 5

Page 7: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 7

Limits to Pipelining • Hazards prevent next instruction from executing

during its designated clock cycle– Structural hazards: attempt to use the same hardware to do two

different things at once

– Data hazards: Instruction depends on result of prior instruction still in the pipeline

– Control hazards: Caused by delay between the fetching of instructions and decisions about changes in control flow (branches and jumps).

Instr.

Order

Time (clock cycles)

RegA

LU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Page 8: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 8

2) The Principle of Locality => Caches ($)

• The Principle of Locality:– Programs access a relatively small portion of the address space at any

instant of time.

• Two Different Types of Locality:– Temporal Locality (Locality in Time): If an item is referenced, it will tend

to be referenced again soon (e.g., loops, reuse)

– Spatial Locality (Locality in Space): If an item is referenced, items whose addresses are close by tend to be referenced soon (e.g., straight-line code, array access)

• For 30 years, HW has relied on locality for memory perf.

P MEM$

Page 9: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 9

Levels of the Memory Hierarchy

CPU Registers100s Bytes300 – 500 ps (0.3-0.5 ns)

L1 and L2 Cache10s-100s K Bytes~1 ns - ~10 ns$1000s/ GByte

Main MemoryG Bytes80ns- 200ns~ $100/ GByte

Disk10s T Bytes, 10 ms (10,000,000 ns)~ $0.25 / GByte

CapacityAccess TimeCost

Tape VaultSemi-infinitesec-min~$1 / GByte

Registers

L1 Cache

Memory

Disk

Tape

Instr. Operands

Blocks

Pages

Files

StagingXfer Unit

prog./compiler1-8 bytes

cache cntlr32-64 bytes

OS4K-8K bytes

user/operatorMbytes

Upper Level

Lower Level

Faster

Larger

L2 Cachecache cntlr64-128 bytesBlocks

Page 10: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 10

3) Focus on the Common Case“Make Frequent Case Fast and Rest Right”

• Common sense guides computer design– Since its engineering, common sense is valuable

• In making a design trade-off, favor the frequent case over the infrequent case

– E.g., Instruction fetch and decode unit used more frequently than multiplier, so optimize it first

– E.g., If database server has 50 disks / processor, storage dependability dominates system dependability, so optimize it 1st

• Frequent case is often simpler and can be done faster than the infrequent case

– E.g., overflow is rare when adding 2 numbers, so improve performance by optimizing more common case of no overflow

– May slow down overflow, but overall performance improved by optimizing for the normal case

• What is frequent case and how much performance improved by making case faster => Amdahl’s Law

Page 11: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 11

4) Amdahl’s Law - Partial Enhancement Limits

( )enhanced

enhancedenhanced

new

oldoverall

Speedup

Fraction Fraction

1

ExTimeExTime

Speedup+−

==1

Best to ever achieve:

Speedupmax = 1

1 - Fractionenhanced( )

( ) ⎥⎦

⎤⎢⎣

⎡+−×=

enhanced

enhancedenhancedoldnew Speedup

FractionFraction ExTime ExTime 1

Example: An I/O bound server gets a new CPU that is 10X faster, but 60% of server time is spent waiting for I/O.

Speedupoverall = 1

1− 0.4( ) + 0.410

= 1

0.64= 1.56

A 10X faster CPU allures, but the server is only 1.6X faster.

Page 12: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 12

5) Processor performance equation

CPU time = Seconds = Instructions x Cycles x Seconds

Program Program Instruction Cycle

CPU time = Seconds = Instructions x Cycles x Seconds

Program Program Instruction CycleCPU time = Inst Count x CPI x Clock Cycle

Program X

Compiler X (X)

Inst. Set. X X

Organization X X

Technology X

Inst count

CPI

Cycle time

Page 13: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 13

What Determines a Clock Cycle?

• At transition edge(s) of each clock pulse, state devices sample and save their present input signals

• Past: 1 cycle = time for signals to pass 10 levels of gates• Today: determined by numerous time-of-flight issues +

gate delays– clock propagation, wire lengths, drivers

Latchor

register

combinationallogic

Page 14: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 14

• Performance Milestones• Processor: ‘286, ‘386, ‘486,

Pentium, Pentium Pro, Pentium 4 (21x,2250x)

• Ethernet: 10Mb, 100Mb, 1000Mb, 10000 Mb/s (16x,1000x)

• Memory Module: 16bit plain DRAM, Page Mode DRAM, 32b, 64b, SDRAM, DDR SDRAM (4x,120x)

• Disk : 3600, 5400, 7200, 10000, 15000 RPM (8x, 143x)

1

10

100

1000

10000

1 10 100

Relative Latency Improvement

Relative BW

Improvement

Processor

Memory

Network

Disk

(Latency improvement = Bandwidth improvement)

CPU high, Memory low(“Memory Wall”)

(Latency = simple operation w/o contention, BW = best-case)

Δ.Latency Lags Δ.Bandwidth (for last 20 yrs)

Page 15: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 15

Summary of Technology Trends

• For disk, LAN, memory, and microprocessor, bandwidth improves by more than the square of latency improvement

– In the time that bandwidth doubles, latency improves by no more than 1.2X to 1.4X

• Lag of gains for latency vs bandwidth probably even larger in real systems, as bandwidth gains multiplied by replicated components

– Multiple processors in a cluster or even on a chip– Multiple disks in a disk array– Multiple memory modules in a large memory – Simultaneous communication in switched local area networks (LANs)

• HW and SW developers should innovate assuming Latency Lags Bandwidth

– If everything improves at the same rate, then nothing really changes – When rates vary, good designs require real innovation

Page 16: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 16

Define and quantify power ( 1 / 2)• For CMOS chips, traditional dominant energy use has

been in switching transistors, called dynamic powerwitchedFrequencySVoltageLoadCapacitivePowerdynamic ×××=

22/1• For mobile devices, energy is a better metric

VoltageLoadCapacitiveEnergydynamic2

×=

• For a fixed task, slowing clock rate (the switching frequency) reduces power, but not energy

• Capacitive load is function of number of transistors connected to output and the technology, which determines the capacitance of wires and transistors

• Dropping voltage helps both, so ICs went from 5V to 1V• To save energy & dynamic power, most CPUs now turn

off clock of inactive modules (e.g. Fltg. Pt. Arith. Unit)• If a 15% voltage reduction causes a 15% reduction in

frequency, what is the impact on dynamic power?New power/old = 0.852 x 0.85 = 0.853 = 0.614 “39% reduction”• Because leakage current flows even when a transistor is

off, now static power important tooVoltageCurrentPower staticstatic ×=

Page 17: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 17

Define and quantity dependability (2/3)• Module reliability = measure of continuous service

accomplishment (or time to failure).1. Mean Time To Failure (MTTF) measures Reliability2. Failures In Time (FIT) = 1/MTTF, the failure rate

• Usually reported as failures per billion hours of operation

Definition: Performance• Performance is in units of things-done per second

1. bigger is better1. If we are primarily concerned with response time

1. " X is N times faster than Y" means

performance(x) = 1 execution_time(x)

N = Performance(X) = Execution_time(Y)

Performance(Y) Execution_time(X) The Speedup = N “mushroom”: The BIG Time the little time

Page 18: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/1,3,8/2009 CSE502-F09, Lec 01-3 - intro 18

And in conclusion …• Computer Science at the crossroads from sequential to

parallel computing– Salvation requires innovation in many fields, including computer architecture

• An architect must track & extrapolate technology

• Bandwidth in disks, DRAM, networks, and processors improves by at least as much as the square of the improvement in Latency

• Quantify dynamic and static power– Capacitance x Voltage2 x frequency, Energy vs. power

• Quantify dependability– Reliability (MTTF, FIT), Availability (99.9…)

• Quantify and summarize performance– Ratios, Geometric Mean, Multiplicative Standard Deviation

• Read Chapter 1, then Appendix A

Page 19: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

CSE 502 Graduate Computer

Architecture

Lec 3-5 – Performance

+ Instruction Pipelining Review

Larry WittieComputer Science, StonyBrook University

http://www.cs.sunysb.edu/~cse502 and ~lw

Slides adapted from David Patterson, UC-Berkeley cs252-s06

9/8,10,15/2009 19CSE502-F09, Lec 03+4+5-perf & pipes

Page 20: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 20

A "Typical" RISC ISA

• 32-bit fixed format instruction (3 formats)

• 32 32-bit GPR (R0 contains zero, DP take pair)

• 3-address, reg-reg arithmetic instruction

• Single address mode for load/store: base + displacement

– no indirection (since it needs another memory access)

• Simple branch conditions (e.g., single-bit: 0 or not?)

• (Delayed branch - ineffective in deep pipelines)

see: SPARC, MIPS, HP PA-Risc, DEC Alpha, IBM PowerPC, CDC 6600, CDC 7600, Cray-1, Cray-2, Cray-3

Page 21: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 21

Example: MIPS

Op

31 26 01516202125

Rs1 Rd immediate

Op

31 26 025

Op

31 26 01516202125

Rs1 Rs2

target

Rd Opx

Register-Register – R Format – Arithmetic operations 561011

Register-Immediate – I Format – All immediate arithmetic ops

Op

31 26 01516202125

Rs1 Rs2/Opx immediate

Branch – I Format – Moderate relative distance conditional branches

Jump / Call – J Format – Long distance jumps

Page 22: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 22

5-Stage MIPS Datapath(has pipeline latches) Figure A.3, Page A-9

MemoryAccess

Write

Back

InstructionFetch

Instr. DecodeReg. Fetch

ExecuteAddr. Calc

ALU

Mem

ory

Reg File

MU

XM

UX

Data

Mem

ory

MU

X

SignExtend

Zero?

IF/ID

ID/E

X

MEM

/WB

EX

/MEM

4

Ad

der

Next SEQ PC Next SEQ PC

RD RD RD WB

Data

• Data stationary control– local decode for each instruction phase / pipeline stage

Next PC

Addre

ss

RS1

RS2

Imm

MU

X

Page 23: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 23

Code SpeedUp Equation for Pipelining

pipelined

dunpipeline

TimeCycle

TimeCycle

CPI stall Pipeline CPI Idealdepth Pipeline CPI Ideal

Speedup ×+×

=

pipelined

dunpipeline

TimeCycle

TimeCycle

CPI stall Pipeline 1depth Pipeline

Speedup ×+

=

Instper cycles Stall Average CPI Ideal CPIpipelined +=

For simple RISC pipeline, Ideal CPI = 1:

Page 24: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 24

Instr.

Order

add r1,r2,r3

sub r4,r1,r3

and r6,r1,r7

or r8,r1,r9

xor r10,r1,r11

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Data Hazard on Register R1 (If No Forwarding)Figure A.6, Page A-17

Time (clock cycles)

IF ID/RF EX MEM WB No forwarding needed since write reg in 1st half cycle, read reg in 2nd half cycle.

Page 25: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 25

• Read After Write (RAW) InstrJ tries to read operand before InstrI writes it

• Caused by a “Dependence” (in compiler nomenclature). This hazard results from an actual need for communicating a new data value.

Three Generic Data Hazards

I: add r1,r2,r3J: sub r4,r1,r3

Page 26: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 26

• Write After Read (WAR) InstrJ writes operand before InstrI reads it

• Called an “anti-dependence” by compiler writers.This results from reuse of the name “r1”.

• Cannot happen in MIPS 5 stage pipeline because:

– All instructions take 5 stages, and

– Register reads are always in stage 2, and

– Register writes are always in stage 5

I: sub r4,r1,r3 J: add r1,r2,r3K: mul r6,r1,r7

Three Generic Data Hazards

Page 27: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 27

Three Generic Data Hazards

• Write After Write (WAW) InstrJ writes operand before InstrI writes it.

• Called an “output dependence” by compiler writersThis also results from the reuse of name “r1”.

• Cannot happen in MIPS 5 stage pipeline because:

– All instructions take 5 stages, and

– Register writes are always in stage 5

• Will see WAR and WAW in more complicated pipes

I: sub r1,r4,r3 J: add r1,r2,r3K: mul r6,r1,r7

Page 28: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 28

Time (clock cycles)

Forwarding to Avoid Data HazardFigure A.7, Page A-19

Inst

r.

Order

add r1,r2,r3

sub r4,r1,r3

and r6,r1,r7

or r8,r1,r9

xor r10,r1,r11

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Need no forwarding since write reg is in 1st half cycle, read reg in 2nd half cycle.

Forwarding of ALU outputs needed as ALU inputs 1 & 2 cycles later.

Forwarding of LW MEM outputs to SW MEM or ALU inputs 1 or 2 cycles later.

Page 29: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 29

HW Datapath Changes (in red) for ForwardingFigure A.23, Page A-37

MEM

/WR

ID/E

X

EX

/MEM

DataMemory

ALU

mux

mux

Registe

rs

NextPC

Immediate

mux

What circuit detects and resolves this hazard?

(From ALU)

To forward ALU output 1 cycle to ALU inputs

To forward ALU, MEM 2 cycles to ALU

(From LW Data Memory)

mux

To forward MEM 1 cycle to SW MEM input

Page 30: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 30

Time (clock cycles)

Forwarding Avoids ALU-ALU & LW-SW Data HazardsFigure A.8, Page A-20

Inst

r.

Order

add r1,r2,r3

lw r4, 0(r1)

sw r4,12(r1)

or r8,r6,r9

xor r10,r9,r11

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Page 31: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 31

Time (clock cycles)

Instr.

Order

lw r1, 0(r2)

sub r4,r1,r6

and r6,r1,r7

or r8,r1,r9

LW-ALU Data Hazard Even with Forwarding Figure A.9, Page A-21

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

No forwarding needed since write reg in 1st half cycle, read reg in 2nd half cycle.

Page 32: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 32

Data Hazard Even with Forwarding(Similar to Figure A.10, Page A-21)

Time (clock cycles)

or r8,r1,r9

Instr.

Order

lw r1, 0(r2)

sub r4,r1,r6

and r6,r1,r7

Reg ALU

DMemIfetch Reg

RegIfetch

ALU

DMem RegBubble

Ifetch

ALU

DMem RegBubble Reg

Ifetch

ALU

DMemBubble Reg

How is this hazard detected?

No forwarding needed since write reg in 1st half cycle, read reg in 2nd half cycle.

Page 33: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 33

Try producing fast code with no stalls for

a = b + c;

d = e – f;

assuming a, b, c, d ,e, and f are in memory.

Slow code:

LW Rb,b

LW Rc,c

ADD Ra,Rb,Rc

SW a,Ra

LW Re,e

LW Rf,f

SUB Rd,Re,Rf

SW d,Rd

Software Scheduling to Avoid Load Hazards

Fast code (no stalls):

LW Rb,b

LW Rc,c

LW Re,e

ADD Ra,Rb,Rc

LW Rf,f

SW a,Ra

SUB Rd,Re,Rf

SW d,Rd

Compiler optimizes for performance. Hardware checks for safety.

Stall ===>

Stall ===>

Page 34: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 34

5-Stage MIPS Datapath(has pipeline latches) Figure A.3, Page A-9

MemoryAccess

Write

Back

InstructionFetch

Instr. DecodeReg. Fetch

ExecuteAddr. Calc

ALU

Mem

ory

Reg File

MU

XM

UX

Data

Mem

ory

MU

X

SignExtend

Zero?

IF/ID

ID/E

X

MEM

/WB

EX

/MEM

4

Ad

der

Next SEQ PC Next SEQ PC

RD RD RD WB

Data

• Simple design put branch completion in stage 4 (Mem)

Next PC

Addre

ss

RS1

RS2

Imm

MU

X

Page 35: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 35

Control Hazard on Branch - Three Cycle Stall

10: beq r1,r3,36

14: and r2,r3,r5

18: or r6,r1,r7

22: add r8,r1,r9

36: xor r10,r1,r11

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

Reg

ALU

DMemIfetch Reg

Reg ALU

DMemIfetch Reg

What do you do with the 3 instructions in between?

How do you do it?

Where is the “commit”?

MEMID/RF

Page 36: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 36

Branch Stall Impact if Commit in Stage 4

• If CPI = 1 and 15% of instructions are branches, Stall 3 cycles => new CPI = 1.45!

• Two-part solution:– Determine sooner whether branch taken or not, AND

– Compute taken branch address earlier

• MIPS branch tests if register = 0 or 0

• MIPS Solution:– Move zero_test to ID/RF (Instr Decode & Register Fetch) stage (2, 4=MEM)

– Add extra adder to calculate new PC (Program Counter) in ID/RF stage

– Result is 1 clock cycle penalty for branch versus 3 when decided in MEM

Page 37: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 37

Ad

der

IF/ID

Pipelined MIPS DatapathFigure A.24, page A-38

MemoryAccess

Write

Back

InstructionFetch

Instr. DecodeReg. Fetch

ExecuteAddr. Calc

ALU

Mem

ory

Reg File

MU

X

Data

Mem

ory

MU

X

SignExtend

Zero?

MEM

/WB

EX

/MEM

4

Ad

der

Next SEQ PC

RD RD RD WB

Data

• Interplay of instruction set design and cycle time.

Next PC

Addre

ss

RS1

RS2

ImmM

UX

ID/E

X

The fast_branch design needs a longer stage 2 cycle time, so the clock is slower for all stages.

Page 38: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 38

Four Branch Hazard Alternatives

#1: Stall until branch direction is clear

#2: Predict Branch Not Taken– Execute the next instructions in sequence

– PC+4 already calculated, so use it to get next instruction

– Nullify bad instructions in pipeline if branch is actually taken

– Nullify easier since pipeline state updates are late (MEM, WB)

– 47% MIPS branches not taken on average

#3: Predict Branch Taken– 53% MIPS branches taken on average

– But have not calculated branch target address in MIPS

» MIPS still incurs 1 cycle branch penalty

» Other machines: branch target known before outcome

Page 39: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 39

Four Branch Hazard Alternatives

#4: Delayed Branch– Define branch to take place AFTER a following instruction

branch instructionsequential successor1

sequential successor2

........sequential successorn

branch target if taken

– 1 slot delay allows proper decision and branch target address in 5 stage pipeline

– MIPS 1st used this (Later versions of MIPS did not; pipeline deeper)

Branch delay of length n

Page 40: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/8,10,15/2009 CSE502-F09, Lec 03+4+5-perf & pipes 40

And In Conclusion: Control and Pipelining• Quantify and summarize performance

– Ratios, Geometric Mean, Multiplicative Standard Deviation

• F&P: Benchmarks age, disks fail, single-point failure• Control via State Machines and Microprogramming• Just overlap tasks; easy if tasks are independent• Speed Up Pipeline Depth; if ideal CPI is 1, then:

• Hazards limit performance on computers:– Structural: need more HW resources– Data (RAW,WAR,WAW): need forwarding, compiler scheduling– Control: delayed branch or branch (taken/not-taken) prediction

• Exceptions and interrupts add complexity• Next time: Read Appendix C• No class Tuesday 9/29/09, when Monday classes will run.

pipelined

dunpipeline

TimeCycle

TimeCycle

CPI stall Pipeline 1depth Pipeline

Speedup ×+

=

Page 41: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

CSE 502 Graduate Computer

Architecture

Lec 6-7 – Memory Hierarchy Review

Larry WittieComputer Science, StonyBrook University

http://www.cs.sunysb.edu/~cse502 and ~lw

Slides adapted from David Patterson, UC-Berkeley cs252-s06

Page 42: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 42

Since 1980, CPU has outpaced DRAM ...

CPU60% per yr2X in 1.5 yrs

DRAM9% per yr2X in 10 yrs

10

DRAM

CPU

Performance(1/latency)

100

1000

1980

2000

1990

Year

Gap grew 50% per year

Q. How do architects address this gap? A. Put smaller, faster “cache”

memories between CPU and DRAM.

Create a “memory hierarchy”.

Page 43: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 43

Apple || (1977)Latencies

Steve WozniakSteve

Jobs

CPU: 1000 ns DRAM: 400 ns

1977: DRAM faster than microprocessors

Page 44: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 44

Memory Hierarchy: Apple iMac G5

iMac G51.6 GHz

1600 (mem: 7.3) x Apple II

07 Reg L1 Inst L1 Data L2 DRAM Disk

Size 1K 64K 32K 512K 256M 80G

Latency

Cycles, Time

1,

0.6 ns

3,

1.9 ns

3,

1.9 ns

11,

6.9 ns

88,

55 ns

107,

12 ms

Let programs address a memory space that scales to

the disk size, at a speed that is usually nearly as fast as

register access

Managed by compiler

Managed by hardware

Managed by OS,hardware,

application

Goal: Illusion of large, fast, cheap memory

Page 45: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 45

iMac’s PowerPC 970 (G5): All caches on-chip

Registers

1/2 KB

1/2 KB

512KL2

L1 (64K Instruction)

L1 (32K Data)

Page 46: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 46

The Principle of Locality

• The Principle of Locality:– Program access a relatively small portion of the address space at

any instant of time.

• Two Different Types of Locality:– Temporal Locality (Locality in Time): If an item is referenced, it will

tend to be referenced again soon (e.g., loops, reuse)

– Spatial Locality (Locality in Space): If an item is referenced, items whose addresses are close by tend to be referenced soon (e.g., straightline code, array access)

• For last 15 years, HW has relied on locality for speed

Locality is a property of programs which is exploited in machine design.

Page 47: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 47

Programs with locality cache well ...

Donald J. Hatfield, Jeanette Gerald: Program Restructuring for Virtual Memory. IBM Systems Journal 10(3): 168-192 (1971)

Time=>

Mem

ory

Ad

dre

ss (

on

e d

ot

per

acc

ess)

SpatialLocality

Temporal Locality

Bad locality behavior

Page 48: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 48

Memory Hierarchy: Terminology

• Hit: data appears in some block in the upper level (example: Block X)

– Hit Rate: the fraction of memory accesses found in the upper level– Hit Time: Time to access the upper level which consists of

RAM access time + Time to determine hit/miss

• Miss: data needs to be retrieved from a block in the lower level (Block Y)

– Miss Rate = 1 - (Hit Rate)– Miss Penalty: Time to replace a block in the upper level +

Time to deliver the block to the upper level

• Hit Time << Miss Penalty(=500 instructions on 21264!)Lower Level

MemoryUpper LevelMemory

To Processor

From ProcessorBlk X

Blk Y

Page 49: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 49

Cache Measures

• Hit rate: fraction found in that level– So high that usually talk about Miss rate– Miss rate fallacy: as MIPS to CPU performance,

miss rate to average memory access time in memory

• Average memory-access time = Hit time + Miss rate x Miss penalty

(ns or clocks)• Miss penalty: time to replace a block from

lower level, including time to replace in CPU– {replacement time: time to make upper-level room for block} – access time: time to lower level

= f(latency to lower level)– transfer time: time to transfer block

=f(BW between upper & lower levels)

Page 50: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 50

4 Questions for Memory Hierarchy

• Q1: Where can a block be placed in the upper level? (Block placement)

• Q2: How is a block found if it is in the upper level? (Block identification)

• Q3: Which block should be replaced on a miss? (Block replacement)

• Q4: What happens on a write? (Write strategy)

Page 51: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 51

Q1: Where can a block be placed in the upper level? • Block 12 placed in 8 block cache:

– Fully associative, direct mapped, 2-way set associative

– S.A. Mapping = Block Number Modulo (Number of Sets)

Cache

01234567 0123456701234567

Memory

111111111122222222223301234567890123456789012345678901

Full MappedDirect Mapped(12 mod 8) = 4

2-Way Assoc(12 mod 4) = 0

Page 52: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 52

Q2: How find block if in upper level = cache? Bits = 18b: tag 8b index: 256 entries/cache 4b: 16 wds/block 2b: 4 Byte/wd

Bits: (One-way) Direct Mapped Cache Data Capacity: 16KB= 256 x 512 / 8

Index => cache entry Location of all possible blocks

Tag for each block:No need to check index, block-offset

Increasing associativity: Shrinks index & expands tag size

Bit Fields in Memory Address Used to Access “Cache” Word

Virtual Memory

Cache Block

Word & Byte OffsetsIn Block

Block (a.k.a. Page) Address

IndexTag

Address (showing bit positions)

DataHit

Data

Tag

V Tag

32

16

=

Index

18 8 Byteoffset

31 14 13 2 1 06 5

4

Block offset

256

entries

512 bits18 bits

Mux

3232 3218

Page 53: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 53

Q3: Which block to replace after a miss? (After start up, cache is nearly always full)

• Easy if Direct Mapped (only 1 block “1 way” per index)• If Set Associative or Fully Associative, must choose:

– Random (“Ran”) Easy to implement, but not best if only 2-way

– LRU (Least Recently Used) LRU is best, but hard to implement if >8-way

Also other LRU approximations better than Random

Miss Rates for 3 Cache Sizes & AssociativitiesAssociativity 2-way 4-way 8-way

DataSize LRU Ran LRU Ran LRU Ran

16 KB 5.2% 5.7% 4.7% 5.3% 4.4% 5.0%

64 KB 1.9% 2.0% 1.5% 1.7% 1.4% 1.5%

256 KB 1.15% 1.17% 1.13% 1.13% 1.12% 1.12%

Random picks => same low miss rate as LRU for large caches

Page 54: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 54

Q4: Write policy: What happens on a write?

Write-Through Write-Back

Policy

Data written to cache block

is also written to next lower-level memory

Write new data only to the cache

Update lower level just before a written block leaves cache,

i.e., is removed

Debugging Easier Harder

Can read misses force writes? No

Yes (may slow read)

Do repeated writes touch lower level? Yes, memory busier No

Additional option -- let writes to an un-cached address allocate a new cache line (“write-

allocate”), else just Write-Through.

Page 55: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 55

Write Buffers for Write-Through Caches

Q. Why a write buffer ?

ProcessorCache

Write Buffer

Lower Level

Memory

Write buffer holds (addresses&) data awaiting write-through to lower levels

A. So CPU not stall for writes

Q. Why a buffer, why not just one register ?

A. Bursts of writes arecommon.Q. Are Read After

Write (RAW) hazards an issue for write buffer?

A. Yes! Drain buffer before next read or check buffer addresses before read-miss.

Page 56: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 56

5 Basic Cache Optimizations

• Reducing Miss Rate1. Larger Block size (reduce Compulsory, “cold”, misses)2. Larger Cache size (reduce Capacity misses)3. Higher Associativity (reduce Conflict misses) (… and

multiprocessors have cache Coherence misses) (4 Cs)

• Reducing Miss Penalty

• Multilevel Caches {total miss rate = π(local miss rate)}

1. Reducing Hit Time (minimal cache latency)2. Giving Reads Priority over Writes, since CPU waiting Read completes before earlier writes in write buffer

Page 57: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 57

The Limits of Physical Addressing

CPU Memory

A0-A31 A0-A31

D0-D31 D0-D31

Programs use “Physical addresses” of memory locations

Data

All programs shared one address space: The physical address space

No way to prevent a program from accessing any machine resource

Machine language programs had to beaware of the machine organization

Simple addressing method of archaic pre-1978 computers

Page 58: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 58

Solution: Add a Layer of Indirection

All user programs run in an standardizedvirtual address space starting at zero

Needs fast(!) Address Translation hardware, managed by the operating system (OS),

maps virtual address to physical memory

Hardware supports “modern” OS features:Memory protection, Address translation, Sharing

“Physical Addresses”

CPU Main Memory

A0-A31 A0-A31

D0-D31 D0-D31

AddressTranslation

Virtual Physical

“Virtual Addresses”

Data

Page 59: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 59

Three Advantages of Virtual Memory

• Translation: – Program can be given consistent view of memory, even though physical

memory is scrambled (pages of programs in any order in physical RAM)– Makes multithreading reasonable (now used a lot!)– Only the most important part of each program (“the Working Set”) must

be in physical memory at any one time.– Contiguous structures (like stacks) use only as much physical memory

as necessary, yet still can grow later as needed without recopying.

• Protection (most important now):– Different threads (or processes) protected from each other.– Different pages can be given special behavior

» (Read Only, Invisible to user programs, Not cached).– Kernel and OS data are protected from access by User programs– Very important for protection from malicious programs

• Sharing:– Can map same physical page to multiple users

(“Shared memory”)

Page 60: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 60

PhysicalMemory Space

• Page table maps virtual page numbers to physical frames (“PTE” = Page Table Entry)

• Virtual memory => treats memory cache for disk

Details of Page Table

Virtual Address (for 4,096 Bytes/page)

Page Table

indexintopagetable

Page Table PtrIn Base Reg

V AccessRights PA

V page no. offset12

table locatedin physicalmemory

(V is valid bit)

Ph page no. offset12

Physical Address

frame

frame

frame

frame

virtual address

Page Table

page

page

page

page

(Byte offset same in VA & PA)

0… … …

Page 61: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 61

All page tables may not fit in memory!

A table for 4KB pages for a 32-bit physical address space (max 4GB) has 1M entries

Each process needs its own address space tables!

P1 index P2 index Page Offset

31 12 11 02122

32 bit virtual address

Two-level Page Tables

Top-level table wired (stays) in main memory

Only a subset of the 1024 second-level tables are in main memory; rest are on disk or unallocated

Page 62: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 62

MIPS Address Translation: How does it work?

“Physical Addresses”

CPU Memory

A0-A31 A0-A31

D0-D31 D0-D31

Data

TLB also containsprotection bits for virtual address

Virtual Physical

“Virtual Addresses”

TranslationLook-Aside

Buffer(TLB)

Translation Look-Aside Buffer (TLB)A small fully-associative cache of

mappings from virtual to physical addresses

Fast common case: If virtual address is in TLB,

process has permission to read/write it.

What is the table

ofmappings that it caches?

Recentlyused VAPAentries.

Page 63: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 63

Can TLB translation overlap cache indexing?

Data out

Virtual Page Number Page Offset

TranslationLook-Aside

Buffer(TLB)

Virtual

Physical

=

Hit

Cache Tag

Having cache index in page offset works, but

Q. What is the downside?A. Inflexibility. Size of cache limited by page size.

ValidCache Tags Cache DataTag Part of Physical Addr = Physical Page Number

Index Byte Select

Cache Block

Cache Block

Page 64: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 64

Problems With Overlapped TLB AccessOverlapped access only works so long as the address bits used to index into the cache do not change as the result of VA translation

This usually limits overlapping to small caches, large page sizes, or high n-way set associative caches if you want a large capacity cache

Example: suppose everything the same except that the cache is increased to 8 KBytes instead of 4 KB:

11 2

00

virt page # disp20 12

cache index

This bit is changedby VA translation, butit is needed for cachelookup.

Solutions: go to 8KByte page sizes; go to 2-way set associative cache; or SW guarantee VA[13]=PA[13]

1K

4 410

2-way set assoc cache

Page 65: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 65

Can CPU use virtual addresses for cache?

“Physical Addresses”

CPU Main Memory

A0-A31 A0-A31

D0-D31 D0-D31

Only use TLB on a cache miss !

TranslationLook-Aside

Buffer(TLB)

Virtual Physical

“Virtual Addresses”

A. Synonym problem. If two address spaces share a physical frame, data may

be in cache twice. Maintaining consistency is a nightmare.

CacheVirtual

D0-D31

Downside: a subtle, fatal problem. What is it? (Aliasing)

Page 66: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 66

Summary #1/3: The Cache Design Space

• Several interacting dimensions– cache size

– block size

– associativity

– replacement policy

– write-through vs write-back

– write allocation

• The optimal choice is a compromise– depends on access characteristics

» workload

» use (I-cache, D-cache, TLB)

– depends on technology / cost

• Simplicity often wins

Associativity

Cache Size

Block Size

Bad

Good

Less More

Factor A Factor B

Page 67: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 67

Summary #2/3: Caches

• The Principle of Locality:– Program access a relatively small portion of the address space at any

instant of time.

» Temporal Locality: Locality in Time

» Spatial Locality: Locality in Space

• Three Major Uniprocessor Categories of Cache Misses:– Compulsory Misses: sad facts of life. Example: cold start misses.

– Capacity Misses: increase cache size

– Conflict Misses: increase cache size and/or associativity.Nightmare Scenario: ping pong effect!

• Write Policy: Write Through vs. Write Back

• Today CPU time is a function of (ops, cache misses) vs. just f(ops): Increasing performance affects Compilers, Data structures, and Algorithms

Page 68: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

9/17-22/2009 CSE502-F09, Lec 06+7-cache VM TLB 68

Summary #3/3: TLB, Virtual Memory• Page tables map virtual address to physical address

• TLBs are important for fast translation

• TLB misses are significant in processor performance– funny times, as most systems cannot access all of 2nd level cache without TLB

misses!

• Caches, TLBs, Virtual Memory all understood by examining how they deal with 4 questions: 1) Where can block be placed?2) How is block found? 3) What block is replaced on miss? 4) How are writes handled?

• Today VM allows many processes to share single memory without having to swap all processes to disk; today VM protection is more important than memory hierarchy benefits, but computers are still insecure

• Short in-class openbook quiz on appendices A-C & Chapter 1 near start of next (9/24) class. Bring a calculator.

(Please put your best email address on your exam.)

Page 69: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

CSE 502 Graduate Computer

Architecture

Lec 8-10 – Instruction Level Parallelism

Larry WittieComputer Science, StonyBrook University

http://www.cs.sunysb.edu/~cse502 and ~lw

Slides adapted from David Patterson, UC-Berkeley cs252-s06

Page 70: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 70

Recall from Pipelining Review

• Pipeline CPI = Ideal pipeline CPI + Structural Stalls + Data Hazard Stalls + Control Stalls

– Ideal pipeline CPI: measure of the maximum performance attainable by the implementation

– Structural hazards: HW cannot support this combination of instructions

– Data hazards: Instruction depends on result of prior instruction still in the pipeline

– Control hazards: Caused by delay between the fetching of instructions and decisions about changes in control flow (branches and jumps)

Page 71: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 71

Instruction Level Parallelism

• Instruction-Level Parallelism (ILP): overlap the execution of instructions to improve performance

• 2 approaches to exploit ILP:1) Rely on hardware to help discover and exploit the parallelism

dynamically (e.g., Pentium 4, AMD Opteron, IBM Power) , and

2) Rely on software technology to find parallelism, statically at compile-time (e.g., Itanium 2)

• Next 3 lectures on this topic

Page 72: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 72

Instruction-Level Parallelism (ILP)• Basic Block (BB) ILP is quite small

– BB: a straight-line code sequence with no branches in except to the entry and no branches out except at the exit

– average dynamic branch frequency 15% to 25% => 4 to 7 instructions execute between a pair of branches

– other problem: instructions in a BB are likely to depend on each other

• To obtain substantial performance enhancements, we must exploit ILP across multiple basic blocks

• Simplest: loop-level parallelism to exploit parallelism among iterations of a loop. E.g.,

for (j=0; j<1000; j=j+1)        x[j+1] = x[j+1] + y[j+1];

for (i=0; i<1000; i=i+4){        x[I+1] = x[I+1] + y[I+1]; x[I+2] = x[I+2] + y[I+2];

x[I+3] = x[I+3] + y[I+3]; x[I+4] = x[I+4] + y[I+4];

} //{Vector HW can make this run much faster.}

Page 73: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 73

Loop-Level Parallelism• Exploit loop-level parallelism to find run-time

parallelism by “unrolling loops” either via1. dynamic branch prediction by CPU hardware or 2. static loop unrolling by a compiler

(Other ways: vectors & parallelism - covered later)• Determining instruction dependence is critical to

Loop Level Parallelism• If two instructions are

– “parallel”, they can execute simultaneously in a pipeline of arbitrary depth without causing any stalls (assuming no structural hazards)

– dependent, they are not parallel and must be executed in order, although they may often be partially overlapped

Page 74: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 74

ILP and Data Dependencies, Hazards• HW/SW must preserve program order: give the same

results as if instructions were executed sequentially in the original order of the source program

– Dependences are a property of programs

• The presence of a dependence indicates the potential for a hazard, but the existence of an actual hazard and the length of any stall are properties of the pipeline

• Importance of the data dependencies1) Indicate the possibility of a hazard2) Determine the order in which results must be calculated3) Set upper bounds on how much parallelism can possibly be exploited

• HW/SW goal: exploit parallelism by preserving program order only where it affects the outcome of the program

Page 75: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 75

• Name dependence: when two instructions use the same register or memory location, called a name, but no data flow between the instructions using that name; there are 2 versions of name dependence, which may cause WAR and WAW hazards, if a name such as “r1” is reused:

• 1. InstrJ may wrongly write operand r1 before InstrI reads it

This “anti-dependence” of compiler writers may cause a Write After Read (WAR) hazard in a pipeline.

• 2. InstrJ may wrongly write operand r1 before InstrI writes it

• This “output dependence” of compiler writers may cause a Write After Write (WAW) hazard in a pipeline.

• Instructions with a name dependence can execute simultaneously if one name is changed by a compiler or by register-renaming in HW.

Name Dependence #1: Anti-dependence

I: sub r4,r1,r3 J: add r1,r2,r5

I: sub r1,r4,r3 J: add r1,r2,r3

Page 76: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 76

Carefully Violate Control Dependencies

• Every instruction is control dependent on some set of branches, and, in general, these control dependencies must be preserved to preserve program orderif p1 {S1;

};if p2 {S2;

}• S1 is control dependent on proposition p1, and S2 is

control dependent on p2 but not on p1.• Control dependence need not always be preserved

– Control dependences can be violated by executing instructions that should not have been, if doing so does not affect program results

• Instead, two properties critical to program correctness are – exception behavior and – data flow

Page 77: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 77

Exception Behavior Is Important

• Preserving exception behavior any changes in instruction execution order must not change how exceptions are raised in program ( no new exceptions)

• Example:DADDU R2,R3,R4BEQZ R2,L1LW R1,-1(R2)

L1:– (Assume branches are not delayed)

• What is the problem with moving LW before BEQZ?{Array overflow: what if R2=0, so -1+[R2] is out of program memory bounds?}

Page 78: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 78

Data Flow Of Values Must Be Preserved

• Data flow: actual flow of data values from instructions that produce results to those that consume them

– branches make flow “dynamic” (since we know details only at runtime); must determine which instruction is supplier of data

• Example:

DADDU R1,R2,R3BEQZ R4,LDSUBU R1,R5,R6L: …OR R7,R1,R8

• OR input R1 depends on which of DADDU or DSUBU? Must preserve data flow on execution

Page 79: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 79

FP Loop: Where are the Hazards?

Loop: L.D F0,0(R1);F0=vector element x[i] ADD.D F4,F0,F2;add scalar from F2 = s S.D 0(R1),F4;store result back into x[i] DADDUI R1,R1,-8;decrement pointer 8B (DblWd) BNEZ R1,Loop ;branch R1!=zero

for (i=1000; i>0; i=i–1)x[i] = x[i] + s;

• First translate into MIPS code: -To simplify loop end, assume 8 is lowest

address, F2=s, and R1 starts with address for x[1000]

Page 80: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 80

FP Loop Showing Stalls

• Loop every 9 clock cycles. How reorder code to minimize stalls?

1 Loop: L.D F0,0(R1) ;F0=vector element

2 stall

3 ADD.D F4,F0,F2 ;add scalar in F2

4 stall

5 stall

6 S.D 0(R1),F4 ;store result

7 DADDUI R1,R1,-8 ;decrement pointer 8B (DW)

8 stall ;assume cannot forward to branch

9 BNEZ R1,Loop ;branch R1!=zeroproduce result use result stalls between

FP ALU op Other FP ALU op 3FP ALU op Store double 2 Load double FP ALU op 1Load double Store double 0Integer op Integer op 0

Page 81: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 81

Revised FP Loop Minimizing Stalls

Loop takes 7 clock cycles, but just 3 for execution (L.D, ADD.D,S.D), 4 for loop overhead; How make faster?

1 Loop: L.D F0,0(R1) 2 DADDUI R1,R1,-8

3 ADD.D F4,F0,F2 4 stall

5 stall

6 S.D 8(R1),F4 ;altered offset 0=>8 when moved DADDUI

7 BNEZ R1,Loop

Swap DADDUI and S.D; change address offset of S.D

Original 9 cycle per loop code 1 Loop: L.D F0,0(R1) ;F0=vector element 2 stall 3 ADD.D F4,F0,F2 ;add scalar in F2 4 stall 5 stall 6 S.D 0(R1),F4 ;store result 7 DADDUI R1,R1,-8 ;decrement pointer

8B 8 stall ;assume cannot forward to

branch 9 BNEZ R1,Loop ;branch R1!

=zero

produce result use result stalls between

FP ALU op Other FP ALU op 3FP ALU op Store double 2 Load double FP ALU op 1Load double Store double 0Integer op Integer op 0

Page 82: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 82

Unroll Loop Four Times (straightforward way gives 7=>6.75 cycles)

How rewrite loop to minimize

stalls?

1 Loop:L.D F0,0(R1)3 ADD.D F4,F0,F26 S.D 0(R1),F4 ;drop DADDUI & BNEZ7 L.D F6,-8(R1)9 ADD.D F8,F6,F212 S.D -8(R1),F8 ;drop DADDUI & BNEZ13 L.D F10,-16(R1)15 ADD.D F12,F10,F218 S.D -16(R1),F12 ;drop DADDUI & BNEZ19 L.D F14,-24(R1)21 ADD.D F16,F14,F224 S.D -24(R1),F1625 DADDUI R1,R1,#-32 ;alter to 4*827 BNEZ R1,LOOP

Four loops take 27 clock cycles, or 6.75 per iteration (Assumes R1 is a multiple of 4)

1 cycle stall

2 cycles stall

1 cycle stall

Page 83: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 83

Unrolled Loop That Minimizes (=0) Stalls

1 Loop:L.D F0,0(R1)

2 L.D F6,-8(R1)

3 L.D F10,-16(R1)

4 L.D F14,-24(R1)

5 ADD.D F4,F0,F2

6 ADD.D F8,F6,F2

7 ADD.D F12,F10,F2

8 ADD.D F16,F14,F2

9 S.D 0(R1),F4

10 S.D -8(R1),F8

11 S.D -16(R1),F12

12 DADDUI R1,R1,#-32

13 S.D 8(R1),F16 ; 8-32 = -24

14 BNEZ R1,LOOP

Four loops take 14 clock cycles, or 3.5 per loop.

1 Loop: L.D F0,0(R1)3 ADD.D F4,F0,F26 S.D0(R1),F4 7 L.DF6,-8(R1)9 ADD.D F8,F6,F212 S.D-8(R1),F8 13 L.DF10,-16(R1)15 ADD.D F12,F10,F218 S.D-16(R1),F12 19 L.DF14,-24(R1)21 ADD.D F16,F14,F224 S.D-24(R1),F1625 DADDUI R1,R1,#-

32 ;4*827 BNEZ R1,LOOP27 cycles

m means cycle m + 1 stall n means cycle n + 2 stalls

Page 84: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 84

Loop Unrolling Detail - Strip Mining

• Do not usually know upper bound of loop

• Suppose it is n, and we would like to unroll the loop to make k copies of the body

• Instead of a single unrolled loop, we generate a pair of consecutive loops:

– 1st executes (n mod k) times and has a body that is the original loop {called “strip mining” of a loop}

– 2nd is the unrolled body surrounded by an outer loop that iterates (| n/k |) times

• For large values of n, most of the execution time will be spent in the | n/k | unrolled loops

Page 85: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 85

Five Loop Unrolling Decisions

• Requires understanding how one instruction depends on another and how the instructions can be changed or reordered given the dependences:

1. Determine if loop unrolling can be useful by finding that loop iterations are independent (except for loop maintenance code)

2. Use different registers to avoid unnecessary constraints forced by using the same registers for different computations

3. Eliminate the extra test and branch instructions and adjust the loop termination and iteration increment/decrement code

4. Determine that loads and stores in unrolled loop can be interchanged by observing that loads and stores from different iterations are independent • Transformation requires analyzing memory addresses and finding that no

pairs refer to the same address

5. Schedule (reorder) the code, preserving any dependences needed to yield the same result as the original code

Page 86: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 86

Three Limits to Loop Unrolling

1. Decrease in amount of overhead amortized with each extra unrolling• Amdahl’s Law

2. Growth in code size • For larger loops, size is a concern if it increases the

instruction cache miss rate

3. Register pressure: potential shortfall in registers created by aggressive unrolling and scheduling• If not possible to allocate all live values to registers, code

may lose some or all of the advantages of loop unrolling

• “Software pipelining” is an older compiler technique to unroll loops systematically.

• Loop unrolling reduces the impact of branches on pipelines; another way is branch prediction.

Page 87: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 87

_ _Compiler Software-Pipelining of V=S*V Loop

l.s f2,0(r1)mul.s f4,f0,f2s.s f4,0(r1)addi r1,r1,4bne r1,r2,Lpl.s f2,0(r1)mul.s f4,f0,f2s.s f4,0(r1)addi r1,r1,4bne r1,r2,Lpl.s f2,0(r1)mul.s f4,f0,f2s.s f4,0(r1)addi r1,r1,4bne r1,r2,Lp

Bg: addi r1,r1,8 l.s f2,-8(r1) mul.s f4,f0,f2 l.s f2,-4(r1)Lp: s.s f4,-8(r1) mul.s f4,f0,f2 l.s f2,0(r1) addi r1,r1,4 bne r1,r2,LpEn: s.s f4,-4(r1) mul.s f4,f0,f2 s.s f4,0(r1)

Software pipelining structure tolerates the long latencies of FltgPt operations { l.s, mul.s, s.s are single precision (SP) floating-pt. Load, Multiply, Store. } {At start: r1=addr [V(0)], r2=addr[V(last)]+4, f0 = scalar SP fltg multiplier.}Instructions in iteration box are in reverse order, from different iterations.If have separate FltgPt function boxes for L, M, S, can overlap S M L triples.Bg: marks prologue starting iterated code; En: marks epilogue to finish code.

Page 88: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 88

Dynamic (Run-Time) Branch Prediction• Why does prediction work?

– Underlying algorithm has regularities– Data that is being operated on has regularities– Instruction sequences have redundancies that are artifacts of way that

humans/compilers solve problems• Is dynamic branch prediction better than static prediction?

– Seems to be – There are a small number of important branches in programs which

have dynamic behavior

• Performance = ƒ(accuracy, cost of misprediction)

• Branch History Table: Lower bits of PC address index table of 1-bit values

– Says whether or not branch taken last time

– No address check

• Problem: 1-bit BHT will cause two mispredictions per loop, (Average for loops is 9 iterations before exit):

– End of loop case, when it exits instead of looping as before

– First time through loop on next time through code, when it predicts exit instead of looping

Page 89: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 89

• Solution: 2-bit scheme where change prediction only if get misprediction twice

• Red: stop, not taken

• Green: go, taken

• Adds hysteresis to decision making process

Dynamic Branch Prediction With 2 Bits

T

T NT

NT

Predict Taken

Predict Not Taken

Predict Taken

Predict Not TakenT

NTT

NT

Page 90: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 90

18%

5%

12%10%

9%

5%

9% 9%

0%1%

0%2%4%6%8%

10%12%14%16%18%20%

eqntottespresso

gccli

spice doducspice fppppmatrix300

nasa7

Misprediction Rate

Branch History Table (BHT) Accuracy

• Mispredict because either:– Make wrong guess for that branch

– Got branch history of wrong branch when index the table (same low address bits used for index).

• 4096 entry

BH table:

Integer Floating Point

Page 91: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 91

Correlated Branch Prediction

• Idea: record m most recently executed branches as taken or not taken, and use that pattern to select the proper n-bit branch history table

• Global Branch History: m-bit shift register keeping Taken/Not_Taken status of last m branches anywhere.

• In general, (m,n) predictor means use record of last m global branches to select between 2m local branch history tables, each with n-bit counters

– Thus, the old 2-bit BHT is a (0,2) predictor

• Each entry in table has m n-bit predictors.

Page 92: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 92

Correlating Branch Predictors

(2,2) predictor with 16 sets of four 2-bit predictions

– Behavior of most recent 2 branches selects between four predictions for next branch, updating just that prediction

Branch address

2-bits per branch predictor

Prediction

2-bit global branch history

4

Page 93: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 93

0%

Fre

quen

cy o

f M

isp

redi

ctio

ns

0%1%

5%6% 6%

11%

4%

6%5%

1%2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

4,096 entries: 2-bits per entry Unlimited entries: 2-bits/entry 1,024 entries (2,2)

Accuracy of Different Schemes

4096 Entries 2-bit BHT (#=4096)Unlimited Entries 2-bit BHT1024 Entries (2,2) BHT (#=4096)

nasa

7

mat

rix3

00

dodu

cd

spic

e

fppp

p

gcc

expr

esso

eqnt

ott li

tom

catv

Page 94: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 94

Tournament Predictors• Multilevel branch predictor

• Use n-bit saturating counter to choose between predictors

• Usual choice is between global and local predictors

Page 95: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 95

Comparing Predictors (Fig. 2.8)

• Advantage: tournament predictor can select the right predictor for a particular branch

– Particularly crucial for integer benchmarks. – A typical tournament predictor will select the global predictor

almost 40% of the time for the SPEC Integer benchmarks and less than 15% of the time for the SPEC FP benchmarks

6.8% 2-bit

3.7% Corr,

2.6% Tourn.

Page 96: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

• Branch target calculation is costly and stalls the instruction fetch one or more cycles.

• BTB stores branch PCs and target PCs the same way as caches store addresses and data blocks.

• The PC of a branch is sent to the BTB

• When a match is found the corresponding Predicted target PC is returned

• If the branch is predicted to be Taken, instruction fetch continues at the returned predicted PC

Branch Target Buffers (BTB)

Page 97: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

Branch Target Buffers

Page 98: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 98

Dynamic Branch Prediction Summary

• Prediction becoming important part of execution

• Branch History Table: 2 bits for loop accuracy

• Correlation: Recently executed branches correlated with next branch

– Either different branches (GA)

– Or different executions of same branches (PA)

• Tournament predictors take insight to next level, by using multiple predictors

– usually one based on global information and one based on local information, and combining them with a selector

– In 2006, tournament predictors using 30K bits are in processors like the Power5 and Pentium 4

• Branch Target Buffer: include branch address & prediction

Page 99: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 99

Advantages of Dynamic Scheduling• Dynamic scheduling - hardware rearranges the

instruction execution to reduce stalls while maintaining data flow and exception behavior

• It handles cases in which dependences were unknown at compile time

– it allows the processor to tolerate unpredictable delays such as cache misses, by executing other code while waiting for the miss to resolve

• It allows code compiled for one pipeline to run efficiently on a different pipeline

• It simplifies the compiler • Hardware speculation, a technique with

significant performance advantages, builds on dynamic scheduling (next lecture)

Page 100: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 100

HW Schemes: Instruction Parallelism

• Key idea: Allow instructions behind stall to proceedDIVD F0,F2,F4ADDD F10,F0,F8SUBD F12,F8,F14

• Enables out-of-order execution and allows out-of-order completion (e.g., SUBD before slow DIVD)

– In a dynamically scheduled pipeline, all instructions still pass through issue stage in order (in-order issue)

• Will distinguish when an instruction begins execution from when it completes execution; between the two times, the instruction is in execution

• Note: Dynamic scheduling creates WAR and WAW hazards and makes exception handling harder

Page 101: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 101

Dynamic Scheduling Step 1

• Simple pipeline had only one stage to check both structural and data hazards: Instruction Decode (ID), also called Instruction Issue

• Split the ID pipe stage of simple 5-stage pipeline into 2 stages to make a 6-stage pipeline:

• Issue—Decode instructions, check for structural hazards

• Read operands—Wait until no data hazards, then read operands

Page 102: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 102

A Dynamic Algorithm: Tomasulo’s

• For IBM 360/91 (before caches!) Long memory latency

• Goal: High Performance without special compilers

• Small number of floating point registers (4 in 360) prevented interesting compiler scheduling of operations

– This led Tomasulo to try to figure out how effectively to get more registers — renaming in hardware!

• Why Study a 1966 Computer?

• The descendants of this have flourished!– Alpha 21264, Pentium 4, AMD Opteron, Power 5, …

Page 103: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 103

Tomasulo Algorithm• Control & buffers distributed with Function Units (FU)

– FU buffers called “reservation stations”; have pending operands

• Registers in instructions replaced by values or pointers to reservation stations(RSs); called register renaming ;

– Renaming avoids WAR, WAW hazards– More reservation stations than registers, so can do optimizations

compilers cannot do without access to the additional internal registers, the reservation stations.

• Results from RSs as leave each FU sent to waiting RSs, not through registers, but over a Common Data Bus that broadcasts results to all FUs and their waiting RSs

– Avoids RAW hazards by executing an instruction only when its operands are available

• Load and Stores treated as FUs with RSs as well• Integer instructions can go past branches (predict

taken), allowing FP ops beyond basic block in FP queue

Page 104: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 104

Tomasulo Organization

FP addersFP adders

Add1Add2Add3

FP multipliersFP multipliers

Mult1Mult2

From Mem FP Registers

Reservation Stations

Common Data Bus (CDB)

To Mem

FP OpsQueue

Load Buffers

Store Buffers

Load1Load2Load3Load4Load5Load6

Page 105: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 105

Three Stages of Tomasulo Algorithm

1. Issue—get instruction from FP Op Queue If reservation station free (no structural hazard),

control issues instr & sends operands (renames registers).

2. Execute—operate on operands (EX) When both operands ready, start to execute;

if not ready, watch Common Data Bus for result

3. Write result—finish execution (WB) Write on Common Data Bus to all awaiting units;

mark reservation station available

• Normal data bus: data + destination (“go to” bus)• Common data bus: data + source (“come from” bus)

– 64 bits of data + 4 bits of Functional Unit source address– Write if matches expected Functional Unit (produces result)– Does the broadcast

• Example speed: After start EX: 2 clocks for LD; 3 for Fl .pt. +,-; 10 for * ; 40 for /.

Page 106: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 106

Reservation Station Components

Op: Operation to perform in the unit (e.g., + or –)

Vj, Vk: Value of source operands for Op– Each store buffer has a V field, for the result to be stored

Qj, Qk: Reservation stations producing source registers (value to be written)

– Note: Qj,Qk=0 => ready– Store buffers only have Qi for RS producing result

Busy: Indicates reservation station or FU is busy

Register result status—Indicates which functional unit will write each register, if one exists. Blank when no pending instructions that will write that register.

Page 107: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 107

Why Can Tomasulo Overlap Iterations Of Loops?

• Register renaming– Multiple iterations use different physical destinations for registers

(dynamic loop unrolling).

• Reservation stations – Permit instruction issue to advance past integer control flow operations

– Also buffer old values of registers - totally avoiding the WAR stall

• Other perspective: Tomasulo building data flow dependency graph on the fly

Page 108: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 108

Tomasulo’s Scheme: Two Major Advantages

1. Distribution of the hazard detection logic– distributed reservation stations and the CDBus

– If multiple instructions waiting on single result and each instruction has other operand, then instructions can be released simultaneously by broadcast on CDB

– If a centralized register file were used, the units would have to read their results from the registers when register buses are available

2. Elimination of stalls for WAW and WAR hazards

Page 109: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 109

Tomasulo Drawbacks

• Complexity– delays of 360/91, MIPS 10000, Alpha 21264,

IBM PPC 620 (in CA:AQA 2/e, before it was in silicon!)

• Many associative stores (CDB) at high speed• Performance limited by Common Data Bus

– Each CDB must go to multiple functional units high capacitance, high wiring density

– Number of functional units that can complete per cycle limited to one!

» Multiple CDBs more FU logic for parallel assoc stores

• Non-precise interrupts!– We will address this later

Page 110: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 110

And In Conclusion … #1

• Leverage Implicit Parallelism for Performance: Instruction Level Parallelism

• Loop unrolling by compiler to increase ILP

• Branch prediction to increase ILP

• Dynamic HW exploiting ILP– Works when cannot know dependence at compile time

– Can hide L1 cache misses

– Code for one pipelined machine runs well on another

Page 111: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/1,8,13/09 (no class 9/29, QuizAns 10/6) CSE502-F09, Lec 08+9+10-ILP 111

And In Conclusion … #2• Reservations stations: renaming to larger set of registers +

buffering source operands– Prevents registers from being bottlenecks

– Avoids WAR, WAW hazards

– Allows loop unrolling in HW

• Not limited to basic blocks (integer units gets ahead, beyond branches)

• Helps cache misses as well

• Lasting Contributions of Tomasulo– Dynamic scheduling

– Register renaming– Load/store disambiguation - do 2 memory operations access same location?

• 360/91 descendants are Intel Pentium 4, IBM Power 5, AMD Athlon/Opteron, …

Page 112: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

CSE 502 Graduate Computer

Architecture

Lec 11 –Simultaneous Multithreading

Larry WittieComputer Science, StonyBrook University

http://www.cs.sunysb.edu/~cse502 and ~lw

Slides adapted from David Patterson, UC-Berkeley cs252-s06

Page 113: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 113

Performance beyond single thread ILP

• There can be much higher natural parallelism in some applications (e.g., Database or Scientific codes)

• Explicit Thread Level Parallelism or Data Level Parallelism

• Thread: a process with its own instructions and data (or much harder on compiler: carefully selected rarely interacting code segments in the same process)

– thread may be one process that is part of a parallel program of multiple processes, or it may be an independent program

– Each thread has all the state (instructions, data, PC, register state, and so on) necessary to allow it to execute

• Data Level Parallelism: Perform identical operations on data, and have lots of data

Page 114: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 114

Thread Level Parallelism (TLP)

• ILP (last lectures) exploits implicit parallel operations within a loop or straight-line code segment

• TLP is explicitly represented by the use of multiple threads of execution that are inherently parallel

• Goal: Use multiple instruction streams to improve 1. Throughput of computers that run many

programs 2. Execution time of multi-threaded programs

• TLP could be more cost-effective to exploit than ILP

Page 115: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 115

New Approach: Mulithreaded Execution

• Multithreading: multiple threads to share the functional units of one processor via overlapped execution

– processor must duplicate independent state of each thread e.g., a separate copy of register file, a separate PC, and if running as independent programs, a separate page table

– memory shared through the virtual memory mechanisms, which already support multiple processes

– HW for fast thread switch (0.1 to 10 clocks) is much faster than a full process switch (100s to 1000s of clocks) that copies state

• When switch among threads?– Alternate instruction per thread (fine grain)

– When a thread is stalled, perhaps for a cache miss, another thread can be executed (coarse grain)

– In cache-less multiprocessors, at start of each memory access

Page 116: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 116

Fine-Grained Multithreading

• Switches between threads on each instruction, causing the execution of multiples threads to be interleaved

• Usually done in a round-robin fashion, skipping any stalled threads

• CPU must be able to switch threads every clock• Advantage is that it can hide both short and long

stalls, since instructions from other threads executed when one thread stalls

• Disadvantage is it slows down execution of individual threads, since a thread ready to execute without stalls will be delayed by instructions from other threads

• Used on Sun’s Niagara chip (with 8 cores, will see later)

Page 117: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 117

Course-Grained Multithreading

• Switches threads only on costly stalls, such as L2 cache misses

• Advantages – Relieves need to have very fast thread-switching– Does not slow down any thread, since instructions from other

threads issued only when the thread encounters a costly stall

• Disadvantage is that it is hard to overcome throughput losses from shorter stalls, because of pipeline start-up costs

– Since CPU normally issues instructions from just one thread, when a stall occurs, the pipeline must be emptied or frozen

– New thread must fill pipeline before instructions can complete

• Because of this start-up overhead, coarse-grained multithreading is efficient for reducing penalty only of high cost stalls, where pipeline refill << stall time

• Used IBM AS/400 (1988, for small to medium business)

Page 118: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 118

For most applications, the execution units stall 80% or more of time during “execution”

From: Tullsen, Eggers, and Levy,

“Simultaneous Multithreading: Maximizing On-chip Parallelism, ISCA 1995. (From U Wash.)

For an 8-way superscalar.

<=#1

<=#2

18 18% CPU usefully busy

Page 119: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 119

Simultaneous Multi-threading ...

1

2

3

4

5

6

7

8

9

M M FX FX FP FP BR CCCycleOne thread, 8 func units

M = Load/Store, FX = Fixed Point, FP = Floating Point, BR = Branch, CC = Condition Codes

1

2

3

4

5

6

7

8

9

M M FX FX FP FP BR CCCycleTwo threads, 8 units

Busy: 13/72 = 18.0% Busy: 30/72 = 41.7%

Page 120: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 120

Simultaneous Multithreading (SMT)

• Simultaneous multithreading (SMT): insight that a dynamically scheduled processor already has many HW mechanisms to support multithreading

– Large set of virtual registers that can be used to hold the register sets of independent threads

– Register renaming provides unique register identifiers, so instructions from multiple threads can be mixed in datapath without confusing sources and destinations across threads

– Out-of-order completion allows the threads to execute out of order, and get better utilization of the HW

• Just need to add a per-thread renaming table and keeping separate PCs

– Independent commitment can be supported by logically keeping a separate reorder buffer for each thread

Source: Micrprocessor Report, December 6, 1999 “Compaq Chooses SMT for Alpha”

Page 121: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 121

Multithreading CategoriesTi

me

(pro

cess

or

cycle

)Pipes: 1 2 3 4

Superscalar

New Thread/cyc

Fine-Grained

Many Cyc/thread

Coarse-Grained

Separate Jobs

Multiprocessing

FUs: 1 2 3 4

Simultaneous

Multithreading

Thread 1

Thread 2Thread 3Thread 4

Thread 5Idle slot

16/48 = 33.3% 27/48 = 56.3% 27/48 = 56.3% 29/48 = 60.4% 42/48 = 87.5%

Page 122: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 122

Design Challenges in SMT

• Since SMT makes sense only with fine-grained implementation, impact of fine-grained scheduling on single thread performance?

– Does designating a preferred thread allow sacrificing neither throughput nor single-thread performance?

– Unfortunately, with a preferred thread, the processor is likely to sacrifice some throughput when the preferred thread stalls

• Larger register file needed to hold multiple contexts

• Try not to affect clock cycle time, especially in – Instruction issue - more candidate instructions need to be

considered

– Instruction completion - choosing which instructions to commit may be challenging

• Ensuring that cache and TLB conflicts generated by SMT do not degrade performance

Page 123: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 123

Power 4 - 1 threadPower 4 - 1 thread

Power 5 - 2 threadsPower 5 - 2 threads

2 fetch (PC),2 initial decodes

2 completes (architected register sets)

See www.ibm.com/servers/eserver/pseries/news/related/2004/m2040.pdfPower5 instruction pipeline (IF = instruction fetch, IC = instruction cache, BP = branch predict, D0 = decode stage 0, Xfer = transfer, GD = group dispatch, MP = mapping, ISS = instruction issue, RF = register file read, EX = execute, EA = compute address, DC =

data caches, F6 = six-cycle floating-point execution pipe, Fmt = data format, WB = write back, and CP = group commit) Page 43.

Page 124: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 124

Changes in Power 5 to support SMT

• Increased associativity of L1 instruction cache and the instruction address translation buffers

• Added per thread load and store queues • Increased size of the L2 (1.92 vs. 1.44 MB) and L3

caches• Added separate instruction prefetch and buffering

per thread• Increased the number of virtual registers from 152

to 240• Increased the size of several issue queues• The Power5 core is about 24% larger than the

Power4 core because of the addition of SMT support

Page 125: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 125

Initial Performance of SMT

• Pentium 4 Extreme SMT yields 1.01 speedup for SPECint_rate benchmark and 1.07 for SPECfp_rate

– Pentium 4 is dual-threaded SMT

– SPECRate requires that each SPEC benchmark be run against a vendor-selected number of copies of the same benchmark

• Running on Pentium 4 with each of 26 SPEC benchmarks paired with every other (26*26 runs) gave speed-ups from 0.90 to 1.58; average was 1.20

• Power 5, 8 processor server 1.23 faster for SPECint_rate with SMT, 1.16 faster for SPECfp_rate

• Power 5 running 2 copies of each application gave speedups between 0.89 and 1.41

– Most gained some

– Floating Pt. applications had most cache conflicts and least gains

Page 126: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 126

Processor Micro architecture Fetch / Issue /

Execute

Funct.Units

Clock Rate (GHz)

Transis-tors

Die size

Power

Intel Pentium

4 Extreme

Speculative dynamically

scheduled; deeply pipelined; SMT

3/3/4 7 int. 1 FP

3.8 125 M 122 mm2

115 W

AMD Athlon 64

FX-57

Speculative dynamically scheduled

3/3/4 6 int. 3 FP

2.8 114 M 115 mm2

104 W

IBM Power5 (1 CPU only)

Speculative dynamically

scheduled; SMT; 2 CPU cores/chip

8/4/8 6 int. 2 FP

1.9 200 M 300 mm2 (est.)

80W (est.)

Intel Itanium 2

Statically scheduled VLIW-style

6/5/11 9 int. 2 FP

1.6 592 M 423 mm2

130 W

Head to Head ILP competition

Page 127: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 127

Normalized Performance: Efficiency

0

5

10

15

20

25

30

35

SPECInt / MTransistors

SPECFP / MTransistors

SPECInt /mm^2

SPECFP /mm^2

SPECInt /Watt

SPECFP /Watt

I tanium 2 Pentium 4 AMD Athlon 64 POWER 5

RankItan

ium2

PentIum4

Athlon

Power5

Int/Trans 4 2 1 3

FP/Trans 4 2 1 3

Int/area 4 2 1 3

FP/area 4 2 1 3

Int/Watt 4 3 1 2

FP/Watt 2 4 3 1

Page 128: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 128

Limits to ILP

• Doubling issue rates above today’s 3-6 instructions per clock, say to 6 to 12 instructions, probably requires a processor to

– issue 3 or 4 data memory accesses per cycle,

– resolve 2 or 3 branches per cycle,

– rename and access more than 20 registers per cycle, and

– fetch 12 to 24 instructions per cycle.

• The complexities of implementing these capabilities is likely to mean sacrifices in the maximum clock rate

– E.g, widest issue processor is the Itanium 2, but it also has the slowest clock rate, despite the fact that it consumes the most power!

Page 129: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/15/09 CSE502-F09, Lec 11-SMT 129

Limits to ILP

• Most techniques for increasing performance increase power consumption

• The key question is whether a technique is energy efficient: does it increase performance faster than it increases power consumption?

• Multiple issue processor techniques all are energy inefficient:1. Issuing multiple instructions incurs some overhead in

logic that grows faster than the issue rate grows2. Growing gap between peak issue rates and sustained

performance

• Number of transistors switching = f(peak issue rate), and performance = f( sustained rate), growing gap between peak and sustained performance increasing energy per unit of performance

Page 130: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

CSE 502 Graduate Computer

Architecture

Lec 12-14 – Vector Computers

Larry WittieComputer Science, StonyBrook University

http://www.cs.sunysb.edu/~cse502 and ~lw

Slides adapted from Krste Asanovic of MIT and David Patterson of UCB, UC-Berkeley cs252-s06

Page 131: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 131

Vector Programming Model

+ + + + + +

[0] [1] [VLR-1]

Vector Arithmetic Instructions

ADDV v3, v1, v2 v3

v2v1

v1Vector Load and Store Instructions

LV v1, r1, r2

Base, r1 Stride, r2Memory

Vector Register

Scalar Registers

r0

r15Vector Registers

v0

v15

[0] [1] [2] [VLRMAX-1]

VLRVector Length Register[63], [127], [255], …

Page 132: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 132

Vector Instruction Set Advantages

• Compact–one short instruction encodes N operations => N*FlOp BandWidth

• Expressive, tells hardware that these N operations:–are independent

–use the same functional unit

–access disjoint registers

–access registers in the same pattern as previous instructions

–access a contiguous block of memory (unit-stride load/store) OR

access memory in a known pattern (strided load/store)

• Scalable–can run same object code on more parallel pipelines or lanes

Page 133: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 133

Vector Code Example

# Scalar Code

LI R4, 64

loop:

L.D F0, 0(R1)

L.D F2, 0(R2)

ADD.D F4, F2, F0

S.D F4, 0(R3)

DADDIU R1, 8

DADDIU R2, 8

DADDIU R3, 8

DSUBIU R4, 1

BNEZ R4, loop

# Vector Code

LI VLR, 64

LV V1, R1

LV V2, R2

ADDV.D V3, V1, V2

SV V3, R3

# C code

for (i=0; i<64; i++)

C[i] = A[i] + B[i];

Page 134: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 134

Vector Arithmetic Execution

• Use deep pipeline (=> fast clock) to execute element operations

• Simplifies control of deep pipeline because elements in vector are independent (=> no hazards!)

V1

V2

V3

V3 <- v1 * v2

Six stage multiply pipeline

Page 135: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 135

Properties of Vector Processors

• Each result independent of previous result=> long pipeline, compiler ensures no dependencies=> high clock rate

• Vector instructions access memory with known pattern=> highly interleaved memory=> amortize memory latency of 64-plus elements=> no (data) caches required! (but use instruction cache)

• Reduces branches and branch problems in pipelines

• Single vector instruction implies lots of work (≈ loop)=> fewer instruction fetches

Page 136: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 136

Common Vector Metrics

• R: MFLOPS rate on an infinite-length vector

– vector “speed of light”

– Real problems do not have unlimited vector lengths, and the start-up penalties encountered in real problems will be larger

– (Rn is the MFLOPS rate for a vector of length n)

• N1/2: The vector length needed to reach one-half of R

– a good measure of the impact of start-up

• NV: Minimum vector length for vector mode faster than scalar mode

– measures both start-up and speed of scalars relative to vectors, quality of connection of scalar unit to vector unit

Page 137: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 137

Vector Execution Time

Time = f(vector length, data dependencies, struct. hazards) • Initiation rate: rate that FU consumes vector elements

(= number of lanes; usually 1 or 2 on Cray T-90)• Convoy: set of vector instructions that can begin

execution on same clock (if no structural or data hazards)• Chime: approximate time for a vector operation• m convoys take m chimes; if each vector length is n, then

they take approx. m x n clock cycles if no chaining (ignores overhead; good approximization for long vectors) and as little as m + n - 1 cycles, if fully chained.

4 convoys, 1 lane, VL=64=> 4 x 64 = 256 clocks(or 4 clocks per result)

1: LV V1,Rx ;load vector X

2: MULV V2,F0,V1 ;vector-scalar mult.

LV V3,Ry ;load vector Y

3: ADDV V4,V2,V3 ;add

4: SV Ry,V4 ;store the result

Page 138: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 138

Memory operations

• Load/store operations move groups of data between registers and memory

• Three types of addressing– Unit stride

» Contiguous block of information in memory» Fastest: always possible to optimize this

– Non-unit (constant) stride» Harder to optimize memory system for all possible strides» Prime number of data banks makes it easier to support

different strides at full bandwidth (Duncan Lawrie patent)– Indexed (gather-scatter)

» Vector equivalent of register indirect» Good for sparse arrays of data» Increases number of programs that vectorize

Page 139: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 139

Interleaved Memory Layout

• Great for unit stride: – Contiguous elements in different DRAMs– Startup time for vector operation is latency of single read

• What about non-unit stride?– Banks above are good for strides that are relatively prime to 8– Bad for: 2, 4– Better: prime number of banks…!

Vector Processor

Un

pip

elin

ed

DR

AM

Un

pip

elin

ed

DR

AM

Un

pip

elin

ed

DR

AM

Un

pip

elin

ed

DR

AM

Un

pip

elin

ed

DR

AM

Un

pip

elin

ed

DR

AM

Un

pip

elin

ed

DR

AM

Un

pip

elin

ed

DR

AM

AddrMod 8= 0

AddrMod 8= 1

AddrMod 8= 2

AddrMod 8= 4

AddrMod 8= 5

AddrMod 8= 3

AddrMod 8= 6

AddrMod 8= 7

Page 140: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 140

How Get Full Bandwidth if Unit Stride?

• Memory system must sustain (# lanes x word) /clock

• Num. memory banks > memory latency to avoid stalls– M banks M words per memory latency L in clocks

– if M < L, then “gap” in memory pipeline:

clock: 0 … L L+1 L+2 … L+M- 1 L+M … 2 L

word: -- … 0 1 2 … M-1 -- … M– may have 1024 banks in SRAM

• If desired throughput greater than one word per cycle– Either more banks (and start multiple requests simultaneously)

– Or wider DRAMS. Only good for unit stride or large data types

• More banks & weird (prime) numbers of banks good to support more strides at full bandwidth

Page 141: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 141

Vectors Are Inexpensive

Multiscalar• N ops per cycle

2) circuitry• HP PA-8000

• 4-way issue• reorder buffer alone:

850K transistors• incl. 6,720 5-bit register

number comparators

Vector• N ops per cycle

2) circuitry• UCB-T0 Integer vector µP

• 24 ops per cycle• 730K transistors total

• only 23 5-bit register number comparators

• Integer, no floating point

Page 142: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 142

Vectors Lower Power

Vector• One inst fetch, decode,

dispatch per vector• Structured register

accesses• Smaller code for high

performance, less power in instruction cache misses

• Bypass cache

• One TLB lookup pergroup of loads or stores

• Move only necessary dataacross chip boundary

Single-issue Scalar

• One instruction fetch, decode, dispatch per operation

• Arbitrary register accesses,adds area and power

• Loop unrolling and software pipelining for high performance increases instruction cache footprint

• All data passes through cache; waste power if no temporal locality

• One TLB lookup per load or store

• Off-chip access is via whole cache lines

Page 143: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 143

Superscalar Energy Efficiency Even Worse

Vector• Control logic grows

linearly with issue width• Vector unit switches

off when not in use

• Vector instructions expose parallelism without speculation

• Software control ofspeculation when desired:

– Whether to use vector mask or compress/expand for conditionals

Superscalar• Control logic grows

quadratically with issue width (nxn hazard chks)

• Control logic consumes energy regardless of available parallelism

• Speculation to increase visible parallelism wastes energy

Page 144: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 144

Vector SupercomputersEpitomized by Cray-1, 1976 (from icy Minnesota):

Scalar Unit + Vector Extensions• Load/Store Architecture*

• Vector Registers• Vector Instructions• Hardwired Control• Highly Pipelined Functional Units*

• Interleaved Memory System• No Data Caches• No Virtual Memory “World’s most costly• 1976 80 M Fl.Op./sec (‘79 160 MFlops) warm loveseat”

(2008 SBU/BNL NY IBM BlueGene: 120,000,000 MFLOPS)

* 2 features of modern instruction-pipeline CPUs

Heat exchanger

Page 145: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 145

Cray-1 (1976)

Single PortedMemory

16 banks of 64-bit words+ 8-bit SECDEDSingle Error CorrectDouble Error Detect

80MW/sec data load/store

320MW/sec instructionbuffer refill

4 Instruction Buffers

64-bitx16NIP

LIP

CIP

(A0)

( (Ah) + j k m )

64T Regs

(A0)

( (Ah) + j k m )

64 B Regs

S0S1S2S3S4S5S6S7

A0A1A2A3A4A5A6A7

Si

Tjk

Ai

Bjk

FP Add

FP Mul

FP Recip

Int Add

Int Logic

Int Shift

Pop Cnt

Sj

Si

Sk

Addr Add

Addr Mul

Aj

Ai

Ak

memory bank cycle 50 ns processor cycle 12.5 ns (80MHz)

V0V1V2V3V4V5V6V7

Vk

Vj

Vi V. Mask

V. Length8 Vector Registers64 Elements Each

NextIP holds next opcode.

Current Instruction Parcel (CIP) register issues 16-bit instructions; CIP + LIP (LowerIP) => 32-bit instructions.

T regs: passive backups for 8 active Scalar regs

B regs: passive backups for 8 active Address regs

Funct. Units

Page 146: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 146

VMIPS Double-Precision Vector Instructions

Figure F.3 The VMIPS vector instructions. Only the double-precision FP operations are shown. In addition to the vector registers, there are two special registers, VLR (discussed in Section F.3) and VM (discussed in Section F.4). These special registers are assumed to live in the MIPS coprocessor 1 space along with the FPU registers. The operations with stride are explained in Section F.3, and the uses of the index creation and indexed load-store operations are explained in Section F.4. (From page F-8 Appendix F Vector Processors of CAQA4e)

Page 147: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 147

Vector Instruction ExecutionADDV C,A,B

C[1]

C[2]

C[0]

A[3] B[3]

A[4] B[4]

A[5] B[5]

A[6] B[6]

Execution using one pipelined functional unit

C[4]

C[8]

C[0]

A[12] B[12]

A[16] B[16]

A[20] B[20]

A[24] B[24]

C[5]

C[9]

C[1]

A[13] B[13]

A[17] B[17]

A[21] B[21]

A[25] B[25]

C[6]

C[10]

C[2]

A[14] B[14]

A[18] B[18]

A[22] B[22]

A[26] B[26]

C[7]

C[11]

C[3]

A[15] B[15]

A[19] B[19]

A[23] B[23]

A[27] B[27]

Four-laneexecution using four pipelined functional units

Page 148: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 148

Vector Unit Structure

Lane

Functional Unit

VectorRegisters

Memory Subsystem

Elements 0, 4, 8, …

Elements 1, 5, 9, …

Elements 2, 6, 10, …

Elements 3, 7, 11, …

Page 149: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 149

Automatic Code Vectorizationfor (i=0; i < N; i++) C[i] = A[i] + B[i];

load

load

add

store

load

load

add

store

Iter. 1

Iter. 2

Scalar Sequential Code

Vectorization is a massive compile-time reordering of operation sequencing requires extensive loop dependence analysis

Vector Instruction

load

load

add

store

load

load

add

store

Iter. 1

Iter. 2

Vectorized Code

Tim

e

Page 150: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 150

Vector StripminingProblem: Vector registers have finite length (64)

Solution: Break longer (than 64) loops into pieces that fit into vector registers, “Stripmining”

ANDI R1, N, 63 # N mod 64 MTC1 VLR, R1 # Do remainderloop: LV V1, RA # Vector load A DSLL R2, R1, 3 # Multiply N%64 *8 DADDU RA, RA, R2 # Bump RA pointer LV V2, RB # Vector load B DADDU RB, RB, R2 # Bump RB pointer ADDV.D V3, V1, V2 # Vector add SV V3, RC # Vector store C DADDU RC, RC, R2 # Bump RC pointer DSUBU N, N, R1 # R1 elements done LI R1, 64 # Vector length is MTC1 VLR, R1 # Set to full 64 BGTZ N, loop # Any more to do?

for (i=0; i<N; i++) C[i] = A[i]+B[i];

+

+

+

A B C

64 elements

Remainder

64 elements

Page 151: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 151

load

Vector Instruction ParallelismChain to overlap execution of multiple vector

instructions– example machine has 32 elements per vector register and 8 lanes

loadmul

mul

add

add

Load Unit Multiply Unit Add Unit

Time

6 issues of instruction

Complete 24 operations/cycle but issue 1 short instruction/cycle

Cycle

1

2

3

4

5

6

7

8

9

10

Page 152: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 152

Vector Chaining

• Vector version of register bypassing– First in revised Cray-1 ‘79, Rpeak: 80 MFlops in ‘76 => 160 MFlops in ‘79

Memory

V1

Load Unit

Mult.

V2

V3

Chain

Add

V4

V5

Chain

LV v1

MULV v3,v1,v2

ADDV v5, v3, v4

Page 153: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 153

Vector Chaining Advantage

• With chaining, can start dependent instruction as soon as first result appears

Load

Mul

Add

Load

Mul

AddTime

• Without chaining, must wait for last element of result to be written before starting dependent instruction

Page 154: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 154

Vector Scatter/Gather

Want to vectorize loops with indirect accesses:for (i=0; i<N; i++)

A[i] = B[i] + C[D[i]]

Indexed load instruction (Gather)LV vD, rD # Load indices in D vector

LVI vC, (rC+vD) # Load indirect from rC base

LV vB, rB # Load B vector

ADDV.D vA, vB, vC # Do add

SV vA, rA # Store result

Page 155: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 155

Vector Conditional Execution

Problem: Want to vectorize loops with conditional code:for (i=0; i<N; i++) if (A[i]>0) then A[i] = B[i];

Solution: Add vector mask (or flag) registers– vector version of predicate registers, 1 bit per element

…and maskable vector instructions– vector operation becomes NOOP at elements where mask bit is clear (0)

Code example:CVM # Turn on all elements

LV vA, rA # Load entire A vector

SGTVS.D vA, F0 # Set bits in mask register where A>0

LV vA, rB # Load B vector into A under mask

SV vA, rA # Store A back to memory under mask

Page 156: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 156

Compress/Expand Operations

• Compress packs non-masked elements from one vector register contiguously at start of destination vector reg.

– population count of mask vector gives packed vector length

• Expand performs inverse operation

M[3]=0

M[4]=1

M[5]=1

M[6]=0

M[2]=0

M[1]=1

M[0]=0

M[7]=1

A[3]

A[4]

A[5]

A[6]

A[7]

A[0]

A[1]

A[2]

M[3]=0

M[4]=1

M[5]=1

M[6]=0

M[2]=0

M[1]=1

M[0]=0

M[7]=1

B[3]

A[4]

A[5]

B[6]

A[7]

B[0]

A[1]

B[2]

Expand

A[7]

A[1]

A[4]

A[5]

Compress

A[7]

A[1]

A[4]

A[5]

Used for density-time conditionals and for general selection operations

Page 157: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 157

Vector Reductions (vector values => one result)

Problem: Loop-carried dependence on reduction variablessum = 0;

for (i=0; i<N; i++)

sum += A[i]; # Loop-carried dependence on sum

Solution: Re-associate operations if possible, use binary tree to perform reduction# Rearrange as:

sum[0:VL-1] = 0 # Vector of VL partial sums

for(i=0; i<N; i+=VL) # Stripmine VL-sized chunks

sum[0:VL-1] += A[i:i+VL-1]; # Vector sum

# Now have VL partial sums in one vector register

do {

VL = VL/2; # Halve vector length

sum[0:VL-1] += sum[VL:2*VL-1] # Halve no. of partials

} while (VL>1)

Page 158: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 158

Vector Summary

• Vector is alternative model for exploiting ILP

• If code is vectorizable, then simpler hardware, more energy efficient, and better real-time model than out-of-order machines

• Design issues include number of lanes, number of functional units, number of vector registers, length of vector registers, exception handling, conditional operations

• Fundamental design issue is memory bandwidth– Especially with virtual address translation and caching

• Will multimedia popularity revive vector architectures?

Page 159: CSE 502 Graduate Computer Architecture  Lec 15 – MidTerm Review

10/22-29/09 CSE502-F09, Lec 12+13+14-Vector 159

And in Conclusion [Vector Processing]

• One instruction operates on vectors of data

• Vector loads get data from memory into big register files, operate, and then vector store

• Have indexed load, store for sparse matrices

• Easy to add vectors to commodity instruction sets

– E.g., Morph SIMD into vector processing

• Vector is a very efficient architecture for vectorizable codes, including multimedia and many scientific matrix applications