memory hierarchy computer organization and assembly languages yung-yu chuang 2007/01/08 with slides...
DESCRIPTION
Reference Chapter 6 from “Computer System: A Programmer’s Perspective”TRANSCRIPT
![Page 1: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/1.jpg)
Memory Hierarchy
Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08with slides by CMU15-213
![Page 2: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/2.jpg)
Announcement• Please submit homeworks if you haven’t by 1/10.• Final project demo time on 1/24 or 1/31. The mechanism for signup will be mailed to the mailing list soon.• Mail your report to TA before your demo time. The length of report depends on your project type. It can be html, pdf, doc, ppt…• TA evaluation today
![Page 3: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/3.jpg)
Reference• Chapter 6 from “Computer System: A
Programmer’s Perspective”
![Page 4: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/4.jpg)
Computer system model• We assume memory is a linear array which
holds both instruction and data, and CPU can access memory in a constant time.
![Page 5: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/5.jpg)
SRAM vs DRAM
Tran. Access Needs per bit time refresh? Cost Applications
SRAM 4 or 6 1X No 100X cache memories
DRAM 1 10X Yes 1X Main memories,frame buffers
![Page 6: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/6.jpg)
The CPU-Memory gap The gap widens between DRAM, disk, and CPU
speeds.
110
1001,000
10,000100,000
1,000,00010,000,000
100,000,000
1980 1985 1990 1995 2000
year
ns
Disk seek timeDRAM access timeSRAM access timeCPU cycle time
register cache memory diskAccess time
(cycles)
1 1-10 50-100 20,000,000
![Page 7: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/7.jpg)
Memory hierarchies• Some fundamental and enduring
properties of hardware and software:– Fast storage technologies cost more per byte,
have less capacity, and require more power (heat!).
– The gap between CPU and main memory speed is widening.
– Well-written programs tend to exhibit good locality.
• They suggest an approach for organizing memory and storage systems known as a memory hierarchy.
![Page 8: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/8.jpg)
Memory system in practice
Larger, slower, and cheaper (per byte) storage devices
registers
on-chip L1cache (SRAM)
main memory(DRAM)
local secondary storage(local disks)
remote secondary storage(tapes, distributed file systems, Web servers)
off-chip L2cache (SRAM)
L0:
L1:
L2:
L3:
L4:
L5:
Smaller, faster, and more expensive (per byte) storage devices
![Page 9: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/9.jpg)
Why does it work?• Most programs tend to access the storage
at any particular level more frequently than the storage at the lower level.
• Locality: tend to access the same set of data items over and over again or tend to access sets of nearby data items.
![Page 10: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/10.jpg)
Why learning it?• A programmer needs to understand this
because the memory hierarchy has a big impact on performance.
• You can optimize your program so that its data is more frequently stored in the higher level of the hierarchy.
• For example, the difference of running time for matrix multiplication could up to a factor of 6 even if the same amount of arithmetic instructions are performed.
![Page 11: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/11.jpg)
Locality• Principle of Locality: programs tend to reuse
data and instructions near those they have used recently, or that were recently referenced themselves.– Temporal locality: recently referenced items are
likely to be referenced in the near future.– Spatial locality: items with nearby addresses
tend to be referenced close together in time.• In general, programs with good locality run faster then programs with poor locality• Locality is the reason why cache and virtual memory are designed in architecture and operating system. Another example is web browser caches recently visited webpages.
![Page 12: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/12.jpg)
Locality example
• Data– Reference array elements in succession (stride-
1 reference pattern):– Reference sum each iteration:
• Instructions– Reference instructions in sequence:– Cycle through loop repeatedly:
sum = 0;for (i = 0; i < n; i++)
sum += a[i];return sum;
Spatial locality
Spatial locality
Temporal locality
Temporal locality
![Page 13: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/13.jpg)
Locality example• Being able to look at code and get a
qualitative sense of its locality is important. Does this function have good locality?
int sum_array_rows(int a[M][N]){ int i, j, sum = 0;
for (i = 0; i < M; i++) for (j = 0; j < N; j++) sum += a[i][j]; return sum;} stride-1 reference pattern
![Page 14: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/14.jpg)
Locality example• Does this function have good locality?
int sum_array_cols(int a[M][N]){ int i, j, sum = 0;
for (j = 0; j < N; j++) for (i = 0; i < M; i++) sum += a[i][j]; return sum;} stride-N reference pattern
![Page 15: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/15.jpg)
Memory hierarchies
Larger, slower, and cheaper (per byte) storage devices
registers
on-chip L1cache (SRAM)
main memory(DRAM)
local secondary storage(local disks)
remote secondary storage(tapes, distributed file systems, Web servers)
off-chip L2cache (SRAM)
L0:
L1:
L2:
L3:
L4:
L5:
Smaller, faster, and more expensive (per byte) storage devices
![Page 16: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/16.jpg)
Cache memories• Cache memories are small, fast SRAM-
based memories managed automatically in hardware.
• CPU looks first for data in L1, then in L2, then in main memory.
• Typical system structure:
mainmemory
I/Obridgebus interfaceL2 data
ALU
register fileCPU chip
SRAM Port system busmemory busL1
cache
![Page 17: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/17.jpg)
Caching in a memory hierarchy
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Larger, slower, cheaper Storage device at level k+1 is partitioned into blocks.
Data is copied between levels in block-sized transfer units
8 9 14 3Smaller, faster, more Expensive device at level k caches a subset of the blocks from level k+1
level k
level k+1
4
4
4 10
10
10
![Page 18: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/18.jpg)
Request14
Request12
General caching concepts• Program needs object d, which is
stored in some block b.• Cache hit
– Program finds b in the cache at level k. E.g., block 14.
• Cache miss– b is not at level k, so level k cache
must fetch it from level k+1. E.g., block 12.
– If level k cache is full, then some current block must be replaced (evicted). Which one is the “victim”?
• Placement policy: where can the new block go? E.g., b mod 4
• Replacement policy: which block should be evicted? E.g., LRU
9 3
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
level k
level k+1
1414
12
14
4*
4*12
12
0 1 2 3
Request12
4*4*12
![Page 19: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/19.jpg)
Type of cache misses• Cold (compulsory) miss: occurs because the
cache is empty.• Capacity miss: occurs when the active cache
blocks (working set) is larger than the cache.• Conflict miss
– Most caches limit blocks at level k+1 to a small subset of the block positions at level k, e.g. block i at level k+1 must be placed in block (i mod 4) at level k.
– Conflict misses occur when the level k cache is large enough, but multiple data objects all map to the same level k block, e.g. Referencing blocks 0, 8, 0, 8, 0, 8, ... would miss every time.
![Page 20: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/20.jpg)
General organization of a cache
• • • B–110
• • • B–110
valid
valid
tag
tagset 0:
B = 2b bytesper cache block
E lines per set
S = 2s sets
t tag bitsper line
Cache size: C = B x E x S data bytes
• • •
• • • B–110
• • • B–110
valid
valid
tag
tagset 1: • • •
• • • B–110
• • • B–110
valid
valid
tag
tagset S-1: • • •
• • •
Cache is an arrayof sets.
Each set containsone or more lines.
Each line holds ablock of data.
1 valid bit per line
![Page 21: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/21.jpg)
Addressing caches
t bits s bits b bits
<tag> <set index><block offset>
0m-1
Address A:
• • •B–110
• • •B–110
v
v
tag
tagset 0: • • •
• • •B–110
• • •B–110
v
v
tag
tagset 1: • • •
• • •B–110
• • •B–110
v
v
tag
tagset S-1: • • •
• • • The word at address A is in the cache ifthe tag bits in one of the <valid> lines in set <set index> match <tag>.
The word contents begin at offset <block offset> bytes from the beginning of the block.
![Page 22: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/22.jpg)
Addressing caches
t bits s bits b bits
<tag> <set index><block offset>
0m-1
Address A:
• • •B–110
• • •B–110
v
v
tag
tagset 0: • • •
• • •B–110
• • •B–110
v
v
tag
tagset 1: • • •
• • •B–110
• • •B–110
v
v
tag
tagset S-1: • • •
• • • 1. Locate the set based on <set index>
2. Locate the line in the set based on <tag>
3. Check that the line is valid4. Locate the data in the line based on
<block offset>
![Page 23: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/23.jpg)
Direct-mapped cache• Simplest kind of cache, easy to build
(only 1 tag compare required per access)• Characterized by exactly one line per set.
valid
valid
valid
tag
tag
tag• • •
set 0:
set 1:
set S-1:
E=1 lines per setcache block
cache block
cache block
Cache size: C = B x S data bytes
![Page 24: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/24.jpg)
Accessing direct-mapped caches• Set selection
– Use the set index bits to determine the set of interest.
t bits s bits0 0 0 0 1
0m-1
b bits
tag set index block offset
selected setvalid
valid
valid
tag
tag
tag• • •
set 0:
set 1:
set S-1:
cache block
cache block
cache block
![Page 25: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/25.jpg)
Accessing direct-mapped caches• Line matching and word selection
– Line matching: Find a valid line in the selected set with a matching tag
– Word selection: Then extract the word
t bits s bits100i0110
0m-1
b bits
tag set index block offset
selected set (i): 1 0110 w3w0 w1 w2
30 1 2 74 5 6
=1?(1) The valid bit must be set
= ?(2) The tag bits in the
cache line must match the tag bits in the address
If (1) and (2), then cache hit
![Page 26: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/26.jpg)
Accessing direct-mapped caches• Line matching and word selection
– Line matching: Find a valid line in the selected set with a matching tag
– Word selection: Then extract the word
t bits s bits100i0110
0m-1
b bits
tag set index block offset
selected set (i): 1 0110 w3w0 w1 w2
30 1 2 74 5 6
(3) If cache hit,block offset selects starting byte.
![Page 27: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/27.jpg)
Direct-mapped cache simulationM=16 byte addresses, B=2 bytes/block, S=4 sets, E=1 entry/set
Address trace (reads):0 [00002], 1 [00012], 7 [01112], 8 [10002], 0 [00002]
xt=1 s=2 b=1
xx x
0 ? ?v tag data
miss
1 0 M[0-1]
hitmiss
1 0 M[6-7]
miss
1 1 M[8-9]
miss
1 0 M[0-1]
![Page 28: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/28.jpg)
What’s wrong with direct-mapped?float dotprod(float x[8], y[8]) { float sum=0.0; for (int i=0; i<8; i++) sum+= x[i]*y[i]; return sum;}
element address set element address setx[0] 0 0 y[0] 32 0x[1] 4 0 y[1] 36 0x[2] 8 0 y[2] 40 0x[3] 12 0 y[3] 44 0x[4] 16 1 y[4] 48 1x[5] 20 1 y[5] 52 1x[6] 24 1 y[6] 56 1x[7] 28 1 y[7] 60 1
two setsblock size=16 bytes
![Page 29: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/29.jpg)
Solution? padding
element address set element Address setx[0] 0 0 y[0] 48 1x[1] 4 0 y[1] 52 1x[2] 8 0 y[2] 56 1x[3] 12 0 y[3] 60 1x[4] 16 1 y[4] 64 0x[5] 20 1 y[5] 68 0x[6] 24 1 y[6] 72 0x[7] 28 1 y[7] 76 0
float dotprod(float x[12], y[8]) { float sum=0.0; for (int i=0; i<8; i++) sum+= x[i]*y[i]; return sum;}
![Page 30: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/30.jpg)
Set associative caches• Characterized by more than one line per
set
E=2lines per set
valid tagset 0:
set 1:
set S-1:
• • •
cache blockvalid tag cache block
valid tag cache blockvalid tag cache block
valid tag cache blockvalid tag cache block
E-way associative cache
![Page 31: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/31.jpg)
Accessing set associative caches• Set selection
– identical to direct-mapped cachevalidvalid
tagtagset 0:
validvalid
tagtagset 1:
validvalid
tagtagset S-1:
• • •
cache blockcache block
cache blockcache block
cache blockcache block
t bits s bits0 0 0 0 1
0m-1
b bits
tag set index block offset
selected set
![Page 32: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/32.jpg)
Accessing set associative caches• Line matching and word selection
– must compare the tag in each valid line in the selected set.
1 0110 w3w0 w1 w2
1 1001selected set (i):
30 1 2 74 5 6
t bits s bits100i0110
0m-1
b bits
tag set index block offset
=1?(1) The valid bit must be set
= ?
(2) The tag bits in one of the cache lines must match the tag bits in the address
If (1) and (2), then cache hit
![Page 33: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/33.jpg)
Accessing set associative caches• Line matching and word selection
– Word selection is the same as in a direct mapped cache
1 0110 w3w0 w1 w2
1 1001selected set (i):
30 1 2 74 5 6
t bits s bits100i0110
0m-1
b bits
tag set index block offset
(3) If cache hit,block offset selects starting byte.
![Page 34: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/34.jpg)
2-Way associative cache simulation
M=16 byte addresses, B=2 bytes/block, S=2 sets, E=2 entry/set
Address trace (reads):0 [00002], 1 [00012], 7 [01112], 8 [10002], 0 [00002]
xxt=2s=1 b=1
x x
0 ? ?v tag data000
miss
1 00 M[0-1]
hitmiss
1 01 M[6-7]
miss
1 10 M[8-9]
hit
![Page 35: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/35.jpg)
Why use middle bits as index?
• High-order bit indexing–adjacent memory
lines would map to same cache entry
–poor use of spatial locality
4-line Cache High-OrderBit Indexing
Middle-OrderBit Indexing
00011011
0000000100100011010001010110011110001001101010111100110111101111
0000000100100011010001010110011110001001101010111100110111101111
![Page 36: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/36.jpg)
What about writes?• Multiple copies of data exist:
– L1– L2– Main Memory– Disk
• What to do when we write?– Write-through– Write-back (need a dirty bit)
• What to do on a replacement?– Depends on whether it is write through or write
back
![Page 37: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/37.jpg)
Multi-level caches• Options: separate data and instruction
caches, or a unified cache
size:speed:$/Mbyte:line size:
200 B3 ns
8 B
8-64 KB3 ns
32 B
128 MB DRAM60 ns$1.50/MB8 KB
30 GB8 ms$0.05/MB
larger, slower, cheaper
MemoryRegs
UnifiedL2
Cache
Processor
1-4MB SRAM6 ns$100/MB32 B
disk
L1 d-cache
L1 i-cache
![Page 38: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/38.jpg)
Intel Pentium III cache hierarchy
Processor Chip
L1 Data1 cycle latency
16 KB4-way assoc
Write-through32B lines
L1 Instruction16 KB, 4-way
32B lines
Regs. L2 Unified128KB--2 MB4-way assocWrite-back
Write allocate32B lines
MainMemory
Up to 4GB
![Page 39: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/39.jpg)
The memory mountain• Read throughput: number of bytes read
from memory per second (MB/s)• Memory mountain
– Measured read throughput as a function of spatial and temporal locality.
– Compact way to characterize memory system performance.
void test(int elems, int stride) { int i, result = 0; volatile int sink; for (i = 0; i < elems; i += stride)
result += data[i]; /* So compiler doesn't optimize away the loop */ sink = result;}
![Page 40: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/40.jpg)
The memory mountain/* Run test(elems, stride) and return read throughput (MB/s) */double run(int size, int stride, double Mhz){ double cycles; int elems = size / sizeof(int);
test(elems, stride); /* warm up the cache */ /* call test(elems,stride) */ cycles = fcyc2(test, elems, stride, 0); /* convert cycles to MB/s */ return (size / stride) / (cycles / Mhz);}
![Page 41: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/41.jpg)
The memory mountain
s1
s3
s5
s7
s9
s11
s13
s15
8m
2m 512k 12
8k 32k 8k
2k
0
200
400
600
800
1000
1200
L1
L2
mem
xeSlopes ofSpatialLocality
Pentium III550 MHz
16 KB on-chip L1 d-cache16 KB on-chip L1 i-cache
512 KB off-chip unifiedL2 cache
Ridges ofTemporalLocality
Working set size(bytes)Stride (words)
Thro
ughp
ut (M
B/se
c)
![Page 42: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/42.jpg)
Ridges of temporal locality• Slice through the memory mountain
(stride=1)– illuminates read throughputs of different
caches and memory
0
200
400
600
800
1000
1200
8m 4m 2m
1024
k
512k
256k
128k 64
k
32k
16k 8k 4k 2k 1k
working set size (bytes)
read
thro
ugpu
t (M
B/s
)
L1 cacheregion
L2 cacheregion
main memoryregion
L2 is unified
callingoverheadis notamortized
![Page 43: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/43.jpg)
A slope of spatial locality• Slice through memory mountain
(size=256KB)– shows cache block size.
0
100
200
300
400
500
600
700
800
s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16
stride (words)
read
thro
ughp
ut (M
B/s
)
one access per cache line
![Page 44: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/44.jpg)
Matrix multiplication example• Major cache effects to consider
–Total cache size•Exploit temporal locality and keep the working set small
(e.g., use blocking)–Block size
•Exploit spatial locality
• Description:–Multiply N x N matrices–O(N3) total operations–Accesses
•N reads per source element•N values summed per destination
– but may be able to hold in register
/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}
Variable sumheld in register
![Page 45: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/45.jpg)
Miss rate analysis for matrix multiply• Assume:
– Line size = 32B (big enough for four 64-bit words)
– Matrix dimension (N) is very large• Approximate 1/N as 0.0
– Cache is not even big enough to hold multiple rows
• Analysis method:– Look at access pattern of inner loop
CA
k
i
B
k
j
i
j
![Page 46: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/46.jpg)
Matrix multiplication (ijk)
/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}
A B C(i,*)
(*,j)(i,j)
Inner loop:
Column-wise
Row-wise Fixed
Misses per Inner Loop Iteration:A B C
0.25 1.0 0.0
![Page 47: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/47.jpg)
Matrix multiplication (jik)
/* jik */for (j=0; j<n; j++) { for (i=0; i<n; i++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum }}
A B C(i,*)
(*,j)(i,j)
Inner loop:
Row-wise Column-wise
FixedMisses per Inner Loop Iteration:
A B C0.25 1.0 0.0
![Page 48: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/48.jpg)
Matrix multiplication (kij)
/* kij */for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}
A B C(i,*)
(i,k) (k,*)
Inner loop:
Row-wise Row-wiseFixed
Misses per Inner Loop Iteration:A B C
0.0 0.25 0.25
![Page 49: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/49.jpg)
Matrix multiplication (ikj)
/* ikj */for (i=0; i<n; i++) { for (k=0; k<n; k++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}
A B C(i,*)
(i,k) (k,*)
Inner loop:
Row-wise Row-wiseFixed
Misses per Inner Loop Iteration:A B C
0.0 0.25 0.25
![Page 50: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/50.jpg)
Matrix multiplication (jki)
/* jki */for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}
A B C
(*,j)(k,j)
Inner loop:(*,k)
Column -wise
Column-wise
Fixed
Misses per Inner Loop Iteration:A B C
1.0 0.0 1.0
![Page 51: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/51.jpg)
Matrix multiplication (kji)
/* kji */for (k=0; k<n; k++) { for (j=0; j<n; j++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}
A B C
(*,j)(k,j)
Inner loop:(*,k)
FixedColumn-wise
Column-wise
Misses per Inner Loop Iteration:A B C
1.0 0.0 1.0
![Page 52: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/52.jpg)
Summary of matrix multiplication
ijk (& jik): • 2 loads, 0 stores• misses/iter = 1.25
kij (& ikj): • 2 loads, 1 store• misses/iter = 0.5
jki (& kji): • 2 loads, 1 store• misses/iter = 2.0
for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }} for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}
![Page 53: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/53.jpg)
Pentium matrix multiply performance
• Miss rates are helpful but not perfect predictors.
• Code scheduling matters, too.
0
10
20
30
40
50
60
25 50 75 100 125 150 175 200 225 250 275 300 325 350 375 400
Array size (n)
Cyc
les/
itera
tion
kjijkikijikjjikijk
kji & jki
kij & ikj
jik & ijk
![Page 54: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/54.jpg)
Improving temporal locality by blocking• Example: Blocked matrix multiplication
– Here, “block” does not mean “cache block”.– Instead, it mean a sub-block within the matrix.– Example: N = 8; sub-block size = 4
C11 = A11B11 + A12B21 C12 = A11B12 + A12B22
C21 = A21B11 + A22B21 C22 = A21B12 + A22B22
A11 A12
A21 A22
B11 B12
B21 B22X =
C11 C12
C21 C22
Key idea: Sub-blocks (i.e., Axy) can be treated just like scalars.
![Page 55: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/55.jpg)
Blocked matrix multiply (bijk)for (jj=0; jj<n; jj+=bsize) { for (i=0; i<n; i++) for (j=jj; j < min(jj+bsize,n); j++) c[i][j] = 0.0; for (kk=0; kk<n; kk+=bsize) { for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; } } }}
![Page 56: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/56.jpg)
Blocked matrix multiply analysis– Innermost loop pair multiplies a 1 X bsize sliver
of A by a bsize X bsize block of B and accumulates into 1 X bsize sliver of C
– Loop over i steps through n row slivers of A & C, using same B
A B C
block reused n times in succession
row sliver accessedbsize times
Update successiveelements of sliver
i ikk
kk jjjj
for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; }
InnermostLoop Pair
![Page 57: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/57.jpg)
Blocked matrix multiply performance• Blocking (bijk and bikj) improves
performance by a factor of two over unblocked versions (ijk and jik)– relatively insensitive to array size.
0
10
20
30
40
50
60
Array size (n)
Cyc
les/
itera
tion
kjijkikijikjjikijkbijk (bsize = 25)bikj (bsize = 25)
![Page 58: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/58.jpg)
Writing cache friendly code•Repeated references to variables are good(temporal locality)
•Stride-1 reference are good (spatial locality)•Examples: cold cache, 4-byte words, 4-word cache blocks
int sum_array_rows(int a[4][8]){ int i, j, sum = 0;
for (i = 0; i < M; i++) for (j = 0; j < N; j++) sum += a[i][j]; return sum;}
int sum_array_cols(int a[4][8]){ int i, j, sum = 0;
for (j = 0; j < N; j++) for (i = 0; i < M; i++) sum += a[i][j]; return sum;}
Miss rate = Miss rate = 1/4 = 25% 100%
![Page 59: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/59.jpg)
Cache-conscious programming• make sure that memory is cache-aligned
• Use union and bitfields to reduce size and increase locality• Split data into hot and cold (list example)
![Page 60: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/60.jpg)
Cache-conscious programming• Prefetching • Blocked 2D array
….
w
h
…….
….
h
w
![Page 61: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/61.jpg)
K-d treestruct KdAccelNode { u_int flags; float split; // Interior u_int aboveChild; // Interior
u_int nPrims; // Leaf MailboxPrim *Primitives; // Leaf}
interior
nleaf
![Page 62: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/62.jpg)
K-d tree8-byte (reduced from 20-byte, 20% gain)struct KdAccelNode { ... union { u_int flags; // Both float split; // Interior u_int nPrims; // Leaf }; union { u_int aboveChild; // Interior MailboxPrim **primitives; // Leaf };}
interior
nleaf
![Page 63: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/63.jpg)
Tree representation
Flag: 0,1,2 (interior x, y, z) 3 (leaf)
S E M
flags
1 8 23
2
n
![Page 64: Memory Hierarchy Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08 with slides by CMU15-213](https://reader036.vdocuments.net/reader036/viewer/2022062600/5a4d1baa7f8b9ab0599ca727/html5/thumbnails/64.jpg)
Concluding observations• Programmer can optimize for cache
performance– How data structures are organized– How data are accessed
• Nested loop structure• Blocking is a general technique
• All systems favor “cache friendly code”– Getting absolute optimum performance is very
platform specific• Cache sizes, line sizes, associativities, etc.
– Can get most of the advantage with generic code• Keep working set reasonably small (temporal locality)• Use small strides (spatial locality)