igpu : exception support and speculative execution on gpus

1

Department of Computer Science

iGPU: Exception Support and Speculative Execution on GPUs

Jaikrishnan Menon, Marc de Kruijf Karthikeyan Sankaralingam

Vertical Research GroupUniversity of Wisconsin−Madison

Presented at ISCA 2012


2

Executive Summary Compiler/hardware co-design for efficient, general-

purpose GPUs

Exception support with 1.5% overhead (no more than 4%)

Demand paging support with 2.5% overhead Context switch (no more than 4%)

Exploiting speculation provides > 10% energy savings


3

Outline Motivation and Background iGPU Mechanisms

General exception handling Context switching Speculation support

iGPU Architecture Software Hardware

Evaluation Conclusion


4

CPU Evolution Retrospective IBM 360 era – precise exceptions as a performance

tradeoff

However, two key shifts in processor design – Virtual memory no longer optional Speculative execution on ILP processors


5

Precise exception handling and speculation was a key enabler for modern CPUs


6

GPU Architectural trends

Significant interest in supporting demand paging Emerging necessity for supporting speculation

More workloads – “irregular” workloads Handling reliability problems

A single unified CPU-GPU address space


7

Need general purpose exception and speculation support for GPUs


8

Why not just borrow CPU ideas? CPUs use buffering to preserve arch. state

Future file, History file, Re-order Buffer …

But GPUs have 1000x as many registers Not practical!


9

Fundamental Challenges1. Well defined restart point in program

GPU pipeline and SIMT model make this hard

2. Preserving architecture state prior to restart Need to save 1000s of registers


10

Key Ideas of our Solution1. Well defined restart point in program

Idempotent code regions Restartable regions

producing same effect

2. Preserving architecture state prior to restart Regions constructed with

small live state: 1 to 3 regs Save only this live state

Creation of restart points

Preservation of necessary state


11

Outline Challenges and Implications iGPU Mechanisms





12

Exception Support Idempotent regions mark restart points Register file provides all the reqd. state! Idempotence guarantees correctness

Implicit checkpoints using idempotence

A BException handler

B

Creation idea


13




Evaluation


14

Context Switch

A B

Exception is page fault

1. Cleanly remove process 1 ?2. Start another process and execute 3. Get page from disk concurrently 4. Restore process 1 ?5. Restart process 1

?

Page-fault handling

B?


15

Context Switch

A B

Exception is page fault

1. Cleanly remove process 1 ?2. Start another process and execute 3. Get page from disk concurrently 4. Restore process 1 ?5. Restart process 1

?

Page-fault handling

B?


Context Switch Must save and restore architectural state

But...GPUs have megabytes of register state Save only live state

Save only live state at points of minimal live state


Context Switch Must save and restore architecture state

But...GPUs have megabytes of register state Save only live state

Save state at points of minimal live state

17

Implicit minimum live state checkpoints using idempotence

A B B

# live registers

23

Candidate cut point

942

B

# live registers

2

Exception handler

Preserve idea


18






19

Speculation Speculation generates state that is wrong

Need even more buffers Recall: buffers are impractical for GPUs

Use idempotence! Reduce re-execution cost by sub-dividing

regions

Implicit checkpoints with low re-execution overhead using idempotence

Tuning the Creation idea


20

Speculation

A B

# live registers: 2

* Region construction details: Idempotent Processing, PLDI ‘12

B1 B2 BB2 CC

Misspeculation


21






22

iGPU Architecture

Compiler

Hardware

Application


23

iGPU Architecture - Software Form regions Preserve state

Creation idea

Preserve idea

state preservation

register re-assignment, moves and

spills

region formation

region marker instructions

Reg. pressure


24

iGPU Architecture - Software

Source Code Compiler

Device Code Generator

Device Code

Kernel Source Code


25




Idempotent Device Code

Kernel Source Code

Region formation


26




Idempotent Device Code

Kernel Source Code

Region formation

State preservation


27

iGPU Architecture - Hardware

…

L2 Cache SIMDProcessor

L1 cache & TLB

General PurposeRegisters

Core

Core Core

Core

Fetch Unit

……

Decode

RPCs

(not to scale)

Creation idea


28


General Purpose Registers

Restart PCRegister

(to scale)

2 RPCs per warp - one each for Sparse and Short regions

Compare to 1024 GPRs per warp

(32 x 32)


29


State preservation handled purely by compiler!Not hardware’s responsibility

Preserve idea


30






31

Evaluation

AES BFS BSC CP DG LIB LPS MUM NN NQU RAY STO WPGeoMean0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

iGPU Runtime Overhead

Region Creation Context Switch and Speculation support overhead

% O

verh

ead


32

Evaluation – Voltage Speculation

AES BFS BSC CP DG LIB LPS MUM NN NQU RAY STO WP GeoMean0

2

4

6

8

10

12

14

16

18

20

Energy Savings on iGPU with Voltage Emergency Prediction

% e

nerg

y sa

ving

s

Vdd reduction : 10% Error rate : 0.01%


33






34

Executive Summary Compiler/hardware co-design for efficient, general-

purpose GPUs

Exception support with 1.5% overhead (no more than 4%)

Demand paging support with 2.5% overhead Context switch (no more than 4%)

Exploiting speculation provides > 10% energy savings


35

Conclusions Exception support for GPUs is practical

Enables better integration with CPUs in CPU-GPU architectures

Speculative execution on GPUs Both for performance and reliability presents interesting possibilities in the context of

“irregular” workloads


36

Questions

igpu : exception support and speculative execution on gpus

Documents

restart process

restart pointsregister

small live state

cpu ideas

history file

statefuture file

mand paging support

page faultcleanly