Combining HWSW Mechanisms to ImproveNUMA Performance of Multi-GPU SystemsVinson Youngdagger Dagger Aamer JaleelDagger Evgeny BolotinDagger Eiman EbrahimiDagger David NellansDagger Oreste VillaDagger

daggerGeorgia Institute of Technology DaggerNVIDIAvyounggatecheduajaleelebolotineebrahimidnellansovillanvidiacom

AbstractmdashHistorically improvement in GPU performance hasbeen tightly coupled with transistor scaling As Moorersquos Lawslows down performance of single GPUs may ultimately plateauTo continue GPU performance scaling multiple GPUs can beconnected using system-level interconnects However limitedinter-GPU interconnect bandwidth (eg 64GBs) can hurt multi-GPU performance when there are frequent remote GPU memoryaccesses Traditional GPUs rely on page migration to service thememory accesses from local memory instead Page migrationfails when the page is simultaneously shared between multipleGPUs in the system As such recent proposals enhance thesoftware runtime system to replicate read-only shared pages inlocal memory Unfortunately such practice fails when there arefrequent remote memory accesses to read-write shared pages Toaddress this problem recent proposals cache remote shared datain the GPU last-level-cache (LLC) Unfortunately remote datacaching also fails when the shared-data working-set exceeds theavailable GPU LLC size

This paper conducts a combined performance analysis of state-of-the-art software and hardware mechanisms to improve NUMAperformance of multi-GPU systems Our evaluations on a 4-node multi-GPU system reveal that the combination of workscheduling page placement page migration page replication andcaching remote data still incurs a 47 slowdown relative to anideal NUMA-GPU system This is because the shared memoryfootprint tends to be significantly larger than the GPU LLCsize and can not be replicated by software because the sharedfootprint has read-write property Thus we show that existingNUMA-aware software solutions require hardware support toaddress the NUMA bandwidth bottleneck We propose CachingRemote Data in Video Memory (CARVE) a hardware mechanismthat stores recently accessed remote shared data in a dedicatedregion of the GPU memory CARVE outperforms state-of-the-art NUMA mechanisms and is within 6 the performanceof an ideal NUMA-GPU system A design space analysis onsupporting cache coherence is also investigated Overall we showthat dedicating only 3 of GPU memory eliminates NUMAbandwidth bottlenecks while incurring negligible performanceoverheads due to the reduced GPU memory capacity

Index TermsmdashGPU Multi-GPU Memory NUMA HBMDRAM-Cache Coherence Page-Migration Page-Replication

I INTRODUCTION

GPU acceleration has improved the performance of HPCsystems [1] [2] [3] [4] and deep learning applications [5][6] [7] Historically transistor scaling has improved singleGPU application performance by increasing the number ofStreaming Multiprocessors (SM) between GPU generationsHowever the end of Moorersquos Law [8] necessitates alternativemechanisms such as multi-GPUs to continue scaling GPUperformance independent of the technology node

Fig 1 Multi-GPU System Low inter-GPU link bandwidth creates Non-Uniform Memory Access (NUMA) bottlenecks

Multi-GPU systems can speed up application performanceby offering 4-8x more computational resources than a single-GPU [9] Figure 1 illustrates a commercially available multi-GPU system (eg DGX [9] [10]) consisting of four GPUseach with their own dedicated path to high-bandwidth localGPU memory (eg 1 TBs [11]) and interconnected usinginter-GPU links (eg NVLink [12]) However due to physicalconstraints these inter-GPU links are likely to have 10-20xlower bandwidth than local GPU memory bandwidth (eg64 GBs compared to 1 TBs) Consequently while multi-GPUs have the potential to provide substantially more computeresources the large bandwidth discrepancy between localmemory bandwidth and inter-GPU bandwidth contributes toNon-Uniform Memory Access (NUMA) behavior that can oftenbottleneck performance

The NUMA bandwidth bottleneck is because the localmemory is unable to satisfy the majority of memory requestsTo address this problem runtime systems can migrate pagesto the local memory Unfortunately such practice fails when apage is concurrently accessed by multiple nodes in the system(ie shared pages) In such situations recent proposals enhancethe runtime system to replicate shared pages [13] [14] in thelocal memory of each node However unbounded replicationcan significantly increase GPU memory capacity pressure (onaverage 24x in our studies) Since GPUs have limited memorycapacity due to memory technology and cost constraints [15]we desire an alternate solution to reduce NUMA bottlenecks

Recent work investigates software and hardware enhance-ments to reduce NUMA performance bottlenecks Specifically

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

e

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 2 Performance of NUMA-GPU relative to an ideal paging mechanism that replicates ALL shared pages There is still a significant performance gap dueto limited inter-GPU link bandwidth

the authors propose work scheduling first-touch page mappingand caching shared pages in the GPU cache hierarchy [16] Werefer to this baseline system as NUMA-GPU Figure 2 comparesthe performance of NUMA-GPU NUMA-GPU enhanced withreplicating read-only shared pages [13] [14] and an ideal upper-bound NUMA-GPU system that replicates all (both read-onlyand read-write) shared pages The x-axis shows the differentworkloads while the y-axis illustrates performance relativeto the ideal NUMA-GPU system that replicates all sharedpages The figure shows that across the 20 workloads studiedeight workloads experience negligible NUMA performancebottlenecks We also see that replicating read-only shared pagesremoves NUMA performance bottlenecks for three workloadsFor the remainder of the workloads the baseline NUMA-GPU system experiences 20-80 slowdown that can only beeliminated by replicating read-write shared pages as well

Read-write pages manifest due to normal program semanticsor due to false-sharing when using large pages [17]1 Whileread-write shared pages can be replicated they must becollapsed on writes to ensure data coherence Unfortunatelythe software overhead of collapsing read-write shared pages(even on occasional writes) can be prohibitively high [13]Consequently developing efficient software techniques toreduce NUMA performance bottlenecks associated with read-write pages is still an open problem

Hardware mechanisms that cache shared pages in the GPUcache hierarchy can also reduce the NUMA performancebottleneck [16] However conventional GPU LLC sizes areunable to capture the large shared application working-set sizeof emerging workloads To address the cache capacity problemrecent papers [19] [20] propose using DRAM caches in multi-node CPU systems As such we also investigate the feasibilityof architecting DRAM caches in multi-node GPU systems

This paper investigates the limitations of combining state-of-the-art software and hardware techniques to address the NUMAperformance bottleneck in multi-GPU systems Overall thispaper makes the following contributions

1) We show that software paging mechanisms commonlyused on GPUs today fail to reduce the multi-GPU NUMAperformance bottleneck Furthermore we also show thatcaching remote data in the GPU cache hierarchy haslimited benefits due to the large shared data footprintThus we show that GPUs must be augmented with large

1GPUs rely on large pages (eg 2MB) to ensure high TLB coverage Reducingthe paging granularity to minimize false read-write sharing can cause severeTLB performance bottlenecks [18]

caches to reduce NUMA overheads To the best of ourknowledge this is the first study that combines state-of-the-art software and hardware techniques (to improve NUMAperformance bottlenecks) on a single NUMA platform(ie a multi-GPU platform in our study)

2) We propose to augment NUMA-GPU with a practicalDRAM cache architecture that increases GPU cachingcapacity by Caching Remote Data in Video Memory(CARVE) CARVE dedicates a small fraction of the GPUmemory to store the contents of remote memory In doingso CARVE transforms the GPU memory into a hybridstructure that is simultaneously configured as OS-visiblememory and a large cache CARVE only caches remotedata in the DRAM cache since there is no latency orbandwidth benefit from caching local data

3) We perform a detailed design space analysis on theimplications of DRAM cache coherence in multi-GPUsystems Specifically we find that conventional softwarecoherence mechanisms used in GPUs today do not scaleto giga-scale DRAM caches This is because softwarecoherence frequently destroys the data locality benefitsfrom DRAM caches Instead we show that GPUs must beextended with a simple hardware coherence mechanismto reap DRAM cache benefits

4) Finally we show that the small loss in GPU memorycapacity due to CARVE can be compensated by allocatingpages in system CPU memory In such situations the useof Unified Memory (UM) enables frequently accessedpages to be migrated between system memory and GPUmemory Thus the performance impact of losing someGPU memory capacity can be tolerable even in situationswhere CARVE provides limited performance benefits

Overall this paper provides an important foundation for furtherreducing NUMA bottlenecks on emerging multi-GPU platformsWe perform detailed performance evaluations with state-of-the-art work scheduling software paging hardware caching andCARVE on a multi-GPU system consisting of four GPUs Weshow that page migration page replication and CARVE incura 49 47 and 6 respective slowdown relative to an idealNUMA-GPU system that satisfies all memory requests locallyThese results show that CARVE is imperative to continuescaling NUMA-GPU performance CARVE provides thesebenefits by incurring only a 3 loss in GPU memory capacityFinally we show that the loss in GPU memory capacity incursnegligible performance overheads

2

TABLE ICHARACTERISTICS OF RECENT NVIDIA GPUS

Fermi Kepler Maxwell Pascal VoltaSMs 16 15 24 56 80BW (GBs) 177 288 288 720 900Transistors (B) 30 71 80 153 211Tech node (nm) 40 28 28 16 12Chip size (mm2) 529 551 601 610 815

II BACKGROUND AND MOTIVATION

A GPU Performance ScalingTable I shows that GPU performance has scaled well overthe past decade due to increasing transistor density and GPUmemory bandwidth For example the recently announced VoltaGPU consists of 80 Streaming Multiprocessors (SMs) on alarge 815 mm2 chip consists of 211 billion transistors [2]Unfortunately the impending end of Moorersquos Law [8] andlimitations in lithography and manufacturing cost [21] threatencontinued GPU performance scaling Without larger or denserdies GPU architects must now investigate alternative techniquesfor GPU performance scaling

Recent advances in packaging [22] signaling [23] [24] andinterconnection technology [25] [26] enable new opportunitiesfor scaling GPU performance For example recent workpropose interconnecting multiple GPUs at a package-level [27]or system-level [16] with high bandwidth interconnectiontechnology (eg GRS [26] or NVLink [12]) These multi-GPU systems provide the programmer the illusion of a singleGPU system However they are non-uniform memory access(NUMA) systems with asymmetric bandwidth to local andremote GPU memory As such software and hardware tech-niques have been proposed to reduce the NUMA bottlenecksWe now discuss these techniques and identify their limitations

B NUMA-aware Multi-GPUsFigure 3 illustrates a NUMA-GPU system where performance isdirectly dependent on the available remote memory bandwidthTraditionally application programmers have manually managedthe placement of both compute and data to ensure high NUMAsystem performance Such practice significantly increasesprogrammer burden and may not necessarily scale to alternateNUMA systems To reduce programmer burden the recentNUMA-GPU [16] proposal presents software and hardwaretechniques to reduce NUMA bottlenecks without the need forprogrammer intervention

NUMA-GPU enhances the GPU runtime by improvingthe Cooperative Thread Array (CTA) scheduling and pageplacement policy Since adjacent CTAs tend to have significantspatial and temporal locality NUMA-GPU exploits inter-CTAdata locality by scheduling a large batch of contiguous CTAs toeach GPU Furthermore NUMA-GPU increases the likelihoodof servicing the majority of memory requests from the localGPU memory by using a First-Touch (FT) page placementpolicy The FT policy allocates pages to the memory of theGPU that first accessed that page If a page were private FTmapping ensures that memory accesses are serviced locally

If a page were already mapped in a remote GPU memory(eg shared data) NUMA-GPU caches remote data in the GPU

Fig 3 Organization and bandwidth of a NUMA-Aware Multi-GPU DistributedCTA-Scheduling and First-Touch Memory Mapping can improve data locality

cache hierarchy to enable future references to be serviced athigh bandwidth To ensure correctness NUMA-GPU extendsexisting software coherence mechanisms to the GPU LLC [28][29] Figure 3 illustrates the software and hardware mechanismsused by NUMA-GPU to reduce NUMA bottlenecks

C NUMA-GPU Performance Bottleneck

NUMA systems experience significant performance bottleneckswhen applications frequently access remote memory A recentstudy showed that remote memory accesses in NUMA systemsis because of false page sharing when using large page sizes(eg to improve TLB coverage [17] [18]) To address thefalse sharing problem the runtime system (or OS) can replicateshared pages locally to avoid remote memory accesses Unfor-tunately page replication increases the application memoryrequirements (on average 24x in our study) Thus pagereplication works best only when there are free memory pagesavailable Since future GPU workloads are expected to entirelyfill GPU memory capacity [15] page replication is expectedto benefit only if the GPU memory is under-utilized

Besides GPU memory capacity pressure page replication isonly limited to read-only shared pages and not read-write sharedpages This is because the software overhead of collapsingread-write shared pages (even on occasional writes) can beextremely expensive [13] Thus page-replication cannot reduceremote memory accesses when a large fraction of remotememory accesses are to read-write shared pages To illustratethis problem on the baseline NUMA-GPU system Figure 4shows the distribution of GPU memory accesses to privatepages read-only shared pages and read-write shared pages

0

20

40

60

80

100

M

em

ory

Ac

ce

ss

es

Private Read-Shared Read-Write-Shared

Cacheline (128B) OS Page (2MB)

Fig 4 Distribution of memory accesses to private and shared pages at pagesize and cache line granularity

3

05

2

8

32

128

512

2048

8192

32768

Me

mo

ry S

ize (

MB

) Aggregate System LLC Capacity

Fig 5 Total memory capacity (across 4-gpus) needed to cover applicationshared working set size on a NUMA-GPU

The figure shows that 40 (up to 100) of GPU memoryaccesses are serviced by read-write pages Consequently pagereplication can not reduce NUMA bottlenecks when read-writeshared pages are frequently accessed

To address the limitations of replicating read-write sharedpages NUMA performance can be improved by caching shareddata in the on-chip last-level-cache (LLC) [16] [30] Toquantify the total LLC space requirements Figure 5 showsthe shared memory footprint for the different workloads Theshared memory footprint represents the total number of uniqueremote pages fetched by the different GPUs The figure clearlyshows that caching remote data in the GPU LLC is unlikelyto benefit since the application shared memory footprint oftensignificantly exceeds the aggregate system LLC capacity

A straightforward mechanism to fully cover the shared mem-ory footprint would be to increase the total LLC capacity in thesystem However Figure 5 shows that the system LLC capacitymust be drastically increased to fit the shared memory footprintof most workloads Unfortunately semiconductor scaling doesnot allow for such large LLCs to be incorporated on to high-performance processor chips Consequently alternative lowoverhead mechanisms are needed to increase the remote datacaching capacity of GPUs

We investigate increasing the GPU caching capacity byCaching Remote Data in Video Memory (CARVE) a hardwaremechanism that statically devotes a small fraction of the localGPU memory to replicate remote shared data CARVE reducesthe granularity of data replication from OS page granularity(ie 2MB) to a finer granularity (ie 128B cacheline) Toensure correct program execution however CARVE requiresan efficient mechanism to maintain coherence between multipleGPUs Thankfully Figure 4 shows that coherence bandwidthis expected to be small because the fine-grain replicationgranularity results in a smaller distribution of memory accessesto read-write shared data (further emphasizing the high degreeof false-sharing when using large page sizes) Before weinvestigate increasing GPU caching capacity we first discussour experimental methodology

III METHODOLOGY

We use an industry proprietary trace-driven performancesimulator to simulate a multi-GPU system with four GPUs inter-connected using high bandwidth links Each GPU in the systemhas a processor and memory hierarchy similar to the NVIDIAPascal GPU [10] We model 64 SMs per GPU that support64 warps each A warp scheduler selects warp instructions

TABLE IIWORKLOAD CHARACTERISTICS

Suite Benchmark Abbr Mem footprint

HPC

AMG 32 AMG 32 GBHPGMG-UVM HPGMG 20 GB

HPGMG-amry-proxy HPGMG-amry 77 GBLulesh-Unstruct-Mesh1 Lulesh 24 MB

Lulesh-s190 Lulesh-s190 37 GBCoMD-xyz64 warp CoMD 910 MBMCB-5M-particles MCB 254 MBMiniAMR-15Kv40 MiniAMR 44 GB

Nekbone-18 Nekbone 10 GBXSBench 17K grid XSBench 44 GB

Euler3D Euler 26 MBSSSP SSSP 42 MB

bfs-road-usa bfs-road 590 MB

MLAlexNet-ConvNet2 AlexNet 96 MB

GoogLeNet-cudnn-Lev2 GoogLeNet 12 GBOverFeat-cudnn-Lev3 OverFeat 88 MB

Other

Bitcoin-Crypto Bitcoin 56 GBOptix-Raytracing Raytracing 150 MB

stream-triad stream-triad 30 GBRandom Memory Access RandAccess 150 GB

each cycle The GPU memory system consists of a multi-levelcache and TLB hierarchy The first level of the cache and TLBhierarchy are private to each SM while the last level cacheand TLB are shared by all SMs We assume software-basedcache coherence across the private caches that is commonlyused in state-of-the-art GPUs [10] today Table III shows thesimulation parameters used in our baseline

We assume our multi-GPU system is connected usingNVLink [12] technology with 64GBs of bandwidth (in one-direction) We also assume each GPU is interconnected to aCPU using NVLink at 32GBs Our infrastructure also includesa detailed DRAM system with 32GB of memory capacity perGPU with 1TBs of local GPU memory bandwidth Each GPUmemory controller supports 128-entry read and write queuesper channel open-page policy minimalist address mappingpolicy [31] and FR-FCFS scheduling policy (prioritizing readsover writes) Writes are issued in batches when the write queuestarts to fill up

We evaluate 20 CUDA benchmarks taken from many appli-cations of interest HPC applications [32] fluid dynamics [33]graph search [34] machine learning deep neural networks [4]and medical imaging We show the memory footprint of ourworkloads in Table II We simulate all of our workloads forfour billion warp-instructions

TABLE IIIBASELINE MULTI-GPU SYSTEM

Number of GPUs 4Total Number of SMs 256Max number of warps 64 per SM

GPU frequency 1GHzOS Page Size 2MBL1 data cache 128KB per SM 128B lines 4 waysTotal L2 cache 32MB 128B lines 16 ways

Inter-GPU interconnect 64 GBs per link uni-directionalCPU-GPU interconnect 32 GBs per GPUTotal DRAM bandwidth 4 TBsTotal DRAM capacity 128GB

4

Fig 6 Architecting a small fraction of GPU memory as a Remote DataCache (RDC) enables caching large remote data working-sets

IV INCREASING GPU CACHE CAPACITY

The NUMA bottleneck occurs when a large fraction of memoryaccesses are serviced by remote memory over low-bandwidthinterconnection links [12] [35] [36] While NUMA has beenwell studied using conventional memory technology [14] [30][37] emerging high bandwidth memory technology enables anovel solution space to tackle the NUMA problem As such weinvestigate hardware mechanisms that dedicate a small fractionof local memory to store the contents of remote memory Doingso enables the remote memory data to be serviced at high localmemory bandwidth In the context of GPUs this approacheffectively increases the caching capacity of a GPU

A Caching Remote Data in Video Memory

We propose Caching Remote Data in Video Memory (CARVE)a hardware mechanism that statically reserves a portion of theGPUs on-package local memory to store remote data Sincethere is no latency or bandwidth advantage to duplicating localmemory data CARVE only stores remote data in the carve-outWe refer to this memory region as a Remote Data Cache (RDC)(see Figure 6) To avoid software overheads the RDC is entirelyhardware-managed and invisible to the programmer and GPUruntime system The amount of GPU memory dedicated forthe RDC can be statically set at system boot time (or kernellaunch time if only a single kernel is executing on the GPU)Thus CARVE transforms the local GPU memory into a hybridstructure that is simultaneously configured both as softwarevisible memory and a hardware-managed cache

CARVE requires minimal changes to the existing GPUdesign On a GPU LLC miss the GPU memory controllerdetermines whether the missing memory address will beserviced by the local GPU memory or a remote GPU memory Ifthe request maps to the local GPU memory the data is fetcheddirectly from local memory and sent to the LLC Otherwisethe memory controller first checks to see if the data is availablein the RDC In the event of a RDC hit the data is servicedlocally without incurring the bandwidth and latency penaltyof a remote GPU memory access However in the event ofan RDC miss the missing data is retrieved from the remote

DRAM ARRAY

2 KB Row Buffer (plus ECC) =

144B = 128-byte Data + 9-byte ECC + 7-byte (Tag+Metadata)

ADDR

DATA(128B)

TAG AND DATA (TAD)

ECC(9B) + TAG(7B) = (16B)

ROW BUFFER

16 x 144 byte TAD = 16 lines

Fig 7 Alloy Cache stores tags with data to enable low latency access Tags canbe stored alongside ECC bits to simplify data alignment and cache controller

GPU memory and returned to the LLC The missing data isalso inserted into the RDC to enable future hits2

Remote Data Cache (RDC) Design The RDC is architectedat a fine (eg 128B) granularity to minimize false sharing (seeFigure 4) In our study we model the RDC as a fine grainAlloy cache [39] (see Figure 7) which organizes its cache asa direct-mapped structure and stores Tags-with-Data In Alloyone access to the cache retrieves both the tag and the dataIf the tag matches (ie a cache hit) the data is then usedto service the request Otherwise the request is forwarded tothe next level of memory We implement Alloy by storing thetag in spare ECC bits3 similar to commercial implementationtoday [40] Note that RDC is not only limited to the Alloydesign and can also be architected using alternate DRAM cachearchitectures [39] [41] [42]

When GPU memory is configured with an RDC CARVErequires a single register in the GPU memory controller tospecify starting location (ie physical address) for the RDCCARVE also requires simple combinational logic to identify thephysical GPU memory location for a given RDC set Finallysince the RDC is architected in local GPU memory the differentRDC sets are interleaved across all GPU memory channels toensure high bandwidth concurrent RDC accesses

Remote Data Cache Evaluation We evaluate the benefits ofCARVE by using a 2GB RDC This amounts to 625 of thebaseline 32GB GPU memory dedicated as an RDC and the

2The RDC can also store data from System Memory However this requiresadditional support for handling CPU-GPU coherence [38]

3We assume GPU memory uses HBM technology HBM provides 16B of ECCto protect 128B of data Assuming ECC at a 16-byte granularity (enoughto protect bus transfers) SECDED requires 89=72 bits to protect the dataThis leaves 56 spare ECC bits that can be used for tag and metadata (RDConly requires 6 bits for tags)

0

01

02

03

04

05

06

07

08

R

em

ote

Ac

ce

ss

es NUMA-GPU

CARVE

Fig 8 Fraction of remote memory accesses CARVE reduces the averagefraction of remote memory accesses from 40 in NUMA-GPU to 8

5

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

eNUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE (No Coherence) NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 9 CARVE performance with zero overhead cache coherence CARVE is able to achieve Ideal-NUMA-GPU speedup

remaining 9375 exposed as operating system (OS) managedmemory (30GB per GPU)

RDC Effect on NUMA Traffic Figure 8 shows the fractionof remote memory accesses with NUMA-GPU and NUMA-GPU enhanced with CARVE The figure shows that manyworkloads on the baseline NUMA-GPU system still suffermore than 70 remote memory accesses (eg XSBench andLulesh) This is because the shared memory footprint of theseworkloads is larger than the GPU LLC CARVE on the otherhand significantly reduces the fraction of remote memoryaccesses On average NUMA-GPU experiences 40 remotememory accesses while CARVE experiences only 8 remotememory accesses In doing so CARVE efficiently utilizeslocal unused HBM bandwidth Consequently the figure showsthat CARVE can significantly reduce the inter-GPU bandwidthbottleneck for most workloads

RDC Performance Analysis (Upper Bound) We first eval-uate the performance potential of CARVE by assuming zero-overhead cache coherence between the RDCs (called CARVE-No-Coherence) This enables us to quantify the benefits ofservicing remote GPU memory accesses in local GPU memoryWe investigate coherence in the next subsection

Figure 9 shows the performance of NUMA-GPU NUMA-GPU with software page replication of read-only shared pagesCARVE-No-Coherence and an ideal NUMA-GPU system thatreplicates all shared pages The x-axis shows the differentworkloads and the y-axis shows the performance relativeto the ideal NUMA-GPU system The figure shows thatCARVE enables workloads like Lulesh Euler SSSP andHPGMG (which experience significant slowdown with NUMA-GPU and NUMA-GPU enhanced with read-only shared pagereplication) to approach the performance of an ideal NUMA-GPU system This is because these workloads experience a20-60 reduction in remote memory accesses (see Figure 8)This shows that extending the remote data caching capacityinto the local GPU memory by using a Remote Data Cache(RDC) significantly reduces the NUMA bandwidth bottleneckCARVE provides these performance improvements withoutrelying on any software support for page replication

While CARVE improves performance significantly acrossmany workloads CARVE can sometimes degrade performanceFor example RandAccess experiences a 10 performancedegradation due to frequent misses in the RDC The ad-

ditional latency penalty of first accessing the RDC thenaccessing remote memory can degrade performance of latency-sensitive workloads However using low-overhead cachehit-predictors [39] can mitigate these performance outliersOverall Figure 9 shows that CARVE significantly bridgesthe performance gap between NUMA-GPU and an idealNUMA-GPU system On average the baseline NUMA-GPUsystem and NUMA-GPU enhanced with read-only shared pagereplication experience a 50 performance gap relative to theideal NUMA-GPU system On the other hand CARVE-No-Coherence experiences only a 5 performance gap relative toan ideal NUMA-GPU system that replicates all shared pagesThese results show significant performance opportunity fromCARVE provided we can efficiently ensure data coherencebetween the RDCs

B Coherence Implications of an RDC in GPUs

CARVE has the potential to significantly improve NUMA-GPUperformance by reducing most of the remote memory accessesHowever since each Remote Data Cache (RDC) stores a copyof remote GPU memory an efficient mechanism is necessary tokeep the copies coherent Fortunately the GPU programmingmodel provides the programmer with an API to explicitlyinsert synchronization points to ensure coherent data accessThus since software coherence [28] is already supported onconventional GPU systems we first investigate whether CARVEcan be easily extended with software coherence

Software CoherenceConventional GPU designs maintain software coherence atkernel boundaries by enforcing two requirements First at theend of each kernel call modified data is written back to memory(ie flush dirty data) Second at the beginning of the nextkernel invocation the GPU cores must read the updated valuesfrom memory Conventional GPUs enforce these requirementsby implementing a write-through L1 cache that is invalidatedat every kernel boundary and a memory-side last-level cachethat is implicitly coherent

TABLE IVKERNEL-LAUNCH DELAY UNDER SOFTWARE COHERENCE

L2 Cache (8MB) RDC (2GB)Cache Invalidate 8MB 16 bank 1cycle 2GB 1024GBs local

4us 2ms =gt0msFlush Dirty 8MB 1024-64GBs 2GB 64GBs remote

8us˜128us 32ms =gt0ms

6

Fig 10 Epoch-counter based invalidation eliminates the penalty of explicitlyinvalidating the whole remote data cache

NUMA-GPUs cache remote data in the GPU LLC and extendsoftware coherence to the GPU LLC by also invalidating theLLC on kernel boundaries [16] Similarly RDC coherence canbe ensured by further extending software-coherence mecha-nisms to the RDC Thus on every kernel boundary the memorycontroller would also invalidate all RDC entries and write backdirty data (if any) to remote GPU memory Table IV comparesthe worst case software coherence overhead for RDC and on-chip LLCs The table shows that invalidating and flushing dirtydata in on-chip LLCs tends to be on the order of microsecondsand can be tolerated within kernel launch latency On theother hand the overhead of invalidating and writing backmillions of RDC lines can take on the order of millisecondsIncurring millisecond latency at every kernel boundary cansignificantly impact application performance Consequentlywe desire architecture support for efficiently invalidating andwriting back RDC lines

Efficient RDC Invalidation Since the RDC tag valid anddirty bits are all stored in GPU memory invalidating all RDClines requires reading and writing GPU memory Dependingon the RDC size RDC invalidation can be a long latency andbandwidth intensive process Alternatively we do not needto physically invalidate all of the lines We simply need toknow if the RDC data is stale (ie from a previous kernel andshould be re-fetched) or up-to-date (ie from current kerneland can be used) If we track the epoch an RDC cacheline wasinstalled in we can easily determine if the RDC data is staleor not Thus an epoch-counter-based invalidation scheme isused to invalidate the RDC instantly (ie 0 ms) [43] [44]

Figure 10 shows our RDC invalidation mechanism We usea 20-bit register to maintain an Epoch Counter (EPCTR) foreach kernelstream running on the GPU RDC insertions storethe current EPCTR value of the kernelstream with each RDCcacheline (in the ECC bits along with the RDC Tag) Bydoing so RDC hits only occur if the tag matches and if theEPCTR stored within the RDC cacheline matches the currentEPCTR value of the associated kernelstream To supportefficient RDC invalidation at kernel boundaries we simplyincrement the EPCTR of the appropriate kernel Thus theRDC hit logic ensures that the newly launched kernel does notconsume a previous kernelsrsquo stale data In the rare event thatthe EPCTR rolls over on an increment the memory controllerphysically resets all RDC cachelines by setting the valid bitand EPCTR to zero Note that an RDC architected in dense

DRAM technology enables large counters per RDC line thatotherwise were impractical with similar mechanisms appliedto on-chip caches [43] [44]

Efficient RDC Dirty Data Flush We also desire an efficientmechanism to flush dirty lines to remote GPU memory at kernelboundaries For a writeback RDC we would need to knowwhich RDC lines must be flushed Since our RDC invalidationmechanism no longer requires physically reading all RDC linesan alternate mechanism is necessary to locate dirty RDC linesWe can use a dirty-map [45] which tracks RDC regions thathave been written to On a kernel boundary the dirty-map canbe used to determine which RDC regions should be flushed toremote GPU memory To avoid the on-chip storage overheadfor the dirty-map we propose a write-through RDC insteadDoing so ensures that dirty data is immediately propagated toremote GPU memory Note that the write bandwidth to remoteGPU memory with a write-through RDC is identical to thebaseline NUMA-GPU configuration with no RDC

Our evaluations with writeback (with dirty-map) and write-through RDC showed that a write-through RDC performs nearlyas well (within 1 performance) as a write-back RDC Thisis because the vast majority of remote data cached in the linegranularity RDC tends to be heavily read-biased (see Figure 4)Consequently for the remainder of this work we assume awrite-through RDC to enable zero-latency dirty data flush andavoid the complexity of maintaining a write-back RDC

Software Coherence Results We now evaluate performanceof extending software coherence to the RDC We refer to thisdesign as CARVE with Software Coherence (CARVE-SWC)Figure 11 compares the performance of CARVE-SWC toCARVE with no coherence overhead (CARVE-No-Coherence)The figure shows that CARVE-SWC enables XSBench toperform similar to CARVE-No-Coherence However despiteeliminating all overheads to maintain software coherence forlarge caches CARVE-SWC removes all performance benefitsof RDC for the remaining workloads Since the only differencebetween CARVE-No-Coherence and CARVE-SWC is theflushing of the RDC between kernels these results suggestsignificant inter-kernel data locality that is exploited by CARVE-No-Coherence While this motivates further research work onimproving CARVE-SWC by efficiently prefetching remotedata at kernel boundaries this is out of the scope of this paperInstead we now investigate hardware coherence mechanismsthat do not require flushing the RDC at kernel boundaries

Hardware CoherenceCARVE-SWC results reveal that it is important to retainRDC data across kernel boundaries While software coherencecan potentially be relaxed to avoid flushing the RDC itwould require additional complexity for maintaining a softwaredirectory to track stale copies of shared data cached remotelyand invalidate them explicitly Since similar functionality isavailable in conventional hardware coherence mechanisms weinvestigate extending the RDC with hardware coherence instead

7

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

e

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 2 Performance of NUMA-GPU relative to an ideal paging mechanism that replicates ALL shared pages There is still a significant performance gap dueto limited inter-GPU link bandwidth

the authors propose work scheduling first-touch page mappingand caching shared pages in the GPU cache hierarchy [16] Werefer to this baseline system as NUMA-GPU Figure 2 comparesthe performance of NUMA-GPU NUMA-GPU enhanced withreplicating read-only shared pages [13] [14] and an ideal upper-bound NUMA-GPU system that replicates all (both read-onlyand read-write) shared pages The x-axis shows the differentworkloads while the y-axis illustrates performance relativeto the ideal NUMA-GPU system that replicates all sharedpages The figure shows that across the 20 workloads studiedeight workloads experience negligible NUMA performancebottlenecks We also see that replicating read-only shared pagesremoves NUMA performance bottlenecks for three workloadsFor the remainder of the workloads the baseline NUMA-GPU system experiences 20-80 slowdown that can only beeliminated by replicating read-write shared pages as well

Read-write pages manifest due to normal program semanticsor due to false-sharing when using large pages [17]1 Whileread-write shared pages can be replicated they must becollapsed on writes to ensure data coherence Unfortunatelythe software overhead of collapsing read-write shared pages(even on occasional writes) can be prohibitively high [13]Consequently developing efficient software techniques toreduce NUMA performance bottlenecks associated with read-write pages is still an open problem

Hardware mechanisms that cache shared pages in the GPUcache hierarchy can also reduce the NUMA performancebottleneck [16] However conventional GPU LLC sizes areunable to capture the large shared application working-set sizeof emerging workloads To address the cache capacity problemrecent papers [19] [20] propose using DRAM caches in multi-node CPU systems As such we also investigate the feasibilityof architecting DRAM caches in multi-node GPU systems

This paper investigates the limitations of combining state-of-the-art software and hardware techniques to address the NUMAperformance bottleneck in multi-GPU systems Overall thispaper makes the following contributions

1) We show that software paging mechanisms commonlyused on GPUs today fail to reduce the multi-GPU NUMAperformance bottleneck Furthermore we also show thatcaching remote data in the GPU cache hierarchy haslimited benefits due to the large shared data footprintThus we show that GPUs must be augmented with large

1GPUs rely on large pages (eg 2MB) to ensure high TLB coverage Reducingthe paging granularity to minimize false read-write sharing can cause severeTLB performance bottlenecks [18]

caches to reduce NUMA overheads To the best of ourknowledge this is the first study that combines state-of-the-art software and hardware techniques (to improve NUMAperformance bottlenecks) on a single NUMA platform(ie a multi-GPU platform in our study)

2) We propose to augment NUMA-GPU with a practicalDRAM cache architecture that increases GPU cachingcapacity by Caching Remote Data in Video Memory(CARVE) CARVE dedicates a small fraction of the GPUmemory to store the contents of remote memory In doingso CARVE transforms the GPU memory into a hybridstructure that is simultaneously configured as OS-visiblememory and a large cache CARVE only caches remotedata in the DRAM cache since there is no latency orbandwidth benefit from caching local data

3) We perform a detailed design space analysis on theimplications of DRAM cache coherence in multi-GPUsystems Specifically we find that conventional softwarecoherence mechanisms used in GPUs today do not scaleto giga-scale DRAM caches This is because softwarecoherence frequently destroys the data locality benefitsfrom DRAM caches Instead we show that GPUs must beextended with a simple hardware coherence mechanismto reap DRAM cache benefits

4) Finally we show that the small loss in GPU memorycapacity due to CARVE can be compensated by allocatingpages in system CPU memory In such situations the useof Unified Memory (UM) enables frequently accessedpages to be migrated between system memory and GPUmemory Thus the performance impact of losing someGPU memory capacity can be tolerable even in situationswhere CARVE provides limited performance benefits

Overall this paper provides an important foundation for furtherreducing NUMA bottlenecks on emerging multi-GPU platformsWe perform detailed performance evaluations with state-of-the-art work scheduling software paging hardware caching andCARVE on a multi-GPU system consisting of four GPUs Weshow that page migration page replication and CARVE incura 49 47 and 6 respective slowdown relative to an idealNUMA-GPU system that satisfies all memory requests locallyThese results show that CARVE is imperative to continuescaling NUMA-GPU performance CARVE provides thesebenefits by incurring only a 3 loss in GPU memory capacityFinally we show that the loss in GPU memory capacity incursnegligible performance overheads

2

TABLE ICHARACTERISTICS OF RECENT NVIDIA GPUS

Fermi Kepler Maxwell Pascal VoltaSMs 16 15 24 56 80BW (GBs) 177 288 288 720 900Transistors (B) 30 71 80 153 211Tech node (nm) 40 28 28 16 12Chip size (mm2) 529 551 601 610 815

II BACKGROUND AND MOTIVATION

A GPU Performance ScalingTable I shows that GPU performance has scaled well overthe past decade due to increasing transistor density and GPUmemory bandwidth For example the recently announced VoltaGPU consists of 80 Streaming Multiprocessors (SMs) on alarge 815 mm2 chip consists of 211 billion transistors [2]Unfortunately the impending end of Moorersquos Law [8] andlimitations in lithography and manufacturing cost [21] threatencontinued GPU performance scaling Without larger or denserdies GPU architects must now investigate alternative techniquesfor GPU performance scaling

Recent advances in packaging [22] signaling [23] [24] andinterconnection technology [25] [26] enable new opportunitiesfor scaling GPU performance For example recent workpropose interconnecting multiple GPUs at a package-level [27]or system-level [16] with high bandwidth interconnectiontechnology (eg GRS [26] or NVLink [12]) These multi-GPU systems provide the programmer the illusion of a singleGPU system However they are non-uniform memory access(NUMA) systems with asymmetric bandwidth to local andremote GPU memory As such software and hardware tech-niques have been proposed to reduce the NUMA bottlenecksWe now discuss these techniques and identify their limitations

B NUMA-aware Multi-GPUsFigure 3 illustrates a NUMA-GPU system where performance isdirectly dependent on the available remote memory bandwidthTraditionally application programmers have manually managedthe placement of both compute and data to ensure high NUMAsystem performance Such practice significantly increasesprogrammer burden and may not necessarily scale to alternateNUMA systems To reduce programmer burden the recentNUMA-GPU [16] proposal presents software and hardwaretechniques to reduce NUMA bottlenecks without the need forprogrammer intervention

NUMA-GPU enhances the GPU runtime by improvingthe Cooperative Thread Array (CTA) scheduling and pageplacement policy Since adjacent CTAs tend to have significantspatial and temporal locality NUMA-GPU exploits inter-CTAdata locality by scheduling a large batch of contiguous CTAs toeach GPU Furthermore NUMA-GPU increases the likelihoodof servicing the majority of memory requests from the localGPU memory by using a First-Touch (FT) page placementpolicy The FT policy allocates pages to the memory of theGPU that first accessed that page If a page were private FTmapping ensures that memory accesses are serviced locally

If a page were already mapped in a remote GPU memory(eg shared data) NUMA-GPU caches remote data in the GPU

Fig 3 Organization and bandwidth of a NUMA-Aware Multi-GPU DistributedCTA-Scheduling and First-Touch Memory Mapping can improve data locality

cache hierarchy to enable future references to be serviced athigh bandwidth To ensure correctness NUMA-GPU extendsexisting software coherence mechanisms to the GPU LLC [28][29] Figure 3 illustrates the software and hardware mechanismsused by NUMA-GPU to reduce NUMA bottlenecks

C NUMA-GPU Performance Bottleneck

NUMA systems experience significant performance bottleneckswhen applications frequently access remote memory A recentstudy showed that remote memory accesses in NUMA systemsis because of false page sharing when using large page sizes(eg to improve TLB coverage [17] [18]) To address thefalse sharing problem the runtime system (or OS) can replicateshared pages locally to avoid remote memory accesses Unfor-tunately page replication increases the application memoryrequirements (on average 24x in our study) Thus pagereplication works best only when there are free memory pagesavailable Since future GPU workloads are expected to entirelyfill GPU memory capacity [15] page replication is expectedto benefit only if the GPU memory is under-utilized

Besides GPU memory capacity pressure page replication isonly limited to read-only shared pages and not read-write sharedpages This is because the software overhead of collapsingread-write shared pages (even on occasional writes) can beextremely expensive [13] Thus page-replication cannot reduceremote memory accesses when a large fraction of remotememory accesses are to read-write shared pages To illustratethis problem on the baseline NUMA-GPU system Figure 4shows the distribution of GPU memory accesses to privatepages read-only shared pages and read-write shared pages

0

20

40

60

80

100

M

em

ory

Ac

ce

ss

es

Private Read-Shared Read-Write-Shared

Cacheline (128B) OS Page (2MB)

Fig 4 Distribution of memory accesses to private and shared pages at pagesize and cache line granularity

3

05

2

8

32

128

512

2048

8192

32768

Me

mo

ry S

ize (

MB

) Aggregate System LLC Capacity

Fig 5 Total memory capacity (across 4-gpus) needed to cover applicationshared working set size on a NUMA-GPU

The figure shows that 40 (up to 100) of GPU memoryaccesses are serviced by read-write pages Consequently pagereplication can not reduce NUMA bottlenecks when read-writeshared pages are frequently accessed

To address the limitations of replicating read-write sharedpages NUMA performance can be improved by caching shareddata in the on-chip last-level-cache (LLC) [16] [30] Toquantify the total LLC space requirements Figure 5 showsthe shared memory footprint for the different workloads Theshared memory footprint represents the total number of uniqueremote pages fetched by the different GPUs The figure clearlyshows that caching remote data in the GPU LLC is unlikelyto benefit since the application shared memory footprint oftensignificantly exceeds the aggregate system LLC capacity

A straightforward mechanism to fully cover the shared mem-ory footprint would be to increase the total LLC capacity in thesystem However Figure 5 shows that the system LLC capacitymust be drastically increased to fit the shared memory footprintof most workloads Unfortunately semiconductor scaling doesnot allow for such large LLCs to be incorporated on to high-performance processor chips Consequently alternative lowoverhead mechanisms are needed to increase the remote datacaching capacity of GPUs

We investigate increasing the GPU caching capacity byCaching Remote Data in Video Memory (CARVE) a hardwaremechanism that statically devotes a small fraction of the localGPU memory to replicate remote shared data CARVE reducesthe granularity of data replication from OS page granularity(ie 2MB) to a finer granularity (ie 128B cacheline) Toensure correct program execution however CARVE requiresan efficient mechanism to maintain coherence between multipleGPUs Thankfully Figure 4 shows that coherence bandwidthis expected to be small because the fine-grain replicationgranularity results in a smaller distribution of memory accessesto read-write shared data (further emphasizing the high degreeof false-sharing when using large page sizes) Before weinvestigate increasing GPU caching capacity we first discussour experimental methodology

III METHODOLOGY

We use an industry proprietary trace-driven performancesimulator to simulate a multi-GPU system with four GPUs inter-connected using high bandwidth links Each GPU in the systemhas a processor and memory hierarchy similar to the NVIDIAPascal GPU [10] We model 64 SMs per GPU that support64 warps each A warp scheduler selects warp instructions

TABLE IIWORKLOAD CHARACTERISTICS

Suite Benchmark Abbr Mem footprint

HPC

AMG 32 AMG 32 GBHPGMG-UVM HPGMG 20 GB

HPGMG-amry-proxy HPGMG-amry 77 GBLulesh-Unstruct-Mesh1 Lulesh 24 MB

Lulesh-s190 Lulesh-s190 37 GBCoMD-xyz64 warp CoMD 910 MBMCB-5M-particles MCB 254 MBMiniAMR-15Kv40 MiniAMR 44 GB

Nekbone-18 Nekbone 10 GBXSBench 17K grid XSBench 44 GB

Euler3D Euler 26 MBSSSP SSSP 42 MB

bfs-road-usa bfs-road 590 MB

MLAlexNet-ConvNet2 AlexNet 96 MB

GoogLeNet-cudnn-Lev2 GoogLeNet 12 GBOverFeat-cudnn-Lev3 OverFeat 88 MB

Other

Bitcoin-Crypto Bitcoin 56 GBOptix-Raytracing Raytracing 150 MB

stream-triad stream-triad 30 GBRandom Memory Access RandAccess 150 GB

each cycle The GPU memory system consists of a multi-levelcache and TLB hierarchy The first level of the cache and TLBhierarchy are private to each SM while the last level cacheand TLB are shared by all SMs We assume software-basedcache coherence across the private caches that is commonlyused in state-of-the-art GPUs [10] today Table III shows thesimulation parameters used in our baseline

We assume our multi-GPU system is connected usingNVLink [12] technology with 64GBs of bandwidth (in one-direction) We also assume each GPU is interconnected to aCPU using NVLink at 32GBs Our infrastructure also includesa detailed DRAM system with 32GB of memory capacity perGPU with 1TBs of local GPU memory bandwidth Each GPUmemory controller supports 128-entry read and write queuesper channel open-page policy minimalist address mappingpolicy [31] and FR-FCFS scheduling policy (prioritizing readsover writes) Writes are issued in batches when the write queuestarts to fill up

We evaluate 20 CUDA benchmarks taken from many appli-cations of interest HPC applications [32] fluid dynamics [33]graph search [34] machine learning deep neural networks [4]and medical imaging We show the memory footprint of ourworkloads in Table II We simulate all of our workloads forfour billion warp-instructions

TABLE IIIBASELINE MULTI-GPU SYSTEM

Number of GPUs 4Total Number of SMs 256Max number of warps 64 per SM

GPU frequency 1GHzOS Page Size 2MBL1 data cache 128KB per SM 128B lines 4 waysTotal L2 cache 32MB 128B lines 16 ways

Inter-GPU interconnect 64 GBs per link uni-directionalCPU-GPU interconnect 32 GBs per GPUTotal DRAM bandwidth 4 TBsTotal DRAM capacity 128GB

4

Fig 6 Architecting a small fraction of GPU memory as a Remote DataCache (RDC) enables caching large remote data working-sets

IV INCREASING GPU CACHE CAPACITY

The NUMA bottleneck occurs when a large fraction of memoryaccesses are serviced by remote memory over low-bandwidthinterconnection links [12] [35] [36] While NUMA has beenwell studied using conventional memory technology [14] [30][37] emerging high bandwidth memory technology enables anovel solution space to tackle the NUMA problem As such weinvestigate hardware mechanisms that dedicate a small fractionof local memory to store the contents of remote memory Doingso enables the remote memory data to be serviced at high localmemory bandwidth In the context of GPUs this approacheffectively increases the caching capacity of a GPU

A Caching Remote Data in Video Memory

We propose Caching Remote Data in Video Memory (CARVE)a hardware mechanism that statically reserves a portion of theGPUs on-package local memory to store remote data Sincethere is no latency or bandwidth advantage to duplicating localmemory data CARVE only stores remote data in the carve-outWe refer to this memory region as a Remote Data Cache (RDC)(see Figure 6) To avoid software overheads the RDC is entirelyhardware-managed and invisible to the programmer and GPUruntime system The amount of GPU memory dedicated forthe RDC can be statically set at system boot time (or kernellaunch time if only a single kernel is executing on the GPU)Thus CARVE transforms the local GPU memory into a hybridstructure that is simultaneously configured both as softwarevisible memory and a hardware-managed cache

CARVE requires minimal changes to the existing GPUdesign On a GPU LLC miss the GPU memory controllerdetermines whether the missing memory address will beserviced by the local GPU memory or a remote GPU memory Ifthe request maps to the local GPU memory the data is fetcheddirectly from local memory and sent to the LLC Otherwisethe memory controller first checks to see if the data is availablein the RDC In the event of a RDC hit the data is servicedlocally without incurring the bandwidth and latency penaltyof a remote GPU memory access However in the event ofan RDC miss the missing data is retrieved from the remote

DRAM ARRAY

2 KB Row Buffer (plus ECC) =

144B = 128-byte Data + 9-byte ECC + 7-byte (Tag+Metadata)

ADDR

DATA(128B)

TAG AND DATA (TAD)

ECC(9B) + TAG(7B) = (16B)

ROW BUFFER

16 x 144 byte TAD = 16 lines

Fig 7 Alloy Cache stores tags with data to enable low latency access Tags canbe stored alongside ECC bits to simplify data alignment and cache controller

GPU memory and returned to the LLC The missing data isalso inserted into the RDC to enable future hits2

Remote Data Cache (RDC) Design The RDC is architectedat a fine (eg 128B) granularity to minimize false sharing (seeFigure 4) In our study we model the RDC as a fine grainAlloy cache [39] (see Figure 7) which organizes its cache asa direct-mapped structure and stores Tags-with-Data In Alloyone access to the cache retrieves both the tag and the dataIf the tag matches (ie a cache hit) the data is then usedto service the request Otherwise the request is forwarded tothe next level of memory We implement Alloy by storing thetag in spare ECC bits3 similar to commercial implementationtoday [40] Note that RDC is not only limited to the Alloydesign and can also be architected using alternate DRAM cachearchitectures [39] [41] [42]

When GPU memory is configured with an RDC CARVErequires a single register in the GPU memory controller tospecify starting location (ie physical address) for the RDCCARVE also requires simple combinational logic to identify thephysical GPU memory location for a given RDC set Finallysince the RDC is architected in local GPU memory the differentRDC sets are interleaved across all GPU memory channels toensure high bandwidth concurrent RDC accesses

Remote Data Cache Evaluation We evaluate the benefits ofCARVE by using a 2GB RDC This amounts to 625 of thebaseline 32GB GPU memory dedicated as an RDC and the

2The RDC can also store data from System Memory However this requiresadditional support for handling CPU-GPU coherence [38]

3We assume GPU memory uses HBM technology HBM provides 16B of ECCto protect 128B of data Assuming ECC at a 16-byte granularity (enoughto protect bus transfers) SECDED requires 89=72 bits to protect the dataThis leaves 56 spare ECC bits that can be used for tag and metadata (RDConly requires 6 bits for tags)

0

01

02

03

04

05

06

07

08

R

em

ote

Ac

ce

ss

es NUMA-GPU

CARVE

Fig 8 Fraction of remote memory accesses CARVE reduces the averagefraction of remote memory accesses from 40 in NUMA-GPU to 8

5

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

eNUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE (No Coherence) NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 9 CARVE performance with zero overhead cache coherence CARVE is able to achieve Ideal-NUMA-GPU speedup

remaining 9375 exposed as operating system (OS) managedmemory (30GB per GPU)

RDC Effect on NUMA Traffic Figure 8 shows the fractionof remote memory accesses with NUMA-GPU and NUMA-GPU enhanced with CARVE The figure shows that manyworkloads on the baseline NUMA-GPU system still suffermore than 70 remote memory accesses (eg XSBench andLulesh) This is because the shared memory footprint of theseworkloads is larger than the GPU LLC CARVE on the otherhand significantly reduces the fraction of remote memoryaccesses On average NUMA-GPU experiences 40 remotememory accesses while CARVE experiences only 8 remotememory accesses In doing so CARVE efficiently utilizeslocal unused HBM bandwidth Consequently the figure showsthat CARVE can significantly reduce the inter-GPU bandwidthbottleneck for most workloads

RDC Performance Analysis (Upper Bound) We first eval-uate the performance potential of CARVE by assuming zero-overhead cache coherence between the RDCs (called CARVE-No-Coherence) This enables us to quantify the benefits ofservicing remote GPU memory accesses in local GPU memoryWe investigate coherence in the next subsection

Figure 9 shows the performance of NUMA-GPU NUMA-GPU with software page replication of read-only shared pagesCARVE-No-Coherence and an ideal NUMA-GPU system thatreplicates all shared pages The x-axis shows the differentworkloads and the y-axis shows the performance relativeto the ideal NUMA-GPU system The figure shows thatCARVE enables workloads like Lulesh Euler SSSP andHPGMG (which experience significant slowdown with NUMA-GPU and NUMA-GPU enhanced with read-only shared pagereplication) to approach the performance of an ideal NUMA-GPU system This is because these workloads experience a20-60 reduction in remote memory accesses (see Figure 8)This shows that extending the remote data caching capacityinto the local GPU memory by using a Remote Data Cache(RDC) significantly reduces the NUMA bandwidth bottleneckCARVE provides these performance improvements withoutrelying on any software support for page replication

While CARVE improves performance significantly acrossmany workloads CARVE can sometimes degrade performanceFor example RandAccess experiences a 10 performancedegradation due to frequent misses in the RDC The ad-

ditional latency penalty of first accessing the RDC thenaccessing remote memory can degrade performance of latency-sensitive workloads However using low-overhead cachehit-predictors [39] can mitigate these performance outliersOverall Figure 9 shows that CARVE significantly bridgesthe performance gap between NUMA-GPU and an idealNUMA-GPU system On average the baseline NUMA-GPUsystem and NUMA-GPU enhanced with read-only shared pagereplication experience a 50 performance gap relative to theideal NUMA-GPU system On the other hand CARVE-No-Coherence experiences only a 5 performance gap relative toan ideal NUMA-GPU system that replicates all shared pagesThese results show significant performance opportunity fromCARVE provided we can efficiently ensure data coherencebetween the RDCs

B Coherence Implications of an RDC in GPUs

CARVE has the potential to significantly improve NUMA-GPUperformance by reducing most of the remote memory accessesHowever since each Remote Data Cache (RDC) stores a copyof remote GPU memory an efficient mechanism is necessary tokeep the copies coherent Fortunately the GPU programmingmodel provides the programmer with an API to explicitlyinsert synchronization points to ensure coherent data accessThus since software coherence [28] is already supported onconventional GPU systems we first investigate whether CARVEcan be easily extended with software coherence

Software CoherenceConventional GPU designs maintain software coherence atkernel boundaries by enforcing two requirements First at theend of each kernel call modified data is written back to memory(ie flush dirty data) Second at the beginning of the nextkernel invocation the GPU cores must read the updated valuesfrom memory Conventional GPUs enforce these requirementsby implementing a write-through L1 cache that is invalidatedat every kernel boundary and a memory-side last-level cachethat is implicitly coherent

TABLE IVKERNEL-LAUNCH DELAY UNDER SOFTWARE COHERENCE

L2 Cache (8MB) RDC (2GB)Cache Invalidate 8MB 16 bank 1cycle 2GB 1024GBs local

4us 2ms =gt0msFlush Dirty 8MB 1024-64GBs 2GB 64GBs remote

8us˜128us 32ms =gt0ms

6

Fig 10 Epoch-counter based invalidation eliminates the penalty of explicitlyinvalidating the whole remote data cache

NUMA-GPUs cache remote data in the GPU LLC and extendsoftware coherence to the GPU LLC by also invalidating theLLC on kernel boundaries [16] Similarly RDC coherence canbe ensured by further extending software-coherence mecha-nisms to the RDC Thus on every kernel boundary the memorycontroller would also invalidate all RDC entries and write backdirty data (if any) to remote GPU memory Table IV comparesthe worst case software coherence overhead for RDC and on-chip LLCs The table shows that invalidating and flushing dirtydata in on-chip LLCs tends to be on the order of microsecondsand can be tolerated within kernel launch latency On theother hand the overhead of invalidating and writing backmillions of RDC lines can take on the order of millisecondsIncurring millisecond latency at every kernel boundary cansignificantly impact application performance Consequentlywe desire architecture support for efficiently invalidating andwriting back RDC lines

Efficient RDC Invalidation Since the RDC tag valid anddirty bits are all stored in GPU memory invalidating all RDClines requires reading and writing GPU memory Dependingon the RDC size RDC invalidation can be a long latency andbandwidth intensive process Alternatively we do not needto physically invalidate all of the lines We simply need toknow if the RDC data is stale (ie from a previous kernel andshould be re-fetched) or up-to-date (ie from current kerneland can be used) If we track the epoch an RDC cacheline wasinstalled in we can easily determine if the RDC data is staleor not Thus an epoch-counter-based invalidation scheme isused to invalidate the RDC instantly (ie 0 ms) [43] [44]

Figure 10 shows our RDC invalidation mechanism We usea 20-bit register to maintain an Epoch Counter (EPCTR) foreach kernelstream running on the GPU RDC insertions storethe current EPCTR value of the kernelstream with each RDCcacheline (in the ECC bits along with the RDC Tag) Bydoing so RDC hits only occur if the tag matches and if theEPCTR stored within the RDC cacheline matches the currentEPCTR value of the associated kernelstream To supportefficient RDC invalidation at kernel boundaries we simplyincrement the EPCTR of the appropriate kernel Thus theRDC hit logic ensures that the newly launched kernel does notconsume a previous kernelsrsquo stale data In the rare event thatthe EPCTR rolls over on an increment the memory controllerphysically resets all RDC cachelines by setting the valid bitand EPCTR to zero Note that an RDC architected in dense

DRAM technology enables large counters per RDC line thatotherwise were impractical with similar mechanisms appliedto on-chip caches [43] [44]

Efficient RDC Dirty Data Flush We also desire an efficientmechanism to flush dirty lines to remote GPU memory at kernelboundaries For a writeback RDC we would need to knowwhich RDC lines must be flushed Since our RDC invalidationmechanism no longer requires physically reading all RDC linesan alternate mechanism is necessary to locate dirty RDC linesWe can use a dirty-map [45] which tracks RDC regions thathave been written to On a kernel boundary the dirty-map canbe used to determine which RDC regions should be flushed toremote GPU memory To avoid the on-chip storage overheadfor the dirty-map we propose a write-through RDC insteadDoing so ensures that dirty data is immediately propagated toremote GPU memory Note that the write bandwidth to remoteGPU memory with a write-through RDC is identical to thebaseline NUMA-GPU configuration with no RDC

Our evaluations with writeback (with dirty-map) and write-through RDC showed that a write-through RDC performs nearlyas well (within 1 performance) as a write-back RDC Thisis because the vast majority of remote data cached in the linegranularity RDC tends to be heavily read-biased (see Figure 4)Consequently for the remainder of this work we assume awrite-through RDC to enable zero-latency dirty data flush andavoid the complexity of maintaining a write-back RDC

Software Coherence Results We now evaluate performanceof extending software coherence to the RDC We refer to thisdesign as CARVE with Software Coherence (CARVE-SWC)Figure 11 compares the performance of CARVE-SWC toCARVE with no coherence overhead (CARVE-No-Coherence)The figure shows that CARVE-SWC enables XSBench toperform similar to CARVE-No-Coherence However despiteeliminating all overheads to maintain software coherence forlarge caches CARVE-SWC removes all performance benefitsof RDC for the remaining workloads Since the only differencebetween CARVE-No-Coherence and CARVE-SWC is theflushing of the RDC between kernels these results suggestsignificant inter-kernel data locality that is exploited by CARVE-No-Coherence While this motivates further research work onimproving CARVE-SWC by efficiently prefetching remotedata at kernel boundaries this is out of the scope of this paperInstead we now investigate hardware coherence mechanismsthat do not require flushing the RDC at kernel boundaries

Hardware CoherenceCARVE-SWC results reveal that it is important to retainRDC data across kernel boundaries While software coherencecan potentially be relaxed to avoid flushing the RDC itwould require additional complexity for maintaining a softwaredirectory to track stale copies of shared data cached remotelyand invalidate them explicitly Since similar functionality isavailable in conventional hardware coherence mechanisms weinvestigate extending the RDC with hardware coherence instead

7

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

TABLE ICHARACTERISTICS OF RECENT NVIDIA GPUS

Fermi Kepler Maxwell Pascal VoltaSMs 16 15 24 56 80BW (GBs) 177 288 288 720 900Transistors (B) 30 71 80 153 211Tech node (nm) 40 28 28 16 12Chip size (mm2) 529 551 601 610 815

II BACKGROUND AND MOTIVATION

A GPU Performance ScalingTable I shows that GPU performance has scaled well overthe past decade due to increasing transistor density and GPUmemory bandwidth For example the recently announced VoltaGPU consists of 80 Streaming Multiprocessors (SMs) on alarge 815 mm2 chip consists of 211 billion transistors [2]Unfortunately the impending end of Moorersquos Law [8] andlimitations in lithography and manufacturing cost [21] threatencontinued GPU performance scaling Without larger or denserdies GPU architects must now investigate alternative techniquesfor GPU performance scaling

Recent advances in packaging [22] signaling [23] [24] andinterconnection technology [25] [26] enable new opportunitiesfor scaling GPU performance For example recent workpropose interconnecting multiple GPUs at a package-level [27]or system-level [16] with high bandwidth interconnectiontechnology (eg GRS [26] or NVLink [12]) These multi-GPU systems provide the programmer the illusion of a singleGPU system However they are non-uniform memory access(NUMA) systems with asymmetric bandwidth to local andremote GPU memory As such software and hardware tech-niques have been proposed to reduce the NUMA bottlenecksWe now discuss these techniques and identify their limitations

B NUMA-aware Multi-GPUsFigure 3 illustrates a NUMA-GPU system where performance isdirectly dependent on the available remote memory bandwidthTraditionally application programmers have manually managedthe placement of both compute and data to ensure high NUMAsystem performance Such practice significantly increasesprogrammer burden and may not necessarily scale to alternateNUMA systems To reduce programmer burden the recentNUMA-GPU [16] proposal presents software and hardwaretechniques to reduce NUMA bottlenecks without the need forprogrammer intervention

NUMA-GPU enhances the GPU runtime by improvingthe Cooperative Thread Array (CTA) scheduling and pageplacement policy Since adjacent CTAs tend to have significantspatial and temporal locality NUMA-GPU exploits inter-CTAdata locality by scheduling a large batch of contiguous CTAs toeach GPU Furthermore NUMA-GPU increases the likelihoodof servicing the majority of memory requests from the localGPU memory by using a First-Touch (FT) page placementpolicy The FT policy allocates pages to the memory of theGPU that first accessed that page If a page were private FTmapping ensures that memory accesses are serviced locally

If a page were already mapped in a remote GPU memory(eg shared data) NUMA-GPU caches remote data in the GPU

Fig 3 Organization and bandwidth of a NUMA-Aware Multi-GPU DistributedCTA-Scheduling and First-Touch Memory Mapping can improve data locality

cache hierarchy to enable future references to be serviced athigh bandwidth To ensure correctness NUMA-GPU extendsexisting software coherence mechanisms to the GPU LLC [28][29] Figure 3 illustrates the software and hardware mechanismsused by NUMA-GPU to reduce NUMA bottlenecks

C NUMA-GPU Performance Bottleneck

NUMA systems experience significant performance bottleneckswhen applications frequently access remote memory A recentstudy showed that remote memory accesses in NUMA systemsis because of false page sharing when using large page sizes(eg to improve TLB coverage [17] [18]) To address thefalse sharing problem the runtime system (or OS) can replicateshared pages locally to avoid remote memory accesses Unfor-tunately page replication increases the application memoryrequirements (on average 24x in our study) Thus pagereplication works best only when there are free memory pagesavailable Since future GPU workloads are expected to entirelyfill GPU memory capacity [15] page replication is expectedto benefit only if the GPU memory is under-utilized

Besides GPU memory capacity pressure page replication isonly limited to read-only shared pages and not read-write sharedpages This is because the software overhead of collapsingread-write shared pages (even on occasional writes) can beextremely expensive [13] Thus page-replication cannot reduceremote memory accesses when a large fraction of remotememory accesses are to read-write shared pages To illustratethis problem on the baseline NUMA-GPU system Figure 4shows the distribution of GPU memory accesses to privatepages read-only shared pages and read-write shared pages

0

20

40

60

80

100

M

em

ory

Ac

ce

ss

es

Private Read-Shared Read-Write-Shared

Cacheline (128B) OS Page (2MB)

Fig 4 Distribution of memory accesses to private and shared pages at pagesize and cache line granularity

3

05

2

8

32

128

512

2048

8192

32768

Me

mo

ry S

ize (

MB

) Aggregate System LLC Capacity

Fig 5 Total memory capacity (across 4-gpus) needed to cover applicationshared working set size on a NUMA-GPU

The figure shows that 40 (up to 100) of GPU memoryaccesses are serviced by read-write pages Consequently pagereplication can not reduce NUMA bottlenecks when read-writeshared pages are frequently accessed

To address the limitations of replicating read-write sharedpages NUMA performance can be improved by caching shareddata in the on-chip last-level-cache (LLC) [16] [30] Toquantify the total LLC space requirements Figure 5 showsthe shared memory footprint for the different workloads Theshared memory footprint represents the total number of uniqueremote pages fetched by the different GPUs The figure clearlyshows that caching remote data in the GPU LLC is unlikelyto benefit since the application shared memory footprint oftensignificantly exceeds the aggregate system LLC capacity

A straightforward mechanism to fully cover the shared mem-ory footprint would be to increase the total LLC capacity in thesystem However Figure 5 shows that the system LLC capacitymust be drastically increased to fit the shared memory footprintof most workloads Unfortunately semiconductor scaling doesnot allow for such large LLCs to be incorporated on to high-performance processor chips Consequently alternative lowoverhead mechanisms are needed to increase the remote datacaching capacity of GPUs

We investigate increasing the GPU caching capacity byCaching Remote Data in Video Memory (CARVE) a hardwaremechanism that statically devotes a small fraction of the localGPU memory to replicate remote shared data CARVE reducesthe granularity of data replication from OS page granularity(ie 2MB) to a finer granularity (ie 128B cacheline) Toensure correct program execution however CARVE requiresan efficient mechanism to maintain coherence between multipleGPUs Thankfully Figure 4 shows that coherence bandwidthis expected to be small because the fine-grain replicationgranularity results in a smaller distribution of memory accessesto read-write shared data (further emphasizing the high degreeof false-sharing when using large page sizes) Before weinvestigate increasing GPU caching capacity we first discussour experimental methodology

III METHODOLOGY

We use an industry proprietary trace-driven performancesimulator to simulate a multi-GPU system with four GPUs inter-connected using high bandwidth links Each GPU in the systemhas a processor and memory hierarchy similar to the NVIDIAPascal GPU [10] We model 64 SMs per GPU that support64 warps each A warp scheduler selects warp instructions

TABLE IIWORKLOAD CHARACTERISTICS

Suite Benchmark Abbr Mem footprint

HPC

AMG 32 AMG 32 GBHPGMG-UVM HPGMG 20 GB

HPGMG-amry-proxy HPGMG-amry 77 GBLulesh-Unstruct-Mesh1 Lulesh 24 MB

Lulesh-s190 Lulesh-s190 37 GBCoMD-xyz64 warp CoMD 910 MBMCB-5M-particles MCB 254 MBMiniAMR-15Kv40 MiniAMR 44 GB

Nekbone-18 Nekbone 10 GBXSBench 17K grid XSBench 44 GB

Euler3D Euler 26 MBSSSP SSSP 42 MB

bfs-road-usa bfs-road 590 MB

MLAlexNet-ConvNet2 AlexNet 96 MB

GoogLeNet-cudnn-Lev2 GoogLeNet 12 GBOverFeat-cudnn-Lev3 OverFeat 88 MB

Other

Bitcoin-Crypto Bitcoin 56 GBOptix-Raytracing Raytracing 150 MB

stream-triad stream-triad 30 GBRandom Memory Access RandAccess 150 GB

each cycle The GPU memory system consists of a multi-levelcache and TLB hierarchy The first level of the cache and TLBhierarchy are private to each SM while the last level cacheand TLB are shared by all SMs We assume software-basedcache coherence across the private caches that is commonlyused in state-of-the-art GPUs [10] today Table III shows thesimulation parameters used in our baseline

We assume our multi-GPU system is connected usingNVLink [12] technology with 64GBs of bandwidth (in one-direction) We also assume each GPU is interconnected to aCPU using NVLink at 32GBs Our infrastructure also includesa detailed DRAM system with 32GB of memory capacity perGPU with 1TBs of local GPU memory bandwidth Each GPUmemory controller supports 128-entry read and write queuesper channel open-page policy minimalist address mappingpolicy [31] and FR-FCFS scheduling policy (prioritizing readsover writes) Writes are issued in batches when the write queuestarts to fill up

We evaluate 20 CUDA benchmarks taken from many appli-cations of interest HPC applications [32] fluid dynamics [33]graph search [34] machine learning deep neural networks [4]and medical imaging We show the memory footprint of ourworkloads in Table II We simulate all of our workloads forfour billion warp-instructions

TABLE IIIBASELINE MULTI-GPU SYSTEM

Number of GPUs 4Total Number of SMs 256Max number of warps 64 per SM

GPU frequency 1GHzOS Page Size 2MBL1 data cache 128KB per SM 128B lines 4 waysTotal L2 cache 32MB 128B lines 16 ways

Inter-GPU interconnect 64 GBs per link uni-directionalCPU-GPU interconnect 32 GBs per GPUTotal DRAM bandwidth 4 TBsTotal DRAM capacity 128GB

4

Fig 6 Architecting a small fraction of GPU memory as a Remote DataCache (RDC) enables caching large remote data working-sets

IV INCREASING GPU CACHE CAPACITY

The NUMA bottleneck occurs when a large fraction of memoryaccesses are serviced by remote memory over low-bandwidthinterconnection links [12] [35] [36] While NUMA has beenwell studied using conventional memory technology [14] [30][37] emerging high bandwidth memory technology enables anovel solution space to tackle the NUMA problem As such weinvestigate hardware mechanisms that dedicate a small fractionof local memory to store the contents of remote memory Doingso enables the remote memory data to be serviced at high localmemory bandwidth In the context of GPUs this approacheffectively increases the caching capacity of a GPU

A Caching Remote Data in Video Memory

We propose Caching Remote Data in Video Memory (CARVE)a hardware mechanism that statically reserves a portion of theGPUs on-package local memory to store remote data Sincethere is no latency or bandwidth advantage to duplicating localmemory data CARVE only stores remote data in the carve-outWe refer to this memory region as a Remote Data Cache (RDC)(see Figure 6) To avoid software overheads the RDC is entirelyhardware-managed and invisible to the programmer and GPUruntime system The amount of GPU memory dedicated forthe RDC can be statically set at system boot time (or kernellaunch time if only a single kernel is executing on the GPU)Thus CARVE transforms the local GPU memory into a hybridstructure that is simultaneously configured both as softwarevisible memory and a hardware-managed cache

CARVE requires minimal changes to the existing GPUdesign On a GPU LLC miss the GPU memory controllerdetermines whether the missing memory address will beserviced by the local GPU memory or a remote GPU memory Ifthe request maps to the local GPU memory the data is fetcheddirectly from local memory and sent to the LLC Otherwisethe memory controller first checks to see if the data is availablein the RDC In the event of a RDC hit the data is servicedlocally without incurring the bandwidth and latency penaltyof a remote GPU memory access However in the event ofan RDC miss the missing data is retrieved from the remote

DRAM ARRAY

2 KB Row Buffer (plus ECC) =

144B = 128-byte Data + 9-byte ECC + 7-byte (Tag+Metadata)

ADDR

DATA(128B)

TAG AND DATA (TAD)

ECC(9B) + TAG(7B) = (16B)

ROW BUFFER

16 x 144 byte TAD = 16 lines

Fig 7 Alloy Cache stores tags with data to enable low latency access Tags canbe stored alongside ECC bits to simplify data alignment and cache controller

GPU memory and returned to the LLC The missing data isalso inserted into the RDC to enable future hits2

Remote Data Cache (RDC) Design The RDC is architectedat a fine (eg 128B) granularity to minimize false sharing (seeFigure 4) In our study we model the RDC as a fine grainAlloy cache [39] (see Figure 7) which organizes its cache asa direct-mapped structure and stores Tags-with-Data In Alloyone access to the cache retrieves both the tag and the dataIf the tag matches (ie a cache hit) the data is then usedto service the request Otherwise the request is forwarded tothe next level of memory We implement Alloy by storing thetag in spare ECC bits3 similar to commercial implementationtoday [40] Note that RDC is not only limited to the Alloydesign and can also be architected using alternate DRAM cachearchitectures [39] [41] [42]

When GPU memory is configured with an RDC CARVErequires a single register in the GPU memory controller tospecify starting location (ie physical address) for the RDCCARVE also requires simple combinational logic to identify thephysical GPU memory location for a given RDC set Finallysince the RDC is architected in local GPU memory the differentRDC sets are interleaved across all GPU memory channels toensure high bandwidth concurrent RDC accesses

Remote Data Cache Evaluation We evaluate the benefits ofCARVE by using a 2GB RDC This amounts to 625 of thebaseline 32GB GPU memory dedicated as an RDC and the

2The RDC can also store data from System Memory However this requiresadditional support for handling CPU-GPU coherence [38]

3We assume GPU memory uses HBM technology HBM provides 16B of ECCto protect 128B of data Assuming ECC at a 16-byte granularity (enoughto protect bus transfers) SECDED requires 89=72 bits to protect the dataThis leaves 56 spare ECC bits that can be used for tag and metadata (RDConly requires 6 bits for tags)

0

01

02

03

04

05

06

07

08

R

em

ote

Ac

ce

ss

es NUMA-GPU

CARVE

Fig 8 Fraction of remote memory accesses CARVE reduces the averagefraction of remote memory accesses from 40 in NUMA-GPU to 8

5

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

eNUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE (No Coherence) NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 9 CARVE performance with zero overhead cache coherence CARVE is able to achieve Ideal-NUMA-GPU speedup

remaining 9375 exposed as operating system (OS) managedmemory (30GB per GPU)

RDC Effect on NUMA Traffic Figure 8 shows the fractionof remote memory accesses with NUMA-GPU and NUMA-GPU enhanced with CARVE The figure shows that manyworkloads on the baseline NUMA-GPU system still suffermore than 70 remote memory accesses (eg XSBench andLulesh) This is because the shared memory footprint of theseworkloads is larger than the GPU LLC CARVE on the otherhand significantly reduces the fraction of remote memoryaccesses On average NUMA-GPU experiences 40 remotememory accesses while CARVE experiences only 8 remotememory accesses In doing so CARVE efficiently utilizeslocal unused HBM bandwidth Consequently the figure showsthat CARVE can significantly reduce the inter-GPU bandwidthbottleneck for most workloads

RDC Performance Analysis (Upper Bound) We first eval-uate the performance potential of CARVE by assuming zero-overhead cache coherence between the RDCs (called CARVE-No-Coherence) This enables us to quantify the benefits ofservicing remote GPU memory accesses in local GPU memoryWe investigate coherence in the next subsection

Figure 9 shows the performance of NUMA-GPU NUMA-GPU with software page replication of read-only shared pagesCARVE-No-Coherence and an ideal NUMA-GPU system thatreplicates all shared pages The x-axis shows the differentworkloads and the y-axis shows the performance relativeto the ideal NUMA-GPU system The figure shows thatCARVE enables workloads like Lulesh Euler SSSP andHPGMG (which experience significant slowdown with NUMA-GPU and NUMA-GPU enhanced with read-only shared pagereplication) to approach the performance of an ideal NUMA-GPU system This is because these workloads experience a20-60 reduction in remote memory accesses (see Figure 8)This shows that extending the remote data caching capacityinto the local GPU memory by using a Remote Data Cache(RDC) significantly reduces the NUMA bandwidth bottleneckCARVE provides these performance improvements withoutrelying on any software support for page replication

While CARVE improves performance significantly acrossmany workloads CARVE can sometimes degrade performanceFor example RandAccess experiences a 10 performancedegradation due to frequent misses in the RDC The ad-

ditional latency penalty of first accessing the RDC thenaccessing remote memory can degrade performance of latency-sensitive workloads However using low-overhead cachehit-predictors [39] can mitigate these performance outliersOverall Figure 9 shows that CARVE significantly bridgesthe performance gap between NUMA-GPU and an idealNUMA-GPU system On average the baseline NUMA-GPUsystem and NUMA-GPU enhanced with read-only shared pagereplication experience a 50 performance gap relative to theideal NUMA-GPU system On the other hand CARVE-No-Coherence experiences only a 5 performance gap relative toan ideal NUMA-GPU system that replicates all shared pagesThese results show significant performance opportunity fromCARVE provided we can efficiently ensure data coherencebetween the RDCs

B Coherence Implications of an RDC in GPUs

CARVE has the potential to significantly improve NUMA-GPUperformance by reducing most of the remote memory accessesHowever since each Remote Data Cache (RDC) stores a copyof remote GPU memory an efficient mechanism is necessary tokeep the copies coherent Fortunately the GPU programmingmodel provides the programmer with an API to explicitlyinsert synchronization points to ensure coherent data accessThus since software coherence [28] is already supported onconventional GPU systems we first investigate whether CARVEcan be easily extended with software coherence

Software CoherenceConventional GPU designs maintain software coherence atkernel boundaries by enforcing two requirements First at theend of each kernel call modified data is written back to memory(ie flush dirty data) Second at the beginning of the nextkernel invocation the GPU cores must read the updated valuesfrom memory Conventional GPUs enforce these requirementsby implementing a write-through L1 cache that is invalidatedat every kernel boundary and a memory-side last-level cachethat is implicitly coherent

TABLE IVKERNEL-LAUNCH DELAY UNDER SOFTWARE COHERENCE

L2 Cache (8MB) RDC (2GB)Cache Invalidate 8MB 16 bank 1cycle 2GB 1024GBs local

4us 2ms =gt0msFlush Dirty 8MB 1024-64GBs 2GB 64GBs remote

8us˜128us 32ms =gt0ms

6

Fig 10 Epoch-counter based invalidation eliminates the penalty of explicitlyinvalidating the whole remote data cache

NUMA-GPUs cache remote data in the GPU LLC and extendsoftware coherence to the GPU LLC by also invalidating theLLC on kernel boundaries [16] Similarly RDC coherence canbe ensured by further extending software-coherence mecha-nisms to the RDC Thus on every kernel boundary the memorycontroller would also invalidate all RDC entries and write backdirty data (if any) to remote GPU memory Table IV comparesthe worst case software coherence overhead for RDC and on-chip LLCs The table shows that invalidating and flushing dirtydata in on-chip LLCs tends to be on the order of microsecondsand can be tolerated within kernel launch latency On theother hand the overhead of invalidating and writing backmillions of RDC lines can take on the order of millisecondsIncurring millisecond latency at every kernel boundary cansignificantly impact application performance Consequentlywe desire architecture support for efficiently invalidating andwriting back RDC lines

Efficient RDC Invalidation Since the RDC tag valid anddirty bits are all stored in GPU memory invalidating all RDClines requires reading and writing GPU memory Dependingon the RDC size RDC invalidation can be a long latency andbandwidth intensive process Alternatively we do not needto physically invalidate all of the lines We simply need toknow if the RDC data is stale (ie from a previous kernel andshould be re-fetched) or up-to-date (ie from current kerneland can be used) If we track the epoch an RDC cacheline wasinstalled in we can easily determine if the RDC data is staleor not Thus an epoch-counter-based invalidation scheme isused to invalidate the RDC instantly (ie 0 ms) [43] [44]

Figure 10 shows our RDC invalidation mechanism We usea 20-bit register to maintain an Epoch Counter (EPCTR) foreach kernelstream running on the GPU RDC insertions storethe current EPCTR value of the kernelstream with each RDCcacheline (in the ECC bits along with the RDC Tag) Bydoing so RDC hits only occur if the tag matches and if theEPCTR stored within the RDC cacheline matches the currentEPCTR value of the associated kernelstream To supportefficient RDC invalidation at kernel boundaries we simplyincrement the EPCTR of the appropriate kernel Thus theRDC hit logic ensures that the newly launched kernel does notconsume a previous kernelsrsquo stale data In the rare event thatthe EPCTR rolls over on an increment the memory controllerphysically resets all RDC cachelines by setting the valid bitand EPCTR to zero Note that an RDC architected in dense

DRAM technology enables large counters per RDC line thatotherwise were impractical with similar mechanisms appliedto on-chip caches [43] [44]

Efficient RDC Dirty Data Flush We also desire an efficientmechanism to flush dirty lines to remote GPU memory at kernelboundaries For a writeback RDC we would need to knowwhich RDC lines must be flushed Since our RDC invalidationmechanism no longer requires physically reading all RDC linesan alternate mechanism is necessary to locate dirty RDC linesWe can use a dirty-map [45] which tracks RDC regions thathave been written to On a kernel boundary the dirty-map canbe used to determine which RDC regions should be flushed toremote GPU memory To avoid the on-chip storage overheadfor the dirty-map we propose a write-through RDC insteadDoing so ensures that dirty data is immediately propagated toremote GPU memory Note that the write bandwidth to remoteGPU memory with a write-through RDC is identical to thebaseline NUMA-GPU configuration with no RDC

Our evaluations with writeback (with dirty-map) and write-through RDC showed that a write-through RDC performs nearlyas well (within 1 performance) as a write-back RDC Thisis because the vast majority of remote data cached in the linegranularity RDC tends to be heavily read-biased (see Figure 4)Consequently for the remainder of this work we assume awrite-through RDC to enable zero-latency dirty data flush andavoid the complexity of maintaining a write-back RDC

Software Coherence Results We now evaluate performanceof extending software coherence to the RDC We refer to thisdesign as CARVE with Software Coherence (CARVE-SWC)Figure 11 compares the performance of CARVE-SWC toCARVE with no coherence overhead (CARVE-No-Coherence)The figure shows that CARVE-SWC enables XSBench toperform similar to CARVE-No-Coherence However despiteeliminating all overheads to maintain software coherence forlarge caches CARVE-SWC removes all performance benefitsof RDC for the remaining workloads Since the only differencebetween CARVE-No-Coherence and CARVE-SWC is theflushing of the RDC between kernels these results suggestsignificant inter-kernel data locality that is exploited by CARVE-No-Coherence While this motivates further research work onimproving CARVE-SWC by efficiently prefetching remotedata at kernel boundaries this is out of the scope of this paperInstead we now investigate hardware coherence mechanismsthat do not require flushing the RDC at kernel boundaries

Hardware CoherenceCARVE-SWC results reveal that it is important to retainRDC data across kernel boundaries While software coherencecan potentially be relaxed to avoid flushing the RDC itwould require additional complexity for maintaining a softwaredirectory to track stale copies of shared data cached remotelyand invalidate them explicitly Since similar functionality isavailable in conventional hardware coherence mechanisms weinvestigate extending the RDC with hardware coherence instead

7

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

05

2

8

32

128

512

2048

8192

32768

Me

mo

ry S

ize (

MB

) Aggregate System LLC Capacity

Fig 5 Total memory capacity (across 4-gpus) needed to cover applicationshared working set size on a NUMA-GPU

The figure shows that 40 (up to 100) of GPU memoryaccesses are serviced by read-write pages Consequently pagereplication can not reduce NUMA bottlenecks when read-writeshared pages are frequently accessed

To address the limitations of replicating read-write sharedpages NUMA performance can be improved by caching shareddata in the on-chip last-level-cache (LLC) [16] [30] Toquantify the total LLC space requirements Figure 5 showsthe shared memory footprint for the different workloads Theshared memory footprint represents the total number of uniqueremote pages fetched by the different GPUs The figure clearlyshows that caching remote data in the GPU LLC is unlikelyto benefit since the application shared memory footprint oftensignificantly exceeds the aggregate system LLC capacity

A straightforward mechanism to fully cover the shared mem-ory footprint would be to increase the total LLC capacity in thesystem However Figure 5 shows that the system LLC capacitymust be drastically increased to fit the shared memory footprintof most workloads Unfortunately semiconductor scaling doesnot allow for such large LLCs to be incorporated on to high-performance processor chips Consequently alternative lowoverhead mechanisms are needed to increase the remote datacaching capacity of GPUs

We investigate increasing the GPU caching capacity byCaching Remote Data in Video Memory (CARVE) a hardwaremechanism that statically devotes a small fraction of the localGPU memory to replicate remote shared data CARVE reducesthe granularity of data replication from OS page granularity(ie 2MB) to a finer granularity (ie 128B cacheline) Toensure correct program execution however CARVE requiresan efficient mechanism to maintain coherence between multipleGPUs Thankfully Figure 4 shows that coherence bandwidthis expected to be small because the fine-grain replicationgranularity results in a smaller distribution of memory accessesto read-write shared data (further emphasizing the high degreeof false-sharing when using large page sizes) Before weinvestigate increasing GPU caching capacity we first discussour experimental methodology

III METHODOLOGY

We use an industry proprietary trace-driven performancesimulator to simulate a multi-GPU system with four GPUs inter-connected using high bandwidth links Each GPU in the systemhas a processor and memory hierarchy similar to the NVIDIAPascal GPU [10] We model 64 SMs per GPU that support64 warps each A warp scheduler selects warp instructions

TABLE IIWORKLOAD CHARACTERISTICS

Suite Benchmark Abbr Mem footprint

HPC

AMG 32 AMG 32 GBHPGMG-UVM HPGMG 20 GB

HPGMG-amry-proxy HPGMG-amry 77 GBLulesh-Unstruct-Mesh1 Lulesh 24 MB

Lulesh-s190 Lulesh-s190 37 GBCoMD-xyz64 warp CoMD 910 MBMCB-5M-particles MCB 254 MBMiniAMR-15Kv40 MiniAMR 44 GB

Nekbone-18 Nekbone 10 GBXSBench 17K grid XSBench 44 GB

Euler3D Euler 26 MBSSSP SSSP 42 MB

bfs-road-usa bfs-road 590 MB

MLAlexNet-ConvNet2 AlexNet 96 MB

GoogLeNet-cudnn-Lev2 GoogLeNet 12 GBOverFeat-cudnn-Lev3 OverFeat 88 MB

Other

Bitcoin-Crypto Bitcoin 56 GBOptix-Raytracing Raytracing 150 MB

stream-triad stream-triad 30 GBRandom Memory Access RandAccess 150 GB

each cycle The GPU memory system consists of a multi-levelcache and TLB hierarchy The first level of the cache and TLBhierarchy are private to each SM while the last level cacheand TLB are shared by all SMs We assume software-basedcache coherence across the private caches that is commonlyused in state-of-the-art GPUs [10] today Table III shows thesimulation parameters used in our baseline

We assume our multi-GPU system is connected usingNVLink [12] technology with 64GBs of bandwidth (in one-direction) We also assume each GPU is interconnected to aCPU using NVLink at 32GBs Our infrastructure also includesa detailed DRAM system with 32GB of memory capacity perGPU with 1TBs of local GPU memory bandwidth Each GPUmemory controller supports 128-entry read and write queuesper channel open-page policy minimalist address mappingpolicy [31] and FR-FCFS scheduling policy (prioritizing readsover writes) Writes are issued in batches when the write queuestarts to fill up

We evaluate 20 CUDA benchmarks taken from many appli-cations of interest HPC applications [32] fluid dynamics [33]graph search [34] machine learning deep neural networks [4]and medical imaging We show the memory footprint of ourworkloads in Table II We simulate all of our workloads forfour billion warp-instructions

TABLE IIIBASELINE MULTI-GPU SYSTEM

Number of GPUs 4Total Number of SMs 256Max number of warps 64 per SM

GPU frequency 1GHzOS Page Size 2MBL1 data cache 128KB per SM 128B lines 4 waysTotal L2 cache 32MB 128B lines 16 ways

Inter-GPU interconnect 64 GBs per link uni-directionalCPU-GPU interconnect 32 GBs per GPUTotal DRAM bandwidth 4 TBsTotal DRAM capacity 128GB

4

Fig 6 Architecting a small fraction of GPU memory as a Remote DataCache (RDC) enables caching large remote data working-sets

IV INCREASING GPU CACHE CAPACITY

The NUMA bottleneck occurs when a large fraction of memoryaccesses are serviced by remote memory over low-bandwidthinterconnection links [12] [35] [36] While NUMA has beenwell studied using conventional memory technology [14] [30][37] emerging high bandwidth memory technology enables anovel solution space to tackle the NUMA problem As such weinvestigate hardware mechanisms that dedicate a small fractionof local memory to store the contents of remote memory Doingso enables the remote memory data to be serviced at high localmemory bandwidth In the context of GPUs this approacheffectively increases the caching capacity of a GPU

A Caching Remote Data in Video Memory

We propose Caching Remote Data in Video Memory (CARVE)a hardware mechanism that statically reserves a portion of theGPUs on-package local memory to store remote data Sincethere is no latency or bandwidth advantage to duplicating localmemory data CARVE only stores remote data in the carve-outWe refer to this memory region as a Remote Data Cache (RDC)(see Figure 6) To avoid software overheads the RDC is entirelyhardware-managed and invisible to the programmer and GPUruntime system The amount of GPU memory dedicated forthe RDC can be statically set at system boot time (or kernellaunch time if only a single kernel is executing on the GPU)Thus CARVE transforms the local GPU memory into a hybridstructure that is simultaneously configured both as softwarevisible memory and a hardware-managed cache

CARVE requires minimal changes to the existing GPUdesign On a GPU LLC miss the GPU memory controllerdetermines whether the missing memory address will beserviced by the local GPU memory or a remote GPU memory Ifthe request maps to the local GPU memory the data is fetcheddirectly from local memory and sent to the LLC Otherwisethe memory controller first checks to see if the data is availablein the RDC In the event of a RDC hit the data is servicedlocally without incurring the bandwidth and latency penaltyof a remote GPU memory access However in the event ofan RDC miss the missing data is retrieved from the remote

DRAM ARRAY

2 KB Row Buffer (plus ECC) =

144B = 128-byte Data + 9-byte ECC + 7-byte (Tag+Metadata)

ADDR

DATA(128B)

TAG AND DATA (TAD)

ECC(9B) + TAG(7B) = (16B)

ROW BUFFER

16 x 144 byte TAD = 16 lines

Fig 7 Alloy Cache stores tags with data to enable low latency access Tags canbe stored alongside ECC bits to simplify data alignment and cache controller

GPU memory and returned to the LLC The missing data isalso inserted into the RDC to enable future hits2

Remote Data Cache (RDC) Design The RDC is architectedat a fine (eg 128B) granularity to minimize false sharing (seeFigure 4) In our study we model the RDC as a fine grainAlloy cache [39] (see Figure 7) which organizes its cache asa direct-mapped structure and stores Tags-with-Data In Alloyone access to the cache retrieves both the tag and the dataIf the tag matches (ie a cache hit) the data is then usedto service the request Otherwise the request is forwarded tothe next level of memory We implement Alloy by storing thetag in spare ECC bits3 similar to commercial implementationtoday [40] Note that RDC is not only limited to the Alloydesign and can also be architected using alternate DRAM cachearchitectures [39] [41] [42]

When GPU memory is configured with an RDC CARVErequires a single register in the GPU memory controller tospecify starting location (ie physical address) for the RDCCARVE also requires simple combinational logic to identify thephysical GPU memory location for a given RDC set Finallysince the RDC is architected in local GPU memory the differentRDC sets are interleaved across all GPU memory channels toensure high bandwidth concurrent RDC accesses

Remote Data Cache Evaluation We evaluate the benefits ofCARVE by using a 2GB RDC This amounts to 625 of thebaseline 32GB GPU memory dedicated as an RDC and the

2The RDC can also store data from System Memory However this requiresadditional support for handling CPU-GPU coherence [38]

3We assume GPU memory uses HBM technology HBM provides 16B of ECCto protect 128B of data Assuming ECC at a 16-byte granularity (enoughto protect bus transfers) SECDED requires 89=72 bits to protect the dataThis leaves 56 spare ECC bits that can be used for tag and metadata (RDConly requires 6 bits for tags)

0

01

02

03

04

05

06

07

08

R

em

ote

Ac

ce

ss

es NUMA-GPU

CARVE

Fig 8 Fraction of remote memory accesses CARVE reduces the averagefraction of remote memory accesses from 40 in NUMA-GPU to 8

5

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

eNUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE (No Coherence) NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 9 CARVE performance with zero overhead cache coherence CARVE is able to achieve Ideal-NUMA-GPU speedup

remaining 9375 exposed as operating system (OS) managedmemory (30GB per GPU)

RDC Effect on NUMA Traffic Figure 8 shows the fractionof remote memory accesses with NUMA-GPU and NUMA-GPU enhanced with CARVE The figure shows that manyworkloads on the baseline NUMA-GPU system still suffermore than 70 remote memory accesses (eg XSBench andLulesh) This is because the shared memory footprint of theseworkloads is larger than the GPU LLC CARVE on the otherhand significantly reduces the fraction of remote memoryaccesses On average NUMA-GPU experiences 40 remotememory accesses while CARVE experiences only 8 remotememory accesses In doing so CARVE efficiently utilizeslocal unused HBM bandwidth Consequently the figure showsthat CARVE can significantly reduce the inter-GPU bandwidthbottleneck for most workloads

RDC Performance Analysis (Upper Bound) We first eval-uate the performance potential of CARVE by assuming zero-overhead cache coherence between the RDCs (called CARVE-No-Coherence) This enables us to quantify the benefits ofservicing remote GPU memory accesses in local GPU memoryWe investigate coherence in the next subsection

Figure 9 shows the performance of NUMA-GPU NUMA-GPU with software page replication of read-only shared pagesCARVE-No-Coherence and an ideal NUMA-GPU system thatreplicates all shared pages The x-axis shows the differentworkloads and the y-axis shows the performance relativeto the ideal NUMA-GPU system The figure shows thatCARVE enables workloads like Lulesh Euler SSSP andHPGMG (which experience significant slowdown with NUMA-GPU and NUMA-GPU enhanced with read-only shared pagereplication) to approach the performance of an ideal NUMA-GPU system This is because these workloads experience a20-60 reduction in remote memory accesses (see Figure 8)This shows that extending the remote data caching capacityinto the local GPU memory by using a Remote Data Cache(RDC) significantly reduces the NUMA bandwidth bottleneckCARVE provides these performance improvements withoutrelying on any software support for page replication

While CARVE improves performance significantly acrossmany workloads CARVE can sometimes degrade performanceFor example RandAccess experiences a 10 performancedegradation due to frequent misses in the RDC The ad-

ditional latency penalty of first accessing the RDC thenaccessing remote memory can degrade performance of latency-sensitive workloads However using low-overhead cachehit-predictors [39] can mitigate these performance outliersOverall Figure 9 shows that CARVE significantly bridgesthe performance gap between NUMA-GPU and an idealNUMA-GPU system On average the baseline NUMA-GPUsystem and NUMA-GPU enhanced with read-only shared pagereplication experience a 50 performance gap relative to theideal NUMA-GPU system On the other hand CARVE-No-Coherence experiences only a 5 performance gap relative toan ideal NUMA-GPU system that replicates all shared pagesThese results show significant performance opportunity fromCARVE provided we can efficiently ensure data coherencebetween the RDCs

B Coherence Implications of an RDC in GPUs

CARVE has the potential to significantly improve NUMA-GPUperformance by reducing most of the remote memory accessesHowever since each Remote Data Cache (RDC) stores a copyof remote GPU memory an efficient mechanism is necessary tokeep the copies coherent Fortunately the GPU programmingmodel provides the programmer with an API to explicitlyinsert synchronization points to ensure coherent data accessThus since software coherence [28] is already supported onconventional GPU systems we first investigate whether CARVEcan be easily extended with software coherence

Software CoherenceConventional GPU designs maintain software coherence atkernel boundaries by enforcing two requirements First at theend of each kernel call modified data is written back to memory(ie flush dirty data) Second at the beginning of the nextkernel invocation the GPU cores must read the updated valuesfrom memory Conventional GPUs enforce these requirementsby implementing a write-through L1 cache that is invalidatedat every kernel boundary and a memory-side last-level cachethat is implicitly coherent

TABLE IVKERNEL-LAUNCH DELAY UNDER SOFTWARE COHERENCE

L2 Cache (8MB) RDC (2GB)Cache Invalidate 8MB 16 bank 1cycle 2GB 1024GBs local

4us 2ms =gt0msFlush Dirty 8MB 1024-64GBs 2GB 64GBs remote

8us˜128us 32ms =gt0ms

6

Fig 10 Epoch-counter based invalidation eliminates the penalty of explicitlyinvalidating the whole remote data cache

NUMA-GPUs cache remote data in the GPU LLC and extendsoftware coherence to the GPU LLC by also invalidating theLLC on kernel boundaries [16] Similarly RDC coherence canbe ensured by further extending software-coherence mecha-nisms to the RDC Thus on every kernel boundary the memorycontroller would also invalidate all RDC entries and write backdirty data (if any) to remote GPU memory Table IV comparesthe worst case software coherence overhead for RDC and on-chip LLCs The table shows that invalidating and flushing dirtydata in on-chip LLCs tends to be on the order of microsecondsand can be tolerated within kernel launch latency On theother hand the overhead of invalidating and writing backmillions of RDC lines can take on the order of millisecondsIncurring millisecond latency at every kernel boundary cansignificantly impact application performance Consequentlywe desire architecture support for efficiently invalidating andwriting back RDC lines

Efficient RDC Invalidation Since the RDC tag valid anddirty bits are all stored in GPU memory invalidating all RDClines requires reading and writing GPU memory Dependingon the RDC size RDC invalidation can be a long latency andbandwidth intensive process Alternatively we do not needto physically invalidate all of the lines We simply need toknow if the RDC data is stale (ie from a previous kernel andshould be re-fetched) or up-to-date (ie from current kerneland can be used) If we track the epoch an RDC cacheline wasinstalled in we can easily determine if the RDC data is staleor not Thus an epoch-counter-based invalidation scheme isused to invalidate the RDC instantly (ie 0 ms) [43] [44]

Figure 10 shows our RDC invalidation mechanism We usea 20-bit register to maintain an Epoch Counter (EPCTR) foreach kernelstream running on the GPU RDC insertions storethe current EPCTR value of the kernelstream with each RDCcacheline (in the ECC bits along with the RDC Tag) Bydoing so RDC hits only occur if the tag matches and if theEPCTR stored within the RDC cacheline matches the currentEPCTR value of the associated kernelstream To supportefficient RDC invalidation at kernel boundaries we simplyincrement the EPCTR of the appropriate kernel Thus theRDC hit logic ensures that the newly launched kernel does notconsume a previous kernelsrsquo stale data In the rare event thatthe EPCTR rolls over on an increment the memory controllerphysically resets all RDC cachelines by setting the valid bitand EPCTR to zero Note that an RDC architected in dense

DRAM technology enables large counters per RDC line thatotherwise were impractical with similar mechanisms appliedto on-chip caches [43] [44]

Efficient RDC Dirty Data Flush We also desire an efficientmechanism to flush dirty lines to remote GPU memory at kernelboundaries For a writeback RDC we would need to knowwhich RDC lines must be flushed Since our RDC invalidationmechanism no longer requires physically reading all RDC linesan alternate mechanism is necessary to locate dirty RDC linesWe can use a dirty-map [45] which tracks RDC regions thathave been written to On a kernel boundary the dirty-map canbe used to determine which RDC regions should be flushed toremote GPU memory To avoid the on-chip storage overheadfor the dirty-map we propose a write-through RDC insteadDoing so ensures that dirty data is immediately propagated toremote GPU memory Note that the write bandwidth to remoteGPU memory with a write-through RDC is identical to thebaseline NUMA-GPU configuration with no RDC

Our evaluations with writeback (with dirty-map) and write-through RDC showed that a write-through RDC performs nearlyas well (within 1 performance) as a write-back RDC Thisis because the vast majority of remote data cached in the linegranularity RDC tends to be heavily read-biased (see Figure 4)Consequently for the remainder of this work we assume awrite-through RDC to enable zero-latency dirty data flush andavoid the complexity of maintaining a write-back RDC

Software Coherence Results We now evaluate performanceof extending software coherence to the RDC We refer to thisdesign as CARVE with Software Coherence (CARVE-SWC)Figure 11 compares the performance of CARVE-SWC toCARVE with no coherence overhead (CARVE-No-Coherence)The figure shows that CARVE-SWC enables XSBench toperform similar to CARVE-No-Coherence However despiteeliminating all overheads to maintain software coherence forlarge caches CARVE-SWC removes all performance benefitsof RDC for the remaining workloads Since the only differencebetween CARVE-No-Coherence and CARVE-SWC is theflushing of the RDC between kernels these results suggestsignificant inter-kernel data locality that is exploited by CARVE-No-Coherence While this motivates further research work onimproving CARVE-SWC by efficiently prefetching remotedata at kernel boundaries this is out of the scope of this paperInstead we now investigate hardware coherence mechanismsthat do not require flushing the RDC at kernel boundaries

Hardware CoherenceCARVE-SWC results reveal that it is important to retainRDC data across kernel boundaries While software coherencecan potentially be relaxed to avoid flushing the RDC itwould require additional complexity for maintaining a softwaredirectory to track stale copies of shared data cached remotelyand invalidate them explicitly Since similar functionality isavailable in conventional hardware coherence mechanisms weinvestigate extending the RDC with hardware coherence instead

7

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

Fig 6 Architecting a small fraction of GPU memory as a Remote DataCache (RDC) enables caching large remote data working-sets

IV INCREASING GPU CACHE CAPACITY

The NUMA bottleneck occurs when a large fraction of memoryaccesses are serviced by remote memory over low-bandwidthinterconnection links [12] [35] [36] While NUMA has beenwell studied using conventional memory technology [14] [30][37] emerging high bandwidth memory technology enables anovel solution space to tackle the NUMA problem As such weinvestigate hardware mechanisms that dedicate a small fractionof local memory to store the contents of remote memory Doingso enables the remote memory data to be serviced at high localmemory bandwidth In the context of GPUs this approacheffectively increases the caching capacity of a GPU

A Caching Remote Data in Video Memory

We propose Caching Remote Data in Video Memory (CARVE)a hardware mechanism that statically reserves a portion of theGPUs on-package local memory to store remote data Sincethere is no latency or bandwidth advantage to duplicating localmemory data CARVE only stores remote data in the carve-outWe refer to this memory region as a Remote Data Cache (RDC)(see Figure 6) To avoid software overheads the RDC is entirelyhardware-managed and invisible to the programmer and GPUruntime system The amount of GPU memory dedicated forthe RDC can be statically set at system boot time (or kernellaunch time if only a single kernel is executing on the GPU)Thus CARVE transforms the local GPU memory into a hybridstructure that is simultaneously configured both as softwarevisible memory and a hardware-managed cache

CARVE requires minimal changes to the existing GPUdesign On a GPU LLC miss the GPU memory controllerdetermines whether the missing memory address will beserviced by the local GPU memory or a remote GPU memory Ifthe request maps to the local GPU memory the data is fetcheddirectly from local memory and sent to the LLC Otherwisethe memory controller first checks to see if the data is availablein the RDC In the event of a RDC hit the data is servicedlocally without incurring the bandwidth and latency penaltyof a remote GPU memory access However in the event ofan RDC miss the missing data is retrieved from the remote

DRAM ARRAY

2 KB Row Buffer (plus ECC) =

144B = 128-byte Data + 9-byte ECC + 7-byte (Tag+Metadata)

ADDR

DATA(128B)

TAG AND DATA (TAD)

ECC(9B) + TAG(7B) = (16B)

ROW BUFFER

16 x 144 byte TAD = 16 lines

Fig 7 Alloy Cache stores tags with data to enable low latency access Tags canbe stored alongside ECC bits to simplify data alignment and cache controller

GPU memory and returned to the LLC The missing data isalso inserted into the RDC to enable future hits2

Remote Data Cache (RDC) Design The RDC is architectedat a fine (eg 128B) granularity to minimize false sharing (seeFigure 4) In our study we model the RDC as a fine grainAlloy cache [39] (see Figure 7) which organizes its cache asa direct-mapped structure and stores Tags-with-Data In Alloyone access to the cache retrieves both the tag and the dataIf the tag matches (ie a cache hit) the data is then usedto service the request Otherwise the request is forwarded tothe next level of memory We implement Alloy by storing thetag in spare ECC bits3 similar to commercial implementationtoday [40] Note that RDC is not only limited to the Alloydesign and can also be architected using alternate DRAM cachearchitectures [39] [41] [42]

When GPU memory is configured with an RDC CARVErequires a single register in the GPU memory controller tospecify starting location (ie physical address) for the RDCCARVE also requires simple combinational logic to identify thephysical GPU memory location for a given RDC set Finallysince the RDC is architected in local GPU memory the differentRDC sets are interleaved across all GPU memory channels toensure high bandwidth concurrent RDC accesses

Remote Data Cache Evaluation We evaluate the benefits ofCARVE by using a 2GB RDC This amounts to 625 of thebaseline 32GB GPU memory dedicated as an RDC and the

2The RDC can also store data from System Memory However this requiresadditional support for handling CPU-GPU coherence [38]

3We assume GPU memory uses HBM technology HBM provides 16B of ECCto protect 128B of data Assuming ECC at a 16-byte granularity (enoughto protect bus transfers) SECDED requires 89=72 bits to protect the dataThis leaves 56 spare ECC bits that can be used for tag and metadata (RDConly requires 6 bits for tags)

0

01

02

03

04

05

06

07

08

R

em

ote

Ac

ce

ss

es NUMA-GPU

CARVE

Fig 8 Fraction of remote memory accesses CARVE reduces the averagefraction of remote memory accesses from 40 in NUMA-GPU to 8

5

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

eNUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE (No Coherence) NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 9 CARVE performance with zero overhead cache coherence CARVE is able to achieve Ideal-NUMA-GPU speedup

remaining 9375 exposed as operating system (OS) managedmemory (30GB per GPU)

RDC Effect on NUMA Traffic Figure 8 shows the fractionof remote memory accesses with NUMA-GPU and NUMA-GPU enhanced with CARVE The figure shows that manyworkloads on the baseline NUMA-GPU system still suffermore than 70 remote memory accesses (eg XSBench andLulesh) This is because the shared memory footprint of theseworkloads is larger than the GPU LLC CARVE on the otherhand significantly reduces the fraction of remote memoryaccesses On average NUMA-GPU experiences 40 remotememory accesses while CARVE experiences only 8 remotememory accesses In doing so CARVE efficiently utilizeslocal unused HBM bandwidth Consequently the figure showsthat CARVE can significantly reduce the inter-GPU bandwidthbottleneck for most workloads

RDC Performance Analysis (Upper Bound) We first eval-uate the performance potential of CARVE by assuming zero-overhead cache coherence between the RDCs (called CARVE-No-Coherence) This enables us to quantify the benefits ofservicing remote GPU memory accesses in local GPU memoryWe investigate coherence in the next subsection

Figure 9 shows the performance of NUMA-GPU NUMA-GPU with software page replication of read-only shared pagesCARVE-No-Coherence and an ideal NUMA-GPU system thatreplicates all shared pages The x-axis shows the differentworkloads and the y-axis shows the performance relativeto the ideal NUMA-GPU system The figure shows thatCARVE enables workloads like Lulesh Euler SSSP andHPGMG (which experience significant slowdown with NUMA-GPU and NUMA-GPU enhanced with read-only shared pagereplication) to approach the performance of an ideal NUMA-GPU system This is because these workloads experience a20-60 reduction in remote memory accesses (see Figure 8)This shows that extending the remote data caching capacityinto the local GPU memory by using a Remote Data Cache(RDC) significantly reduces the NUMA bandwidth bottleneckCARVE provides these performance improvements withoutrelying on any software support for page replication

While CARVE improves performance significantly acrossmany workloads CARVE can sometimes degrade performanceFor example RandAccess experiences a 10 performancedegradation due to frequent misses in the RDC The ad-

ditional latency penalty of first accessing the RDC thenaccessing remote memory can degrade performance of latency-sensitive workloads However using low-overhead cachehit-predictors [39] can mitigate these performance outliersOverall Figure 9 shows that CARVE significantly bridgesthe performance gap between NUMA-GPU and an idealNUMA-GPU system On average the baseline NUMA-GPUsystem and NUMA-GPU enhanced with read-only shared pagereplication experience a 50 performance gap relative to theideal NUMA-GPU system On the other hand CARVE-No-Coherence experiences only a 5 performance gap relative toan ideal NUMA-GPU system that replicates all shared pagesThese results show significant performance opportunity fromCARVE provided we can efficiently ensure data coherencebetween the RDCs

B Coherence Implications of an RDC in GPUs

CARVE has the potential to significantly improve NUMA-GPUperformance by reducing most of the remote memory accessesHowever since each Remote Data Cache (RDC) stores a copyof remote GPU memory an efficient mechanism is necessary tokeep the copies coherent Fortunately the GPU programmingmodel provides the programmer with an API to explicitlyinsert synchronization points to ensure coherent data accessThus since software coherence [28] is already supported onconventional GPU systems we first investigate whether CARVEcan be easily extended with software coherence

Software CoherenceConventional GPU designs maintain software coherence atkernel boundaries by enforcing two requirements First at theend of each kernel call modified data is written back to memory(ie flush dirty data) Second at the beginning of the nextkernel invocation the GPU cores must read the updated valuesfrom memory Conventional GPUs enforce these requirementsby implementing a write-through L1 cache that is invalidatedat every kernel boundary and a memory-side last-level cachethat is implicitly coherent

TABLE IVKERNEL-LAUNCH DELAY UNDER SOFTWARE COHERENCE

L2 Cache (8MB) RDC (2GB)Cache Invalidate 8MB 16 bank 1cycle 2GB 1024GBs local

4us 2ms =gt0msFlush Dirty 8MB 1024-64GBs 2GB 64GBs remote

8us˜128us 32ms =gt0ms

6

Fig 10 Epoch-counter based invalidation eliminates the penalty of explicitlyinvalidating the whole remote data cache

NUMA-GPUs cache remote data in the GPU LLC and extendsoftware coherence to the GPU LLC by also invalidating theLLC on kernel boundaries [16] Similarly RDC coherence canbe ensured by further extending software-coherence mecha-nisms to the RDC Thus on every kernel boundary the memorycontroller would also invalidate all RDC entries and write backdirty data (if any) to remote GPU memory Table IV comparesthe worst case software coherence overhead for RDC and on-chip LLCs The table shows that invalidating and flushing dirtydata in on-chip LLCs tends to be on the order of microsecondsand can be tolerated within kernel launch latency On theother hand the overhead of invalidating and writing backmillions of RDC lines can take on the order of millisecondsIncurring millisecond latency at every kernel boundary cansignificantly impact application performance Consequentlywe desire architecture support for efficiently invalidating andwriting back RDC lines

Efficient RDC Invalidation Since the RDC tag valid anddirty bits are all stored in GPU memory invalidating all RDClines requires reading and writing GPU memory Dependingon the RDC size RDC invalidation can be a long latency andbandwidth intensive process Alternatively we do not needto physically invalidate all of the lines We simply need toknow if the RDC data is stale (ie from a previous kernel andshould be re-fetched) or up-to-date (ie from current kerneland can be used) If we track the epoch an RDC cacheline wasinstalled in we can easily determine if the RDC data is staleor not Thus an epoch-counter-based invalidation scheme isused to invalidate the RDC instantly (ie 0 ms) [43] [44]

Figure 10 shows our RDC invalidation mechanism We usea 20-bit register to maintain an Epoch Counter (EPCTR) foreach kernelstream running on the GPU RDC insertions storethe current EPCTR value of the kernelstream with each RDCcacheline (in the ECC bits along with the RDC Tag) Bydoing so RDC hits only occur if the tag matches and if theEPCTR stored within the RDC cacheline matches the currentEPCTR value of the associated kernelstream To supportefficient RDC invalidation at kernel boundaries we simplyincrement the EPCTR of the appropriate kernel Thus theRDC hit logic ensures that the newly launched kernel does notconsume a previous kernelsrsquo stale data In the rare event thatthe EPCTR rolls over on an increment the memory controllerphysically resets all RDC cachelines by setting the valid bitand EPCTR to zero Note that an RDC architected in dense

DRAM technology enables large counters per RDC line thatotherwise were impractical with similar mechanisms appliedto on-chip caches [43] [44]

Efficient RDC Dirty Data Flush We also desire an efficientmechanism to flush dirty lines to remote GPU memory at kernelboundaries For a writeback RDC we would need to knowwhich RDC lines must be flushed Since our RDC invalidationmechanism no longer requires physically reading all RDC linesan alternate mechanism is necessary to locate dirty RDC linesWe can use a dirty-map [45] which tracks RDC regions thathave been written to On a kernel boundary the dirty-map canbe used to determine which RDC regions should be flushed toremote GPU memory To avoid the on-chip storage overheadfor the dirty-map we propose a write-through RDC insteadDoing so ensures that dirty data is immediately propagated toremote GPU memory Note that the write bandwidth to remoteGPU memory with a write-through RDC is identical to thebaseline NUMA-GPU configuration with no RDC

Our evaluations with writeback (with dirty-map) and write-through RDC showed that a write-through RDC performs nearlyas well (within 1 performance) as a write-back RDC Thisis because the vast majority of remote data cached in the linegranularity RDC tends to be heavily read-biased (see Figure 4)Consequently for the remainder of this work we assume awrite-through RDC to enable zero-latency dirty data flush andavoid the complexity of maintaining a write-back RDC

Software Coherence Results We now evaluate performanceof extending software coherence to the RDC We refer to thisdesign as CARVE with Software Coherence (CARVE-SWC)Figure 11 compares the performance of CARVE-SWC toCARVE with no coherence overhead (CARVE-No-Coherence)The figure shows that CARVE-SWC enables XSBench toperform similar to CARVE-No-Coherence However despiteeliminating all overheads to maintain software coherence forlarge caches CARVE-SWC removes all performance benefitsof RDC for the remaining workloads Since the only differencebetween CARVE-No-Coherence and CARVE-SWC is theflushing of the RDC between kernels these results suggestsignificant inter-kernel data locality that is exploited by CARVE-No-Coherence While this motivates further research work onimproving CARVE-SWC by efficiently prefetching remotedata at kernel boundaries this is out of the scope of this paperInstead we now investigate hardware coherence mechanismsthat do not require flushing the RDC at kernel boundaries

Hardware CoherenceCARVE-SWC results reveal that it is important to retainRDC data across kernel boundaries While software coherencecan potentially be relaxed to avoid flushing the RDC itwould require additional complexity for maintaining a softwaredirectory to track stale copies of shared data cached remotelyand invalidate them explicitly Since similar functionality isavailable in conventional hardware coherence mechanisms weinvestigate extending the RDC with hardware coherence instead

7

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

000

020

040

060

080

100R

ela

tive

Pe

rfo

rma

nc

eNUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE (No Coherence) NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 9 CARVE performance with zero overhead cache coherence CARVE is able to achieve Ideal-NUMA-GPU speedup

remaining 9375 exposed as operating system (OS) managedmemory (30GB per GPU)

RDC Effect on NUMA Traffic Figure 8 shows the fractionof remote memory accesses with NUMA-GPU and NUMA-GPU enhanced with CARVE The figure shows that manyworkloads on the baseline NUMA-GPU system still suffermore than 70 remote memory accesses (eg XSBench andLulesh) This is because the shared memory footprint of theseworkloads is larger than the GPU LLC CARVE on the otherhand significantly reduces the fraction of remote memoryaccesses On average NUMA-GPU experiences 40 remotememory accesses while CARVE experiences only 8 remotememory accesses In doing so CARVE efficiently utilizeslocal unused HBM bandwidth Consequently the figure showsthat CARVE can significantly reduce the inter-GPU bandwidthbottleneck for most workloads

RDC Performance Analysis (Upper Bound) We first eval-uate the performance potential of CARVE by assuming zero-overhead cache coherence between the RDCs (called CARVE-No-Coherence) This enables us to quantify the benefits ofservicing remote GPU memory accesses in local GPU memoryWe investigate coherence in the next subsection

Figure 9 shows the performance of NUMA-GPU NUMA-GPU with software page replication of read-only shared pagesCARVE-No-Coherence and an ideal NUMA-GPU system thatreplicates all shared pages The x-axis shows the differentworkloads and the y-axis shows the performance relativeto the ideal NUMA-GPU system The figure shows thatCARVE enables workloads like Lulesh Euler SSSP andHPGMG (which experience significant slowdown with NUMA-GPU and NUMA-GPU enhanced with read-only shared pagereplication) to approach the performance of an ideal NUMA-GPU system This is because these workloads experience a20-60 reduction in remote memory accesses (see Figure 8)This shows that extending the remote data caching capacityinto the local GPU memory by using a Remote Data Cache(RDC) significantly reduces the NUMA bandwidth bottleneckCARVE provides these performance improvements withoutrelying on any software support for page replication

While CARVE improves performance significantly acrossmany workloads CARVE can sometimes degrade performanceFor example RandAccess experiences a 10 performancedegradation due to frequent misses in the RDC The ad-

ditional latency penalty of first accessing the RDC thenaccessing remote memory can degrade performance of latency-sensitive workloads However using low-overhead cachehit-predictors [39] can mitigate these performance outliersOverall Figure 9 shows that CARVE significantly bridgesthe performance gap between NUMA-GPU and an idealNUMA-GPU system On average the baseline NUMA-GPUsystem and NUMA-GPU enhanced with read-only shared pagereplication experience a 50 performance gap relative to theideal NUMA-GPU system On the other hand CARVE-No-Coherence experiences only a 5 performance gap relative toan ideal NUMA-GPU system that replicates all shared pagesThese results show significant performance opportunity fromCARVE provided we can efficiently ensure data coherencebetween the RDCs

B Coherence Implications of an RDC in GPUs

CARVE has the potential to significantly improve NUMA-GPUperformance by reducing most of the remote memory accessesHowever since each Remote Data Cache (RDC) stores a copyof remote GPU memory an efficient mechanism is necessary tokeep the copies coherent Fortunately the GPU programmingmodel provides the programmer with an API to explicitlyinsert synchronization points to ensure coherent data accessThus since software coherence [28] is already supported onconventional GPU systems we first investigate whether CARVEcan be easily extended with software coherence

Software CoherenceConventional GPU designs maintain software coherence atkernel boundaries by enforcing two requirements First at theend of each kernel call modified data is written back to memory(ie flush dirty data) Second at the beginning of the nextkernel invocation the GPU cores must read the updated valuesfrom memory Conventional GPUs enforce these requirementsby implementing a write-through L1 cache that is invalidatedat every kernel boundary and a memory-side last-level cachethat is implicitly coherent

TABLE IVKERNEL-LAUNCH DELAY UNDER SOFTWARE COHERENCE

L2 Cache (8MB) RDC (2GB)Cache Invalidate 8MB 16 bank 1cycle 2GB 1024GBs local

4us 2ms =gt0msFlush Dirty 8MB 1024-64GBs 2GB 64GBs remote

8us˜128us 32ms =gt0ms

6

Fig 10 Epoch-counter based invalidation eliminates the penalty of explicitlyinvalidating the whole remote data cache

NUMA-GPUs cache remote data in the GPU LLC and extendsoftware coherence to the GPU LLC by also invalidating theLLC on kernel boundaries [16] Similarly RDC coherence canbe ensured by further extending software-coherence mecha-nisms to the RDC Thus on every kernel boundary the memorycontroller would also invalidate all RDC entries and write backdirty data (if any) to remote GPU memory Table IV comparesthe worst case software coherence overhead for RDC and on-chip LLCs The table shows that invalidating and flushing dirtydata in on-chip LLCs tends to be on the order of microsecondsand can be tolerated within kernel launch latency On theother hand the overhead of invalidating and writing backmillions of RDC lines can take on the order of millisecondsIncurring millisecond latency at every kernel boundary cansignificantly impact application performance Consequentlywe desire architecture support for efficiently invalidating andwriting back RDC lines

Efficient RDC Invalidation Since the RDC tag valid anddirty bits are all stored in GPU memory invalidating all RDClines requires reading and writing GPU memory Dependingon the RDC size RDC invalidation can be a long latency andbandwidth intensive process Alternatively we do not needto physically invalidate all of the lines We simply need toknow if the RDC data is stale (ie from a previous kernel andshould be re-fetched) or up-to-date (ie from current kerneland can be used) If we track the epoch an RDC cacheline wasinstalled in we can easily determine if the RDC data is staleor not Thus an epoch-counter-based invalidation scheme isused to invalidate the RDC instantly (ie 0 ms) [43] [44]

Figure 10 shows our RDC invalidation mechanism We usea 20-bit register to maintain an Epoch Counter (EPCTR) foreach kernelstream running on the GPU RDC insertions storethe current EPCTR value of the kernelstream with each RDCcacheline (in the ECC bits along with the RDC Tag) Bydoing so RDC hits only occur if the tag matches and if theEPCTR stored within the RDC cacheline matches the currentEPCTR value of the associated kernelstream To supportefficient RDC invalidation at kernel boundaries we simplyincrement the EPCTR of the appropriate kernel Thus theRDC hit logic ensures that the newly launched kernel does notconsume a previous kernelsrsquo stale data In the rare event thatthe EPCTR rolls over on an increment the memory controllerphysically resets all RDC cachelines by setting the valid bitand EPCTR to zero Note that an RDC architected in dense

DRAM technology enables large counters per RDC line thatotherwise were impractical with similar mechanisms appliedto on-chip caches [43] [44]

Efficient RDC Dirty Data Flush We also desire an efficientmechanism to flush dirty lines to remote GPU memory at kernelboundaries For a writeback RDC we would need to knowwhich RDC lines must be flushed Since our RDC invalidationmechanism no longer requires physically reading all RDC linesan alternate mechanism is necessary to locate dirty RDC linesWe can use a dirty-map [45] which tracks RDC regions thathave been written to On a kernel boundary the dirty-map canbe used to determine which RDC regions should be flushed toremote GPU memory To avoid the on-chip storage overheadfor the dirty-map we propose a write-through RDC insteadDoing so ensures that dirty data is immediately propagated toremote GPU memory Note that the write bandwidth to remoteGPU memory with a write-through RDC is identical to thebaseline NUMA-GPU configuration with no RDC

Our evaluations with writeback (with dirty-map) and write-through RDC showed that a write-through RDC performs nearlyas well (within 1 performance) as a write-back RDC Thisis because the vast majority of remote data cached in the linegranularity RDC tends to be heavily read-biased (see Figure 4)Consequently for the remainder of this work we assume awrite-through RDC to enable zero-latency dirty data flush andavoid the complexity of maintaining a write-back RDC

Software Coherence Results We now evaluate performanceof extending software coherence to the RDC We refer to thisdesign as CARVE with Software Coherence (CARVE-SWC)Figure 11 compares the performance of CARVE-SWC toCARVE with no coherence overhead (CARVE-No-Coherence)The figure shows that CARVE-SWC enables XSBench toperform similar to CARVE-No-Coherence However despiteeliminating all overheads to maintain software coherence forlarge caches CARVE-SWC removes all performance benefitsof RDC for the remaining workloads Since the only differencebetween CARVE-No-Coherence and CARVE-SWC is theflushing of the RDC between kernels these results suggestsignificant inter-kernel data locality that is exploited by CARVE-No-Coherence While this motivates further research work onimproving CARVE-SWC by efficiently prefetching remotedata at kernel boundaries this is out of the scope of this paperInstead we now investigate hardware coherence mechanismsthat do not require flushing the RDC at kernel boundaries

Hardware CoherenceCARVE-SWC results reveal that it is important to retainRDC data across kernel boundaries While software coherencecan potentially be relaxed to avoid flushing the RDC itwould require additional complexity for maintaining a softwaredirectory to track stale copies of shared data cached remotelyand invalidate them explicitly Since similar functionality isavailable in conventional hardware coherence mechanisms weinvestigate extending the RDC with hardware coherence instead

7

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

Fig 10 Epoch-counter based invalidation eliminates the penalty of explicitlyinvalidating the whole remote data cache

NUMA-GPUs cache remote data in the GPU LLC and extendsoftware coherence to the GPU LLC by also invalidating theLLC on kernel boundaries [16] Similarly RDC coherence canbe ensured by further extending software-coherence mecha-nisms to the RDC Thus on every kernel boundary the memorycontroller would also invalidate all RDC entries and write backdirty data (if any) to remote GPU memory Table IV comparesthe worst case software coherence overhead for RDC and on-chip LLCs The table shows that invalidating and flushing dirtydata in on-chip LLCs tends to be on the order of microsecondsand can be tolerated within kernel launch latency On theother hand the overhead of invalidating and writing backmillions of RDC lines can take on the order of millisecondsIncurring millisecond latency at every kernel boundary cansignificantly impact application performance Consequentlywe desire architecture support for efficiently invalidating andwriting back RDC lines

Efficient RDC Invalidation Since the RDC tag valid anddirty bits are all stored in GPU memory invalidating all RDClines requires reading and writing GPU memory Dependingon the RDC size RDC invalidation can be a long latency andbandwidth intensive process Alternatively we do not needto physically invalidate all of the lines We simply need toknow if the RDC data is stale (ie from a previous kernel andshould be re-fetched) or up-to-date (ie from current kerneland can be used) If we track the epoch an RDC cacheline wasinstalled in we can easily determine if the RDC data is staleor not Thus an epoch-counter-based invalidation scheme isused to invalidate the RDC instantly (ie 0 ms) [43] [44]

Figure 10 shows our RDC invalidation mechanism We usea 20-bit register to maintain an Epoch Counter (EPCTR) foreach kernelstream running on the GPU RDC insertions storethe current EPCTR value of the kernelstream with each RDCcacheline (in the ECC bits along with the RDC Tag) Bydoing so RDC hits only occur if the tag matches and if theEPCTR stored within the RDC cacheline matches the currentEPCTR value of the associated kernelstream To supportefficient RDC invalidation at kernel boundaries we simplyincrement the EPCTR of the appropriate kernel Thus theRDC hit logic ensures that the newly launched kernel does notconsume a previous kernelsrsquo stale data In the rare event thatthe EPCTR rolls over on an increment the memory controllerphysically resets all RDC cachelines by setting the valid bitand EPCTR to zero Note that an RDC architected in dense

DRAM technology enables large counters per RDC line thatotherwise were impractical with similar mechanisms appliedto on-chip caches [43] [44]

Efficient RDC Dirty Data Flush We also desire an efficientmechanism to flush dirty lines to remote GPU memory at kernelboundaries For a writeback RDC we would need to knowwhich RDC lines must be flushed Since our RDC invalidationmechanism no longer requires physically reading all RDC linesan alternate mechanism is necessary to locate dirty RDC linesWe can use a dirty-map [45] which tracks RDC regions thathave been written to On a kernel boundary the dirty-map canbe used to determine which RDC regions should be flushed toremote GPU memory To avoid the on-chip storage overheadfor the dirty-map we propose a write-through RDC insteadDoing so ensures that dirty data is immediately propagated toremote GPU memory Note that the write bandwidth to remoteGPU memory with a write-through RDC is identical to thebaseline NUMA-GPU configuration with no RDC

Our evaluations with writeback (with dirty-map) and write-through RDC showed that a write-through RDC performs nearlyas well (within 1 performance) as a write-back RDC Thisis because the vast majority of remote data cached in the linegranularity RDC tends to be heavily read-biased (see Figure 4)Consequently for the remainder of this work we assume awrite-through RDC to enable zero-latency dirty data flush andavoid the complexity of maintaining a write-back RDC

Software Coherence Results We now evaluate performanceof extending software coherence to the RDC We refer to thisdesign as CARVE with Software Coherence (CARVE-SWC)Figure 11 compares the performance of CARVE-SWC toCARVE with no coherence overhead (CARVE-No-Coherence)The figure shows that CARVE-SWC enables XSBench toperform similar to CARVE-No-Coherence However despiteeliminating all overheads to maintain software coherence forlarge caches CARVE-SWC removes all performance benefitsof RDC for the remaining workloads Since the only differencebetween CARVE-No-Coherence and CARVE-SWC is theflushing of the RDC between kernels these results suggestsignificant inter-kernel data locality that is exploited by CARVE-No-Coherence While this motivates further research work onimproving CARVE-SWC by efficiently prefetching remotedata at kernel boundaries this is out of the scope of this paperInstead we now investigate hardware coherence mechanismsthat do not require flushing the RDC at kernel boundaries

Hardware CoherenceCARVE-SWC results reveal that it is important to retainRDC data across kernel boundaries While software coherencecan potentially be relaxed to avoid flushing the RDC itwould require additional complexity for maintaining a softwaredirectory to track stale copies of shared data cached remotelyand invalidate them explicitly Since similar functionality isavailable in conventional hardware coherence mechanisms weinvestigate extending the RDC with hardware coherence instead

7

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

000

020

040

060

080

100

Re

lati

ve

Pe

rfo

rma

nc

eNUMA-GPU CARVE-SWC CARVE-HWC CARVE (No Coherence)

Fig 11 Performance of CARVE with software coherence and hardware coherence Software coherence eliminates all of the benefits that CARVE offerHardware coherence enables CARVE to reduce NUMA performance bottlenecks

Since we desire a simple hardware coherence mechanism withminimal design complexity we augment our RDC design withthe simple GPU-VI cache coherence protocol [46]

GPU-VI is a directory-less protocol that implements (a)write-through caches and (b) broadcasts write-invalidates toall remote caches on store requests Unfortunately sending awrite-invalidate on every store request can significantly increasenetwork traffic in the system In general write-invalidatesare only required for read-write shared cachelines and canbe avoided for private read-write cachelines Thus write-invalidate traffic can be reduced by identifying read-writeshared cachelines To that end we dynamically identify read-write shared cachelines using an In-Memory Sharing Tracker(IMST) for every memory location at a cacheline granularityWe propose a 2-bit IMST that is stored in the spare ECCspace of each cacheline at the home node Figure 12 shows thefour possible states tracked by the IMST uncached privateread-shared and read-write shared

Figure 12 also shows the possible transitions between thedifferent sharing states of a cacheline These transitions areperformed by the GPU memory controller on reads and writesfrom local and remote GPUs For example a remote readtransitions the sharing state to read-shared Similarly a remotewrite transitions the state to read-write shared Since the IMSTtracks the global sharing behavior of a cacheline a cachelinecan perpetually stay in the read-write shared or read-sharedstate To avoid this problem we probabilistically (eg 1)transition the sharing state to private on local GPU writes(after broadcasting invalidates) Note that the IMST differsfrom conventional cache coherence states like MESIMOESIwhich track the instantaneous cacheline state during residencyin the cache The IMST on the other hand monitors the globalsharing behavior of a cacheline beyond cache residency

Local ReadWrite orRemote ReadWrite

Remote Read

Local ReadWriteLocalRemote Read

LocalRemote Write

Read-only or read-write lineonly used by local GPU

Read-only line shared by multiple GPUs

Read-write lineshared by multiple GPUsUn-cached line

private

Remote WriteRemote Read

Local Write

Local Write RW-shared

R-shared

Probabilistic Update to trackdynamic cacheline behavior

Remote Write

un-cached

Local ReadWrite

Fig 12 In-Memory Sharing Tracker (IMST) Identifying read-write sharedcachelines reduces write-invalidate traffic in GPU-VI The IMST is storedin-memory at a cacheline granularity in spare ECC bits at the home node

We refer to RDC extended with GPU-VI coherence andIMST as CARVE with Hardware Coherence (CARVE-HWC)CARVE-HWC leverages the IMST to avoid write-invalidatesfor private cachelines When the GPU memory controller atthe home node receives a write request it consults the IMSTto determine the sharing properties of the cacheline To avoidbandwidth and latency for reading an IMST entry we storea copy of the IMST-entry along with the cacheline in theGPU cache hierarchy Thus when the IMST-entry identifiesthe cacheline as private a write-invalidate broadcast is avoidedHowever if the cacheline is detected to be in read-write sharingstate a write-invalidate broadcast is generated In our work wefind that CARVE-HWC introduces negligible write-invalidatetraffic from read-write shared lines since the fine grain RDCenables the majority of memory accesses to reference eitherprivate data or read-only shared data (as evident from Figure 4)

Hardware Coherence Results Figure 11 shows that CARVE-HWC restores RDC benefits lost under CARVE-SWC (egLulesh Euler and HPGMG) In fact CARVE-HWC nearsideal system performance which replicates all shared pages

C CARVE Summary

CARVE significantly improves NUMA-GPU performance byextending the caching capacity of the GPU cache hierarchyHowever CARVE requires an efficient and high-performancecache coherence mechanism to ensure correct data executionOur investigations in this section show that conventionalsoftware coherence mechanisms do not scale with increasingGPU caching capacity This is because flushing the cachehierarchy between kernel boundaries removes all inter-kerneldata locality benefits offered by a high capacity cache hierarchyConsequently we find that hardware coherence is necessary toreap the performance benefits of a high capacity GPU cachehierarchy Our performance evaluations corroborate with resultsfrom previous work on GPU L1 caches [47] [48] [49] andconclude that implementing hardware coherence may be aworthwhile endeavor in future NUMA-GPU systems

V RESULTS AND ANALYSIS

Thus far we have shown CARVE can significantly improveNUMA-GPU performance However the performance ofNUMA-GPU enhanced with CARVE depends on the RemoteData Cache size the NUMA-bandwidth differential between

8

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

000

100

200

300

400

500P

erf

Rela

tive t

o S

ing

le G

PU

NUMA-GPU NUMA-GPU + Replicating-ReadOnly-SharedPages CARVE-HWC NUMA-GPU + Replicating-ALL-SharedPages (Ideal)

Fig 13 Performance comparison of CARVE with software support for replicating shared pages CARVE outperforms read-only shared page replication andnears the performance of an ideal NUMA-GPU system that replicates all pages locally

local and remote nodes as well as the potential impact offorcing some application data into the slower system memory(to make room for the Remote Data Cache) This sectionanalyzes the trade-offs associated with these issues as well asexplores the performance sensitivity of our proposals to eachof these factors For the remainder of this section we onlypresent CARVE-HWC since it performs the best

A Results Summary

Figure 13 compares the performance of NUMA-GPU NUMA-GPU enhanced with page replication of read-only sharedpages NUMA-GPU enhanced with CARVE and an idealNUMA-GPU system that replicates all shared pages The x-axis shows the different workloads while the y-axis representsthe performance speedup relative to a single GPU Overall weobserve that the baseline NUMA-GPU system only achievesa 25x speedup relative to a single-GPU system NUMA-GPU enhanced with software support to replicate read-onlyshared pages achieves a 275x speedup However NUMA-GPU enhanced with CARVE outperforms both systems byproviding a 36x speedup This is because CARVE services thecontents of remote read-only and read-write shared pages fromthe local GPU memory In doing so NUMA-GPU enhancedwith CARVE nears the performance of an ideal NUMA-GPUsystem that provides a 37x speedup These results showthat CARVE significantly improves NUMA-GPU performancewithout relying on any software support for page replication

B Sensitivity to Remote Data Cache Size

With any caching proposal the question of appropriate cachesize comes into question particularly in a proposal like CARVEwhere the cache capacity could be sized to consume as littleor as much of the GPU memory size Table V(a) sheds lighton this question by illustrating the performance sensitivity ofCARVE across a variety of remote data cache (RDC) sizeswhere NUMA speed-up is multi-GPU speed-up relative to asingle-GPU system We investigate four RDC sizes per GPU05GB 1GB 2GB and 4GB On average we observe thatCARVE has minimal sensitivity to RDC size suggesting thatdedicating only 15 of total GPU memory capacity to remotedata caches (ie 2GB total out of 128GB total capacity inour system) can improve performance an additional 38 overan existing NUMA-GPU However we observed that someworkloads like XSBench MCB and HPGMG can observe anadditional 40-75 speedup when using an aggregate 8GB

RDC This suggests that a runtime mechanism to decide theappropriate RDC size will be an important factor for furthermulti-GPU performance optimization

C Impact of GPU Memory Capacity Loss

NUMA-GPUs enable scaling unmodified GPU applicationsacross multiple GPUs without requiring any programmerinvolvement Since the application footprint remains the samedespite scaling across multiple GPUs the aggregate systemhas an abundance of available on-package GPU memorycapacity Consequently carving out a small fraction of localGPU memory has no impact on the applicationrsquos ability tofit inside the memory of the NUMA-GPU However whenapplication designers optimize their workloads for NUMA-GPUsystems and increase their workload footprints to maximize allavailable GPU memory capacity this condition may no longerhold true When generalizing the CARVE proposal to multi-GPU systems where the application is hand optimized in bothapplication footprint and memory placement CARVE may infact force some fraction of the application footprint to spill intosystem memory resulting in GPU memory over-subscriptionthat must be handled by software managed runtime systemslike NVIDIArsquos Unified Memory [15]

To quantify this phenomenon Table V(b) shows the geomet-ric mean slowdown across all workloads when 0 15 36 and 12 of the application memory footprint is placedin system memory under a Unified Memory like policy Weobserve that on average a small GPU memory carve-out (15)has minimal performance degradation (1) while increasedRDC sizes begin to substantially hurt workload throughput Theperformance of software based paging systems between systemmemory and GPU memory is an active area of research withperformance improving rapidly [38] [50] Because softwarepaging between system memory and GPU memory focuseson the cold end of the application data footprint whileCARVE focuses on the hottest shared portion we believe

TABLE VPERFORMANCE SENSITIVITY TO RDC SIZE

Aggregate (a) NUMA (b) SlowdownMemory Speed-up due to

Carve-Out (over 1-GPU) Carve-OutNUMA-GPU 000 253x 100x

CARVE-05GB 15 350x 096xCARVE-1GB 312 355x 094xCARVE-2GB 625 361x 083xCARVE-4GB 125 365x 076x

9

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

000

050

100

150

200

250

300

350

400

32GBs 64GBs 128GBs 256GBs

Sp

ee

du

p

Baseline Replicate-ReadOnly CARVE Replicate-All (Ideal)

(default config)

Fig 14 Performance sensitivity of NUMA-GPU to different inter-GPU linkbandwidths normalized to single GPU We illustrate baseline NUMA-GPUNUMA-GPU enhanced with replicating read-only shared pages CARVE andan ideal system that replicates all shared pages

these techniques may remain largely orthogonal though clearlythey may have some intersection in implementation

Taking these trade-offs into account we conclude thatCARVE is likely to be a worthwhile trade-off in the face ofgrowing NUMA effects in multi-GPU systems While CARVEhas a high potential for reducing NUMA traffic CARVEmay degrade performance in situations where the entire GPUmemory is required by the application Similar to dynamiccaching solutions better understanding of this trade-off andhow rapidly the dynamic balance between these two opposingforces can be balanced will be key to designing an optimalmulti-GPU memory system

D Sensitivity to Inter-GPU Link Bandwidth

Inter-GPU link bandwidth is one of the largest performancecritical factors in future multi-GPU systems While GPU toGPU connections continue to improve [12] [51] ultimatelythe link bandwidth between GPUs will always trail the localGPU memory bandwidth To understand the importance ofremote data caching as the ratio of inter-GPU to local memorybandwidth changes we examine the performance gains ofCARVE under a variety of inter-GPU bandwidths in Figure 14In the figure x-axis varies the single direction inter-GPUlink bandwidth while the y-axis illustrates geometric meanof speedups across all workloads evaluated in this study

We observe that the baseline NUMA-GPU performancedepends directly on the available inter-GPU link bandwidthThis is because existing NUMA-GPU design (despite cachingremote data in the GPU LLC) is unable to entirely capture theshared working set of most workloads Conversely CARVEresults in a system that is largely insensitive to a range ofNUMA-GPU bandwidths performing very close to an ideal-NUMA-GPU with infinite NUMA bandwidth in all cases Thisis because CARVE dedicates local memory space to storeremote data and hence converts nearly all remote accesses tolocal accesses In doing so CARVE eliminates the NUMAbottleneck with a minor decrease in GPU memory capacity

As multi-GPU systems scale the NUMA-bandwidth avail-able is unlikely to grow at the same rate as local memorybandwidth Despite improvements in high speed signalingtechnology [26] there simply is not enough package perimeteravailable to enable linear scaling of NUMA bandwidth whendoubling or quadrupling the number of closely coupled GPUs in

a system So while CARVE is still able to improve performanceas the ratio of NUMA to local memory bandwidth grows it isworth noting that CARVErsquos relative performance will actuallyincrease should the NUMA link get slower relative to localmemory bandwidth (as shown by moving from 64GBs to32GBs links in Figure 14)

E Scalability of CARVE

NUMA-GPU problems exacerbate as the number of nodesin a multi-GPU system increase In such situations CARVEcan scale to arbitrary node counts by caching remote shareddata locally in the RDC Since the addition of each GPUnode increases total GPU memory capacity the total memorycapacity loss from CARVE is minimal However increasingnode counts require an efficient hardware coherence mechanismA directory-less hardware coherence mechanism (like the oneused in this paper) can incur significant network traffic overheadfor large multi-node systems that experience frequent read-writesharing In such scenarios a directory-based hardware coher-ence mechanism may be more efficient [19] [20] Dependingon the workload sharing behavior this suggests continuedresearch on efficient hardware coherence mechanisms in multi-node CARVE-enabled GPU systems

VI RELATED WORK

Before drawing conclusions we now compare our work toexisting work on multi-GPUs Non-Uniform Memory Accessand giga-scale DRAM cache architectures

A Multi-GPUs Systems and Optimization

Multi-GPUs are already used to scale GPU performance for awide range of workloads [5] [6] [7] [32] A common methodis to use system-level integration [3] [4] [9] [10] Multi-nodeGPU systems have also been studied and are employed inthe context of high-performance computing and data-centerapplications [52] [53] [54] [55] However programming multi-GPU systems require explicit programmer involvement usingsoftware APIs such as Peer-2-Peer access [56] or a combinationof MPI and CUDA [57] This paper explores a transparentMulti-GPU approach that enables a multi-GPU to be utilized asa single GPU (which enables running unmodified single-GPUcode) and uses hardware and driver level support to maintainperformance [16] [58] [59] [60]

B Techniques to Reduce NUMA Effects

Existing work has also investigated caching remote data inlocal caches to reduce NUMA traffic CC-NUMA [30] S-COMA [14] Reactive NUMA [37] use different mechanismsand granularity for caching remote memory in on-chip cachesCC-NUMA stores remote data in fine-granularity on-chipcaches S-COMA stores remote data at page-granularity inmemory with software support and R-NUMA switches betweenfine-granularity and page-granularity

Our work has a similar goal to prior NUMA proposalsas we also target reducing remote traffic by storing remote

10

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

memory contents locally However we find that shared working-set sizes are much larger than conventional GPU last-levelcaches Unfortunately semiconductor scaling will not allow forlarger on-chip caches to be incorporated into high-performanceprocessor chips Additionally we desire to reduce programmereffort in software-based page replication Our insight is that wecan dedicate a portion of the existing GPU memory to store thecontents of remote memory CARVE can be implemented withminimal additional hardware while maintaining programmerand software transparency

Software page replication can reduce NUMA effects byreplicating pages on each remote node [13] [14] Carrefour [13]proposes to solve NUMA problems with software-based pagereplication and migration If accesses are heavily read biased(gt99) it uses page replication to service requests to shareddata If there is heavy memory imbalance it uses page migrationto maintain similar bandwidth out of all memories Whilesoftware-based techniques can work they rely on softwaresupport for page protections Changing page protection ofshared pages can incur high latency to collapse the replicatedpages [13] Finally page replication can significantly reduceGPU memory capacity which tends to be limited and costly

C DRAM Cache Architectures

Emerging high bandwidth memory technology have enabledDRAM caches to be an important research topic [39] [41][42] [61] [62] Such work typically targets heterogeneousmemory systems where a high bandwidth memory technology(eg HBM) is used to cache low bandwidth denser memorytechnology (eg DDR PCM 3DXPoint [63]) Our work onthe other hand targets a GPU system that consists of a singlememory technology (eg HBM GDDR) Specifically wetransform the conventional GPU memory into a hybrid structurethat is simultaneously configured as OS-visible memory as wellas a data store for frequently accessed remote shared data

DRAM Caches have also seen industry application withthe High-Bandwidth Cache-Controller (HBCC) in the AMDVega system [64] and MCDRAM in the Intel KnightsLanding (KNL) system [40] AMDrsquos HBCC also supports videomemory to be configured as a page-based cache for systemmemory KNL uses a high-bandwidth memory technologycalled MCDRAM and allows the user to statically configureMCDRAM as cache memory or some combination of bothThe purpose of HBCC and MCDRAM is to reduce accessesto the next level in the heterogeneous memory hierarchy (egDDR or PCM) While CARVE has design similarities toMCDRAM CARVE is designed to reduce inter-GPU remotememory accesses and must also consider inter-GPU coherence

To ensure correct execution of multi-GPU applicationsCARVE also requires support for maintaining coherenceExisting approaches CANDY [19] and C3D [20] proposecoherent DRAM caches for multi-socket CPU systems Theseproposals target maintaining coherence for HBM-based cachesin front of traditional DIMM-based memory CANDY proposesa full coherence directory in the DRAM-cache and cachingcoherence entries in an on-die coherence directory C3D

proposes a clean DRAM cache to simplify coherence andreduce the latency of coherent memory accesses CARVE onthe other hand extends elements of these existing coherenceproposals to a multi-GPU system and also proposes architecturesupport for extending software coherence

VII CONCLUSIONS

GPU performance has scaled well over the past decade dueto increasing transistor density However the slowdown ofMoorersquos Law poses significant challenges for continuous GPUperformance scaling Consequently multi-GPU systems havebeen proposed to help meet the insatiable demand for GPUthroughput and are commonly used in HPC data center andeven workstation class machines For many workloads multi-GPU execution means taking a leap of faith and re-writing theGPU application to support multi-GPU communications onlythen to profile and optimize the application to avoid NUMAperformance bottlenecks With the advent of transparent multi-GPU technology where a single application is spread acrossmultiple GPUs for execution NUMA effects on GPUs mayimpact all GPU workloads not just those explicitly optimizedfor NUMA-GPU systems

NUMA performance bottlenecks are primarily due tofrequent remote memory accesses over the low bandwidthinterconnection network Significant research has investigatedsoftware and hardware techniques to avoid remote memoryaccesses Our evaluations on a multi-GPU system reveal that thecombination of page migration page replication and cachingremote data still incurs significant slowdown relative to anideal NUMA GPU system This is because the shared memoryfootprint tends to be much larger than the GPU LLC and cannot be replicated by software because the shared footprinthas read-write property Thus we show that GPUs must beaugmented with large caches to improve NUMA performanceWe investigate a hardware mechanism that increases GPUcaching capacity by storing recently accessed remote shareddata in a dedicated region of the GPU memory We showthat this mechanism can significantly outperform state-of-the-art software and hardware mechanisms while incurring onlya minor cost to GPU memory capacity We then investigatethe implications of maintaining GPU cache coherence whenincreasing the GPU caching capacity Interestingly we findthat conventional GPU software coherence does not scale tolarge GPU caches As such we show that hardware coherenceis necessary to reap the full benefits of increasing GPU cachingcapacity Overall we show that caching remote shared data in adedicated region of the GPU memory can improve multi-GPUperformance within 6 of an ideal multi-GPU system whileincurring negligible performance impact due to the loss in GPUmemory capacity

ACKNOWLEDGMENTS

We thank Moinuddin Qureshi for providing helpful organiza-tional comments on an earlier draft of this work We also thankSam Duncan Wishwesh Gandhi Hemayet Hossain and LackyShah for their feedback over the course of this work

11

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

REFERENCES

[1] InsideHPC ldquoTop500 shows growing momentum for acceleratorsrdquo 2015accessed 2017-09-19 [Online] Available httpsinsidehpccom201511top500-shows-growing-momentum-for-accelerators

[2] NVIDIA ldquoInside Volta The worldrsquos most advanced data centerGPUrdquo Jun 2017 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallinside-volta

[3] CG Willard A Snell and M Feldman ldquoHPC application supportfor GPU computingrdquo 2015 accessed 2017-09-19 [Online] Availablehttpwwwintersect360comindustryreportsphpid=131

[4] ORN Laboratory ldquoTitan The worldrsquos 1 open science supercomputerrdquo 2016 accessed 2017-09-19 [Online] Available httpswwwolcfornlgovtitan

[5] NVIDIA ldquoNVIDIA cuDNN GPU accelerated deep learningrdquo 2016accessed 2017-09-19 [Online] Available httpsdevelopernvidiacomcudnn

[6] A Lavin ldquoFast algorithms for convolutional neural networksrdquo CoRRvol abs150909308 2015 accessed 2017-09-19 [Online] Availablehttparxivorgabs150909308

[7] K Simonyan and A Zisserman ldquoVery deep convolutional networksfor large-scale image recognitionrdquo CoRR vol abs14091556 2014accessed 2017-09-19 [Online] Available httparxivorgabs14091556

[8] P Bright ldquoMoorersquos law really is dead this timerdquo 2016 accessed2017-09-19 [Online] Available httparstechnicacominformation-technology201602moores-law-really-is-dead-this-time

[9] NVIDIA ldquoNVIDIA DGX-1 The fastest deep learning systemrdquo 2017accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforalldgx-1-fastest-deep-learning-system

[10] NVIDIA ldquoNVIDIA Tesla P100 the most advanced datacenteraccelerator ever builtrdquo 2016 accessed 2017-09-19 [Online]Available httpsimagesnvidiacomcontentpdfteslawhitepaperpascal-architecture-whitepaperpdf

[11] J Standard ldquoHigh bandwidth memory (HBM) dramrdquo JESD235A2015 [Online] Available httpswwwjedecorgsitesdefaultfilesdocsJESD235Apdf

[12] NVIDIA ldquoNVIDIA NVlink high-speed interconnectrdquo 2017 accessed2017-09-19 [Online] Available httpwwwnvidiacomobjectnvlinkhtml

[13] M Dashti A Fedorova J Funston F Gaud R Lachaize B LepersV Quema and M Roth ldquoTraffic management A holistic approach tomemory placement on numa systemsrdquo in ASPLOS rsquo13 New York NYUSA ACM 2013

[14] A Saulasbury T Wilkinson J Carter and A Landin ldquoAn argument forsimple comardquo in HPCA rsquo95 January 1995

[15] NVIDIA ldquoBeyond GPU memory limits with unified memory on Pascalrdquo2016 accessed 2017-09-19 [Online] Available httpsdevblogsnvidiacomparallelforallbeyond-gpu-memory-limits-unified-memory-pascal

[16] U Milic O Villa E Bolotin A Arunkumar E Ebrahimi A JaleelA Ramirez and D Nellans ldquoBeyond the socket NUMA-aware GPUsrdquoin MICRO rsquo17 Oct 2017

[17] F Gaud B Lepers J Decouchant J Funston A Fedorova andV Quema ldquoLarge pages may be harmful on NUMA systemsrdquo in USENIX-ATC rsquo14 Berkeley CA USA USENIX Association 2014

[18] R Ausavarungnirun J Landgraf V Miller S Ghose J Gandhi CJRossbach and O Mutlu ldquoMosaic A GPU memory manager withapplication-transparent support for multiple page sizesrdquo in MICRO rsquo172017

[19] C Chou A Jaleel and MK Qureshi ldquoCANDY Enabling coherentDRAM caches for multi-node systemsrdquo in MICRO rsquo16 Oct 2016

[20] CC Huang R Kumar M Elver B Grot and V Nagarajan ldquoC3DMitigating the NUMA bottleneck via coherent DRAM cachesrdquo in MICRO

rsquo16 Oct 2016[21] P Bright ldquoThe twinscan nxt1950i dual-stage immersion

lithography systemrdquo 2016 accessed 2017-09-19 [Online] Avail-able httpswwwasmlcomproductssystemstwinscan-nxttwinscan-nxt1950iens46772dfp product id=822

[22] A Kannan NE Jerger and GH Loh ldquoEnabling interposer-baseddisintegration of multi-core processorsrdquo in MICRO rsquo15 Dec 2015

[23] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[24] M Mansuri JE Jaussi JT Kennedy TC Hsueh S Shekhar G Bal-

amurugan F OrsquoMahony C Roberts R Mooney and B Casper ldquoAscalable 0128-to-1tbs 08-to-26pjb 64-lane parallel io in 32nm cmosrdquoin ISSCC rsquo13 Feb 2013

[25] D Dreps ldquoThe 3rd generation of IBMrsquos elastic interface on POWER6TMrdquo2007 IEEE Hot Chips 19 Symposium (HCS) vol 00 2007

[26] JW Poulton WJ Dally X Chen JG Eyles TH Greer SG Tell JMWilson and CT Gray ldquoA 054 pjb 20 gbs ground-referenced single-ended short-reach serial link in 28 nm cmos for advanced packagingapplicationsrdquo IEEE Journal of Solid-State Circuits vol 48 Dec 2013

[27] A Arunkumar E Bolotin B Cho U Milic E Ebrahimi O VillaA Jaleel CJ Wu and D Nellans ldquoMCM-GPU Multi-Chip-ModuleGPUs for continued performance scalabilityrdquo in ISCA rsquo17 New YorkNY USA ACM 2017

[28] NVIDIA ldquoNVIDIArsquos next generation CUDA compute ar-chitecture Fermirdquo 2009 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFfermi white papersNVIDIA Fermi Compute Architecture Whitepaperpdf

[29] NVIDIA ldquoNVIDIArsquos next generation CUDA compute archi-tecture Kepler gk110rdquo 2012 accessed 2017-09-19 [Online]Available httpswwwnvidiacomcontentPDFkeplerNVIDIA-Kepler-GK110-Architecture-Whitepaperpdf

[30] P Stenstrom T Joe and A Gupta ldquoComparative performance evaluationof cache-coherent NUMA and COMA architecturesrdquo in ISCA rsquo92 1992

[31] D Kaseridis J Stuecheli and LK John ldquoMinimalist Open-page ADRAM page-mode scheduling policy for the many-core erardquo in MICRO

rsquo11 New York NY USA ACM 2011[32] LLN Laboratory ldquoCoral benchmarksrdquo 2014 accessed 2017-09-19

[Online] Available httpsascllnlgovCORAL-benchmarks[33] S Che M Boyer J Meng D Tarjan JW Sheaffer SH Lee and

K Skadron ldquoRodinia A benchmark suite for heterogeneous computingrdquoin ISSWC rsquo09 Washington DC USA IEEE Computer Society 2009

[34] M Kulkarni M Burtscher C Cascaval and K Pingali ldquoLonestar Asuite of parallel irregular programsrdquo in ISPASS rsquo09 April 2009

[35] Intel ldquoAn introduction to the Intel RcopyQuickPath Inter-connectrdquo 2009 accessed 2017-09-19 [Online] Avail-able httpswwwintelcomcontentwwwusenioquickpath-technologyquick-path-interconnect-introduction-paperhtml

[36] TH Consortium ldquoMeeting the io bandwidth challenge Howhypertransport technology accelerates performance in key applicationsrdquo2002 accessed 2017-09-19 [Online] Available httpwwwopteronicscompdfHTC WP01 AppsOverview 1001pdf

[37] B Falsafi and DA Wood ldquoReactive NUMA A design for unifyingS-COMA and CC-NUMArdquo in ISCA rsquo97 New York NY USA ACM1997

[38] N Agarwal D Nellans E Ebrahimi TF Wenisch J Danskin andSW Keckler ldquoSelective GPU caches to eliminate CPU-GPU HW cachecoherencerdquo in HPCA rsquo16 March 2016

[39] MK Qureshi and GH Loh ldquoFundamental latency trade-off in archi-tecting dram caches Outperforming impractical sram-tags with a simpleand practical designrdquo in MICRO rsquo12 IEEE Computer Society 2012

[40] A Sodani R Gramunt J Corbal HS Kim K Vinod S ChinthamaniS Hutsell R Agarwal and YC Liu ldquoKnights Landing Second-generation Intel Xeon Phi productrdquo IEEE Micro vol 36 Mar 2016

[41] D Jevdjic S Volos and B Falsafi ldquoDie-stacked DRAM caches forservers Hit ratio latency or bandwidth have it all with footprint cacherdquoin ISCA rsquo13 New York NY USA ACM 2013

[42] D Jevdjic GH Loh C Kaynak and B Falsafi ldquoUnison cache Ascalable and effective die-stacked dram cacherdquo in MICRO rsquo14 IEEE2014

[43] X Yuan R Melhem and R Gupta ldquoA timestamp-based selectiveinvalidation scheme for multiprocessor cache coherencerdquo in ICPP rsquo96vol 3 IEEE 1996

[44] SL Min and JL Baer ldquoDesign and analysis of a scalable cachecoherence scheme based on clocks and timestampsrdquo IEEE Transactionson Parallel and Distributed Systems vol 3 1992

[45] J Sim GH Loh H Kim M OrsquoConnor and M Thottethodi ldquoA mostly-clean dram cache for effective hit speculation and self-balancing dispatchrdquoin MICRO rsquo12 IEEE 2012

[46] I Singh A Shriraman WWL Fung M OrsquoConnor and TM AamodtldquoCache coherence for GPU architecturesrdquo in HPCA rsquo13 2013

[47] BA Hechtman S Che DR Hower Y Tian BM Beckmann MDHill SK Reinhardt and DA Wood ldquoQuickRelease A throughput-oriented approach to release consistency on GPUsrdquo in HPCA rsquo14 Feb2014

12

[48] MD Sinclair J Alsop and SV Adve ldquoEfficient GPU synchronizationwithout scopes Saying no to complex consistency modelsrdquo in MICROrsquo15 Dec 2015

[49] J Alsop MS Orr BM Beckmann and DA Wood ldquoLazy releaseconsistency for GPUsrdquo in MICRO rsquo16 Oct 2016

[50] T Zheng D Nellans A Zulfiqar M Stephenson and SW KecklerldquoTowards high performance paged memory for GPUsrdquo in HPCA rsquo16March 2016

[51] NVIDIA ldquoNVIDIA nvswitch the worlds highest-bandwidth on-node switchrdquo 2018 accessed 2018-08-31 [Online] Availablehttpsimagesnvidiacomcontentpdfnvswitch-technical-overviewpdf

[52] Intel ldquoThe Xeon X5365rdquo 2007 accessed 2017-09-19 [Online]Available httparkintelcomproducts30702Intel-Xeon-Processor-X5365-8M-Cache-3 00-GHz-1333-MHz-FSB

[53] IBM ldquoIBM zEnterprise 196 technical guiderdquo 2011 accessed 2017-09-19 [Online] Available httpwwwredbooksibmcomredbookspdfssg247833pdf

[54] IBM ldquoIBM Power Systems deep diverdquo 2012 accessed2017-09-19 [Online] Available httpwww-05ibmcomczeventsfebannouncement2012pdfpower architecturepdf

[55] AMD ldquoAMD server solutions playbookrdquo 2012 accessed 2017-09-19[Online] Available httpwwwamdcomDocumentsAMD OpteronServerPlaybookpdf

[56] NVIDIA ldquoMulti-GPU programmingrdquo 2011 accessed 2017-09-19[Online] Available httpwwwnvidiacomdocsIO116711sc11-multi-

gpupdf[57] NVIDIA ldquoMPI solutions for GPUsrdquo 2016 accessed 2017-09-19

[Online] Available httpsdevelopernvidiacommpi-solutions-gpus[58] T Ben-Nun E Levy A Barak and E Rubin ldquoMemory access patterns

the missing piece of the multi-gpu puzzlerdquo in SC rsquo15 Nov 2015[59] J Cabezas L Vilanova I Gelado TB Jablin N Navarro and WmW

Hwu ldquoAutomatic parallelization of kernels in shared-memory multi-gpunodesrdquo in ICS rsquo15 New York NY USA ACM 2015

[60] J Lee M Samadi Y Park and S Mahlke ldquoTransparent cpu-gpucollaboration for data-parallel kernels on heterogeneous systemsrdquo inPACT rsquo13 Piscataway NJ USA IEEE Press 2013

[61] V Young PJ Nair and MK Qureshi ldquoDICE Compressing DRAMcaches for bandwidth and capacityrdquo in ISCA rsquo17 New York NY USAACM 2017

[62] V Young C Chou A Jaleel and MK Qureshi ldquoAccord Enablingassociativity for gigascale dram caches by coordinating way-install andway-predictionrdquo in ISCA rsquo18 June 2018

[63] Intel and Micron ldquoIntel and Micron produce breakthroughmemory technologyrdquo 2015 accessed 2017-09-19 [Online]Available httpsnewsroomintelcomnews-releasesintel-and-micron-produce-breakthrough-memory-technology

[64] AMD ldquoRadeonrsquos next-generation vega architecturerdquo 2017 accessed2017-09-19 [Online] Available httpradeoncom downloadsvega-whitepaper-11617pdf

13