volunteer computing with gpus david p. anderson space sciences laboratory u.c. berkeley
TRANSCRIPT
Volunteer Computing with GPUs
David P. Anderson
Space Sciences LaboratoryU.C. Berkeley
Outline
• Volunteer computing
• BOINC
• Supporting GPUs in BOINC
• BOINC projects using GPUs
• Problems and issues
Volunteer computing
• Early projects
– 1997: GIMPS, distributed.net
– 1999: SETI@home, Folding@home
• Today
– 50 projects
– 500K volunteers
– 900K computers
– 10 PetaFLOPS
Other types of high-throughput computing
• Clusters– volunteer computing is 10x-100x cheaper
• Clouds– volunteer computing is 100x-1000x cheaper
– but it’s cost-effective to run a volunteer computing server in a cloud
• Grids
– same idea as volunteer computing, but within/among organizations
GPUs and volunteer computing
• Current PetaFLOPS breakdown:
• Potential: ExaFLOPS by 2010– 4M GPUs * 1 TFLOPS * 0.25 availability
Processor type0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5 4.6
2.42.2
1.2
NVIDIACPUPS3 (Cell)ATI
BOINC
• Middleware for volunteer computing
– client, server, web
• Based at UC Berkeley Space Sciences Lab
• Open source (LGPL)
• NSF-funded since 2002
• http://boinc.berkeley.edu
BOINC: volunteers and projects
volunteers projects
CPDN
LHC@home
WCGattachments
The BOINC computing ecosystem
• Better research should get more computing power
• Public is the Decider
The world’scomputing
power
Currentscientificresearch
The public
Science areas using BOINC• Biology
– protein study, genetic analysis• Medicine
– drug discovery, epidemiology• Physics
– LHC, nanotechnology, quantum computing• Astronomy
– data analysis, cosmology, galactic modeling• Environment
– climate modeling, ecosystem simulation• Math• Graphics rendering
Application types
• Computing-intensive analysis of large data
• Physical simulations
• Genetic algorithms
– GA-inspired optimization
• Sensor networks
– Quake-Catcher Network: distributed seismography
Software overview
client
apps
screensaver
GUI
scheduler
MySQL
data server
daemons
volunteer host
project serverHTTP
Client: job scheduling
• Queue lots of jobs– to avoid starvation– for variety
• Job scheduling– Round-robin time-slicing– Earliest deadline first
Client: work fetch policy
• When? Who? How much?• Goals
– maintain enough work– minimize scheduler requests– honor resource shares
• per-project debt
CPU 0
CPU 3
CPU 2
CPU 1
maxmin
Scheduling: server
• Possible outcomes of a job:– success– runs but returns wrong answer– doesn’t run, returns wrong answer (hacker)– crashes, client reports it– never hear from client again
• Job delay bounds• Replicated computing
– homogeneous replication
Good estimates matter
• If overestimate:– won’t send work to some computers– some computers will run out of work
• If underestimate:– wasted computation– volunteers may not get credit for work done
Server abstractionsapplications
Win32
Win64
Mac OS X
app versions
jobs
instances
Scheduler overview
MySQL
feederschedulers
share-memoryjob cache
client
GPUs: the general situation (client)
• One or more GPUs
– possibly different RAM, speed, compute capability etc.
– some driver version
• CPUs, memory
• Operating system
• Availability info
• Attached to a set of projects with various resource shares
GPUs: the general situation (server)
• Possibly multiple versions of a given app for a given platform. Each version may have:
– HW/SW requirements
– resource usage (#GPUs, #CPUs)
– host-specific performance characteristics
• Jobs with varying FLOPs estimates and deadlines
• Things may change over time
The general situation: issues
• Client:
– how does work fetch policy change?
– how to describe GPU resources?
– how to schedule GPU (and other) jobs?
• Server:
– how to decide what jobs to send and what app version to use?
– how to estimate job runtime?
Representing GPU resources
• Hosts may have multiple GPUs with different characteristics
• But most don’t, so represent GPU resources as (count, characteristics)
Scheduling GPU jobs
• CPU and main memory resources are virtualized, but GPU is not
• Job scheduling for GPUs:
– EDF if in danger of deadline miss, else FIFO
– Preempt by kill
– Wait for exit before start next job
• BOINC client allocates GPUs, tells apps which instance(s) to use (-- device N)
Keeping users happy
• GPU apps ruin GUI response
• User pref: “Don’t use GPU while computer in use” (default: on)
• Ideal:– use non-display GPUs all the time– use display GPU if not running graphics app
(Aero counts as a graphics app)
Work fetch for GPUs: goals
• Separate work buffers for different resource types
• Resource shares apply to combined resource types
Example: projects A, B have same resource share; A has CPU and GPU jobs, B has only GPU jobs
GPU
CPU A
BA
Generalized work fetch policy
• For each resource type
– per-project backoff
– per-project debt• accumulate only while not backed off
• A project’s overall debt is weighted average of resource debts
• Get work from project with highest overall debt
How to choose app version?
• App versions can have project-supplied “planning function”
• Inputs:– host description
• Outputs:
– Whether host can run app version
– Resource usage (#CPUs, #GPUs)
– expected FLOPS
App version selection
• Call planning function for platform’s app versions
• Skip versions that use resources for which no work is being requested
• Use the version with highest expected FLOPS
• Repeat this when a resource request is satisfied
GPUs and anonymous platform
• Anonymous platform mechanism
– volunteer supplies app versions
– why? security, optimization, unsupported platforms
• App version description can include GPU usage
– BOINC client schedules jobs accordingly
– works even if project has no GPU apps
Einstein@home
• Gravitational waves; gravitational pulsars
SETI@home
Milkyway@home
GPUGRID.net
AQUA@home
• D-Wave Systems
• Simulation of “adiabatic quantum algorithms” for binary quadratic optimization
Collatz Conjecture
• even N → N/2• odd N → 3N + 1• always goes to 1?
% hosts with an NVIDIA GPU
Total Windows Linux0
20000
40000
60000
80000
100000
120000
19% 20%
5%
81%
80%
95%
GPUno GPU
Number of GPUs per host
0
5000
10000
15000
20000
25000 23632
126056 37 22 6 1
1234568
Video RAM
0
2000
4000
6000
8000
10000
12000
14000
16000
6308
15002
3595
87
<512 MB512-1023 MB1024-2047 MB2048+ MB
GPU model
Model0
2000
4000
6000
8000
10000
12000
14000
GeForce 8600 GTGeForce 9600 GTGeForce 8800 GTGeForce 9800 GTGeForce GTX 260GeForce 8800 GTSGeForce 8500 GTother
Driver version
Version0
1000
2000
3000
4000
5000
6000
7000
8000
9000
190.xx182.xx186.xx185.xxother
Productivity
Credit/day0
100
200
300
400
500
600
700
800
hosts with GPUhosts without GPU
Future work for NVIDIA
• Preemptive prioritized scheduling of GPU
• Provide mechanisms so that
– BOINC can avoid running GPU apps if memory or processors not available
– BOINC can quit running apps if something else wants GPU
• Fix bug where Windows services can’t see GPUs
Future work for BOINC
• Maintain GPU availability separately
• Add GPUs to simulators
– server simulator
– client simulator
• Add GPUs to homogeneous redundancy framework
Conclusion
GPU computing+ volunteer computing= the future of scientific computing