![Page 1: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/1.jpg)
The Beauty and Joy of Computing
Distributed Computing
http://www.sfgate.com/technology/article/Tile-startup-finds-a-GPS-solution-for-lost-
items-4895278.php!
Tile startup finds a GPS solution for lost items Good: Find your stuff! Bad: How many do you have on you?
UC Berkeley EECS Sr Lecturer SOE
Dan Garcia
![Page 2: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/2.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (2)
Garcia
§ Basics ú Memory ú Network
§ Distributed Computing ú Themes ú Challenges
§ Solution! MapReduce ú How it works ú Our implementation
Lecture Overview
![Page 3: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/3.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (3)
Garcia
Memory Hierarchy Processor
Size of memory at each level
Increasing Distance from
Processor Level 1 Level 2
Level n
Level 3 . . .
Higher
Lower
Levels in memory hierarchy
![Page 4: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/4.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (4)
Garcia
Memory Hierarchy Details § If level closer to Processor, it is:
ú Smaller ú Faster ú More expensive ú subset of lower levels …contains most recently used data
§ Lowest Level (usually disk) contains all available data (does it go beyond the disk?)
§ Memory Hierarchy Abstraction presents the processor with the illusion of a very large & fast memory
![Page 5: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/5.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (5)
Garcia
Networking Basics § source encodes and destination decodes
content of the message § switches and routers use the destination in
order to deliver the message, dynamically
Internet
source destination
Network interface device
Network interface device
![Page 6: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/6.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (6)
Garcia
Networking Facts and Benefits § Networks connect
computers, sub-networks, and other networks. ú Networks connect
computers all over the world (and in space!)
ú Computer networks... support asynchronous and
distributed communication enable new forms of
collaboration
![Page 7: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/7.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (7)
Garcia
Performance Needed for Big Problems § Performance terminology
ú the FLOP: FLoating point OPeration ú “flops” = # FLOP/second is the standard metric for computing power
§ Example: Global Climate Modeling ú Divide the world into a grid (e.g. 10 km spacing) ú Solve fluid dynamics equations for each point & minute
Requires about 100 Flops per grid point per minute
ú Weather Prediction (7 days in 24 hours): 56 Gflops
ú Climate Prediction (50 years in 30 days): 4.8 Tflops
§ Perspective ú Intel Core i7 980 XE Desktop Processor
~100 Gflops Climate Prediction would take ~5 years
www.epm.ornl.gov/chammp/chammp.html
en.wikipedia.org/wiki/FLOPS
![Page 8: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/8.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (8)
Garcia
§ Supercomputing – like those listed in top500.org ú Multiple processors “all in one box / room” from one vendor that
often communicate through shared memory ú This is often where you find exotic architectures
§ Distributed computing ú Many separate computers (each with independent CPU, RAM, HD,
NIC) that communicate through a network Grids (heterogenous computers across Internet) Clusters (mostly homogeneous computers all in one room)
Google uses commodity computers to exploit “knee in curve” price/performance sweet spot
ú It’s about being able to solve “big” problems, not “small” problems faster These problems can be data (mostly) or CPU intensive
What Can We Do? Use Many CPUs!
![Page 9: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/9.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (9)
Garcia
Distributed Computing Themes § Let’s network many disparate machines into
one compute cluster § These could all be the same (easier) or very
different machines (harder) § Common themes
ú “Dispatcher” gives jobs & collects results ú “Workers” (get, process, return) until done
§ Examples ú SETI@Home, BOINC, Render farms ú Google clusters running MapReduce
en.wikipedia.org/wiki/Distributed_computing
![Page 10: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/10.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (10)
Garcia
Peer Instruction
1. Writing & managing SETI@Home is relatively straightforward; just hand out & gather data
2. The majority of the world’s computing power lives in supercomputer centers
12!a) FF!b) FT!c) TF!d) TT!
![Page 11: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/11.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (11)
Garcia
1. The heterogeneity of the machines, handling machines that fail, falsify data. FALSE
2. Have you considered how many PCs + game devices exist? Not even close. FALSE
Peer Instruction Answer
1. Writing & managing SETI@Home is relatively straightforward; just hand out & gather data
2. The majority of the world’s computing power lives in supercomputer centers
12!a) FF!b) FT!c) TF!d) TT!
![Page 12: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/12.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (12)
Garcia
Distributed Computing Challenges § Communication is fundamental difficulty
ú Distributing data, updating shared resource, communicating results, handling failures
ú Machines have separate memories, so need network ú Introduces inefficiencies: overhead, waiting, etc.
§ Need to parallelize algorithms, data structures ú Must look at problems from parallel standpoint ú Best for problems whose compute times >> overhead
en.wikipedia.org/wiki/Embarrassingly_parallel
![Page 13: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/13.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (13)
Garcia
§ Functions as Data § Higher-Order Functions § Useful HOFs (you can build your own!)
ú map Reporter over List! Report a new list, every element E of List becoming Reporter(E)
ú keep items such that Predicate from List! Report a new list, keeping only elements E of List if Predicate(E)!
ú combine with Reporter over List! Combine all the elements of List with Reporter(E)! This is also known as “reduce”!
§ Acronym example ú keep è map è combine
Review
![Page 14: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/14.jpg)
combine with Reporter over List a" b"
c"d"
![Page 15: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/15.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (15)
Garcia
§ We told you “the beauty of pure functional programming is that it’s easily parallelizable” ú Do you see how you could
parallelize this? ú Reducer should be associative
and commutative
§ Imagine 10,000 machines ready to help you compute anything you could cast as a MapReduce problem! ú This is the abstraction Google is
famous for authoring ú It hides LOTS of difficulty of
writing parallel code! ú The system takes care of load
balancing, dead machines, etc.
Google’s MapReduce Simplified en.wikipedia.org/wiki/MapReduce
1 20 3 10
* * * *
1 400 9 100 + +
401 109 +
510 Output:
Input:
Note: only two data
types!
![Page 16: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/16.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (16)
Garcia
MapReduce Advantages/Disadvantages § Now it’s easy to program for many CPUs
ú Communication management effectively gone ú Fault tolerance, monitoring
machine failures, suddenly-slow machines, etc are handled
ú Can be much easier to design and program! ú Can cascade several (many?) MapReduce tasks
§ But … it might restrict solvable problems ú Might be hard to express problem in MapReduce ú Data parallelism is key
Need to be able to break up a problem by data chunks
ú Full MapReduce is closed-source (to Google) C++ Hadoop is open-source Java-based rewrite
![Page 17: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/17.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (17)
Garcia
a) Dividing problem up b) Shipping it out /
Getting it back c) Verifying the result d) Putting results together e) Depends on problem
What contributes to overhead the most?
![Page 18: The Beauty and Joy of Computingcs10/sp14/Lectures... · " “flops” = # FLOP/second is the standard metric for computing power ! Example: Global Climate Modeling " Divide the world](https://reader033.vdocuments.net/reader033/viewer/2022052103/603e1d2c025a5a6a0b2c5325/html5/thumbnails/18.jpg)
UC Berkeley “The Beauty and Joy of Computing” : Distributed Computing (18)
Garcia
§ Systems and networks enable and foster computational problem solving
§ MapReduce is a great distributed computing abstraction ú It removes the onus of
worrying about load balancing, failed machines, data distribution from the programmer of the problem
ú (and puts it on the authors of the MapReduce framework)
Summary