Fundamentals of MapReduce

Jimmy LinThe iSchoolUniversity of Maryland

Monday, March 30, 2009

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United StatesSee http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details

Some material adapted from slides by Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed Computing Seminar, 2007 (licensed under Creation Commons Attribution 3.0 License)

This afternoon... Introduction to MapReduce

MapReduce “killer app” #1: text retrieval

MapReduce “killer app” #2: graph algorithms

Introduction to MapReduce

Introduction to MapReduce: Topics Functional programming

MapReduce

Distributed file system

Functional Programming MapReduce = functional programming meets distributed

processing on steroids Not a new idea… dates back to the 50’s (or even 30’s)

What is functional programming? Computation as application of functions Theoretical foundation provided by lambda calculus

How is it different? Traditional notions of “data” and “instructions” are not applicable Data flows are implicit in program Different orders of execution are possible

Exemplified by LISP and ML

Overview of Lisp Lisp ≠ Lost In Silly Parentheses

We’ll focus on particular a dialect: “Scheme”

Lists are primitive data types

Functions written in prefix notation

(+ 1 2) 3(* 3 4) 12(sqrt (+ (* 3 3) (* 4 4))) 5(define x 3) x(* x 5) 15

'(1 2 3 4 5)

'((a 1) (b 2) (c 3))

Functions Functions = lambda expressions bound to variables

Syntactic sugar for defining functions Above expressions is equivalent to:

Once defined, function can be applied:

(define (foo x y) (sqrt (+ (* x x) (* y y))))

(define foo (lambda (x y) (sqrt (+ (* x x) (* y y)))))

(foo 3 4) 5

Other Features In Scheme, everything is an s-expression

No distinction between “data” and “code” Easy to write self-modifying code

Higher-order functions Functions that take other functions as arguments

(define (bar f x) (f (f x)))

(define (baz x) (* x x))

(bar baz 2) 16

Doesn’t matter what f is, just apply it twice.

Recursion is your friend Simple factorial example

Even iteration is written with recursive calls!

(define (factorial n) (if (= n 1) 1 (* n (factorial (- n 1)))))

(factorial 6) 720

(define (factorial-iter n) (define (aux n top product) (if (= n top) (* n product) (aux (+ n 1) top (* n product))))

(aux 1 n 1))

(factorial-iter 6) 720

Lisp MapReduce? What does this have to do with MapReduce?

After all, Lisp is about processing lists

Two important concepts in functional programming Map: do something to everything in a list Fold: combine results of a list in some way

Map Map is a higher-order function

How map works: Function is applied to every element in a list Result is a new list

f f f f f

Fold Fold is also a higher-order function

How fold works: Accumulator set to initial value Function applied to list element and the accumulator Result stored in the accumulator Repeated for every item in the list Result is the final value in the accumulator

f f f f f final value

Initial value

Map/Fold in Action Simple map example:

Fold examples:

Sum of squares:

(map (lambda (x) (* x x)) '(1 2 3 4 5)) '(1 4 9 16 25)

(fold + 0 '(1 2 3 4 5)) 15(fold * 1 '(1 2 3 4 5)) 120

(define (sum-of-squares v) (fold + 0 (map (lambda (x) (* x x)) v)))

(sum-of-squares '(1 2 3 4 5)) 55

Lisp MapReduce Let’s assume a long list of records: imagine if...

We can parallelize map operations We have a mechanism for bringing map results back together in

the fold operation

That’s MapReduce!

Observations: No limit to map parallelization since maps are indepedent We can reorder folding if the fold function is commutative and

associative

Typical Problem Iterate over a large number of records

Extract something of interest from each

Shuffle and sort intermediate results

Aggregate intermediate results

Generate final output

Key idea: provide an abstraction at the point of these two operations

Map

Reduce

MapReduce Programmers specify two functions:

map (k, v) → <k’, v’>*reduce (k’, v’) → <k’, v’>* All v’ with the same k’ are reduced together

Usually, programmers also specify:partition (k’, number of partitions ) → partition for k’ Often a simple hash of the key, e.g. hash(k’) mod n Allows reduce operations for different keys in parallel

Implementations: Google has a proprietary implementation in C++ Hadoop is an open source implementation in Java

mapmap map map

Shuffle and Sort: aggregate values by keys

reduce reduce reduce

k1 k2 k3 k4 k5 k6v1 v2 v3 v4 v5 v6

ba 1 2 c c3 6 a c5 2 b c7 9

a 1 5 b 2 7 c 2 3 6 9

r1 s1 r2 s2 r3 s3

Recall these problems? How do we assign work units to workers?

What if we have more work units than workers?

What if workers need to share partial results?

How do we aggregate partial results?

How do we know all the workers have finished?

What if workers die?

MapReduce Runtime Handles scheduling

Assigns workers to map and reduce tasks

Handles “data distribution” Moves the process to the data

Handles synchronization Gathers, sorts, and shuffles intermediate data

Handles faults Detects worker failures and restarts

Everything happens on top of a distributed FS (later)

“Hello World”: Word Count

Map(String input_key, String input_value): // input_key: document name // input_value: document contents for each word w in input_values: EmitIntermediate(w, "1");

Reduce(String key, Iterator intermediate_values): // key: a word, same for input and output // intermediate_values: a list of counts int result = 0; for each v in intermediate_values: result += ParseInt(v); Emit(AsString(result));

split 0

split 1

split 2

split 3

split 4

worker

worker

worker

worker

worker

Master

UserProgram

outputfile 0

outputfile 1

(1) fork (1) fork (1) fork

(2) assign map(2) assign reduce

(3) read(4) local write

(5) remote read(6) write

Inputfiles

Mapphase

Intermediate files(on local disk)

Reducephase

Outputfiles

Redrawn from Dean and Ghemawat (OSDI 2004)

Bandwidth Optimization Issue: large number of key-value pairs

Solution: use “Combiner” functions Executed on same machine as mapper Results in a “mini-reduce” right after the map phase Reduces key-value pairs to save bandwidth

Skew Problem Issue: reduce is only as fast as the slowest map

Solution: redundantly execute map operations, use results of first to finish Addresses hardware problems... But not issues related to inherent distribution of data

How do we get data to the workers?

Compute Nodes

NAS

SAN

What’s the problem here?

Distributed File System Don’t move data to workers… Move workers to the data!

Store data on the local disks for nodes in the cluster Start up the workers on the node that has the data local

Why? Not enough RAM to hold all the data in memory Disk access is slow, disk throughput is good

A distributed file system is the answer GFS (Google File System) HDFS for Hadoop

GFS: Assumptions Commodity hardware over “exotic” hardware

High component failure rates Inexpensive commodity components fail all the time

“Modest” number of HUGE files

Files are write-once, mostly appended to Perhaps concurrently

Large streaming reads over random access

High sustained throughput over low latency

GFS slides adapted from material by Dean et al.

GFS: Design Decisions Files stored as chunks

Fixed size (64MB)

Reliability through replication Each chunk replicated across 3+ chunkservers

Single master to coordinate access, keep metadata Simple centralized management

No data caching Little benefit due to large data sets, streaming reads

Simplify the API Push some of the issues onto the client

Redrawn from Ghemawat et al. (SOSP 2003)

Application

GSF Client

GFS master

File namespace/foo/bar

chunk 2ef0

GFS chunkserver

Linux file system

…

GFS chunkserver

Linux file system

…

(file name, chunk index)

(chunk handle, chunk location)

Instructions to chunkserver

Chunkserver state(chunk handle, byte range)

chunk data

Single Master We know this is a:

Single point of failure Scalability bottleneck

GFS solutions: Shadow masters Minimize master involvement

• Never move data through it, use only for metadata (and cache metadata at clients)

• Large chunk size• Master delegates authority to primary replicas in

data mutations (chunk leases) Simple, and good enough!

Master’s Responsibilities (1/2) Metadata storage

Namespace management/locking

Periodic communication with chunkservers Give instructions, collect state, track cluster health

Chunk creation, re-replication, rebalancing Balance space utilization and access speed Spread replicas across racks to reduce correlated failures Re-replicate data if redundancy falls below threshold Rebalance data to smooth out storage and request load

Master’s Responsibilities (2/2) Garbage Collection

Simpler, more reliable than traditional file delete Master logs the deletion, renames the file to a hidden name Lazily garbage collects hidden files

Stale replica deletion Detect “stale” replicas using chunk version numbers

Metadata Global metadata is stored on the master

File and chunk namespaces Mapping from files to chunks Locations of each chunk’s replicas

All in memory (64 bytes / chunk) Fast Easily accessible

Master has an operation log for persistent logging of critical metadata updates Persistent on local disk Replicated Checkpoints for faster recovery

Mutations Mutation = write or append

Must be done for all replicas

Goal: minimize master involvement

Lease mechanism: Master picks one replica as primary; gives it a “lease” for

mutations Primary defines a serial order of mutations All replicas follow this order Data flow decoupled from control flow

Parallelization Problems How do we assign work units to workers?

What if we have more work units than workers?

What if workers need to share partial results?

How do we aggregate partial results?

How do we know all the workers have finished?

What if workers die?

How is MapReduce different?

Questions?

MapReduce “killer app” #1:Text Retrieval

Text Retrieval: Topics Introduction to information retrieval (IR)

Boolean retrieval

Ranked retrieval

Text retrieval with MapReduce

The Information Retrieval Cycle

SourceSelection

Search

Query

Selection

Results

Examination

Documents

Delivery

Information

QueryFormulation

Resource

source reselection

System discoveryVocabulary discoveryConcept discoveryDocument discovery

The Central Problem in Search

SearcherAuthor

Concepts Concepts

Query Terms Document Terms

Do these represent the same concepts?

“tragic love story” “fateful star-crossed romance”

Architecture of IR Systems

DocumentsQuery

Hits

RepresentationFunction

RepresentationFunction

Query Representation Document Representation

ComparisonFunction Index

offlineonline

How do we represent text? Remember: computers don’t “understand” anything!

“Bag of words” Treat all the words in a document as index terms for that document Assign a “weight” to each term based on “importance” Disregard order, structure, meaning, etc. of the words Simple, yet effective!

Assumptions Term occurrence is independent Document relevance is independent “Words” are well-defined

What’s a word?

天主教教宗若望保祿二世因感冒再度住進醫院。這是他今年第二度因同樣的病因住院。 - باسم الناطق ريجيف مارك وقال

قبل - شارون إن اإلسرائيلية الخارجيةبزيارة األولى للمرة وسيقوم الدعوة

المقر طويلة لفترة كانت التي تونس،لبنان من خروجها بعد الفلسطينية التحرير لمنظمة الرسمي

1982عام .

Выступая в Мещанском суде Москвы экс-глава ЮКОСа заявил не совершал ничего противозаконного, в чем обвиняет его генпрокуратура России.

भा�रत सरका�र ने आर्थि� का सर्वे�क्षण में� विर्वेत्ती�य र्वेर्ष� 2005-06 में� स�त फ़ी�सदी� विर्वेका�स दीर हा�सिसल कारने का� आकालने विकाय� हा! और कार स#धा�र पर ज़ो'र दिदीय� हा!

日米連合で台頭中国に対処…アーミテージ前副長官提言

조재영 기자 = 서울시는 25 일 이명박 시장이 ` 행정중심복합도시 '' 건설안에 대해 ` 군대라도 동원해 막고싶은 심정 '' 이라고 말했다는 일부 언론의 보도를 부인했다 .

Sample DocumentMcDonald's slims down spudsFast-food chain to reduce certain types of fat in its french fries with new cooking oil.

NEW YORK (CNN/Money) - McDonald's Corp. is cutting the amount of "bad" fat in its french fries nearly in half, the fast-food chain said Tuesday as it moves to make all its fried menu items healthier.

But does that mean the popular shoestring fries won't taste the same? The company says no. "It's a win-win for our customers because they are getting the same great french-fry taste along with an even healthier nutrition profile," said Mike Roberts, president of McDonald's USA.

But others are not so sure. McDonald's will not specifically discuss the kind of oil it plans to use, but at least one nutrition expert says playing with the formula could mean a different taste.

Shares of Oak Brook, Ill.-based McDonald's (MCD: down $0.54 to $23.22, Research, Estimates) were lower Tuesday afternoon. It was unclear Tuesday whether competitors Burger King and Wendy's International (WEN: down $0.80 to $34.91, Research, Estimates) would follow suit. Neither company could immediately be reached for comment.

…

16 × said

14 × McDonalds

12 × fat

11 × fries

8 × new

6 × company, french, nutrition

5 × food, oil, percent, reduce, taste, Tuesday

…

“Bag of Words”

Boolean Retrieval Users express queries as a Boolean expression

AND, OR, NOT Can be arbitrarily nested

Retrieval is based on the notion of sets Any given query divides the collection into two sets:

retrieved, not-retrieved Pure Boolean systems do not define an ordering of the results

Representing Documents

The quick brown fox jumped over the lazy dog’s back.

Document 1

Document 2

Now is the time for all good men to come to the aid of their party.

the

isfor

to

of

quick

brown

fox

over

lazy

dog

back

now

time

all

good

men

come

jump

aid

their

party

00110110110010100

11001001001101011

Term Doc

ume

nt 1

Doc

ume

nt 2

Stopword List

Inverted Index

quick

brown

fox

over

lazy

dog

back

now

time

all

good

men

come

jump

aid

their

party

00110000010010110

01001001001100001

Term

Doc

1D

oc 2

00110110110010100

11001001001000001

Doc

3D

oc 4

00010110010010010

01001001000101001

Doc

5D

oc 6

00110010010010010

10001001001111000

Doc

7D

oc 8

quick

brown

fox

over

lazy

dog

back

now

time

all

good

men

come

jump

aid

their

party

4 82 4 61 3 71 3 5 72 4 6 83 53 5 72 4 6 831 3 5 7

1 3 5 7 8

2 4 82 6 8

1 5 72 4 6

1 36 8

Term Postings

Boolean Retrieval To execute a Boolean query:

Build query syntax tree

For each clause, look up postings

Traverse postings and apply Boolean operator

Efficiency analysis Postings traversal is linear (assuming sorted postings) Start with shortest posting first

( fox or dog ) and quick

fox dog

ORquick

AND

foxdog 3 5

3 5 7

foxdog 3 5

3 5 7OR = union 3 5 7

Extensions Implementing proximity operators

Store word offset in postings

Handling term variations Stem words: love, loving, loves … lov

Strengths and Weaknesses Strengths

Precise, if you know the right strategies Precise, if you have an idea of what you’re looking for Implementations are fast and efficient

Weaknesses Users must learn Boolean logic Boolean logic insufficient to capture the richness of language No control over size of result set: either too many hits or none When do you stop reading? All documents in the result set are

considered “equally good” What about partial matches? Documents that “don’t quite match”

the query may be useful also

Ranked Retrieval Order documents by how likely they are to be relevant to

the information need Estimate relevance(q, di) Sort documents by relevance Display sorted results

User model Present hits one screen at a time, best results first At any point, users can decide to stop looking

How do we estimate relevance? Assume document is relevant if it has a lot of query terms Replace relevance (q, di) with sim(q, di) Compute similarity of vector representations

Vector Space Model

Assumption: Documents that are “close together” in vector space “talk about” the same things

t1

d2

d1

d3

d4

d5

t3

t2

θ

φ

Therefore, retrieve documents based on how close the document is to the query (i.e., similarity ~ “closeness”)

Similarity Metric How about |d1 – d2|?

Instead of Euclidean distance, use “angle” between the vectors It all boils down to the inner product (dot product) of vectors

kj

kj

dd

dd

)cos(

n

i ki

n

i ji

n

i kiji

kj

kjkj

ww

ww

dd

ddddsim

1

2,1

2,

1 ,,),(

Term Weighting Term weights consist of two components

Local: how important is the term in this document? Global: how important is the term in the collection?

Here’s the intuition: Terms that appear often in a document should get high weights Terms that appear in many documents should get low weights

How do we capture this mathematically? Term frequency (local) Inverse document frequency (global)

TF.IDF Term Weighting

ijiji n

Nw logtf ,,

jiw ,

ji ,tf

N

in

weight assigned to term i in document j

number of occurrence of term i in document j

number of documents in entire collection

number of documents with term i

TF.IDF Example

4

5

6

3

1

3

1

6

5

3

4

3

7

1

2

1 2 3

2

3

2

4

4

0.301

0.125

0.125

0.125

0.602

0.301

0.000

0.602

tfidf

complicated

contaminated

fallout

information

interesting

nuclear

retrieval

siberia

1,4

1,5

1,6

1,3

2,1

2,1

2,6

3,5

3,3

3,4

1,2

0.301

0.125

0.125

0.125

0.602

0.301

0.000

0.602

complicated

contaminated

fallout

information

interesting

nuclear

retrieval

siberia

4,2

4,3

2,3 3,3 4,2

3,7

3,1 4,4

Sketch: Scoring Algorithm Initialize accumulators to hold document scores

For each query term t in the user’s query Fetch t’s postings For each document, scoredoc += wt,d wt,q

Apply length normalization to the scores at end

Return top N documents

MapReduce it? The indexing problem

Must be relatively fast, but need not be real time For Web, incremental updates are important

The retrieval problem Must have sub-second response For Web, only need relatively few results

Indexing: Performance Analysis Inverted indexing is fundamental to all IR models

Fundamentally, a large sorting problem Terms usually fit in memory Postings usually don’t

How is it done on a single machine?

How large is the inverted index? Size of vocabulary Size of postings

Vocabulary Size: Heaps’ Law

KnV V is vocabulary sizen is corpus size (number of documents)K and are constants

Typically, K is between 10 and 100, is between 0.4 and 0.6

When adding new documents, the system is likely to have seen most terms already… but the postings keep growing

George Kingsley Zipf (1902-1950) observed the following relation between frequency and rank

In other words: A few elements occur very frequently Many elements occur very infrequently

Zipfian distributions: English words Library book checkout patterns Website popularity (almost anything on the Web)

Postings Size: Zipf’s Law

crf or

r

cf f = frequency

r = rankc = constant

Word Frequency in English

the 1130021 from 96900 or 54958of 547311 he 94585 about 53713to 516635 million 93515 market 52110a 464736 year 90104 they 51359in 390819 its 86774 this 50933and 387703 be 85588 would 50828that 204351 was 83398 you 49281for 199340 company 83070 which 48273is 152483 an 76974 bank 47940said 148302 has 74405 stock 47401it 134323 are 74097 trade 47310on 121173 have 73132 his 47116by 118863 but 71887 more 46244as 109135 will 71494 who 42142at 101779 say 66807 one 41635mr 101679 new 64456 their 40910with 101210 share 63925

Frequency of 50 most common words in English (sample of 19 million words)

Does it fit Zipf’s Law?

the 59 from 92 or 101of 58 he 95 about 102to 82 million 98 market 101a 98 year 100 they 103in 103 its 100 this 105and 122 be 104 would 107that 75 was 105 you 106for 84 company 109 which 107is 72 an 105 bank 109said 78 has 106 stock 110it 78 are 109 trade 112on 77 have 112 his 114by 81 but 114 more 114as 80 will 117 who 106at 80 say 113 one 107mr 86 new 112 their 108with 91 share 114

The following shows rf1000 r is the rank of word w in the sample f is the frequency of word w in the sample

MapReduce: Index Construction Map over all documents

Emit term as key, (docid, tf) as value Emit other information as necessary (e.g., term position)

Reduce Trivial: each value represents a posting! Might want to sort the postings (e.g., by docid or tf)

MapReduce does all the heavy lifting!

Query Execution MapReduce is meant for large-data batch processing

Not suitable for lots of real time operations requiring low latency

The solution: “the secret sauce” Most likely involves document partitioning Lots of system engineering: e.g., caching, load balancing, etc.

MapReduce: Query Execution High-throughput batch query execution:

Instead of sequentially accumulating scores per query term: Have mappers traverse postings in parallel, emitting partial score

components Reducers serve as the accumulators, summing contributions for

each query term

MapReduce does all the heavy lifting Replace random access with sequential reads Amortize over lots of queries Examine multiple postings in parallel

Questions?

MapReduce “killer app” #2:Graph Algorithms

Graph Algorithms: Topics Introduction to graph algorithms and graph representations

Single Source Shortest Path (SSSP) problem Refresher: Dijkstra’s algorithm Breadth-First Search with MapReduce

PageRank

What’s a graph? G = (V,E), where

V represents the set of vertices (nodes) E represents the set of edges (links) Both vertices and edges may contain additional information

Different types of graphs: Directed vs. undirected edges Presence or absence of cycles

Graphs are everywhere: Hyperlink structure of the Web Physical structure of computers on the Internet Interstate highway system Social networks

Some Graph Problems Finding shortest paths

Routing Internet traffic and UPS trucks

Finding minimum spanning trees Telco laying down fiber

Finding Max Flow Airline scheduling

Identify “special” nodes and communities Breaking up terrorist cells, spread of avian flu

Bipartite matching Monster.com, Match.com

And of course... PageRank

Graphs and MapReduce Graph algorithms typically involve:

Performing computation at each node Processing node-specific data, edge-specific data, and link

structure Traversing the graph in some manner

Key questions: How do you represent graph data in MapReduce? How do you traverse a graph in MapReduce?

Representing Graphs G = (V, E)

A poor representation for computational purposes

Two common representations Adjacency matrix Adjacency list

Adjacency Matrices

Represent a graph as an n x n square matrix M n = |V| Mij = 1 means a link from node i to j

1 2 3 4

1 0 1 0 1

2 1 0 1 1

3 1 0 0 0

4 1 0 1 0

1

2

3

4

Adjacency Matrices: Critique Advantages:

Naturally encapsulates iteration over nodes Rows and columns correspond to inlinks and outlinks

Disadvantages: Lots of zeros for sparse matrices Lots of wasted space

Adjacency Lists

Take adjacency matrices… and throw away all the zeros

1 2 3 4

1 0 1 0 1

2 1 0 1 1

3 1 0 0 0

4 1 0 1 0

1: 2, 42: 1, 3, 43: 14: 1, 3

Adjacency Lists: Critique Advantages:

Much more compact representation Easy to compute over outlinks Graph structure can be broken up and distributed

Disadvantages: Much more difficult to compute over inlinks

Single Source Shortest Path Problem: find shortest path from a source node to one or

more target nodes

First, a refresher: Dijkstra’s Algorithm

Dijkstra’s Algorithm Example

0

10

5

2 3

2

1

9

7

4 6

Example from CLR

Dijkstra’s Algorithm Example

0

10

5

10

5

2 3

2

1

9

7

4 6

Example from CLR

Dijkstra’s Algorithm Example

0

8

5

14

7

10

5

2 3

2

1

9

7

4 6

Example from CLR

Dijkstra’s Algorithm Example

0

8

5

13

7

10

5

2 3

2

1

9

7

4 6

Example from CLR

Dijkstra’s Algorithm Example

0

8

5

9

7

10

5

2 3

2

1

9

7

4 6

Example from CLR

Dijkstra’s Algorithm Example

0

8

5

9

7

10

5

2 3

2

1

9

7

4 6

Example from CLR

Single Source Shortest Path Problem: find shortest path from a source node to one or

more target nodes

Single processor machine: Dijkstra’s Algorithm

MapReduce: parallel Breadth-First Search (BFS)

Finding the Shortest Path First, consider equal edge weights

Solution to the problem can be defined inductively

Here’s the intuition: DistanceTo(startNode) = 0 For all nodes n directly reachable from startNode,

DistanceTo(n) = 1 For all nodes n reachable from some other set of nodes S,

DistanceTo(n) = 1 + min(DistanceTo(m), m S)

From Intuition to Algorithm A map task receives

Key: node n Value: D (distance from start), points-to (list of nodes reachable

from n)

p points-to: emit (p, D+1)

The reduce task gathers possible distances to a given p and selects the minimum one

Multiple Iterations Needed This MapReduce task advances the “known frontier” by

one hop Subsequent iterations include more reachable nodes as frontier

advances Multiple iterations are needed to explore entire graph Feed output back into the same MapReduce task

Preserving graph structure: Problem: Where did the points-to list go? Solution: Mapper emits (n, points-to) as well

Visualizing Parallel BFS

1

2 2

2

3

3

3

3

4

4

Termination Does the algorithm ever terminate?

Eventually, all nodes will be discovered, all edges will be considered (in a connected graph)

When do we stop?

Weighted Edges Now add positive weights to the edges

Simple change: points-to list in map task includes a weight w for each pointed-to node emit (p, D+wp) instead of (p, D+1) for each node p

Does this ever terminate? Yes! Eventually, no better distances will be found. When distance

is the same, we stop Mapper should emit (n, D) to ensure that “current distance” is

carried into the reducer

Comparison to Dijkstra Dijkstra’s algorithm is more efficient

At any step it only pursues edges from the minimum-cost path inside the frontier

MapReduce explores all paths in parallel Divide and conquer Throw more hardware at the problem

General Approach MapReduce is adapt at manipulating graphs

Store graphs as adjacency lists

Graph algorithms with for MapReduce: Each map task receives a node and its outlinks Map task compute some function of the link structure, emits value

with target as the key Reduce task collects keys (target nodes) and aggregates

Iterate multiple MapReduce cycles until some termination condition Remember to “pass” graph structure from one iteration to next

Random Walks Over the Web Model:

User starts at a random Web page User randomly clicks on links, surfing from page to page

What’s the amount of time that will be spent on any given page?

This is PageRank

Given page x with in-bound links t1…tn, where C(t) is the out-degree of t is probability of random jump N is the total number of nodes in the graph

PageRank: Defined

n

i i

i

tC

tPR

NxPR

1 )(

)()1(

1)(

X

t1

t2

tn

…

Computing PageRank Properties of PageRank

Can be computed iteratively Effects at each iteration is local

Sketch of algorithm: Start with seed PRi values

Each page distributes PRi “credit” to all pages it links to Each target page adds up “credit” from multiple in-bound links to

compute PRi+1

Iterate until values converge

PageRank in MapReduce

Map: distribute PageRank “credit” to link targets

...

Reduce: gather up PageRank “credit” from multiple sources to compute new PageRank value

Iterate untilconvergence

PageRank: Issues Is PageRank guaranteed to converge? How quickly?

What is the “correct” value of , and how sensitive is the algorithm to it?

What about dangling links?

How do you know when to stop?

Questions?