batch graph processing frameworks

Comparison of Graph Processing Frameworks

Alex Averbuch

Swedish Institute of Computer Science

[email protected]

January 25, 2012

Alex Averbuch Big Graph Processing 1 / 36

Frameworks Compared

• Pregel: a system for large-scale graph processing. G. Malewicz,M.H. Austern, AJ Bik, J.C. Dehnert, I. Horn, N. Leiser, and G.Czajkowski. PODC, 2009.

• Signal/collect: graph algorithms for the (semantic) web. P.Stutz, A. Bernstein, and W. Cohen. The Semantic Web - ISWC,2010.


Background — Big Graphs Everywhere

• Real world web and social graphs continue to grow• 2008 → Google estimates number of web pages at 1 trillion• March 2011 → LinkedIn has over 120 million registered users• September 2011 → Twitter has over 100 million active users• September 2011 → Facebook has over 800 million active users

Data: The New Oil

• Relevant, personalized user information relies on graph algorithms

• Popularity rank → determine popular users, news, jobs, etc.• Shortest paths → find how users, groups, etc. are connected• Clustering → discover related people, groups, interests, etc.


Background — Big Graphs Everywhere

• Real world web and social graphs continue to grow• 2008 → Google estimates number of web pages at 1 trillion• March 2011 → LinkedIn has over 120 million registered users• September 2011 → Twitter has over 100 million active users• September 2011 → Facebook has over 800 million active users

Data: The New Oil

• Relevant, personalized user information relies on graph algorithms• Popularity rank → determine popular users, news, jobs, etc.• Shortest paths → find how users, groups, etc. are connected• Clustering → discover related people, groups, interests, etc.


Background — The Vertex Centric Model

Definition: Vertex Centric Graph Computing Model

• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages

• computation proceeds in iterations. in each iteration, vertices:

1 perform some computation2 communicate with other vertices







*4 2

1

1

*







iteration 0

0 - - -4 2

1

1

1

4







iteration 0

0 4 1 -4 2

1

1







iteration 1

0 4 1 -4 2

1

1

5

3







iteration 1

0 3 1 54 2

1

1







iteration 2

0 3 1 54 2

1

1

4







iteration 2

0 3 1 44 2

1

1







0 3 1 44 2

1

1


Pregel — Contributions

• parallel programming model (for processing graphs)

• distributed execution model (for processing graphs)

• (limited) evaluation → using big data sets


Pregel — Overview

• vertex centric graph computing model

• in each iteration a compute function is invoked on each vertex

1 reads messages sent to it in previous iteration2 modifies its state & local graph topology3 sends messages to other vertices4 votes to halt (to become inactive)

Vote to halt

Message received

Active Inactive

Vertex State Machine


Pregel — Programming Model (Vertex & Edge)

• Vertex (v)• v.id → unique identifier• v.state → arbitrary vertex state• v.outEdges : List[Edge] → list of edges that have v as source• v.compute() : per iteration, calculates new state

1 reads incoming messages, from previous iteration2 sends (unbounded number of) messages to other vertices3 if destination non-existent, call handler (create vertex/remove edge)4 modifies its state and that of its outgoing edges5 adds/removes edges to/from outEdges

6 votes to halt

• Edge (e)• e.targetId → identifier of target vertex• e.state → arbitrary edge state• no associated computation


Pregel — Programming Model (Combiner & Aggregator)

• Combiner

• combines multiple messages into one (like reducer in M/R)• combined using commutative & associative function• reduces network traffic & message buffer size• e.g. in SSSP vertex only cares about length of shortest path

• Aggregator

• globally shared/aggregated state

1 vertices write to aggregator variable locally2 globally aggregated value available to all vertices in next iteration

• aggregated using commutative & associative function• pre-defined aggregators: min, max, sum


Pregel — Programming Model (Combiner & Aggregator)

• Combiner

• combines multiple messages into one (like reducer in M/R)• combined using commutative & associative function• reduces network traffic & message buffer size• e.g. in SSSP vertex only cares about length of shortest path

• Aggregator• globally shared/aggregated state

1 vertices write to aggregator variable locally2 globally aggregated value available to all vertices in next iteration

• aggregated using commutative & associative function• pre-defined aggregators: min, max, sum


Pregel — Programming Model (Topology Mutations)

• determinism in the presence of conflicts is achieved by:1 partial ordering

1 remove edges2 remove vertices (implicitly removes edges)3 add vertices4 add edges

2 conflict handlers• example conflict → vertices with same ID created simultaneously• extend conflict handler() of Vertex class• same handler called for all conflict types

• most topology changes are seen in next iteration

• self mutations (remove out edge, remove self) are immediate


Pregel — Programming Model (Topology Mutations)

• determinism in the presence of conflicts is achieved by:1 partial ordering

1 remove edges2 remove vertices (implicitly removes edges)3 add vertices4 add edges

2 conflict handlers• example conflict → vertices with same ID created simultaneously• extend conflict handler() of Vertex class• same handler called for all conflict types

• most topology changes are seen in next iteration• self mutations (remove out edge, remove self) are immediate


Pregel — Programming Model Example — Vertex

Code: Vertex program for Single Source Shortest Path (SSSP)

class ShortestPathVertex : public Vertex <int , int , int > {void Compute(MessageIterator* msgs) {

// initializationint mindist = IsSource(vertex_id ()) ? 0 : INF;

// read incoming messages & update mindistfor (; !msgs ->Done(); msgs ->Next())

mindist = min(mindist , msgs ->Value());

// send updated mindist to neighborsif (mindist < GetValue ()) {

*MutableValue () = mindist;OutEdgeIterator iter = GetOutEdgeIterator ();for (; !iter.Done(); iter.Next())

SendMessageTo(iter.Target (),mindist + iter.GetValue ());

}

// deactivate unless/until another message arrivesVoteToHalt ();

}};


Pregel — Execution Model

• vertex scheduling: all active vertices, per iteration

• termination: no active vertices & no messages in transit

Scheduler: Pregel (Bulk Synchronous Parallel)

while (∃v ∈ V : v.active = true) dofor all v ∈ V parallel do

if (v.active = true) thenv.compute()


Pregel — Execution — Without Combiner

4

*

-

-

-

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

mode: pregel



4

0

-

-

-

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

3

1

iteration: 0computing vertices: 13messages: 3

total operations: 13total messages: 3

mode: pregel



4

0

1

3

1

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

4

2

2

2

5

5



mode: pregel



4

0

1

2

1

4

2

5

2

5

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

6

6

4

6

7

7



mode: pregel

7



4

0

1

2

1

4

2

4

2

5

6

6

7

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

5

6

7

910

8



mode: pregel



4

0

1

2

1

4

2

4

2

5

5

6

7

7

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

6



mode: pregel



4

0

1

2

1

4

2

4

2

5

5

6

7

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1



mode: pregel



4

0

1

2

1

4

2

4

2

5

5

6

7

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

total iterations: 5total operations: 31total messages: 23

mode: pregel


Pregel — Programming Model Example — Combiner

Code: Combiner program for Single Source Shortest Path (SSSP)

class MinIntCombiner : public Combiner <int > {virtual void Combine(MessageIterator* msgs) {

// initializationint mindist = INF;

// read messages & update mindistfor (; !msgs ->Done(); msgs ->Next())

mindist = min(mindist , msgs ->Value());

// only emit minimum message value (distance)Output("combined_source", mindist);

}};


Pregel — Execution — With Combiner

4

*

-

-

-

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

mode: pregel (combined)



4

0

-

-

-

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

3

1






4

0

1

3

1

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

4

2

2

2

5

5






4

0

1

2

1

4

2

5

2

5

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

6

6

4

6

7

7



7




4

0

1

2

1

4

2

4

2

5

6

6

7

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

5

6

7

910

8






4

0

1

2

1

4

2

4

2

5

5

6

7

7

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

6






4

0

1

2

1

4

2

4

2

5

5

6

7

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1





Pregel — Execution — Comparison

• combiner vs no combiner• algorithm → Single Source Shortest Path (SSSP)• sample graph → 13 vertices / 19 edges• cost → iterations, operations, message buffers

Results: Cost comparison of execution modes (SSSP)

Iterations Operations Messages

Pregel 5 31 23Pregel + Combiner 5 31 18


Pregel — Architecture

Worker

PartitioninQ

outQ Worker

PartitioninQ

outQ Worker

PartitioninQ

outQ

Graph Dataset

Loading &Checkpointing

(Combined) Messages

MasterSynchronization &Aggregatation


Pregel — Typical Program

• Client

1 load input data into workers2 notify master to “start processing”3 wait for master to complete

4 extract result data from workers

• Master1 repeat until no active workers

• signal workers to process• wait for all workers to finish• update active-worker count

2 notify client about completion

• Worker1 repeat until inactive

• wait for “start iteration” from master• read data from in-queue• perform local processing• write data to out-queue & transmit• update active/inactive status• notify master


Pregel — Typical Program

• Client

1 load input data into workers2 notify master to “start processing”3 wait for master to complete4 extract result data from workers

• Master1 repeat until no active workers

• signal workers to process• wait for all workers to finish• update active-worker count

2 notify client about completion

• Worker1 repeat until inactive

• wait for “start iteration” from master• read data from in-queue• perform local processing• write data to out-queue & transmit• update active/inactive status• notify master


Pregel — Fault Tolerance

• logging1 checkpointing → state persisted at beginning of every n-th iteration

• master persists → progress of execution, aggregate values• workers persist → vertex values, edge values, messages

2 confined recovery → workers log out-messages from their partitions

• failure detection• heart beats• worker gets no heartbeat from master → worker terminates• master gets no heartbeat from worker → marks worker as failed

• failure recovery• partition(s) belonging to failed worker(s) are reassigned• lost partitions recovered from checkpoints• missing iterations recomputed (using logged messages)


Pregel — Evaluation — Scaling

• algorithm → Single Source Shortest Path (SSSP)

• hardware → 800 worker tasks scheduled on 300 multicore machines

• graph• random, log-normal out-degree distribution (mean = 127.1)• up to 1,000,000,000 vertices / 127,000,000,000 edges

Results: Scalability of Pregel (SSSP)

200 400 600 800 1000

100

200

300

400

500

600

700

800

Number of vertices (millions)

Ru

nti

me

(sec

ond

s)


Signal/Collect — Contributions

• parallel programming model (for processing graphs)

• parallel execution model (for processing graphs)

• (limited) evaluation → benefits of various scheduling policies


Signal/Collect — Programming Model (Vertex)

• Vertex (v)• v.id → unique identifier• v.state → arbitrary vertex state• v.lastSignalState → v.state at time of last signal()• v.outEdges : List[edge] → list of edges that have v as source• v.signalMap : Map(vid,signal) → last received messages

• vid - identifier of sender vertex• signal - last received signal from that vertex

• v.uncollectedSignals : List[signal] → list of signalsreceived since collect() was last executed

• v.collect() : calculates new vertex state

1 collect incoming signals2 process those signals (possibly using v.state)3 return new vertex state


Signal/Collect — Programming Model (Edge)

• Edge (e)• e.source → source vertex• e.sourceId → identifier of source vertex (e.source.id)• e.targetId → identifier of target vertex• e.state → arbitrary edge state• e.signal() → calculates the signal to send, then sends it

• signals are sent along edges of the compute graph


Signal/Collect — Programming Model Example

Code: Single Source Shortest Path (SSSP)

class Location(id: Any , initialState: Int) extends Vertex {def collect: Int = min(state , min(uncollectedSignals))

}

class Path(sourceId: Any , targetId: Int) extends Edge {def signal: Int = source.state + weight

}

• vertex state → shortest known path length to vertex from source

• edge state (weight) → path length of that individual edge

• signal → shortest known path length from source through edge


Signal/Collect — Execution Model

• different Execution Modes (scheduling policies)

1 synchronous2 synchronous score-guided3 asynchronous4 asynchronous scheduled

• execution mode dictates when signal & collect are called

Definition: Internal methods used by execution engine

procedure v.executeSignalOperation()lastSignalState ← statefor all e ∈ outGoingEdges do

e.target.uncollectedSignals.append(e.signal())e.target.signalMap.put(e.sourceId,e.signal())

procedure v.executeCollectOperation()state ← collect()uncollectedSignals ← Nil


Signal/Collect — Execution — Synchronous

• vertex scheduling: all vertices (unordered) per iteration

• termination: all iterations

Scheduler: Synchronous

for i ← 1 to num iterations dofor all v ∈ V parallel do

v.executeSignalOperation()

for all v ∈ V parallel dov.executeCollectOperation()



4

*

-

-

-

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

mode: synchronous



4

0

1

3

1

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

-

-

-

-

-

3

1 -

-

-

-

-

-

-

--

-

-

iteration: 0signaling vertices: 13collecting vertices: 13messages: 19


mode: synchronous



4

0

1

2

1

4

2

5

2

5

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

4

-

2

-

2

3

1 2

5

-

5

-

-

-

--

-

-



mode: synchronous



4

0

1

2

1

4

2

4

2

5

6

6

7

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

4

6

2

7

2

3

1 2

5

6

4

6

7

-

--

-

7



mode: synchronous



4

0

1

2

1

4

2

4

2

5

5

6

7

7

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

4

6

2

7

2

3

1 2

5

6

4

5

6

7

910

8

7



mode: synchronous



4

0

1

2

1

4

2

4

2

5

5

6

7

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

4

6

2

7

2

3

1 2

5

6

4

5

6

6

910

8

7



mode: synchronous



4

0

1

2

1

4

2

4

2

5

5

6

7

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1


mode: synchronous


Signal/Collect — Execution — Synchronous Guided

• vertex scheduling: all active vertices (unordered) per iteration

• termination: all iterations or (no signal and no collect)





• extension v.signalScore() “importance for vertex to signal”• may change ⇐⇒ v.state changes• default → 1 if changed, 0 otherwise

• extension v.collectScore() “importance for vertex to collect”• may change ⇐⇒ v.uncollectedSignals changes• default → v.uncollectedSignals.size()





Scheduler: Synchronous Score-Guided

done ← falsewhile (iterations < num iterations) ∧ (done = false) do

done ← trueiterations← iterations + 1for all v ∈ V parallel do

if v.signalScore() > signal threshold thendone ← falsev.executeSignalOperation()

for all v ∈ V parallel doif v.collectScore() > collect threshold then

done ← falsev.executeCollectOperation()



4

*

-

-

-

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0



4

0

1

3

1

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

3

1






4

0

1

2

1

4

2

5

2

5

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

4

2

2

2

5

5






4

0

1

2

1

4

2

4

2

5

6

6

7

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

6

6

4

6

7

7

7 iteration: 2signaling vertices: 6collecting vertices: 5messages: 7





4

0

1

2

1

4

2

4

2

5

5

6

7

7

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

5

6

7

910

8






4

0

1

2

1

4

2

4

2

5

5

6

7

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

6






4

0

1

2

1

4

2

4

2

5

5

6

7

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1




Signal/Collect — Execution — Asynchronous

• vertex scheduling: random

• termination: max operations or (no signal and no collect)





• no guarantee on order of execution• some vertices may signal while others collect

• no guarantee that all vertices are executed same amount of times

• asynchronous mode not usable for every algorithm• use when correctness not dependent on strict execution order





Scheduler: Asynchronous

ops← 0while (ops < num ops) ∧

(∃v ∈ V :

(v.signalScore > signal threshold) ∨ (v.collectScore > collect threshold))

doS ← random subset of Vfor all v ∈ S do

next ← random(signal/collect)if (next = signal) ∧ (v.signalScore > signal threshold) then

v.executeSignalOperation()ops← ops + 1

else if (next = collect) ∧ (v.collectScore > collect threshold) thenv.executeCollectOperation()ops← ops + 1


Signal/Collect — Execution — Asynchronous Scheduled

• vertex scheduling: scheduler-dependent

• termination: scheduler-dependent

• scheduler: schedules vertices’ signal & collect operations

• e.g. “eager” scheduler → tries to signal right after collection





• scheduler: schedules vertices’ signal & collect operations• e.g. “eager” scheduler → tries to signal right after collection

Scheduler: “Eager”

for all v ∈ V doif (v.collectScore() > collect threshold) then

v.executeCollectOperation()if (v.signalScore() > signal threshold) then

v.executeSignalOperation()





• scheduler: schedules vertices’ signal & collect operations

• e.g. “eager” scheduler → tries to signal right after collection

• benefit of this execution mode depends on:

1 impact on convergence (number of operations)2 cost of operations3 cost of scheduling



Scheduler: “SSSP” — minimize messages & computation

Signal← {v source} // sorted setwhile (Signal 6= {}) do

for top k v ∈ Signal dov.executeSignalOperation()Signal.remove(v)

for all v ∈ V doif (v.collectScore > collect threshold) then

v.executeCollectOperation()if (v.signalScore > signal threshold) then

Signal.put(v)

if (v destination.state ≤ min(Signal)) thenSignal← {}

// returns distance of shortest next stepfunction signalSort(v)

Distances← {∞}for all e ∈ v.outEdges do

Distances.put(e.signal)

return min(Distances)



4

*

-

-

-

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0



4

0

1

3

1

-

-

-

-

-

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

1

3

1




2

5

2



4

0

1

3

1

-

-

-

2

5

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

2

5




2

5

5

7



4

0

1

2

1

4

2

-

2

5

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

4

2

2




67

4

6

7



4

0

1

2

1

4

2

4

2

5

-

-

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

4




67

5

6

7



4

0

1

2

1

4

2

4

2

5

5

6

-

*

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1



5

6


67

6

9

6

7



4

0

1

2

1

4

2

4

2

5

5

6

-

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

6




67

6

9

7



4

0

1

2

1

4

2

4

2

5

5

6

-

6

1

3

1

32

1 5

2

1

1

4

2

3

1

3

2

1

1

iterations: 5total operations: 18total messages: 12



Signal/Collect — Execution Modes — Comparison

• basic comparison of execution modes• algorithm → Single Source Shortest Path (SSSP)• sample graph → 13 vertices / 19 edges• cost → iterations, operations, messages

Results: Cost comparison of execution modes (SSSP)

Iterations Operations Messages

Pregel 5 31* 23Pregel + Combiner 5 31* 18Synchronous 4 130 95Synchronous Guided 4 34 23Asynchronous Scheduled 5 18 12


Signal/Collect — Evaluation — Scaling

• algorithm → Single Source Shortest Path (SSSP)

• graph• random• log-normal out-degree distribution• longest path length 5• 1,000,000 vertices / 94,000,000 edges

Results: Scalability of Signal/Collect (SSSP)

0.00

50.00

100.00

150.00

200.00

250.00

300.00

1 2 3 4 5 6 7 8

Tim

e (

secon

ds)

# Worker Threads

AverageFastestSlowest

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7 8

Sp

eed

up

# Worker Threads

Measured Speedup(Average)

(a) Scaling with additional workers (b) Parallel speedup


Signal/Collect — Evaluation — Execution Modes

Results: Comparison of Execution Modes (PageRank)

0100200300400500600700800900

1,000

Synchronous Score-GuidedSynchronous

"Eager" Score-GuidedAsynchronous

"Above Average" Score-GuidedAsynchronous

0

50,000

100,000

150,000

200,000

250,000

300,000




0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000




0

10,000

20,000

30,000

40,000

50,000

60,000




Average Computation Time (ms) Average # of Signal Operations Executed

Average Computation Time (ms) Average # of Signal Operations Executed

PageRank on SwetoDblp citations22,387 vertices (publications) / 112,303 edges (citations)

PageRank on a generated graph100,000 vertices / 1,284,495 edges / log-normal out-degree distribution


Signal/Collect — Evaluation — Convergence

• algorithm → Vertex Coloring

• graph• random• log-normal out-degree distribution• 100,000 vertices / 554,118 edges

• experiment• measure time for algorithm to converge• results are averaged over 10 runs• noted as “did not converge” (8) when time exceeded one minute

Results: Vertex Coloring convergence times (ms)

Number of colors 5 6 7 8 9 10 11

Asynchronous“Eager”

8 12,870 1,690 1,392 1,243 1,218 1,046

SynchronousScore-Guided

8 8 8 8 8 2,856 1,876


batch graph processing frameworks

Technology

worker marks worker

social graphs continue

implicitly removes edges

computational units edges

message buer size

signal collect programming model

real world web

id unique identier