batch graph processing frameworks
Post on 11-Nov-2014
732 Views
Preview:
DESCRIPTION
TRANSCRIPT
Comparison of Graph Processing Frameworks
Alex Averbuch
Swedish Institute of Computer Science
averbuch@sics.se
January 25, 2012
Alex Averbuch Big Graph Processing 1 / 36
Frameworks Compared
• Pregel: a system for large-scale graph processing. G. Malewicz,M.H. Austern, AJ Bik, J.C. Dehnert, I. Horn, N. Leiser, and G.Czajkowski. PODC, 2009.
• Signal/collect: graph algorithms for the (semantic) web. P.Stutz, A. Bernstein, and W. Cohen. The Semantic Web - ISWC,2010.
Alex Averbuch Big Graph Processing 2 / 36
Background — Big Graphs Everywhere
• Real world web and social graphs continue to grow• 2008 → Google estimates number of web pages at 1 trillion• March 2011 → LinkedIn has over 120 million registered users• September 2011 → Twitter has over 100 million active users• September 2011 → Facebook has over 800 million active users
Data: The New Oil
• Relevant, personalized user information relies on graph algorithms
• Popularity rank → determine popular users, news, jobs, etc.• Shortest paths → find how users, groups, etc. are connected• Clustering → discover related people, groups, interests, etc.
Alex Averbuch Big Graph Processing 3 / 36
Background — Big Graphs Everywhere
• Real world web and social graphs continue to grow• 2008 → Google estimates number of web pages at 1 trillion• March 2011 → LinkedIn has over 120 million registered users• September 2011 → Twitter has over 100 million active users• September 2011 → Facebook has over 800 million active users
Data: The New Oil
• Relevant, personalized user information relies on graph algorithms
• Popularity rank → determine popular users, news, jobs, etc.• Shortest paths → find how users, groups, etc. are connected• Clustering → discover related people, groups, interests, etc.
Alex Averbuch Big Graph Processing 3 / 36
Background — Big Graphs Everywhere
• Real world web and social graphs continue to grow• 2008 → Google estimates number of web pages at 1 trillion• March 2011 → LinkedIn has over 120 million registered users• September 2011 → Twitter has over 100 million active users• September 2011 → Facebook has over 800 million active users
Data: The New Oil
• Relevant, personalized user information relies on graph algorithms• Popularity rank → determine popular users, news, jobs, etc.• Shortest paths → find how users, groups, etc. are connected• Clustering → discover related people, groups, interests, etc.
Alex Averbuch Big Graph Processing 3 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
*4 2
1
1
*
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
iteration 0
0 - - -4 2
1
1
1
4
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
iteration 0
0 4 1 -4 2
1
1
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
iteration 1
0 4 1 -4 2
1
1
5
3
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
iteration 1
0 3 1 54 2
1
1
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
iteration 2
0 3 1 54 2
1
1
4
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
iteration 2
0 3 1 44 2
1
1
Alex Averbuch Big Graph Processing 4 / 36
Background — The Vertex Centric Model
Definition: Vertex Centric Graph Computing Model
• computations execute on a compute graph• same topology as that of data graph• vertices are computational units• edges are communication channels• vertices interact with other vertices using messages
• computation proceeds in iterations. in each iteration, vertices:
1 perform some computation2 communicate with other vertices
0 3 1 44 2
1
1
Alex Averbuch Big Graph Processing 4 / 36
Pregel — Contributions
• parallel programming model (for processing graphs)
• distributed execution model (for processing graphs)
• (limited) evaluation → using big data sets
Alex Averbuch Big Graph Processing 5 / 36
Pregel — Overview
• vertex centric graph computing model
• in each iteration a compute function is invoked on each vertex
1 reads messages sent to it in previous iteration2 modifies its state & local graph topology3 sends messages to other vertices4 votes to halt (to become inactive)
Vote to halt
Message received
Active Inactive
Vertex State Machine
Alex Averbuch Big Graph Processing 6 / 36
Pregel — Programming Model (Vertex & Edge)
• Vertex (v)• v.id → unique identifier• v.state → arbitrary vertex state• v.outEdges : List[Edge] → list of edges that have v as source• v.compute() : per iteration, calculates new state
1 reads incoming messages, from previous iteration2 sends (unbounded number of) messages to other vertices3 if destination non-existent, call handler (create vertex/remove edge)4 modifies its state and that of its outgoing edges5 adds/removes edges to/from outEdges
6 votes to halt
• Edge (e)• e.targetId → identifier of target vertex• e.state → arbitrary edge state• no associated computation
Alex Averbuch Big Graph Processing 7 / 36
Pregel — Programming Model (Combiner & Aggregator)
• Combiner
• combines multiple messages into one (like reducer in M/R)• combined using commutative & associative function• reduces network traffic & message buffer size• e.g. in SSSP vertex only cares about length of shortest path
• Aggregator
• globally shared/aggregated state
1 vertices write to aggregator variable locally2 globally aggregated value available to all vertices in next iteration
• aggregated using commutative & associative function• pre-defined aggregators: min, max, sum
Alex Averbuch Big Graph Processing 8 / 36
Pregel — Programming Model (Combiner & Aggregator)
• Combiner
• combines multiple messages into one (like reducer in M/R)• combined using commutative & associative function• reduces network traffic & message buffer size• e.g. in SSSP vertex only cares about length of shortest path
• Aggregator• globally shared/aggregated state
1 vertices write to aggregator variable locally2 globally aggregated value available to all vertices in next iteration
• aggregated using commutative & associative function• pre-defined aggregators: min, max, sum
Alex Averbuch Big Graph Processing 8 / 36
Pregel — Programming Model (Topology Mutations)
• determinism in the presence of conflicts is achieved by:1 partial ordering
1 remove edges2 remove vertices (implicitly removes edges)3 add vertices4 add edges
2 conflict handlers• example conflict → vertices with same ID created simultaneously• extend conflict handler() of Vertex class• same handler called for all conflict types
• most topology changes are seen in next iteration
• self mutations (remove out edge, remove self) are immediate
Alex Averbuch Big Graph Processing 9 / 36
Pregel — Programming Model (Topology Mutations)
• determinism in the presence of conflicts is achieved by:1 partial ordering
1 remove edges2 remove vertices (implicitly removes edges)3 add vertices4 add edges
2 conflict handlers• example conflict → vertices with same ID created simultaneously• extend conflict handler() of Vertex class• same handler called for all conflict types
• most topology changes are seen in next iteration• self mutations (remove out edge, remove self) are immediate
Alex Averbuch Big Graph Processing 9 / 36
Pregel — Programming Model Example — Vertex
Code: Vertex program for Single Source Shortest Path (SSSP)
class ShortestPathVertex : public Vertex <int , int , int > {void Compute(MessageIterator* msgs) {
// initializationint mindist = IsSource(vertex_id ()) ? 0 : INF;
// read incoming messages & update mindistfor (; !msgs ->Done(); msgs ->Next())
mindist = min(mindist , msgs ->Value());
// send updated mindist to neighborsif (mindist < GetValue ()) {
*MutableValue () = mindist;OutEdgeIterator iter = GetOutEdgeIterator ();for (; !iter.Done(); iter.Next())
SendMessageTo(iter.Target (),mindist + iter.GetValue ());
}
// deactivate unless/until another message arrivesVoteToHalt ();
}};
Alex Averbuch Big Graph Processing 10 / 36
Pregel — Execution Model
• vertex scheduling: all active vertices, per iteration
• termination: no active vertices & no messages in transit
Scheduler: Pregel (Bulk Synchronous Parallel)
while (∃v ∈ V : v.active = true) dofor all v ∈ V parallel do
if (v.active = true) thenv.compute()
Alex Averbuch Big Graph Processing 11 / 36
Pregel — Execution — Without Combiner
4
*
-
-
-
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
mode: pregel
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Execution — Without Combiner
4
0
-
-
-
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
3
1
iteration: 0computing vertices: 13messages: 3
total operations: 13total messages: 3
mode: pregel
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Execution — Without Combiner
4
0
1
3
1
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
4
2
2
2
5
5
iteration: 1computing vertices: 3messages: 6
total operations: 16total messages: 9
mode: pregel
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Execution — Without Combiner
4
0
1
2
1
4
2
5
2
5
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
6
6
4
6
7
7
iteration: 2computing vertices: 6messages: 7
total operations: 22total messages: 16
mode: pregel
7
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Execution — Without Combiner
4
0
1
2
1
4
2
4
2
5
6
6
7
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
5
6
7
910
8
iteration: 3computing vertices: 5messages: 6
total operations: 27total messages: 22
mode: pregel
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Execution — Without Combiner
4
0
1
2
1
4
2
4
2
5
5
6
7
7
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
6
iteration: 4computing vertices: 3messages: 1
total operations: 30total messages: 23
mode: pregel
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Execution — Without Combiner
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
iteration: 5computing vertices: 1messages: 0
total operations: 31total messages: 23
mode: pregel
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Execution — Without Combiner
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
total iterations: 5total operations: 31total messages: 23
mode: pregel
Alex Averbuch Big Graph Processing 12 / 36
Pregel — Programming Model Example — Combiner
Code: Combiner program for Single Source Shortest Path (SSSP)
class MinIntCombiner : public Combiner <int > {virtual void Combine(MessageIterator* msgs) {
// initializationint mindist = INF;
// read messages & update mindistfor (; !msgs ->Done(); msgs ->Next())
mindist = min(mindist , msgs ->Value());
// only emit minimum message value (distance)Output("combined_source", mindist);
}};
Alex Averbuch Big Graph Processing 13 / 36
Pregel — Execution — With Combiner
4
*
-
-
-
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — With Combiner
4
0
-
-
-
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
3
1
iteration: 0computing vertices: 13messages: 3
total operations: 13total messages: 3
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — With Combiner
4
0
1
3
1
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
4
2
2
2
5
5
iteration: 1computing vertices: 3messages: 6
total operations: 16total messages: 9
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — With Combiner
4
0
1
2
1
4
2
5
2
5
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
6
6
4
6
7
7
iteration: 2computing vertices: 6messages: 5
total operations: 22total messages: 14
7
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — With Combiner
4
0
1
2
1
4
2
4
2
5
6
6
7
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
5
6
7
910
8
iteration: 3computing vertices: 5messages: 3
total operations: 27total messages: 17
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — With Combiner
4
0
1
2
1
4
2
4
2
5
5
6
7
7
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
6
iteration: 4computing vertices: 3messages: 1
total operations: 30total messages: 18
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — With Combiner
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
iteration: 5computing vertices: 1messages: 0
total operations: 31total messages: 18
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — With Combiner
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
total iterations: 5total operations: 31total messages: 18
mode: pregel (combined)
Alex Averbuch Big Graph Processing 14 / 36
Pregel — Execution — Comparison
• combiner vs no combiner• algorithm → Single Source Shortest Path (SSSP)• sample graph → 13 vertices / 19 edges• cost → iterations, operations, message buffers
Results: Cost comparison of execution modes (SSSP)
Iterations Operations Messages
Pregel 5 31 23Pregel + Combiner 5 31 18
Alex Averbuch Big Graph Processing 15 / 36
Pregel — Architecture
Worker
PartitioninQ
outQ Worker
PartitioninQ
outQ Worker
PartitioninQ
outQ
Graph Dataset
Loading &Checkpointing
(Combined) Messages
MasterSynchronization &Aggregatation
Alex Averbuch Big Graph Processing 16 / 36
Pregel — Typical Program
• Client
1 load input data into workers2 notify master to “start processing”3 wait for master to complete
4 extract result data from workers
• Master1 repeat until no active workers
• signal workers to process• wait for all workers to finish• update active-worker count
2 notify client about completion
• Worker1 repeat until inactive
• wait for “start iteration” from master• read data from in-queue• perform local processing• write data to out-queue & transmit• update active/inactive status• notify master
Alex Averbuch Big Graph Processing 17 / 36
Pregel — Typical Program
• Client
1 load input data into workers2 notify master to “start processing”3 wait for master to complete
4 extract result data from workers
• Master1 repeat until no active workers
• signal workers to process• wait for all workers to finish• update active-worker count
2 notify client about completion
• Worker1 repeat until inactive
• wait for “start iteration” from master• read data from in-queue• perform local processing• write data to out-queue & transmit• update active/inactive status• notify master
Alex Averbuch Big Graph Processing 17 / 36
Pregel — Typical Program
• Client
1 load input data into workers2 notify master to “start processing”3 wait for master to complete
4 extract result data from workers
• Master1 repeat until no active workers
• signal workers to process• wait for all workers to finish• update active-worker count
2 notify client about completion
• Worker1 repeat until inactive
• wait for “start iteration” from master• read data from in-queue• perform local processing• write data to out-queue & transmit• update active/inactive status• notify master
Alex Averbuch Big Graph Processing 17 / 36
Pregel — Typical Program
• Client
1 load input data into workers2 notify master to “start processing”3 wait for master to complete
4 extract result data from workers
• Master1 repeat until no active workers
• signal workers to process• wait for all workers to finish• update active-worker count
2 notify client about completion
• Worker1 repeat until inactive
• wait for “start iteration” from master• read data from in-queue• perform local processing• write data to out-queue & transmit• update active/inactive status• notify master
Alex Averbuch Big Graph Processing 17 / 36
Pregel — Typical Program
• Client
1 load input data into workers2 notify master to “start processing”3 wait for master to complete4 extract result data from workers
• Master1 repeat until no active workers
• signal workers to process• wait for all workers to finish• update active-worker count
2 notify client about completion
• Worker1 repeat until inactive
• wait for “start iteration” from master• read data from in-queue• perform local processing• write data to out-queue & transmit• update active/inactive status• notify master
Alex Averbuch Big Graph Processing 17 / 36
Pregel — Fault Tolerance
• logging1 checkpointing → state persisted at beginning of every n-th iteration
• master persists → progress of execution, aggregate values• workers persist → vertex values, edge values, messages
2 confined recovery → workers log out-messages from their partitions
• failure detection• heart beats• worker gets no heartbeat from master → worker terminates• master gets no heartbeat from worker → marks worker as failed
• failure recovery• partition(s) belonging to failed worker(s) are reassigned• lost partitions recovered from checkpoints• missing iterations recomputed (using logged messages)
Alex Averbuch Big Graph Processing 18 / 36
Pregel — Fault Tolerance
• logging1 checkpointing → state persisted at beginning of every n-th iteration
• master persists → progress of execution, aggregate values• workers persist → vertex values, edge values, messages
2 confined recovery → workers log out-messages from their partitions
• failure detection• heart beats• worker gets no heartbeat from master → worker terminates• master gets no heartbeat from worker → marks worker as failed
• failure recovery• partition(s) belonging to failed worker(s) are reassigned• lost partitions recovered from checkpoints• missing iterations recomputed (using logged messages)
Alex Averbuch Big Graph Processing 18 / 36
Pregel — Fault Tolerance
• logging1 checkpointing → state persisted at beginning of every n-th iteration
• master persists → progress of execution, aggregate values• workers persist → vertex values, edge values, messages
2 confined recovery → workers log out-messages from their partitions
• failure detection• heart beats• worker gets no heartbeat from master → worker terminates• master gets no heartbeat from worker → marks worker as failed
• failure recovery• partition(s) belonging to failed worker(s) are reassigned• lost partitions recovered from checkpoints• missing iterations recomputed (using logged messages)
Alex Averbuch Big Graph Processing 18 / 36
Pregel — Evaluation — Scaling
• algorithm → Single Source Shortest Path (SSSP)
• hardware → 800 worker tasks scheduled on 300 multicore machines
• graph• random, log-normal out-degree distribution (mean = 127.1)• up to 1,000,000,000 vertices / 127,000,000,000 edges
Results: Scalability of Pregel (SSSP)
200 400 600 800 1000
100
200
300
400
500
600
700
800
Number of vertices (millions)
Ru
nti
me
(sec
ond
s)
Alex Averbuch Big Graph Processing 19 / 36
Signal/Collect — Contributions
• parallel programming model (for processing graphs)
• parallel execution model (for processing graphs)
• (limited) evaluation → benefits of various scheduling policies
Alex Averbuch Big Graph Processing 20 / 36
Signal/Collect — Programming Model (Vertex)
• Vertex (v)• v.id → unique identifier• v.state → arbitrary vertex state• v.lastSignalState → v.state at time of last signal()• v.outEdges : List[edge] → list of edges that have v as source• v.signalMap : Map(vid,signal) → last received messages
• vid - identifier of sender vertex• signal - last received signal from that vertex
• v.uncollectedSignals : List[signal] → list of signalsreceived since collect() was last executed
• v.collect() : calculates new vertex state
1 collect incoming signals2 process those signals (possibly using v.state)3 return new vertex state
Alex Averbuch Big Graph Processing 21 / 36
Signal/Collect — Programming Model (Edge)
• Edge (e)• e.source → source vertex• e.sourceId → identifier of source vertex (e.source.id)• e.targetId → identifier of target vertex• e.state → arbitrary edge state• e.signal() → calculates the signal to send, then sends it
• signals are sent along edges of the compute graph
Alex Averbuch Big Graph Processing 22 / 36
Signal/Collect — Programming Model Example
Code: Single Source Shortest Path (SSSP)
class Location(id: Any , initialState: Int) extends Vertex {def collect: Int = min(state , min(uncollectedSignals))
}
class Path(sourceId: Any , targetId: Int) extends Edge {def signal: Int = source.state + weight
}
• vertex state → shortest known path length to vertex from source
• edge state (weight) → path length of that individual edge
• signal → shortest known path length from source through edge
Alex Averbuch Big Graph Processing 23 / 36
Signal/Collect — Execution Model
• different Execution Modes (scheduling policies)
1 synchronous2 synchronous score-guided3 asynchronous4 asynchronous scheduled
• execution mode dictates when signal & collect are called
Definition: Internal methods used by execution engine
procedure v.executeSignalOperation()lastSignalState ← statefor all e ∈ outGoingEdges do
e.target.uncollectedSignals.append(e.signal())e.target.signalMap.put(e.sourceId,e.signal())
procedure v.executeCollectOperation()state ← collect()uncollectedSignals ← Nil
Alex Averbuch Big Graph Processing 24 / 36
Signal/Collect — Execution Model
• different Execution Modes (scheduling policies)
1 synchronous2 synchronous score-guided3 asynchronous4 asynchronous scheduled
• execution mode dictates when signal & collect are called
Definition: Internal methods used by execution engine
procedure v.executeSignalOperation()lastSignalState ← statefor all e ∈ outGoingEdges do
e.target.uncollectedSignals.append(e.signal())e.target.signalMap.put(e.sourceId,e.signal())
procedure v.executeCollectOperation()state ← collect()uncollectedSignals ← Nil
Alex Averbuch Big Graph Processing 24 / 36
Signal/Collect — Execution — Synchronous
• vertex scheduling: all vertices (unordered) per iteration
• termination: all iterations
Scheduler: Synchronous
for i ← 1 to num iterations dofor all v ∈ V parallel do
v.executeSignalOperation()
for all v ∈ V parallel dov.executeCollectOperation()
Alex Averbuch Big Graph Processing 25 / 36
Signal/Collect — Execution — Synchronous
4
*
-
-
-
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
mode: synchronous
Alex Averbuch Big Graph Processing 26 / 36
Signal/Collect — Execution — Synchronous
4
0
1
3
1
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
-
-
-
-
-
3
1 -
-
-
-
-
-
-
--
-
-
iteration: 0signaling vertices: 13collecting vertices: 13messages: 19
total operations: 26total messages: 19
mode: synchronous
Alex Averbuch Big Graph Processing 26 / 36
Signal/Collect — Execution — Synchronous
4
0
1
2
1
4
2
5
2
5
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
4
-
2
-
2
3
1 2
5
-
5
-
-
-
--
-
-
iteration: 1signaling vertices: 13collecting vertices: 13messages: 19
total operations: 52total messages: 38
mode: synchronous
Alex Averbuch Big Graph Processing 26 / 36
Signal/Collect — Execution — Synchronous
4
0
1
2
1
4
2
4
2
5
6
6
7
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
4
6
2
7
2
3
1 2
5
6
4
6
7
-
--
-
7
iteration: 2signaling vertices: 13collecting vertices: 13messages: 19
total operations: 78total messages: 57
mode: synchronous
Alex Averbuch Big Graph Processing 26 / 36
Signal/Collect — Execution — Synchronous
4
0
1
2
1
4
2
4
2
5
5
6
7
7
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
4
6
2
7
2
3
1 2
5
6
4
5
6
7
910
8
7
iteration: 3signaling vertices: 13collecting vertices: 13messages: 19
total operations: 104total messages: 76
mode: synchronous
Alex Averbuch Big Graph Processing 26 / 36
Signal/Collect — Execution — Synchronous
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
4
6
2
7
2
3
1 2
5
6
4
5
6
6
910
8
7
iteration: 4signaling vertices: 13collecting vertices: 13messages: 19
total operations: 130total messages: 95
mode: synchronous
Alex Averbuch Big Graph Processing 26 / 36
Signal/Collect — Execution — Synchronous
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
total iterations: 4total operations: 130total messages: 95
mode: synchronous
Alex Averbuch Big Graph Processing 26 / 36
Signal/Collect — Execution — Synchronous Guided
• vertex scheduling: all active vertices (unordered) per iteration
• termination: all iterations or (no signal and no collect)
Alex Averbuch Big Graph Processing 27 / 36
Signal/Collect — Execution — Synchronous Guided
• vertex scheduling: all active vertices (unordered) per iteration
• termination: all iterations or (no signal and no collect)
• extension v.signalScore() “importance for vertex to signal”• may change ⇐⇒ v.state changes• default → 1 if changed, 0 otherwise
• extension v.collectScore() “importance for vertex to collect”• may change ⇐⇒ v.uncollectedSignals changes• default → v.uncollectedSignals.size()
Alex Averbuch Big Graph Processing 27 / 36
Signal/Collect — Execution — Synchronous Guided
• vertex scheduling: all active vertices (unordered) per iteration
• termination: all iterations or (no signal and no collect)
Scheduler: Synchronous Score-Guided
done ← falsewhile (iterations < num iterations) ∧ (done = false) do
done ← trueiterations← iterations + 1for all v ∈ V parallel do
if v.signalScore() > signal threshold thendone ← falsev.executeSignalOperation()
for all v ∈ V parallel doif v.collectScore() > collect threshold then
done ← falsev.executeCollectOperation()
Alex Averbuch Big Graph Processing 27 / 36
Signal/Collect — Execution — Synchronous Guided
4
*
-
-
-
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 28 / 36
Signal/Collect — Execution — Synchronous Guided
4
0
1
3
1
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
3
1
iteration: 0signaling vertices: 1collecting vertices: 3messages: 3
total operations: 4total messages: 3
mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 28 / 36
Signal/Collect — Execution — Synchronous Guided
4
0
1
2
1
4
2
5
2
5
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
4
2
2
2
5
5
iteration: 1signaling vertices: 3collecting vertices: 6messages: 6
total operations: 13total messages: 9
mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 28 / 36
Signal/Collect — Execution — Synchronous Guided
4
0
1
2
1
4
2
4
2
5
6
6
7
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
6
6
4
6
7
7
7 iteration: 2signaling vertices: 6collecting vertices: 5messages: 7
total operations: 24total messages: 16
mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 28 / 36
Signal/Collect — Execution — Synchronous Guided
4
0
1
2
1
4
2
4
2
5
5
6
7
7
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
5
6
7
910
8
iteration: 3signaling vertices: 4collecting vertices: 3messages: 6
total operations: 31total messages: 22
mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 28 / 36
Signal/Collect — Execution — Synchronous Guided
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
6
iteration: 4signaling vertices: 2collecting vertices: 1messages: 1
total operations: 34total messages: 23
mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 28 / 36
Signal/Collect — Execution — Synchronous Guided
4
0
1
2
1
4
2
4
2
5
5
6
7
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
total iterations: 4total operations: 34total messages: 23
mode: synchronous-guidedscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 28 / 36
Signal/Collect — Execution — Asynchronous
• vertex scheduling: random
• termination: max operations or (no signal and no collect)
Alex Averbuch Big Graph Processing 29 / 36
Signal/Collect — Execution — Asynchronous
• vertex scheduling: random
• termination: max operations or (no signal and no collect)
• no guarantee on order of execution• some vertices may signal while others collect
• no guarantee that all vertices are executed same amount of times
• asynchronous mode not usable for every algorithm• use when correctness not dependent on strict execution order
Alex Averbuch Big Graph Processing 29 / 36
Signal/Collect — Execution — Asynchronous
• vertex scheduling: random
• termination: max operations or (no signal and no collect)
Scheduler: Asynchronous
ops← 0while (ops < num ops) ∧
(∃v ∈ V :
(v.signalScore > signal threshold) ∨ (v.collectScore > collect threshold))
doS ← random subset of Vfor all v ∈ S do
next ← random(signal/collect)if (next = signal) ∧ (v.signalScore > signal threshold) then
v.executeSignalOperation()ops← ops + 1
else if (next = collect) ∧ (v.collectScore > collect threshold) thenv.executeCollectOperation()ops← ops + 1
Alex Averbuch Big Graph Processing 29 / 36
Signal/Collect — Execution — Asynchronous Scheduled
• vertex scheduling: scheduler-dependent
• termination: scheduler-dependent
• scheduler: schedules vertices’ signal & collect operations
• e.g. “eager” scheduler → tries to signal right after collection
Alex Averbuch Big Graph Processing 30 / 36
Signal/Collect — Execution — Asynchronous Scheduled
• vertex scheduling: scheduler-dependent
• termination: scheduler-dependent
• scheduler: schedules vertices’ signal & collect operations• e.g. “eager” scheduler → tries to signal right after collection
Scheduler: “Eager”
for all v ∈ V doif (v.collectScore() > collect threshold) then
v.executeCollectOperation()if (v.signalScore() > signal threshold) then
v.executeSignalOperation()
Alex Averbuch Big Graph Processing 30 / 36
Signal/Collect — Execution — Asynchronous Scheduled
• vertex scheduling: scheduler-dependent
• termination: scheduler-dependent
• scheduler: schedules vertices’ signal & collect operations
• e.g. “eager” scheduler → tries to signal right after collection
• benefit of this execution mode depends on:
1 impact on convergence (number of operations)2 cost of operations3 cost of scheduling
Alex Averbuch Big Graph Processing 30 / 36
Signal/Collect — Execution — Asynchronous Scheduled
Scheduler: “SSSP” — minimize messages & computation
Signal← {v source} // sorted setwhile (Signal 6= {}) do
for top k v ∈ Signal dov.executeSignalOperation()Signal.remove(v)
for all v ∈ V doif (v.collectScore > collect threshold) then
v.executeCollectOperation()if (v.signalScore > signal threshold) then
Signal.put(v)
if (v destination.state ≤ min(Signal)) thenSignal← {}
// returns distance of shortest next stepfunction signalSort(v)
Distances← {∞}for all e ∈ v.outEdges do
Distances.put(e.signal)
return min(Distances)
Alex Averbuch Big Graph Processing 31 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
*
-
-
-
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
0
1
3
1
-
-
-
-
-
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
1
3
1
iteration: 0signaling vertices: 1collecting vertices: 3messages: 3
total operations: 4total messages: 3
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
2
5
2
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
0
1
3
1
-
-
-
2
5
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
2
5
iteration: 1signaling vertices: 1collecting vertices: 2messages: 2
total operations: 7total messages: 5
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
2
5
5
7
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
0
1
2
1
4
2
-
2
5
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
4
2
2
iteration: 2signaling vertices: 1collecting vertices: 3messages: 3
total operations: 11total messages: 8
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
67
4
6
7
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
0
1
2
1
4
2
4
2
5
-
-
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
4
iteration: 3signaling vertices: 1collecting vertices: 1messages: 1
total operations: 13total messages: 9
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
67
5
6
7
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
0
1
2
1
4
2
4
2
5
5
6
-
*
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
iteration: 4signaling vertices: 1collecting vertices: 2messages: 2
total operations: 16total messages: 11
5
6
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
67
6
9
6
7
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
0
1
2
1
4
2
4
2
5
5
6
-
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
6
iteration: 5signaling vertices: 1collecting vertices: 1messages: 1
total operations: 18total messages: 12
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
67
6
9
7
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution — Asynchronous Scheduled
4
0
1
2
1
4
2
4
2
5
5
6
-
6
1
3
1
32
1 5
2
1
1
4
2
3
1
3
2
1
1
iterations: 5total operations: 18total messages: 12
mode: asynchronous-scheduledscoreSignal: state != lastCollectedStatescoreCollect: v.uncollectedSignals() > 0
Alex Averbuch Big Graph Processing 32 / 36
Signal/Collect — Execution Modes — Comparison
• basic comparison of execution modes• algorithm → Single Source Shortest Path (SSSP)• sample graph → 13 vertices / 19 edges• cost → iterations, operations, messages
Results: Cost comparison of execution modes (SSSP)
Iterations Operations Messages
Pregel 5 31* 23Pregel + Combiner 5 31* 18Synchronous 4 130 95Synchronous Guided 4 34 23Asynchronous Scheduled 5 18 12
Alex Averbuch Big Graph Processing 33 / 36
Signal/Collect — Evaluation — Scaling
• algorithm → Single Source Shortest Path (SSSP)
• graph• random• log-normal out-degree distribution• longest path length 5• 1,000,000 vertices / 94,000,000 edges
Results: Scalability of Signal/Collect (SSSP)
0.00
50.00
100.00
150.00
200.00
250.00
300.00
1 2 3 4 5 6 7 8
Tim
e (
secon
ds)
# Worker Threads
AverageFastestSlowest
1
2
3
4
5
6
7
8
1 2 3 4 5 6 7 8
Sp
eed
up
# Worker Threads
Measured Speedup(Average)
(a) Scaling with additional workers (b) Parallel speedup
Alex Averbuch Big Graph Processing 34 / 36
Signal/Collect — Evaluation — Execution Modes
Results: Comparison of Execution Modes (PageRank)
0100200300400500600700800900
1,000
Synchronous Score-GuidedSynchronous
"Eager" Score-GuidedAsynchronous
"Above Average" Score-GuidedAsynchronous
0
50,000
100,000
150,000
200,000
250,000
300,000
Synchronous Score-GuidedSynchronous
"Eager" Score-GuidedAsynchronous
"Above Average" Score-GuidedAsynchronous
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
Synchronous Score-GuidedSynchronous
"Eager" Score-GuidedAsynchronous
"Above Average" Score-GuidedAsynchronous
0
10,000
20,000
30,000
40,000
50,000
60,000
Synchronous Score-GuidedSynchronous
"Eager" Score-GuidedAsynchronous
"Above Average" Score-GuidedAsynchronous
Average Computation Time (ms) Average # of Signal Operations Executed
Average Computation Time (ms) Average # of Signal Operations Executed
PageRank on SwetoDblp citations22,387 vertices (publications) / 112,303 edges (citations)
PageRank on a generated graph100,000 vertices / 1,284,495 edges / log-normal out-degree distribution
Alex Averbuch Big Graph Processing 35 / 36
Signal/Collect — Evaluation — Convergence
• algorithm → Vertex Coloring
• graph• random• log-normal out-degree distribution• 100,000 vertices / 554,118 edges
• experiment• measure time for algorithm to converge• results are averaged over 10 runs• noted as “did not converge” (8) when time exceeded one minute
Results: Vertex Coloring convergence times (ms)
Number of colors 5 6 7 8 9 10 11
Asynchronous“Eager”
8 12,870 1,690 1,392 1,243 1,218 1,046
SynchronousScore-Guided
8 8 8 8 8 2,856 1,876
Alex Averbuch Big Graph Processing 36 / 36
top related