RisGraph: A Real-Time Streaming System for Evolving Graphs

Guanyu FengTsinghua University

Zixuan MaTsinghua University

Daixuan LiTsinghua University

Xiaowei ZhuTsinghua University

Yanzheng CaiTsinghua University

Wentao HanTsinghua University

Wenguang ChenTsinghua University

AbstractGraphs in the real world are constantly changing and of

large scale. In processing these evolving graphs, the com-bination of update workloads (updating vertices and edgesin a streaming manner) and analytical (performing graphalgorithms incrementally) workloads is ubiquitous. Through-put, latency, and granularity are three key requirements inprocessing evolving graphs with such combined workloads.Although there are several streaming systems proposed forevolving graphs to improve latency. They usually use batch-update model to improve throughput but hurt granularity. It isstill challenging to fulfill all the requirements simultaneously,especially for power-law graphs because they are difficult tobe partitioned.

We analyze the computational cost on synthesized power-law graphs and realistic evolving graphs from public datasets.We find that the affected areas are usually small for eachupdate, and there are scheduling opportunities for combinedworkloads. Based on these observations, we design a real-timestreaming system for incremental graph computing calledRisGraph. Our novel design on scheduling, trade-offs on datastructures and the computing engine make RisGraph satisfythe three requirements at the same time. The evaluation showsRisGraph can ingest millions of updates per second and its99.9%ile latency is within 20 milliseconds for graphs withhundreds of millions of vertices and billions of edges on asingle commodity machine.

1 Introduction

Graph computing techniques have been developing rapidly inrecent years. On one hand, the scale of graph-structured datacould be enormous, with the numbers of vertices and edgesusually more than hundreds of millions. On the other hand,most realistic graphs are constantly changing [10], which areknown as evolving graphs.

There are two different kinds of workloads for evolvinggraphs: update workloads and analytical workloads. In update

workloads, vertices and edges are continuously being updatedand read. In analytical workloads, different graph algorithms,like shortest path and connected components, are executed toextract information from the current graph data.

The combination of these two types of workloads is ubiq-uitous. Take the following two as examples:

• In the e-commerce and personal finance domain, it isnecessary to know the security and legality of a transac-tion (i.e., maybe a few updates on the graph) in time todecide whether it is acceptable. For example, cycles mayindicate fake transactions [47], and a short distance fromblacklists is suspicious. Detecting these patterns requiressupport for efficient updates and real-time analysis onevolving graphs.

• In the social network and video-sharing domain, illegalinformation needs to be banned promptly. Some illegalitems newly reported or detected are a type of updateworkload. Real-time graph analysis can help to find outmore related illegal information as soon as possible aftersuch updates are applied.

To achieve this combined workloads, throughput, latency,and granularity are the three key requirements. Through-put is always a high priority for systems to support evolvinggraphs. Latency is also critical here as the results of analysison evolving graphs need to be updated within the latency con-straint, which depends on the application scenario. For typicalfraud detection scenarios, decisions must be made in severalmilliseconds [47]. If the execution time of either updates oranalysis is too long, the detection will become inaccurate oreven useless. The granularity of updates and analysis is alsopivotal. In some scenarios, users require detailed informationfrom analysis after fine-grained updates are applied. Batchupdates are not suitable if the detailed information is essen-tial, since it is difficult to isolate the impact of a single updatewithin a batch or query the states after performing a specificupdate within a batch. Some meaningful details will be lost,for example, finding out which specific behavior causes a user

1

arX

iv:2

004.

0080

3v1

[cs

.DB

] 2

Apr

202

0

to be banned or marked as suspicious. It is non-trivial becauseresults may be caused by indirect modifications.

It is difficult to fulfill all the requirements above, and theproblem is still not well-solved because the performance gapbetween updates and analysis could be significant. A singleupdate only costs several microseconds, while the time ofperforming even simple analysis like BFS is on the order ofseconds [30,54,63]. The dilemma here is that analytical work-loads need to scan a large amount of data or even the entiregraph, while update workloads only touch a small neighboringarea.

Existing solutions. Recent graph systems have made con-siderable progress for evolving graph processing, but they stillhave limitations and shortcomings.

One naive solution is to use existing graph engines andrecompute new results on the whole graph after updates areapplied. Recent researches accelerate graph analysis with effi-cient and compact data layout [54], utilizing accelerators [53],scaling to large distributed clusters [63] and even using su-percomputers [35]. However, if the graph is very large, thenETL (Extract, Transform and Load) time would limit the per-formance of this approach and thus fail to match the latencyrequirement.

Recent papers have proposed two different ways to re-duce ETL cost. One is adapting graph engines to support up-dates [30] (usually in the form of batch updates), and the otheris making databases support efficient analysis [64]. These twomethods focus on update workloads, but they still suffer fromrecomputing on the whole updated graph.

Incremental computing is another approach. It leveragesprevious results and some auxiliary information to reduceuseless computing and accelerate analytical performance.There have been quite some incremental streaming sys-tems [17, 40, 41, 45, 50, 52, 57, 58] proposing various com-puting models but all these systems suffered from the coarseupdate granularity. These streaming systems with batch up-dates can achieve sufficient throughput but the latency andgranularity issues are still unsolved.

Tornado [39] is a real-time streaming system but the cor-rectness of Tornado is questionable [58]. GraphS [47] dynami-cally detects cycles based on its indexes, targeting fine-grainedupdates, and is able to guarantee a 20 ms latency for 99.9%updates. The indexes from GraphS fulfill all requirements onits specific problem, but it is ad-hoc and depth-limited.

Some papers propose hierarchical algorithms. This cate-gory of methods works effectively on specific graphs that canbe easily partitioned such as road networks [49]. However,not all graphs can be easily partitioned. PowerGraph [23]indicates that power-law graphs, also called scale-free net-works, are difficult to be partitioned [23]. The degrees ofvertices in power-law graphs follow the power law distribu-tion. Many real-world graphs, especially most large-scale real-world graphs follow the power law, e.g., social networks [43],web graphs [9] and financial graphs [47].

Open Challenges. The gap between existing solutions andideal combined workloads prompts us to find a solution. Wefocus on power-law graphs, which are widespread but not wellsolved. A natural question is whether a real-time streamingsystem could efficiently handle power-law graphs. It seemsimpractical to solve the latency problem on power-law graphsbecause of the highest-degree vertices, hubs. Once a hub ismodified, it will cause numerous related changes, resulting ina huge latency. Power-law graphs and hubs challenge us moreseverely. Existing literature proposed streaming systems withbatch updates. Latency is not quite essential for batch-updatesystems, but it is crucial for real-time streaming systems.

Our Contributions. We design a real-time streaming sys-tem called RisGraph. It targets a class of widely used algo-rithms, monotonic algorithms, whose dependence of resultscan be expressed by a tree or a forest, such as Breadth FirstSearch (BFS), Single Source Shortest Path (SSSP) and Min/-Max Label Propagation. The incremental computing modelis inspired by KickStarter [58], a state-of-the-art streamingsystem with batch updates.

RisGraph can support both fine-grained updates and in-cremental analysis. For graphs with hundreds of millions ofvertices and billions of edges, RisGraph can ingest millionsof updates per second while ensuring more than 99.9% up-dates to be responded within 20 milliseconds, and present theimpact of each update in analysis.

The contributions of this paper are the following:

• To verify the feasibility, we analyze the computationalcost with fine-grained updates on several common publicdatasets and synthetic power-law graphs. We give anoptimistic answer that our goal is achievable. From ouranalysis, we draw several insights, which guide designsof RisGraph. (Section 3)

• Based on our insights, we dig up scheduling opportuni-ties for RisGraph. We design a scheduler, which can sig-nificantly increase throughput under latency constraintswithout compromising consistency and granularity. (Sec-tion 3 and Section 4.5)

• We make novel trade-offs derived from our insights. Thegraph storage supports both efficient updates and anal-ysis. The data structure for dependence tree is friendlyto our scheduler. Our computing engine is optimized forincremental computing with fine-grained updates. Weadopt techniques different from most graph computingframeworks. (Section 4)

2 Background

Given a graph G, affected area [20, 21] (AFF) is the areainspected by an incremental algorithm with updates ∆G. Toconsider the computational complexities , incremental algo-rithm costs [22] are affected or determined by |AFF| and |∆G|.

2

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

Ⅰ Ⅱ Ⅲ Ⅳ Ⅴ

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

0 1

2 3

Label = 0

Label = 1

Label = 3

Figure 1: An example of the model in RisGraph on the weak connected component algorithm.

Vertices are the emphasis of AFF because when a vertex isadded to AFF, the related edges are usually in AFF. We divideAFF into three categories, Updated set, Invalidated set andPropagated set.

Updated set refers to the vertices involved by updates ∆G.For edge modifications (vs,vt) and vertex modifications vx in∆G, the incremental algorithm should check if these modifica-tions change their output, in other words, vs, vt and vx are inUpdated set. There may be some inconsistency in the resultswhen updates occur. Some vertices need to be reset and re-computed if the correctness cannot recover based on existingvalues. These vertices build up Invalidate set. The algorithmwill propagate changes from Updated set and Invalidate set.Propagated set is defined by the vertices which are checkedor modified during the process of convergence.

The difficulty of locating AFF lies in the Invalidate setbecause locating Invalidate set usually requires recording andcomparing with the previous calculation process or informa-tion rather than just the results. For example, if we need todelete an edge and incrementally maintain the single-sourcereachability, we can’t indicate which vertex will be unreach-able just from the previous reachability.

GraphBolt [40] and KickStarter [58] exploit dependencegraph to represent the value of a vertex depends on which ver-tices. An edge (vs,vt) in dependence graph indicates that theresult of vs depends on the result of vt and the edge betweenvs and vt . vs should be re-computed if vt is in the Invalidateset, or the edge (vt ,vs) is deleted. We adopt the model fromKickStarter. It works when the dependence graph is a tree ora forest, and the assumption always holds for general mono-tonic graph algorithms, such as Reachability, Shortest Path,Weak Connected Components, Widest Path, etc.

Next, we will give an example of how the model in Ris-Graph works on the weak connected component algorithm inFigure 1. The algorithm is to give a label to each vertex onan undirected graph, and propagates labels iteratively. Eachvertex maintains and propagates the minimal visible labeluntil all vertices converge. Vertices in the same connectedcomponent will hold the same label. There are four verticesand five states in Figure 1. Vertex labels are initialized bythe vertex IDs in circles. Labels are represented by the col-ors of the vertices, as shown in the legend on the right. Weadopt black straight lines to express edges in the graphs, anddark blue arc arrows to indicate the dependence tree or the

forest. Black and dark blue dashed lines are newly addededges. Light-colored dash-dot lines are deleted edges. Romannumerals I to V indicate the initial state and the state aftermodifications. The other sub-graphs are intermediate statesand the process of state changes is indicated by arrows.

At State I, the label of v0 is 0 (marked by white), and thelabels of v1, v2 and v3 are 1 (marked by light green). (v2,v1)and (v3,v1) are the dependence tree because labels of v2 andv3 are propagated from v1.

After adding the edge linked v2 and v3, State II is similar toStage I, shown in the figure. The dependence tree from State Iis still acceptable at State II.

Then, adding an edge with v0 and v2 makes v2 meet asmaller label 0. v2 will update its label from 1 to 0, at thesame time v2 needs to delete the dependence (v2,v1) and adda new dependence (v2,v0). Then v2 iteratively propagates itsnew label to v1 and v3. Dependence (v1,v2) and (v3,v2) willestablish and (v3,v1) will be deleted. After the propagation,all vertices have label 0, meaning that all vertices belong to aconnected component. Sub-figures from State II to State IIIshow this process.

As a result of removing the edge between v1 and v2, the de-pendence (v1,v2) is invalid. v1 resets its label to the vertex ID1. Then v1 pulls labels from its neighborhoods, sets its labelby the minimal labels, adds a dependence, and propagates itsnew label. In this example, v1 get 0 from v3. State IV is theresult affected by this modification.

Finally, cut the link between v2 and v3. v3 need to be resetbecause the dependence (v3,v2) is invalid. The invalidationof v3 will make v1 invalid because of the dependence (v1,v3).The labels of v1 and v3 are 1 (light green) and 3 (dark blue)now. v1 and v3 propagate their new labels, therefore, v3 labelbecomes 1 with a dependence (v3,v1), as shown in State V.

3 Analysis and Insights

Analyzing the size of AFF is necessary for RisGraph becauseincremental computing does not provide substantial changesin algorithm theoretical complexity compared to direct re-computing. Especially when dealing with power-law graphs,hubs make the problem more critical. Intuitively, it is morelikely that modifying an edge affects the hubs. Once a hubis modified, a large number of vertices may be affected. The

3

size of AFF is not quite important for batch-update systems,so this problem is not well solved. AFF generated by multipleupdates in a batch can be processed in parallel, the intersectingparts of AFF (such as the impact of the same hub) need tobe processed only once. However, it is crucial for real-timestreaming systems.

3.1 Methodology

We only need to focus on adding and deleting edges becauseadding vertices does not produce AFF (isolated points inthe model cannot affect other vertices), and deleting verticescan be represented by a series of edge deletions. We analyzethe AFF of edges (AFFE ) and the AFF of vertices (AFFV ).AFFE can be estimated by AFFV , generally, when vi belongsto AFFV , the edge related to vi will belong to AFFE . So, wemainly analyze AFFV .

For a graph G0 with a dependence tree or forest T 0, afteradding an undirected edge linked with vi and v j, a new graphG1 and its dependence tree T 1 forms. For the new graph G1,if (vi,v j) and (v j,vi) are not in T 1, the edge only affects twovertices vi and v j but don’t change any value, so |AFFV | is 2,and |AFFE | is 1. If (vi,v j) is in T 1 and T 1

i is the subtree of vi(including vi) in T 1, then |AFFV | must be less than or equalto

∣∣T 1i

∣∣+1, because all vertices that depend on the value ofvi in G1 are in T 1

i . The same conclusion also works whendeleting an edge based on G1 and constructing G2 and T 2.Based on the discussion above, we can estimate |AFFV | on asingle version G1 (the whole graph), run algorithms to buildup T 1, and analyze |AFFV | and |AFFE | based on related

∣∣T 1i

∣∣for a set of updates. |AFFE | is computed by the sum of edgesassociated with each vertex in the subtree from T 1.

3.2 Analysis

Breadth First Search (BFS), Single Source Shortest Path(SSSP), Single Source Widest Path (SSWP) and Weak Con-nected Component (WCC) are the four algorithms we analyze.We randomly select the root of BFS, SSSP and SSWP for hun-dreds of times and also run multiple times for each algorithm.To avoid the impact of root selection and the randomness ofparallelism, we use the result with the largest |AFFV | (theaverage |AFFV |) in analysis.

We choose Kronecker graph generator with noising [33,51]and parameters from Graph500 [1] for synthesized graphs.We generate graphs with 64M vertices, and change the aver-age degrees from 1 to 64, up to 4B edges. Table 1 lists fivereal-world datasets used in our analysis. Twitter-2010 andStackoverflow are social networks. UK2007-05 and Subdo-main are web graphs. Wiki is an interaction network.

Figure 2 shows the results on synthesized power-lawgraphs. We only present results of WCC which lead to thelargest |AFFV | due to the page limit. Table 2 summarizes the

Graphs Vertices EdgesWiki (WK) [48] 2.13M 9.00MStackOverflow (SO) [34] 2.60M 63.5MTwitter-2010 (TT) [34] 41.7M 1.47BSubdomain (SD) [7] 102M 2.04BUK-2007 (UK) [11, 12] 106M 3.74B

Table 1: Real-world datasets used in the experiment

1 2 4 8 16 32 64Average degree

0

5

10

15

20

25

Affe

cted

Are

a

Edge Vertex

Figure 2: |AFFV | and |AFFE | against scale of synthesizedgraph

|AFFV | and |AFFE | for real-world power-law graphs. Com-paring to the size of datasets, the average size of AFF isextremely small. KickStarter’s incremental processing model,which we use in RisGraph, can obviously reduce computingtime. Therefore, we have opportunities to design and imple-ment a real-time streaming system.

Only 0.01% updates will trigger AFFE which is larger than10K. Therefore, the affected areas are small in most cases.The conclusion further illustrates the feasibility of real-timeincremental computing.

However, the affected areas could be very large. For thelargest one-millionth updates, the average |AFFV | is 1.53Mand the average |AFFE | is 195M on Twitter-2010. Althoughthe probability seems quite small, it actually often happens,in the case that the system supports high throughput.

Table 2 shows the probability of modifying the dependencetree. The probability is only about 10% to 20%. For synthe-sized graphs, the probabilities increase from 4% to 23% asthe average degree grows from 1 to 64. Obviously, when anupdate does not hit the dependence tree, it will not affectthe results. These updates can be applied in parallel with-out breaking the consistency and correctness of the results.Therefore, we can schedule updates to achieve a trade-offbetween latency and throughput.

3.3 InsightsAfter analyzing the results of many algorithms and datasets,we find that the average and most of |AFFV | and |AFFE | aresmall. We can give an optimistic conclusion that the modelused by RisGraph can work efficiently for most real-world

4

BFS SSSP SSWP WCC|AFFV | |AFFE | TP |AFFV | |AFFE | TP |AFFV | |AFFE | TP |AFFV | |AFFE | TP

WK 13.12 421.84 0.2223 17.64 423.91 0.1592 4.28 68.21 0.1961 17.18 572.82 0.2187SO 3.61 143.13 0.0866 3.28 139.79 0.0387 1.52 54.08 0.0426 3.34 224.76 0.0432TT 3.55 253.05 0.0460 1.29 20.76 0.0308 1.14 12.06 0.0312 9.04 1048.73 0.1832SD 1.26 18.58 0.0437 1.37 17.37 0.0403 1.27 223.46 0.0400 8.96 675.33 0.0434UK 1.03 5.65 0.0301 1.66 61.26 0.0348 28.09 2033.57 0.0344 1.44 51.28 0.0315

Table 2: |AFFV |, |AFFE | and the probability of modifying the dependence tree (TP) in real-world datasets

add/del_edge (edge) → version_idupdate_edge (edge, new_edge_data) → version_idadd/del_vertex ( vertex_id ) → version_idget_value ( version_id , vertex_id )→ valueget_parent ( version_id , vertex_id )→ edgeget_current_version () → version_idget_modified_vertices ( version_id ) → vertex_idsclear_history ( version_id )

Table 3: APIs of RisGraph

power-law graphs. From this perspective, RisGraph shouldbe designed to optimize small AFF situations.

But, AFF could be huge sometimes. An update may affectmillions of edges and vertices. We need to optimize RisGraphfor these updates to reach millions of throughput and to pro-vide acceptable tail latency.

The possibility of trade-off shows that RisGraph can po-tentially achieve higher throughput by rescheduling updatesfrom different sessions. To achieve such capabilities, Ris-Graph needs to support quickly classifying updates into twocategories: updates that may modify results (called unsafeupdates) and updates that do not modify results (called safeupdates).

Based on the findings and insights above, we summarizeour guidelines when designing RisGraph. RisGraph needs tobe optimized for small AFF, efficient in large AFF with smallprobability, and friendly to the scheduler.

4 Design and Implementation

RisGraph maintains the multi-version results of a specificalgorithm. Table 3 shows the APIs of RisGraph. When Ris-Graph receives an update, RisGraph will modify the graph,calculate new results, and return a version ID to the user(the first part of Table 3). Users can get the results and thedependence tree of any version according to the version IDand vertex ID by get_value and get_parent. RisGraph sup-ports querying the current version (get_current_version)and querying which vertices have been modified in a spe-cific version (get_modified_vertices). Users can also useclear_history to delete all historical versions before a cer-tain version.

A series of user-defined functions (UDF) listed in Table 4constitute the user’s algorithms. init_value defines the initialvalue of each vertex. In addition to calculating the results at

init_value (vid) → init_valueneed_update (edge, src_value , dst_value )→ is_neededupdate_value (edge, src_value , dst_value )→ new_valueequal_value (edge, src_value , dst_value )→ is_equal

Table 4: UDFs to describe the algorithm in RisGraph

Classifier

GraphTree & Value History

SchedulerComputing

EngineNew

VersionUpdate

CurrentVersion

Storage

Safe

Unsafe

Figure 3: The overview of RisGraph

RisGraph startup, the initial values are also used when ver-tices being reset. need_update and update_value are twofunctions used to update the results monotonously (e.g., relax-ation in the shortest path algorithm). need_update uses theweight of a directed edge and the source vertex value to de-cide whether the destination vertex value should be updated.update_value uses the same information as need_updatebut returns the new value. equal_value is very similar toneed_update. It checks whether the destination vertex valueis equal to the value calculated from the source vertex valueand the weight on the edge.

The overview of RisGraph is shown in Figure 3. RisGraphcontains several modules: storage, computing engine, classi-fier, and scheduler. Two modules connected by dashed lineswill cooperate. Arrows indicate how an update will be pro-cessed into a version ID that can be used via the APIs.

Three parts make up the storage modules: graph storage,tree and value storage, and history storage. Graph storagemaintains the current graph structure and supports efficientmodification and traversal. Tree and value storage stores thecurrent dependence tree and the current result of each vertex.Updates of results will be directly applied into tree and valuestorage during incremental computing. Changes of values anddependence trees will generate new versions, and be tracedby history storage. Users can directly interact with the historystorage through the APIs related to version ID.

The classifier module classifies updates into two categories,

5

0 1

2 3

Top-Down

0 1

2 3

0 1

2 3

0 1

2 3

Bottom-Up

Figure 4: An example of maintaining the dependence tree.

safe updates and unsafe updates, based on the informationin the tree and value storage. Safe updates will not affect re-sults. Safe updates include edge modifications that will notchange the results and all vertex modifications. Edge modi-fications that will not change the results are edge additionswith need_update return false, and edge deletions off the de-pendence tree. And isolated vertices will not affect the resultsin our model, so all vertex modifications are safe updates.Correspondingly, unsafe updates may modify results. Theclassifier writes unsafe updates into a first-in, first-out (FIFO)queue of the scheduler. The classifier applies safe updates intograph storage in parallel, and gives users the current versionID. It will not break the consistency if these updates belongto different sessions.

The computing engine leverages information in the storagemodule to incrementally compute unsafe updates, modifiesresults and writes new results into the storage module. Thecomputing engine processes unsafe updates in the FIFO queueone by one. However, the computing for a single update is inparallel.

The scheduler performs scheduling under the user’s lim-itation, based on information of updates in the FIFO queueand the situation of the computing engine. The schedulerwill control the computing engine and classifier to periodi-cally alternate between safe and unsafe updates, and achievea balanced trade-off between throughput and latency.

4.1 Tree and Value StorageIn RisGraph, each vertex maintains at most one dependenceedge. The value of the vertex is determined by its dependenceedge and the other vertex targeted by this edge. These edgesform the dependence tree, and each edge is the parent pointer(bottom-up pointer) in the tree. So, the dependence tree ofRisGraph is stored by parent pointer tree [59].

The usual method is maintaining top-down pointers fromthe parent to children. Top-down pointers are friendly forwalking down because the parent vertex can locate all itschildren vertices by scanning the edges on the dependencetree rather than scanning all its edges on the graph. However,bottom-up pointers perform better in other aspects.

Querying whether an edge is on the dependence tree isthe most common operation used by the classifier modulein RisGraph. Using bottom-up pointer is simpler and morestraightforward than using top-down pointers. It is necessary

to query whether the parent has a child with top-down pointers,which requires complex data structures. But the parent pointertree only needs to check the parent pointer of the child.

And, the modifications on the parent pointer tree duringcomputing are much more lightweight than the tree repre-sented by top-down pointers. With top-down pointers, updat-ing the value of a vertex requires locking three vertices, themodified vertex, the current parent vertex, and the new par-ent after modification. However, the parent pointer tree onlyneeds to lock one vertex (the modified vertex) in this case.Figure 4 shows an example. It needs to modify (lock) v0,v1, and v2 when changing the parent of v2 from v1 to v0 bytop-down pointers. However, the tree with bottom-up point-ers only needs to modify v2. This format can eliminate thelock contention caused by hubs when modifying the depen-dence tree because each vertex maintains at most one edge.Atomic operations (when the length is less than the maximumCAS length) and Hardware Transactional Memory (HTM)can make a vertex modification lock-free if packing the parentpointer and the value of a vertex together.

4.2 Graph Storage

Graph is stored in adjacency lists composed by an array ofarrays and indexes in RisGraph. Each vertex maintains anadjacency list in a dynamic size array (doubling capacitywhen full). Each element in the list represents a directed edgefrom the vertex, consists of the destination vertex ID, theweight on the edge and the number of its identical edges.Edges are identical if the destination and the weight are boththe same. Each vertex also contains an index, which representsthe location of the edge in the list. The key of an edge is apair of its destination vertex ID and its weight. We only makeindexes for the vertices whose degree is greater than a certainthreshold because of memory consumption.

For the operation of adding an edge, RisGraph first checksin the index whether the edge exists in the adjacency list. If theedge exists, RisGraph only needs to modify the number of theedge. If not, RisGraph appends an element to the adjacencylist and updates the index. For deleting an edge, our systemmodifies the number of edges after searching from the index.RisGraph keeps zero edges but does not really delete themin the list. Our system will recycle these zero edges andtheir indexes when doubling the adjacency list. RisGraphwill recycle the vertex ID into a pool when deleting a vertex.When adding a vertex, RisGraph will assign a new vertex IDor use an ID from the recycling pool.

Such a data structure can support both efficient fine-grainedupdates and incremental computing. RisGraph uses Hash Ta-bles as the indexes because there are no range operationsfor all the modifications and queries on indexes. Our datastructure including adjacency lists and indexes can ensurethe average O(1) time complexity of insertions and deletions.The computing engine can directly access the adjacency lists

6

without involving the indexes. Adjacency lists can ensure thatall the outgoing edges of a vertex can be continuously stored,which is very important for the computing performance. Nat-urally, our data structure is not as compact as CompressedSparse Row (CSR) used by some streaming systems withbatch updates [40]However, the cost of modification on CSRis a disaster for real-time streaming systems with fine-grainedupdates. Rebuilding the CSR once will take seconds. Withour data structure, RisGraph only takes a few microsecondsper modification. Meanwhile, the conclusions in Section 3show that AFF is frequently small, therefore our data structureshould be optimized towards faster modifications first. Somesystems, for example, LiveGraph [64] and GraphOne [30]have shown that the array of arrays is comparable with CSRin graph computing.

There are many different data structures that can replaceHash Table for indexes, such as BTree and ARTree [32] (theadaptive radix tree). According to the theoretical complexityand our experiments, the performance of Hash Table is thebest. To maximize performance, we choose Hash Table, eventhough it is non-optimal in memory consumption. A trendis that the capacity of memory is increasing, and the priceis decreasing. Byte-addressable non-volatile memories canfurther alleviate this problem. We compare the performanceand memory footprint of Hash Table, BTree and ARTree inthe evaluation. Excessive comparisons of different indexes arebeyond the scope of our paper. RisGraph is also flexible. Userscan select internal indexes from configurations or implementcustom indexes by a little coding.

4.3 History StorageRisGraph maintains a linked list from the new version to theold version for each vertex, to implement the history storage.This method is very similar to the version chaining in a multi-versioned database, which is generally efficient in practice[60]. A potential problem is that there may be many versionsof a single vertex, so clients would have to traverse a longversion chain to access an old version. But our experimentsshow that it almost never happens. This conclusion has beensupported on all algorithms and all datasets used in Section 3.For billions of edges, after adding and deleting 10% of theedges, the longest version chain is still shorter than 100. Skiplists can solve the traversal trouble, but it will slow down theperformance of updates. Sticking to higher performance, wechoose not to use skip lists and keep our design. The historystorage is only responsible for recording short-term historicalinformation. Changes can be exported to external key-valuestores or time series databases in streaming.

4.4 Computing EngineThe papers aimed at graph computing have discussed variousmethods in parallelization schemes, traversal directions, data

10K 100K 1M 10MNumber of active vertices

1K

10K

100K

1M

10M

100M

1B

Num

ber o

f edg

esfro

m a

ctiv

e ve

rtice

s

linear-classifiervertex-paralleledge-parallel

Figure 5: The performance comparison of edge-parallel andvertex-parallel on UK-2007

layouts, etc. These works inspired us. The goal of these papersis to optimize computing for the whole graph, so the designsand trade-offs do not fully meet the requirements of the incre-mental computing scenario. From our analysis, incrementalcomputing only involves few vertices in most cases, whichis different from traditional graph computing. Therefore, wemake some modifications to fit in the incremental computingscenario.

We always choose sparse arrays to store the active vertices,and we will convert the sparse arrays to bitmaps only whenwe have to perform pull operations like ligra [54]. Sparsearrays can avoid useless vertices checking and the overheadof clearing the bitmaps. We create a separate sparse array foreach thread, which helps eliminate the overhead of synchro-nization and contention in multiple-threads.

For the most commonly used push operations, it is bet-ter to use edge-parallel than using vertex-parallel in manycases. This is different from the common conclusions in graphcomputing [61]. RisGraph rarely activates a lot of verticesin incremental computing according to our analysis in Sec-tion 3. Vertex-parallel hardly achieves sufficient parallelismand good load balancing when few vertices are active. Thence,finer-grained parallelism (edge-parallel) is potentially better.

Figure 5 shows the results of comparing edge-parallel andvertex-parallel on UK-2007 dataset running various algo-rithms. The x-coordinate is the number of active vertices. They-coordinate is the out-degrees of active vertices. We takethe natural logarithm of the two parameters and keep onedecimal place. We average the time of push operations, andonly keep the results where the difference is greater than 20%.Red dots mean edge-parallel is better. Blue crosses are wherevertex-parallel is better.

When there are fewer active vertices and more active edges(top left corner of the figure), edge-parallel is better. So, wecan hybrid edge-parallel and vertex-parallel, and potentiallyget better performance. The black straight line1 is the linearclassifier trained by machine learning methods. In our evalua-

1The equation is −2.21lnx+0.58lny = 14.52. x is the number of activevertices and y is out-degrees of the active vertices.

7

tions, using the linear classifier to hybrid edge-parallel andvertex-parallel can give RisGraph better performance thanusing one strategy alone in most cases.

4.5 SchedulerThe goal of the scheduler is to achieve higher throughputwhile satisfying latency constraints. The constraints requirea given percentage of updates to be applied in a specifiedlatency. The possibility of scheduling is that the safe updatescan be executed in parallel we mentioned in Section 3. Inorder to ensure consistency, the scheduler can only schedulebetween different sessions. The scheduler should guaranteethat RisGraph stops classifying updates and applying safeupdates when performing unsafe updates, and vice versa. Oth-erwise, the results will be incorrect. More safe updates whichcan be executed in parallel are obviously useful to improvethroughput. However, consistency and correctness, and la-tency constraints limit the scheduling opportunity.

The scheduler controls the computing engine and the clas-sifier to execute alternately. Each time when the schedulerswitches to unsafe updates, it will wait for the computingengine to process all unsafe updates in the queue. The sched-uler aborts parallel classification and turns to process safeupdates according to two strategies. One is when the ear-liest unsafe update in the queue almost exceeds the targetlatency. Another one is when the number of unprocessedunsafe updates reached a threshold. This threshold is dynam-ically adjusted based on historical information. If the ratioof qualified updates (under the latency limitation) is higherthan the constraints for multiple consecutive switchings, thescheduler will slowly increase the threshold. If the ratio islower than the constraints, the scheduler will quickly decreasethe threshold. Both increasing and decreasing of the thresholdare exponential changes.

5 Evaluation

5.1 Experimental SetupWe use four algorithms to evaluate our system, includingWeak Connected Components (WCC), Breath First Search(BFS), Single Source Shortest Paths (SSSP), and SingleSource Widest Paths (SSWP). Table 1 lists five real-worlddatasets used in the evaluation. Similar to KickStarter [58] andGraphBolt [40], some edges from the dataset is pre-populatedbefore updates. We load 90% edges first, select 10% edgesas the deletion updates from loaded edges, and treat the re-maining (10%) edges as the addition updates. the If datasetsare timestamped (StackOverflow and Wiki2), we will choosethe latest 10% as the addition set and the oldest 10% as thedeletion set. If not, we will randomly select edges as updates.

2Since the Wiki dataset is a bipartite graph, we treat each edge as anundirected edge.

We alternately stream the addition set and the deletion set. Wepreload 90% of the graph rather than 50% used in previous lit-erature because selecting only half of the edges might changethe characteristics of these graphs. For example, loading onlythe base 50% edges of the UK-2007, the average degree isreduced by half (17.7), with 7M vertices (7.2% of the totalvertices) being isolated.

All experiments are conducted on a dual-socket server. Theserver has two Intel Xeon Gold 6126 CPU (12 cores), 576GBmain memory, Intel P3608 4TB SSD, and runs 64-bit Ubuntu18.04 with kernel 4.15.

5.2 Comparison of Implementation Choices

We evaluate various implementations of the modules dis-cussed in Section 4. The scheduler and history storage aredisabled in this part. We classify all updates into safe updatesand unsafe updates, apply all safe updates in parallel, and thenapply unsafe updates one by one. The purpose is to clearlyshow the impact of different designs.

Computing Engine We first evaluate our computing en-gine by comparing it with the performance of vertex-parallel,edge-parallel and our hybrid-parallel strategies. We focuson unsafe updates, and keep the adjacency lists in arrays toeliminate the impact of data structures.

SO 1.74 1.08 0.86 0.80

SD 1.21 1.60 1.18 1.06

TT 1.48 1.27 1.00 1.29

UK 1.20 0.94 1.46 0.98

WK 1.63 1.28 1.23 1.57BFS SSSP SSWP WCC

(a) edge-parallel

1.76 1.18 1.04 0.99 SO

1.19 1.67 1.21 1.09 SD

1.50 1.26 1.02 1.39 TT

1.25 1.38 1.45 1.14 UK

1.73 1.34 1.54 1.99 WKBFS SSSP SSWP WCC

(b) hybrid-parallel

Figure 6: Speedup compared with vertex-parallel.

Figure 6 lists the speedup compared with vertex-parallel.We only measure the slowest 1% updates because they willmainly affect RisGraph’s tail latency. According to Figure 6a,edge-parallel is better than vertex-parallel in most cases,which validates our discussion in Section 4.4. hybrid-parallelcan better deal with the situation when edge-parallel can’thandle well. It can accelerate computing up to 1.99 times,except for WCC on StackOverflow (a slight drop of 0.8%).

We calculate geometric averages for the speedups of allalgorithms and datasets, in order to represent the overall per-formance of different parallelization schemes. The perfor-mance of edge-parallel outperforms 22% compared to vertex-parallel. Our hybrid strategy can achieve a greater improve-ment, reaching 1.33 times of the vertex-parallel performance.We get similar results for all unsafe updates, with 9.6% and11.6% performance advantage.

8

Graph Storage Then, we evaluate 6 different data struc-tures for the graph storage, which can be divided into twocategories. IA_SUFFIX means the adjacency lists are storedin arrays and corresponding indexes. IO_SUFFIX representsthat only indexes are used to store the edges. We borrow andmodify open-source implementations of 3 different indexes,Hash Table3 (HASH), BTree4 (BT) and ARTree5 (ART).

Index with Array Index OnlyARTree BTree Hash ARTree BTree Hash

Safe 0.92 0.76 1.00 1.00 0.82 1.11Unsafe 0.94 0.98 1.00 0.35 0.73 0.81Overall 0.94 0.88 1.00 0.47 0.76 0.90

Table 5: Overall performance of various data structures

Table 5 shows the relative overall performance measuredfrom various data structures. The baseline is IA_HASH usedby RisGraph. We calculate the geometric average of the rela-tive performance to reflect the overall performance.

For pure edge addition and deletion operations (safe up-dates), IA_HASH and IO_HASH provide higher performance,mainly because the time complexity of Hash Table is betterthan other indexes. IO_HASH reduces the overhead by about11% compared to IA_HASH because IO_HASH does notmaintain additional compact adjacency lists. It is worth to paya little overhead in order to optimize the computing becauseunsafe updates (with computing) take an average 1.89 timeslonger than safe updates, and unsafe updates will seriously af-fect tail latency. The additional adjacency lists give IA_HASHa 19% advantage over IO_HASH for unsafe updates, whichwill be greater if compared with IO_ART and IO_BT. Overall,IA_HASH can provide the best performance.

Tree Storage We also evaluate the impact of the tree stor-age. Table 6 lists the performance of the top-down tree relativeto our parent pointer tree. The aggregated performance of thetop-down tree is only 53% of the parent pointer trees. Whenrunning SSSP on Wiki, the parent pointer tree’s performanceis more than 7 times faster than the top-down tree. Accordingto the results, potential lock contention on hubs can lead todramatic performance loss.

Memory Consumption Table 7 shows the memory con-sumption of RisGraph. We compare our memory footprintwith raw datas (16 Bytes per edge) and geometrically av-eraged the emory consumption of various algorithms anddatasets. In general, RisGraph expands 3.91 times on un-weighted graphs and 4.75 times on weighted graphs. Themain overhead comes from our index, but this is necessary tosupport both fast additions and deletions. In order to supportefficient bi-direct traversal at the same time, we maintain acopy of a directed edge at two vertices, which doubles the

3https://github.com/sparsehash/sparsehash, dense version4https://github.com/abseil/abseil-cpp5https://github.com/philipbecker/cpp-art

SO SD TT UK WKBFS 0.71 0.73 0.63 0.86 0.18SSSP 0.83 0.76 0.66 0.88 0.14SSWP 0.86 0.84 0.82 0.89 0.30WCC 0.89 0.80 0.80 0.80 0.80

Table 6: Performance of the top-down tree on unsafe updates

Index with Array Index OnlyARTree BTree Hash ARTree BTree Hash

Unweighted 4.40 2.62 3.91 4.05 2.25 3.508B_Weight 4.70 3.43 4.75 4.24 2.94 4.22

Table 7: RisGraph’s memory consumption relative to raw data

memory occupation. Comparing IA_HASH with IO_HASH,the adjacency lists do not occupy too much memory becausethis data structure is quite compact, up to the size of raw data.The gap is actually less than 1 because this part only takesmore memory for the high-degree vertices. We only create in-dex for vertices whose degree exceeds the threshold, to avoiduseless indexes taking too much memory.

Replacing the Hash Table with a BTree can effectivelyreduce memory usage by nearly 1.5 times of the raw data.According to the previous Table 5, using BTree will lose 11%in performance. Since our system is an in-memory system,how to scale for larger datasets is an issue. We try to extendRisGraph to support out-of-core. We use mmap to build aprototype supports swapping to disks. We choose BTree asthe index and run UK-2014 [11, 12] (788M vertices, 47.6Bedges, 710GB raw data). On WCC, it can process 262K safeupdates per second. The average time of unsafe updates is147us, and the P999 latency is 2091us. It shows a positivesignal. To efficiently support out-of-core could be our futurework.

5.3 Performance of RisGraphWe evaluate the performance of RisGraph by a group of em-ulated users like TPC-C [6]. We build up some clients tointeract with RisGraph. Each client maintains multiple ses-sions. Sessions represent emulated users. Users will senda single update (addition or deletion an edge) and wait forresults (a timestamp) to be returned. Users will repeat theoperations continuously (without thinking time). To eliminatethe impact of the network, we use embedded clients. All mod-ules in our system will be enabled, including scheduler andhistory storage.

The latency constraint is that at least 99.9% of all updatesshould receive a response in 20 milliseconds. Such a strictlatency constraint requires our system to provide sufficientreal-time ability. The concept of emulated users restricts thescheduling of updates. All requests issued by the same usercannot be reordered. All updates in our evaluation are addi-tions and deletions with only a single edge, which evaluatethe performance with the minimal update granularity.

9

0 1.0E6 2.0E6 3.0E6 4.0E6 5.0E6Throughput (op/s)

0200400600800

1000120014001600

Late

ncy

(us)

SSSDTT

UKWK

(a) BFS

0 1E6 2E6 3E6 4E6Throughput (op/s)

0

500

1000

1500

2000

2500

Late

ncy

(us)

SSSDTT

UKWK

(b) SSSP

0 1E6 2E6 3E6 4E6 5E6Throughput (op/s)

0

400

800

1200

1600

Late

ncy

(us)

SSSDTT

UKWK

(c) SSWP

0 1E6 2E6 3E6 4E6 5E6 6E6Throughput (op/s)

0400800

12001600200024002800

Late

ncy

(us)

SSSDTT

UKWK

(d) WCC

Figure 7: RisGraph’s throughput and latency trends, with different number of sessions.

Figure 7 indicates average latency and throughput whendoubling the number of sessions from 48 (the number of thehardware threads in our server) until the peak throughput isreached, or the constraints cannot be satisfied, which is up to3072 (64×48) sessions. Black crosses show where the latencyconstraints are not fulfilled. RisGraph can effectively utilizethe possibility of scheduling and reach higher throughputunder the constraint of latency. With more sessions, RisGraphgets more opportunities to schedule updates, so RisGraph canget higher throughput.

Table 8 lists detailed metrics when throughput reaches thepeak (T. represents Throughput). RisGraph’s throughput canreach millions or nearly millions of updates per second, andthe 99.9th percentile response time is under 20ms. The resultsshow that our designs can well address throughput, latency,and granularity requirements. It is noticeable that Wiki is thesmallest one of all the datasets, but the throughput on Wikiis less than 1 million per second under each algorithm. Themain reason is that Wiki is the one with the highest possibilityof modifying tree edges (verified in Section 3), resulting inlittle scheduling opportunities.

5.4 RisGraph with Batch Updates

We evaluate the performance of RisGraph with other stream-ing systems with batch updates. The goal is to evaluate theperformance of our system when the scenarios allow coarse-grained updates. During the evaluation, our system worksas follows. After classifying updates, RisGraph applies safeupdates in parallel first, then processes unsafe updates one by

one. RisGraph stores the changes of results for each update.So, our system can still give more detailed information withina batch, including the order of processing updates (the ver-sion ID), and the modification on the results caused by eachupdate. We choose KickStarter as the baseline in this evalua-tion. KickStarter is designed for graph incremental computingand uses the same computing model as RisGraph. As far aswe know, KickStarter is a state-of-the-art streaming systemwith batch updates when the algorithms can be expressed byits model. Because KickStarter is still not open source (asof January 2020), we use the binaries shared by the authorsof KickStarter (only BFS and SSSP). We include the timeof modifying the graph structure and results, and ignore thereading time for KickStarter. We compare the performanceof two systems with different sizes of batch updates, from 2(one edge addition and one edge deletion) to 200M.

Figure 8 shows the performance speedup of RisGraph (lefty-axis) and the throughput (op/s, right y-axis) with variousbatch sizes. Due to the limitation of the length of the paper,we can only show the results of BFS and SSSP on one graph.The other figures are generally similar to Figure 8.

Our system outperforms KickStarter a lot when processingsmall batches, up to 20688 times (SSSP on Subdomain). Asthe batch size increases, the performance of our system andKickStarter will improve, and our advantages will graduallydecrease. We keep the superiority, except when BFS on UK-2007 with 200M batch size. At that point, we are slightlyslower than KickStarter by 38.2%. But the processing time ismore than 32 seconds. It seriously hurts the granularity andthe latency. Furthermore, it is already longer than recomputing

10

BFS SSSP SSWP WCCT. (op/s) Mean (us) P999 (ms) T. (op/s) Mean (us) P999 (ms) T. (op/s) Mean (us) P999 (ms) T. (op/s) Mean (us) P999 (ms)

SO 3.41 M 224.86 15.34 3.34 M 883.76 11.39 2.08 M 738.05 6.70 5.51 M 543.87 7.18SD 3.73 M 408.06 15.13 1.92 M 398.18 18.03 3.79 M 398.07 17.13 4.16 M 367.46 12.35TT 4.94 M 287.39 16.07 3.95 M 361.78 18.04 4.92 M 284.80 16.58 5.81 M 239.49 12.98UK 3.09 M 242.43 16.80 1.32 M 145.42 12.11 2.48 M 298.29 19.37 5.80 M 480.10 18.25WK 0.97 M 376.84 13.54 0.62 M 1,144.47 17.17 0.86 M 842.43 10.68 0.83 M 1,856.06 18.36

Table 8: Metrics when RisGraph reaches peak performance

2E0 2E1 2E2 2E3 2E4 2E5 2E6 2E7 2E8Batch size

0.11

101001K

10K100K

1M

Spee

dup

RGKS

Speedup

0M1M2M3M4M5M6M7M

Thro

ughp

ut(a) BFS on UK-2007

2E0 2E1 2E2 2E3 2E4 2E5 2E6 2E7 2E8Batch size

0.11

101001K

10K100K

1M

Spee

dup

SYSKS

Speedup

0.0M0.5M1.0M1.5M2.0M2.5M3.0M3.5M

Thro

ughp

ut

(b) SSSP on UK-2007

Figure 8: Performance speedup of RisGraph (Abbreviated as RG in the figure) and throughput with various batch sizes.

with efficient graph computing frameworks. Ligra [54] takesabout 1.5s to calculate BFS on UK-2007. So, incrementalcomputing is no longer suitable for such a large batch.

Our graph storage supports both efficient updates and ana-lytics. According to the logs of KickStarter, the modificationson the graph are much slower than RisGraph when the batchis small, so KickStarter cannot provide O(1) time complex-ity like RisGraph. We adopt a hybrid execution strategy ofedge-parallel and vertex-parallel, which provides better per-formance than the common vertex-parallel strategy.

6 Related Work

Graph computing on static graphs A large number ofsystems [8, 16, 23, 24, 31, 35–37, 44, 53, 54, 56, 61, 63, 65]focus on graph computing on static graphs. These systems aredesigned for fast graph analytics on whole graphs, But theysuffer from the ETL overhead on evolving graphs.

Dynamic graph stores Graph databases [2–5, 13, 15, 18,25, 55] mainly target transactional workloads, which rarelyquery and modify a large number of vertices or edges. Somerecent works [19,28,29,62,64] propose to optimize analyticalworkloads in graph databases as well. Several graph engines[26,27,30,38,42,46] are designed to support evolving graphs.These graph engines usually do not support ACID and usebatch-update to achieve better update throughput. They areall limited by recomputing on the whole graph.

Incremental graph computing systems There are manyincremental graph computing systems with batch updatesthat provide shorter computing time than recomputing.GraphInc [14] automatically supports the incremental compu-tation on the top of the Pregel model. Kineograph [17] creates

a series of consistent snapshots to accommodate algorithmsthat require static graphs. Naiad [41, 45] carries out a newcomputational model that supports executing iterative andincremental computations with low latency as well as par-allel computing. GraphIn [50], based on GAS model, usesfixed-size batches of updates to process incremental graphsand offers a heuristic optimization to decide between staticand dynamic graph execution. The paper [57] from Vora etal. uses techniques to amortize fetching and processing costs.KickStarter [58] proves that trimming approximate values canefficiently lead to correct results as well when edge deletionsoccur. GraPU [52] exploits the advantages of batch process-ing to identify the affected data and to get intermediate valueswhich can accelerate computation. Graphbolt [40] minimizesredundant computations on evolving graphs while guaran-teeing BSP semantics. Tornado [39] is a real-time streamingsystem but the correctness of Tornado is an issue. GraphS [47]is an ad-hoc real-time streaming system, which targets cycledetection.

7 Conclusion

In this paper, we present RisGraph, a real-time streamingsystem that supports fine-grained updates and incrementalanalysis for evolving graphs. Our analysis shows in this sce-nario, the affected areas are small in most cases, but couldbe very large in some special cases. And there are schedul-ing opportunities herein. We design RisGraph based on theseinsights, and it achieves the throughput, latency, and granular-ity requirements simultaneously for combined workloads inprocessing evolving graphs.

11

References

[1] Graph 500 | large-scale benchmarks. [Online; accessed6-January-2020; https://graph500.org].

[2] Graph database | multi-model database | orientdb.[Online; accessed 6-January-2020; http://orientdb.com/].

[3] Multi-model highly available nosql database - arangodb.[Online; accessed 6-January-2020; http://orientdb.com/].

[4] Neo4j graph platform - the leader in graph databases.[Online; accessed 6-January-2020; https://neo4j.com/].

[5] Titan: Distributed graph database. [Online; accessed6-January-2020; http://titan.thinkaurelius.com/].

[6] TPC-c - overview of the TPC-c benchmark. [Online; ac-cessed 6-January-2020; http://www.tpc.org/tpcc/detail.asp].

[7] WDC - download the 2012 hyperlink graph. [Online;accessed 6-January-2020; http://webdatacommons.org/hyperlinkgraph/2012-08/download.html].

[8] Zhiyuan Ai, Mingxing Zhang, Yongwei Wu, XuehaiQian, Kang Chen, and Weimin Zheng. Squeezing outall the value of loaded data: An out-of-core graph pro-cessing system with reduced disk i/o. In 2017 USENIXAnnual Technical Conference (USENIX ATC 17), pages125–137, 2017.

[9] Réka Albert, Hawoong Jeong, and Albert-LászlóBarabási. Internet: Diameter of the world-wide web.nature, 401(6749):130, 1999.

[10] Timothy G Armstrong, Vamsi Ponnekanti, DhrubaBorthakur, and Mark Callaghan. Linkbench: a databasebenchmark based on the facebook social graph. InProceedings of the 2013 ACM SIGMOD InternationalConference on Management of Data, pages 1185–1196.ACM, 2013.

[11] Paolo Boldi, Marco Rosa, Massimo Santini, and Sebas-tiano Vigna. Layered label propagation: A multireso-lution coordinate-free ordering for compressing socialnetworks. In Proceedings of the 20th international con-ference on World wide web, pages 587–596. ACM, 2011.

[12] Paolo Boldi and Sebastiano Vigna. The webgraph frame-work i: compression techniques. In Proceedings of the13th international conference on World Wide Web, pages595–602. ACM, 2004.

[13] Nathan Bronson, Zach Amsden, George Cabrera, PrasadChakka, Peter Dimov, Hui Ding, Jack Ferris, AnthonyGiardullo, Sachin Kulkarni, Harry Li, et al. TAO: Face-book’s distributed data store for the social graph. InPresented as part of the 2013 USENIX Annual Techni-cal Conference (USENIX ATC 13), pages 49–60, 2013.

[14] Zhuhua Cai, Dionysios Logothetis, and GeorgosSiganos. Facilitating real-time graph mining. In Pro-ceedings of the fourth international workshop on Clouddata management, pages 1–8. ACM, 2012.

[15] Andrew Carter, Andrew Rodriguez, Yiming Yang, andScott Meyer. Nanosecond indexing of graph data withhash maps and vlists. In Proceedings of the 2019 In-ternational Conference on Management of Data, pages623–635, 2019.

[16] Rong Chen, Jiaxin Shi, Yanzhe Chen, Binyu Zang, Haib-ing Guan, and Haibo Chen. Powerlyra: Differenti-ated graph computation and partitioning on skewedgraphs. ACM Transactions on Parallel Computing(TOPC), 5(3):13, 2019.

[17] Raymond Cheng, Ji Hong, Aapo Kyrola, Youshan Miao,Xuetian Weng, Ming Wu, Fan Yang, Lidong Zhou, FengZhao, and Enhong Chen. Kineograph: taking the pulseof a fast-changing and connected world. In Proceed-ings of the 7th ACM european conference on ComputerSystems, pages 85–98. ACM, 2012.

[18] Ayush Dubey, Greg D Hill, Robert Escriva, andEmin Gün Sirer. Weaver: a high-performance, trans-actional graph database based on refinable timestamps.Proceedings of the VLDB Endowment, 9(11):852–863,2016.

[19] Jing Fan, Adalbert Gerald Soosai Raj, and Jignesh M Pa-tel. The case against specialized graph analytics engines.In CIDR, 2015.

[20] Wenfei Fan, Chunming Hu, and Chao Tian. Incremen-tal graph computations: Doable and undoable. In Pro-ceedings of the 2017 ACM International Conference onManagement of Data, pages 155–169. ACM, 2017.

[21] Wenfei Fan, Jianzhong Li, Jizhou Luo, Zijing Tan, XinWang, and Yinghui Wu. Incremental graph patternmatching. In Proceedings of the 2011 ACM SIGMODInternational Conference on Management of data, pages925–936. ACM, 2011.

[22] Wenfei Fan, Xin Wang, and Yinghui Wu. Incrementalgraph pattern matching. ACM Transactions on DatabaseSystems (TODS), 38(3):18, 2013.

12

[23] Joseph E Gonzalez, Yucheng Low, Haijie Gu, DannyBickson, and Carlos Guestrin. Powergraph: Distributedgraph-parallel computation on natural graphs. In Pre-sented as part of the 10th USENIX Symposium on Oper-ating Systems Design and Implementation (OSDI 12),pages 17–30, 2012.

[24] Joseph E Gonzalez, Reynold S Xin, Ankur Dave, DanielCrankshaw, Michael J Franklin, and Ion Stoica. Graphx:Graph processing in a distributed dataflow framework.In 11th USENIX Symposium on Operating Systems De-sign and Implementation (OSDI 14), pages 599–613,2014.

[25] Sairam Gurajada, Stephan Seufert, Iris Miliaraki, andMartin Theobald. Triad: a distributed shared-nothing rdfengine based on asynchronous message passing. In Pro-ceedings of the 2014 ACM SIGMOD international con-ference on Management of data, pages 289–300. ACM,2014.

[26] Wentao Han, Youshan Miao, Kaiwei Li, Ming Wu, FanYang, Lidong Zhou, Vijayan Prabhakaran, WenguangChen, and Enhong Chen. Chronos: a graph engine fortemporal graph analysis. In Proceedings of the NinthEuropean Conference on Computer Systems, page 1.ACM, 2014.

[27] Anand Padmanabha Iyer, Li Erran Li, Tathagata Das,and Ion Stoica. Time-evolving graph processing at scale.In Proceedings of the Fourth International Workshopon Graph Data Management Experiences and Systems,page 5. ACM, 2016.

[28] Alekh Jindal, Samuel Madden, Malú Castellanos, andMeichun Hsu. Graph analytics using vertica relationaldatabase. In 2015 IEEE International Conference onBig Data (Big Data), pages 1191–1200. IEEE, 2015.

[29] Hideaki Kimura, Alkis Simitsis, and Kevin Wilkinson.Janus: Transaction processing of navigation and analyticgraph queries on many-core servers. 2017.

[30] Pradeep Kumar and H Howie Huang. Graphone: A datastore for real-time analytics on evolving graphs. In 17thUSENIX Conference on File and Storage Technologies(FAST 19), pages 249–263, 2019.

[31] Aapo Kyrola, Guy Blelloch, and Carlos Guestrin.Graphchi: Large-scale graph computation on just a PC.In Presented as part of the 10th USENIX Symposium onOperating Systems Design and Implementation (OSDI12), pages 31–46, 2012.

[32] Viktor Leis, Alfons Kemper, and Thomas Neumann. Theadaptive radix tree: Artful indexing for main-memorydatabases. In ICDE, volume 13, pages 38–49, 2013.

[33] Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg,Christos Faloutsos, and Zoubin Ghahramani. Kroneckergraphs: An approach to modeling networks. Journal ofMachine Learning Research, 11(Feb):985–1042, 2010.

[34] Jure Leskovec and Andrej Krevl. SNAP Datasets: Stan-ford large network dataset collection. http://snap.stanford.edu/data, June 2014.

[35] Heng Lin, Xiaowei Zhu, Bowen Yu, Xiongchao Tang,Wei Xue, Wenguang Chen, Lufei Zhang, Torsten Hoe-fler, Xiaosong Ma, Xin Liu, et al. Shentu: processingmulti-trillion edge graphs on millions of cores in sec-onds. In Proceedings of the International Conferencefor High Performance Computing, Networking, Storage,and Analysis, page 56. IEEE Press, 2018.

[36] Yucheng Low, Joseph Gonzalez, Aapo Kyrola, DannyBickson, Carlos Guestrin, and Joseph Hellerstein.GraphLab: A New Framework for Parallel MachineLearning. In Proceedings of the Twenty-Sixth Confer-ence on Uncertainty in Artificial Intelligence, UAI’10,pages 340–349, Arlington, Virginia, USA, 2010. AUAIPress. event-place: Catalina Island, CA.

[37] Steffen Maass, Changwoo Min, Sanidhya Kashyap,Woonhak Kang, Mohan Kumar, and Taesoo Kim. Mo-saic: Processing a trillion-edge graph on a single ma-chine. In Proceedings of the Twelfth European Con-ference on Computer Systems, pages 527–543. ACM,2017.

[38] Peter Macko, Virendra J Marathe, Daniel W Margo, andMargo I Seltzer. Llama: Efficient graph analytics usinglarge multiversioned arrays. In 2015 IEEE 31st Interna-tional Conference on Data Engineering, pages 363–374.IEEE, 2015.

[39] Ahmed R Mahmood, Ahmed M Aly, Thamir Qadah,El Kindi Rezig, Anas Daghistani, Amgad Madkour,Ahmed S Abdelhamid, Mohamed S Hassan, Walid GAref, and Saleh Basalamah. Tornado: A distributedspatio-textual stream processing system. Proceedingsof the VLDB Endowment, 8(12):2020–2023, 2015.

[40] Mugilan Mariappan and Keval Vora. Graphbolt:Dependency-driven synchronous processing of stream-ing graphs. In Proceedings of the Fourteenth EuroSysConference 2019, page 25. ACM, 2019.

[41] Frank McSherry, Derek Murray, Rebecca Isaacs, andMichael Isard. Differential dataflow. In Proceedings ofCIDR 2013, January 2013.

[42] Youshan Miao, Wentao Han, Kaiwei Li, Ming Wu, FanYang, Lidong Zhou, Vijayan Prabhakaran, Enhong Chen,and Wenguang Chen. Immortalgraph: A system for

13

storage and analysis of temporal graphs. ACM Trans.Storage, 11(3), July 2015.

[43] Alan Mislove, Massimiliano Marcon, Krishna P Gum-madi, Peter Druschel, and Bobby Bhattacharjee. Mea-surement and analysis of online social networks. InProceedings of the 7th ACM SIGCOMM conference onInternet measurement, pages 29–42. ACM, 2007.

[44] Anurag Mukkara, Nathan Beckmann, Maleen Abey-deera, Xiaosong Ma, and Daniel Sanchez. Exploitinglocality in graph analytics through hardware-acceleratedtraversal scheduling. In 2018 51st Annual IEEE/ACM In-ternational Symposium on Microarchitecture (MICRO),pages 1–14. IEEE, 2018.

[45] Derek G Murray, Frank McSherry, Rebecca Isaacs,Michael Isard, Paul Barham, and Martín Abadi. Na-iad: a timely dataflow system. In Proceedings of theTwenty-Fourth ACM Symposium on Operating SystemsPrinciples, pages 439–455. ACM, 2013.

[46] Vijayan Prabhakaran, Ming Wu, Xuetian Weng, FrankMcSherry, Lidong Zhou, and Maya Haradasan. Manag-ing large graphs on multi-cores with graph awareness. InPresented as part of the 2012 USENIX Annual TechnicalConference (USENIX ATC 12), pages 41–52, 2012.

[47] Xiafei Qiu, Wubin Cen, Zhengping Qian, You Peng,Ying Zhang, Xuemin Lin, and Jingren Zhou. Real-timeconstrained cycle detection in large dynamic graphs.Proceedings of the VLDB Endowment, 11(12):1876–1888, 2018.

[48] Ryan Rossi and Nesreen Ahmed. The network datarepository with interactive graph analytics and visual-ization. In Twenty-Ninth AAAI Conference on ArtificialIntelligence, 2015.

[49] Peter Sanders and Dominik Schultes. Highway hierar-chies hasten exact shortest path queries. In EuropeanSymposium on Algorithms, pages 568–579. Springer,2005.

[50] Dipanjan Sengupta, Narayanan Sundaram, Xia Zhu,Theodore L Willke, Jeffrey Young, Matthew Wolf, andKarsten Schwan. Graphin: An online high performanceincremental graph processing framework. In Euro-pean Conference on Parallel Processing, pages 319–333.Springer, 2016.

[51] Comandur Seshadhri, Ali Pinar, and Tamara G Kolda.An in-depth analysis of stochastic kronecker graphs.Journal of the ACM (JACM), 60(2):13, 2013.

[52] Feng Sheng, Qiang Cao, Haoran Cai, Jie Yao, andChangsheng Xie. Grapu: Accelerate streaming graph

analysis through preprocessing buffered updates. In Pro-ceedings of the ACM Symposium on Cloud Computing,pages 301–312, 2018.

[53] Xuanhua Shi, Zhigao Zheng, Yongluan Zhou, Hai Jin,Ligang He, Bo Liu, and Qiang-Sheng Hua. Graph pro-cessing on gpus: A survey. ACM Computing Surveys(CSUR), 50(6):81, 2018.

[54] Julian Shun and Guy E Blelloch. Ligra: a lightweightgraph processing framework for shared memory. InACM Sigplan Notices, volume 48, pages 135–146. ACM,2013.

[55] Wen Sun, Achille Fokoue, Kavitha Srinivas, AnastasiosKementsietsidis, Gang Hu, and Guotong Xie. Sqlgraph:An efficient relational-based property graph store. InProceedings of the 2015 ACM SIGMOD InternationalConference on Management of Data, pages 1887–1901.ACM, 2015.

[56] Keval Vora. LUMOS: Dependency-driven disk-basedgraph processing. In 2019 USENIX Annual TechnicalConference (USENIX ATC 19), pages 429–442, 2019.

[57] Keval Vora, Rajiv Gupta, and Guoqing Xu. Synergis-tic analysis of evolving graphs. ACM Transactions onArchitecture and Code Optimization (TACO), 13(4):32,2016.

[58] Keval Vora, Rajiv Gupta, and Guoqing Xu. Kickstarter:Fast and accurate computations on streaming graphsvia trimmed approximations. ACM SIGOPS OperatingSystems Review, 51(2):237–251, 2017.

[59] Wikipedia contributors. Parent pointer tree —Wikipedia, the free encyclopedia, 2019. [Online; ac-cessed 16-January-2020].

[60] Yingjun Wu, Joy Arulraj, Jiexi Lin, Ran Xian, and An-drew Pavlo. An empirical evaluation of in-memorymulti-version concurrency control. Proceedings of theVLDB Endowment, 10(7):781–792, 2017.

[61] Yunming Zhang, Mengjiao Yang, Riyadh Baghdadi,Shoaib Kamil, Julian Shun, and Saman Amarasinghe.Graphit: A high-performance graph dsl. Proceedings ofthe ACM on Programming Languages, 2(OOPSLA):121,2018.

[62] Kangfei Zhao and Jeffrey Xu Yu. All-in-one: Graphprocessing in rdbmss revisited. In Proceedings of the2017 ACM International Conference on Management ofData, pages 1165–1180. ACM, 2017.

[63] Xiaowei Zhu, Wenguang Chen, Weimin Zheng, and Xi-aosong Ma. Gemini: A computation-centric distributed

14

graph processing system. In 12th USENIX Sympo-sium on Operating Systems Design and Implementation(OSDI 16), pages 301–316, 2016.

[64] Xiaowei Zhu, Guanyu Feng, Marco Serafini, XiaosongMa, Jiping Yu, Lei Xie, Ashraf Aboulnaga, and Wen-guang Chen. Livegraph: A transactional graph storagesystem with purely sequential adjacency list scans. arXivpreprint arXiv:1910.05773, 2019.

[65] Xiaowei Zhu, Wentao Han, and Wenguang Chen. Grid-graph: Large-scale graph processing on a single machineusing 2-level hierarchical partitioning. In 2015 USENIXAnnual Technical Conference (USENIX ATC 15), pages375–386, 2015.

15