Structure and models of real-world graphs and networks

Jure LeskovecMachine Learning Department

Carnegie Mellon Universityjure@cs.cmu.edu

http://www.cs.cmu.edu/~jure/

Jure Leskovec

Networks (graphs)

Jure Leskovec

Examples of networks

• Internet (a)• citation network (b)• World Wide Web (c)

(b) (c)(a)

(d)

(e)

• sexual network (d)• food web (e)

Jure Leskovec

Networks of the real-world (1)• Information networks:

– World Wide Web: hyperlinks– Citation networks– Blog networks

• Social networks: people + interactios– Organizational networks– Communication networks – Collaboration networks– Sexual networks – Collaboration networks

• Technological networks:– Power grid– Airline, road, river networks– Telephone networks– Internet– Autonomous systems

Florence families Karate club network

Collaboration networkFriendship network

Jure Leskovec

Networks of the real-world (2)

• Biological networks– metabolic networks– food web– neural networks– gene regulatory

networks

• Language networks– Semantic networks

• Software networks• …

Yeast proteininteractions

Semantic network

Language network XFree86 network

Jure Leskovec

Types of networks

• Directed/undirected• Multi graphs (multiple edges between nodes)• Hyper graphs (edges connecting multiple nodes)• Bipartite graphs (e.g., papers to authors)• Weighted networks• Different type nodes and edges• Evolving networks:

– Nodes and edges only added– Nodes, edges added and removed

Jure Leskovec

Traditional approach• Sociologists were first to study networks:

– Study of patterns of connections between people to understand functioning of the society

– People are nodes, interactions are edges – Questionares are used to collect link data

(hard to obtain, inaccurate, subjective)– Typical questions: Centrality and connectivity

• Limited to small graphs (~10 nodes) and properties of individual nodes and edges

Jure Leskovec

New approach (1)

• Large networks (e.g., web, internet, on-line social networks) with millions of nodes

• Many traditional questions not useful anymore: – Traditional: What happens if a node U is removed? – Now: What percentage of nodes needs to be removed

to affect network connectivity?

• Focus moves from a single node to study of statistical properties of the network as a whole

• Can not draw (plot) the network and examine it

Jure Leskovec

New approach (2)

• How the network “looks like” even if I can’t look at it?

• Need statistical methods and tools to quantify large networks

• 3 parts/goals:– Statistical properties of large networks– Models that help understand these properties– Predict behavior of networked systems based on

measured structural properties and local rules governing individual nodes

Jure Leskovec

Statistical properties of networks

• Features that are common to networks of different types:– Properties of static networks:

• Small-world effect• Transitivity or clustering• Degree distributions (scale free networks)• Network resilience• Community structure• Subgraphs or motifs

– Temporal properties:• Densification• Shrinking diameter

Jure Leskovec

Small-world effect (1)

• Six degrees of separation (Milgram 60s)– Random people in Nebraska were asked to send letters to

stockbrokes in Boston– Letters can only be passed to first-name acquantices– Only 25% letters reached the goal– But they reached it in about 6 steps

• Measuring path lengths: – Diameter (longest shortest path): max dij

– Effective diameter: distance at which 90% of all connected pairs of nodes can be reached

– Mean geodesic (shortest) distance l

or

Jure Leskovec

Small-world effect (2)

• Distribution of shortest path lengths

• Microsoft Messenger network – 180 million people– 1.3 billion edges– Edge if two people

exchanged at least one message in one month period

0 5 10 15 20 25 3010

0

101

102

103

104

105

106

107

108

Distance (Hops)

Num

ber

of n

odes

Pick a random node, count how many

nodes are at distance

1,2,3... hops

7

Jure Leskovec

Small-world effect (3)• Fact:

– If number of vertices within distance r grows exponentially with r, then mean shortest path length l increases as log n

• Implications:– Information (viruses) spread quickly– Erdos numbers are small– Peer to peer networks (for navigation purposes)

• Shortest paths exists • Humans are able to find the paths:

– People only know their friends– People do not have the global knowledge of the network

• This suggests something special about the structure of the network– On a random graph short paths exists but no one would be able to find

them

Jure Leskovec

Transitivity or Clustering• “friend of a friend is a friend”• If a connects to b, and b to c,

then with high probability a connects to c.

• Clustering coefficient C:

C = 3*number of triangles / number of connected triples

• Alternative definition:

Ci = triangles connected to vertex i / number triples centered on vertex i– Clustering coefficient:

Ci=1, 1, 1/6, 0, 0

Jure Leskovec

• Clustering coefficient scales as

• It is considerably higher than in a random graph

• It is speculated that in real networks:

C=O(1) as n→∞ In Erdos-Renyi random graph: C=O(n-1)

Transitivity or Clustering (2)

Synonyms network

World Wide Web

Jure Leskovec

Degree distributions (1)

• Let pk denote a fraction of nodes with degree k• We can plot a histogram of pk vs. k• In a Erdos-Renyi random graph degree

distribution follows Poisson distribution• Degrees in real networks are heavily skewed to

the right• Distribution has a long tail of values that are far

above the mean• Heavy (long) tail:

– Amazon sales– word length distribution, …

Jure Leskovec

Detour: how long is the long tail?

This is not directly related to graphs, but it nicely explains

the “long tail” effect. It shows that there is big

market for niche products.

Jure Leskovec

100

101

102

103

10-6

10-5

10-4

10-3

10-2

0 200 400 600 800 100010

-6

10-5

10-4

10-3

10-2

0 200 400 600 800 10000

0.5

1

1.5

2

2.5

3

3.5x 10

-3

Degree distributions (2)

• Many real world networks contain hubs: highly connected nodes

• We can easily distinguish between exponential and power-law tail by plotting on log-lin and log-log axis

• In scale-free networks maximum degree scales as n1/(α-1)

Degree distribution ina blog network

lin-lin log-lin

log-log

pk

pk

k

kk

Jure Leskovec

Poisson vs. Scale-free network

Poisson network Scale-free (power-law) network

Function is scale free if:f(ax) = b f(x)

(Erdos-Renyi random graph)

Degree distribution is Poisson

Degree distribution is Power-law

Jure Leskovec

Degree distribution

number of people a person talks to on a

Microsoft Messenger

Node degree

Cou

nt

X

Highest degree

Jure Leskovec

Network resilience (1)

• We observe how the connectivity (length of the paths) of the network changes as the vertices get removed

• Vertices can be removed:– Uniformly at random– In order of decreasing degree

• It is important for epidemiology– Removal of vertices

corresponds to vaccination

Jure Leskovec

Network resilience (2)• Real-world networks are resilient to random attacks

– One has to remove all web-pages of degree > 5 to disconnect the web– But this is a very small percentage of web pages

• Random network has better resilience to targeted attacks

Fraction of removed nodes

Mea

n pa

th le

ngth

Random network

Fraction of removed nodes

Internet (Autonomous systems)

Randomremoval

Preferentialremoval

Jure Leskovec

Community structure

• Most social networks show community structure– groups have higher density of edges

within than accross groups– People naturally divide into groups

based on interests, age, occupation, …

• How to find communities:– Spectral clustering (embedding into a

low-dim space)– Hierarchical clustering based on

connection strength– Combinatorial algorithms– Block models– Diffusion methods

Friendship network of children in a school

Jure Leskovec

MSN MessengerDistribution of

Connected components in MSN Messenger

network

X

Largest component

Size (number of nodes)

Cou

nt

Growth of largest component over time in a citation

network

• Graphs have a “giant component ”• Distribution of connected

components follows a power law

Jure Leskovec

Network motifs (1)

• What are the building blocks (motifs) of networks?

• Do motifs have specific roles in networks?• Network motifs detection process:

– Count how many times each subgraph appears– Compute statistical significance for each subgraph –

probability of appearing in random as much as in real network

3 node motifs

Jure Leskovec

Network motifs (2)

• Biological networks– Feed-forward loop– Bi-fan motif

• Web graph:– Feedback with two

mutual diads– Mutual diad– Fully connected triad

Jure Leskovec

Network motifs (3)

Transcription networks

Signal transduction networks

WWW and friendship networks

Word adjacency networks

Jure Leskovec

Networks over time: Densification• A very basic question: What is the

relation between the number of nodes and the number of edges in a network?

• Networks are becoming denser over time

• The number of edges grows faster than the number of nodes – average degree is increasing

a … densification exponent: 1 ≤ a ≤ 2:– a=1: linear growth – constant out-

degree (assumed in the literature so far)

– a=2: quadratic growth – clique

Internet

Citations

N(t)

N(t)

E(t)

E(t)a=1.2

a=1.7

Jure Leskovec

Densification & degree distribution

• How does densification affect degree distribution?

• Given densification exponent a, the degree exponent is:– (a) For γ=const over time, we obtain

densification only for 1<γ<2, then γ=a/2

– (b) For γ<2 degree distribution has to evolve according to:

• Power-law: y=b xγ, for γ<2 E[y] = ∞

Degree exponent over time

γ(t)

pk=kγ

γ(t)

a=1.1

a=1.6

(a)

(b)

Jure Leskovec

Shrinking diameters

• Intuition says that distances between the nodes slowly grow as the network grows (like log n)

• But as the network grows the distances between nodes slowly decrease

Internet

Citations

Models of network generation and evolution

Jure Leskovec

Recap (1)• Last time we saw:

– Large networks (web, on-line social networks) are here

– Many traditional questions not useful anymore– We can not plot the network so we need statistical

methods and tools to quantify large networks– 3 parts/goals:

• Statistical properties of large networks• Models that help understand these properties• Predict behavior of networked systems based on measured

structural properties and local rules governing individual nodes

Jure Leskovec

Recap (2)

• We also so features that are common to networks of various types:

• Properties of static networks:– Small-world effect– Transitivity or clustering– Degree distributions (scale free networks)– Network resilience– Community structure– Subgraphs or motifs

• Temporal properties:– Densification– Shrinking diameter

Jure Leskovec

Outline for today

• We will see the network generative models for modeling networks’ features:– Erdos-Renyi random graph– Exponential random graphs (p*) model– Small world model– Preferential attachment– Community guided attachment– Forest fire model

• Fitting models to real data– How to generate a synthetic realistic looking network?

Jure Leskovec

(Erodos-Renyi) Random graphs

• Also known as Poisson random graphs or Bernoulli graphs– Given n vertices connect each pair i.i.d. with

probability p

• Two variants:– Gn,p: graph with m edges appears with probability pm(1-

p)M-m, where M=0.5n(n-1) is the max number of edges– Gn,m: graphs with n nodes, m edges

• Very rich mathematical theory: many properties are exactly solvable

Jure Leskovec

Properties of random graphs

• Degree distribution is Poisson since the presence and absence of edges is independent

• Giant component: average degree k=2m/n:– k=1-ε: all components are of size log n

– k=1+ε: there is 1 component of size n • All others are of size log n

• They are a tree plus an edge, i.e., cycles

• Diameter: log n / log k

!)1(

k

ezpp

k

np

zkknk

k

Jure Leskovec

Evolution of a random graphfo

r no

n-G

CC

ver

tices

k

Jure Leskovec

Subgraphs in random graphsExpected number of

subgraphs H(v,e) in Gn,p is

a

pnp

a

v

v

nXE

eve

!)(

a... # of isomorphic graphs

Jure Leskovec

Random graphs: conclusion

• Pros:– Simple and tractable model– Phase transitions– Giant component

• Cons:– Degree distribution– No community structure– No degree correlations

• Extensions:– Configuration model

• Random graphs with arbitrary degree sequence• Excess degree: Degree of a vertex of the end of

random edge: qk = k pk

Configuration model

Jure Leskovec

Exponential random graphs(p* models)

• Comes from social sciences

• Let εi set of measurable properties of a graph (number of edges, number of nodes of a given degree, number of triangles, …)

• Exponential random graph model defines a probability distribution over graphs:

Examples of εi

Jure Leskovec

Exponential random graphs• Includes Erdos-Renyi as a special

case• Assume parameters βi are

specified– No analytical solutions for the model– But can use simulation to sample the

graphs:• Define local moves on a graph:

– Addition/removal of edges– Movement of edges– Edge swaps

• Parameter estimation: – maximum likelihood

• Problem:– Can’t solve for transitivity (produces

cliques)– Used to analyze small networks

Example of parameter estimates:

Jure Leskovec

Small-world model• Used for modeling network transitivity• Many networks assume some kind of geographical

proximity• Small-world model:

– Start with a low-dimensional regular lattice– Rewire:

• Add/remove edges to create shortcuts to join remote parts of the lattice

• For each edge with prob p move the other end to a random vertex

• Rewiring allows to interpolate between regular lattice and random graph

Jure Leskovec

Small-world model

• Regular lattice (p=0):– Clustering coefficient

C=(3k-3)/(4k-2)=3/4– Mean distance L/4k

• Almost random graph (p=1):– Clustering coefficient C=2k/L– Mean distance log L / log k

• No power-law degree distribution

Rewiring probability p

Degree distribution

Jure Leskovec

Models of evolution

• Models of network evolution:– Preferential attachment– Edge copying model– Community Guided Attachment– Forest Fire model

• Models for realistic network generation:– Kronecker graphs

Jure Leskovec

Preferential attachment

• Models the growth of the network• Preferential attachment (Price 1965, Albert & Barabasi

1999):– Add a new node, create m out-links– Probability of linking a node ki is

proportional to its degree

• Based on Herbert Simon’s result– Power-laws arise from “Rich get richer” (cumulative advantage)

• Examples (Price 1965 for modeling citations):– Citations: new citations of a paper are proportional to the

number it already has

Jure Leskovec

Preferential attachment

• Leads to power-law degree distributions

• But: – all nodes have equal (constant) out-

degree– one needs a complete knowledge of the

network

• There are many generalizations and variants, but the preferential selection is the key ingredient that leads to power-laws

3 kpk

Jure Leskovec

Edge copying model

• Copying model:– Add a node and choose k the number of edges to add– With prob β select k random vertices and link to them– With prob 1-β edges are copied from a randomly

chosen node

• Generates power-law degree distributions with exponent 1/(1-β)

• Generates communities• Related Random-surfer model

Jure Leskovec

Community guided attachment

• Want to model/explain densification in networks

• Assume community structure

• One expects many within-group friendships and fewer cross-group ones

Self-similar university community structure

CS Math Drama Music

Science Arts

University

Jure Leskovec

Community guided attachment• Assuming cross-community linking probability

• The Community Guided Attachment leads to Densification Power Law with exponent

– a … densification exponent– b … community tree branching factor– c … difficulty constant, 1 ≤ c ≤ b

• If c = 1: easy to cross communities– Then: a=2, quadratic growth of edges – near clique

• If c = b: hard to cross communities– Then: a=1, linear growth of edges – constant out-degree

Jure Leskovec

Forest Fire Model

• Want to model graphs that density and have shrinking diameters

• Intuition:– How do we meet friends at a party?– How do we identify references when

writing papers?

Jure Leskovec

Forest Fire Model for directed graphs

• The model has 2 parameters: – p … forward burning probability– r … backward burning probability

• The model:– Each turn a new node v arrives– Uniformly at random chooses an “ambassador” w– Flip two geometric coins to determine the number in-

and out-links of w to follow (burn)– Fire spreads recursively until it dies– Node v links to all burned nodes

Jure Leskovec

Forest Fire Model• Simulation experiments• Forest Fire generates graphs that densify

and have shrinking diameter

densification diameter

1.32

N(t)

E(t)

N(t)

diam

eter

Jure Leskovec

Forest Fire Model

• Forest Fire also generates graphs with heavy-tailed degree distribution

in-degree out-degree

count vs. in-degree count vs. out-degree

Jure Leskovec

Forest Fire: Parameter Space• Fix backward

probability r and vary forward burning probability p

• We observe a sharp transition between sparse and clique-like graphs

• Sweet spot is very narrow

Sparse graph

Clique-like

graphIncreasing

diameter

Decreasing diameter

Constant

diameter

Jure Leskovec

Kronecker graphs

• Want to have a model that can generate a realistic graph:– Static Patterns

• Power Law Degree Distribution• Small Diameter• Power Law Eigenvalue and Eigenvector Distribution

– Temporal Patterns• Densification Power Law• Shrinking/Constant Diameter

• For Kronecker graphs all these properties can actually be proven

Jure LeskovecAdjacency matrix

Kronecker Product – a Graph

Intermediate stage

Adjacency matrix

Jure Leskovec

Kronecker Product – Definition

• The Kronecker product of matrices A and B is given by

• We define a Kronecker product of two graphs as a Kronecker product of their adjacency matrices

N x M K x L

N*K x M*L

Jure Leskovec

Stochastic Kronecker Graphs

• Create N1N1 probability matrix P1

• Compute the kth Kronecker power Pk

• For each entry puv of Pk include an edge (u,v) with probability puv

0.5 0.2

0.1 0.3

P1

Instance

Matrix G2

0.25 0.10 0.10 0.04

0.05 0.15 0.02 0.06

0.05 0.02 0.15 0.06

0.01 0.03 0.03 0.09

P2

flip biased

coins

Kronecker

multiplication

Jure Leskovec

Fitting Kronecker to Real Data

• Given a graph G and Kronecker matrix P1 we can calculate probability that P1 generated G: P(G|P1): 0.25 0.10 0.10 0.04

0.05 0.15 0.02 0.06

0.05 0.02 0.15 0.06

0.01 0.03 0.03 0.09

0.5 0.2

0.1 0.3

P1Pk

1 1 0 0

1 1 1 0

0 1 1 1

0 0 1 1G

σ… node labeling

P(G|P1)

Jure Leskovec

Fitting Kronecker: 2 challenges

• Invariance to node labeling σ (there are N! labelings)

• Calculating P(G|P1) takes O(N2) (since one needs to consider every cell of adjacency matrix)

0.25 0.10 0.10 0.04

0.05 0.15 0.02 0.06

0.05 0.02 0.15 0.06

0.01 0.03 0.03 0.09

0.5 0.2

0.1 0.3

P1 Pk

1 1 0 0

1 1 1 0

0 1 1 1

0 0 1 1G

P(G|P1)

==

1

2

3

4

2

1

4

3

)(),|()|( 11

PPGPPGP

Jure Leskovec

Fitting Kronecker: Solutions

• Node Labeling: can use MCMC sampling to average over (all) node labelings

• P(G|P1) takes O(N2): Real graphs are sparse, so calculate P(Gempty) and then “add” the edges. This takes O(E).

0.25 0.10 0.10 0.04

0.05 0.15 0.02 0.06

0.05 0.02 0.15 0.06

0.01 0.03 0.03 0.09

P=

1 1 0 0

1 1 1 0

0 1 1 1

0 0 1 1

G=

σ… node labeling

Jure Leskovec

Experiments on real AS graphDegree distribution Hop plot

Network valueAdjacency matrix eigen values

Jure Leskovec

Why fitting generative models?

• Parameters tell us about the structure of a graph• Extrapolation: given a graph today, how will it

look in a year?• Sampling: can I get a smaller graph with similar

properties?• Anonymization: instead of releasing real graph

(e.g., email network), we can release a synthetic version of it

Processes taking place on networks

Jure Leskovec

Epidemiological processes

• The simplest way to spread a virus over the network– S: Susceptible– I: Infected– R: Recovered (removed)

• SIS model: 2 parameters– β … virus birth rate– δ … virus death (recovery) rate

• SIR model: as one gets cured, he or she can not get infected again

SIS model

ζit depends on β and topology

Jure Leskovec

Epidemic threshold for SIS model• How infectious the virus

needs to be to survive in the network?

• First results on power-law networks suggested that any virus will prevail

• New result that works for any topology:

• For s>1 virus prevails• For s<1 virus dies

λ1… largest eigen value of graph adjacency matrix

Jure Leskovec

Navigation in small-world networks

• Milgram’s experiment showed:– (a) short paths exist in networks– (b) humans are able to find them

• Assume the following setting:– Nodes of a graph are scattered on a plane– Given starting node u and we want to reach target

node v– A small world navigation algorithm navigates the

network by always navigating to a neighbor that is closest (in Manhattan distance) to target node v

u

v

Jure Leskovec

Navigation in small-world networks

• Start with random lattice:– Each node connects with their 4

immediate neighbors– Long range links are added with

probability proportional to the distance between the points (p(u,v) ~ dα)

• Can be show that only for α=2 delivery time is poly-log in number of nodes n

Deliver time T < nβ

Network creation

Jure Leskovec

Navigation in a real-world network

• Take a social network of 500k bloggers where for each blogger we know their geographical location

• Pick two nodes at random and geographically greedy navigate the network

• Results:– 13% success rate (vs. 18%

for Milgram)

Distribution of path lengths

Friendships vs. distance

Jure Leskovec

Navigation in real-world network

• Geographical distance may not be the right kind of distance

• Since population is non-uniform let’s use rank based friendship distance:

i.e., we measure the distance d(u,v) by the number of people living closer to v than u does

• Then

And the proof still works

Jure Leskovec

• Some references used to prepare this talk:– The Structure and Function of Complex Networks, by Mark Newman– Statistical mechanics of complex networks, by Reka Albert and Albert-Laszlo

Barabasi– Graph Mining: Laws, Generators and Algorithms, by Deepay Chakrabarti and

Christos Faloutsos – An Introduction to Exponential Random Graph (p*) Models for Social Networks

by Garry Robins, Pip Pattison, Yuval Kalish and Dean Lusher– Graph Evolution: Densification and Shrinking Diameters, by Jure Leskovec, Jon

Kleinberg and Christos Faloutsos– Realistic, Mathematically Tractable Graph Generation and Evolution, Using

Kronecker Multiplication, by Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg and Christos Faloutsos

– Navigation in a Small World, by Jon Kleinberg– Geographic routing in social networks, by David Liben-Nowell, Jasmine Novak,

Ravi Kumar, Prabhakar Raghavan, and Andrew Tomkins

– Some plots and slides borrowed from Lada Adamic, Mark Newman, Mark Joseph, Albert Barabasi, Jon Kleinberg, David Lieben-Nowell, Sergi Valverde and Ricard Sole

Jure Leskovec

Rough random materialthat did not make it into the

presentation

Jure Leskovec

Bow-tie structure of the web

SCC56 M

OUT44 M

IN44 M

Broder & al. WWW 2000, Dill & al. VLDB 2001

DISC17 M

TENDRILS44M

Jure Leskovec

Study of 3 websites• study over three universities’ publicly indexable Web

sites

Jure Leskovec

Australia

In- and out-degree distributions

Jure Leskovec

In- and out-degree distributions

New Zealand

Jure Leskovec

United Kingdom

In- and out-degree distributions

Jure Leskovec

We assume this node would like to connect to a centrally located node; a node whose distances to other nodes is minimized.

dij is the Euclidean distance

hj is some measure of the “centrality” of node j

α is a parameter – a function of the final number n of points, gauging the relative importance of the two objectives

Jure Leskovec

Fabrikant et al. define 3 possible measures of “centrality”

1. The average number of hops from other nodes

2. The maximum number of hops from another node

3. The number of hops from a fixed center of the tree

Jure Leskovec

α is the crux of the theorem!

Why? Here are some examples:

Fabrikant&al

Jure Leskovec

If α is too low, then the Euclidian distances become unimportant, and the network resembles a star:

Fabrikant&al

Jure Leskovec

But if α grows at least as fast as √n, where n is the final number of points, then distance becomes too important, and minimum spanning trees with high degree occur, but with exponentially vanishing probability – thus not a power law.

if α is anywhere in between, we have a power law

Through a rather complex and elaborate proof, Fabrikant&al prove this initial assumption will produce a power law distribution – I’ll save you the math!