minimizing interference and selection bias in network ... · current approaches to a/b testing in...

Minimizing Interference and Selection Biasin Network Experiment Design

Zahra Fatemi, Elena ZhelevaDepartment of Computer Science, University of Illinois at Chicago

Chicago, IL, USA{zfatem2, ezheleva}@uic.edu

Abstract

Current approaches to A/B testing in networks focus on limit-ing interference, the concern that treatment effects can ”spillover” from treatment nodes to control nodes and lead to bi-ased causal effect estimation. Prominent methods for networkexperiment design rely on two-stage randomization, in whichsparsely-connected clusters are identified and cluster random-ization dictates the node assignment to treatment and con-trol. Here, we show that cluster randomization does not en-sure sufficient node randomization and it can lead to selectionbias in which treatment and control nodes represent differentpopulations of users. To address this problem, we propose aprincipled framework for network experiment design whichjointly minimizes interference and selection bias. We intro-duce the concepts of edge spillover probability and clustermatching and demonstrate their importance for designing net-work A/B testing. Our experiments on a number of real-worlddatasets show that our proposed framework leads to signif-icantly lower error in causal effect estimation than existingsolutions.

IntroductionThe gold standard for inferring causality is the use of con-trolled experiments, also known as A/B tests and random-ized controlled trials, in which experimenters can assigntreatment (e.g. prompt users to vote) to a random sub-set of a population of interest and compare their outcomewith the outcome of a control group, randomly selectedfrom the same population (e.g., a group of users who werenot prompted to vote). Through randomization, the exper-imenter can control for confounding variables that are notpresent in the data but can impact the treatment and out-come assignment (e.g. distance from voting polls location)and to assess whether the treatment can cause the targetvariable (e.g. vote) to change. Controlled experiments arewidely used in the social and biological sciences (Imbensand Rubin 2015), and have numerous applications in indus-try (Varian 2016; Kohavi et al. 2013), from understandingthe impact of personalization algorithms to measuring in-cremental revenue due to ads.

While it is straightforward to randomly assign treatmentand control to units that are i.i.d., it is much harder to do for

Copyright c© 2020, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

units that interact with each other. One of the challenges innetwork experiment design is dealing with interference (orspillover), the problem of treatment ”spilling over” from atreated node to a control node through a shared edge, e.g.,information flowing from person to person in an online so-cial network, and diseases spreading between people inter-acting in the same physical space. The presence of inter-ference breaks the Stable Unit Treatment Value Assumption(SUTVA), the assumption that one unit’s outcomes are un-affected by another unit’s treatment assignment, and chal-lenges the validity of causal inference (Imbens and Rubin2015). Since SUTVA is hard to guarantee in real-world sce-narios, recent research on causal inference from graphs fo-cuses on designing controlled experiments in a way thatminimizes interference.

Prominent methods for interference minimization in con-trolled experiments rely on graph clustering (Eckles, Karrer,and Ugander 2017; Pouget-Abadie et al. 2018; Saveski et al.2017; Ugander et al. 2013). Graph clustering aims to finddensely connected clusters of nodes, such that few edges ex-ist across clusters (Schaeffer 2007). The basic idea of apply-ing it to causal inference is that little interference can occurbetween nodes in different clusters. Treatment and control isassigned at the cluster level, and the cluster assignment dic-tates the node assignment within each cluster, an experimentdesign known as two-stage randomization (Basse and Feller2018).

In this work, we make the key observation that there isan inherent tradeoff between interference and selection biasin cluster-based randomization based on the chosen num-ber of clusters (as demonstrated in Figure 1). Due to theheterogeneity of real-world graphs, discovered clusters canbe very different from each other, and the nodes in theseclusters may not represent the same underlying population.For example, a treatment cluster may represent ”predom-inantly Democrats from Oklahoma” while a control clus-ter may represent ”predominantly Republicans from NewYork.” Therefore, cluster randomization can lead to selec-tion bias in the data with causal effects that are confoundedby the difference in node features of each cluster, rather thanthe presence or absence of a treatment. Ideally, treatmentand control groups should represent the same populations,e.g., there should be clusters of ”predominantly Democratsfrom Oklahoma” assigned to both treatment and control. A

arX

iv:2

004.

0722

5v1

[cs

.LG

] 1

5 A

pr 2

020

common method for dealing with selection bias in observa-tional treatment and control data is matching, where nodesare matched based on their similarity and then assignedrandomly to treatment and control (Stuart 2010). However,there are no methods for incorporating node matching intomatching graph clusters, and our work is the first to proposesuch a method.

Our second main contribution is in introducing the con-cept of ”edge spillover probability” and account for it in thedesign. Clustering a connected graph component is guaran-teed to leave edges between clusters, therefore removing in-terference completely is impossible. At the same time, somenode pairs are more likely to interact than others and assign-ing such pairs to different treatment groups is more likely tolead to undesired spillover (and biased causal effect estima-tion) than separating pairs with low probability of interac-tion.

The goal of our work is to develop a framework for net-work experiment design in a way that minimizes both selec-tion bias and interference. We propose CMatch, a two-stageframework that achieves this goal through a novel objectivefunction for matching clusters and combining node match-ing with weighted graph clustering. While the idea of usinggraph clustering to address interference is not new (Uganderet al. 2013; Saveski et al. 2017; Eckles, Karrer, and Ugan-der 2017), incorporating node matching and edge spilloverprobabilities into it is novel.

Related WorkCausal inference models are studied by multiple disciplines,including statistics (Imbens and Rubin 2015), computer sci-ence (Pearl 2009), and the social sciences (Varian 2016).Here, we review relevant work on causal inference forgraphs.

Graph mining. Our framework relies on three prepro-cessing steps which can leverage existing graph mining al-gorithms for edge strength estimation, graph clustering, andnode representation learning. The goal of edge (or tie or rela-tionship) strength estimation is to assign each existing edgea numeric value which corresponds to a metric of interest fora pair of nodes, such as how close of friends two people areor how likely they are to communicate. Existing approachesrely on topological proximity (Liben-Nowell and Kleinberg2007), supervised models on node attributes (Gilbert andKarahalios 2009), or latent variable models (Li et al. 2010).Graph clustering aims to find subgraph clusters with highintra-cluster and low inter-cluster edge density (Yang andLeskovec 2015; Zhou, Cheng, and Yu 2009). A number ofalgorithms exist for weighted graph clustering (Schaeffer2007). Node representation learning approaches range fromgraph motifs (Milo et al. 2002) to embedding representa-tions (Hamilton, Ying, and Leskovec 2017) and statisticalrelational learning (SRL) (Rossi et al. 2012).

Dealing with interference bias. Recent work that ad-dresses interference in graphs relies on separating data sam-ples through graph clustering (Backstrom and Kleinberg2011; Eckles, Karrer, and Ugander 2017; Gui et al. 2015;Pouget-Abadie et al. 2018; Saveski et al. 2017; Ugander etal. 2013), relational d-separation (Lee and Honavar 2016;

Maier et al. 2010; Maier et al. 2013; Marazopoulou, Maier,and Jensen 2015; Rattigan, Maier, and Jensen 2011), or se-quential randomization design (Toulis and Kao 2013). Sinceour work is closest to the approaches based on graph clus-tering, we use these approaches as baselines in our exper-iments. None of the existing approaches account for inter-ference heterogeneity and the fact that different edges canhave different spillover probabilities; this is one of our maincontributions.

Matching and selection bias. In controlled experiments,the treatment assignment is randomized by the experimenter,whereas in estimating causal effects from observational data,the process by which the treatment is assigned is not de-cided by the experimenter and is often unknown. Match-ing is a prominent method for mimicking randomizationin observational data by pairing treated units with similaruntreated units. Then, the causal effect of interest is esti-mated based on the matched pairs, rather than the full setof units present in the data, thus reducing the selection biasin observational data (Stuart 2010). There are two main ap-proaches to matching, fully blocked and propensity scorematching (PSM) (Stuart 2010). Fully blocked matching se-lects pairs of units whose distance in covariate space is un-der a pre-determined distance threshold. PSM models thetreatment variable based on the observed covariates andmatches units which have the same likelihood of treatment.The few research articles that look at the problem of match-ing for relational domains (Oktay, Taylor, and Jensen 2010;Arbour et al. 2014) consider SRL data representations. Noneof them consider cluster matching for two-stage designwhich is one of our contributions.

Network experiment designThe goal of designing network experiments is to ensure re-liable causal effect estimation in controlled experiments forgraphs. As a running example, imagine that we are interestedto test whether showing a social media post about the ben-efits of voting would lead to a higher voter turnout. In thissection, we present our data model, give a brief overviewof the potential outcomes framework for causal inference,and present the challenges with causal effect estimation ingraphs. Then, we describe the causal effects of interest andformally define the problem of network experiment design.

Data modelA graph G = (V,E) consists of a set of n nodes V and a setof edges E = {eij} where eij denotes that there is an edgebetween node vi ∈ V and node vj ∈ V . Let Ni denote theset of neighbors for node vi, i.e. set of nodes that share anedge with vi. Let vi.X denote the pre-treatment node fea-ture variables (e.g., Twitter user features) for unit vi. Letvi.Y denote the outcome variable of interest for each nodevi (e.g., voting), and vi.T ∈ {0, 1} denote whether node vi(e.g., social media user) has been treated (e.g., shown a postabout the benefits of voting), vi.T = 1, or not, vi.T = 0.V1 and V0 indicate the sets of units in treatment and con-trol groups, respectively. For simplicity, we assume that bothvi.T and vi.Y are binary variables. The edge spillover prob-

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08RMSE

0.15

0.20

0.25

0.30

0.35

0.40Eu

clide

an d

istan

ce b

etwe

en

trea

tmen

t and

con

trol n

odes

2

10

100

300500 800

10002708

ClusteredRandomized

Figure 1: The tradeoff between selection bias (distance) andundesired spillover (RMSE) in cluster-based randomization;each data point is annotated with the number of clusters.

ability eij .p refers to the probability of interference occur-ring between two nodes.

Potential outcomes frameworkThe fundamental problem of causal inference is that we canobserve the outcome of a target variable for an individualvi in either the treatment or control group but not in both.Let vi.y(1) and vi.y(0) denote the potential outcomes ofvi.y if unit vi were assigned to the treatment or controlgroup, respectively. The treatment effect (or causal effect)is the difference g(i) = vi.y(1) − vi.y(0). Since we cannever observe the outcome of a unit under both treatmentand control simultaneously, the effect µ̂ of a treatment on anoutcome is typically calculated through averaging outcomesover treatment and control groups via difference-in-means:µ̂ = V1.Y − V0.Y (Stuart 2010). For the treatment effect tobe estimable, the following identifiability assumptions haveto hold:

• Stable unit treatment value assumption (SUTVA) refersto the assumption that the outcomes vi.y(1) and vi.y(0)are independent of the treatment assignment of otherunits: {vi.y(1), vi.y(0)} ⊥⊥ vj .T, ∀vj 6= vi ∈ V .• Ignorability (Imbens and Rubin 2015) – also known

as conditional independence (Pearl 2009) and absenceof unmeasured confoundness – is the assumption thatall variables vi.X that can influence both the treat-ment and outcome vi.Y are observed in the data andthere are no unmeasured confounding variables that cancause changes in both the treatment and the outcome:{vi.y(1), vi.y(0)} ⊥⊥ vi.T | vi.X .

• Overlap is the assumption that each unit assigned to thetreatment or control group could have been assigned tothe other group. This is also known as positivity assump-tion: P (vi.T |vi.X) > 0 for all units and all possible Tand X .

Challenges with causal effect estimation in graphsChallenge no. 1: It is hard to separate a graph into treat-ment and control nodes without leaving edges across. Thepresence of interference breaks the SUTVA assumption andleads to biased causal effect estimation in relational data.

Two-stage experimental design addresses this problem byfinding groups of units that are unlikely to interact with eachother (stage 1) and then randomly assigning each group totreatment and control (stage 2). Clustering has been pro-posed as a way to discover such groups that are stronglyconnected within but loosely connected across, thus findingtreatment and control subgraphs that have low probability ofspillover from one to the other. However, due to the densityof real-world graphs, graph clustering techniques can leaveas many as 65% to 79% of edges as inter-cluster edges (Ta-ble 2 in (Saveski et al. 2017)). Leaving these edges acrosstreatment and control nodes would lead to a large amount ofspillover. Incorporating information about the edge proba-bility of spillover into the clustering helps alleviate this prob-lem and is one of the main contributions of our work.

Challenge no. 2: There is a tradeoff between interfer-ence and selection bias in cluster-based network experi-ments. While randomization of i.i.d. units in controlled ex-periments can guarantee ignorability and overlap, two-stagedesign does not. One of the key observations in our work isthat dependent on the number of clusters, there is a tradeoffbetween interference and selection bias in terms of the treat-ment and control group not representing the same underly-ing distribution. Figure 1 illustrates this tradeoff for Cora,one of the datasets in our experiments, using reLDG as theclustering method. When a network is separated into veryfew clusters, the Euclidean distance between nodes in treat-ment and control clusters is larger than the Euclidean dis-tance when a lot of clusters are produced over the same net-work (e.g., 0.4 vs. 0.18 for 2 and 1000 clusters). This is in-tuitive because as the clusters get smaller and smaller, theirrandomization gets closer to mimicking full node random-ization (shown as a star). At the same time, a larger numberof clusters translates to a higher likelihood of edges betweentreatment and control nodes, which leads to higher unde-sired spillover and causal effect estimation error (e.g., 0.015vs. 0.059 for 2 and 1000 clusters).

Types of causal effects in networksIn real-world scenarios, we are interested in estimating theTotal Treatment Effect (TTE). Let Z ∈ {0, 1}N be the treat-ment assignment vector of all nodes. TTE is defined as theoutcome difference between two alternative universes, onein which all nodes are assigned to treatment (Z1 = {1}N )and one in which all nodes are assigned to control (Z0 ={0}N ) (Ugander et al. 2013; Saveski et al. 2017):

TTE =1

N

∑vi∈V

(vi.Y (Z1)− vi.Y (Z0)).

TTE is estimated as averages over the treatment and controlgroup, and it accounts for two types of effects, individualeffects (IE) and peer effects (PE):

ˆTTE = V1.Y − V0.Y = IE(V ) + PE(V1)− PE(V0).

Average individual effects (IE) reflect the difference inoutcome between treated and untreated subjects which canbe attributed to the treatment alone. They are estimated as:

IE(V ) = Evi∈V

[vi.Y |vi.T = 1]− Evi∈V

[vi.Y |vi.T = 0].

Average peer effects (PE) reflect the difference in outcomethat can be attributed to influence by other subjects in theexperiment. Let Ni.π denote the vector of treatment assign-ments to node vi’s neighbors Ni. Average PE is estimatedas having neighbors with a treatment vector:

PE(V ) = Evi∈V

[vi.Y |vi.T = t,Ni.π]

− Evi∈V

[vi.Y |vi.T = t,Ni = ∅].

Here, we distinguish between two types of peer effects,allowable peer effects (APE) and unallowable peer effects(UPE). Allowable peer effects are peer effects that occurwithin the same treatment group, and they are a natural con-sequence of network interactions. For example, if a socialmedia company wants to introduce a new feature (e.g., nudg-ing users to vote), it would introduce that feature to all usersand the total effect of the feature would include both indi-vidual and peer effects. Unallowable peer effects are peereffects that occur across treatment groups and contribute toundesired spillover and incorrect causal effect estimation.

For each node vi in treatment group t, we have two typesof neighbors: 1) neighborsN t

i in the same treatment class asnode vi with treatment assignment setN t

i .π; 2) set of neigh-bors in a different treatment class N t

i (t 6= t) with treatmentassignment denoted by N t

i .π. The APE is defined as:

APE(V ) = Evi∈V

[vi.Y |vi.T = t,N ti .π]

− Evi∈V

[vi.Y |vi.T = t,N ti = ∅],

and the UPE is defined as:

UPE(V ) = Evi∈V

[vi.Y |vi.T = t,N ti .π]

− Evi∈V

[vi.Y |vi.T = t,N ti = ∅].

Ideally, we would like to measure TTE = IE(V ) +APE(V1)−APE(V0). Due to undesired spillover in a con-trolled experiment, what we are able to measure insteadis the overall effect that comprises of both allowable andunallowable peer effects TTE = IE(V ) + APE(V1) −APE(V0) + UPE(V1) − UPE(V0). Therefore, when wedesigning an experiment for minimum interference, we areinterested in setting it up in a way that makes UPE(V1) = 0and UPE(V0) = 0.

There are two types of pairwise interference that canoccur, direct interference and contagion (Ogburn, Vander-Weele, and others 2014). What we have described so far iscausal effect estimation for direct interference which refersto a treated node vi (vi.X = 1) influencing the outcome ofanother node vj . For example, a treated social media userwho sees the post decides to share it with another user whoends up voting as a result. Contagion refers to the outcomeof node vi influencing another node vj to have the same out-come. For example, a social media user who votes can con-vince another user to vote. The above definitions of peer ef-fect can also be defined for contagion by conditioning themon neighbor outcomes, rather than neighbor treatments. Weleave this exercise to the reader.

Problem definitionThe goal of network experiment design is to minimize bothunallowable peer effects and selection bias in node assign-ment to treatment and control. More formally:

Problem 1 (Network experiment design) Given a graphG = (V ,E), a set of attributes V.X associated with eachnode and a set of spillover probabilities E.P associatedwith the graph edges, we want to construct two sets of nodes,the control nodes V0 ∈ V and the treatment nodes V1 ∈ Vsuch that:

a. V0 ∩ V1 = ∅b. |V0|+|V1| is maximizedc. θ = UPE(V1)− UPE(V0) is minimizedd. V0.X and V1.X are identically distributed

This problem definition describes the desired qualities ofthe experiment design at a high level. The first componentensures that the treatment and control nodes do not over-lap. The second component aims to keep as many nodes aspossible from V in the final design. The third componentminimizes unallowable spillover. The fourth component re-quires that there is no selection bias between the treatmentand control groups. The second and third component are atodds with one another and require a trade off because thelower θ, the lower the number of selected nodes for the ex-periment |V0|+|V1|. As we showed in Figure 1, there is alsoa tradeoff between the third and fourth component. In thenext section, we propose a solution to this problem.

CMatch: a network experiment designframework

Our network experiment design framework CMatch, illus-trated in Figure 2, has two main goals: 1) spillover minimiza-tion which it achieves through weighted graph clustering,and 2) selection bias minimization which it achieves throughcluster matching. Clusters in each matched pair are assignedto different treatments, thus achieving covariate balance be-tween treatment and control. The first goal addresses part cof Problem 1 and the second goal addresses part d. Whilethe first goal can be achieved with existing graph miningalgorithms, solving for the second one requires developingnovel approaches. To achieve the second goal, we proposean objective function, which can be solved with maximumweighted matching, and present the nuances of operational-izing each step.

Step 1: Interference minimization throughweighted graph clusteringExisting cluster-based techniques for network experimentdesign assume unweighted graphs (Backstrom and Klein-berg 2011; Eckles, Karrer, and Ugander 2017; Gui et al.2015; Saveski et al. 2017; Ugander et al. 2013) and do notconsider that different edges can have different likelihoodof spillover. Incorporating information about the edge prob-ability of spillover into the clustering helps alleviate thisproblem and is one of the main contributions of our work.In order to minimize undesired spillover, we operationalize

1

54

98

76

2 3

13

2

8

9

7

4

5

6

61

54

98

76

2 3

1

5

2

4

37

9

8

Control

Control

Treatment Treatment

Clustering

Node M

atchin

g

Input CMatch Output

Cluster Matching

12

12

12

1211

11

11

11

10

10

1010

Figure 2: Illustration of CMatch framework for minimizing interference and selection bias in controlled experiments. Input: agraph of nodes and the connection between them. CMatch: node and cluster matching; the dashed circles indicates the clusters.Matched nodes are represented with a similar circle border. Output: assigning the matched cluster pairs to treatment and controlrandomly; circles with the same color represent matched clusters.

minimizing θ as minimizing the edges, and more specifi-cally the edge spillover probabilities, between treatment andcontrol nodes: θ̂ =

∑∀vi∈V0,∀vj∈V1

eij .p. To achieve this,CMatch creates graph clusters for two-stage design by em-ploying two functions, edge spillover probability estimationand weighted graph clustering.

Edge spillover probability estimation. We consideredge strength, how strong the relationship between twonodes is, as a proxy for edge spillover probability. This re-flects the notion that the probability of a person influencinga close friend to do something is higher than the probabilityof influencing an acquaintance. We can use common graphmining techniques to calculate edge strength, including onesbased on topological proximity (Liben-Nowell and Klein-berg 2007), supervised classification (Gilbert and Karahalios2009), or latent variable models (Li et al. 2010).

Weighted graph clustering. In order to incorporate edgestrength into clustering, we can use any existing weightedgraph clustering algorithm (Enright, van Dongen, andOuzounis 2002; Schaeffer 2007; Yang and Leskovec 2015).In our experiments, we use a prominent non-parametric al-gorithm, the Markov Clustering Algorithm (MCL) (Enright,van Dongen, and Ouzounis 2002) which applies the ideaof random walk for clustering graphs and produces non-overlapping clusters. We also compare this algorithm withreLDG which was the basis of previous work (Saveski et al.2017). One of the advantages of MCL is that it automati-cally finds the optimal number of clusters, rather than requir-ing it as input. The main idea behind MCL is that nodes inthe same cluster are connected with higher-weighted short-est paths than nodes in different clusters.

Step 2: Selection bias minimization through clustermatchingRandomizing treatment assignment over clusters in a two-stage design does not guarantee that nodes within thoseclusters would represent random samples of the popula-tion. We propose to address this selection bias problemby cluster matching and balancing covariates across treat-ment and control clusters. While methods for matchingnodes exist (Stuart 2010; Oktay, Taylor, and Jensen 2010;Arbour et al. 2014), this work is the first to propose methodsfor matching clusters.

Objective function. The goal of cluster matching is tofind pairs of clusters with similar node covariate distribu-tions and assign them to different treatment groups. We pro-pose to capture this through a maximum weighted match-ing objective over a cluster graph in which each discoveredcluster from step 1 is a node and edges between clusters rep-resent their similarity. Suppose that graph G is partitionedinto C = {c1, c2, ..., cg} clusters. We define A ∈ {0, 1},such that aij = 1 if two clusters ci and cj are matched, elseaij = 0. wi,j ∈ R represents the similarity between twoclusters ci and cj . Then the objective function of CMatchis as follows:

argmaxA

g∑i=1

g∑j=i+1

(aij · wij)

subject to ∀ci ∈ C,|ci|∑j=1

aij ≤ 1, aij ∈ {0, 1}.

(1)

This objective function maps to a maximum weighted

matching problem for which there is a linear-time ap-proximation algorithm (Duan and Pettie 2014) and apolynomial-time exact algorithm with O(N2.376) (Muchaand Sankowski 2004; Harvey 2009).

Solution. In order to operationalize the solution to this ob-jective, the main question that needs to be addressed is: whatdoes it mean for two clusters to be similar? We propose tocapture this cluster similarity through matched nodes. Themore nodes can be matched based on their covariates acrosstwo clusters, the more similar two clusters are. Thus, theoperationalization comes down to the following three ques-tions which we address next:

1. What constitutes a node match?

2. How are node matches taken into consideration in com-puting the pairwise cluster weights (cluster similarity)?

3. Given a cluster weight, what constitutes a potential clustermatch, and thus an edge in the cluster graph?

Once these three questions are addressed, the cluster graphcan be built and an existing maximum weighted matching al-gorithm can be applied on it to find the final cluster matches.

Node Matching. The goal of node matching is to reducethe imbalance between treatment and control groups due totheir different feature distributions. Given a node represen-tation, fully blocked matching would look for the most sim-ilar nodes based on that representation (Stuart 2010). It isimportant to note that propensity score matching does notapply here because it models the probability of treatmentin observational data and treatment is unknown at the timeof designing a controlled experiment. In its simplest form,a node can be represented as a vector of attributes, includ-ing node-specific attributes, such as demographic character-istics, and structural attributes, such as node degree. For anytwo nodes, it is possible to apply an appropriate similaritymeasure sim(vi, vj), in order to match two nodes, includingcosine similarity, Jaccard similarity or Euclidean distance.

We consider two different options to match a pair of nodesin different clusters (and ignore matches within the samecluster):

• Threshold-based node matching (TNM): Node vk incluster ci is matched with node vl from a different clustercj if the pairwise similarity of nodes sim(vk, vl) > α.The threshold α can vary from 0, which liberally matchesall pairs of nodes, to the maximum possible similaritywhich matches nodes only if they are exactly the same.In our experiments, we set α based on the covariate dis-tribution of each dataset and consider different quartilesof pairwise similarity as thresholds. This allows for eachnode to have multiple possible matches across clusters.

• Best node matching (BNM): Node vk in cluster ci ismatched with only one node vl which is most similar tovk in the whole graph; vl should be in a different cluster.This is a very conservative matching approach in whicheach node is uniquely matched but allows the matching tobe asymmetric.

Cluster Weights. After the selection of a node matchingmechanism, we are ready to define the pairwise similarity

of clusters which is the basis of cluster matching. We con-sider three simple approaches and three more expensive ap-proaches which require maximum weighted matching be-tween nodes:

• Euclidean distance (E): This approach is the simplest ofall because it does not consider node matches and it sim-ply calculates the Euclidean distance between the nodeattribute vector means of two clusters.

• Matched node count (C): The first approach counts thenumber of matched nodes in each pair of clusters ci and cjand consider the count as the clusters’ pairwise similarity:wij =

∑|ci|k=1

∑|cj |l=1 r

ijkl. A node in cluster ci can have

multiple matched nodes in cj .• Matched node average similarity (S): Instead of the

count, this approach considers the average similarity be-tween matched nodes across two clusters ci and cj :

wij =∑|ci|

k=1

∑|cj |l=1 rijkl·sim(vk,vl)∑|ci|

k=1

∑|cj |l=1 rijkl

.

These first two approaches allow a single node to bematched with multiple nodes in another cluster and eachof those matches to count towards the cluster pair weight.In order to distinguish this from a more desirable case inwhich multiple nodes in one cluster are matched to multiplenodes in another cluster, we propose approaches that alloweach node to be considered only once in the matches thatcount towards the weight. For each pair of clusters, we builda node graph in which an edge is formed between nodesvi and vj in the two clusters and the weight of this edgeis sim(vi, vj). Maximum weighted matching will find thebest possible node matches in the two clusters. We considerthree different variants for calculating the cluster pair weightbased on the maximum weighted matching of nodes:

• Maximum matched node count (MC): This method cal-culates the cluster weight the same way as C except thatthe matches (whether rijkl is 0 or 1) are based on the max-imum weighted matching result.

• Maximum matched node average similarity (MS): Thismethod calculates the cluster weight the same way as Sexcept that the node matches are based on the maximumweighted matching result.

• Maximum matched node similarity sum (MSS): Thismethod calculates the cluster weight similarly to MS ex-cept that it does not average the node similarity: wij =∑|ci|

k=1

∑|cj |l=1 r

ijkl · sim(vk, vl).

Cluster Graph. Once the cluster similarities of have beendetermined, we need to decide what similarity constitutes apotential cluster match. Such potential matches are added asedges in the cluster graph which is considered for maximumweighted matching. We consider three different options:

• Threshold-based cluster matching (TCM): Cluster ciis considered as a potential match of cluster cj if theirweight wi,j > β. The threshold β can vary from 0, whichallows all pairs of clusters to be potential matches, to themaximum possible similarity which allows matching be-tween clusters only if they are exactly the same. In our

experiments, we set β based on the distribution of pair-wise similarities and their quartiles as thresholds.

• Greedy cluster matching (GCM): For each cluster ci, asorted list of the similarities between ci and all other clus-ters is defined. Cluster ci is considered a potential matchonly to the cluster with the highest similarity value in thelist.

The last step inCMatch runs maximum weighted match-ing on the cluster graph. For every matched cluster pair, itassigns one cluster to treatment and the other one to controlat random. This completes the network experiment design.

ExperimentsIn this section, we evaluate the advantages of CMatchin minimizing interference and selection bias compared tostate-of-the-art methods.

Semi-synthetic dataWe consider four real-world attributed network datasets. Ta-ble 1 shows the basic dataset characteristics. The 50 Womendataset (Michell and Amos 1997) incorporates the smoking,alcohol, sport and drug habits of 50 students. Hamsterster(Zheleva et al. 2008) describes the online social network ofhamsters and their attributes.Cora andCiteseer (Sen et al.2008) are two citation networks with binary bag-of-wordsattributes for each article.

We assume that the underlying probability of activating anode (changing the outcome) due to treatment and allowablepeer effects in the treatment group is 0.4 and the underlyingprobability of activating a control node due to treatment andallowable peer effects is 0.2 which makes the true causal ef-fect TTE = 0.2. Based on these probabilities, we randomlyassign each node as activated or not. For each inactivatednodes, we simulate two types of interference consideringboth fixed values (0.1 and 0.5) and values based on the edgeweights for e.p:

1. Direct interference: each treated neighbor of a controlnode activates the node with unallowable spillover proba-bility of e.p.

2. Contagion: inactive treated and untreated nodes get acti-vated with the unallowable spillover probability of e.p ifthey are connected to at least one activated node in a dif-ferent treatment class.

Table 1: Number of nodes, edges and attributes in thedatasets

Dataset Nodes Edges Attributes

Citeseer 3,312 4,675 3709Cora 2,708 5,278 1440Hamsterster 2,059 10,943 650 Women 50 74 4

Main algorithms and baselinesAll our baseline and main algorithm variants take an at-tributed graph as an input and produce a set of clusters, eachassigned to treatment, control, or none. For graph cluster-ing, we considered two main algorithms, Restreaming Lin-ear Deterministic Greedy (reLDG) (Nishimura and Ugander2013) and Markov Clustering Algorithm (MCL) (Enright,van Dongen, and Ouzounis 2002). reLDG takes as input anunweighted graph and desired number of clusters and pro-duces a graph clustering. reLDG was reported to performvery well in state-of-the-art methods for network experimentdesign (Saveski et al. 2017). MCL is a non-parametric al-gorithm which takes as input a weighted graph and producesa graph clustering. The edge weights which correspond tothe probabilities of spillover are estimated based on nodepair similarity using one minus the normalized L2 norm:1− L2(vi.x, vj .x).

The main algorithms and baselines are:• Randomized: This algorithm assigns nodes to treat-

ment and control randomly, ignoring the network.• CR (Saveski et al. 2017): The Completely Randomized

(CR) algorithm was used as a baseline in (Saveski et al.2017). The algorithm clusters the unweighted graph us-ing reLDG algorithm, assigns similar clusters to the samestrata and assigns nodes in strata to treatment and controlin a randomized fashion

• CBRreLDG (Saveski et al. 2017): Cluster-based Ran-domized assignment (CBR) is the main algorithm pro-posed by (Saveski et al. 2017). The algorithm clustersthe unweighted graph using reLDG, assigns similar clus-ters to the same strata and randomly picks clusters withinthe same strata as treatment or control.

• CBRMCL: A variant of CBR that we introduce forthe sake of fairness which uses MCL for weighted-graphclustering.

• Match: This algorithm matches nodes using maximumweighted matching algorithm and then randomly assignsnodes in each matched pair to treatment and control atrandom without considering clustering.

• CMatchreLDG: This method uses our CMatchframework but works on an unweighted graph. It usesreLDG for graph clustering.

• CMatchMCL: This is our proposed technique whichuses MCL for weighted graph clustering.CMatch uses the maximum weight matching func-

tion from the NetworkX Python library.

Experimental setupWe run a number of experiments varying the underly-ing spillover assumptions, clustering algorithms, number ofclusters, and node matching algorithms. Our experimentalsetup measures the desired properties for network exper-iment design, as described in Problem 1 and follows theexperimental setups in existing work (Arbour et al. 2014;Eckles, Karrer, and Ugander 2017; Maier et al. 2013; Stuart2010; Saveski et al. 2017).

0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09RMSE

0.14

0.16

0.18

0.20

0.22

0.24

0.26

0.28

0.30Eu

clide

an d

istan

ce b

etwe

en

trea

tmen

t and

con

trol n

odes

20

100

500

1885

RandomizedCRCBR_MCLCBR_reLDGMatchCMatch_reLDGTNM2+C+TCM1TNM3+S+TCM3TNM0+MS+TCM0TNM2+MSS+TCM3TNM2+C+TCM2

Figure 3: The tradeoff between selection bias (distance) andundesirable spillover (RMSE) in CMatchMCL variants (la-beled with methods applied in) and baselines in Cora datasetfor e.p = edge-weight; CBRreLDG is annotated with thenumber of clusters

To measure the strength of interference bias in differentestimators, we report on two metrics:

1. Root Mean Squared Error (RMSE) of the total treatmenteffect calculated as:

RMSE =

√√√√ 1

S

S∑s=1

((τ̂s − τs)2)

where S is the number of runs and τs and τ̂s are the trueand estimated causal effect in run s, respectively. We setS = 10 in all experiments. Error can be attributed to un-desired spillover only.

2. The number of edges and sum of edge weights betweentreatment and control nodes assigned by each algorithm.

To show the selection bias, we want to assess how differenttreatment vs. control nodes are. We compute the Euclideandistance between the attribute vector mean of treated andthat of untreated nodes. We show the average and standarddeviation over 10 runs.

We run all 115 possible combinations of CMatch op-tions for node matching, cluster weights and cluster graphfor each dataset. We consider four different values for thethreshold α in TNM: 0 (TNM0), first (TNM1), second(TNM2) and third (TNM3) quantile of pairwise nodes’ sim-ilarity distribution where sim(vi, vj)= (1- the normalizedL2 norm). For TCM, we consider four different β values: 0(TCM0), first (TCM1), second (TCM2) and third (TCM3)quantile of the pairwise clusters’ similarity distribution foreach dataset. We use TNM2 + C + TCM2 in all the experi-ments of CMatchreLDG.

Unless otherwise specified, the number of clusters isthe same for all CBR and CMatch versions based on theoptimal determination by MCL as optimal for each re-spective dataset. The number of clusters determined byMCL is 2, 497 for Citeseer, 1, 885 for Cora, 1, 056 forHamsterster and 20 in 50 Women dataset.

ResultsTradeoff between interference and selection bias inCMatch variants and baselines: Given the large number

of CMatch option combinations (115), we first find whichones of these combinations have a good tradeoff betweenRMSE and Euclidean distance (between treatment and con-trol) with e.p = edge-weight. Based on these experiments,we notice that 1) methods with stricter cluster thresholds(TCM2 and TCM3) tend to have lower error, 2) stricternode match thresholds (TNM2 and TNM3) have lower er-ror than others for S and MSS and 3) MS has high erroracross thresholds. Due to space constraints, we are showingdetailed results for Cora only.

Fig. 3 shows the results for the CMatch variants withthe best tradeoffs and their better performance when com-pared to the baselines for Cora. Full CMatch resultscan be found in Table 2. The figure clearly shows thatthe selection bias decreases at the expense of interfer-ence bias. For example, while the Euclidean distance forTNM0+MS+TCM0 is low (0.155) when compared toTNM2+C+TCM2 (0.253), its RMSE is higher, 0.048 vs.0.01. The comparison between CBRreLDG with differentpossible number of clusters is consistent with the tradeoffshown in Fig.1. CBRreLDG with the highest error (anno-tated with 1885) and CMatchMCL have the same num-ber of clusters. It is intuitive that the Match method hasthe least selection bias, because all nodes have their bestmatches. However, similar to the Randomized method, itsuffers from high interference bias (RMSE) because of thehigh density of edges between treatment and control nodes.

Interference evaluation for contagion: We choose twoCMatch variants with low estimation errors: TNM2 +MSS + TCM3 and TNM2 + C + TCM2, denoted byCMatchMCLMSS

and CMatchMCLCrespectively, and

compare their causal effect estimation error with the base-lines. The first method uses a simpler cluster weight as-signment while the second one uses the expensive maxi-mum weighted matching of nodes. Fig. 4 shows that bothvariants of CMatchMCL get significantly lower error thanother methods, especially in Citeseer and Cora with 75.5%and 81.8% estimated error reduction in comparison toCBRreLDG for e.p = edge-weight. CMatchMCLMSS

hashigher error thanCMatchMCLC

in most of the experimentswhich is expected as shown in Figure 3. Randomized andMatch approaches have similar performance in all datasetsbecause of their similarity in node randomization approach.We also notice that CBRreLDG has the highest estima-tion error in Hamsterster data which confirms that clusteringhas a significant effect on the unallowable spillover. Mean-while, CMatchreLDG outperforms other baselines in somedatasets (Citeseer) and but not in others (Hamsterster and 50Women). In Citeseer, the CR method gets the largest esti-mation error.

Fig. 4 also shows that the higher the unallowable spilloverprobability, the larger the estimation error but also the bet-ter our method becomes relative to the baselines. For exam-ple, by increasing the unallowable spillover probability from0.1 to 0.5 in Citeseer, the estimation error increases from0.005 to 0.02 for CMatchMCLC

and from 0.023 to 0.086for CBRreLDG.

Interference evaluation for direct interference: Fig. 5shows the difference between RMSE of different estima-

Citeseer Cora Hamsterster 50 Women0.00

0.05

0.10

0.15

0.20

e.p = edge-weight


0.02

0.04

0.06

0.08

e.p = 0.1


0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200e.p = 0.5

RandomizedCRCBR_reLDGCBR_MCLMatchCMatch_reLDGCMatch_MCL_MSSCMatch_MCL_C

RMSE

of T

otal

Effe

ct

Figure 4: RMSE of total effect in the presence of contagion considering different unallowable spillover probabilities in alldatasets; CMatchMCLC

achieves the lowest error in all datasets


0.1

0.2

0.3

0.4

0.5

RMSE

of T

otal

Effe

ct

RandomizedCRCBR_reLDGCBR_MCLMatchCMatch_reLDGCMatch_MCL_MSSCMatch_MCL_C

Figure 5: RMSE of total effect in the presence of di-rect interference (e.p = edge-weight). CMatchMCLC

andCMatchMCLMSS obtain the lowest RMSE for all datasets.

tors over the presence of direct interference for e.p = edge-weight. In four datasets, both variants of CMatchMCL getthe lowest estimation error in comparison to baseline meth-ods. For example, CMatchMCLC

’s error is approximatelyhalf of the error of CBRreLDG (0.06 vs. 0.13 for Citeseer,0.1 vs. 0.22 for Cora, 0.31 vs. 0.54 for Hamsterster, 0.15vs. 0.36 for 50 Women). Similar to contagion, Match andRandomized methods have similar estimation errors.

Potential spillover evaluation: Table 3 shows the po-tential spillover between treatment and control nodes as-signed by different methods. This applies to both con-tagion and direct interference. CMatch has the lowestsum of edges and edge weights between treatment andcontrol nodes across all datasets. The difference betweenCMatchMCLC

and the baselines in Cora and Citeseer issubstantial: CMatchMCLC

has between 13.5% and 34.8%lower number of edges between treatment and control acrossdatasets.

Selection bias evaluation for contagion. In this ex-periment, we look at the relationship between number ofclusters and the difference between treatment and controlnodes with and without cluster matching. Fig. 6 shows the

20 100 10000.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Citeseer

20 100 10000.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35Cora

20 100 10000.00

0.05

0.10

0.15

0.20

Hamsterster

10 20 300.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

50 Women

reLDGCBR_reLDGCMatch_reLDG

Eucli

dean

dist

ance

bet

ween

tr

eatm

ent a

nd c

ontro

l nod

es

Number of Clusters

Figure 6: Euclidean distance between the attribute vectormeans of treatment and control nodes for different numberof clusters. The higher the number of clusters, the lower theselection bias.

Euclidean distance between the average of treatment andcontrol nodes’ attributes in CMatchreLDG, CBRreLDG

and reLDG for three different number of clusters and un-allowable spillover probability e.p = edge-weight. SinceCMatchreLDG optimizes for selection bias directly, it is notsurprising that it results in treatment and control nodes thathave more similar feature distributions than the other twomethods. In Citeseer the differences are more subtle than inthe other datasets. Error bars show the variance of averagesover 10 runs which confirms the low variance of estimationsin all datasets except in 50 Women, which is a small dataset.

Sensitivity to spillover probability metrics. Our lastexperiment compares metrics for calculating the spilloverprobability, Cosine similarity, Jaccard similarity and theL2-based similarity used in all other experiments. Wereport on RMSE of total effect using CMatchMCLC

and CMatchMCLMSSmethods under contagion. Figure

7 shows that CMatchMCLCwith L2-based similarity

obtains the least error in all datasets except for Cite-seer where Cosine similarity has slightly lower error. ForCMatchMCLMSS

, Cosine similarity has the lowest RMSEin Citeseer and 50 Women dataset, while Euclidean similar-ity has the lowest error in the other datasets. Jaccard similar-ity has the highest estimation error in all almost all cases.

Table 2: The tradeoff between selection bias (distance) and undesirable spillover (RMSE) in CMatch variants in Cora dataset.CMatchMCL variants used in Fig. 3 are in bold.

TCM0 TCM1 TCM2 TCM3 GCMRMSE ED RMSE ED RMSE ED RMSE ED RMSE ED

C

TNM0 0.052 0.184 0.007 0.267 0.017 0.263 0.014 0.26 0.048 0.789TNM1 0.055 0.176 0.051 0.177 0.008 0.258 0.012 0.26 0.031 0.6TNM2 0.054 0.171 0.042 0.171 0.01 0.253 0.017 0.251 0.036 0.591TNM3 0.043 0.175 0.043 0.175 0.173 0.046 0.018 0.231 0.034 0.592BNM 0.012 0.262 0.037 0.481 0.049 0.485 0.059 0.479 0.025 0.274

S

TNM0 0.056 0.16 0.058 0.159 0.048 0.16 0.056 0.162 0.035 0.34TNM1 0.055 0.16 0.053 0.162 0.057 0.165 0.054 0.166 0.026 0.31TNM2 0.056 0.162 0.054 0.168 0.048 0.165 0.033 0.183 0.039 0.292TNM3 0.057 0.169 0.041 0.174 0.024 0.198 0.015 0.211 0.021 0.275BNM 0.014 0.253 0.017 0.264 0.02 0.27 0.027 0.303 0.014 0.277

MC

TNM0 0.049 0.177 0.015 0.261 0.01 0.262 0.008 0.263 0.042 0.189TNM1 0.055 0.173 0.052 0.174 0.01 0.257 0.012 0.253 0.040 0.191TNM2 0.047 0.171 0.051 0.177 0.013 0.261 0.007 0.263 0.024 0.211TNM3 0.047 0.173 0.049 0.178 0.051 0.176 0.011 0.249 0.012 0.244BNM N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

MS


MSS


E N/A 0.051 0.178 0.05 0.18 0.031 0.203 0.012 0.242 0.042 0.718

Table 3: Percentage of edges (and edge weights) between treatment and control nodes. The lower the number, the lower proba-bility of undesired spillover.Dataset Randomized CR CBRreLDG CBRMCL Match CMatchreLDG CMatchMCLC

Citeseer 49.9%(50%) 35.9% (36.3%) 39.8% (38.4%) 38.9%(38.4%) 53.9% (56.6%) 35.8% (34.4%) 7.5%(7.2%)Cora 49.7%(49.7%) 37.6%(37.6%) 43.4%(42.8%) 38.9% (33.6%) 51.8%(53.3%) 38.7% (38.2%) 8.6%(9.1%)Hamsterster 50.2%(50.1%) 31.7%(30.4%) 48.3%(48.3%) 35.1% (34.7%) 50% (50.1%) 43.3% (44.4%) 34.8%(34.4%)50 Women 48.5%(48.1%) 31.8%(30.5%) 36.6%(34.3%) 18.3%(11.4%) 52.5%(52.7%) 16%(18.6%) 12.8%(9.7%)

Citeseer Cora Hamsterster 50 women0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

RMSE

of T

otal

Effe

ct

CMatch_MCL_C_L2CMatch_MCL_C_coCMatch_MCL_C_jaCMatch_MCL_MSS_L2CMatch_MCL_MSS_coCMatch_MCL_MSS_ja

Figure 7: RMSE of total effect in the presence of contagionusing three different similarity methods to calculate spilloverprobability: Cosine (co), Jaccard (ja) and L2 similarity.

Conclusion

We presented CMatch, the first optimization frameworkthat minimizes both interference and selection bias incluster-based network experiment design. We demonstratedthe tradeoff between causal effect estimation error and dis-tance between treatment and control groups, as well as thevalue of combining weighted graph clustering with clus-ter matching. Our experiments on four real-world networkdatasets showed that CMatch reduces the causal effect es-timation error by 8.6% to 91.4% when compared to state-of-the-art techniques. Some possible extensions of our frame-work include understanding the impact of network structuralproperties on estimation, jointly optimizing for interferenceand selection bias, and developing frameworks that are ableto mitigate for multiple-hop diffusions.

References[Arbour et al. 2014] Arbour, D. T.; Marazopoulou, K.;Garant, D.; and Jensen, D. D. 2014. Propensity score match-ing for causal inference with relational data. In UAI Work-shop on causal inference, 25–34.

[Backstrom and Kleinberg 2011] Backstrom, L., and Klein-berg, J. 2011. Network bucket testing. In WWW.

[Basse and Feller 2018] Basse, G., and Feller, A. 2018.Analyzing two-stage experiments in the presence of inter-ference. Journal of the American Statistical Association113(521):41–55.

[Duan and Pettie 2014] Duan, R., and Pettie, S. 2014.Linear-time approximation for maximum weight matching.J. ACM 61(1):1:1–1:23.

[Eckles, Karrer, and Ugander 2017] Eckles, D.; Karrer, B.;and Ugander, J. 2017. Design and analysis of experimentsin networks: Reducing bias from interference. Journal ofCausal Inference.

[Enright, van Dongen, and Ouzounis 2002] Enright, A. J.;van Dongen, S.; and Ouzounis, C. A. 2002. An efficientalgorithm for large-scale detection of protein families. Nu-cleic acids research 30 7:1575–84.

[Gilbert and Karahalios 2009] Gilbert, E., and Karahalios,K. 2009. Predicting tie strength with social media. In CHI,211–220. ACM.

[Gui et al. 2015] Gui, H.; Xu, Y.; Bhasin, A.; and Han, J.2015. Network a/b testing: From sampling to estimation.In WWW.

[Hamilton, Ying, and Leskovec 2017] Hamilton, W. L.;Ying, R.; and Leskovec, J. 2017. Representation learningon graphs: Methods and applications. TKDE.

[Harvey 2009] Harvey, N. J. A. 2009. Algebraic algo-rithms for matching and matroid problems. SIAM J. Comput.39(2):679–702.

[Imbens and Rubin 2015] Imbens, G., and Rubin, D. 2015.Causal Inference in Statistics, Social and Biomedical Sci-ences: An Introduction. Cambridge Univ Press.

[Kohavi et al. 2013] Kohavi, R.; Deng, A.; Frasca, B.;Walker, T.; Xu, Y.; and Pohlmann, N. 2013. Online con-trolled experiments at large scale. In KDD.

[Lee and Honavar 2016] Lee, S., and Honavar, V. 2016. OnLearning Causal Models from Relational Data. In AAAI,3263–3270.

[Li et al. 2010] Li, L.; Chu, W.; Langford, J.; and Schapire,R. E. 2010. A contextual-bandit approach to personalizednews article recommendation. In WWW.

[Liben-Nowell and Kleinberg 2007] Liben-Nowell, D., andKleinberg, J. 2007. The link-prediction problem for socialnetworks. JASIST 58(7):1019–1031.

[Maier et al. 2010] Maier, M.; Taylor, B.; Oktay, H.; andJensen, D. 2010. Learning causal models of relational do-mains. In AAAI.

[Maier et al. 2013] Maier, M.; Marazopoulou, K.; Arbour,D.; and Jensen, D. 2013. A sound and complete algorithmfor learning causal models from relational data. In UAI.

[Marazopoulou, Maier, and Jensen 2015] Marazopoulou,K.; Maier, M.; and Jensen, D. 2015. Learning the structureof causal models with rel. and temporal dependence. InUAI.

[Michell and Amos 1997] Michell, L., and Amos, A. 1997.Girls, pecking order and smoking. Social Science &Medicine 44(12):1861 – 1869.

[Milo et al. 2002] Milo, R.; Shen-Orr, S.; Itzkovitz, S.; Kash-tan, N.; Chklovskii, D.; and Alon, U. 2002. Network mo-tifs: simple building blocks of complex networks. Science298(5594):824–827.

[Mucha and Sankowski 2004] Mucha, M., and Sankowski, P.2004. Maximum matchings via gaussian elimination. In45th Annual IEEE Symposium on Foundations of ComputerScience, 248–255.

[Nishimura and Ugander 2013] Nishimura, J., and Ugander,J. 2013. Restreaming graph partitioning: Simple versatilealgorithms for advanced balancing. In KDD.

[Ogburn, VanderWeele, and others 2014] Ogburn, E. L.;VanderWeele, T. J.; et al. 2014. Causal diagrams forinterference. Statistical science 29(4):559–578.

[Oktay, Taylor, and Jensen 2010] Oktay, H.; Taylor, B.; andJensen, D. 2010. Causal discovery in social media usingquasi-experimental designs. In KDD Workshop on SocialMedia Analytics.

[Pearl 2009] Pearl, J. 2009. Causality. Cambridge UnivPress.

[Pouget-Abadie et al. 2018] Pouget-Abadie, J.; Mirrokni, V.;Parkes, D. C.; and Airoldi, E. M. 2018. Optimizing cluster-based randomized experiments under monotonicity. In Pro-ceedings of the 24th ACM SIGKDD International Confer-ence on Knowledge Discovery and Data Mining, KDD 18,20902099. New York, NY, USA: Association for Comput-ing Machinery.

[Rattigan, Maier, and Jensen 2011] Rattigan, M. J.; Maier,M. E.; and Jensen, D. D. 2011. Relational blocking forcausal discovery. In AAAI.

[Rossi et al. 2012] Rossi, R.; McDowell, L.; Aha, D.; andNeville, J. 2012. Transforming graph data for statisticalrelational learning. JAIR 45:363–441.

[Saveski et al. 2017] Saveski, M.; Pouget-Abadie, J.; Saint-Jacques, G.; Duan, W.; Ghosh, S.; Xu, Y.; and Airoldi, E.2017. Detecting network effects: Randomizing over ran-domized experiments. In KDD.

[Schaeffer 2007] Schaeffer, S. E. 2007. Graph clustering.Computer science review 1(1):27–64.

[Sen et al. 2008] Sen, P.; Namata, G. M.; Bilgic, M.; Getoor,L.; Gallagher, B.; and Eliassi-Rad, T. 2008. Collective clas-sification in network data. AI Magazine 29(3):93–106.

[Stuart 2010] Stuart, E. A. 2010. Matching methods forcausal inference: A review and a look forward. Statisticalscience 25(1):1.

[Toulis and Kao 2013] Toulis, P., and Kao, E. 2013. Estima-tion of causal peer influence effects. In ICML, 1489–1497.

[Ugander et al. 2013] Ugander, J.; Karrer, B.; Backstrom, L.;and Kleinberg, J. 2013. Graph cluster randomization: Net-work exposure to multiple universes. In KDD.

[Varian 2016] Varian, H. 2016. Causal inference in eco-nomics and marketing. PNAS.

[Yang and Leskovec 2015] Yang, J., and Leskovec, J. 2015.Defining and evaluating network communities based onground-truth. KAIS 42(1):181–213.

[Zheleva et al. 2008] Zheleva, E.; J.; Kuter, U.; and Getoor,L. 2008. Using friendship ties and family circles for linkprediction. SNAKDD.

[Zhou, Cheng, and Yu 2009] Zhou, Y.; Cheng, H.; and Yu,J. X. 2009. Graph clustering based on structural/attributesimilarities. Proc. VLDB Endow. 2(1):718729.

minimizing interference and selection bias in network ... · current approaches to a/b testing in...

Documents