the indian chefs process
TRANSCRIPT
The Indian Chefs Process
Patrick Dallaire1 Luca Ambrogioni2 Ludovic Trottier1 Umut Güçlü2 Max Hinne2
Philippe Giguère1 Brahim Chaib-Draa1 Marcel van Gerven2 Francois Laviolette1
1Université Laval 2Radboud University
Abstract
This paper introduces the Indian Chefs Pro-cess (ICP), a Bayesian nonparametric prioron the joint space of infinite directed acyclicgraphs (DAGs) and orders that generalizesIndian Buffet Processes. As our constructionshows, the proposed distribution relies on a la-tent Beta Process controlling both the ordersand outgoing connection probabilities of thenodes, and yields a probability distribution onsparse infinite graphs. The main advantageof the ICP over previously proposed Bayesiannonparametric priors for DAG structures is itsgreater flexibility. To the best of our knowl-edge, the ICP is the first Bayesian nonpara-metric model supporting every possible DAG.We demonstrate the usefulness of the ICP onlearning the structure of deep generative sig-moid networks as well as convolutional neuralnetworks.
1 Introduction
In machine learning and statistics, the directed acyclicgraph (DAG) is a common modelling choice for express-ing relationships between objects. Prime examples ofDAG-based graphical models include Bayesian net-works, feed-forward neural networks, causal networks,deep belief networks, dynamic Bayesian networks andhidden Markov models, to name a few. Learning the un-known structure of these models presents a significantlearning challenge, a task that is often avoided by fixingthe structure to a large and hopefully sufficiently ex-pressive model. Structure learning is a model selectionproblem in which ones estimates the underlying graph-ical structure of the model. Over the years, researchershave explored a great variety of approaches to thisproblem [13, 23, 2, 15, 17, 18, 24], from frequentist toBayesian, and some using pure heuristic-based search,but the vast majority is limited to finite parametricmodels.
Bayesian nonparametric learning methods are appeal-ing alternatives to their parametric counterparts, be-cause they offer more flexibility when dealing withgenerative models of unknown dimensionality [10]. In-stead of looking for specific finite-dimensional models,the idea is rather to define probability measures oninfinite-dimensional spaces and then infer the finite sub-set of active dimensions explaining the data. Over thepast years, there has been extensive work on construct-ing flexible Bayesian nonparametric models for varioustypes of graphical models, allowing complex hiddenstructures to be learned from data. For instance, [12]developed a model for infinite latent conditional ran-dom fields while [11] proposed an infinite mixture offully observable finite-dimensional Bayesian networks.In the case of time series, [3] developed the infinitehidden Markov random field model and [5] proposedan infinite dynamic Bayesian network with factoredhidden states. Another interesting model is the in-finite factorial dynamical model of [25] representingthe hidden dynamics of a system with infinitely manyindependent hidden Markov models.
The problem of learning networks containing hiddenstructures with Bayesian nonparametric methods hasalso received attention. The cascading Indian BuffetProcess (CIBP) of [1] is a Bayesian nonparametric priorover infinitely deep and infinitely broad layered net-work structures. However, the CIBP does not allowconnections from non-adjacent layers, yielding a re-stricted prior over infinite DAGs. The extended CIBP(ECIBP) is an extension of the previous model whichseeks to correct this limitation and support a largerset of DAG structures [4]. However, the ECIBP hassome drawbacks: the observable nodes are confined toa unique layer placed at the bottom of the network,which prevents learning the order of the nodes or haveobservable inputs. An immediate consequence of thisis the impossibility for an observable unit to be theparent of any hidden unit or any other observable unit,which considerably restricts the support of the priorover DAGs and make their application to deep learningvery problematic.
In the context of deep learning, structure learning is
arX
iv:2
001.
1065
7v1
[cs
.LG
] 2
9 Ja
n 20
20
often part of the optimization. Recently, [26] proposeda method that enforces the model to dynamically learnmore compact structures by imposing sparsity throughregularization. While sparsity is obviously an interest-ing property for large DAG-based models, their methodignores the epistemic uncertainty about the structure.Structure learning for probabilistic graphical modelscan also be applied in deep learning. For instance, [22]have demonstrated that deep network structures can belearned throught the use of Bayesian network structurelearning strategies. To our knowledge, no Bayesiannonparametric structure learning methods have beenapplied to deep learning models.
This paper introduces the Indian Chefs Process (ICP),a new Bayesian nonparametric prior for general DAG-based structure learning, which can equally be appliedto perform Bayesian inference in probabilistic graphicalmodels and deep learning. The proposed distributionhas a support containing all possible DAGs, admitshidden and observable units, is layerless and enforcessparsity. We present its construction in Section 2 anddescribe a learning method based on Markov ChainMonte Carlo in Section 3. In Section 4, we use the ICPas a prior in two Bayesian structure learning experi-ments: in the first, we compute the posterior distribu-tion on the structure and parameters of a deep gener-ative sigmoid networks and in the second we performstructure learning in convolutional neural networks.
2 Bayesian nonparametric directedacylic graphs
In this section, we construct the probability distributionover DAGs and orders by adopting the methodologyfollowed by [8]. We first define a distribution over finite-dimensional structures, while the final distribution isobtained by evaluating it as the structure size growsto infinity.
Let G = (V,Z) be a DAG where V = {1, . . . ,K} is theset of nodes and Z ∈ {0, 1}K×K is the adjacency matrix.We introduce an ordering θ on the nodes so that thedirection of an edge is determined by comparing theorder value of each node. A connection Zki = 1 isonly allowed when θk > θi, meaning that higher ordernodes are parents and lower order nodes are children.Notice that this constraint is stronger than acyclicitysince all (Z,θ) combinations respecting the order valueconstraint are guaranteed to be acyclic, but an acyclicgraph can violate the ordering constraint.
We assume that both the adjacency matrix Z and theordering θ are random variables and develop a Bayesianframework reflecting our uncertainty. Accordingly, weassign a popularity parameter πk and an order value
θk, called reputation, to every node k in G based onthe following model:
θk ∼ U(0, 1) (1)
πk | α, γ, φ,K ∼ Beta(αγK
+ φI(k ∈ O), α− αγ
K
)(2)
Zki | πk, θk, θi ∼ Bernoulli (πkI(θk > θi)) . (3)
Here, I denotes the indicator function, U(a, b) denotesthe uniform distribution on interval [a, b] and O ⊆ Vis the set of observed nodes. In this model, the pop-ularities reflected by π control the outgoing connec-tion probability of the nodes, while respecting the to-tal order imposed by θ. Moreover, the Beta priorparametrization in Eq. (2) is motivated by the BetaProcess construction of [21], where Eq. (1) becomes thebase distribution, and is convenient when evaluatingthe limit in Section 2.1. Also, α and γ correspond tothe usual parameters defining a Beta Process and thepurpose of the new parameter φ is to control the pop-ularity of the observable nodes and ensure a non-zeroconnection probability when required.
Under this model, the conditional probability of theadjacency matrix Z given the popularities π = {πk}Kk=1
and order values θ = {θk}Kk=1 is:
p(Z | π,θ) =K∏k=1
K∏i=1
p(Zki | πk, θk, θi) . (4)
The adjacency matrix Z may contain connections fornodes that are not of interest, i.e. nodes that are notancestors of any observable nodes. Formally, we defineA ⊆ V as the set of active nodes, which contains allobservable nodes O and the ones having a directedpath ending at an observable node.
When solely considering connections from A to A, i.e.the adjacency submatrix ZAA of the A-induced sub-graph of G, Eq. (4) simplifies to:
p(ZAA | π,↓,θ) =∏k∈A
πmk
k (1− πk)↓k−mk , (5)
where mk =∑i∈A Zki denotes the number of out-
going connections from node k to any active nodes,↓k=
∑j∈A I(θj < θk) denotes the number of active
nodes having an order value strictly lower than θkand ↓= {↓k}Kk=1. At this point, we marginalize outthe popularity vector π in Eq. (5) with respect to theprior, by using the conjugacy of the Beta and Binomial
distributions, and we get:
p(ZAA | α, γ,φ,↓,θ) =∏k∈H
[αγK ]mk [α− αγK ]↓k−mk
α↓k(6)
∏k∈O
[αγK + φ]mk [α− αγK ]↓k−mk
[α+ φ]↓k,
where xn = x(x+1) . . . (x+ n− 1) is the Pochhammersymbol denoting the rising factorial and H = A \O isthe set of active hidden nodes.
The set of active nodes A contains all observable nodesas well as their ancestors, which means there existsa part of the graph G that is disconnected from A.Let us denote by I = V \A the set of inactive nodes.Considering that the A-induced subgraph is effectivelymaximal, then this subgraph must be properly isolatedby some envelope of no-connections ZIA containingonly zeros. The joint probability of submatrices ZAAand ZIA is:
p(ZAA,ZIA | α, γ, φ,↓,θ) =
p(ZAA | α, γ, φ,↓,θ) ·∏k∈I
[α− αγK ]↓k
α↓k(7)
where the number of negative Bernoulli trials ↓k de-pends on θk itself and θA. Notice that since the sub-matrices ZAI and ZII contain uninteresting and unob-served binary events, they are trivially marginalizedout of p(Z).
One way to simplify Eq. (7) is to marginalize out theorder values θI of the inactive nodes with respect to(1). To do so, we first sort the active node ordersascendingly in vector θ↗A and augment it with theextrema θ↗0 = 0 and θ↗K++1 = 1, where we introduceK+ = |A| to denote the number of active nodes. Weslightly abuse notation here since these extrema donot refer to any nodes and are only used to computeinterval lengths. This provides us with all relevantinterval boundaries, including the absolute boundariesimplied by Eq. (1). We refer to the jth smallest valueof this vector as θ↗j . Based on the previous notation,the probability for an inactive node to lie between twoactive nodes is simply θ↗j+1 − θ
↗j . Using this notation,
we have the following marginal probability:
p(Z↗AA, ZIA,θ↗A | α, γ, φ) =
(K −D)K+−D
K+!
K+∑j=0
(θ↗j+1 − θ↗j )
[α(1− γK )]j
αj
K−
∏k∈H
[αγK ]mk [α− αγK ]↓k−mk
α↓k(8)
∏k∈O
[αγK + φ]mk [α− αγK ]↓k−mk
[α+ φ]↓k,
where K− = |I| denotes the number of inactive nodesand xn = x(x− 1) . . . (x−n+1) symbolizes the fallingfactorial. Due to the exchangeability of our model,the joint probability on both the adjacency matrixand active order values can cause problems regardingthe index k of the nodes. One way to simplify thisis to reorder the adjacency matrix according to θ↗A ,which we denote Z↗AA. By using this many-to-onetransformation, we obtain a probability distributionon an equivalence class of DAGs that is analog to thelof function used by [8]. The number of permutationmapping to this sorted representation is accounted forby the normalization constant (K −D)K
+−D(K+!)−1.
2.1 From Finite to Infinite DAGs
An elegant way to construct Bayesian nonparametricmodels is to consider the infinite limit of finite para-metric Bayesian models [20]. Following this idea, werevisit the model of Section 2 so that G now containsinfinitely many nodes. To this end, we evaluate thelimit as K → ∞ of Eq. (8), yielding the followingprobability distribution:
p(Z↗AA, ZIA,θ↗A |α, γ, φ,O) =
1
K+!exp
−αγ K+∑j=1
(θ↗j+1 − θ↗j )[ψ(α+ j)− ψ(α)
]∏k∈H
αγ(mk − 1)!
(α+ ↓k −mk)mk
∏k∈O
φmkα↓k−mk
[α+ φ]↓k, (9)
where ψ is the digamma function. Eq. (9) is the pro-posed marginal probability distribution on the jointspace of infinite DAGs and continuous orders. TheIndian Chefs Process1 is a stochastic process for gener-ating random DAGs from (9), making it an equivalentrepresentative of this distribution.
1Described in the supplementary material along withsome properties of the distribution. This content is notnecessary to reproduce the experimental results.
2.2 The Indian Chefs Process
Now that we have the marginal probability distribu-tion (9), we want to draw random active subgraphsfrom it. This section introduces the Indian chefs process(ICP), a stochastic process serving this purpose. In theICP metaphor, chefs draw inspiration from other chefs,based on their popularity and reputation, to createthe menu of their respective restaurant. This createsinspiration maps representable with directed acyclicgraphs. ICP defines two types of chefs: 1) star chefs(corresponding to observable variables) which are intro-duced iteratively and 2) regular chefs (correspondingto hidden variables) which appear only when anotherchef selects them as a source of inspiration.
The ICP starts with an empty inspiration map as itsinitial state. The infinitely many chefs can be thoughtof as lying on a unit interval of reputations. Everychef has a fraction of the infinitely many chefs abovehim and this fraction is determined by the chef’s ownreputation.
The general procedure at iteration t is to introducea new star chef, denoted i, within a fully specifiedmap of inspiration representing the connections of thepreviously processed chefs. The very first step is todraw a reputation value from θi ∼ U(0, 1) to determinethe position of the star chef in the reputation interval.Once chef i is added, sampling the new inspirationconnections is done in three steps.
Backward proposal Step one consists in proposingstar chef i as an inspiration to all the ↓i chefs havinga lower reputation than chef i. To this end, we canfirst sample the total number of inspiration connectionswith:
qi ∼ Binomial(↓i,
φ
α+ φ
), (10)
and then uniformly pick one of the(↓iqi
)possible config-
urations of inspiration connections.
Selecting existing chefs In step two, chef i consid-ers any already introduced chefs of higher reputation.The probability for candidate chef k to become aninspiration for i is:
Zki ∼ Bernoulli(
mk + φI(k ∈ star chefs)α+ ↓k −1 + φI(k ∈ star chefs)
),
(11)where ↓k includes the currently processed chef i.
Selecting new chefs The third step allows chef i toconsider completely new regular chefs as inspirationsin every single interval above i. The number of newregular chefs Knew
j to add in the jth reputation interval
above i follows probability distribution:
Knewj ∼ Poisson
((θ↗j+1 − θ
↗j )αγ
α+ ↓j −1
), (12)
where the new regular chefs are independently assigneda random reputation drawn from U(θ↗j , θ
↗j+1). The reg-
ular chefs introduced during this step will be processedone by one using step two and three. Once all newlyintroduced regular chefs have been processed, the nextiteration t+ 1 can begin with step one, a step reservedto star chefs only.
2.3 Some properties of the distribution
To better understand the effect of the hyperparameterson the graph properties, we performed an empiricalstudy of some relations between the hyperparameters,the expected number of active nodes E[K+|α, γ] andthe expected number of active edges E[E+|α, γ], whereE+ is the number of elements in ZAA. Figure 1(a)depicts level curves of E[K+|α, γ] for the case of only1 observable placed at θk = 0. The figure shows thatseveral combinations of α and γ leads to the sameexpected number of active nodes. Notice that fixingone hyperparameter, either α or γ, and selecting theexpected number of nodes, one can retrieve the secondhyperparameter that matches the relationship. We usedthis fact in the construction of Figure 1(b) where theunshown parameter γ could be calculated. In Figure1(b), we illustrate the effect of α on E[E+|α, γ] whichessentially shows that smaller values of α increase thegraph density.
When using Bayesian nonparametric models, we areactually assuming that the generative model of the datais infinite-dimensional and that only a finite subset ofthe parameters are involved in producing a finite set ofdata. The effective number of parameters explainingthe data corresponds to the model complexity and usu-ally scales logarithmically with respect to the samplesize. Unlike most Bayesian nonparametric models, theICP prior scales according to the number of observednodes added to the network. In Figure 1, we show howthe expected number of active hidden nodes increasesas function of the number of observable nodes.
2.4 Connection to the Indian Buffet Process
There exists a close connection between the IndianChefs Process (ICP) and the Indian Buffet Process(IBP). In fact, our model can be seen as a generalizationof the IBP. Firstly, all realizations of the IBP receive apositive probability under the ICP. Secondly, the two-parameter IBP [9] is recovered, at least conceptually,when altering the prior on order values (see Eq. (1)) so
that all observed nodes are set to reputation θ = 0 andall hidden nodes are set to reputation θ = 1. This way,connections are prohibited between hidden nodes andbetween observable nodes, while hidden-to-observableconnections are still permitted.
3 Structure Learning
In this section, we present a Markov Chain Monte Carlo(MCMC) algorithm approximating the exact ICP priordistribution with samples. We propose a reversiblejump MCMC algorithm producing random walks onEq. (9) [7]. This algorithm works in three phases: thefirst resamples graph connections without adding orremoving any nodes, the second phase is a birth-deathprocess on nodes and the third one only involves theorder.
The algorithm itself uses the notion of singleton andorphan nodes. A node is a singleton when it onlyhas a unique active child. Thus, removing its uniqueconnection would disconnect the node from the activesubgraph. Moreover, a node is said to be an orphan ifit does not have any parents.
Within model moves on adjacency matrix: Webegin by uniformly selecting a node i from the activesubgraph. Here, the set of parents to consider fori comprises all non-singleton active nodes having anorder value greater than θi. This set includes bothcurrent parents and candidate parents. Then, for eachparent k, we Gibbs sample the connections using thefollowing conditional probability:
p(Z↗ki = 1|Z↗¬kiAA ,θA) =
m¬ik + φI(k ∈ O)
α+ ↓k −1 + φI(k ∈ O),
(13)where m¬ik is the number of outgoing connections ofnode k excluding connections going to node i andZ↗¬kiAA has element ki removed. Also, all connections
not respecting the order are prohibited and thereforehave an occurrence probability of 0, along with (trans-dimentional) singleton parent moves.
Trans-dimensional moves on adjacency matrix:We begin with a random uniform selection of nodei in the active subgraph and, with equal probability,propose either a birth or a death move.
In the birth case, we activate node k by connecting it tonode i. Its order θk is determined by uniformly selectingan insertion interval above θi. Assuming node i is alsothe ith element in θ↗A , we have ↑i= K+− i+1 possibleintervals, including zero-length intervals. Let us assumethat j and j + 1 are the two nodes between which k isto be inserted. Then, we obtain the candidate order
value of the new node by sampling θk ∼ U(θ↗j , θ↗j+1).
The Metropolis-Hastings acceptance ratio here is:
abirth = min
{1,p(Z ′
↗A′A′ , Z
′I′A′ ,θ
′↗A′ |α, γ, φ,O)
p(Z↗AA, ZIA,θ↗A |α, γ, φ,O)
·(θ↗j+1 − θ
↗j )(↑i +1)K+
K∗i + 1
},
(14)
where K∗i is the number of singleton-orphan parentsof i and ↑i=
∑j∈A I(θj > θi) is the number of active
nodes above i.
In the death case, we uniformly select one of the K∗isingleton-orphan parents of i if K∗i > 0 and simply donothing in case there exists no such node. Let k be theparent to disconnect and consequently deactivate. TheMetropolis-Hastings acceptance ratio for this move is:
adeath = min
{1,p(Z ′
↗A′A′ , Z
′I′A′ ,θ
′↗A′ |α, γ, φ,O)
p(Z↗AA, ZIA,θ↗A |α, γ, φ,O)
· K∗i
(θ↗j+1 − θ↗j )(K+ − 1) ↑i
}.
(15)
If accepted, node k is removed from the active sub-graph.
Moves on order values: We re-sample the ordervalue of randomly picked node i. This operation isdone by finding the lowest order valued parent of ialong with its highest order valued children, whichwe respectively denote l and h. Next, the candidateorder value is sampled according to θi ∼ U(θl, θh) andaccepted with Metropolis-Hasting acceptance ratio:
aorder = min
{1,p(Z↗AA, ZIA,θ
′↗A |α, γ, φ,O)
p(Z↗AA, ZIA,θ↗A |α, γ, φ,O)
},
(16)which proposes a new total order θ respecting thepartial order imposed by the rest of the DAG.
4 Experiments
The ICP distribution (9) can be used as a prior tolearn the structure of any DAG-based model involv-ing hidden units. In particular, one can introduce apriori knowledge about the structure by fixing the or-der values of some observed units. Feedforward neuralnetworks, for instance, can be modelled by imposingθk = 1 for all input units and θk = 0 for the outputunits. On the other hand, generative models can bedesigned by placing all observed units at θk = 0, pre-venting interconnections between them and forcing theabove generative units to explain the data. In section
0 1 2 3 4 51
2
3
4
5
6
7
8
γα
K+ = 2
K+ = 4
K+ = 5
K+ = 3
(a)
0 1 2 3 4 50
5
10
15
expe
cted
num
ber
of c
onne
ctio
ns
α
K+ = 2
K+ = 5
K+ = 4
K+ = 3
(b)
2 4 6 8 100.5
1
1.5
2
2.5
3
number of observed variables
num
ber
of h
idde
n va
riabl
es
α = 0.5
α = 1.0
α = 1.5
α = 2.0
(c)
Figure 1: Empirical study of hyperpameters. Figure (a) shows the expected number of active nodes as a functionof α and γ. Figure (b) shows that once we know the expected K+ from α and γ, we can find the expected numberof connections. Figure (c) shows the influence of α (with γ = 1) on the complexity (number of hidden nodes) asfunction of the number of observable nodes.
4.1, we use the ICP as a prior to learn the structuresof a generative neural network by approximating thefull posterior for 9 datasets. In section 4.2, we use theICP to learn the structure of a convolutional neuralnetwork (CNN) in a Bayesian learning framework.
4.1 Bayesian nonparametric generativesigmoid network
The network used in this section is the Nonlinear Gaus-sian Belief Network (NLGBN) [6], which is basicallya generative sigmoid network. In this model, the out-put of a unit ui depends on a weighted sum of itsparents, where Wki represents the weight of parentunit uk, Zki indicates whether uk is a parent of ui andbi is a bias. The weighted sum is then corrupted bya zero mean Gaussian noise of precision ρi, so thatai ∼ N (bi +
∑k ZkiWkiuk, 1/ρi). The noisy preactiva-
tion ai is then passed through a sigmoid nonlinearity,producing the output value ui. It turns out that thedensity function of this random output ui can be rep-resented in closed-form, a property used to form thelikelihood function given the data. An ICP prior isplaced on the structure represented by Z along with pri-ors γ ∼ Gamma(0.5, 0.5), 1/α ∼ Gamma(0.5, 0.5) andφ ∼ Gamma(0.5, 0.5). To complete the prior on param-eters, we specify ρk ∼ Gamma(0.5, 0.5), bk ∼ N (0, 1)and Wki ∼ N (0, 1).
The inference is done with MCMC where structureoperator are given in 3 and we refer to [1] for the pa-rameter and activation operators. The Markov chainexplores the space of structures by creating and de-stroying edge and nodes, which means that posteriorsamples are of varying size and shape, while remaininginfinitely layered due to θk ∈ [0, 1]. We also simulatethe random activations uk and add them into the chainstate.
This experiment aims at reproducing the generativeprocess of synthetic data sources. In the learning phase,we simulate the posterior distribution conditioned on
2000 training points. Fantasy data from the posteriorare generated by first sampling a model from set ofposterior network samples and then one point is gen-erated from the selected model. Figure 2 shows 2000test samples from the true distribution along with thesamples generated from the posterior accounting forthe model uncertainty.
Next, we compare the ICP (with observables at θk = 0)against other Bayesian nonparametric approaches: TheCascading Indian Buffet Process [1] and the ExtendedCIBP [4]. The inference for these models was donewith an MCMC algorithm similar to the one used forthe ICP and we used similar priors for the parametersto ensure a fair comparison. The comparison metricused in this experiments is the Hellinger distance (HD),a function quantifying the similarity between two prob-ability densities. Table 1 shows the HD’s betweenfantasy datasets generated and groundtruth.
4.2 Bayesian nonparametric convolutionalneural networks
So far, we introduced the ICP as a prior on the space ofdirected acyclic graphs. In this section we will use thisformalism in order to construct a prior on the space ofconvolutional neural architectures. The fundamentalbuilding blocks of (2D) convolutional networks aretensors T whose entries encode the presence of localfeatures in the input image. A convolutional neuralnetwork can be described as a sequence of convolutionoperators acting on these tensors followed by entry-wisenonlinearity f .
In our nonparametric model, a convolutional network isconstructed from a directed acyclic graph. Each nodeof the graph represents a tensor T (i). The entries ofthis tensor are given by the following:
T (i) = ReLu
∑k∈Parents(i)
W (ki) ? T (k)
. (17)
−2 −1 0 1 2−1.5
−1
−0.5
0
0.5
1
1.5
(a) groundtruth−2 −1 0 1 2
−1.5
−1
−0.5
0
0.5
1
1.5
(b) fantasy−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
(c) groundtruth−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
(d) fantasy−2 −1 0 1 2
−1.5
−1
−0.5
0
0.5
1
1.5
(e) groundtruth−2 −1 0 1 2
−1.5
−1
−0.5
0
0.5
1
1.5
(f) fantasy
Figure 2: Resulting fantasy data generated from the posterior on 3 toy datasets.
Table 1: Hellinger distance between the fantasy data from posterior models and the test set. Dimensionality ofthe data is in parenthesis. The baseline shows the distance between the training and test sets, representing thebest achievable distance since the two come from the true source.
Data set Ring (2) Two Moons (2) Pinwheel (2) Geyser (2) Iris (4) Yeast (8) Abalone (9) Cloud (10) Wine (12)
ICP 0.0402 0.0342 0.0547 0.0734 0.2666 0.3817 0.1379 0.1495 0.3629CIBP 0.0493 0.0469 0.0692 0.1246 0.2667 0.4056 0.1502 0.1713 0.4079ECIBP 0.0419 0.0450 0.0685 0.1171 0.2632 0.3840 0.1470 0.1501 0.3855
Baseline 0.0312 0.0138 0.0436 0.0234 0.1930 0.3059 0.1079 0.1299 0.3387
whereW (ki) is a tensor of convolutional weights and ? isthe discrete convolution operator. In most hand-craftedarchitectures, the spatial dimensions of the tensor arecourse-grained as the depth increases while the numberof channels (each representing a local feature of theinput) increases. In the ICP, the depth of a node i isrepresented by its popularity θi. In order to encodethe change of shape in the nonparametric prior, we setthe number of channels to be a function of θ:
Nc(θ) = 2bNbins(1−θ))c +N0 , (18)
where Nbins is the number of different possible tensorshapes and N0 is the number of channels of the lowestlayers. Similarly, the number of pixels is given by:
Np(θ) = 2−bNbins(1−θ))cM , (19)
where M is the number of pixels in the original image.The shape of the weight tensors W (ki) is determinedby the shape of parent and child tensor.
In a classification problem, the nonparametric convolu-tional network is connected to the data through twoobserved nodes. The input node X stores the inputimages and we set θX = 1. On the other hand, for theoutput node we set θY = 0, and have it receive inputthrough fully connected layers:
Y = SoftMax
∑k∈Parents(Y )
∑a,b,c
V (k)a T
(k)abc
, (20)
where V (k) is a tensor of weights. During samplingof a connection Zki in the directed acyclic graph, thelog acceptance ratio needs to be summed to the logmarginal-likelihood ratio:
log alk = log p(y | DAG,X)−log p(y | DAGproposal, X) .(21)
In this paper, we use a point estimate of the log modelevidence:
p(y | DAG, x) ≈ p(y | DAG, {W (ki)}, x) , (22)
where W (jk) are the parameters of the net-work optimized using Adam (alpha=0.1, beta1=0.9,beta2=0.999, eps=1e-08, eta=1.0) [14].
We performed architecture sampling on the MNISTdataset. For computational reasons, we restricted thedataset to the first three classes (1,2 and 3). We sam-pled the DAGs using the MCMC sampler introducedin section 3 with prior parameters α = 1, γ = 20 andφ = 5. For each sampled DAG, we trained the inducedconvolutional architecture until convergence (540 iter-ations, batch size equal to 100). The number of binsin the partition of the range of the popularity was fiveand the number of channels of the first convolutionallayer was four. We ran 15 independent Markov chainsin order to sample from multiple modes. Each chainconsisted of 300 accepted samples. After sampling, allchains were merged, resulting in a total of 4500 sampledarchitectures.
Figure 3A shows accuracy and descriptive statistics ofthe sampled convolutional architectures. In all thesestatistics, we only considered nodes that receive inputfrom the input node (directly or indirectly) as the re-maining nodes do not contribute to the forward passof the network. The network width is quantified asthe total number of directed paths between input andoutput nodes, while depth is quantified as the maxi-mal directed path length. The sampler reaches a widerange of different architectures, whose number of layersrange from three to fifteen, and whose average degreerange from one to four. Some examples of architecturesare shown in Figure 3C. Interestingly, the correlationbetween the number of nodes, degree, width and depth
0.6 1.00.8 0 155 10 41 2 7.52.5 5.0 0 2010
0.6 1.00.8 0 155 10 41 2 7.52.5 5.0
0.6 1.00.8 0 155 10 41 2
0.6 1.00.8 0 155 10
0.6 1.00.8
A B
0.6 1.00.8 0.6 1.00.8 0.6 1.00.8 0.6 1.00.8
C
ReputationMean Std Min Max
input
output
123
47
56
output
input
12
input
output
1
Accu
racy
#Nod
esAv
g. D
egre
eD
epth
Wid
th
WidthDepthAvg. Degree#NodesAccuracy
Figure 3: Statistics of the sampled convolutional ar-chitectures. A) Histograms and bivariate density plotsof test set accuracy, number of nodes, average degreewidth and depth. The three colored crosses denotethe statistics of the three visualized networks. B) His-togram of the mean, standard deviation, min and maxof the popularity values.
and accuracy is very low. Most likely, this is due tothe simple nature of the MNIST task. The ensembleaccuracy (0.95), obtained by averaging the label prob-abilities over all samples, is higher that the averageaccuracy (0.91), but lower that the maximum accu-racy (0.99). Figure 3B shows the histograms of mean,standard deviation, minimum and maximum of thepopularity values in the networks.
5 Conclusion and Future Work
This paper introduces the Indian Chefs Process (ICP),a Bayesian nonparametric distribution on the jointspace of infinite directed acyclic graphs and orders.The model allows for a novel way of learning the struc-ture of deep learning models. As a proof-of-concept,we have demonstrated how the ICP can be used tolearn the architecture of convolutional deep networkstrained on the MNIST data set. However. for morerealistic applications, several efficiency improvementsare required. Firstly, the inference procedure over themodel parameters could be performed using stochasticHamiltonian MCMC [16]. This removes the need tofully train the network for every sampled DAG. An-other possible improvement is to add deep learningspecific sampling moves. For example, it is possible toinclude a increase-depth move that replaces a connec-tion with a path comprised by two connections and a
latent node. Future applications of the ICP may ex-tend beyond deep learning architectures. For example,the ICP may serve as a basis for nonparametric causalinference, where a DAG structure is learned when theexact number of relevant variables is not known a pri-ori, or when certain relevant input variables are notobserved [19].
References
[1] R. P. Adams, H. M. Wallach, and Z. Ghahramani.Learning the structure of deep sparse graphicalmodels. In Proceedings of the 13th InternationalConference on Artificial Intelligence and Statistics,2010.
[2] S. Banerjee and S. Ghosal. Bayesian structurelearning in graphical models. Journal of Multi-variate Analysis, 136:147–162, 2015.
[3] S. P. Chatzis and G. Tsechpenakis. The infinitehidden Markov random field model. Neural Net-works, IEEE Transactions on, 21(6):1004–1014,2010.
[4] P. Dallaire, P. Giguere, and B. Chaib-draa. Learn-ing the structure of probabilistic graphical modelswith an extended cascading Indian buffet process.In Twenty-Eighth AAAI Conference on ArtificialIntelligence, 2014.
[5] F. Doshi, D. Wingate, J. Tenenbaum, and N. Roy.Infinite dynamic Bayesian networks. In Proceed-ings of the 28th International Conference on Ma-chine Learning (ICML-11), pages 913–920, 2011.
[6] B. J. Frey. Continuous sigmoidal belief networkstrained using slice sampling. In Advances in NeuralInformation Processing Systems 9, pages 452–458.MIT Press, 1997.
[7] P. J. Green and D. I. Hastie. Reversible jumpMCMC. Genetics, 155(3):1391–1403, 2009.
[8] T. Griffiths and Z. Ghahramani. Infinite latentfeature models and the indian buffet process. Ad-vances in neural information processing systems,18:475, 2006.
[9] T. L. Griffiths and Z. Ghahramani. The indianbuffet process: An introduction and review. TheJournal of Machine Learning Research, 12:1185–1224, 2011.
[10] N. Hjort, C. Holmes, P. Muller, and S. Walker. Aninvitation to bayesian nonparametrics. BayesianNonparametrics, 28:1, 2009.
[11] K. Ickstadt, B. Bornkamp, M. Grzegorczyk,J. Wieczorek, M. R. Sheriff, H. E. Grecco, andE. Zamir. Nonparametric Bayesian Networks. Ox-ford University Press, 2010.
[12] Y. Jiang and A. Saxena. Infinite latent conditionalrandom fields. In Proceedings of the IEEE Interna-tional Conference on Computer Vision Workshops,pages 262–266, 2013.
[13] M. I. Jordan. Learning in graphical models, vol-ume 89. Springer Science & Business Media, 1998.
[14] D. P. Kingma and J. Ba. Adam: A methodfor stochastic optimization. arXiv preprintarXiv:1412.6980, 2014.
[15] T.-Y. Kwok and D.-Y. Yeung. Constructive algo-rithms for structure learning in feedforward neuralnetworks for regression problems. IEEE transac-tions on neural networks, 8(3):630–645, 1997.
[16] Y.-A. Ma, T. Chen, and E. Fox. A completerecipe for stochastic gradient MCMC. In Advancesin Neural Information Processing Systems, pages2917–2925, 2015.
[17] V. Mansinghka, C. Kemp, T. Griffiths, andJ. Tenenbaum. Structured priors for structurelearning. arXiv preprint arXiv:1206.6852, 2012.
[18] A. Mohammadi, E. C. Wit, et al. Bayesian struc-ture learning in sparse gaussian graphical models.Bayesian Analysis, 10(1):109–138, 2015.
[19] K. Mohan, J. Pearl, and J. Tian. Graphical modelsfor inference with missing data. In C. J. C. Burges,L. Bottou, M. Welling, Z. Ghahramani, and K. Q.Weinberger, editors, Advances in Neural Infor-mation Processing Systems 26, pages 1277–1285.Curran Associates, Inc., 2013.
[20] P. Orbanz and Y. W. Teh. Bayesian nonparametricmodels. In Encyclopedia of Machine Learning,pages 81–89. Springer, 2010.
[21] J. Paisley. Machine Learning with Dirichlet andBeta Process Priors: Theory and Applications.PhD thesis, Duke University, Durham, North Car-olina, 2010.
[22] R. Y. Rohekar, S. Nisimov, Y. Gurwicz, G. Koren,and G. Novik. Constructing deep neural networksby bayesian network structure learning. In Ad-vances in Neural Information Processing Systems,pages 3047–3058, 2018.
[23] M. Schmidt, A. Niculescu-Mizil, K. Murphy,et al. Learning graphical model structure usingl1-regularization paths. In AAAI, volume 7, pages1278–1283, 2007.
[24] D. G. R. Tervo, J. B. Tenenbaum, and S. J. Gersh-man. Toward the neural implementation of struc-ture learning. Current opinion in neurobiology,37:99–105, 2016.
[25] I. Valera, F. Ruiz, L. Svensson, and F. Perez-Cruz.Infinite factorial dynamical model. In Advancesin Neural Information Processing Systems, pages1657–1665, 2015.
[26] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li.Learning structured sparsity in deep neural net-works. In Advances in neural information process-ing systems, pages 2074–2082, 2016.