for graphs deep learning · other works: gin (xu et al., 2019) graph isomorphism network as...
TRANSCRIPT
![Page 1: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/1.jpg)
Deep Learning for GraphsTrends & Open Questions
Federico Errica
![Page 2: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/2.jpg)
Outline
● What are graphs?
● Why ML on graphs?
● How?
● Some models
● Open questions (Projects / Theses)
2
![Page 3: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/3.jpg)
The What
3
● Vertex u → Entity○ Continuous / Discrete attributes
● Edge (u,v) → Relationship○ Continuous / Discrete attributes ( {0,1} still most common )○ Directed / Undirected
● Generalizations: Multigraphs & Hypergraphs
![Page 4: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/4.jpg)
The What (cont.)
4
● Representation Learning on graphs○ Vertex & Graph embeddings
● Supervised○ Vertex/Graph classification/regression
● Unsupervised○ Link Prediction○ Clustering
![Page 5: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/5.jpg)
A visual example
5
Leow, Yao Yang, Thomas Laurent, and Xavier Bresson. GraphTSNE: A Visualization Technique for Graph-Structured Data. ICLR Workshop (2019).https://leowyy.github.io/graphtsne
![Page 6: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/6.jpg)
The Why
6
RnxmHave → fun → with → Machine → Learning
Flat Sequences
Trees
Graphs
![Page 7: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/7.jpg)
The Why (cont.)
7
● Handle cyclic structures ○ No recursion!
● Variable size
● Variable shape
● Efficiency
● No more feature engineering○ i.e. kernel methods
Me The feature engineering guy
![Page 8: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/8.jpg)
The How (in a nutshell)
● Neighborhood Aggregation to the rescue● Use layering to spread context between vertices
● How can we aggregate neighbors?○ Permutation-invariant functions over (multi)-sets
● How many layers do we need?8
![Page 9: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/9.jpg)
Resemblance to CNNs● Convolution as neighborhood aggregation
○ On regular grids
● Layers increases the local receptive field of each vertex
9
![Page 10: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/10.jpg)
What we do: NN4G (Micheli, 2009)
● Constructive approach○ Cascade Correlation
● Aggregation function○ Sum
● The first GNN!
10
![Page 11: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/11.jpg)
What we do: GraphESN (Gallicchio & Micheli, 2010)
● Does not require training but for the output layer
● Let the Reservoir reach convergence
● Train a linear readout
11
![Page 12: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/12.jpg)
What we do: CGMM (Bacciu, Errica & Micheli, 2018)● A deep stack of probabilistic layers
● Unsupervised constructive approach
● Switching Parent approximation○ Borrowed from
Hidden Tree Markov Models ● It works well
○ State-of-the-art accuracycompared to GNNs
● CGMM exploits layering
12
ℓ
![Page 13: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/13.jpg)
DiffPool (Ying et al., 2018)
● Differentiable Pooling technique
13
![Page 14: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/14.jpg)
On Aggregation Functions
14
From “On the Limitations of Representing Functions on Sets”, Wagstaff et al., 2019
Neighborhood aggregation!
M neighbors → M neurons!
![Page 15: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/15.jpg)
Other works: GIN (Xu et al., 2019)
● Graph Isomorphism Network○ As powerful as 1-dim WL test of graph isomorphism○ But it can exploit continuous attributes
● Not able to distinguish k-regular graphs, k > 1
● Astonishing results (under revision by me and Marco..)
● Very nice theorems for aggregation on multi-sets○ Similar to Wagstaff et al., 2019
● We can build other GNNs from here (Thesis?)
15
![Page 16: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/16.jpg)
Open Questions (nice & challenging)
16
● Automatize hyper-parameter selections
● New pooling strategies
● CGMM extensions
● Design a new GNN
● Implement a Graph Neural Network (using Pytorch Geometrics) ○ Choose from a list of possible models
![Page 17: for Graphs Deep Learning · Other works: GIN (Xu et al., 2019) Graph Isomorphism Network As powerful as 1-dim WL test of graph isomorphism But it can exploit continuous attributes](https://reader033.vdocuments.net/reader033/viewer/2022042408/5f23655282f33144e45bc58c/html5/thumbnails/17.jpg)
Thank you!
17
You can reach out to me via:
Email: [email protected] (anytime)
Office: Room 328, Department of Computer Science (late September)
Website: http://pages.di.unipi.it/errica/
Powered by CIML